Release Notes for Feb 13, 2015
QDS now offers Spark as a service and allows SparkCommand to be run via QDS. QDS also offers notebook interface for Spark. Every new account would have a pre-created Spark cluster. One can create additional clusters for Spark via Control Panel. QDS supports Spark SQL, Scala, Python and Shell commands to be run against Spark clusters. Spark SQL can use hive metastore for table metadata. QDS also supports autoscaling for Spark clusters.
Pig & HCatalog integration
QDS allows using Pig with HCatalog now. This allows using Hive Table definition with Pig.
QDS now supports RedShift command and allows queries against RedShift.
Roles & Groups
Roles and Groups support is available in Analyze v2
QDS now supports multiple clusters and fail scheduler for GCE. Node bootstrap can also be edited via UI. Support for more types of instances has been added.
All other minor changes and bug fixes are listed below
* HAD-304: Hadoop FairScheduler UI has been enhanced to show jobs that are queued for initialization. In addition, in the advanced mode, users can force Job Initialization
* HAD-297: Name Specified in Qubole Command is now used as part of the Hadoop Job Name for Shell Commands.
* HAD-335: Improvements in Hadoop JobTracker to speed up job scheduling by reducing contention on global JT lock
* HAD-333: Failed Reducers would leak files containing intermediate spilled data - gradually filling up disk drives. This leak has been plugged.
* HAD-236/331: mapreduce.user.classpath.first option is now supported in Qubole Hadoop.
* HAD-182: S3 Objects with a trailing ‘/’ are now handled correctly by Hadoop S3 Native FileSystem.
* HAD-261: Map-Reduce should filter directories beginning with “_” character
* HAD-292: Hadoop Job may be limited in parallelism because of max mapper limits in it’s FairScheduler pool. This is now taken into account by the Hadoop auto-scaling logic.
* HAD-298: Uninitialized Hadoop jobs (in PREP state) are now terminated automatically when the jobclient is killed.
* QBOL-3739: Fixes to allow FairScheduler configuration to be pushed correctly from UI into running clusters.
* QBOL-3670: Limit number of EBS volumes and number that can be attached to a cluster
* Allow longer cluster bringup timeouts when using encrypted ephemeral storage
* QBOL-3666: Fix S3 Bytes read metrics when S3: URIs are used.
* HAD-242: Fix timeout errors when Shell Commands are submitted to newly created clusters with encrypted ephemeral drives
* PRES-312: Introducing array_agg function
* PRES-310: CTAS and Inserts in Presto can now write array type columns
* PRES-315: Presto’s Hive connector now compacts tuple domains of ranges only if the ranges’ count is smaller than certain size. This improves performance of IN operator with list size less than 100 (configurable).
* UI-1121: Cluster disallow termination is now available for presto cluster
* PRES-328: Handle bad ORC statistics
* PRES-313: Disable SSL when reading from S3
* PRES-308: Fix inserts into partitioned tables
* PRES-85: Allow dropping of hive tables from Presto
* QHIVE-297: Create View statement should not bring up a cluster
* QHIVE-559: Recover partitions should ignore directories with null status
* UI-1130: V2 CPL does not load in many customers accounts
* UI-1155: Fixed redshift query text
* UI-1139 - V2 CPL credential warning fix
* UI-1133 Fix Workflows in scheduler when DB Export is used
* QBOL-3698: Qubole Logs and Results can now be written out with S3 Server Side Encryption
* UI-1044: Enable Analyze V2 for all users
* UI-1072: Fixed multiple user selection in history search
* UI-1097 - replace defloc with trailing '/' in account setting
* UI-1027: Fix Analyze V2 layout on mac chrome 39.0.2171.95
* UI-1034 Fixed cache invalidation as it wasn't removing users from a group properly
* UI-1078 Reload the users list if groups have changed
* UI-1093 Valid error message is displayed for duplicate role names
* UI-925: Remember cluster selection across command types
* UI-1076 - V2 Control Panel: Customer ssh key not populated
* UI-1054 : Trigger email alerts if command hits configurable timeout.
* UI-1122: Overview dashboard QCUH calculations are now more accurate
* UI-1117 Block placement policy should default to True anytime Spot Instance purchasing options are selected
* UI-987: fix for drag and drop of columns brings database name
* UI-1021: fix for history filter reset button not working
* QBOL-3614: Per user AWS credentials support for Shell Commands.
* QBOL-3655: Trim trailing slash from default storage location