Qubole Release Notes 27-July-2016

Release Version: 35.0.0

For details of what has changed in this version of QDS, see What is New and List of Changes and Bug Fixes in this Release.

What is New

Support for new EBS Volume Types and Sizes

QDS clusters now support additional EBS volume types. You can now use Throughput Optimized HDD (st1) and Cold HDD (sc1) volumes. This is in addition to General Purpose SSD (gp2) and Magnetic (standard) volume types.

QDS now supports EBS volume sizes of up to 16 TB (previously, the limit per volume was 500
GB). QDS uses the ext4 file system by default for volume disks larger than 500 GB to improve
speed in formatting during cluster bootup. For volume disks less than 500 GB, QDS still defaults
to using ext3 file system. You can opt in to using ext4 for volumes smaller than 500 GB by
emailing [e-mail].

QDS UI Enhancements

These are enhancements in the QDS UI:

• Qubole now supports keyword and schema autocomplete in Hive, Presto, and Spark SQL offerings. Use the Ctrl+Space key combination in the online editors to get a list of suggestions. It takes care of the aliases given to tables and queries using tables across different databases.
• Qubole now supports creating either a new IAM Keys or IAM Roles account from within the Control Panel.
• Two new buttons, refresh Hive Schema and refresh Hive Table, have been added to the Data Import command type in the Analyze page’s query composer.
• Syntax help text for the Override Presto Configuration Variables and Override Spark Configuration Variables fields in the Control Panel > Cluster Configuration page.

Angular Interpreter in Notebooks

Qubole has introduced the Angular interpreter in Notebooks. It can be used in creating interpreter variables using customized JS libraries to render customized graphs and visualizations. See Angular Interpreter for more information.

List of Changes and Bug Fixes in this Release

AIRFLOW

Change
QBOL-5455: Fixed broken links, pages and Create/Update/Delete operations on the Airflow Webserver Dashboard

AWS CLUSTER MANAGEMENT

New   ACM252: Added support for new AWS EBS disk types: st1 and sc1 to be used in clusters. Please note, all the cluster disks are formatted using ext4 filesystem if st1 and sc1 volumes are used since they are at least 500 GB in size.

The EBS disk size limit has been increased to 16TB from 500 GB.

Contact [e-mail] to enable the ext4 filesystem for volume sizes less
than 500 GB.

Change   ACM-291: Improvements to avoid formatting ebs disks by using pre-formatted EBS snapshots.

Change   ACM-357: DataDog API and APP token now implicitly enables Ganglia in the cluster. Earlier it was necessary to explicitly enable Ganglia monitoring for a cluster to get metrics into DataDog. Although the Ganglia monitoring is enabled, its link may not be visible in the cluster’s UI resources list.

Change  Reduced cluster start time for i2, r3, c3, and m3 family of instances.

HADOOP

Fix   HAD-541: Files older than 3 days in the S3 buffer directory (identified by the fs.s3.buffer.dir option) will be removed to clear up storage capacity.

HADOOP-2

Fix   HADTWO-428: Added s3distcp support for IAM roles
Fix   HADTWO-529: clouddistcp will now work with spaces in source or destination paths.
Change   HADTWO-508: Minimum cluster size of cluster is made pushable. With this change, you can change the minimum cluster size of a running cluster.
Change   HADTWO-522: S3 listing taking a long time for Hadoop 2. This issue has been fixed.

HBASE

New   HBAS-156: Added monit services for HBase master, zookeeper and regionserver processes. The monit daemon will try to restart these processes automatically in case they are terminated due to some reason.
Fix   HBAS-149: The pid files for master, regionserver, and zookeeper process in HBase are now stored in /media/ephemeral0/pids/hbase directory. Also, the bug in restarting the daemons for the three processes due to pid file-name mismatch is resolved.
Fix   HBAS-167: Added auto-purging properties for periodic removal of zookeeper snapshots and transaction logs in HBase. Only last three snapshots and transaction logs are saved and the cleanup runs every hour. This will help to keep in check the storage used by zookeeper logs with a fallback of last three snapshots for recovery.
Change   HBAS-175: Increased the timeout for HBase table snapshots to ensure they complete for large tables. Previously, the timeout was 60 seconds (1 min) for both master and regionserver. Now, the timeout is 900 seconds (15 mins) for master and 600 seconds (10 mins) for regionserver.

HIVE 0.13

Fix   HIVE-1449: Fixed an issue which caused CTAS statements to throw errors, when the table is stored as ORC File format and select clause has NULL/VOID type column. Related to OS JIRA - HIVE-11217
Fix   HIVE-1478: Fixed an issue which caused queries of type group by...having with table alias to fail with ClassCastException. OS JIRA - HIVE-8398
Fix   HIVE-1487: Fix to handle spaces in the partition path when bucketing is enforced with dynamic partitioning.
Fix   HIVE-1491: Increasing the hive.exec.reducers.bytes.per.reducer to 1G to improve reducer estimation
Change   ROLL-22: Change in Hive to allow deletion of jar by default.

HIVE 1.2

Fix   HIVE-1291: Fixed an issue which caused NPE in DynamicPartFileRecordWriterContainer on using null partition keys.
Related OS JIRA - HIVE-11470
Fix   HIVE-1449: Fixed an issue which caused CTAS statements throw errors, when the table is stored as ORC File format and select clause has NULL/VOID type column. Related OS JIRA - HIVE-11217
Fix   HIVE-1476: Fixed an issue which caused drop table commands for managed table to succeed even in case of failure in actual data deletion.
Fix   HIVE-1478: Fixed an issue which caused queries of type group by..having with table alias to fail with ClassCastException. OS JIRA - HIVE-8398
Fix   HIVE-1487: Fix to handle spaces in the partition path when bucketing is enforced with dynamic partitioning.
Fix   HIVE-1491: Increasing the hive.exec.reducers.bytes.per.reducer to 1G to improve reducer estimation
Fix   HIVE-1495: Changed log level in Avro serde package to reduce the frequency of logs printing schema details.
Change   ROLL-22: Change in Hive to allow deletion of jar by default

HBASE

New   HBAS-156: Added monit services for HBase master, zookeeper and regionserver processes. The monit daemon will try to restart these processes automatically in case they are terminated due to some reason.
Fix   HBAS-149: The pid files for master, regionserver, and zookeeper process in HBase are now stored in /media/ephemeral0/pids/hbase directory. Also, the bug in restarting the daemons for the three processes due to pid file-name mismatch is resolved.
Fix   HBAS-167: Added auto-purging properties for periodic removal of zookeeper snapshots and transaction logs in HBase. Only last three snapshots and transaction logs are saved and the cleanup runs every hour. This will help to keep in check the storage used by zookeeper logs with a fallback of last three snapshots for recovery.
Change   HBAS-175: Increased the timeout for HBase table snapshots to ensure they complete for large tables. Previously, the timeout was 60 seconds (1 min) for both master and regionserver. Now, the timeout is 900 seconds (15 mins) for master and 600 seconds (10 mins) for regionserver.

PRESTO

Fix   PRES-663: Fixed a bug in block caching returned EOFException wrongly when the server is under a heavy load.
Fix   PRES-675: This fixes the issue of queries failing with the
Error committing write to Hive error
Fix   PRES-689: Strict logging in RubiX's BookKeeper server to prevent frequent instance freezes

QDS

New   UI-3443: Qubole now supports creating either a new IAM Keys or IAM Roles account from within the Control Panel.

New   UI-3734: Introducing the Format button in the online HiveQL editor to make a query more readable and investigate the underlying structure of sub queries.
Fix   UI-3675: Scheduled job instances that have Hive or S3 dependency now display this information in Scheduler and Analyze UI.
Fix   UI-3948: The issue of Persistent Security Groups being disabled if there is no account update permission has been fixed.
Fix   UI-4010: The job-level DAG visualisation is now a working link and takes you to the job details page.
Change   MW-17: Support to throttle command submission to ensure fair scheduling. Throttling is based on global waiting command limit and per account waiting command limit
Change   UI-3533: Two new buttons, refresh Hive Schema and refresh Hive Table, have been added to the Data Import command type in the Analyze page’s query composer.
Change   UI-3770: Qubole now supports keyword and schema autocomplete in Hive, Presto, and Spark SQL offerings. Use the Ctrl+Space key combination in the online editors to get a list of suggestions. It takes care of the aliases given to tables and queries using tables across different databases.
Change   UI-3896: Added syntax help text for the Override Spark
Configuration Variables field in the Control Panel > Cluster Configuration page.
Change   UI-3933: Added syntax help text for the Override Presto
Configuration field in the Control Panel > Cluster Configuration page.

SPARK

Fix   SPAR-1126: The job-level DAG visualisation is now a working link and takes you to the job details page.

TEZ

Fix   QTEZ-46: Fixed application tracking URL in Tez UI.

ZEPPELIN/NOTEBOOKS

New   ZEP-395: Qubole has introduced a new Angular interpreter. You can include custom JS libraries and customize UI.

List of Hotfixes Since 06th July 2016

Fix   HIVE-1494: Passing user name instead of email as UGI for hivecli and shellcli

Fix  QBOL-5562: Fixed the issue with shell commands with parameters for the Korea region.