Release Version: 35.0.0
For details of what has changed in this version of QDS, see What is New and List of Changes and Bug Fixes in this Release.
What is New
Support for new EBS Volume Types and Sizes
QDS clusters now support additional EBS volume types. You can now use Throughput Optimized HDD (st1) and Cold HDD (sc1) volumes. This is in addition to General Purpose SSD (gp2) and Magnetic (standard) volume types.
QDS now supports EBS volume sizes of up to 16 TB (previously, the limit per volume was 500
GB). QDS uses the ext4 file system by default for volume disks larger than 500 GB to improve
speed in formatting during cluster bootup. For volume disks less than 500 GB, QDS still defaults
to using ext3 file system. You can opt in to using ext4 for volumes smaller than 500 GB by
emailing help@qubole.com.
QDS UI Enhancements
These are enhancements in the QDS UI:
- Qubole now supports keyword and schema autocomplete in Hive, Presto, and Spark SQL offerings. Use the Ctrl+Space key combination in the online editors to get a list of suggestions. It takes care of the aliases given to tables and queries using tables across different databases.
- Qubole now supports creating either a new IAM Keys or IAM Roles account from within the Control Panel.
- Two new buttons, refresh Hive Schema and refresh Hive Table, have been added to the Data Import command type in the Analyze page’s query composer.
- Syntax help text for the Override Presto Configuration Variables and Override Spark Configuration Variables fields in the Control Panel > Cluster Configuration page.
Angular Interpreter in Notebooks
Qubole has introduced the Angular interpreter in Notebooks. It can be used in creating interpreter variables using customized JS libraries to render customized graphs and visualizations. See Angular Interpreter for more information.
List of Changes and Bug Fixes in this Release
AIRFLOW
Change
QBOL-5455: Fixed broken links, pages and Create/Update/Delete operations on the Airflow Webserver Dashboard
AWS CLUSTER MANAGEMENT
New ACM252: Added support for new AWS EBS disk types: st1 and sc1 to be used in clusters. Please note, all the cluster disks are formatted using ext4 filesystem if st1 and sc1 volumes are used since they are at least 500 GB in size.
The EBS disk size limit has been increased to 16TB from 500 GB.
Contact help@qubole.com to enable the ext4 filesystem for volume sizes less
than 500 GB.
Change ACM-291: Improvements to avoid formatting ebs disks by using pre-formatted EBS snapshots.
Change ACM-357: DataDog API and APP token now implicitly enables Ganglia in the cluster. Earlier it was necessary to explicitly enable Ganglia monitoring for a cluster to get metrics into DataDog. Although the Ganglia monitoring is enabled, its link may not be visible in the cluster’s UI resources list.
Change Reduced cluster start time for i2, r3, c3, and m3 family of instances.
HADOOP
Fix HAD-541: Files older than 3 days in the S3 buffer directory (identified by the fs.s3.buffer.dir option) will be removed to clear up storage capacity.
HADOOP-2
Fix HADTWO-428: Added s3distcp support for IAM roles
Fix HADTWO-529: clouddistcp will now work with spaces in source or destination paths.
Change HADTWO-508: Minimum cluster size of cluster is made pushable. With this change, you can change the minimum cluster size of a running cluster.
Change HADTWO-522: S3 listing taking a long time for Hadoop 2. This issue has been fixed.
HBASE
New HBAS-156: Added monit services for HBase master, zookeeper and regionserver processes. The monit daemon will try to restart these processes automatically in case they are terminated due to some reason.
Fix HBAS-149: The pid files for master, regionserver, and zookeeper process in HBase are now stored in /media/ephemeral0/pids/hbase directory. Also, the bug in restarting the daemons for the three processes due to pid file-name mismatch is resolved.
Fix HBAS-167: Added auto-purging properties for periodic removal of zookeeper snapshots and transaction logs in HBase. Only last three snapshots and transaction logs are saved and the cleanup runs every hour. This will help to keep in check the storage used by zookeeper logs with a fallback of last three snapshots for recovery.
Change HBAS-175: Increased the timeout for HBase table snapshots to ensure they complete for large tables. Previously, the timeout was 60 seconds (1 min) for both master and regionserver. Now, the timeout is 900 seconds (15 mins) for master and 600 seconds (10 mins) for regionserver.
HIVE 0.13
Fix HIVE-1449: Fixed an issue which caused CTAS statements to throw errors, when the table is stored as ORC File format and select clause has NULL/VOID type column. Related to OS JIRA - HIVE-11217
Fix HIVE-1478: Fixed an issue which caused queries of type group by...having with table alias to fail with ClassCastException. OS JIRA - HIVE-8398
Fix HIVE-1487: Fix to handle spaces in the partition path when bucketing is enforced with dynamic partitioning.
Fix HIVE-1491: Increasing the hive.exec.reducers.bytes.per.reducer to 1G to improve reducer estimation
Change ROLL-22: Change in Hive to allow deletion of jar by default.
HIVE 1.2
Fix HIVE-1291: Fixed an issue which caused NPE in DynamicPartFileRecordWriterContainer on using null partition keys.
Related OS JIRA - HIVE-11470
Fix HIVE-1449: Fixed an issue which caused CTAS statements throw errors, when the table is stored as ORC File format and select clause has NULL/VOID type column. Related OS JIRA - HIVE-11217
Fix HIVE-1476: Fixed an issue which caused drop table commands for managed table to succeed even in case of failure in actual data deletion.
Fix HIVE-1478: Fixed an issue which caused queries of type group by..having with table alias to fail with ClassCastException. OS JIRA - HIVE-8398
Fix HIVE-1487: Fix to handle spaces in the partition path when bucketing is enforced with dynamic partitioning.
Fix HIVE-1491: Increasing the hive.exec.reducers.bytes.per.reducer to 1G to improve reducer estimation
Fix HIVE-1495: Changed log level in Avro serde package to reduce the frequency of logs printing schema details.
Change ROLL-22: Change in Hive to allow deletion of jar by default
HBASE
New HBAS-156: Added monit services for HBase master, zookeeper and regionserver processes. The monit daemon will try to restart these processes automatically in case they are terminated due to some reason.
Fix HBAS-149: The pid files for master, regionserver, and zookeeper process in HBase are now stored in /media/ephemeral0/pids/hbase directory. Also, the bug in restarting the daemons for the three processes due to pid file-name mismatch is resolved.
Fix HBAS-167: Added auto-purging properties for periodic removal of zookeeper snapshots and transaction logs in HBase. Only last three snapshots and transaction logs are saved and the cleanup runs every hour. This will help to keep in check the storage used by zookeeper logs with a fallback of last three snapshots for recovery.
Change HBAS-175: Increased the timeout for HBase table snapshots to ensure they complete for large tables. Previously, the timeout was 60 seconds (1 min) for both master and regionserver. Now, the timeout is 900 seconds (15 mins) for master and 600 seconds (10 mins) for regionserver.
PRESTO
Fix PRES-663: Fixed a bug in block caching returned EOFException wrongly when the server is under a heavy load.
Fix PRES-675: This fixes the issue of queries failing with the
Error committing write to Hive error
Fix PRES-689: Strict logging in RubiX's BookKeeper server to prevent frequent instance freezes
QDS
New UI-3443: Qubole now supports creating either a new IAM Keys or IAM Roles account from within the Control Panel.
New UI-3734: Introducing the Format button in the online HiveQL editor to make a query more readable and investigate the underlying structure of sub queries.
Fix UI-3675: Scheduled job instances that have Hive or S3 dependency now display this information in Scheduler and Analyze UI.
Fix UI-3948: The issue of Persistent Security Groups being disabled if there is no account update permission has been fixed.
Fix UI-4010: The job-level DAG visualisation is now a working link and takes you to the job details page.
Change MW-17: Support to throttle command submission to ensure fair scheduling. Throttling is based on global waiting command limit and per account waiting command limit
Change UI-3533: Two new buttons, refresh Hive Schema and refresh Hive Table, have been added to the Data Import command type in the Analyze page’s query composer.
Change UI-3770: Qubole now supports keyword and schema autocomplete in Hive, Presto, and Spark SQL offerings. Use the Ctrl+Space key combination in the online editors to get a list of suggestions. It takes care of the aliases given to tables and queries using tables across different databases.
Change UI-3896: Added syntax help text for the Override Spark
Configuration Variables field in the Control Panel > Cluster Configuration page.
Change UI-3933: Added syntax help text for the Override Presto
Configuration field in the Control Panel > Cluster Configuration page.
SPARK
Fix SPAR-1126: The job-level DAG visualisation is now a working link and takes you to the job details page.
TEZ
Fix QTEZ-46: Fixed application tracking URL in Tez UI.
ZEPPELIN/NOTEBOOKS
New ZEP-395: Qubole has introduced a new Angular interpreter. You can include custom JS libraries and customize UI.
List of Hotfixes Since 06th July 2016
Fix HIVE-1494: Passing user name instead of email as UGI for hivecli and shellcli
Fix QBOL-5562: Fixed the issue with shell commands with parameters for the Korea region.
Comments