Release Version: 43.37.0
For details of what has changed in this version of QDS, see What is New, New Beta Features, and List of Changes and Bug Fixes in this Release.
What is New
QDS Supports Refreshing New VPCs and Subnets in Cluster Configuration through APIs
QDS supports refreshing the newly created VPCs and subnets in the cluster configuration through a REST API call.
Enabling Hive Server 2 and Setting Hive versions in Hadoop 2 Clusters
QDS supports enabling Hive Server 2 on the Hadoop 2 cluster UI and also setting Hive versions 1.2.0 or 2.1.1 in the Hadoop 2 cluster UI. With this release, Hadoop 2 cluster will be known as Hadoop 2 (Hive) cluster. For more information, see documentation.
Hive Server 2 is Generally Available and Hive 2.1.1 is supported as a beta version with this release. For more information, see documentation.
Workload-defined Scaling Limits
Qubole's YARN-based auto-scaling for Hadoop2 and Spark clusters now honors maximum limits set for each queue in the Fair Scheduler. This prevents unnecessary upscaling where even the requested application cannot use resources that are available due to upscaling. Using this, administrators can now set auto-scaling limits for different workloads using the Fair-Scheduler configuration. To enable this feature on your Qubole account, create a ticket with Qubole Support.
New Enhancements in Presto
These are the new enhancements in Presto in this QDS release:
- RubiX’s version is upgraded to 0.2.8.
- Presto 0.157 is Generally Available in QDS.
- QDS supports HTTPS in Presto 0.157.
- Presto 0.119 is an unsupported version with this release.
New Enhancements in Notebooks
These are the new enhancements in Notebooks in this QDS release:
- A visual folder location picker in Notebooks. For more information, see documentation.
- REST APIs for Notebook actions such as assigning ACLs, cloning a notebook, and binding a notebook to a cluster. For more information, see Notebook API and Object Policy API.
- Notebook command APIs can have other parameters. Parameterized Notebook APIs use dynamic forms to populate parameter values. For more information, see documentation.
- You can collapse/expand the side panel in Notebook folders.
- After the Qubole Notebook’s version is upgraded to the latest open-source stable version, the Lock feature is currently unavailable on notebooks. If you want to use this feature, create a ticket with Qubole Support to enable the older version on your account.
Qubole Supports File and Hive Partition Sensors
Qubole has a few enhancements in this QDS release that are listed below:
- QDS does not display clusters with deny ACL read permissions on the UI.
- QDS provides an option to filter schedule jobs by multiple users in the Scheduler UI, which now includes disabled users.
New Beta Features
List of Changes and Bug Fixes in this Release
Change QBOL-6061: File and Hive Partition sensors have been added in Airflow.
AWS CLUSTER MANAGEMENT
New ACM-872: QDS allows newly created VPCs and subnets to be specified in the cluster configuration through REST APIs.
Fix ACM-500: Better handling of cluster configuration with increased robustness.
Fix ACM-979: The issue where queries failing due to missing master node in DB has been fixed.
Change ACM-774 and ACM-1067: Improvements to reduce the number of EC2 calls made on clusters.
Change QBOL-6072: QDS does not display clusters with deny ACL read permissions on the UI.
Fix HADTWO-724: Logs related to S3A file system in Hadoop 2 have been added.
Fix HADTWO-833: The resource manager on Hadoop 2 clusters can restart gracefully in case of errors without losing information on running jobs. To enable this feature, set the following hadoop configuration in your hadoop overrides:
Fix HADTWO-838: The Tez jobs that were failing with unavailability of Name Node as the error, has been fixed now.
Fix HADTWO-839: Upgraded the version of AWS SDK being used by S3A FS to 1.10.77 from 1.10.6.
Fix HADTWO-853: Fixed issue where applications would not run on certain nodes because of nodemanager being registered with an incorrect name. In these cases, it is seen that the node registers itself with the Resource Manager as localhost instead of registering with its DNS name.
Change HADTWO-608: Qubole's Yarn based auto-scaling works accordingly by considering the Fair Scheduler’s maximum limit.
Change HADTWO-830: Handled the case in which 1 byte file ending with/was not considered as a folder with the S3A filesystem.
New HIVE-2003: Displays a warning when a user is writing to dynamic partitions instead of static partitions.
New HIVE-2004: Displays a warning to the user when the conditions of hive.mapred.mode=strict gets violated.
New UI-5463: Hive version and Hive Server configuration support has been added in the Hadoop 2 cluster configuration UI.
Fix HIVE-1145: Support alter table partition concatenate and truncate. Also pulled HIVE-12522: Wrong FS error during Tez merge files when warehouse and scratchdir are on different File System.
Fix HIVE-1435: The issue where Hive queries were not starting has been fixed now.
Fix HIVE-1527: TMP tables get cleaned up now.
Fix HIVE-1767: Killing an Hadoop job on QDS successfully kills an MR job.
Fix HIVE-1910: Pulled in open source HIVE-9499 fix to handle NPE when querying non-partitioned tables with hive.limit.query.max.table.partition set.
Fix HIVE-1971: To make OS Hive executable work by encouraging Hive jars to be ahead of H2 jars in the classpath.
Fix HIVE-1993: Adding retries in case of JDOException while initializing the HiveMetastoreClient.
Fix HIVE-2076: Fixed an issue causing logs to be missing for Hive Server 2 queries intermittently.
Change HIVE-1729: Added support for Authorization on Hive Server 2.
Change HIVE-1958: Tez is supported on the beta version of Hive Server 2.
Fix QPIG-69: A pig script can successfully work with a script location on an Hadoop 2 cluster.
Fix PRES-863: Starting Presto version v0.142, optimized parquet reader is enabled by default in QDS Presto.
Fix PRES-978: Fixed an issue due to which queries with duplicate alias in result were failing.
Fix PRES-981: Following changes have been made to Presto 0.157 to be consistent with Hive:
All tables under non-default DB go to the location provided by DB instead of defloc/warehouse.
Directories are created if DB location does not exist instead of failing the queries.
Fix PRES-990: Fixed the addPartition logic in 0.157 to copy the SerDe properties from table into Partition.
Fix PRES-1004: Upgraded Rubix version to 0.2.8.
Change PRES-994: QDS supports HTTPS in Presto 0.157.
Change PRES-1012: Presto 0.157 is Generally Available in QDS.
Fix MUL-441: Resolves a bug for parallel-master-slave cluster bring up feature.
Fix MW-551: The issue where a user on the EU environment was unable to login after disabling the account has been resolved.
Fix MW-605: The issue where a system admin could not start the cluster, has been fixed.
Fix QBOL-5904:The command status that was not updated in the UI gets updated now.
Fix QBOL-5924: The issue related to special characters in the metastore credentials has been fixed.
Fix SEC-666: When a user of a QDS account is disabled, that account admin will receive an email notification with the details of periodic jobs of the disabled user if any.
Fix UI-4901: Improvements in Usage charts.
Change UI-3151: Display enhancements for selected tabs within Control Panel.
Change UI-4293: QDS provides an option to filter schedule jobs by multiple users in the Scheduler UI, which now includes disabled users.
Change UI-5097: Recent commands section will start showing up commands run from the UI by the current Qubole user. This section will show up under My Unsaved Commands for a maximum period of 30 days with the default time period of 24 hours and Last 48 Hours as the third option. This is along with Saved Commands in Workspace of the Analyze UI page.
Change UI-5474: A UI option to configure Dedicated Instances in the Cluster UI configuration when a VPC is selected.
Fix QTEZ-96: Fixed an issue in CombineHiveInputFormat which caused the SELECT operation on an empty partitioned table to fail.
Fix QTEZ-132: Keeping track of lost nodes in case of stopping containers.
New UI-4420: A visual folder location picker in Notebooks.
New ZEP-645: Pulled latest stable Zeppelin from the open source.
New ZEP-718: Notebook command APIs can have other parameters. Parameterized Notebook APIs use dynamic forms to populate parameter values.
New ZEP-761: REST APIs for Notebook actions such as assigning ACLs, cloning a notebook, and binding a notebook to a cluster.
Fix UI-4774: Fixed the issue with Filter on a New Notebook page on the Firefox browser.
Fix UI-5668: Fixed an issue where it used to display a message to configure the cluster even when the cluster is configured.
Fix ZEP-822: Notebooks residing in a folder which have space in the name will work.
Change UI-5503: You can collapse/expand the side panel in Notebook folders.
Change UI-5564: Enhancements in Notebook folders.
List of Hotfixes Since 12th April 2017
New ACM-993: QDS supports i3 ec2 instance types which include i3.xlarge, i3.2xlarge, i3.4xlarge, i3.8xlarge and i3.16xlarge.
New ACM-1105: QDS supports m4.16xlarge instance type.
New SPAR-1386: QDS now supports Spark-2.1.0 as the latest version.
Fix ZEP-759: The issue where a few paragraphs had been missing has been resolved.
Change ZEP-803: The exact time of execution is displayed in each paragraph.