Qubole Release Notes for QDS Version R43 11-May-2017

Release Version: 43.37.0

For details of what has changed in this version of QDS, see What is New, New Beta Features, and List of Changes and Bug Fixes in this Release.


What is New

QDS Supports Refreshing New VPCs and Subnets in Cluster Configuration through APIs

QDS supports refreshing the newly created VPCs and subnets in the cluster configuration through a REST API call.

Enabling Hive Server 2 and Setting Hive versions in Hadoop 2 Clusters

QDS supports enabling Hive Server 2 on the Hadoop 2 cluster UI and also setting Hive versions 1.2.0 or 2.1.1 in the Hadoop 2 cluster UI. With this release, Hadoop 2 cluster will be known as Hadoop 2 (Hive) cluster. For more information, see documentation

Hive Server 2 is Generally Available and Hive 2.1.1 is supported as a beta version with this release. For more information, see documentation

Workload-defined Scaling Limits
Qubole's YARN-based auto-scaling for Hadoop2 and Spark clusters now honors maximum limits set for each queue in the Fair Scheduler. This prevents unnecessary upscaling where even the requested application cannot use resources that are available due to upscaling. Using this, administrators can now set auto-scaling limits for different workloads using the Fair-Scheduler configuration. To enable this feature on your Qubole account, create a ticket with Qubole Support.

New Enhancements in Presto
These are the new enhancements in Presto in this QDS release:

  • RubiX’s version is upgraded to 0.2.8.
  • Presto 0.157 is Generally Available in QDS.
  • QDS supports HTTPS in Presto 0.157.
  • Presto 0.119 is an unsupported version with this release.

New Enhancements in Notebooks

These are the new enhancements in Notebooks in this QDS release:

  • A visual folder location picker in Notebooks. For more information, see documentation.
  • REST APIs for Notebook actions such as assigning ACLs, cloning a notebook, and binding a notebook to a cluster. For more information, see Notebook API and Object Policy API.
  • Notebook command APIs can have other parameters. Parameterized Notebook APIs use dynamic forms to populate parameter values. For more information, see documentation.
  • You can collapse/expand the side panel in Notebook folders.
  • After the Qubole Notebook’s version is upgraded to the latest open-source stable version, the Lock feature is currently unavailable on notebooks. If you want to use this feature, create a ticket with Qubole Support to enable the older version on your account.

Qubole Supports File and Hive Partition Sensors 

Qubole supports file and Hive partition sensors that Airflow uses to programatically monitor workflows. For more information, see Sensors and Sensor APIs

UI Enhancements
Qubole has a few enhancements in this QDS release that are listed below:

  • QDS does not display clusters with deny ACL read permissions on the UI.
  • QDS provides an option to filter schedule jobs by multiple users in the Scheduler UI, which now includes disabled users.

New Beta Features


None


List of Changes and Bug Fixes in this Release

AIRFLOW

Change   QBOL-6061: File and Hive Partition sensors have been added in Airflow.

AWS CLUSTER MANAGEMENT

New   ACM-872: QDS allows newly created VPCs and subnets to be specified in the cluster configuration through REST APIs.
Fix   ACM-500: Better handling of cluster configuration with increased robustness.
Fix   ACM-979: The issue where queries failing due to missing master node in DB has been fixed.
Change   ACM-774 and ACM-1067: Improvements to reduce the number of EC2 calls made on clusters.
Change   QBOL-6072: QDS does not display clusters with deny ACL read permissions on the UI.


HADOOP 2

Fix   HADTWO-724: Logs related to S3A file system in Hadoop 2 have been added.
Fix   HADTWO-833: The resource manager on Hadoop 2 clusters can restart gracefully in case of errors without losing information on running jobs. To enable this feature, set the following hadoop configuration in your hadoop overrides:
yarn.resourcemanager.recovery.enabled=true
Fix   HADTWO-838: The Tez jobs that were failing with unavailability of Name Node as the error, has been fixed now.
Fix   HADTWO-839: Upgraded the version of AWS SDK being used by S3A FS to 1.10.77 from 1.10.6.
Fix   HADTWO-853: Fixed issue where applications would not run on certain nodes because of nodemanager being registered with an incorrect name. In these cases, it is seen that the node registers itself with the Resource Manager as localhost instead of registering with its DNS name.

Change   HADTWO-608: Qubole's Yarn based auto-scaling works accordingly by considering the Fair Scheduler’s maximum limit.
Change   HADTWO-830: Handled the case in which 1 byte file ending with/was not considered as a folder with the S3A filesystem.


HIVE 1.2

New   HIVE-2003: Displays a warning when a user is writing to dynamic partitions instead of static partitions.
New   HIVE-2004: Displays a warning to the user when the conditions of hive.mapred.mode=strict gets violated.
New   UI-5463: Hive version and Hive Server configuration support has been added in the Hadoop 2 cluster configuration UI.
Fix   HIVE-1145: Support alter table partition concatenate and truncate. Also pulled HIVE-12522: Wrong FS error during Tez merge files when warehouse and scratchdir are on different File System.
Fix   HIVE-1435: The issue where Hive queries were not starting has been fixed now.
Fix   HIVE-1527: TMP tables get cleaned up now.
Fix   HIVE-1767: Killing an Hadoop job on QDS successfully kills an MR job.
Fix   HIVE-1910: Pulled in open source HIVE-9499 fix to handle NPE when querying non-partitioned tables with hive.limit.query.max.table.partition set.
Fix   HIVE-1971: To make OS Hive executable work by encouraging Hive jars to be ahead of H2 jars in the classpath.
Fix   HIVE-1993: Adding retries in case of JDOException while initializing the HiveMetastoreClient.
Fix   HIVE-2076: Fixed an issue causing logs to be missing for Hive Server 2 queries intermittently.
Change   HIVE-1729: Added support for Authorization on Hive Server 2.
Change   HIVE-1958: Tez is supported on the beta version of Hive Server 2.

PIG

Fix   QPIG-69: A pig script can successfully work with a script location on an Hadoop 2 cluster.

 

PRESTO

Fix   PRES-863: Starting Presto version v0.142, optimized parquet reader is enabled by default in QDS Presto.
Fix   PRES-978: Fixed an issue due to which queries with duplicate alias in result were failing.
Fix   PRES-981: Following changes have been made to Presto 0.157 to be consistent with Hive:
All tables under non-default DB go to the location provided by DB instead of defloc/warehouse.
Directories are created if DB location does not exist instead of failing the queries.

Fix   PRES-990: Fixed the addPartition logic in 0.157 to copy the SerDe properties from table into Partition.
Fix   PRES-1004: Upgraded Rubix version to 0.2.8.
Change   PRES-994: QDS supports HTTPS in Presto 0.157.
Change   PRES-1012: Presto 0.157 is Generally Available in QDS.


QDS

Fix   MUL-441: Resolves a bug for parallel-master-slave cluster bring up feature.
Fix   MW-551: The issue where a user on the EU environment was unable to login after disabling the account has been resolved.
Fix   MW-605: The issue where a system admin could not start the cluster, has been fixed.
Fix   QBOL-5904:The command status that was not updated in the UI gets updated now.
Fix   QBOL-5924: The issue related to special characters in the metastore credentials has been fixed.
Fix   SEC-666: When a user of a QDS account is disabled, that account admin will receive an email notification with the details of periodic jobs of the disabled user if any.
Fix   UI-4901: Improvements in Usage charts.
Change   UI-3151: Display enhancements for selected tabs within Control Panel.
Change   UI-4293: QDS provides an option to filter schedule jobs by multiple users in the Scheduler UI, which now includes disabled users.

Change   UI-5097: Recent commands section will start showing up commands run from the UI by the current Qubole user. This section will show up under My Unsaved Commands for a maximum period of 30 days with the default time period of 24 hours and Last 48 Hours as the third option. This is along with Saved Commands in Workspace of the Analyze UI page.

Change   UI-5474: A UI option to configure Dedicated Instances in the Cluster UI configuration when a VPC is selected. 


TEZ

Fix   QTEZ-96: Fixed an issue in CombineHiveInputFormat which caused the SELECT operation on an empty partitioned table to fail.
Fix   QTEZ-132: Keeping track of lost nodes in case of stopping containers.


ZEPPELIN/NOTEBOOKS

New   UI-4420: A visual folder location picker in Notebooks.
New   ZEP-645: Pulled latest stable Zeppelin from the open source.
New   ZEP-718: Notebook command APIs can have other parameters. Parameterized Notebook APIs use dynamic forms to populate parameter values.
New   ZEP-761: REST APIs for Notebook actions such as assigning ACLs, cloning a notebook, and binding a notebook to a cluster.
Fix   UI-4774: Fixed the issue with Filter on a New Notebook page on the Firefox browser.
Fix   UI-5668: Fixed an issue where it used to display a message to configure the cluster even when the cluster is configured.
Fix   ZEP-822: Notebooks residing in a folder which have space in the name will work.
Change   UI-5503: You can collapse/expand the side panel in Notebook folders.
Change   UI-5564: Enhancements in Notebook folders.

 

List of Hotfixes Since 12th April 2017


New   ACM-993: QDS supports i3 ec2 instance types which include i3.xlarge, i3.2xlarge, i3.4xlarge, i3.8xlarge and i3.16xlarge.
New   ACM-1105: QDS supports m4.16xlarge instance type.
New   SPAR-1386: QDS now supports Spark-2.1.0 as the latest version.
Fix   ZEP-759: The issue where a few paragraphs had been missing has been resolved.
Change   ZEP-803: The exact time of execution is displayed in each paragraph.

 

Have more questions? Submit a request

Comments

Powered by Zendesk