Qubole Release Notes for QDS Version R45 12-Jul-2017

Release Version: 45.35.0

For details of what has changed in this version of QDS, see What is New, New Beta Features, and List of Changes and Bug Fixes in this Release.

What is New

New in AWS Cluster Management
These are the new enhancements in AWS clusters in this QDS release:

  • Qubole now supports the automatic endpoint discovery of an S3 bucket in the S3A file system.
  • Qubole clusters, data stores and custom metastores support non-default SSH port and user on the Bastion node. Create a ticket on Qubole Support to get this feature enabled.
  • To enhance security in clusters, Qubole has added support to remove wide-open outbound (egress) rules in the cluster security group. Create a ticket on Qubole Support to get this feature enabled.
  • User-specific details such as the user who has shut down/restarted a cluster, will be soon available in the Cluster UI.

New Enhancements in Hive
This is a new enhancement in Hive in this QDS release:

  • Qubole supports Ganglia metrics for HiveServer2.

New Enhancements in Notebooks

These are the new enhancements in Notebooks in this QDS release:

  • The query editor in Analyze has a new field Arguments for a Spark notebook command and Arguments is also supported in the Scheduler UI for a Spark notebook command. For more information, see the Analyze documentation and Scheduler documentation for notebooks.
  • The websocket port has been changed to port 443. For more information, see the documentation
  • Jobs and Stages pages of the Spark UI will display the information of the notebook ID, name and user information, related to the paragraph that resulted in the Spark job/stage. 
  • Qubole Notebook allows setting UseLegacySql to false.
  • Enabled the lock feature in notebooks. For more information, see the documentation.

 

UI Enhancements
Qubole has a few enhancements in this QDS release that are listed below:

  • The issue of using long query tags in the Analyze > History filter has been fixed. The entire-length of the tag can be seen in the Tag field of the History filter.
  • Object Store is added to resource permission that helps a user to give granular permissions to read, upload and download. The permission will be honored from this QDS release. 
  • Results for a command can now be opened in a new tab. 
  • The Logs UI in Analyze will only have Error and Warning filter for all command types.
  • Information about node types selected while configuring a cluster is now displayed in the Cluster Summary pane and in the Cluster Settings page.


New Beta Features

QDS Supports StreamX Cluster Type

QDS now supports configuring the StreamX cluster type in the Clusters UI page.

Qubole Hive Authorization Supports S3 Storage Permissions Check

Qubole Hive Authorization now supports S3 storage permission check when DDL/DML contains the S3 URI path.


List of Changes and Bug Fixes in this Release

AWS CLUSTER MANAGEMENT

New Features

New   MW-646: Qubole clusters, data stores and custom metastores support non-default SSH port and user on the Bastion node.


Bug Fixes

Fix   ACM-394: Spot Instance errors that occurred during upscaling where the adding nodes process used to be stuck, have been fixed.
Fix   ACM-537: The issue where the cluster start failed and later the cluster marked up as 0 nodes has been fixed.
Fix   ACM-1163: The issue when there was an exception while inserting table name in the Hive metastore has been fixed.
Fix   ACM-1222: The issue of cluster downscaling process being stuck has been fixed.
Fix   ACM-1226: To enhance security in clusters, Qubole has added support to remove wide-open outbound (egress) rules in the cluster security group. Create a ticket on Qubole Support to get this feature enabled.
Fix   ACM-1279: The issue in which the cluster with i3.4x large instance type was considering only one SSD out of 2 for DFS capacity has been fixed.

Enhancements

Change   ACM-100: User-specific details such as the user who has shut down/restarted a cluster, will be soon available in the Cluster UI.

HADOOP 2

Bug Fixes

Fix   HADTWO-640: Fixed an issue in which removeNode API did not work for Hadoop2 clusters intermittently.
Fix   HADTWO-800: There is an edge case in which if all the nodes that we want to decommission do not have the instance IDs, then Qubole would fail to remove them from yarn.exclude, yarn.gs.exclude and dfs.exclude files. This commit fixes it. Now, if Qubole does not find instance IDs for nodes that it wants to terminate, then QDS clears the corresponding IP addresses from yarn and dfs exclude files.
Fix   HADTWO-919: Improved performance when using st1/sc1 disks by setting xen_blkfront.max_indirect_segments to 256 and increasing readahead to 1MB.


Enhancements

Change   HADTWO-898: Qubole now supports the automatic endpoint discovery of a bucket in the S3A file system.
Change   HADTWO-930: Autoscaling events in Cluster APIs are displayed in a list.


HIVE 1.2

New Features

New   HIVE-2110: Qubole Hive Authorization now supports S3 storage permission check when DDL/DML contains the S3 URI path.
New   HIVE-2180: Upgraded the Opex Json jars to 1.3.8 version.


Bug Fixes

Fix   HIVE-1995: Open source commit for HIVE-15023: SimpleFetchOptimizer needs to optimize limit=0.
Fix   HIVE-1996: Qubole does not allow making EXTERNAL TABLES transactional.
Fix   HIVE-2122: Beeline should escape semi-colon in queries.

Enhancements

Change   HIVE-1754: Qubole supports Ganglia metrics for HiveServer2.

 


PRESTO

Bug Fixes

Fix   PRES-831: The query failure to fetch metadata has been fixed.


Enhancements

Change   PRES-1030: Presto notebook’s JDBC version is the same as the version in its associated cluster.


QDS

New Features

New   MW-954: Users can now see if there was a Spot loss during the runtime of the command.

New   UI-4617: The issue of using long query tags in the Analyze > History filter has been fixed. The entire-length of the tag can be seen in the Tag field of the History filter.
New   UI-5960: Object Store is added to resource permission that helps a user to give granular permissions to read, upload and download. The permission will be honored from this QDS release.


Bug Fixes

Fix   UI-5720: The first column heading is changed from Instance ID to Cluster ID in the Cluster Usage report.
Fix   UI-5770: Command runs, logs, and results are displayed for Saved Commands.


Enhancements

Change   UI-5516: When a user resizes the query editor, types a query and runs it, the query editor goes back to its original size after submission. This is now fixed this to retain the query editor size even after submission.
Change   UI-5526: Results for a command can now be opened in a new tab.
Change   UI-5676: The Logs UI in Analyze will only have Error and Warning filter for all command types.
Change   UI-5829: Information about node types selected while configuring a cluster is now displayed in the Cluster Summary pane and in the Cluster Settings page.
Change   UI-5984: In case of a large result set, QDS displays:

  • This message is for commands that return tabular results

Only 100 rows are displayed here.To view larger result set, please download it.

  • This message is for commands that do not return tabular results

Only 1000 rows are displayed here.To view larger result set, please download it.


SPARK
New Features

New   SPAR-1730: Error Code Summary in GET API response for failed Spark command-line options. This feature is available for beta access. Create a ticket with Qubole Support to get this feature enabled on your account.

 

Enhancements

Change   SPAR-1544: The Spark History Server memory has been increased.
Change   SPAR-1629: The FileSystem issue while using S3 URL as checkpoint for structured streaming has been fixed.


STREAMX
New Features

New   UI-5482: There is a new UI to configure StreamX new cluster type in the Clusters page.


ZEPPELIN/NOTEBOOKS

New Features

New   UI-6133: The query editor in Analyze has a new field Arguments for a Spark notebook command.
New   MW-986: Macros are supported in Scheduler for Spark notebooks. In addition, The Spark Notebooks’ scheduler also has the new field Arguments.  For more information, see the Analyze documentation and Scheduler documentation for notebooks.


Bug Fixes

Fix   ZEP-879: The issue where R programs could not be terminated from a notebook has been fixed.
Fix   ZEP-880: The issue where clusters were auto terminating when R programs were running in a notebook has been fixed.
Fix   ZEP-1009: The issue where a notebook was showing that job is running even after individual jobs were completed has been fixed.
Fix   ZEP-1101: The issue in which Python Spark jobs submitted on Zeppelin 0.6.2 were failing has been fixed.


Enhancements

Change   ZEP-188: Enabled the lock feature in notebooks. For more information, see the documentation
Change   ZEP-907: The websocket port has been changed to port 443. For more information, see the documentation
Change   ZEP-948: Jobs and Stages page of the Spark UI will display the information of the notebook ID, name, and user information, related to the paragraph that resulted in the Spark job/stage.
Change   ZEP-954: Qubole Notebook allows setting UseLegacySql to false.
Change   ZEP-1051: A Stop button has been added on the Interpreters page to stop Spark interpreters, which would free resources.
Change   ZEP-1000: Clicking the Restart button in the Interpreters page restarts the Spark interpreter.

 

 

List of Hotfixes Since 14th June 2017


Fix   HADTWO-918: Fixed an issue in which downscaling thread gets killed sometimes due to logger exception as a result of which downscaling of the cluster got stuck.
Fix   MW-1105: The data store connection issue has been fixed.
Fix   PRES-831: The query failure to fetch meta data has been fixed.
Fix   PRES-1096: Rubix has been upgraded to 0.2.11 which minimizes the number of open files.
Fix   PRES-1099: Upgraded the Opex Json jars to 1.3.8 version.
Fix   UI: 6257: The Schedulerp page option from the home page of the QDS UI that was not visible to a set of users is visible now.
Change   HIVE-1978: Zeppelin support for HiveServer2. This feature is available for beta access.  For more information, see the documentation
Change   UI-5986: A link to the HiveServer2 UI has been added under the Resources section in the New Clusters page. However, It is available only when the Hive version is 2.1.1.
Change   ZEP-640: pyspark paragraphs will provide a Spark-specific Python autocomplete suggestion.
Change   ZEP-1023: Zeppelin support for HiveServer2.

 

Have more questions? Submit a request

Comments

Powered by Zendesk