Qubole Release Notes for QDS Version R41 28-Feb-2017

Release Version: 41.25.0

For details of what has changed in this version of QDS, see What is New, New Beta Features, and List of Changes and Bug Fixes in this Release.

An Heads-up on the Next QDS Feature Roll-out

After this QDS release, Qubole will begin rolling out new Clusters UI as a separate page that is at the same-level as Control Panel and Analyze. Among other things, it features a wizard like interface for editing cluster settings, integrated charts for showing cluster usage, a cluster nodes report (On-Demand vs Spot), and also cluster start logs. If you would like to enable this page for your account, please contact help@qubole.com.

 

What is New

Decommissioning Change in Hadoop 2 and Spark Clusters

Hadoop 2/Spark clusters will not decommission to 1 node once they have upscaled. When the minimum size is 1, the cluster starts with a single node, but once upscaled, it will never go back to one node. This is because decommissioning slows down immensely if there is one usable node left for HDFS, potentially leaving several nodes hanging around, doing no work.
This behaviour may be overridden by setting mapred.allow.single.worker.node to true. However, this setting needs a cluster restart.

 

Support for P2, R4, and X1 Instance Types

These three types of instances are supported from this QDS release:

  • Qubole has added support for P2 instances provided by AWS. P2 instances provide GPU based parallel processing ideal for machine learning, high performance databases and other server side workloads.
  • Qubole has added support for R4 instance family provided by AWS. These instances are the next generation of memory optimized instances with larger L3 cache and faster memory. The R4 instances are supported only on ap-south-1, eu-west-1, us-east-1, us-east-2, us-west-1, and us-west-2 regions. 
  • Qubole has added support for X1 instances provided by AWS. X1 instances are memory optimized instances designed for running large scale, in-memory applications and databases.

 

QDS Supported in Two Additional AWS Regions

AWS recently made a new region available in Ohio as us-east-2; QDS now supports creating/launching clusters in the us-east-2 region.

QDS is also supported on the AWS ap-south-1 region.

Enhancements in Notebooks

These are the new enhancements in Notebooks in this QDS release:

  • On the bottom-left of the Notebooks UI, there is a pin button that you can use to hide and unhide the left pane/navigation bar of the Notebooks UI. 
  • Changing the name of the notebook inline reflects the new name under the folders in the left pane/navigation bar of the Notebooks UI.
  • Notebook name instead of notebook ID are displayed in successful acknowledgements. 
  • There is a change in the order in the Notebooks UI’s navigation bar.
  • The message that shows Zeppelin being started has changed and it does not show the 502 error when cluster is started (even when Zeppelin is still unavailable).
  • The import icon to import a notebook on the Notebooks UI has changed.
  • You can rename, move, and delete folders from the Notebooks UI. For more information, see Move Rename and Delete Operations Supported on Notebook Folders.
  • You can move a notebook to another folder by dragging it from its parent folder and dropping it in the other folder. For more information, see Move Rename and Delete Operations Supported on Notebook Folders.
  • The important information while deleting folder is shown in small text.
  • You can add a public maven repository using zeppelin.dep.additionalRemoteRepository. To use private repositories, add the repository in the Repositories section of the Interpreters page. 
  • Jars in the specified repos can be accessed using %dep from the Dependencies section.

 

Enhancements in Presto

In this QDS release, Rubix has been upgraded to the 0.2.5 version.

 

UI Enhancements

Qubole has a few enhancements in this QDS release that are listed below:

  • Saved queries will now the show loading icon if they have a command running.
  • Create Schema in the Explore steps have been changed and you can only edit through the Analyze now.
  • EBS-based upscaling configuration can now be set via the new cluster settings UI. Since the new clusters page is not globally available yet, contact help@qubole.com if you would like to enable this for your account.


New Beta Features

Automatic Termination of a Presto Cluster with Invalid Configuration

A Presto cluster terminates automatically on invalid configuration. This feature is available for beta access. Contact help@qubole.com to enable this feature.

 

Move Rename and Delete Operations Supported on Notebook Folders

You can rename, move and delete folders from the Notebooks UI. This feature is available for beta access. Contact help@qubole.com to enable this feature for the account.

 

List of Changes and Bug Fixes in this Release

AIRFLOW

Fix  QBOL-5931: These issues have been fixed:
Airflow cluster bringup failing due to special characters in airflow configuration
Broken links in Airflow emails
Improve validations while saving airflow configuration

AWS CLUSTER MANAGEMENT

New   ACM-491: Qubole has added support for X1 instances provided by AWS. X1 instances are memory optimized instances designed for running large scale, in-memory applications and databases.
New   ACM-695: AWS recently made a new region available in Ohio as us-east-2;
QDS now supports creating/launching clusters in us-east-2 also.
New   ACM-783: Qubole has added support for R4 instance family provided by AWS. These instances are the next generation of memory optimized instances with larger L3 cache and faster memory. The R4 instances are supported only on ap-south-1, eu-west-1, us-east-1, us-east-2, us-west-1, and us-west-2 regions. 
New   ACM-869: Qubole has added support for P2 instances provided by AWS. P2 instances provide GPU based parallel processing ideal for machine learning, high performance databases and other server side workloads.
Fix   ACM-770: The VolumeLimit Exceeded error while bringing up a cluster would be now be caught and surfaced to the UI.
Fix   ACM-865: In case of missing tags on the cluster instance, it will still setup the cron to backup logs into S3.
Fix   ACM-873: Fixed an issue where nodes in heterogeneous cluster fail to come up with custom SSH key.
Fix   ACM-901: QDS is supported on the AWS ap-south-1 region.


HADOOP 2

Fix   HADTWO-723: Hadoop 2/Spark clusters will not decommission to 1 node once they have upscaled. When the minimum size is 1, the cluster starts with a single node, but once upscaled, it will never go back to one node. This is because decommissioning slows down immensely if there is one usable node left for HDFS, potentially leaving several nodes hanging around, doing no work.
This behaviour may be overridden by setting mapred.allow.single.worker.node to true. However, this setting needs a cluster restart.
Fix   HADTWO-748: Fixed the issue in NativeS3FileSystem due to which last modified time for liststatus and getfilestatus APIs, is different.
Fix   HADTWO-762: Fixed full plain text log links for IAM-Role-based accounts.
Change   HADTWO-631: Changed the default value for mapreduce.job.reduce.slowstart.completedmaps to 0.5.
Change   HADTWO-678: Pulled in patches from open-source Hadoop for S3A optimizations.
Change   HADTWO-687: Making the default for mapreduce.task.timeout as 600000 (same as Hadoop1).
Change   HADTWO-695: Decommissioning the Node from HDFS when it is about to be lost due to spot loss.
Change   HADTWO-739: Added optimised ListStatusRecursively in the S3A FileSystem.
Change   HADTWO-743: Added ListStatusRecursively in NativeS3FileSystem and provided default implementation in FileSystem.


HIVE 1.2


New   PER-67: Caching ORC metadata in a Redis cache on the cluster master.
Fix   HIVE-1849: Fixed an issue due to which query on view results used to fail with table not found error if view is created with subquery alias (CTE). Related OS issue : HIVE-10698
Change   HIVE-1745: Picked OS patches for HIVE-10555: Improve windowing spec of range based windowing to support additional range formats.
Change   HIVE-1765: Show stacktraces for SemanticExceptions as well.
Change   HIVE-1780: Patch pulled in from OS:HIVE-10559: IndexOutOfBoundsException with RemoveDynamicPruningBySize (Wei Zhang via Gunther Hagleitner)
Change   HIVE-1814: Fixed updates and deletes with transaction enabled

PRESTO

Fix   PRES-729: Presto cluster terminates automatically on invalid configuration. This feature is available for beta access. Contact help@qubole.com to enable this feature.
Fix   PRES-886: QDS converts all queries into INSERT OVERWRITE DIRECTORY which was causing failures when query result had data type not supported by INSERT operator. This change adds the support to write all data types when INSERT OVERWRITE DIRECTORY is used.
Fix   PRES-907: Added support for non-partitioned date type Parquet column.
Change   PRES-881: Rubix has been upgraded to the 0.2.5 version.
Change   PRES-883: A Presto cluster will have Presto 0.142 as the default version.


QDS

New   UI-4683: EBS-based upscaling configuration can now be set via the new cluster settings UI. Since the new clusters page is not globally available yet, contact help@qubole.com if you would like to enable this for your account.
Fix   MUL-251: Support for better error handling for disk storage account settings.
Fix   UI-4803: Qubole has fixed an issue where it used to truncate column name where tablename is prefix of the column name.
Fix   UI-4822: Fixed an issue where create schema/tables from the Explore UI using a CTRL-A file on S3 was not working.
Fix   UI-4944: Fixed a bug where landing on Analyze via permalinks always opens last visited tab instead of History.
Change   UI-4873: Saved queries will now the show loading icon if they have a command running.
Change   UI-5113: Create Schema in the Explore steps have been changed and you can only edit through the Analyze now.


SPARK

Fix   SPAR-1372: The job link in the paragraph will allow user to the view the corresponding job UI even when after application is stopped.
Change   SPAR-1373: Qubole supports Spark-2.0.2 as the latest version.

 

TEZ

New   QTEZ-110: Offline logs support for Tez jobs
New   QTEZ-111: Added support of roles in the Tez offline UI.
Fix   QTEZ-92: Open-source fix for HIVE-13092 - Initializing Extractor object only for primitive types.

 

ZEPPELIN/NOTEBOOKS

New   ZEP-653: There is an improvement in notebook-loading when two notebooks on the same running cluster are opened consecutively from the folders in the Notebooks UI.
Fix   ZEP-600: You can add a public maven repository using zeppelin.dep.additionalRemoteRepository. To use private repositories, add the repository in the Repositories section of the Interpreters page.
Jars in the specified repos can be accessed using %dep from the Dependencies section.
Change   UI-4334: On the bottom-left of the Notebooks UI, there is a pin button that you can use to hide and unhide the left pane/navigation bar of the Notebooks UI.
Change   UI-4763 and UI-5064: You can move a notebook to another folder by dragging it from its parent folder and dropping it in the other folder.
Change   UI-4785: Changing the name of the notebook inline reflects the new name under the folders in the left pane/navigation bar of the Notebooks UI.
Change   UI-4787: Notebook name instead of notebook ID are displayed in successful acknowledgements.
Change   UI-4834: There is a change in the order in the Notebooks UI’s navigation bar.
Change   UI-4896: The message that shows Zeppelin being started has changed and it does not show the 502 error when cluster is started (even when Zeppelin is still unavailable).
Change   UI-4970: The import icon to import a notebook on the Notebooks UI has changed.
Change   UI-5046 and UI-5163: You can rename, move and delete folders from the Notebooks UI. This feature is available for beta access. Contact help@qubole.com to enable this feature for the account.
Change   UI-5078: The important information while deleting folder is shown in small text.


List of Hotfixes Since 31st January 2017


Fix   UI-5214: Fixed saved query permalinks.
Fix   UI-5215: Fixed Workspace permalinks that were not displayed.

Have more questions? Submit a request

Comments

Powered by Zendesk