Qubole Release Notes 21-Sep-2016

Release Version: 37.0.0

For details of what has changed in this version of QDS, see What is New and List of Changes and Bug Fixes in this Release.

What is New

Support for GPU-based Instances

Qubole now supports GPU-based instances, g2.2xlarge and g2.8xlarge.


New Features in Notebooks

These are the new notebook features in this QDS release:

  • Qubole supports import/export of a notebook from the Notebooks UI page.
  • Qubole notebooks now supports the checkbox for multi-selection that is available in the open-source dynamic form features. See checkbox for more information.

 

Changing Default Version of Python to Python 2.7

Qubole now provides both Python 2.7 and Python 2.6 in cluster nodes as a first step towards migration to Python 2.7. Add the following snippet in the node bootstrap script to make Python 2.7 as the default version for a cluster.

source /usr/lib/hustler/bin/qubole-bash-lib.sh
make-python2.7-system-default

Contact help@qubole.com to enable this feature for the Qubole account. Qubole recommends you to test all applications/jobs which use Python before switching the default Python version to Python 2.7 at the account level.

UI Enhancements

These are the UI enhancements:

  • In the Explore UI page, while creating Table Schema to Hive from the Amazon S3 data, the following changes are done:
  1. In the Sample data, there is a checkbox to select if you want to skip the header row and use them   as column names. 
  2. In the Create Table Schema to Hive (2/4) dialog, selecting the checkbox automatically selects the     first row as unique column headings. This dialog also contains a Hive table description text field. 
  • The Hive Table description field is now available for the user to fill or edit for a particular table in the Explore UI page.


New Beta Features

Viewing Cluster-start Logs

Qubole now provides a feature to see all cluster-start logs irrespective of whether it was started from UI, by clicking the start button or it was started when a query/command was run. To enable this feature for the Qubole account, contact help@qubole.com.


List of Changes and Bug Fixes in this Release

AWS CLUSTER MANAGEMENT

New   ACM-145: Qubole now provides a feature to see all cluster-start logs irrespective of whether it was started from UI, by clicking the start button or it was started when a query/command was run. To enable this feature for the Qubole account, contact help@qubole.com.
New   ACM-58: As a first step towards migration to Python 2.7 by default, Qubole is providing both Python 2.7 and Python 2.6 in cluster nodes. Add the following snippet in the node bootstrap script to make Python 2.7 as the default version for a cluster.
source /usr/lib/hustler/bin/qubole-bash-lib.sh
make-python2.7-system-default

This feature can be enabled at an account level by contacting help@qubole.com. Test all applications/jobs which use Python before switching the default Python version to Python 2.7 at the account level.
Fix   ACM-368: Hive on master queries now work for clusters launched with unique SSH keys.
Change   ACM-165: Qubole now supports GPU based instances (g2.2xlarge and g2.8xlarge). In addition to NVIDIA drivers which enable full functionality and performance of the NVIDIA GPUs, these instances also have CUDA toolkit installed. THe CUDA toolkit drastically increases the computing performance with parallel computing.


HADOOP-2

New   HADTWO-506: Added support to kill hadoop application from the UI.
Fix   HADTWO-403: Fixed the issue to show Application IDs beyond 9999 in the RM UI.
Change   HADTWO-555: Adding support to do log aggregation before removing a node while downscaling the cluster.
Change   HADTWO-576: Fixing issues while recommissioning fully decommissioned nodes as part the cluster upscaling.
Change   HADTWO-579: Added GetSample jar in Hadoop 2. This jar is responsible for fetching sample files from Amazon S3.

 

HBASE

Change   HBAS-179: Maximum heap-size for HBase processes is now configurable using HBase cluster overrides. The hbase_max_heap property can be set with a custom value (in MB). If configured, the HBase master and region server processes start with the specified heap size.


HIVE 0.13

Fix   HIVE-1162 : Qubole provides flags to control s3 path filtering. s3.ipp.ignoretemppaths.nonrestrictive (boolean) enables to discard S3 path, which starts with _SUCCESS, _tmp and a dot (.) . A user can also provide path fields as a comma-separated list with the help of flag s3.inputpathprocessor.filters, which are to be discarded.
Fix   HIVE-1529: Populate the Serde information for Textfile and sequencefile storageformats.

 

HIVE 1.2

Fix   HIVE-1162 : Qubole provides flags to control s3 path filtering. s3.ipp.ignoretemppaths.nonrestrictive (boolean) enables to discard S3 path, which starts with _SUCCESS, _tmp and a dot (.) . A user can also provide path fields as a comma-separated list with the help of flag s3.inputpathprocessor.filters, which are to be discarded.
Fix   HIVE-1528: Added support for cleaning up data written by the INSERT INTO command in the event of failure.

 

PRESTO 0.119

Fix   PRES-672: Fix to show Presto autoscaling logs in the UI
Change   PRES-700: Query Tracker URLs are now listed in Resource tab for presto queries in the Analyze UI page.


PRESTO 0.142

Fix   PRES-672: Fix to show Presto autoscaling logs in the UI
Change   PRES-700: Query Tracker URLs are now listed in Resource tab for presto queries in the Analyze UI page.

 

QDS

New   UI-3622: In the Explore UI page, while creating Table Schema to Hive from the Amazon S3 data, the following changes are done:

  • In the Sample data, there is a checkbox to select if you want to skip the header row and use them as column names. 
  • In the Create Table Schema to Hive (2/4) dialog, selecting the checkbox automatically selects the first row as unique column headings. This dialog also contains a Hive table description text field.

New   UI-3937: Qubole supports import/export of a notebook from the Notebooks UI page.
Fix   UI-3636: For jobs generating huge amount of logs, last 1000 lines are displayed by default, with options to download the entire file.
Fix   UI-4153: Fixed an issue where Notebook search was broken for a cluster label
Fix   UI-4269: There was a case where Data Import command in the query composer was displayed when the Analyze UI was opened. This case has been fixed. If user had not composed a data import command before, the Analyze query composer UI displays the Hive Command.
Change   UI-3846: The Hive Table description field is now available for the user to fill or edit for a particular table in the Explore UI page.

 


TEZ

Fix   QTEZ-71: Showing summary of completed job for Hive on Tez queries
Fix   QTEZ-80: Upgraded Kyro version to 3.0.x . Related OS JIRAs: HIVE-11428 and HIVE-12175.
Fix   QTEZ-81: Logs are now accessible for the tasks which ran on already downscaled nodes.

 


ZEPPELIN/NOTEBOOKS

New   ZEP-446: Qubole notebooks now supports the checkbox for multi-selection that is available in the open-source dynamic form features. See checkbox for more information.

 


List of Hotfixes Since 01st September 2016


Fix   HIVE-1579: Throwing an exception as soon as the transfer of the files fails.
Fix   QBOL-5709: Do not escape from the shell command submitted inside an Airflow cluster.
Fix   UI-4262: A bug which caused the Run button to be disabled even if the correct permissions were present has been resolved.

 

 List of Hotfixes After 21st September 2016

Fix   HADTWO-627: Fixed issues in FairScheduler caused by dot in the username.

Have more questions? Submit a request

Comments

Powered by Zendesk