Qubole Release Notes 06-July-2016

Release Version: 34.0.0


For details of what has changed in this version of QDS, see What is New, New Beta Features, and List of Changes and Bug Fixes in this Release.


What is New

 

Airflow in Qubole

 

Qubole has introduced support for hosted Airflow clusters. Airflow is an open source data pipeline and workflow management platform. See Airflow in Qubole, quick start guide, and cluster API parameters description for more information. 

 

Notebook Enhancements

 

Qubole has made significant enhancements that includes exporting notebook results and exporting a notebook as PDF. See Notebooks for more information. 

 

Hive Authorization support for Presto commands

 

Hive Authorization can now be used for Presto commands. Hive authorization can be used to provide table-level access control to Qubole users. See Qubole Hive Authorization and catalog Hive properties for more information.

 

Qubole Help Center Search Result

 

Search functionality in the Qubole Help Center (icon near top right corner) now returns results from both Knowledge Base and Qubole Documentation. See Search Field for more information. 

 

New Beta Features

 

Adding a Custom SSH Key Pair for an Account

 

Qubole now supports adding a custom SSH key pair on a cluster/account to provide security. This feature is not enabled by default. Send an email to help@qubole.com to enable this feature on a Qubole account.

 

GitHub Integration with Notebooks


Qubole now supports GitHub version-control integration with notebooks. This feature is available for Beta access and is not enabled by default. Send an email to help@qubole.com to enable this feature on a Qubole account.


List of Changes and Bug Fixes in this Release

AWS CLUSTER MANAGEMENT

New  ACM-92: Allow option to add an SSH key pair per cluster/account. Write an email to help@support.com to enable it on an account.

Fix   ACM-248: Added entries in the /etc/fstab for all the non-encrypted volumes that are mounted while starting up a cluster. This to ensure that the non-encrypted volumes are available even after the node is rebooted.

Fix   ACM-334: Allow EBS disks to be used with r3, c3, and m3 instance types in ap-northeast-2 and eu-central-1 AWS regions.

Change  ACM-350: Clusters now have their root disk on SSDs (GP2) for improved performance.

   

HADOOP-2

 

Fix    HADTWO-470: Qubole has now reduced the maximum number of retries that an Application Master can do and the timeout interval that an IPC client can do, before killing the containers on lost nodes to speed up this cleanup process for the lost nodes.

Fix   HADTWO-503: Resource Localization fails on a cluster due to existing cache directories.

Change  HADTWO-231: Added support of EBS disks as regular volumes in Hadoop 2.

Change  HADTWO-430: Support for encrypted communication in Hadoop

Change  HADTWO-446: Decommissioning nodes will now be shown on the Resource manager UI. These nodes do not pick up new tasks as they will be downscaled shortly.

 


HBASE

 

Fix    HBAS-168: Fixes the issue of HBase snapshot restoration failure

 


HIVE 1.2

 

Fix    HIVE-1436: Fixed an issue due to which hive.mapred.local.mem was not respected to configure child JVM heap size

 


HIVE 0.13

 

Fix   HIVE-1431 and HIVE-1460: The Hive Execution Engine is now set to mr (by default) for Hive 0.13

 


PRESTO

New   PRES-535: Presto supports hive authorization now. Grants and Roles have to be set up via Hive and Presto can use those to figure out if a user is allowed to perform the action or not.

Fix   PRES-587: The query.max-memory-per-node property does not apply on a master node. As a result, now Qubole supports having smaller master nodes than slave nodes, which previously led to Presto server failure when bringing up slave nodes with optimised memory configurations.

Fix   PRES-592: Fixed a memory leak in thrift metastore which was degrading performance in Presto and Spark clusters, which did not use metastore caching.

Fix   PRES-624: Fixed leak of HTTP clients in Block Caching, which was exhausting file descriptors.

Change  PRES-626: Presto used to throw invalid exception while reading empty ORC files. This fix pulls in a solution for this from open source.

Change  PRES-636: Presto now supports Date columns as Partitions when table is in the Parquet format.

 

 

QDS


New   QBOL-5250: Added ojdbc6 jar in sqoop package for Oracle db import/export support. Changes made in qweez package to include the jar in sqoop while compiling.

New   QBOL-5273: Airflow Cluster Resources Links - Airflow Webserver and Celery Dashboard.

New   UI-3297: New Airflow type cluster is available under the Control Panel > Clusters section.

New   UI-3794: GitHub integration with Notebooks. This feature is available for beta access. Write an email to help@qubole.com to enable this feature on a Qubole account.

Fix   QBOL-5416: Printing duplicate logs. Fixed the issue in the logging utility.

Fix   UI-3521: The Quick Tour option is no longer visible on the Help Center for disabled accounts.

Fix   UI-3531: Fixes an issue where the status of the cluster does not automatically get updated when a cluster is started in Notebooks.

Fix   UI-3550: The Whitelist IP option under the Control Panel UI now enables deleting multiple IP addresses at once.

Fix   UI-3630: Fixed an issue where there is no option to edit and update credit card details once subscription plan is changed from Standard Plan to Elastic Plan in the Control Panel > Subscription and Payment section.

With this fix, you can now change your credit card details even after changing your plans from Standard to Elastic.

Fix   UI-3650: Fixed an issue where on the new Scheduler page, the info icon pop-overs are not showing help tooltips.

 

Change  QBOL-4201: Support to disable an active account

Change  QBOL-5178: Users can now get the version numbers of the individual components of the product from the url api.qubole.com/about.

Change  QBOL-5229: Disable and Enable users by email in the REST API calls using the user endpoint.

Change  QBOL-5286: Uploading Airflow logs to an S3 location via s3cmd sync and disabling Airflow's inbuilt functionality to upload logs to an S3 location.

Change  QBOL-5378: Create default Qubole airflow connection (qubole_default) without password while launching an Airflow cluster.

Change  UI-3495: Search functionality in Help Center (icon near top right corner) now returns results from Knowledge Base and Qubole Documentation.

Change  UI-3552: Fixed an issue, where the repo tab (saved queries) was not working on the Analyze page when we land on analyze with a cluster label filter i.e, when the url was something like https://api.qubole.com/v2/analyze?cluster_label=xyz.

Change  UI-3643: Notebook action buttons will be hidden when cluster is down (readonly mode)

Change  UI-3669: In the Help Center, Search a topic will fetch and display results only from the Qubole Data Services category in Knowledge Base. Earlier, the results also included results from the Archived Articles section.

 


SPARK

 

New   SPAR-1001: Change: Idle executors those which have cache blocks of RDD will also be downscaled. This can be turned off by setting spark.qubole.autoscaling.memory.downscaleCachedExecutors to false

Change  SPAR-1047: Set the slave node type of a Spark cluster in a new account to r3.2xlarge and the master node type to r3.xlarge.

Change  SPAR-1052: Show the reason for executor termination in driver logs. The pattern to look for is removeExecutor.

 


TEZ

 

Fix   QTEZ-63: Show datetime in the Analyze UI job progress for TEZ jobs

 

 

ZEPPELIN/NOTEBOOKS

 

New   ZEP-311: Notebooks can be exported in a PDF format.

New   ZEP-340: Github integration with notebooks. This feature is available for Beta access. Send an email to help@qubole.com to get this feature enabled on a Qubole account.

 

Fix   ZEP-249: Fixed loading of dependencies at interpreter startup.

Fix   ZEP-328: Fixed the problem where the Scheduler already terminated error used to appear and the user could not use Notebooks after that.

Fix   ZEP-349: Fixed the deadlock in Notebooks where the entire UI becomes unresponsive.

 


List of Hotfixes Since 23rd June 2016


Fix   ACM-388: For users in default VPC account and they have not specified VPC, this causes the upscaling of nodes (spots only) get stuck. This change fixes the issue where autoscaling/downscaling will not be stuck.

Fix   ACM-419: Removed the bucket_location parameter from the s3cfg file which was causing notebook synchronization to fail if the user did not have getBucketLocation permission in AWS.

Fix   UI-3819: Fixed the issue of inability to re-run the failed steps in a workflow after deleting the success steps.

Fix   UI-2502: Fixed a bug where "clusters are not available" is displayed even though clusters are available.

Have more questions? Submit a request

Comments

Powered by Zendesk