Release Version: 34.0.0
For details of what has changed in this version of QDS, see What is New, New Beta Features, and List of Changes and Bug Fixes in this Release.
What is New
Airflow in Qubole
Qubole has introduced support for hosted Airflow clusters. Airflow is an open source data pipeline and workflow management platform. See Airflow in Qubole, quick start guide, and cluster API parameters description for more information.
Qubole has made significant enhancements that includes exporting notebook results and exporting a notebook as PDF. See Notebooks for more information.
Hive Authorization support for Presto commands
Hive Authorization can now be used for Presto commands. Hive authorization can be used to provide table-level access control to Qubole users. See Qubole Hive Authorization and catalog Hive properties for more information.
Qubole Help Center Search Result
Search functionality in the Qubole Help Center (icon near top right corner) now returns results from both Knowledge Base and Qubole Documentation. See Search Field for more information.
New Beta Features
Adding a Custom SSH Key Pair for an Account
Qubole now supports adding a custom SSH key pair on a cluster/account to provide security. This feature is not enabled by default. Send an email to firstname.lastname@example.org to enable this feature on a Qubole account.
GitHub Integration with Notebooks
Qubole now supports GitHub version-control integration with notebooks. This feature is available for Beta access and is not enabled by default. Send an email to email@example.com to enable this feature on a Qubole account.
List of Changes and Bug Fixes in this Release
AWS CLUSTER MANAGEMENT
New ACM-92: Allow option to add an SSH key pair per cluster/account. Write an email to firstname.lastname@example.org to enable it on an account.
Fix ACM-248: Added entries in the /etc/fstab for all the non-encrypted volumes that are mounted while starting up a cluster. This to ensure that the non-encrypted volumes are available even after the node is rebooted.
Fix ACM-334: Allow EBS disks to be used with r3, c3, and m3 instance types in ap-northeast-2 and eu-central-1 AWS regions.
Change ACM-350: Clusters now have their root disk on SSDs (GP2) for improved performance.
Fix HADTWO-470: Qubole has now reduced the maximum number of retries that an Application Master can do and the timeout interval that an IPC client can do, before killing the containers on lost nodes to speed up this cleanup process for the lost nodes.
Fix HADTWO-503: Resource Localization fails on a cluster due to existing cache directories.
Change HADTWO-231: Added support of EBS disks as regular volumes in Hadoop 2.
Change HADTWO-430: Support for encrypted communication in Hadoop
Change HADTWO-446: Decommissioning nodes will now be shown on the Resource manager UI. These nodes do not pick up new tasks as they will be downscaled shortly.
Fix HBAS-168: Fixes the issue of HBase snapshot restoration failure
Fix HIVE-1436: Fixed an issue due to which hive.mapred.local.mem was not respected to configure child JVM heap size
Fix HIVE-1431 and HIVE-1460: The Hive Execution Engine is now set to mr (by default) for Hive 0.13
New PRES-535: Presto supports hive authorization now. Grants and Roles have to be set up via Hive and Presto can use those to figure out if a user is allowed to perform the action or not.
Fix PRES-587: The query.max-memory-per-node property does not apply on a master node. As a result, now Qubole supports having smaller master nodes than slave nodes, which previously led to Presto server failure when bringing up slave nodes with optimised memory configurations.
Fix PRES-592: Fixed a memory leak in thrift metastore which was degrading performance in Presto and Spark clusters, which did not use metastore caching.
Fix PRES-624: Fixed leak of HTTP clients in Block Caching, which was exhausting file descriptors.
Change PRES-626: Presto used to throw invalid exception while reading empty ORC files. This fix pulls in a solution for this from open source.
Change PRES-636: Presto now supports Date columns as Partitions when table is in the Parquet format.
New QBOL-5250: Added ojdbc6 jar in sqoop package for Oracle db import/export support. Changes made in qweez package to include the jar in sqoop while compiling.
New QBOL-5273: Airflow Cluster Resources Links - Airflow Webserver and Celery Dashboard.
New UI-3297: New Airflow type cluster is available under the Control Panel > Clusters section.
New UI-3794: GitHub integration with Notebooks. This feature is available for beta access. Write an email to email@example.com to enable this feature on a Qubole account.
Fix QBOL-5416: Printing duplicate logs. Fixed the issue in the logging utility.
Fix UI-3521: The Quick Tour option is no longer visible on the Help Center for disabled accounts.
Fix UI-3531: Fixes an issue where the status of the cluster does not automatically get updated when a cluster is started in Notebooks.
Fix UI-3550: The Whitelist IP option under the Control Panel UI now enables deleting multiple IP addresses at once.
Fix UI-3630: Fixed an issue where there is no option to edit and update credit card details once subscription plan is changed from Standard Plan to Elastic Plan in the Control Panel > Subscription and Payment section.
With this fix, you can now change your credit card details even after changing your plans from Standard to Elastic.
Fix UI-3650: Fixed an issue where on the new Scheduler page, the info icon pop-overs are not showing help tooltips.
Change QBOL-4201: Support to disable an active account
Change QBOL-5178: Users can now get the version numbers of the individual components of the product from the url api.qubole.com/about.
Change QBOL-5229: Disable and Enable users by email in the REST API calls using the user endpoint.
Change QBOL-5286: Uploading Airflow logs to an S3 location via s3cmd sync and disabling Airflow's inbuilt functionality to upload logs to an S3 location.
Change QBOL-5378: Create default Qubole airflow connection (qubole_default) without password while launching an Airflow cluster.
Change UI-3495: Search functionality in Help Center (icon near top right corner) now returns results from Knowledge Base and Qubole Documentation.
Change UI-3552: Fixed an issue, where the repo tab (saved queries) was not working on the Analyze page when we land on analyze with a cluster label filter i.e, when the url was something like https://api.qubole.com/v2/analyze?cluster_label=xyz.
Change UI-3643: Notebook action buttons will be hidden when cluster is down (readonly mode)
Change UI-3669: In the Help Center, Search a topic will fetch and display results only from the Qubole Data Services category in Knowledge Base. Earlier, the results also included results from the Archived Articles section.
New SPAR-1001: Change: Idle executors those which have cache blocks of RDD will also be downscaled. This can be turned off by setting spark.qubole.autoscaling.memory.downscaleCachedExecutors to false
Change SPAR-1047: Set the slave node type of a Spark cluster in a new account to r3.2xlarge and the master node type to r3.xlarge.
Change SPAR-1052: Show the reason for executor termination in driver logs. The pattern to look for is removeExecutor.
Fix QTEZ-63: Show datetime in the Analyze UI job progress for TEZ jobs
New ZEP-311: Notebooks can be exported in a PDF format.
New ZEP-340: Github integration with notebooks. This feature is available for Beta access. Send an email to firstname.lastname@example.org to get this feature enabled on a Qubole account.
Fix ZEP-249: Fixed loading of dependencies at interpreter startup.
Fix ZEP-328: Fixed the problem where the Scheduler already terminated error used to appear and the user could not use Notebooks after that.
Fix ZEP-349: Fixed the deadlock in Notebooks where the entire UI becomes unresponsive.
List of Hotfixes Since 23rd June 2016
Fix ACM-388: For users in default VPC account and they have not specified VPC, this causes the upscaling of nodes (spots only) get stuck. This change fixes the issue where autoscaling/downscaling will not be stuck.
Fix ACM-419: Removed the bucket_location parameter from the s3cfg file which was causing notebook synchronization to fail if the user did not have getBucketLocation permission in AWS.
Fix UI-3819: Fixed the issue of inability to re-run the failed steps in a workflow after deleting the success steps.
Fix UI-2502: Fixed a bug where "clusters are not available" is displayed even though clusters are available.