Qubole Release Notes 12-May-2016

Release Version Number: 32.0.0

 

For details of the changes in this QDS version, see What is New and List of Changes and Bug Fixes in this Release.

 

What is New

 

AWS CloudFront Content Delivery Network

CDN has been enabled on api.qubole.com on 16th May 2016 as informed earlier (that is mentioned below).

Qubole will enable AWS CloudFront content delivery network (CDN) on api.qubole.com on 16th May 2016.


New Features/Enhancements in Spark

  • Spark version 1.6.1 is available with this QDS release
  • Customer Data Stores registered in QDS can now be accessed automatically via Spark clusters
  • Spark/Zeppelin Application UI is accessible through a Spark UI button available in a Spark Notebook. The UI shows the list of jobs’ status.
  • Notebook Library which contains a list of sample notes that Qubole provides

 

Support for Private Subnets in VPC

QDS now supports launching clusters in private subnets:

  • Pig, Presto, and Hive on master commands are now supported. 
  • Hive, Hadoop, Shell and Spark commands were already supported in a previous QDS release.
  • UI and SDK support will be available in a forthcoming QDS release.
  • This feature is not available by default. Write an email to help@qubole.com for more information.

 

Datadog Cloud Monitoring on Hadoop-1 Clusters

Qubole now supports Datadog cloud monitoring service on Hadoop-1 clusters.

Datadog is not supported in all accounts by default. To enable this feature in a

QDS account, write an email to help@qubole.com.

 

New Version of Columnar Cache for Presto Clusters

A new version of data cache is added to Presto which improves warm-up times

with respect to existing File Caching for columnar formats. It is recommended to

use the data cache for IO intensive workloads that access same data multiple

times. To enable data cache, write an email to help@qubole.com.

 

List of Changes and Bug Fixes in this Release

 

CLUSTER MANAGEMENT

 

New ACM-50: Supports Pig, presto and Hive on master commands in Private Subnets.

New ACM-90: Select AZ upfront based on spot prices for pure Spot clusters - please contact solutions@qubole.com to enable this feature

 

HADOOP-1


Fix HAD-549: Cluster doesn't downscale even on reducing max cluster size.

Fix HAD-573: Enable Lz4 codec by default in Hadoop1

 

HADOOP-2

 

Fix HADTWO-427: Aggressive autoscaling was happening for long running mapper tasks

Fix HADTWO-410: DeadLock in MR job due to mapper starvation and no-preemption

Fix HADTWO-406: Adding IPP Optimisation in S3A FileSystem.

Fix HADTWO-402: Adding support for streaming multipart upload in S3A Filesystem

Fix HADTWO-419: AMRMClientImpl does not update AMRM token properly

Fix HADTWO-420: AMRestart causing timeouts due to bad yarn.resourcemanager.pending.app.timeout.minutes calculation.

  

HIVE 0.13

Fix HIVE-1341: Printing exception traceback along with detailed error message for Hive errors.

Fix HIVE-1330: Changing the stats temp directory to MR scratch directory. 
 

HIVE 1.2

 

Fix HADTWO-417: Adding support of S3A FS in Hive1.2

Fix HIVE-1330: changing the stats temp directory to MR scratch directory.

Fix HIVE-1341: Printing exception traceback along with detailed error message for Hive errors.

Fix HIVE-1364: Disabling fs check by default for Hive Authorization.

 

PRESTO 

 

New PRES-112: Improve display of progress information in console logs

New PRES-504: Add support for new block caching filesystem

New PRES-593: Adding property to enable server side encryption for hive.

New PRES-569: Expose hadoop overrides section for Presto clusters

Fix PRES-589: Disabling the feature of running small queries via CLI for Presto, after this change each Select or With query will be converted to insert-overwrite-directory irrespective of the limit clause.

 

QUARK

 

New NEZ-125: Upgrade Nezha to Quark 5.x 

 

SPARK 

 

New SPAR-875: Support for querying Data Stores from Spark

New SPAR-923: Adding spark version 1.6.1

Fix SPAR-739: Make --jars option work with Spark SQL

Change SPAR-865: Speed up SparkSQL split computation

 

ZEPPELIN/NOTEBOOK 

 

New ZEP-144: Ability to access spark UI from Zeppelin

 

QDS

 

New QBOL-3549: Ability to add users and roles to groups by email ID and role name 

New QBOL-4694: Expose new API endpoint for commands that has both status and results

New UI-2888: Added warning when a query that runs on presto does not autoscale the cluster

New UI-3266: Datadog fields in the Account Settings page

New UI-3345: Notebook Library UI Integration

New UI-3485: Search historical commands via API

Fix ACM-180: Add timestamps to UDF log files

Fix MOIR-11: Monitoring of Autoscaling Process in Discovery Server

Fix QBOL-5085: Shell Command to list S3 Files/Archives failing for an IAM Roles based account

Fix QPIG-33: Pig15 is available from shell command

Fix UI-1239: v1 Scheduler - Improper selection and error message for TimeZone fix

Fix UI-1952: Reset does not reset emails in account settings details

Fix UI-3188 Scheduler v2: Fix for System user is able to Suspend someone else’s job.

Fix UI-3261: Command Name wiped when syntax error and tooltip problem

Fix UI-3332: Command Template Popup icon not working properly

Fix UI-3369 - Remove two default location for AWS roles

Fix UI-3370: Scheduler v2 show rerun only on hover

Fix UI-3386: Scheduler V2: Removing last tag is not getting saved

Fix UI-3405: V1 Scheduler table metadata modal not usable.

Fix UI-3419: FCN UI shows null for cluster selection

Fix UI-3430 Removal of qubole user id from permalinks.

Fix UI-3441: Account Reset doesn't work properly.

Fix UI-3444: Change from Dashboard to Usage for uniformity.

Fix UI-3449: Support Hadoop-2 clusters in dbimport/export use hadoop cluster selection

Fix UI-3451: Hitting JS error while loading queries from Scheduler History

Fix UI-3456: Scroll to the top automatically on error in scheduler

Fix UI-3499: Mouse Hover on Scheduler Runs, hides the Suspend, Kill, Clone, and Edit Links

Fix UI-3508: Tags keyword search for instance not working

Fix UI-3559: Templates : Not seeing status icon for Templates in History section.

Fix UI-3590: Command ID Link on Scheduler v2 Page Redirects to Compose Query on Analyze

Fix UI-3599: Logs are not refreshed when switching between commands in the history under a filter

Change QBOL-5092: Notebook Library Improvement

 

List of Known Issues

 

These are the known issues:

  • SPAR-942: Support for remote py Files. The --py-files option is completely broken in 1.6.x.
  • UI-3657: Notebook - Example Notebook should clone and not copy on clicking the Copy to My Notebook option.

 

List of Hotfixes Since 03rd May 2016

 

Fix UI-3615: Support Plan Options have changed

Fix UI-3618: Sign-up using SAML is broken

Fix UI-3478: add composite index in query_hists table for account_id + submit_time

Fix ACM-233: Add capability to send cluster health check failure alert to customers

 

List of Hotfixes After QDS Release on 12th May 2016

 

Fix HIVE-1413 - hive1.2 - Dynamic Partition loading detecting wrong partitions

Fix HIVE-1390 - [Hive 0.13] Parquet JOIN with ORC not combining files ( resulting in ton of mappers )

Fix HAD-587 - Remove Lz4codec for codec list COMPLETE

 

Have more questions? Submit a request

Comments

Powered by Zendesk