Qubole Release Notes - 06-Aug-2015

Release Notes for QDS Release on 06 August 2015

Major Enhancements

 

Spark 1.4.0 is available

Spark-1.4.0 is generally available. QDS now supports SparkR support via the UI and SDK. You can also look at the open source release notes here. You can continue to use spark-1.3.1 by following the Spark Downgrade Steps.

 

Multi-Tenant Spark History Server

Qubole now supports accessing logs of spark applications even after the cluster is terminated. Qubole provides this ability using a multi-tenant spark history server. Look forward to a blog about this in the near future.

 

Granular permissions to access Cluster resources

In the Roles pane of Control Panel, Cluster resource had only read and manage permissions before, with the manage permission encompassing many different operations. With this update, users now have the ability to create roles for specific cluster operations with permissions such as create, read, update, delete, start, terminate, and clone. See Roles.

 

Cluster Nodes vs Time Chart

In the Overview (now known as Usage) dashboard, Qubole has added a new chart that now displays how many nodes were spawned over a time period. This will be rolled out gradually over the next few days.

Hbase Self-service UI

In the Clusters pane of Control Panel, Qubole now allows you to create a HBase cluster and edit its configuration and delete Hbase clusters. See HBase in Qubole for more information.

 

Logs Results Widget rewrite

The logs and results section in analyze has been rewritten to fix some critical bugs. This is not open to all users and will be rolled out to everyone gradually.

 

Reduce 500 errors from QDS API

 

All other minor changes and bug fixes are as listed below.

 

HADOOP

 

  • HAD-377: Optimizing IPP for avoiding bad prefixes while handling multiple tables

 

 

HADOOP-2

 

  • HADTWO-266: Allow AMs to be scheduled on volatile nodes when the minimum size of the cluster is small and autoscaled nodes are 100% spot.

  • HADTWO-295: Mark ElasticSearch related params as final for H2

 

 

HIVE-13

 

  • QTEZ-19: Print Tracking URL for tez queries.

  • HIVE-968: Fix for external tables creation after managed tables default to ORC.

 

 

PRESTO

 

  • PRES-387:. Fixing wrong results and perf with kinesis connector

 

 

QDS

 

  • QBOL-4314: Edit API for schedulers

  • push ganglia metrics to datadog

  • QBOL-4178: Adding support to add support plan through the super admin console

  • Add support for restrict_ssh in api

  • Set default datadog alerts

  • Enable node rebalancing by default

  • Refresh VPC button does not refresh the VPCs or Subnet

  • UI-2198: Fixed Log download does not work for cancelled/failed commands

  • HBAS-111: Changing UI-tooltip message and ‘node-count’ label to ‘region server’ count

  • UI-2205: Show Password is not working in Data Store added under Explore

  • UI-1766: Rewrite logs/results widget in analyze

  • UI-1976: Unable to delete tags from past command instances.

  • UI-2091: UI-2087 Hbase Config override and set as default fix

  • UI-2093: Do not setup scheduled snaphsot for Qubole-managed clusters

  • UI-2011: Confirm Query bug in templates

  • UI-1765: Load Hive Bootstrap when needed

  • UI-609: Create table in Hive Wizard UI improvements

  • UI-1987: Analyze - FSP UI issue fix

  • UI-1537: Added Support for non-default Schema in Scheduler Dependencies

  • UI-1912: Explore - While creating hive schema from S3 bucket, if delimiter is switched, it shows improper columns

  • UI-1922: Changed Overview dashboard to Usage dashboard in Qubole

  • HADTWO-248: Pass stable-spot settings to cluster.

  • UI-1830: Added Ganglia monitoring to resource column in cluster tab

  • A new api to return job-tracker-url of all clusters across all accounts of a user

  • UI-1825: Popover on hover fix

  • QBOL-4166: Fix hadoop2/spark/hbase reset

  • Cluster label will be sent in email when any cluster configuration is changed including label

  • UI-1813: DB export/import issue fixed in Analyze

  • UI-1703: Help Buttons in Templates screen leads to 404 page and require clarification

  • HIVE-886: Fix for dbimport/export failures

  • UI-1276: Fix for Explore - 'Qubole supports SSL mode for postgres' is shown when not required

  • UI-1609: Improving page load time for Analyze

  • UI-1222: Changed repository to datastore in Explore

  • UI-1590: Validation Stable spot bid percentage must be greater than volatile node bid percentage

  • UI-1527: Help on Volatile and Stable spot instance settings

  • SQL must take Spark submit arguments specified by user

  • UI-2048: Command template edit/clone/new action redirect to landing page

  • UI-1854: Help Widget Phase II

  • QBOL-4098: Account Creation with current plans

  • QPIG-17: pig-on-hadoop2 commands supported as top level command.

 

SPARK

 

  • SPAR-254, SPAR-251: Compress spark event logs by default.

  • SPAR-249: Move spark event logs to S3 on program exit.

  • SPAR-448: Add support for SparkR in Analyze.

  • SPAR-443: Clean up primary jar at job exit.

  • SPAR-208: Allow avro to be used by spark. Make changes for working with Qubole-hive13. Also changes for allowing cluster mode to deal with hive queries.

  • SPAR-174: Configure /usr/lib/spark/lib as the standard jar destination for spark. Any jar in this dir is available to both driver and executor. We also use this mechanism to make hive_contrib.jar (JsonSerde) accessible to SparkSQL.

  • SPAR-330: Use shaded kryo jar in spark-hive shim code.

  • SPAR-502: Create kinesis assembly and make this available in Spark Clusters.

 

 

 

 

SPARK Notebooks

 

  • SPAR-357: If settings are not correct for an interpreter, then an error message should be communicated to user.

  • Update Akka version in Zeppelin.

  • Approach to add jars in Zeppelin statically and dynamically.

Have more questions? Submit a request

Comments

Powered by Zendesk