What is New
Support for Private Subnets in VPC:
QDS now supports launching clusters in private subnets:
- In this release, Hive, Hadoop, Shell and Spark commands are supported.
- In a forthcoming release, support for Pig, Presto and Hive on master commands will be added.
- This feature is not yet available for self-service access. Please contact firstname.lastname@example.org to learn more.
Spark 1.6.0 (beta version) is now available
The Presto version is upgraded to 0.119
Revamped Scheduler UI
The user interface for scheduled jobs has been revamped to be lighter, faster, and simpler. This interface will be rolled out gradually to all users.
Ability to edit cluster node bootstrap
Editing node bootstrap files can now be done directly in the UI. In Control Panel > Clusters pane, click the dropdown in the Actions column to find the option to edit node bootstrap file. This will also be rolled out gradually to all users.
Tagging support for Notebooks
User can tag a notebook for better manageability. User can also search for notebooks by tags.
Minor UI Enhancements
These are the minor UI enhancements:
- Revamped global navigation
- Monospace font for Logs tab in Analyze UI with an option to wrap text for better readability
- Ability to see Job Tracker URLs in Resources tab in Analyze UI for Hadoop-2 and Spark jobs
- Several bug fixes
List of Changes and Bug Fixes in this Release
New Adding Sync Functionality in Hadoop-2
Fix Autoscaling implementation for TEZ jobs
Fix Job submission is failing frequently due to an HDFS replication failure
Fix Closing the inputStream, before doing the seek() operation
Fix Added support for S3A filesystem in Hadoop 2
Fix Minimum Cluster size option has been made refreshable.
Fix Show node loss messages when node is lost due to spot prices or any other reason.
Fix Change default decommissioning start time to 47 minutes
Fix Do not timeout after 5 successive retries of NM fails. This messes up recommissioning.
Change Fix TEZ UI by using SOCKS proxy in YARN TimelineClient to communicate with YARN ATS
Fix Fixed case-sensitivity issue in aggregate function throwing IndexOutOfBoundsException
Fix Added map task progress for TEZ queries. This was required for TEZ autoscaling to calculate the number of nodes to autoscale.
Fix Schema and Table name can not be null in Cubes/Partitions
New Add support for ElasticSearch Storage Handler
New Package avro jars to make avro tables accessible in Presto
New Add Authentication and SSL Encryption over JDBC Drivers
New Added `hive.allow-drop-external-table`config parameter to disable/enable dropping of external tables from Presto
New Allow compression while inserting into Hive text tables, controlled via`hive.exec_compress_output=true` session config
Fix Fixes to make Presto work with DBVisualizer
Fix Process queries with only comments ('--') correctly
Fix Disabled `redistribute-writes` by default to prevent wrong results in OrderBy queries
Fix Fix NPE in ranking empty objects (Backport of PIG-3726, QPIG-30)
Fix Fix setStoreFuncUDFContextSignature NPE (Backport of PIG-3120, QPIG-26)
Fix Fix ConfigurationUtil.getLocalFSProperties returning inconsistent property set (Backport of PIG-3677, QPIG-26)
New Added new login and signup pages (UI-2884)
New Hive Authorization
New Adding command exceptions in log file (QBOL-4768)
New Support streaming logs via sdk (QBOL-4902)
New Spark Job Server UI in analyze spark command (UI-2582)
New Added Node Bootstrap editor (UI-2696)
New Use metriks-addons v3.1.4 to report rails metrics (QDSOP-687)
New Support for private subnets
New Added account_settings for bulk delete rollout.
New Permalinks and filters support for saved queries (UI-2327)
New Move Presto to run on Hadoop-2 clusters
New Scheduler support for saved queries (UI-1954)
Fix Explore Table Search does not work (UI-3080)
Fix Removed history popover (UI-2759)
Fix Fix VPC and subnet validations for a cluster outside the US-East region
Fix Only 1 DS-Set is allowed per account.
Fix Fix issues with blank data during results download (QBOL-4963)
Fix Removed alias from count(*) type select clauses (UI-1908)
Fix The job history server gets killed due to OOM issues.
Fix Creating a thrift server executable to start a HiveServer with Hadoop-2 and Hive2 jars.
Fix Analyze - Improve history list first time load results.
Fix Replacing non valid chars from result set (QBOL-4809)
Fix Account limits flag to control qubole.mysql.compatibility.check
Fix No stdout if mapreduce.map.memory.mb > yarn.app.mapreduce.am.resource.mb
Fix cluster labels get reset for spark commands in edit mode ( UI-2670)
Fix Validation fix for cluster nodes (QBOL-4778)
Fix Use current_user email if no other valid email is provided while creating Scheduled Job (QBOL-4756)
Fix Composer actions in analyze gets hidden when the screen resolution is
Fix Fix issue with account_id in query hist authorization (UI-2607)
Fix Access Control for user attributes update (QBOL-4744)
Fix Fixing cpl for roles enabled accounts (QBOL-4741)
Fix Explore - My Amazon S3 Custom path refresh not working (UI-2577)
Fix Explore meta data cache to use schema name for caching (UI-2416)
Fix Keep user selection in the logs results tab in analyze (UI-2546)
Fix Saved Queries to remember cluster label and fsp value (UI-2452)
Fix Not able to start clusters which needs tunnel server after switching to IAM Roles
Fix Updating storage setting should not update the persistent security group at account/cluster level
Change UI-2630: Added warning for 0 EBS volume selection for c4/m4 instance types
Change UI-2780: Remove left navigation and move to top (behind the setting) Restructured help widget
Change QBOL-4895: Optimize mysql query to fetching sessions in cleanup worker
Change Monit support for the following daemons: ResourceManager, NameNode, Yarn ApplicationTimelineServer, MapReduce JobHistoryServer in addition to NodeManager and DataNode
Change SPAR-748: Allow delete of notes from active cluster
Change QBOL-4080: Fix md_cmd for spark SQL commands
New SPAR-844: Support Spark 1.6.0
New SPAR-476: Support Scala-2.11 for Spark 1.5.1
New SPAR-298: Capture Spark S3 related metrics at end of a job
New SPAR-802: Add a wrapper class for HiveThriftServer2
New SPAR-786: Add helper command /usr/lib/spark/bin/qcmd
Fix SPAR-809: Increase max-connections of httpclient library
Fix SPAR-733: Bundle avro-1.7.7 into spark assembly
Fix SPAR-824: Add conf dir to head of classpath to allow proper logging
Fix SPAR-150: Add Hadoop native libs to Spark libpath
Fix SPAR-835: Fix insert overwrite over dynamic partitions
Change SPAR-808: Use qubole-hive-0.13 to talk to the metastore server
Change SPAR-786: Add back kafka jar into spark assembly
Change SPAR-784: Compile spark with qubole-hive-1.2 instead of apache-hive-1.2
Change SPAR-758: Build spark on a Darwin/Mac OS
Change SPAR-780: Monit support for Spark history server
Fix Autoscaling implementation for TEZ jobs.
Fix Application UI link is navigated to application specific page for running a job
Fix Proxify links in application tracking URLs.
New Added Tagging support in FCN UI (ZEP-91)
Fix Allow zeppelin to start when there are empty JSON. (ZEP-183)
Fix Do not sync when note.json is empty (ZEP-184)
Fix Fix state of icons when note feature is disabled (ZEP-178)
List of Known Issues
The Spark version 1.6.0 does not support autoscaling and writes to avro files (SPAR-893 and SPAR-894).