Qubole Release Notes - 30-Sep-2015

Release Notes for QDS Release on 30th September 2015

New Features and Enhancements

 

Data Exchange

Qubole has introduced Data Exchange, a feature that enables an organization to share data and analytics with partners. With this feature, you can still retain complete control over access, permissions, and usage rights while granting access to the partners or members of other organizations. See Data Exchange for more information.

 

Spark Features and Enhancements

These are the new features:

In the Spark cluster settings user interface, a new option has been introduced to select a required Spark version from a list of supported Spark versions. See Spark cluster configuration for more information.

  • A new option to override default Spark configuration variables has been added in the Spark Cluster settings user interface. See Spark cluster configuration for more information.

  • Qubole Scheduler and Templates user interface now supports Spark commands. See Spark Schedule and Command template for more information.

 

HBase Features

These are the new features:

  • Qubole provides Hannibal HBase, which is an interface to monitor an HBase cluster, HBase Master to see the master and region server details, and a HBase shell to run queries. See Monitoring an HBase Cluster for more information.

  • An HBase cluster management interface to add, replace, and remove cluster nodes. See Managing an HBase Cluster for more information.

  • An HBase user interface to create backups and schedules. See HBase Backup and Schedule for more information.

 

Improved cluster credential configuration within the QDS user interface

 

All other minor changes and bug fixes are as listed below.

 

HADOOP

 

  • Recommission excluded and graceful shutdown nodes

 

 

HADOOP-2

 

  • Implemented the JobUpdater equivalent in Hadoop-2

  • Block downscaling if HDFS is running low on space

  • Enable recommissioning of decommissioning nodes while upscaling.

 

 

HBASE

  • Upgrade jackson version so that, it does not break against hadoop new version

 

 

HIVE-13

 

  • HIVE-1028: hive log dir perms for hive on master

  • HIVE-1106: Added a try catch block to detect the FileNotFoundExceptions in Warehouse.isEmpty(). Fix while deleting nested partitions

  • HIVE-1097: Updating the openx json serde version which has the fix to read sequence files

  • HIVE-1075: Upgrading to OS openx jars

  • HIVE-1056: Do not package hive hbase handlers

  • HIVE-821: Changes to HadoopJobExecHelper to make it compatible with new Hive 0.13

  • HAD-458: Comparator logic for computing summaries is made same as that of IPP comparator logic

  • HIVE-878 and HIVE-918: Fix NPE during stats collection

 

 

PIG-11

 

  • Changes in pig11/bin to use hive13 jars.

 

 

PRESTO

 

  • PRES-417: Support non-native Hive tables created via JDBC storage handler

  • PRES-435: JDBC StorageHandler improvement: Cancel Queries in backend DB when presto query cancelled.

  • PRES-436: Added knobs to control predicate pushdown compulsion in JDBC splits.

  • PRES-162: adding support of openx json serde

  • PRES-444 : Add gc logging

  • PRES-427: Enable CPU timers

  • PRES-378: Delete files in bulk for IOV

  • PRES-429: changing logging level for JVM metrics module

 

 

QDS

 

  • UI-2429: Add UI support to specify spark version

  • SPAR-599: Allow users to override recommended spark configuration

  • UI-1898: Add reordering and remove buttons for sub commands in workflow

  • SCHED-58: Concurrency of scheduled jobs will not be more than 2 for trial accounts

  • QBOL-4167: API for inviting new users

  • QBOL-4463: Raw result download option

  • Exposes API to get public ami-ids.

  • Add support for d2 node types

  • NEZ-22 Support for Nezha and SqlCommand

  • UI-2070: Added front end support for account cloning

  • state field added in json reponse to differentiate the hive-tables

  • Account creation API

  • UI-1799 and UI-1800: Manage and Backup for Hbase Command

  • Cluster start as ClusterManageCommand

  • UI-1743: HBase CRUD in control panel

  • QBOL-3929: New User Experience - Plan and Credential Management related changes

  • Metriks for collecting application metrics in tapp

  • HAD-443: Uses light hdfs check in scrub

  • QBOL-4185: Saves reason in case of cluster health-check-failure.

  • Add overcommit configuration file

  • UI-806: Added New Email Notifications for Comments Section.

  • Account-feature-based activation of recommissioning

  • Store/list the snapshot backup ids once backup is over.

  • UI-879: Added edit and remove comment functionality to Comments

  • QBOL-4666: Updates socksify version to make it compatible with Ruby 2.0

  • Add support for space in custom ec2 tags

  • UI-2474: presto form not getting submitted

  • PRES-473: Fixing query tracker page for presto queries

  • Fixes the failure when trying to update cluster settings by label.

  • SPAR-564: supporting --packages, --proxy-user and --repositories in spark commands from Analyze

  • QBOL-4635: Fix cluster termination for expired sessions

  • UI-2382: Bug fix: while refreshing manage/backup cluster page should redirect to same page

  • UI-2280: Font fix and fitting manage-cluster action list on screen

  • UI-2440: Fix for logs download link not working for failed commands

  • UI-2412 and UI- 2436: For new login, welcome screen looks weird.

  • UI-2401: AWS secret not getting saved intermittently.

  • Fixes the failure when trying to update cluster settings by label.

  • SPAR-563: Fixes related to handling of spark cluster config overrides

  • UI-2438: Input support plan details through the modal (pop-up)

  • QBOL-4564: Qubole-Managed accounts shouldn't accept 'Customer Public ssh Key'

  • UI-2409: Templates and Scheduler - s3 location for presto does not show input box

  • HIVE-869: A subtle message saying that we are using Hive tier

  • Fixes for UI-515 and UI-519 MapReduce widget issues

  • UI-1373: Added support for more data types sorting in results table

  • UI-2386: UI issue in refresh icon for Explore/Analyze

  • SPAR-542: Ensure use of proper hive metastore warehouse dir

  • UI-2217: Tutorials Integration - Phase I

  • UI-2123: Better way to show terminate reason

  • not able to create user using invitation link

  • UI-2321: Link to pricing page in payments and subscription

  • UI-2294: Manage account update/clear option not available, roles and keys account issues

  • SCHED-74: Scheduler frequency must greater than 0

  • QBOL-4524: Clean up of Mysql connections in the hive_scripts

  • HBAS-120: HBASE cluster should not be considered to be spark in UI

  • UI-1478 and UI-912: Tool tips for S3 upload/download to S3

  • UI-2174: Presto Default Config change based on recommendation

  • Fix issue with Mysql connection

  • UI-2283: Resources tab issues fix

  • UI-2304: Making cloned account as non default

  • HIVE-1019: Making use of can_use_qhs() to control the commands to hive server and it will be controller through account_limits

  • UI-2092: Fixed carriage return behaviour on signup page

  • UI-1546: No cancel button in new/edit/clone template

  • UI-113: redirecting user to login page if sign up with an email that has already signed up

  • UI-1614: Schedule - Nominal Date picker of Filter section in Show Instances is not working properly

  • QPIG-18: Changes in pigcli.py to automatically include hcatalog for pig2

  • UI-1501 Wrongly showing error message of interval less than 60 days

  • SPAR-503: change maximum attempts in Spark to 1

  • Use dry run when reading security groups to check if credentials are valid

  • HBAS-112: Converting parameters hash of cluster-manage-commands to string

  • HBAS-113: Forcing user to choose ssh-keys on UI for Hbase Cluster

  • UI-2207: Warning message thrown while stopping Clusters

  • HBAS-105: Displaying warning about cluster-node change while running add/remove node for Hbase Cluster

  • UI-1845: Notification emails are overflowing in the account settings

  • UI-2021: Fix the case of missing spark notebook link

  • Custom EC2 tags should not be pushable

  • UI-2174: Default Presto Cluster Config Seems Off

  • QTEZ-31: Start YARN timelineserver and Tez UI on hadoop2 clusters

  • UI-2202 Control Panel Minor Fixes

  • Fixed editing of templates from Pig Command to Hive Command

  • Do not print the hgc_master command

  • UI-1766: rewrite logs/results widget in analyze

  • UI-1346 Explore - In Add Repository, Info link leads to 404 error

  • HAD-469: Fix for various issues hit by Indix

  • HBase: Override Hbase Configuration Variables is not getting saved.

  • UI-2091 and UI-2087: Hbase Config override and set as default fix

  • UI-1186: Improving the DBImport Compose Pane.

  • HBase: Override Hbase Configuration Variables is not getting saved.

  • UI-2093: Do not set up scheduled snapshot for qubole managed clusters

  • QBOL-4109: JT URL across accounts

  • UI-2010: SHOW CREATE TABLE: map columns cut off in UI

  • UI-2091 and UI-2087: Hbase Config override and set as default fix

  • Displaying error for node-count 0 while creating HBase cluster

  • Do not allow changing cluster type when Hbase is involved

  • UI-1783: remove c1.medium from master type

  • UI-1793: disable presto popup

  • UI-950: CSS files not getting minified

  • Aws Availability zone is pushable in cluster details page

  • Use ShellCommand for backup

  • Schema support in fetching S3 location from hive metastore

  • HIVE-956: Use hive 1.2 using account limits

  • UI-2011: Confirm Query bug in templates

  • Fix typo for hbase zeppelin interpreters

  • UI-1178: Fixed broken help links in scheduler new/edit/clone/list

  • UI-1483: Search box in Exploring Database and Tables - challenge using it

  • UI-2002: Change error code for refresh_nodes

  • UI-1986: When Analyze page is refreshed after a while, it shows a irrelevant modal box.

  • Logging details who aborted the query

  • IP whitelisting issue with /32 mask

  • UI-1865: Allow switching accounts from the "My Accounts"

  • UI-1928: Upgrade font-awesome and replace add user icon

  • Reposition override to S3 button

  • QBOL-4075: Remove usage of spark.use_hadoop2_clusters_for_spark global setting.

  • Make Hbase Backup/restore executable anywhere

  • UI-2391: Added tooltip to main menu icons

  • QBOL-4535:Remove redirect loops from IP Whitelisted accounts

  • QBOL-3874: Performance improvement in account limits

  • QBOL-4445: Making Account Name as Mandatory argument for account clone

  • Look at global settings only if the account settings are inline

  • UI-1644 and UI-1955: Control Panel Account settings UX Improvement

  • UI-2048: command template edit/clone/new action redirect to landing page

  • UI-1984: Permalink and Clone Icons icon fix

  • HADTWO-273: Improve Hadoop-2 cluster startup time

  • QBOL-4174: Do not ssh as root to defaultdj

  • Make hbase interpreter default and remove other interpreters from zepplin for hbase

  • HBAS-70: Fix tests for Cluster Manage commands

 

SPARK

  • Added spark-defaults.conf.qubole to spark repo for SPAR-599

  • SPAR-624: Remove the twitter assembly from spark package since the correct package should not have twitter assembly

Have more questions? Submit a request

Comments

Powered by Zendesk