Error: SemanticException Cannot merge files

Symptom

While running a query in Hive an error is returned which contains the following phrase:

'FAILED: SemanticException Cannot merge files. Please contact Qubole Support to enable move operation on this FileSystem.'

Cause

By default the Hive Merge File functionality is disabled for S3 and Qubole generally does not recommend using this hive merge file feature due to performance problems on S3 file system. This feature is designed for classic HDFS and setting up these parameters will have an impact on the job run time as it will increase the running time of the queries. In S3 the "move" operation is not supported and S3 filesystem does copy and delete operations and these are increased when using the Hive Merge File feature.

Action

This feature can be enabled with: set hive.allow.move.on.s3=true;

As an alternative consider increasing the split size (or) using the DISTRIBUTE BY feature (or) using ORDER BY feature depending on the requirement.

Have more questions? Submit a request

Comments

Powered by Zendesk