Most of the time s3 eventual consistency bites us in listing API. Typically when we do a listing of a directory in s3 in which we have recently deleted or added some files, the listing API can either show us “ghost” files: files which have been deleted but still present in the listing or miss “conceived” files: files which have been added but were not listed by the listing API.
Ghost files cause problems because when doing any operations with such a file, we are likely to fail with FileNotFound exception.
Conceived files usually don’t cause any problem in the Spark job level. They cause problems because of missing data in the final results.
At Qubole we have addressed these issues at some of the problematic areas. Here is how you can identify which solution would be appropriate for your use case.
Qubole Spark uses DirectFileOutputCommitter by default. This helps in avoiding listing API which is used in the jobCommit phase of the usual FileOutputCommitter. By writing directly to the final location DirectFileOutputCommitter avoids the jobCommit phase all together. But when doing insert into/append operations on spark tables, spark forces the use of FileOutputCommitter. FileOutputCommitter v1 does a copy operation in the jobCommit phase which not only takes lots of time, but is also likely to fail because of eventual consistency issues.
Qubole offers following two settings for dealing with eventual consistency issues in the FileOutputCommitter:
If using FileOutputCommitter v1, set:
Parallel merge paths makes the jobCommit phase run in multiple threads, making is faster and also retries various procedures if they fail, instead of giving up on first error.
Or use FileOutputCommitter v2:
Writing to Hive Tables
Eventual consistency issues are also common when writing to hive tables from spark. Hive tables writes are done by first spark writing the results to a temporary location on s3, followed by rename/copy operations to move files from temporary s3 location to final table location. If writing to hive tables, please set:
This setting improves the speed of file copy operations by parallelising it using threadpool and also handle common failure scenarios by automatically ignoring or retrying operations as appropriate.
(Available from Jan 2018)
This setting does away with writing to temp location and copying files, instead it directly writes files to final table location. It will not only improve speed of writing to hive tables from spark, but by avoiding writes and s3 listing, will avoid running into eventual consistency issues.
Reading from recently written tables
Many times we have seen users first write to a location and then immediately read it in the next line of scala/python code. Such tables are susceptible to eventual consistency failures. Usually this appears in the form of FileNotFound exception. If possible, avoid such code patterns. The following setting will come handy if you cannot avoid the pattern:
As you can guess we are simply ignoring the “ghost” files here, which were listed by the s3 listing API. Another reason for this issue could be concurrent delete by some other job.
Another advanced API for generally ignoring all kind of errors during read operations is:
From spark documentation:If
spark.sql.files.ignoreCorruptFiles is true, the Spark jobs will continue to run when encountering corrupted or non-existing and contents that have been read will still be returned
DFOC & speculation
By default Qubole disables speculation. DirectFileOutputCommitter (Qubole default) doesn’t work well with speculation. If you have a need to enable speculation, please make sure to disable DirectFileOutputCommitter.
Many thanks to Rohit Karlupia for his help and guidance.
- On eventually consistent file listing: http://chakpak.blogspot.com/2017/07/on-eventually-consistent-file-listing.html
- From FAQs at AWS: http://aws.amazon.com/s3/faqs/:
- Eventual Consistency -- http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyMode
- Regions - http://docs.aws.amazon.com/general/latest/gr/rande.html
- Best Practices - http://docs.aws.amazon.com/redshift/latest/dg/managing-data-consistency.html
- Additional How To: https://qubole.zendesk.com/hc/en-us/articles/204486155