How To: Write To S3 Bucket

By default, when you write an object to an s3 bucket, you are the only one able to read that file. This is also true even if you write an object to a bucket you do not own. In many cases this can be problematic as the bucket owner would not have access to read files in their bucket written by other users. To allow the bucket owner to read data written by a non-bucket owner apply this setting:

In Hive:

set fs.s3n.acl=bucket-owner-full-control;

In Hadoop:

fs.s3n.acl=bucket-owner-full-control

Have more questions? Submit a request

Comments

  • Avatar
    Mike Roark

    sc._jsc.hadoopConfiguration().set("fs.s3a.acl","bucket-owner-full-control")

  • Avatar
    Sandeep Badam

    Just to add, Any of these will work:

    sc._jsc.hadoopConfiguration().set("fs.s3a.acl","bucket-owner-full-control")
    sc._jsc.hadoopConfiguration().set("fs.s3n.acl","BucketOwnerFullControl")
    sc._jsc.hadoopConfiguration().set("fs.s3a.acl","BucketOwnerFullControl")

  • Avatar
    Kim Chew

    How about in Spark? I have tried this in Notebook but no effects,

    spark.sparkContext.hadoopConfiguration.set("fs.s3n.acl","bucket-owner-full-control")

  • Avatar
    Sandeep Badam

    Kim, as we discussed there was a restriction in writing to your cross-AWS account. It was resolved after acquiring required permissions.

    Otherwise these statements will work at Spark job level:

    sc._jsc.hadoopConfiguration().set("fs.s3a.acl","bucket-owner-full-control")
    sc._jsc.hadoopConfiguration().set("fs.s3n.acl","BucketOwnerFullControl")
    sc._jsc.hadoopConfiguration().set("fs.s3a.acl","BucketOwnerFullControl")

  • Avatar
    Minesh Patel

    Update- Recommendation is to use the following for Hadoop2 / s3a:

    fs.s3a.acl.default=BucketOwnerFullControl

     

    For s3n:

    fs.s3n.acl=bucket-owner-full-control

    Edited by Minesh Patel
Powered by Zendesk