Enable feature : Https communication for S3A

By default, Qubole implementation of S3A talks to AWS S3 buckets using HTTP protocol. If you would like to change it to https, here is the process in 3 simple steps:

 

1. Create a ticket with Qubole Support to enable an account feature.

2. You have to add a set of permissions for the bucket. A sample S3 policy would be:

{ 
"Version": "2018-03-15", 
"Id": "some_policy", 
"Statement": [ 
{ 
"Sid": "HTTPSOnly", 
"Effect": "Deny", 
"Principal": "*", 
"Action": "s3:*", 
"Resource": [ 
"arn:aws:s3:::<bucket_path>/*", 
"arn:aws:s3:::<bucket_path>" 
], 
"Condition": { 
"Bool": { 
"aws:SecureTransport": "false" 
} 
} 
} 
] 
}

3. The last step is to add below Hadoop overrides for the cluster and restart it:

fs.s3.https.only=true
fs.s3a.connection.ssl.enabled=true

Let the Qubole support team know in case you see any problems. 

Have more questions? Submit a request

Comments

Powered by Zendesk