com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: XXXXXXXX)

Question

From few days back i am receiving this exception when i try to push files to my S3Bucket. Ealier everything seems to work and i am sure there is no code changes from my side.

com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden
(Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden;
Request ID: XXXXXXXXXXXX),
S3 Extended Request ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3699)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:999)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:977)

.... ....

I came across many such Q related to com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden following those

1. I have installed NTP on my server to solve any time related issue.
2. I have also added endPointUrl for "AmazonS3Client" object to the code which i feel may solve my problem.

Anything else i can try to solve this issue

i am using aws-java-sdk:1.9.10 for pushing files to S3 Bucket.

Solution

Most likely your instance has not been launched with an IAM instance profile role that has access to S3.

All access to AWS services must be signed with access key and secret. When you do this from your local machine the DefaultCredentialsProviderChain uses the access key and secret defined in your .aws/credentials file.

When you launch an EC2 instance in AWS it also needs to sign the requests to services, like s3. However, it does this by retrieving it's credentials from an internal metadata service.

So what you do is create an IAM instance profile that your instance will assume when it starts up. This IAM instance profile, like other IAM profiles for user's for example, defines what the instance has access to.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

Comments

[NetRocks]

I am having a similar problem trying to access S3 from pyspark. Background: My goal is to learn Spark using Python. Don't want to jump into EMR or HDSF etc yet. So, I created a S3 bucket put some files in and then created simple Ubuntu EC2 instance but forgot to attach IAM role. I got pyspark to work but I am trying to access S3 file using Spark and getting the above error. I also got the hadoop-aws-2.7.1.jar & aws-java-sdk-1.7.4.jar files because I didn't install Hadoop. I created IAM role with no permission & assigned to my still running EC2 instance. Still getting error. Any thoughts?