AWS Lambda: An error occurred (403) when calling the HeadObject operation: Forbidden

Question

I have a lambda function making a s3 HeadObject call. Even though there has been a custom policy made for this lambda function, I keep getting an 403 error whenever this HeadObject call is made. There is specifically;

An error occurred (403) when calling the HeadObject operation: Forbidden

My policy very clearly allows GetObject calls for the bucket in question. Not sure what the problem is. I have triple check that it all lines up. The line in question is:

    s3 = boto3.client('s3')
    local_file_path = '/tmp/' + key_name.split('/')[-1] + '_REMOTE.json'
    response = s3.head_object(Bucket=environ['OUTPUT_BUCKET'], Key=OUTPUT_FILE_NAME)

Let me know if I can provide more info to help

Solution 1

Since you said the file is missing its an expected behaviour if you're missing s3:ListBucket permissions

You need the s3:GetObject permission for this operation. For more information, go to Specifying Permissions in a Policy in the Amazon Simple Storage Service Developer Guide. If the object you request does not exist, the error Amazon S3 returns depends on whether you also have the s3:ListBucket permission.

If you have the s3:ListBucket permission on the bucket, Amazon S3 will return a HTTP status code 404 ("no such key") error.

If you don’t have the s3:ListBucket permission, Amazon S3 will return a HTTP status code 403 ("access denied") error.

Link to the doco

There is also a "different" eventual consistency behaviour for doing HEAD before uploading the object

Amazon S3 Data Consistency Model Amazon S3 provides read-after-write consistency for PUTS of new objects in your S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD or GET request to the key name (to find if the object exists) before creating the object, Amazon S3 provides eventual consistency for read-after-write.

Solution 2

I found the solution on this link, so all the credit goes to the author.

Basically you might need to check you have the right permissions ( s3:GetObject and s3:ListBucket at least ) for the operation and the resource expression matches the ARN plus a path e.g this -> "Resource":"arn:aws:s3:::BUCKET_NAME/*" rather than this -> "Resource":"arn:aws:s3:::BUCKET_NAME".