Cloudfront returns 403 for a minute after upload

Question

I have a cloudfront distribution in front of an s3 bucket. Cloudfront urls are restricted, and keys must be url signed. The s3 bucket is also restricted, allowing only cloudfront to read

Sometimes, after I successfully upload an image to my s3, cloudfront returns a 403 when I subsequently try to retrieve the image. This lasts for about a minute, after which cloudfront starts serving the image properly.

This happens even after the image is successfully uploaded, and I can manually view it by going to the s3 console.

Solution

Something is causing your files to be requested from CloudFront before they are present in the bucket. The default configuration of CloudFront causes this to be negatively-cached for up to 5 minutes.

By default, when your origin returns an HTTP 4xx or 5xx status code, CloudFront caches these error responses for five minutes and then submits the next request for the object to your origin to see whether the problem that caused the error has been resolved and the requested object is now available.

^{— http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/custom-error-pages.html}

If the browser, or anything else, tries to download the file from that particular CloudFront edge (or any edge, if the request happens to also go through the regional edge -- CloudFront now has two edge layers) before the upload into S3 is complete, S3 will return an error, and CloudFront -- at that edge location -- will cache that error and remember, for the next 5 minutes, not to bother trying again.

This timer is configurable.

You can specify the error-caching duration—the Error Caching Minimum TTL—for each 4xx and 5xx status code that CloudFront caches. For a procedure, see Configuring Error Response Behavior.

^{— http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/custom-error-pages.html}

Set the value to 0 for any error codes, like 403, where you want to disable the error cache.

The content in this answer is adapted from my original post on Stack Overflow.

Comments

[Admin]

The 403 is probably cached from before the upload completed.

[Admin]

add /index.html in your url & check?

[f.khantsis]

so why is it a 403 and not 404?

[Michael - sqlbot]

It's part of the security model. With S3, by default, a user does not have permission to even know whether or not a specific object exists, even if they are authorized to download some objects from the bucket. 403 Forbidden, in this case, means "the server has declined to tell you whether or not an object actually exists at this URL." It is possible to configure S3 to return a 404 with an error message saying "No Such Key," but this option must be configured very carefully -- otherwise, it is possible for a user to download the entire list of everything in the bucket.

[f.khantsis]

you'd think that since the url is signed only for object a, it can know whether or not object a exists without access to meta-data about object b

[Michael - sqlbot]

That's not the issue, though. CloudFront is allowing, then S3 is denying. There are two independent layers. The CloudFront signed URL is validated by the origin access identity's keys to see whether the URL is authorized at CloudFront (also, CloudFront signed URLs can be wildcards)... but the bucket policy at S3 validates what actions the origin access identity is allowed to request on behalf of the requesting user. It's actually the origin access identity that is not being allowed to know whether the object doesn't exist or whether access is denied by S3 for some other reason.