Retrieve bucket's objects without knowing bucket's region with AWS S3 REST API

Question

I'm trying to retrieve a bucket's objects with AWS S3 REST API (I'm not using the SDK) but unfortunately I'm facing the following problem : I don't know its region when I do the call to the API.

Here's what I do :

• I build a request to "https:// [bucketname] .s3.amazonaws.com/"
• I sign the request using a default region set to "us-west-1" (using the AWS4 signature process)
• But imagine the bucket I'm trying to GET is set to the region "us-west-2", AWS is throwing me an error

Therefore here's my question:

Is there any way to get a list of the objects of a bucket without knowing its region ? Or is there any simple way to get the region of a bucket ?

INFO : I've read there are two style to call a bucket, "path-style" and "virtual-hosted-style", but whenever I send the request to "https://s3.amazonaws.com/ [bucketname]" instead it gives me a redirectpermanent error...

Solution 1

AWS V4 authentication requires that you know the bucket's region so that you can correctly sign the request. Otherwise:

HTTP/1.1 400 Bad Request

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AuthorizationHeaderMalformed</Code>
  <Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'</Message>
  <Region>us-west-2</Region>
  <RequestId>xxxx</RequestId>
  <HostId>xxxx</HostId>
</Error>

V2 signatures were not region-specific, so there was previously a simple way you could learn the bucket's region using a V2 request:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html

However, there apprears to be a catch-22 with V4, since you have to know the bucket's region before you can make the call to discover the bucket's region.

The first solution, then, is to capture the <Region> returned in the error response and use that region for signing future requests for the bucket. Obviously, you'd want to cache this information, since not doing so would lower performance and increase costs.

Alternately, there is a way to ask the US-Standard region (us-east-1) about the location of any bucket, in any region, using this URL format only:

https://s3.amazonaws.com/bucket-name?location

Sign this request with the us-east-1 region and you will get a response, wherever the bucket happens to be. Note that if the bucket is in us-east-1 (US-Standard) the LocationConstraint is returned empty.

<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  us-west-2
</LocationConstraint>

You cannot use this URL construct for actually listing bucket objects in other regions, since https://s3.amazonaws.com/ always routes your request to US-Standard (us-east-1) but you can use it to discover the region of any bucket, since US-Standard has that information available.

---

Update/Additonal

At some point, S3 appears to have added a new response header, x-amx-bucket-region: which appears to be an undocumented addition to the REST API, and appears to be added to many S3 error responses, particularly 403 errors.

This seems like a useful mechanism for self-correction if you receive an error response.

Also, the information above about interrogating the US-Standard region about the location of any bucket anywhere is still accurate, and the same request should work if sent to any regional S3 REST endpoint, not just US-Standard, since all regions are aware of the location of all other buckets: e.g. https://s3-us-west-2.amazonaws.com/bucket-name?location would ask US-West-2 (Oregon), which also has the same information available, for any bucket globally. It might be desirable in some cases to interrogate your nearest region.

Solution 2

In Ruby, you can pass the region as such when getting the resource:

s3 = Aws::S3::Resource.new(region: 'us-west-2')

Solution 3

You could get the region from the exception -> additional details

            AmazonS3 client = AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(credentials)).withRegion(Regions.US_EAST_1).build();
            Regions bucketRegion = Regions.US_EAST_1;    
            try {
                client.getBucketLocation(bucket.getName());
            } catch (AmazonS3Exception e) {
                bucketRegion = Regions.fromName(e.getAdditionalDetails().get("Region"));
            }

Its not the best way of doing it but right now there is no alternative to do it.

Comments

[Asticode]

Thanks for the answer :)

[richb]

I found the "s3.amazonaws.com/bucket-name?location" get only works for the bucket owner (Access Denied otherwise), so it's not a general solution.