AWS Elasticsearch Service IAM Role based Access Policy

Question

I have been struggling to figure out how to communicate with the Amazon ES service from my EC2 instances.

The documentation clearly states that the Amazon ES service supports IAM User & Role based access policies. http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-access-policies

However, when I have this access policy for my ES domain:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:role/my-ec2-role"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:123456789:domain/myDomain/*"
    }
  ]
}

I can't log into an ec2 instance and run a curl to hit my elasticsearch cluster.

Trying to do a simple curl of the _search API:

curl "http://search-myDomain.es.amazonaws.com/_search"

Produces an authentication error response:

{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:us-west-2:123456789:domain/myDomain/_search"}

Just to be extra safe I put the AmazonESFullAccess Policy on my IAM Role, still doesn't work.

I must be missing something, because being able to programmatically interact with Elasticsearch from ec2 instances that use an IAM Role is essential to getting anything accomplished with the Amazon ES Service.

I also see this contradictory statement in the docs.

IAM-based Policy Example You create IAM-based access policies by using the AWS IAM console rather than the Amazon ES console. For information about creating IAM-based access policies, see the IAM documentation.

That link to IAM documentation, is to the home page of IAM and contains exactly zero information about how to do it. Anyone got a solution for me?

Solution 1

When using IAM service with AWS, you must sign your requests. curl doesn't support signed requests (which consists of hashing the request and adding a parameter to the header of the request). You can use one of their SDK's that has the signing algorithm built in, and then submit that request.

See: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/what-is-amazon-elasticsearch-service.html#signing-requests

You can find the SDKs for popular languages here: http://aws.amazon.com/tools/

Solution 2

First, you said you can't login to an EC2 instance to curl the ES instance? You can't login? Or you can't curl it from EC2?

I have my Elasticsearch (Service) instance open to the world (with nothing on it) and am able to curl it just fine, without signing. I changed the access policy to test, but unfortunately it takes forever to come back up after changing it...

My policy looks like this:

{   "Version": "2012-10-17",   "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:843348267853:domain/myDomain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:843348267853:domain/myDomain"
    }   
   ] 
}

I realize this isn't exactly what you want, but start off with this (open to the world), curl from outside AWS and test it. Then restrict it, that way you're able to isolate the issues.

Also, I think you have an issue with the "Principal" in your access policy. You have your EC2 Role. I understand why you're doing that, but I think the Principal requires a USER, not a role.

See below:

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-access-policies

Principal

Specifies the AWS account or IAM user that is allowed or denied access to a resource. Specifying a wildcard (*) enables anonymous access to the domain, which is not recommended. If you do enable anonymous access, we strongly recommend that you add an IP-based condition to restrict which IP addresses can submit requests to the Amazon ES domain.

EDIT 1

To be clear, you added the AmazonESFullAccess policy to the my-ec2-role? If you're going to use IAM access policies, I don't think you can have a resource based policy attached to it (which is what you're doing).

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_compare-resource-policies.html

For some AWS services, you can grant cross-account access to your resources. To do this, you attach a policy directly to the resource that you want to share, instead of using a role as a proxy. The resource that you want to share must support resource-based policies. Unlike a user-based policy, a resource-based policy specifies who (in the form of a list of AWS account ID numbers) can access that resource.

Possibly try removing the access policy altogether?

Solution 3

Why you don't create a proxy with elastic ip and allow your proxy to access your ES? Basically exists three forms that you can limit access in your ES:

• Allow everyone
• White IP list
• Signing the access key and secret key provided by AWS.

I'm using two forms, in my php apps I prefer to use proxy behind the connection to ES and in my nodejs app I prefer to sign my requests using the http-aws-es node module. It's useful to create a proxy environment because my users needs to access the kibana interface to see some reports and it's possible because they have configured the proxy in their browsers =)

I must recommend to you close the access to your ES indexes, because it's pretty easy to delete them, curl -XDELETE https://your_es_address/index anyone can do it but you can say: "how the others users will get my ES address?" and I will answer you: "Security based in dimness isn't a real security"

My security access policy is basically something like it:

http://pastebin.com/EUKT1ekX

Solution 4

I encountered this issue recently and the root problem is that none of the Amazon SDKs yet support calling Elasticsearch operations like search, put, etc.

The only workaround at the moment is to execute requests directly against the endpoint using signed requests: http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

The example here is for calling EC2, but it can be modified to instead call against Elasticsearch. Just modify the "service" value to "es". From there, you have to fill in values for

• the endpoint (which is the full URL of your cluster including operation without request parameters)
• the host (the part between https:// and your canonical URI like /_status
• the canonical uri which is the URI after the first / inclusive (like /_status) but without the query string
• the request parameters (everything after ? inclusive)

Note that I've only managed to get this working so far using AWS credentials as the assumption is that you pass in an access key and secret key to the various signing calls (access_key and secret_key in the example). It should be doable using IAM roles but you'll have to call into the security token service first to get temporary credentials that can be used to sign the request. Until you do that, be sure to edit your access policy on the Elasticsearch cluster to allow user creds (user/

View more solutions

Comments

[Rick Fletcher]

Possible duplicate of Proper Access Policy for Amazon Elastic Search Cluster

[Karup]

@nackjicholson Was your problem solved? Somewhat in the same situation right now!

[Berlin]

why you wrote the same twice in your policy ?

[Berlin]

with open to the world policy not working, any advise ? curl -XGET 'search-aa-kfrqkt7hi.sa-east-1.es.amazonaws.com/_cluster/…' {"Message":"Your request: '/_cluster/state' is not allowed."}

[Cyril Duchon-Doris]

"but unfortunately it takes forever to come back up after changing it..." deserves to be bolded and highlighted. Most IAM changes on Amazon are instantaneous BUT when it comes to ElasticSearch...

[azizj]

Can you do it via AWS CLI?

[John Jones]

FWIW, AWS sigv4 was added to curl recently.