How to talk to aws elasticsearch service using elastic java client?

Question

I have set up a elasticsearch server using AWS elasticsearch service (Not EC2). It gave me an endpoint https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com/ and if I click this endpoint(Note that there is no port specified) I can get the expected

{
  status: 200,
  name: "Mastermind",
  cluster_name: "xxxx",
  version: {
    number: "1.5.2",
    build_hash: "yyyyyy",
    build_timestamp: "2015-04-27T09:21:06Z",
    build_snapshot: false,
    lucene_version: "4.10.4"
  },
  tagline: "You Know, for Search"
}

The question is how do I get this through the elasticsearch java client without a port number? The sample code I get is

Client client = TransportClient.builder().build()
    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("host1"), 9300));

If I use this code and just replace "host1" with my endpoint, I'll get "NoNodeAvailableException"

ps: The java client version I'm using is 2.0.0.

Edit I finally decided to go with Jest, a 3rd party REST client. But what Brooks answered below is also very helpful - AWS do use port 80 for http and 443 for https. The blocker for me was the firewall I guess.

Edit2 The AWS ES service documentation explicitly says:
The service supports HTTP on port 80, but does not support TCP transport.

Solution 1

Believe it or not, AWS doesn't launch Elasticsearch using 9200 and 9300. It's launched via plain old port 80.

So, to demonstrate, try this...

curl -XPOST "http://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:80/myIndex/myType" -d '["name":"Edmond"}'

Or

curl -XPOST "https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:443/myIndex/myType" -d '["name":"Edmond"}'

It should respond with: {"_index":"myIndex","_type":"myType","_id":"SOME_ID_#","_version":1,"created":true}

Check in Kibana and you'll see it's there.

So, then in your code, it should be:

Client client = TransportClient.builder().build()
    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("xxx-xxxxxxxx.us-west-2.es.amazonaws.com"), 80));

Unfortunately, I don't off-hand know how to transmit encrypted via SSL/HTTPS using the transport client. You could try using regular REST calls instead using JERSEY.

Finally, make sure your Elasticsearch access policy is configured properly. Something along the lines of:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:yyyyyyy:domain/myDomain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:yyyyyyyyy:domain/myDomain"
    }
  ]
}

NOTE: The above access policy is completely wide open and is not recommended for anything remotely close to production. Just so you know....

Solution 2

Managed elastic search service in AWS does not provide the port for the transport protocol until now.
This question has been answered here ,

Elastic Transport client on AWS Managed ElasticSearch1

There is also a discussion in the AWS forum regarding the transport protocol. Here is the link

What is the port for the transport protocol ?

Solution 3

After lot of search i found an example which used a GET request, so I made minor changes to it for allowing POST requests so that complex queries can be submitted via POST body. The implementation is available at https://github.com/dy10/aws-elasticsearch-query-java

Apart from properly configuring access to you AWS ES (i.e. dont open it to Public), make sure to use https (the above code uses http; just replace http with https in the code and it will work).

Another useful looking but partial implementation is at https://github.com/aws/aws-sdk-java/issues/861

Comments

[Edmond]

yes. curl gave me Unknown SSL protocol error in connection to xxxxxxx.us-west-2.es.amazonaws.com:-9847. not sure what 9847 is.

[Edmond]

I think you're probably right. I can get response by hitting xxx.amazonaws.com:80. but I still don't know how to do that in code. BTW, InetAddress.getByName(String) cannot take "http" or "https", otherwise it will give "Unknown host" error.

[Brooks]

Ah, my apologies. Yeah, if you're using https, then it would be port 443. If you choose to go with http, then it would be port 80. I've updated my answer accordingly. By the way, make sure you have set the access policy for elasticsearch properly. I will update my answer for this as well.

[Brooks]

Right, I forgot about that. Unfortunately, I don't know how to transmit encrypted using transport client. As I mentioned in the latest edit to my answer, you would be able to do so easily using REST calls with JERSEY, but then obviously lose out on the benefits of the transport client. There are wrappers out there that support SSL for Elastic Found, but I haven't seen one for AWS. For testing purposes, I would stick with in the clear via port 80 while continuing to search for a way to encrypt. Java has to have a way, I just don't know it.

[dy10]

This is just an example of how to send HTTP POST queries to AWS ElasticSearch service. The EsQuery.java file contains the main method and you can run it directly, after replacing the 4 static String variables with real value.

[Artjom B.]

Your comment should be included in your answer. Comments are short-lived and can be deleted at any time. Please edit your answer to the missing information.

[wikier]

Good to see your access details hardcoded in the client,

[dy10]

did u read the comments 2 lines above of credentials? Those are fake credentials. It is just there so people who are less conversant can easily identify what kind of value is expected there.