Elastic Transport client on AWS Managed ElasticSearch

Question

I am trying to use the AWS Managed ElasticSearch for my project.

I have followed https://aws.amazon.com/blogs/aws/new-amazon-elasticsearch-service/ and i am able to start an instance and which is successfull. But I am unable to connect to the same instance from my service using elasticsearch transport client.

I know transport client supposed to connect on to the 9300 port, and that port i am unable to turn on through the aws console.

Here is the code that I am using to connect, which is successfully able to connect to my elastic search setup on an ec2 machine on the 9300 port.

ImmutableSettings.Builder settings = ImmutableSettings.settingsBuilder();
                    settings.put("cluster.name", "my-cluster-name")
                            .put("client.transport.nodes_sampler_interval", "15s")
                            .put("client.transport.ping_timeout", "15s")
                            .put("client.transport.sniff", true)
                            .put("client.transport.ignore_cluster_name", false).build();
                    client = new TransportClient(settings)
                        .addTransportAddress(
                            new InetSocketTransportAddress(
                                        env.getProperty("elastic-host-url-provided-by-aws",80)
                            ));

I am getting the Exception

org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:305) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:200) at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106) at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:98)

I suspect that this error is since I am connecting the TransportClient over the HTTP port. But I don't know, what is the TCP port for aws managed elastic search instance. I searched in aws documents, and i couldnt find any. If some have used TransportClient to connect with Amazon ES let me know.

NB: I have verified that the elasticsearch java jar version that I am using is as same as same with the server. And from my system I am able to access the Kibana and the ES HTTP PORTS with out any issue.

Solution 1

AWS related Forum link.

Here are the restrictions for AWS ElasticSearch service:

TCP transport The service supports HTTP on port 80, but does not support TCP transport.

Solution 2

The documentation section about missing support to TCP transport for AWS Elasticsearch 5.x domains can be found here:

TCP transport

The service supports HTTP on port 80, but does not support TCP transport.

Comments

[Ysak]

found in the forum that its only possible to connect using http client

[Ysak]

no transport ports are open from this service

[constpetrov]

could you please add a forum link here? I wanted to use the same setup as your.

[Faheem Sohail]

You could use JEST which is a java wrapper around the REST API github.com/searchbox-io/Jest

[Ysak]

I had faced lots of issue while having bulks reads and writes over rest api. many time it used to get crashed. Anyway I dont think I can use Jest, which I already thought to use, but couldn't.

[draeron]

My experience using Jest is that you have to throttle your requests and limit youself to 1-3 uploading threads.