AWS S3 Event notification using Lambda function in Java

Question

I am trying to use Lambda function for S3 Put event notification. My Lambda function should be called once I put/add any new JSON file in my S3 bucket. The challenge I have is there are not enough documents for this to implement such Lambda function in Java. Most of doc I found are for Node.js

I want, my Lambda function should be called and then inside that Lambda function, I want to consume that added json and then send that JSON to AWS ES Service.

But what all classes I should use for this? Anyone has any idea about this? S3 abd ES are all setup and running. The auto generated code for lambda is `

@Override
public Object handleRequest(S3Event input, Context context) {
    context.getLogger().log("Input: " + input);

    // TODO: implement your handler
    return null;
}

What next??

Solution 1

Finally, here are the steps for S3 --> Lambda --> ES integration using Java.

1. Have your S3, Lamba and ES created on AWS. Steps are here.

2. Use below Java code in your lambda function to fetch a newly added object in S3 and send it to ES service.

   public Object handleRequest(S3Event input, Context context) {
   AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain());        
   
   for (S3EventNotificationRecord record : input.getRecords()) {
       String s3Key = record.getS3().getObject().getKey();
       String s3Bucket = record.getS3().getBucket().getName();
       context.getLogger().log("found id: " + s3Bucket+" "+s3Key);
       // retrieve s3 object
       S3Object object = s3Client.getObject(new GetObjectRequest(s3Bucket, s3Key));
       InputStream objectData = object.getObjectContent();
   
       //Start putting your objects in AWS ES Service
       String esInput = "Build your JSON string here using S3 objectData";
   
       HttpClient httpClient = new DefaultHttpClient();
   
       HttpPut putRequest = new HttpPut(AWS_ES_ENDPOINT + "/{Index_name}/{product_name}/{unique_id}" );
   
       StringEntity input = new StringEntity(esInput);
       input.setContentType("application/json");
       putRequest.setEntity(input);
   
       httpClient.execute(putRequest);
       httpClient.getConnectionManager().shutdown();
   
   }        
   return "success";}
   

3. Use either Postman or Sense to create Actual index & corresponding mapping in ES.

4. Once done, download and run proxy.js on your machine. Make sure you setup ES Security steps suggested in this post

5. Test setup and Kibana by running http://localhost:9200/_plugin/kibana/ URL from your machine.

6. All is set. Go ahead and set your dashboard in Kibana. Test it by adding new objects in your S3 bucket

Solution 2

Handling S3 events in Lambda can be done, but you have to keep in mind, the the S3Event object only transports the reference to the object and not the object itself. To get to the actual object you have to invoke the AWS SDK yourself. Requesting a S3 Object within a lambda function would look like this:

public Object handleRequest(S3Event input, Context context) {
    AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain());        

    for (S3EventNotificationRecord record : input.getRecords()) {
        String s3Key = record.getS3().getObject().getKey();
        String s3Bucket = record.getS3().getBucket().getName();
        context.getLogger().log("found id: " + s3Bucket+" "+s3Key);
        // retrieve s3 object
        S3Object object = s3Client.getObject(new GetObjectRequest(s3Bucket, s3Key));
        InputStream objectData = object.getObjectContent();
        //insert object into elasticsearch
    }        
    return null;
}

Now the rather difficult part to insert this object into ElasticSearch. Sadly the AWS SDK does not provide any functions for this. The default approach would be to do a REST call against the AWS ES endpoint. There are various samples out their on how to proceed with calling an ElasticSearch instance.

Some people seem to go with the following project:

Jest - Elasticsearch Java Rest Client

Comments

[NGR]

Thanks Jens. I also completed the simpler part and struggling with the difficult one. Trying to use Jest only to add message to ES. But somehow it seems like that part is missing. Everything like Search, get etc has shown. But not the Add document. Below line is giving error to me DocumentMapper documentMapper = new DocumentMapper.Builder("groupId", null, rootObjectMapperBuilder).build(null);

[jens walter]

To be honest, I never used the DocumentMapper, since this step has to be executed only once, I usually generated my mapping through the Kibana GUI. Apart from that, I only used the API to append Documents to the Index.

[NGR]

Thanks for your help Jens. I used Apache HTTP for REST API and by using Postman, create my index and mapping. It's working.