Apache camel to aggregate multiple REST service responses

Question

I m new to Camel and wondering how I can implement below mentioned use case using Camel,

We have a REST web service and lets say it has two service operations callA and callB. Now we have ESB layer in the front that intercepts the client requests before hitting this actual web service URLs.

Now I m trying to do something like this - Expose a URL in ESB that client will actually call. In the ESB we are using Camel's Jetty component which just proxies this service call. So lets say this URL be /my-service/scan/

Now on receiving this request @ESB, I want to call these two REST endpoints (callA and callB) -> Get their responses - resA and resB -> Aggregate it to a single response object resScan -> return to the client.

All I have right now is -

<route id="MyServiceScanRoute">
<from uri="jetty:http://{host}.{port}./my-service/scan/?matchOnUriPrefix=true&amp;bridgeEndpoint=true"/>
<!-- Set service specific headers, monitoring etc. -->  
<!-- Call performScan -->
<to uri="direct:performScan"/>
</route>

<route id="SubRoute_performScan">
<from uri="direct:performScan"/>
<!--  HOW DO I??
Make callA, callB service calls. 
Get their responses resA, resB.
Aggregate these responses to resScan
 -->
</route>

Solution 1

I think that you unnecessarily complicate the solution a little bit. :) In my humble opinion the best way to call two independed remote web services and concatenate the results is to:

• call services in parallel using multicast
• aggregate the results using the GroupedExchangeAggregationStrategy

The routing for the solution above may look like:

from("direct:serviceFacade")
  .multicast(new GroupedExchangeAggregationStrategy()).parallelProcessing()
    .enrich("http://google.com?q=Foo").enrich("http://google.com?q=Bar")
  .end();

Exchange passed to the direct:serviceFacadeResponse will contain property Exchange.GROUPED_EXCHANGE set to list of results of calls to your services (Google Search in my example).

And that's how could you wire the direct:serviceFacade to Jetty endpoint:

from("jetty:http://0.0.0.0:8080/myapp/myComplexService").enrich("direct:serviceFacade").setBody(property(Exchange.GROUPED_EXCHANGE));

Now all HTTP requests to the service URL exposed by you on ESB using Jetty component will generate responses concatenated from the two calls to the subservices.

Further considerations regarding the dynamic part of messages and endpoints

In many cases using static URL in endpoints is insufficient to achieve what you need. You may also need to prepare payload before passing it to each web service.

Generally speaking - the type of routing used to achieve dynamic endpoints or payloads parameters in highly dependent on the component you use to consume web services (HTTP, CXFRS, Restlet, RSS, etc). Each component varies in the degree and a way in which you can configure it dynamically.

If your endpoints/payloads should be affected dynamically you could also consider the following options:

Preprocess copy of exchange passed to each endpoint using the onPrepareRef option of the Multicast endpoint. You can use it to refer to the custom processor that will modify the payload before passing it to the Multicast's endpoints. This may be good way to compose onPrepareRef with Exchange.HTTP_URI header of HTTP component.

Use Recipient List (which also offers parallelProcessing as the Multicast does) to dynamically create the REST endpoints URLs.

Use Splitter pattern (with parallelProcessing enabled) to split the request into smaller messages dedicated to each service. Once again this option could work pretty well with Exchange.HTTP_URI header of HTTP component. This will work only if both sub-services can be defined using the same endpoint type.

As you can see Camel is pretty flexible and offers you to achieve your goal in many ways. Consider the context of your problem and choose the solution that fits you the best.

If you show me more concrete examples of REST URLs you want to call on each request to the aggregation service I could advice you which solution I will choose and how to implement it. The particularly important is to know which part of the request is dynamic. I also need to know which service consumer you want to use (it will depend on the type of data you will receive from the services).

Solution 2

This looks like a good example where the Content Enricher pattern should be used. Described here

<from uri="direct:performScan"/>
   <enrich uri="ServiceA_Uri_Here" strategyRef="aggregateRequestAndA"/>
   <enrich uri="ServiceA_Uri_Here" strategyRef="aggregateAandB"/>
</route>

The aggregation strategies has to be written in Java (or perhaps some script language, Scala/groovy? - but that I have not tried).

The aggregation strategy just needs to be a bean that implements org.apache.camel.processor.aggregate.AggregationStrategy which in turn requires you to implement one method:

Exchange aggregate(Exchange oldExchange, Exchange newExchange);

So, now it's up to you to merge the request with the response from the enrich service call. You have to do it twice since you have both callA and callB. There are two predefined aggregation strategies that you might or might not find usefull, UseLatestAggregationStrategy and UseOriginalAggregationStrategy. The names are quite self explainatory.

Good luck

Comments

[Rishi]

Thanks for the pointers! Really appreciate it. Let me start reading abt the Content Enricher pattern and alike.

[Rishi]

After a round of discussions with team, we found out that the Enricher option would be costly as its sequential (and we might have more than 2 service calls). Now I m evaluating the options of using - 1. Composed Message Processor EIP/ Splitter-Aggregator 2. Or Scatter-Gather EIP 3. Or Recipient List with an Agregator strategy Any inputs? Which one is the right fit for the use case?

[Petter Nordlander]

The issue is to keep the request and the aggregated response in the same route, since you want to reply to the HTTP Call. The most appealing choice seems to be recipientList with an aggregation strategy specified and the "parallellProcessing" option set to true. The tricky part is to write the aggregation strategy, since it might receive the answers in any order. I know nothing about the payload, but it should be fairly straight forward. camel.apache.org/aggregator.html has some examples how to aggregate multiple responses, for instance the highest number in a sequence of responses. gl

[Rishi]

Looks like in both - Multipcast and Recipient list pattern the 'same' message is routed to multiple consumers. But in this case, two different web service calls ( callA and callB ) are entirely different. So I m going to try something like this - <split> <to uri="direct:callA" /> <to uri="direct:callB" /> <to uri="seda:aggregateScan" /> </split> <route> <from uri="direct:callA"/> <!-- Call webservice service --> </route> <!-- same route for B --> <!-- aggregation route and strategy bean ref -->

[Petter Nordlander]

Yes, that sounds appropriate, given the payload differs in the call and not only the URL. Anyway, you can't overuse the "direct" transport in Camel. It is extremly useful as it allows to do things like this and to modularize your routes, build up reusable sub-routes etc. The overhead is simply a method-call (or so, they say).

[Henryk Konsek]

Thank you for accepting my answer. Upvoting is welcome as well :P .

[Petter Nordlander]

Nice demo of the multicast directive, Henryk. However, seems like there is a need to prepare the payload, not only the URL of the request to each service. Can you do that with multicast as well? Isn't multicast for same msg to multiple endpoints?

[Rishi]

@Petter As I m hitting the REST end point, its pretty much just the URL (GET request). But I have one question here, I need the parameters to be passed to individual service calls. example: 0.0.0.0:8080/myapp/myComplexService?keyword=abc so this 'keyword' parameter has to be passed in individual service calls -> myserver.com/serviceA?keyword=abc & myserver.com/serviceB?keyword=abc

[Petter Nordlander]

@Rishi, Henryk demonstrated that above. Look at the URIs, q=Foo and q=Bar. That would represent your keyword=abc.

[Rishi]

@Petter Actually what want to say is - all url parameters passed by the client should be passed (carried forward) when I make the calls to services. What I see in the example is a hardcoded param just for simplified demonstration (two service calls will be always with q=foo & q=bar). After reading the Camel docs, I find the http component can be useful in this regard.. in that way I'll forward the URL params that client has sent to a different endpoint. Does this sound correct?

[Henryk Konsek]

@Rishi, Petter, your concerns can be addressed with Dynamic Recipent List or Spiltter. It also depends which component will you use to consume web services (since the components varies in the degree of ability to dynamically configure consumer). I must go to the bed now, but I'll return the this thread tomorrow. Stay tuned :) .

[Henryk Konsek]

@Petter "there is a need to prepare the payload, not only the URL of the request to each service. Can you do that with multicast as well?" You can preprocess the copy of the exchange passed to each Multicast's endpoint using the 'onPrepareRef' option of Multicast. This is the way to dynamically affect each multicasted message.

[Henryk Konsek]

I've updated the answer to put a light on other possible options. If you provide me more details regarding your case (what parts of REST URLs are dynamic and the components you want to use to consume the web services) I could provide you more detailed solution.

[Rishi]

@HenrykKonsek esbhost/myapp/… is the request coming from client. I need all the URL parameters from this request to be carried over when I m making calls to individual service (either using multicast/ splitter/ RecipientList). example: 1. host/serviceA/endpoint/… 2. host/serviceB/endpoint/… and so forth. Its a GET request (REST) from client, so all I need is to make sure that URL params should be passed to sub calls to serviceA and serviceB .

[Rishi]

@Petter,Henryk Update from my side - I was able to spawn the requests using multicast. I m checking GroupedExchangeAggregationStrategy as my aggregation strategy. So here the response from individual REST web services is XML.

[Petter Nordlander]

Nice!Great to hear about your progress:)

[Rishi]

Update #2 - Aggregation strategy worked! End 2 end flow is working fine (needs refactoring though). We used multicast and custom aggregation strategy. I still need to test out the correlation-id case.. but for now atleast the basic use case is working. Thanks @Petter,Henryk for valuable pointers.

[Jahan Zinedine]

/** * Aggregate all exchanges into a single combined Exchange holding all the aggregated exchanges * in a {@link List} of {@link Exchange} as the message body. * <p/> * <b>Important:</b> This strategy is not to be used with the <a href="http://camel.apache.org/content-enricher.html">Content Enricher</a> EIP * which is enrich or pollEnrich. * * @version */ public class GroupedExchangeAggregationStrategy extends AbstractListAggregationStrategy<Exchange>

[mac]

I see a comment inside GroupedExchangeAggregationStrategy.java that <b>Important:</b> This strategy is not to be used with the <a href="camel.apache.org/content-enricher.html">Content Enricher</a> EIP * which is enrich or pollEnrich. But in the answer it is used with enrich method, so is it okay to use with enrich method?

[ddsultan]

@HenrykKonsek, If recipientList is used to generate URL endpoints, how would one best process the responses?