Service Oriented Architecture - AMQP or HTTP

Question

A little background.

Very big monolithic Django application. All components use the same database. We need to separate services so we can independently upgrade some parts of the system without affecting the rest.

We use RabbitMQ as a broker to Celery.

Right now we have two options:

1. HTTP Services using a REST interface.
2. JSONRPC over AMQP to a event loop service

My team is leaning towards HTTP because that's what they are familiar with but I think the advantages of using RPC over AMQP far outweigh it.

AMQP provides us with the capabilities to easily add in load balancing, and high availability, with guaranteed message deliveries.

Whereas with HTTP we have to create client HTTP wrappers to work with the REST interfaces, we have to put in a load balancer and set up that infrastructure in order to have HA etc.

With AMQP I can just spawn another instance of the service, it will connect to the same queue as the other instances and bam, HA and load balancing.

Am I missing something with my thoughts on AMQP?

Solution 1

At first,

• REST, RPC - architecture patterns, AMQP - wire-level and HTTP - application protocol which run on top of TCP/IP
• AMQP is a specific protocol when HTTP - general-purpose protocol, thus, HTTP has damn high overhead comparing to AMQP
• AMQP nature is asynchronous where HTTP nature is synchronous
• both REST and RPC use data serialization, which format is up to you and it depends of infrastructure. If you are using python everywhere I think you can use python native serialization - pickle which should be faster than JSON or any other formats.
• both HTTP+REST and AMQP+RPC can run in heterogeneous and/or distributed environment

So if you are choosing what to use: HTTP+REST or AMQP+RPC, the answer is really subject of infrastructure complexity and resource usage. Without any specific requirements both solution will work fine, but i would rather make some abstraction to be able switch between them transparently.

You told that your team familiar with HTTP but not with AMQP. If development time is an important time you got an answer.

If you want to build HA infrastructure with minimal complexity I guess AMQP protocol is what you want.

I had an experience with both of them and advantages of RESTful services are:

• they well-mapped on web interface
• people are familiar with them
• easy to debug (due to general purpose of HTTP)
• easy provide API to third-party services.

Advantages of AMQP-based solution:

• damn fast
• flexible
• cost-effective (in resources usage meaning)

Note, that you can provide RESTful API to third-party services on top of your AMQP-based API while REST is not a protocol but rather paradigm, but you should think about it building your AQMP RPC api. I have done it in this way to provide API to external third-party services and provide access to API on those part of infrastructure which run on old codebase or where it is not possible to add AMQP support.

If I am right your question is about how to better organize communication between different parts of your software, not how to provide an API to end-users.

If you have a high-load project RabbitMQ is damn good piece of software and you can easily add any number of workers which run on different machines. Also it has mirroring and clustering out of the box. And one more thing, RabbitMQ is build on top of Erlang OTP, which is high-reliable,stable platform ... (bla-bla-bla), it is good not only for marketing but for engineers too. I had an issue with RabbitMQ only once when nginx logs took all disc space on the same partition where RabbitMQ run.

UPD (May 2018): Saurabh Bhoomkar posted a link to the MQ vs. HTTP article written by Arnold Shoon on June 7th, 2012, here's a copy of it:

I was going through my old files and came across my notes on MQ and thought I’d share some reasons to use MQ vs. HTTP:

• If your consumer processes at a fixed rate (i.e. can’t handle floods to the HTTP server [bursts]) then using MQ provides the flexibility for the service to buffer the other requests vs. bogging it down.
• Time independent processing and messaging exchange patterns — if the thread is performing a fire-and-forget, then MQ is better suited for that pattern vs. HTTP.
• Long-lived processes are better suited for MQ as you can send a request and have a seperate thread listening for responses (note WS-Addressing allows HTTP to process in this manner but requires both endpoints to support that capability).
• Loose coupling where one process can continue to do work even if the other process is not available vs. HTTP having to retry.
• Request prioritization where more important messages can jump to the front of the queue.
• XA transactions – MQ is fully XA compliant – HTTP is not.
• Fault tolerance – MQ messages survive server or network failures – HTTP does not.
• MQ provides for ‘assured’ delivery of messages once and only once, http does not.
• MQ provides the ability to do message segmentation and message grouping for large messages – HTTP does not have that ability as it treats each transaction seperately.
• MQ provides a pub/sub interface where-as HTTP is point-to-point.

UPD (Dec 2018): As noticed by @Kevin in comments below, it's questionable that RabbitMQ scales better then RESTful servies. My original answer was based on simply adding more workers, which is just a part of scaling and as long as single AMQP broker capacity not exceeded, it is true, though after that it requires more advanced techniques like Highly Available (Mirrored) Queues which makes both HTTP and AMQP-based services have some non-trivial complexity to scale at infrastructure level.

After careful thinking I also removed that maintaining AMQP broker (RabbitMQ) is simpler than any HTTP server: original answer was written in Jun 2013 and a lot of changed since that time, but the main change was that I get more insight in both of approaches, so the best I can say now that "your mileage may vary".

Also note, that comparing both HTTP and AMQP is apple to oranges to some extent, so please, do not interpret this answer as the ultimate guidance to base your decision on but rather take it as one of sources or as a reference for your further researches to find out what exact solution will match your particular case.

Solution 2

The irony of the solution OP had to accept is, AMQP or other MQ solutions are often used to insulate callers from the inherent unreliability of HTTP-only services -- to provide some level of timeout & retry logic and message persistence so the caller doesn't have to implement its own HTTP insulation code. A very thin HTTP gateway or adapter layer over a reliable AMQP core, with option to go straight to AMQP using a more reliable client protocol like JSONRPC would often be the best solution for this scenario.

Comments

[jreid42]

We ended up going with HTTP/REST. I really wanted to go the AMQP route because it fit so nicely into our architecture but my team didnt want to try something new so that's a bummer. So much more work is needed for development of a redudant and highly available SOA system using HTTP instead of AMQP and RPC.

[rayman]

@pinepain I think one thing to mention (and correct me if Iam wrong) is that with AMQP you can actually push messages to the destination where as with HTTP you cant (working on request-response method)

[pinepain]

@rayman HTTP and AMQP are different concepts, so I'd prefer not use such criteria for their comparison.

[rayman]

@pinepain I agree but still thats a capability which should be considered when thinking about solution messaging solution (to http or not to http)

[pinepain]

@rayman exactly, AMQP is very different from HTTP by many factors, like already mentioned advanced routing, connection multiplexing (which added in http2) and so on. Same for HTTP, caching, proxying, methods and so on. My original point is HTTP and AMQP are on the different level and comparing them may be like comparing car and train: while both are vehicles, they are different in many aspects.

[Saurabh Bhoomkar]

Here is a good read from the comparison point of view :blogs.perficient.com/ibm/2012/06/07/mq-vs-http

[Kevin]

@pinepain I know this is an old question, but as a clarification why does AMQP scale better than HTTP? Intuitively, AMQP forces all traffic through a broker and I would assume that the broker then becomes a bottleneck especially if all message passing in the architecture is based on funneling traffic through a broker? In that sense HTTP would seen to scale better even if inferior in other respects?

[pinepain]

@Kevin Thank you for a good question. At that time of writing here's what was in my mind: speaking about particular implementation of AMQP - RabbitMQ, scaling horizontally would mean adding a node in a cluster and enabling queue mirroring, then scaling application horizontally would mean adding more workers, so eventually we have no SPOF, while with REST solutions, we normally have to have LB in front of it which became a real bottleneck. Nowadays I would say that with service discovery and k8s it shouldn't make any significant difference in terms of scaling infra nowadays.

[pinepain]

@Kevin Though as long as broker/LB is not a bottleneck, scaling application with adding more workers to AMQP looks easier to me, and giving that AMQP and MQ in general fits a long-running tasks processing far more better and used for tasks which doesn't require realtime processing, broker is rarely a bottleneck, unlike LB in HTTP.

[Kevin]

I don't know one way or the other but it would be surprising to me if a broker scales better than a load balancer. I would assume it generally scales worse but message queues generally provide desirable extra features on top of HTTP and that would be a key trade off.

[pinepain]

In case of RabbitMQ, if clustering used and queue has HA option on, single broker is rarely and issue (at some cost, obviously). I see your point, I agree that scales better is a subjective opinion which may be influenced by experience with technologies in question and to state this I should backed it by more facts. What I'm going to do is to alter original answer to point that both solutions scales well. The issue comparing HTTP and AMQP is that it's like apple to oranges to some extent. Thanks a lot for bringing this up! Merry Xmas!

[jreid42]

I have a lot more experience then when I originally asked this question. I am very glad that my team stopped me and we stuck with the standard approach. For what it's worth though I have personally 100% abandoned using REST in any future projects and try to exclusively use gRPC or Cap'n Proto. When I need a message queue I use NATS and if I need guarantees I use NATS streaming. These are all just personal choices. I think at the end of the day unless you have extreme requirements any solution that is well thought out and tested will work just fine.