Why use AMQP/ZeroMQ/RabbitMQ

21,871

Solution 1

what makes them better than writing your own library?

When rolling out the first version of your app, probably nothing: your needs are well defined and you will develop a messaging system that will fit your needs: small feature list, small source code etc.

Those tools are very useful after the first release, when you actually have to extend your application and add more features to it. Let me give you a few use cases:

  • your app will have to talk to a big endian machine (sparc/powerpc) from a little endian machine (x86, intel/amd). Your messaging system had some endian ordering assumption: go and fix it
  • you designed your app so it is not a binary protocol/messaging system and now it is very slow because you spend most of your time parsing it (the number of messages increased and parsing became a bottleneck): adapt it so it can transport binary/fixed encoding
  • at the beginning you had 3 machine inside a lan, no noticeable delays everything gets to every machine. your client/boss/pointy-haired-devil-boss shows up and tell you that you will install the app on WAN you do not manage - and then you start having connection failures, bad latency etc. you need to store message and retry sending them later on: go back to the code and plug this stuff in (and enjoy)

  • messages sent need to have replies, but not all of them: you send some parameters in and expect a spreadsheet as a result instead of just sending and acknowledges, go back to code and plug this stuff in (and enjoy.)

  • some messages are critical and there reception/sending needs proper backup/persistence/. Why you ask ? auditing purposes

And many other use cases that I forgot ...

You can implement it yourself, but do not spend much time doing so: you will probably replace it later on anyway.

Solution 2

That's very much like asking: why use a database when you can write your own?

The answer is that using a tool that has been around for a while and is well understood in lots of different use cases, pays off more and more over time and as your requirements evolve. This is especially true if more than one developer is involved in a project. Do you want to become support staff for a queueing system if you change to a new project? Using a tool prevents that from happening. It becomes someone else's problem.

Case in point: persistence. Writing a tool to store one message on disk is easy. Writing a persistor that scales and performs well and stably, in many different use cases, and is manageable, and cheap to support, is hard. If you want to see someone complaining about how hard it is then look at this: http://www.lshift.net/blog/2009/12/07/rabbitmq-at-the-skills-matter-functional-programming-exchange

Anyway, I hope this helps. By all means write your own tool. Many many people have done so. Whatever solves your problem, is good.

Solution 3

I'm considering using ZeroMQ myself - hence I stumbled across this question.

Let's assume for the moment that you have the ability to implement a message queuing system that meets all of your requirements. Why would you adopt ZeroMQ (or other third party library) over the roll-your-own approach? Simple - cost.

Let's assume for a moment that ZeroMQ already meets all of your requirements. All that needs to be done is integrating it into your build, read some doco and then start using it. That's got to be far less effort than rolling your own. Plus, the maintenance burden has been shifted to another company. Since ZeroMQ is free, it's like you've just grown your development team to include (part of) the ZeroMQ team.

If you ran a Software Development business, then I think that you would balance the cost/risk of using third party libraries against rolling your own, and in this case, using ZeroMQ would win hands down.

Perhaps you (or rather, your partner) suffer, as so many developers do, from the "Not Invented Here" syndrome? If so, adjust your attitude and reassess the use of ZeroMQ. Personally, I much prefer the benefits of Proudly Found Elsewhere attitude. I'm hoping I can proud of finding ZeroMQ... time will tell.

EDIT: I came across this video from the ZeroMQ developers that talks about why you should use ZeroMQ.

Solution 4

what makes them better than writing your own library?

Message queuing systems are transactional, which is conceptually easy to use as a client, but hard to get right as an implementor, especially considering persistent queues. You might think you can get away with writing a quick messaging library, but without transactions and persistence, you'd not have the full benefits of a messaging system.

Persistence in this context means that the messaging middleware keeps unhandled messages in permanent storage (on disk) in case the server goes down; after a restart, the messages can be handled and no retransmit is necessary (the sender does not even know there was a problem). Transactional means that you can read messages from different queues and write messages to different queues in a transactional manner, meaning that either all reads and writes succeed or (if one or more fail) none succeeds. This is not really much different from the transactionality known from interfacing with databases and has the same benefits (it simplifies error handling; without transactions, you would have to assure that each individual read/write succeeds, and if one or more fail, you have to roll back those changes that did succeed).

Solution 5

Before writing your own library, read the 0MQ Guide here: http://zguide.zeromq.org/page:all

Chances are that you will either decide to install RabbitMQ, or else you will make your library on top of ZeroMQ since they have already done all the hard parts.

Share:
21,871
Evil Spork
Author by

Evil Spork

Programmer, writer, artist, game dev, nerd.

Updated on July 06, 2022

Comments

  • Evil Spork
    Evil Spork almost 2 years

    as opposed to writing your own library.

    We're working on a project here that will be a self-dividing server pool, if one section grows too heavy, the manager would divide it and put it on another machine as a separate process. It would also alert all connected clients this affects to connect to the new server.

    I am curious about using ZeroMQ for inter-server and inter-process communication. My partner would prefer to roll his own. I'm looking to the community to answer this question.

    I'm a fairly novice programmer myself and just learned about messaging queues. As i've googled and read, it seems everyone is using messaging queues for all sorts of things, but why? What makes them better than writing your own library? Why are they so common and why are there so many?

  • Evil Spork
    Evil Spork over 14 years
    I've been thinking about this for a short while. What exactly do you mean by Transactions and Persistence? I understand the words but not the context here. Simply stated, it rings of buzz words.
  • pmf
    pmf over 14 years
    I would not call these buzzwords; persistence and transactions have been known under these names for several decades. I've edited my answer to elaborate these points in the context of messaging.
  • Evil Spork
    Evil Spork over 14 years
    Interesting, but what makes these so hard to get right? From my perspective it seems, if I'm writing a server and client, you send the message, make sure the message is received in full, and keep a duplicate of the message stored to disk. That is the end of it. why are there so many versions of such a simplistic system? I mean alot of them are fairly small code wise, once you have a comprehension of how they work it doesnt seem to require that much to implement. So it comes back to why use a pre-written messaging queue.
  • Evil Spork
    Evil Spork over 14 years
    ahh, thank you very much. This certainly gives me some food for thought.
  • ddimitrov
    ddimitrov almost 14 years
    Transactions are very easy to implement if you don't care about latency and/or correctness. See here for an overview of some typical implementation challenges: jroller.com/pyrasun/category/XA
  • Evil Spork
    Evil Spork almost 14 years
    i tend to agree with you up to a point.. ive been working with a few programs over the last few months in C++ since this question was asked.. and ive run in to the hell that is 3rd party libs.. some are relatively painless.. but alot of them add more trouble than they are worth.. just trying to install them. I still am unsure as to why anyone would have requirements for zeroMQ.. one of the first things i learned to write was how to send a message from one program to another through TCP and sockets for a very simple chat program, it was relatively easy..
  • Daniel Paull
    Daniel Paull almost 14 years
    Completely agree that many (maybe most?) third party libs are more trouble than they are worth. I guess I've found some real gems over the years that I'm more than happy to work with. I've just compiled ZeroMQ and run their tests. It's looks pretty good. It doesn't have "gem" status just yet, but we'll see...
  • Daniel Paull
    Daniel Paull almost 14 years
    Just to follow up - I've found ZeroMQ to be quite good. It certainly is very light weight and fast. I don't like the way that their Socket has thread affinity as I want to use two threads to access the socket - one for reading messages and one for writing messages. It think that the "ZeroMQ" way of doing this is to use in-process message queues to marshal messages between threads... more thought required.
  • Evil Spork
    Evil Spork almost 14 years
    Interesting, please share more with more experience!
  • Daniel Paull
    Daniel Paull almost 14 years
    I spent yesterday writing an alternative messaging implementation for my app based on Boost.Asio (boost.org/doc/libs/1_37_0/doc/html/boost_asio.html). This fits my needs much better. I've not measured it's performance yet, but I expect it will be quite good. I think it's just as easy to use ZeroMQ when, assuming you don't need all their fruit (like pubish/subscribe pattern), but has far better documentation.
  • Robin Green
    Robin Green almost 13 years
    According to the official zeromq guide, zeromq doesn't provide persistence. The guide shows you how to do this, but I'd rather have something that has persistence baked into the framework, not using some poorly-tested tutorial code!
  • Robin Green
    Robin Green almost 13 years
    If I've understood the official zeromq guide correctly, it is not persistent. You have to build persistence on top of it.
  • Robin Green
    Robin Green almost 13 years
    They have done some "hard parts" in tutorial fashion in the guide (e.g. persistence). I would rather use a battle-tested implementation than some poorly-tested tutorial code.
  • Michael Dillon
    Michael Dillon almost 13 years
    If you want battle-tested implementation, then you would choose AMQP instead of ZMQ and use something like RabbitMQ
  • O.O
    O.O almost 12 years
    Disagree on "when rolling out the first version of your app, probably nothing" You finish by saying the first version will get rewritten with a library anyway, may as well start with something you don't end up throwing away. Otherwise, I agree.
  • ppi
    ppi over 11 years
    @O.O most of the time the initial features needed are small and you cannot always justify importing 3rd party stuff easily: need approval, security screening of the code etc. depending on who you work for these take much more time than dev time. Hence you start small, do it yourself with the tools already in the comnpany, and then when your code is in prod, heavily used, using the 3rd party stuff becomes easier. Your mileage may vary, but usually companies don't want use a gazillion 3rd party stuff/framework/library.
  • Jeff Hammond
    Jeff Hammond over 9 years
    Or a compiler. Why bother using a compiled language when one can write pure, unadulterated assembly language all the time? :-)