Message Queueing with AMQP
My basic intention for using a message queue was to decouple requests from fulfillment, in several ways:
- logically, in the architecture of the overall system
- physically, so requestors and actors can be network distant
My use case which was a low-volume, asynchronous task queue; the details are out of scope for this article, but using a message queue had various benefits:
- reusing others' round wheels instead of reinventing square ones
- making each component of the overall system independent but interoperable
- the system became distributed 'for free'
- it became easy to turn a 'pull' action into a 'push'
Basically, from a development perspective, the result of using AMQP and a message broker is that I can write code that deals directly with the data that I care about using abstract interfaces and not deal with transports.
From a systems/operations perspective, I get reliability, modularity and the ability to (re)provision and administer my service without interrupting availability. Using a mature broker makes it easier to move to clustering/HA/federation.
Basic Concepts
I started poking around in likely places for hints and came up with ZeroMQ (aka 0MQ). Although I was able to make ZeroMQ do something minimally useful very easily, I soon noticed that it does not support message persistence so I chose a message queueing system with a broker that does. For my use case, that was required.
Broker
There are several message broker projects, but not knowing much about them or reasons to choose between them, I picked RabbitMQ, mostly because it seemed that it was easier to get going than Apache Qpid. They both appear to be mature, active projects that would do the job.
It is extremely easy to install on modern unix systems, pick one:
yum install librabbitmq rabbitmq-server apt-get install librabbitmq0 rabbitmq-server brew install rabbitmq
Then start it up, according to your system's preferred service management approach.
Enable RabbitMQ's management plugin:
rabbitmq-plugins enable rabbitmq_management
Restart rabbitmq-server and look at the web interface: http://localhost:15672. The default credentials are guest/guest.
Message Queue
The concept of a Message Queue is very simple:
- a Queue, so things go in one end and out the other, first-in-first-out (FIFO)
- a Message Queue, so the "things" are messages.
A single instance of a Message Queue server such as RabbitMQ can manage lots of queues.
Exchange
An exchange is a router for messages: messages have routing keys, the exchange has bindings which map routing keys to queues so each message gets into the correct queue.
- Fanout exchanges route all incoming messages to every queue
- Direct exchanges route based on an exact string match (the routing key for the message and the queue must match exactly)
- Topic exchanges route based on a basic pattern match with just 2 metacharacters: '#' matches one or more words, '*' matches exactly one word, '.' is the word delimiter.