Actions: | Security

AllGoodBits.org

Navigation: Home | Services | Tools | Articles | Other

Message Queueing with AMQP

My basic intention for using a message queue was to decouple requests from fulfillment, in several ways:

My use case which was a low-volume, asynchronous task queue; the details are out of scope for this article, but using a message queue had various benefits:

Basically, from a development perspective, the result of using AMQP and a message broker is that I can write code that deals directly with the data that I care about using abstract interfaces and not deal with transports.

From a systems/operations perspective, I get reliability, modularity and the ability to (re)provision and administer my service without interrupting availability. Using a mature broker makes it easier to move to clustering/HA/federation.

Basic Concepts

I started poking around in likely places for hints and came up with ZeroMQ (aka 0MQ). Although I was able to make ZeroMQ do something minimally useful very easily, I soon noticed that it does not support message persistence so I chose a message queueing system with a broker that does. For my use case, that was required.

Broker

There are several message broker projects, but not knowing much about them or reasons to choose between them, I picked RabbitMQ, mostly because it seemed that it was easier to get going than Apache Qpid. They both appear to be mature, active projects that would do the job.

It is extremely easy to install on modern unix systems, pick one:

yum install librabbitmq rabbitmq-server apt-get install librabbitmq0 rabbitmq-server brew install rabbitmq

Then start it up, according to your system's preferred service management approach.

Enable RabbitMQ's management plugin:

rabbitmq-plugins enable rabbitmq_management

Restart rabbitmq-server and look at the web interface: http://localhost:15672. The default credentials are guest/guest.

Message Queue

The concept of a Message Queue is very simple:

  • a Queue, so things go in one end and out the other, first-in-first-out (FIFO)
  • a Message Queue, so the "things" are messages.

A single instance of a Message Queue server such as RabbitMQ can manage lots of queues.

Exchange

An exchange is a router for messages: messages have routing keys, the exchange has bindings which map routing keys to queues so each message gets into the correct queue.

  • Fanout exchanges route all incoming messages to every queue
  • Direct exchanges route based on an exact string match (the routing key for the message and the queue must match exactly)
  • Topic exchanges route based on a basic pattern match with just 2 metacharacters: '#' matches one or more words, '*' matches exactly one word, '.' is the word delimiter.

Send Messages

There are many libraries for messaging, most major languages will have something suitable; my recent messaging project used Python. Kombu provides synchronous messaging support, apparently txAMQP is the one to look at for asynchronous support if you're into Twisted.