There are lots of options when it comes to choosing a message queue for your application. The guys at queues.io have a very comprehensive summary of the options.
There are some times, however, where you don’t need something as heavyweight as RabbitMQ or Amazon SQS. A popular alternative is to build a queue using Redis. A simple implementation will use
LPUSH to push messages onto the queue, and
BRPOP pull them off, respectively. Whilst this is useful for a basic FIFO queue, it does not support delayed messages – pushing a message to be pulled from the queue at a later date.
This post explores how a simple message queue can be built to support delayed messages, using Redis.
We’ll be using the following Redis data structures for our queue:
- A hash to store the message with a unique message ID.
- A sorted set to store the message ID with a timestamp, or message due date.
Adding a message
To add a message, we first need to generate a unique ID for the message. You’ll likely already know how to generate an ID in your programming language of choice – for example, if you are using node.js, you could use node-uuid or bson-objectid.
Once you have this unique ID, store it with your (stringified) message in the hash with
We should then insert the message ID with the message due date timestamp into the sorted set:
Now that we have our messages being produced, we need a consumer.
To receive a message, we need to poll the sorted set to see if any messages are past their due date. We do this with a periodic
ZRANGEBYSCORE call, along with the current timestamp.
If this returns a message ID, we know that this message is ready for consumption. We lookup the message from the hash:
And finally, we remove the message from the sorted set and the hash, to ensure it won’t be received again.
Note that this is a very simple queue implementation, and is sufficient for basic purposes. For a production application, however, it’s likely that you’ll need a way to retry messages if a consumer crashes before the message has been processed, and perhaps some way of determining the total number of messages in the queue.
If you are using node.js, I would recommend taking a look at Redis Simple Message Queue. This implements the basic algorithm described above, and adds extra features such as message retrying.