Message queues are a software architecture pattern that solves a lot of issues that distributed systems face. When I first started my career, I had to learn what a message brokering system (RabbitMQ, specifically) was and how I’d go about using it for the system I was building.
» This is a free preview of what you can expect with a subscription to Python Snacks Pro - you’ll receive one article like this once per month taking a deep dive into Python-adjacent topics such as system design, data science, data engineering, DevOps, and much more. Check out all of the perks to Python Snacks Pro.
Table of Contents
What are messages?
If you’ve ever looked up an analogy for message queuing, you’ll often see the analogy of the mail system:
You write a letter to your friend - often referred to as the “publisher” or “producer”.
The post office is responsible for delivering it - often referred to as the “message queue”
Your friend receives the letter - generally referred to as the “consumer”.
Think of messages as the letter you’re sending to your friend - it contains something that your friend could use, whether if that would just be a nice little note or a gift card.
In software, these messages are dictionaries:
{
"task": "send_email",
"to": "[email protected]"
}What are message queues?
Message queues are like a “holding area” for your messages - publishers will send a message to this queue, and consumers will read the message:
» Related reading: what is a queue
Say you’re working on building a data pipeline that’s relatively straight-forward: you’re pulling raw data into your system, transforming it and writing it to a database, then sending the data downstream to customers:
At the surface, this works and is a valid solution if we were to stuff it into one script and call it a day. However, there’s a few major issues we can pick up right off the bat:
If your code throws an error writing to the database, you won’t be able to send the data downstream to customers since that code won’t run.
If you have multiple raw data ingest sources, you may want to have multiple threads running to ingest, process, and send, especially it’s a real-time system.
This is where message queues come in - they allow you to send data to another part of your system so that your services do one thing and one thing only:
The biggest advantages you get from the pub/sub architecture (“message queues”) are as follows:
Decoupling - all services (green boxes above) don’t have knowledge of each other. You can swap out, scale, or rewrite either side without touching the other.
Resilience - If a consumer goes down, messages continue to queue up and then get processed when the service recovers. Nothing is lost and the producer never knew there was a problem.
Scalability - you can spin up multiple consumers (multi-threaded or multi-processing) to chew through a backlog in parallel, or add multiple producers without any coordination logic. The queue absorbs the load.
Enjoying this article? Consider upgrading you subscription to Pro to receive these kinds of deep dives once per month, access to unique coding challenges to test your Python skills, and access to an exclusive community of Python developers.
Types of Delivery Guarantees
When a broker sends a message to a consumer, you’ll often think that messages are delivered once and only once. However, this isn’t the case sometimes and by default you could have the message sent more than once.
Message brokers allow you to configure message delivery with one of three methods: at-most-once, at-least-once, and exactly-once.
At-most-once
The message is delivered zero or one times, never more. The broker fires it off and doesn’t care about it afterwards.
This is the fastest option in terms of latency, as there’s no retries or acknowledgements it needs to wait on. However, the trade-off is risky: if your consumer crashes at the wrong moment, the message is gone forever.
At-least-once
With this delivery method, the message is delivered one or more times, which is generally the default (and generally what you’ll stick with).
The reason being is that this method of delivery reduces the risk of you losing your message in case something happens. It’s important to note that duplicate delivery is normal and is not an error.
This means that you should build your consumer to be okay with seeing the same message more than once (you might’ve heard of the term “idempotency”).
For instance, if you’re sending an email to someone and the message gets delivered twice, you may accidentally send them 2 emails instead of one. To safeguard against this, message consumers often come with a message ID. You can store this message ID in a SQLite database, redis, or in-memory dictionary depending upon your needs.
Exactly-once
Within its own boundaries, a broker can offer exactly-once (SQS FIFO and Kafka both advertise it). But the moment processing reaches outside the broker such as sending an email, charging a card, or calling an external API, guaranteeing a single delivery becomes effectively impossible.
So in practice you don't rely on a broker setting; you implement it yourself with the method above: at-least-once delivery plus a stored message ID to deduplicate.
Common bugs with working with queues
Queues solve a lot of problems, but where there’s problems that are being solved introduces new problems; if you’re not careful you’ll end up shooting yourself in the foot with these issues.
Backpressure/consumer lag
Backpressure happens when you push too many messages and there aren’t enough consumers to consume and process the message. This isn’t a message broker problem, but rather a problem with either your consumer or publisher.
There’s a few ways to diagnose if you’re experiencing backpressure:
The number of messages sitting in the queue increases over time.
Check the age of the oldest message. In SQS, you’ll look at
ApproximateAgeOfOldestMessage.
To handle it, there’s a few ways to fix it:
Lower the number of publishers/publishing frequency
Increase the number of consumers and/or increase your batch read size (
MaxNumberOfMessages).
No dead-letter queue
A dead letter queue (DLQ) is designed to hold messages that have malformed, bug-inducing, and/or failed processed messages for you to quickly diagnose what went wrong.
Not having one introduces the possibility of having messages that cause bugs to go unnoticed, which propagate to issues that won’t get caught until well later down the line.
Visibility Timeout
It’s important to note that a message becomes invisible for a visibility timeout window. This means that when you consume the message (sqs.receive_message), the message isn’t deleted in the queue until you call the appropriate delete_message method.
If the application were to crash or the processing takes longer than the timeout, the message will reappear in the queue.
Enjoying this article? Consider upgrading you subscription to Pro to receive these kinds of deep dives once per month, access to unique coding challenges to test your Python skills, and access to an exclusive community of Python developers.
Python Examples for publishing and consuming
There are several kinds of message queuing systems, such as Kafka and RabbitMQ. These examples use Amazon SQS, but the same concept applies for different brokers.
Before we can do anything, we need to be able to communicate with our AWS services by setting up a boto3 client (boto3 is the package to handle these things):
import boto3
import json
AWS_REGION = "us-east-1"
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/my-queue"
# The queue we'll be publishing and consuming to/from.
sqs = boto3.client("sqs", region_name=AWS_REGION)Publishing
To publish, or send, a message, our code may look something such as:
def publish(message: dict) -> str:
response = sqs.send_message(
QueueUrl=QUEUE_URL,
MessageBody=json.dumps(message),
)
return response["MessageId"]
# Example of publishing a message
message = {"text" : "Hello, world!", "action" : "print"}
msg_id = publish(message)Consuming
Once a message is published, we can consume it with the following code:
def consume(max_messages: int = 10, wait_seconds: int = 5):
# Step 1: consume the message
response = sqs.receive_message(
QueueUrl=QUEUE_URL,
MaxNumberOfMessages=max_messages,
WaitTimeSeconds=wait_seconds, # long polling
)
messages = response.get("Messages", [])
for msg in messages:
# Step 2: read the message
body = json.loads(msg["Body"])
print("Received:", body)
# Step 3: perform all processing
process_message(body)
# Step 4: Delete from queue (important)
sqs.delete_message(
QueueUrl=QUEUE_URL,
ReceiptHandle=msg["ReceiptHandle"],
)I want to point out something important here: the ordering. There’s a very specific reason as to why I highlight individual steps in this exact order: if you delete (“ack”) the message before the work is finished and the consumer crashes, the message is gone.
Happy coding!
📧 Join the Python Snacks Newsletter! 🐍
Want even more Python-related content that’s useful? Here’s 3 reasons why you should subscribe the Python Snacks newsletter:
Get Ahead in Python with bite-sized Python tips and tricks delivered straight to your inbox, like the one above.
Exclusive Subscriber Perks: Receive a curated selection of up to 6 high-impact Python resources, tips, and exclusive insights with each email.
Get Smarter with Python in under 5 minutes. Your next Python breakthrough could just an email away.
You can unsubscribe at any time.
Interested in starting a newsletter or a blog?
Do you have a wealth of knowledge and insights to share with the world? Starting your own newsletter or blog is an excellent way to establish yourself as an authority in your field, connect with a like-minded community, and open up new opportunities.
If TikTok, Twitter, Facebook, or other social media platforms were to get banned, you’d lose all your followers. This is why you should start a newsletter: you own your audience.
This article may contain affiliate links. Affiliate links come at no cost to you and support the costs of this blog. Should you purchase a product/service from an affiliate link, it will come at no additional cost to you.
