[ad_1]
|
Lots of of hundreds of shoppers use Amazon Easy Queue Service (SQS) to construct message-based functions to decouple and scale microservices, distributed programs, and serverless apps. When a message can’t be efficiently processed by the queue client, you may configure SQS to retailer it in a dead-letter queue (DLQ).
As a software program developer or architect, you’d like to look at and evaluate unconsumed messages in your DLQs to determine why they couldn’t be processed, establish patterns, resolve code errors, and finally reprocess these messages within the authentic queue. The life cycle of those unconsumed messages is a part of your error-handling workflow, which is usually handbook and time consuming.
Right now, I’m completely happy to announce the overall availability of a brand new enhanced DLQ administration expertise for SQS commonplace queues that permits you to simply redrive unconsumed messages out of your DLQ to the supply queue.
This new performance is obtainable within the SQS console and helps you give attention to the vital part of your error dealing with workflow, which consists of figuring out and resolving processing errors. With this new improvement expertise, you may simply examine a pattern of the unconsumed messages and transfer them again to the unique queue with a click on, and with out writing, sustaining, and securing any customized code. This new expertise additionally takes care of redriving messages in batches, decreasing general prices.
DLQ and Lambda Processor Setup
Should you’re already snug with the DLQ setup, then skip the setup and leap into the brand new DLQ redrive expertise.
First, I create two queues: the supply queue and the dead-letter queue.
I edit the supply queue and configure the Lifeless-letter queue part. Right here, I choose the DLQ and configure the Most receives, which is the variety of occasions after which a message is reprocessed earlier than being despatched to the DLQ. For this demonstration, I’ve set it to 1. Which means each failed message goes to the DLQ instantly. In a real-world setting, you would possibly need to set a better quantity relying in your necessities and based mostly on what a failure means with respect to your utility.
I additionally edit the DLQ to ensure that solely my supply queue is allowed to make use of this DLQ. This configuration is elective: when this Redrive enable coverage is disabled, any SQS queue can use this DLQ. There are circumstances the place you need to reuse a single DLQ for a number of queues. However normally it’s thought of finest practices to setup unbiased DLQs per supply queue to simplify the redrive part with out affecting price. Take into account that you’re charged based mostly on the variety of API calls, not the variety of queues.
As soon as the DLQ is appropriately arrange, I want a processor. Let’s implement a easy message client utilizing AWS Lambda.
The Lambda operate written in Python will iterate over the batch of incoming messages, fetch two values from the message physique, and print the sum of those two values.
import json
def lambda_handler(occasion, context):
for document in occasion['Records']:
payload = json.masses(document['body'])
value1 = payload['value1']
value2 = payload['value2']
value_sum = value1 + value2
print("the sum is %s" % value_sum)
return "OK"
The code above assumes that every message’s physique accommodates two integer values that may be summed, with out coping with any validation or error dealing with. As you may think about, this may result in hassle in a while.
Earlier than processing any messages, you need to grant this Lambda operate sufficient permissions to learn messages from SQS and configure its set off. For the IAM permissions, I take advantage of the managed coverage named AWSLambdaSQSQueueExecutionRole, which grants permissions to invoke sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:GetQueueAttributes.
I take advantage of the Lambda console to arrange the SQS set off. I might obtain the identical from the SQS console too.
Now I’m able to course of new messages utilizing Ship and obtain messages for my supply queue within the SQS console. I write {"value1": 10, "value2": 5} within the message physique, and choose Ship message.
After I have a look at the CloudWatch logs of my Lambda operate, I see a profitable invocation.
START RequestId: 637888a3-c98b-5c20-8113-d2a74fd9edd1 Model: $LATEST
the sum is 15
END RequestId: 637888a3-c98b-5c20-8113-d2a74fd9edd1
REPORT RequestId: 637888a3-c98b-5c20-8113-d2a74fd9edd1 Period: 1.31 ms Billed Period: 2 ms Reminiscence Measurement: 128 MB Max Reminiscence Used: 39 MB Init Period: 116.90 ms Troubleshooting powered by DLQ Redrive
Now what if a distinct producer begins publishing messages with the unsuitable format? For instance, {"value1": "10", "value2": 5}. The primary quantity is a string and that is fairly prone to develop into an issue in my processor.
In actual fact, that is what I discover within the CloudWatch logs:
To determine what’s unsuitable within the offending message, I take advantage of the brand new SQS redrive performance, choosing DLQ redrive in my dead-letter queue.
I take advantage of Ballot for messages and fetch all unconsumed messages from the DLQ.
After which I examine the unconsumed message by choosing it.
The issue is obvious, and I resolve to replace my processing code to deal with this case correctly. Within the perfect world, that is an upstream problem that needs to be fastened within the message producer. However let’s assume that I can’t management that system and it’s critically vital for the enterprise that I course of this new kind of messages.
Subsequently, I replace the processing logic as follows:
import json
def lambda_handler(occasion, context):
for document in occasion['Records']:
payload = json.masses(document['body'])
value1 = int(payload['value1'])
value2 = int(payload['value2'])
value_sum = value1 + value2
print("the sum is %s" % value_sum)
# do some extra stuff
return "OK"
Now that my code is able to course of the unconsumed message, I begin a brand new redrive activity from the DLQ to the supply queue.
By default, SQS will redrive unconsumed messages to the supply queue. However you may additionally specify a distinct vacation spot and supply a customized velocity to set the utmost variety of messages per second.
I watch for the redrive activity to finish by monitoring the redrive standing within the console. This new part at all times exhibits the standing of most up-to-date redrive activity.
The message has been moved again to the supply queue and efficiently processed by my Lambda operate. Every little thing seems to be wonderful in my CloudWatch logs.
START RequestId: 637888a3-c98b-5c20-8113-d2a74fd9edd1 Model: $LATEST
the sum is 15
END RequestId: 637888a3-c98b-5c20-8113-d2a74fd9edd1
REPORT RequestId: 637888a3-c98b-5c20-8113-d2a74fd9edd1 Period: 1.31 ms Billed Period: 2 ms Reminiscence Measurement: 128 MB Max Reminiscence Used: 39 MB Init Period: 116.90 ms Out there Right now at No Extra Value
Right now you can begin leveraging the brand new DLQ redrive expertise to simplify your improvement and troubleshooting workflows, with none extra price. This new console expertise is obtainable in all AWS Areas the place SQS is obtainable, and we’re wanting ahead to listening to your suggestions.
Try the DLQ redrive documentation right here.
— Alex
[ad_2]
