Messaging

AWS Intermediate/Advanced 8 put an API in front of a Lambda function that answers a request and finishes. Real systems also do work that should not make the caller wait, must survive a failure, or must reach several places. Four services cover that: SQS holds messages until a consumer takes them, SNS pushes one message to many subscribers, EventBridge routes events to targets by their content, and Step Functions runs a multi-step workflow with retries and error paths. This chapter explains each from the documentation and then tests what can be tested on a laptop. Event-driven architecture as a design idea is not re-taught here; the AWS-specific rules and limits are.

Which Service for Which Job?

The comparison below uses only facts from the pages I read; the last column is where I add a judgement, marked as such.

A common chain, and the one this chapter builds, is: something publishes to an SNS topic; a filtered subscription puts matching messages on an SQS queue; a Lambda function drains the queue; messages that keep failing land in a dead-letter queue; and a state machine handles the multi-step part with retries. EventBridge sits beside it for events that come from AWS itself or from other applications.

Can the Emulator Be Trusted?

I ran the same questions against the documentation and the emulators and counted the agreements. The result is mixed, and it decides how the rest of the chapter uses the emulator: good for behaviour, poor for limits, and with one real defect.

The details follow in each service's section. The practical reading: use the emulators to test how your code behaves (does my consumer cope with a redelivery, does my pattern match my event), and use something else for limits and for validity (cfn-lint, your own checks, and ultimately the real service).

SQS: Queues, Visibility and Dead Letters

The rules, from the documentation

• Visibility timeout. A received message "remains in the queue but becomes temporarily invisible"; the default is 30 seconds, the range 0 seconds to 12 hours. If you do not delete it before the timeout, it becomes visible again. ChangeMessageVisibility can extend or shorten the timeout, and setting it to 0 makes the message available at once. "Extending the timeout doesn't reset this 12-hour limit."
• At-least-once. "There's no absolute guarantee that a message won't be delivered more than once during the visibility timeout period." Design consumers so that seeing a message twice is harmless.
• Dead-letter queues. A redrive policy gives a maxReceiveCount: "the number of times a consumer can receive a message from a source queue before it is moved to a dead-letter queue". The dead-letter queue must be in the same account and Region. For a standard queue the message keeps its original enqueue timestamp, so "it is a best practice to always set the retention period of a dead-letter queue to be longer than the retention period of the original queue"; for FIFO queues the timestamp resets. The page warns against a dead-letter queue on a FIFO queue where the exact order must survive.
• Polling. Short polling (the default) "queries a subset of servers" and a call "might not return all of your messages"; long polling (a wait time above 0, maximum 20 seconds) queries all servers and returns as soon as a message is available.
• FIFO. MessageGroupId is required; messages in a group are delivered in order, and while one is in flight "subsequent messages in that group are not made available until the in-flight message is either deleted or the visibility timeout expires". A resend with the same deduplication ID inside the 5-minute interval adds nothing; content-based deduplication hashes the body but "not the attributes of the message". Each partition is limited to 300 transactions per second per action (3,000 messages per second with batching of 10) unless high-throughput mode is on.
• Limits. Message size 1 byte to 1,048,576 bytes (1 MiB); retention 60 seconds to 14 days; delay 0 to 15 minutes; 10 messages per batch; 10 message attributes; group ID up to 128 characters; batch entry ID up to 80 characters (alphanumerics, hyphen, underscore); about 120,000 in-flight messages on a standard queue (OverLimit under short polling, silently nothing under long polling).
• Price (partial). The page I read gave no price table. It says every action is a request, each 64 KB chunk of a payload is billed as one request (a 1 MiB message is 16 requests), and the first million requests a month are free. FIFO and standard requests have different rates that I could not read.

Run here: behaviour against the emulator

Each check sends real SQS API calls to moto and compares the result with the sentence above. Two mistakes of mine are worth keeping: my first FIFO ordering check asked for 10 messages in one call and got the whole group back at once (g1-1, g1-2 and g2-1), which looked like a failure; the page I read does not say whether one call may return several messages of a group, so I split it into an unscored information line and a one-message-at-a-time check that matches the quoted sentence.

All ten scored checks agreed. The limits are another matter. I sent one value just inside each documented limit and one just outside:

moto enforced only 6 of the 12 limits. It accepted a visibility timeout of 43,201 seconds, retention of 59 and 1,209,601 seconds, 11 message attributes, a 129-character group ID and a control character in the body. A test suite on this emulator would pass code that the real service rejects. This is why the template tests later in the chapter use cfn-lint, which caught the queue-attribute ranges.

At-least-once in practice: a slow consumer

Run here: a queue with a 2-second visibility timeout, a message that takes 3.5 seconds to process, and two workers. The first worker is still busy when the message reappears and the second takes it:

The naive consumer charged order A-1001 twice. The idempotent one saw the repeat at 2.1 seconds, recognised the order ID and skipped it, so the charge happened once. Here the "already processed" check is an in-memory set; in a real system it must be shared and atomic (a conditional write to a database is the usual tool; not run here). A first version of this demo crashed because I forgot to ask for the ApproximateReceiveCount attribute: the workers died after the side effect and before the delete, so the first numbers were meaningless until I fixed and re-ran it.

SNS: Fan-Out and Filter Policies

From the documentation: "By default, an Amazon SNS topic subscriber receives every message that's published to the topic." A subscription can carry a filter policy; with scope MessageAttributes it is matched against the attributes, with MessageBody against the message's JSON body. "If any single property in this policy doesn't match an attribute assigned to the message, the policy rejects the message." Raw message delivery to SQS strips the SNS metadata so "the message is sent as is" (otherwise the receiver gets a JSON envelope); with raw delivery to an SQS queue at most 10 message attributes can be sent, and messages with more are discarded. The pricing page says SNS does not charge for per-message delivery to SQS and Lambda, but gave no price table, so the publish price is not stated here.

Run here: the page's own example message and its two policies (one that should accept it, one that should reject it because it requires an encrypted attribute the message does not have), in both scopes, plus twelve operators one at a time:

moto agreed everywhere, including the anything-but, numeric and exists operators. Two caveats. First, my message delivered to the queue without any queue access policy; the documentation page on fan-out to SQS did not load through my fetch tool (it returned general knowledge instead), so from my own understanding, not from a page I read, the real service needs the queue's policy to allow sns.amazonaws.com to send. The emulator cannot tell you that you forgot it. Second, the template in this chapter includes such a policy and cfn-lint cannot tell if it is missing (see the template section).

EventBridge: Rules and Event Patterns

From the documentation: an event pattern "has the same structure as the events they match" and either matches or does not. All fields in the pattern must match; the values of a field are an array of alternatives (OR); if the event's value is an array, the pattern matches when "the intersection of the event pattern array and the event array is non-empty". The operators page lists, among others: prefix, suffix, equals-ignore-case, anything-but (with values, lists, prefix, suffix, equals-ignore-case and wildcard), numeric, exists, cidr, wildcard (event buses only) and $or. Numbers are compared as 64-bit doubles, exact only up to ±2⁵³; a pattern with more than 1,000 $or combinations is rejected; if a key is repeated in a pattern "the last reference will be the one used". Null and empty strings "are not interchangeable". A warning on the page is worth repeating: a rule can trigger the very change it detects and loop, with "higher-than-expected charges and throttling".

Delivery, from the docs: if a target cannot be reached, EventBridge "retries sending the event for 24 hours and up to 185 times with an exponential back off and jitter" by default (both settable per target); after that "the event is dropped" unless the target has a dead-letter queue. Price (page, Region not stated, from worked examples): custom events $1.00 per million; AWS management events are free to ingest; Scheduler invocations are free for 14,000,000 a month and then $1.00 per million.

Run here: a reference matcher and the emulator

Trying to test patterns, I found that moto's TestEventPattern is a stub: it returned an empty 200 reply for a matching pattern and for a non-matching one alike. The test therefore had to use the real path: a rule on a fresh bus, an SQS queue as target, put_events, and a look at whether a message arrived. As a second opinion I wrote a reference matcher that implements only what the pages above state:

Then 43 cases, each with the outcome the docs' example or sentence implies:

The reference matcher agreed with the documentation on all 43 cases (so it is a fair second opinion); moto agreed on 42. The one disagreement is a real defect in moto 5.2.4 for something the docs show directly ("time": [{"prefix": "2017-10-02"}] is the docs' own example). I narrowed it down:

A prefix on source, on detail-type and on the same text inside detail all matched; an exact value or prefix on the top-level time never did, while exists on time did. moto stores the time exactly as sent, so I could not tell the cause from outside; the lesson is that an emulator can fail on one field and agree on the rest. Not tested: IPv6 addresses, pattern size limits, the wildcard complexity limit, archives and replay, schedules.

Step Functions: Workflows with Retry and Catch

Standard or Express, from the docs: Standard workflows run up to a year, are "exactly-once", keep history for 90 days after closing, and are billed per state transition; Express workflows run up to 5 minutes, are "at-least-once" (asynchronous) or "at-most-once" (synchronous), and are billed by requests, duration and memory. Express workflows do not support the job-run (.sync) or callback (.waitForTaskToken) integration patterns, Distributed Map or activities. The workflow type cannot be changed after creation. Quotas worth knowing: a Standard execution fails if its history reaches 25,000 events; task, state and execution input or output is limited to 256 KiB; names are limited to 80 characters; 1,000,000 open executions per account per Region.

Errors, from the docs. Any state except Pass and Wait can fail, and "Step Functions defaults to failing the entire state machine execution". Task, Parallel and Map states can have Retry (a list of retriers) and Catch. A retrier has ErrorEquals, IntervalSeconds (default 1), MaxAttempts (default 3; 0 means never retry), BackoffRate (default 2.0), an optional MaxDelaySeconds (above 0) and JitterStrategy (FULL or NONE, default NONE). States.ALL "must appear alone in the ErrorEquals array and must appear in the last retrier"; it does not catch States.DataLimitExceeded or States.Runtime; States.TaskFailed matches any known error except States.Timeout. Retries run first and the matching catcher second, and "retries are treated as state transitions" for billing.

Run here: retry timing on Step Functions Local

Step Functions Local calls no real services when a mock configuration scripts each Task's result. The docs describe the file: StateMachines with TestCases, and MockedResponses numbered by attempt ("0-1" is attempts 0 and 1) with Return or Throw; every attempt a retry might make needs a mocked response, or the execution fails. The state machine, the mock file and the results:

The waits matched the pages in every scenario: 1 then 2 seconds; 1, 2, 4 when all retries are used before the catcher takes over; 3, 5, 5 with MaxDelaySeconds 5 and 3, 6, 12 without it (the docs' own two examples); an error not listed in ErrorEquals was not retried at all; and the docs' "retry everything except States.Timeout" policy did not retry a timeout and did retry another error. The two JitterStrategy FULL runs gave different waits (1, 3, 8 and 2, 3, 1 seconds), each inside its documented range of 0 to 2, 0 to 4 and 0 to 8; both runs produced whole seconds, which the page does not promise. The execution status was SUCCEEDED after a catch because the fallback state was a Pass state: a handled failure is a success as far as the execution status goes, so alarms on status alone will miss it.

Cost sketch

From the pricing page (a table was missing from the text, so these come from its examples): Standard is $0.000025 per state transition in US East (N. Virginia), with 4,000 free a month, and "each retry will be charged as an additional state transition". Express is $1.00 per million requests (N. Virginia) plus $0.00001667 per GB-second (Region not stated). Counting my own transitions as states entered plus retries (AWS's count may differ by one):

A path with three retries costs 2.5 times a path without any, so retries are not free; and the same page's free tier covers only 1,000 executions of a four-transition path.

What validates a state machine definition?

I changed one thing at a time in a definition with a Task, Choice, Wait, Parallel, Map, Succeed and Fail state, and asked two checkers: Step Functions Local (it validates on CreateStateMachine) and cfn-lint (the definition in a CloudFormation DefinitionString). The "expect" column says what the docs or the ASL (Amazon States Language) specification say; for the specification I rely on my own reading, because I did not fetch it this session:

Step Functions Local rejected 21 of the 22 invalid definitions; cfn-lint rejected 12. cfn-lint missed a Next pointing at no state, an unreachable state, both Wait mistakes, a Catch to a missing state, and every retry-field mistake the error-handling page defines (a jitter strategy of HALF, MaxDelaySeconds 0, States.ALL not last or not alone). Both accepted a Retry with IntervalSeconds 0 although the page says the value is "a positive integer"; I do not know what the real service does. AWS's own alternatives need an account: the TestState API (the Step Functions Local page recommends it instead of Step Functions Local) and the ValidateStateMachineDefinition API (it appears in the quotas page; I did not read how it behaves).

Lambda as a Queue Consumer

From the documentation (Lambda SQS pages):

• Set the queue's visibility timeout "to at least six times the configuration timeout on your function"; with a batching window, six times the timeout plus MaximumBatchingWindowInSeconds. The function timeout must not exceed the queue's visibility timeout; Lambda checks this and returns an error.
• Set maxReceiveCount to at least 5 on a queue read by Lambda, which gives Lambda a few chances to retry before a message goes to the dead-letter queue.
• Batch size up to 10,000 on a standard queue and 10 on a FIFO queue; above 10 you must set a batching window of at least 1 second (standard queues only). A batch is also limited by the 6 MB synchronous payload.
• By default, an error fails the whole batch and every message in it returns to the queue after the visibility timeout. With ReportBatchItemFailures the function returns {"batchItemFailures": [{"itemIdentifier": "id2"}]} and only those messages return. An empty or null list is a complete success; an invalid JSON response, an empty or null itemIdentifier, a wrong key name or an ID that does not exist is a complete failure; an exception fails the whole batch. For a FIFO queue the function "should stop processing messages after the first failure and return all failed and unprocessed messages".

Run here: a handler that reports only the records it cannot process, run in a real Lambda Python 3.13 container with sam local invoke and a five-record event made from the SAM-generated SQS fixture (which has a single record with the body Hello from SQS!; I copied the record five times and changed the IDs and bodies):

The function returned exactly the documented shape for records 2 and 4. What this run does not show is what the Lambda service then does with the response and the queue: that is the event source mapping, which only exists on AWS, so the rules in the list above are from the documentation.

The Whole Design as a Template

Run here: one CloudFormation template for the chain (queue with a dead-letter queue, a FIFO queue, a topic with a filtered subscription and its queue policy, an event bus with a rule and an audit queue, the Lambda consumer and its mapping, and a state machine):

It linted clean with cfn-lint 1.57.2 (exit status 0). My first draft of this template had a mistake that cfn-lint did not report: I set the queue's visibility timeout to 180 seconds with a 30-second function and a 5-second batching window, but the docs' rule is 6 × 30 + 5 = 185. I found it when I reread the Lambda page against the template, not through a tool, and then wrote the small checker below for the rules that need two resources together. I changed one thing at a time in the template (30 changes):

cfn-lint reported an error for 15 of 30 and passed or only warned for 15. It caught all the single-property ranges (the SQS limits above, a FIFO name without .fifo, batch sizes, a non-numeric retry count, a dead-letter queue that does not exist or is the queue itself). It missed everything that needs knowledge of two resources, or of what a string means: a function timeout longer than the queue's visibility timeout, a timeout shorter than six times the function's, a batch of 100 with no window, a subscription endpoint that is the queue URL instead of its ARN, RawMessageDelivery: yes, a filter-policy scope or operator that does not exist, a missing queue policy, a queue policy that trusts the wrong service, an event pattern with an operator that does not exist, a callback state in an Express workflow, and a state machine definition that is broken or points at a missing state.

Two of those misses I probed further, because the definition was inside Fn::Sub. With a plain DefinitionString cfn-lint also passed a definition that is not valid JSON, a Next to a missing state and an Express workflow with .waitForTaskToken; wrapped in Fn::Sub it printed only W1020 (a substitution rule). So the substitution is not the cause: this version does not check those things at all.

The checker I wrote reads the template, follows Ref and GetAtt between the mapping, the queue and the function, and applies the Lambda page's rules:

It flagged the three mutations cfn-lint missed in this group (function timeout above the queue's visibility timeout, a visibility timeout below six times the function's, a batch of 100 with no window) and my own 180, and I added three rules for good measure (FIFO batch size, maxReceiveCount below 5, a dead-letter queue that does not outlive its source). It is a sketch for this template's shape, not a general tool: a project that wants these checks should keep a script like it next to the template, with tests like the mutations above.

Against Real AWS (not run)

Level 2, needs an account. The boto3 calls used in this chapter are the same ones you would make against the real services: drop the endpoint_url, set credentials and a Region, and the code in sqs_behaviour.py, sns_tests.py, eb_pattern_tests.py and consumer_demo.py runs against AWS (queues and topics have names you must make unique, and the tests leave their resources behind, so a real run needs a clean-up step I did not write). There is no AWS CLI on this machine, so I give no CLI commands. The template deploys with aws cloudformation deploy or sam deploy (AWS Intermediate/Advanced 4 and 8), and costs money only for requests and state transitions at the rates above; set a budget alert first (AWS Fundamentals 9) and delete the stack afterwards.

Cost and Clean-Up

Nothing in this chapter created anything in AWS. On this machine: the scratch folder C:/lwaws/c9 (a Python environment and test files), the moto server process (stopped at the end) and the Step Functions Local container (removed at the end), and three Docker images: the Step Functions Local image (1.89 GB) and the two Lambda images from chapter 8 (795 MB and 834 MB). SAM's local invoke leaves no container behind (checked with docker ps -a).

Practice