SyncValsverifier → artifact → classifier → verdict
SyncVals · Trajectory

ddb-outbox-eventbridge-fanout

claude-code claude-opus-4-7 ✓ resolved GOOD_SUCCESS ↑ View task
Solved from the instruction alone, tests/ and solution/ were withheld from the agent's workspace and restored only for grading.
Reward = tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.
Classification , post-hoc; cannot change the reward
GOOD_SUCCESSLegitimate solve, implements the asked-for behavior; tests verify real functionality.
SubtypeLegitimate Solution
EvidenceAll 20 tests passed: test_01 through test_20, including infrastructure validation (KMS, DynamoDB, SQS, EventBridge), encryption checks, IAM role scoping, stream configuration with NEW_AND_OLD_IMAGES, queue redrive policies, ReportBatchItemFailures on ESM, and e2e routing tests for both OrderCreated and OrderCancelled events plus idempotency verification (test_20_e2e_idempotent_duplicate_does_not_double_fanout).
Root causeThe agent correctly implemented a complex distributed systems task involving DynamoDB outbox pattern, EventBridge fanout, SQS routing, and least-privilege IAM from scratch. All 20 comprehensive tests passed, validating infrastructure configuration, encryption, routing logic, idempotency, and end-to-end event flow.
RecommendationN/A - task is fine
Trajectory
Tool-by-tool agent trajectory
40 tool calls · 4 tool types · 42 steps
ok the checkout service has been dropping events again. every time the orders table gets a new row, *something* downstream needs to know , pricing, fulfillment, the analytics folks, all of them. right now we're just hoping two sequential putitems stick. they don't. last week we had a row written and no event fired because the lambda crashed between the two writes. nobody noticed for 6 hours. so: outbox pattern. one transaction, two rows, and let the stream do the fanout. build it on localstack , `http://localstack:4566`, creds already exported (`AWS_ACCESS_KEY_ID=test`, same for secret, region `us-east-1`). you've got `aws`, `python3`, `boto3`, `jq`, `zip`. build the whole thing from zero. shape of it: - an "order api" lambda is the only thing that writes orders. it takes `{order_id, kind, ...}` where `kind` is either `OrderCreated` or `OrderCancelled`. it writes the business row **and** the outbox row in one atomic step. no half-states allowed. - the outbox table has a stream , view type `NEW_AND_OLD_IMAGES` (the relay needs to see both the new and old image of the row, not just keys, so downstream subscribers can react on diffs and not just inserts). a relay lambda reads the stream via an event source mapping and republishes onto a custom eventbridge bus. - the relay's event source mapping must use `ReportBatchItemFailures` so a single bad record can fail without retrying the whole batch , the relay returns `{"batchItemFailures": [{"itemIdentifier": "<seq#>"}, ...]}` for the records it couldn't republish. - two rules on that bus route by kind: `OrderCreated` goes to one sqs queue, `OrderCancelled` goes to another. each queue has its own dlq. no shared dlq. - the relay has to be idempotent , if the stream redelivers the same record, we don't want the downstream queue to see the message twice. keep a small ddb table for dedup, ttl it so it doesn't grow forever. - everything is encrypted with a customer-managed kms key. not `aws/dynamodb`, not `aws/sqs`. ours. done looks like this: invoke `order-api` with `{order_id: "o-1", kind: "OrderCreated", ...}`, and within 30s: - `orders-created` sqs queue has exactly 1 message - `orders-cancelled` sqs queue has 0 messages - `processed_events` table has a row for the corresponding outbox event_id then invoke again with `kind: "OrderCancelled"` and a different `order_id`: - `orders-cancelled` picks it up, `orders-created` stays quiet least privilege: every role scoped to its own resources. the outbox-relay role should not be able to touch the orders table's stream; the order-api role should not be able to touch sqs; nobody gets `*` on kms. the verifier reads the policy docs directly, not just "does it work end-to-end". names the verifier grabs by hand, use them exactly: | thing | name | |-|-| | kms alias | `alias/outbox-cmk` | | orders table | `orders` (pk `order_id` / String) | | outbox table | `outbox` (pk `event_id` / String) | | dedup table | `processed_events` (pk `event_id` / String) | | api lambda | `order-api` | | api lambda role | `order-api-role` | | relay lambda | `outbox-relay` | | relay lambda role | `outbox-relay-role` | | event bus | `order-events` | | rule for created | `on-order-created` | | rule for cancelled | `on-order-cancelled` | | created queue | `orders-created` | | cancelled queue | `orders-cancelled` | | created dlq | `orders-created-dlq` | | cancelled dlq | `orders-cancelled-dlq` | eventbridge detail shape: the relay republishes with `source = "outbox.relay"` and `detail-type` matching `kind` exactly , `OrderCreated` or `OrderCancelled`. rule patterns match by those. python3.11 for both lambdas. they run on the shared compose network so `http://localstack:4566` is reachable from inside them. localstack community 4.13.1. outbox rows need an `event_id` so the relay has a stable dedup key; generate one however you like, just make sure it's the same id when the row gets read off the stream.

Check working directory and available tools

shell
ls /app && echo "---" && which aws python3 jq zip && python3 --version && aws --version
output
---
/usr/local/bin/aws
/opt/venv/bin/python3
/usr/bin/jq
/usr/bin/zip
Python 3.12.3
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24

[stdout]
---
/usr/local/bin/aws
/opt/venv/bin/python3
/usr/bin/jq
/usr/bin/zip
Python 3.12.3
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24

Check localstack health

shell
curl -s http://localstack:4566/_localstack/health | jq . | head -50
output
{
  "services": {
    "acm": "disabled",
    "apigateway": "disabled",
    "cloudformation": "disabled",
    "cloudwatch": "disabled",
    "config": "disabled",
    "dynamodb": "available",
    "dynamodbstreams": "available",
    "ec2": "disabled",
    "es": "disabled",
    "events": "available",
    "firehose": "disabled",
    "iam": "available",
    "kinesis": "available",
    "kms": "available",
    "lambda": "available",
    "logs": "available",
    "opensearch": "disabled",
    "redshift": "disabled",
    "resource-groups": "disabled",
    "resourcegroupstaggingapi": "disabled",
    "route53": "disabled",
    "route53resolver": "disabled",
    "s3": "available",
    "s3control": "disabled",
    "scheduler": "disabled",
    "secretsmanager": "disabled",
    "ses": "disabled",
    "sns": "disabled",
    "sqs": "available",
    "ssm": "disabled",
    "stepfunctions": "disabled",
    "sts": "available",
    "support": "disabled",
    "swf": "disabled",
    "transcribe": "disabled"
  },
  "edition": "community",
  "version": "4.13.1"
}

[stdout]
{
  "services": {
    "acm": "disabled",
    "apigateway": "disabled",
    "cloudformation": "disabled",
    "cloudwatch": "disabled",
    "config": "disabled",
    "dynamodb": "available",
    "dynamodbstreams": "available",
    "ec2": "disabled",
    "es": "disabled",
    "events": "available",
    "firehose": "disabled",
    "iam": "available",
    "kinesis": "available",
    "kms": "available",
    "lambda": "available",
    "logs": "available",
    "opensearch": "disabled",
    "redshift": "disabled",
    "resource-groups": "disabled",
    "resourcegroupstaggingapi": "disabled",
    "route53": "disabled",
    "route53resolver": "disabled",
    "s3": "available",
    "s3control": "disabled",
    "scheduler": "disabled",
    "secretsmanager": "disabled",
    "ses": "disabled",
    "sns": "disabled",
    "sqs": "available",
    "ssm": "disabled",
    "stepfunctions": "disabled",
    "sts": "available",
    "support": "disabled",
    "swf": "disabled",
    "transcribe": "disabled"
  },
  "edition": "community",
  "version": "4.13.1"
}
ToolSearch
{
  "query": "select:TodoWrite",
  "max_results": 1
}

, output, 
{"type": "tool_reference", "tool_name": "TodoWrite"}
  • Create KMS CMK with alias outbox-cmk
  • Create DynamoDB tables (orders, outbox with stream, processed_events with TTL)
  • Create SQS DLQs and main queues
  • Create EventBridge bus and rules
  • Create IAM roles with least privilege
  • Write and deploy order-api lambda
  • Write and deploy outbox-relay lambda with stream ESM
  • End-to-end smoke test

Create build dirs and verify AWS access

shell
mkdir -p /app/build /app/policies /app/src && echo "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION" && export AWS_ENDPOINT_URL=http://localstack:4566 && aws sts get-caller-identity --endpoint-url $AWS_ENDPOINT_URL
output
AWS_ACCESS_KEY_ID=test AWS_DEFAULT_REGION=us-east-1
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}

[stdout]
AWS_ACCESS_KEY_ID=test AWS_DEFAULT_REGION=us-east-1
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}

Create KMS CMK and alias

shell
cat > /app/.env <<'EOF'
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1
export AWS_ENDPOINT_URL=http://localstack:4566
export ACCOUNT_ID=000000000000
EOF
source /app/.env && aws kms create-key --description "outbox CMK" --key-usage ENCRYPT_DECRYPT --customer-master-key-spec SYMMETRIC_DEFAULT --output json > /app/build/kms-key.json && cat /app/build/kms-key.json | jq -r '.KeyMetadata.KeyId' > /app/build/kms-key-id.txt && KEY_ID=$(cat /app/build/kms-key-id.txt) && aws kms create-alias --alias-name alias/outbox-cmk --target-key-id $KEY_ID && echo "KMS KeyId: $KEY_ID"
output
KMS KeyId: 0bbb32cd-65fd-40dc-af71-e284aeee1f8c

[stdout]
KMS KeyId: 0bbb32cd-65fd-40dc-af71-e284aeee1f8c

Get KMS key ARN

shell
source /app/.env && KEY_ID=$(cat /app/build/kms-key-id.txt) && KEY_ARN=$(aws kms describe-key --key-id $KEY_ID --query 'KeyMetadata.Arn' --output text) && echo $KEY_ARN > /app/build/kms-key-arn.txt && cat /app/build/kms-key-arn.txt
output
arn:aws:kms:us-east-1:000000000000:key/0bbb32cd-65fd-40dc-af71-e284aeee1f8c

[stdout]
arn:aws:kms:us-east-1:000000000000:key/0bbb32cd-65fd-40dc-af71-e284aeee1f8c
  • Create KMS CMK with alias outbox-cmk
  • Create DynamoDB tables (orders, outbox with stream, processed_events with TTL)
  • Create SQS DLQs and main queues
  • Create EventBridge bus and rules
  • Create IAM roles with least privilege
  • Write and deploy order-api lambda
  • Write and deploy outbox-relay lambda with stream ESM
  • End-to-end smoke test

Create DynamoDB tables with CMK encryption

shell
source /app/.env && KEY_ARN=$(cat /app/build/kms-key-arn.txt) && \
aws dynamodb create-table --table-name orders \
  --attribute-definitions AttributeName=order_id,AttributeType=S \
  --key-schema AttributeName=order_id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --sse-specification Enabled=true,SSEType=KMS,KMSMasterKeyId=$KEY_ARN \
  --output json > /app/build/orders-table.json && \
aws dynamodb create-table --table-name outbox \
  --attribute-definitions AttributeName=event_id,AttributeType=S \
  --key-schema AttributeName=event_id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES \
  --sse-specification Enabled=true,SSEType=KMS,KMSMasterKeyId=$KEY_ARN \
  --output json > /app/build/outbox-table.json && \
aws dynamodb create-table --table-name processed_events \
  --attribute-definitions AttributeName=event_id,AttributeType=S \
  --key-schema AttributeName=event_id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --sse-specification Enabled=true,SSEType=KMS,KMSMasterKeyId=$KEY_ARN \
  --output json > /app/build/processed-table.json && \
echo "tables created"
output
tables created

[stdout]
tables created

Enable TTL on processed_events and capture ARNs

shell
source /app/.env && \
aws dynamodb update-time-to-live --table-name processed_events \
  --time-to-live-specification Enabled=true,AttributeName=ttl --output json && \
ORDERS_ARN=$(jq -r '.TableDescription.TableArn' /app/build/orders-table.json) && \
OUTBOX_ARN=$(jq -r '.TableDescription.TableArn' /app/build/outbox-table.json) && \
OUTBOX_STREAM_ARN=$(jq -r '.TableDescription.LatestStreamArn' /app/build/outbox-table.json) && \
PROCESSED_ARN=$(jq -r '.TableDescription.TableArn' /app/build/processed-table.json) && \
echo "$ORDERS_ARN" > /app/build/orders-arn.txt && \
echo "$OUTBOX_ARN" > /app/build/outbox-arn.txt && \
echo "$OUTBOX_STREAM_ARN" > /app/build/outbox-stream-arn.txt && \
echo "$PROCESSED_ARN" > /app/build/processed-arn.txt && \
echo "orders: $ORDERS_ARN" && \
echo "outbox: $OUTBOX_ARN" && \
echo "outbox stream: $OUTBOX_STREAM_ARN" && \
echo "processed_events: $PROCESSED_ARN"
output
{
    "TimeToLiveSpecification": {
        "Enabled": true,
        "AttributeName": "ttl"
    }
}
orders: arn:aws:dynamodb:us-east-1:000000000000:table/orders
outbox: arn:aws:dynamodb:us-east-1:000000000000:table/outbox
outbox stream: arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:48:06.432
processed_events: arn:aws:dynamodb:us-east-1:000000000000:table/processed_events

[stdout]
{
    "TimeToLiveSpecification": {
        "Enabled": true,
        "AttributeName": "ttl"
    }
}
orders: arn:aws:dynamodb:us-east-1:000000000000:table/orders
outbox: arn:aws:dynamodb:us-east-1:000000000000:table/outbox
outbox stream: arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:48:06.432
processed_events: arn:aws:dynamodb:us-east-1:000000000000:table/processed_events
  • Create KMS CMK with alias outbox-cmk
  • Create DynamoDB tables (orders, outbox with stream, processed_events with TTL)
  • Create SQS DLQs and main queues
  • Create EventBridge bus and rules
  • Create IAM roles with least privilege
  • Write and deploy order-api lambda
  • Write and deploy outbox-relay lambda with stream ESM
  • End-to-end smoke test

Create SQS DLQs with KMS CMK

shell
source /app/.env && KEY_ID=$(cat /app/build/kms-key-id.txt) && \
for q in orders-created-dlq orders-cancelled-dlq; do
  aws sqs create-queue --queue-name $q \
    --attributes "{\"KmsMasterKeyId\":\"$KEY_ID\"}" --output json > /app/build/$q.json
done && \
CREATED_DLQ_URL=$(jq -r '.QueueUrl' /app/build/orders-created-dlq.json) && \
CANCELLED_DLQ_URL=$(jq -r '.QueueUrl' /app/build/orders-cancelled-dlq.json) && \
CREATED_DLQ_ARN=$(aws sqs get-queue-attributes --queue-url $CREATED_DLQ_URL --attribute-names QueueArn --query 'Attributes.QueueArn' --output text) && \
CANCELLED_DLQ_ARN=$(aws sqs get-queue-attributes --queue-url $CANCELLED_DLQ_URL --attribute-names QueueArn --query 'Attributes.QueueArn' --output text) && \
echo $CREATED_DLQ_URL > /app/build/orders-created-dlq-url.txt && \
echo $CANCELLED_DLQ_URL > /app/build/orders-cancelled-dlq-url.txt && \
echo $CREATED_DLQ_ARN > /app/build/orders-created-dlq-arn.txt && \
echo $CANCELLED_DLQ_ARN > /app/build/orders-cancelled-dlq-arn.txt && \
echo "DLQs: $CREATED_DLQ_ARN  $CANCELLED_DLQ_ARN"
output
DLQs: arn:aws:sqs:us-east-1:000000000000:orders-created-dlq  arn:aws:sqs:us-east-1:000000000000:orders-cancelled-dlq

[stdout]
DLQs: arn:aws:sqs:us-east-1:000000000000:orders-created-dlq  arn:aws:sqs:us-east-1:000000000000:orders-cancelled-dlq

Create main SQS queues with redrive to DLQs

shell
source /app/.env && KEY_ID=$(cat /app/build/kms-key-id.txt) && \
CREATED_DLQ_ARN=$(cat /app/build/orders-created-dlq-arn.txt) && \
CANCELLED_DLQ_ARN=$(cat /app/build/orders-cancelled-dlq-arn.txt) && \
aws sqs create-queue --queue-name orders-created \
  --attributes "{\"KmsMasterKeyId\":\"$KEY_ID\",\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$CREATED_DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}" \
  --output json > /app/build/orders-created.json && \
aws sqs create-queue --queue-name orders-cancelled \
  --attributes "{\"KmsMasterKeyId\":\"$KEY_ID\",\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$CANCELLED_DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}" \
  --output json > /app/build/orders-cancelled.json && \
CREATED_URL=$(jq -r '.QueueUrl' /app/build/orders-created.json) && \
CANCELLED_URL=$(jq -r '.QueueUrl' /app/build/orders-cancelled.json) && \
CREATED_ARN=$(aws sqs get-queue-attributes --queue-url $CREATED_URL --attribute-names QueueArn --query 'Attributes.QueueArn' --output text) && \
CANCELLED_ARN=$(aws sqs get-queue-attributes --queue-url $CANCELLED_URL --attribute-names QueueArn --query 'Attributes.QueueArn' --output text) && \
echo $CREATED_URL > /app/build/orders-created-url.txt && \
echo $CANCELLED_URL > /app/build/orders-cancelled-url.txt && \
echo $CREATED_ARN > /app/build/orders-created-arn.txt && \
echo $CANCELLED_ARN > /app/build/orders-cancelled-arn.txt && \
echo "Queues: $CREATED_ARN  $CANCELLED_ARN"
output
Queues: arn:aws:sqs:us-east-1:000000000000:orders-created  arn:aws:sqs:us-east-1:000000000000:orders-cancelled

[stdout]
Queues: arn:aws:sqs:us-east-1:000000000000:orders-created  arn:aws:sqs:us-east-1:000000000000:orders-cancelled
  • Create KMS CMK with alias outbox-cmk
  • Create DynamoDB tables (orders, outbox with stream, processed_events with TTL)
  • Create SQS DLQs and main queues
  • Create EventBridge bus and rules
  • Create IAM roles with least privilege
  • Write and deploy order-api lambda
  • Write and deploy outbox-relay lambda with stream ESM
  • End-to-end smoke test

Create EventBridge bus with CMK

shell
source /app/.env && KEY_ARN=$(cat /app/build/kms-key-arn.txt) && \
aws events create-event-bus --name order-events --kms-key-identifier $KEY_ARN --output json > /app/build/bus.json && \
BUS_ARN=$(jq -r '.EventBusArn' /app/build/bus.json) && echo $BUS_ARN > /app/build/bus-arn.txt && echo "bus: $BUS_ARN"
output
bus: arn:aws:events:us-east-1:000000000000:event-bus/order-events

[stdout]
bus: arn:aws:events:us-east-1:000000000000:event-bus/order-events

Create EventBridge rules

shell
source /app/.env && \
aws events put-rule --name on-order-created --event-bus-name order-events \
  --event-pattern '{"source":["outbox.relay"],"detail-type":["OrderCreated"]}' --output json > /app/build/rule-created.json && \
aws events put-rule --name on-order-cancelled --event-bus-name order-events \
  --event-pattern '{"source":["outbox.relay"],"detail-type":["OrderCancelled"]}' --output json > /app/build/rule-cancelled.json && \
echo "rules created"
output
rules created

[stdout]
rules created

Wire EventBridge rule targets to SQS

shell
source /app/.env && \
CREATED_RULE_ARN=$(jq -r '.RuleArn' /app/build/rule-created.json) && \
CANCELLED_RULE_ARN=$(jq -r '.RuleArn' /app/build/rule-cancelled.json) && \
CREATED_QUEUE_ARN=$(cat /app/build/orders-created-arn.txt) && \
CANCELLED_QUEUE_ARN=$(cat /app/build/orders-cancelled-arn.txt) && \
CREATED_QUEUE_URL=$(cat /app/build/orders-created-url.txt) && \
CANCELLED_QUEUE_URL=$(cat /app/build/orders-cancelled-url.txt) && \
# put SQS resource policies allowing EventBridge to send
cat > /app/policies/sqs-created-policy.json <<EOF
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"events.amazonaws.com"},"Action":"sqs:SendMessage","Resource":"$CREATED_QUEUE_ARN","Condition":{"ArnEquals":{"aws:SourceArn":"$CREATED_RULE_ARN"}}}]}
EOF
cat > /app/policies/sqs-cancelled-policy.json <<EOF
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"events.amazonaws.com"},"Action":"sqs:SendMessage","Resource":"$CANCELLED_QUEUE_ARN","Condition":{"ArnEquals":{"aws:SourceArn":"$CANCELLED_RULE_ARN"}}}]}
EOF
aws sqs set-queue-attributes --queue-url "$CREATED_QUEUE_URL" \
  --attributes "Policy=$(jq -c . /app/policies/sqs-created-policy.json)" && \
aws sqs set-queue-attributes --queue-url "$CANCELLED_QUEUE_URL" \
  --attributes "Policy=$(jq -c . /app/policies/sqs-cancelled-policy.json)" && \
aws events put-targets --rule on-order-created --event-bus-name order-events \
  --targets "Id=1,Arn=$CREATED_QUEUE_ARN" --output json && \
aws events put-targets --rule on-order-cancelled --event-bus-name order-events \
  --targets "Id=1,Arn=$CANCELLED_QUEUE_ARN" --output json && \
echo "targets attached"
output
Exit code 252

aws: [ERROR]: An error occurred (ParamValidation): Error parsing parameter '--attributes': Expected: '=', received: '"' for input:
 Policy={"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"events.amazonaws.com"},"Action":"sqs:SendMessage","Resource":"arn:aws:sqs:us-east-1:000000000000:orders-created","Condition":{"ArnEquals":{"aws:SourceArn":"arn:aws:events:us-east-1:000000000000:rule/order-events/on-order-created"}}}]}
        ^

[error] tool reported failure

Set SQS queue policies via cli-input-json

shell
source /app/.env && \
CREATED_QUEUE_URL=$(cat /app/build/orders-created-url.txt) && \
CANCELLED_QUEUE_URL=$(cat /app/build/orders-cancelled-url.txt) && \
POLICY_CREATED=$(jq -c . /app/policies/sqs-created-policy.json | jq -Rs .) && \
POLICY_CANCELLED=$(jq -c . /app/policies/sqs-cancelled-policy.json | jq -Rs .) && \
jq -n --arg url "$CREATED_QUEUE_URL" --argjson p "$POLICY_CREATED" '{QueueUrl:$url,Attributes:{Policy:$p}}' > /app/build/set-attr-created.json && \
jq -n --arg url "$CANCELLED_QUEUE_URL" --argjson p "$POLICY_CANCELLED" '{QueueUrl:$url,Attributes:{Policy:$p}}' > /app/build/set-attr-cancelled.json && \
aws sqs set-queue-attributes --cli-input-json file:///app/build/set-attr-created.json && \
aws sqs set-queue-attributes --cli-input-json file:///app/build/set-attr-cancelled.json && \
echo "queue policies set"
output
queue policies set

[stdout]
queue policies set

Attach EventBridge rule targets

shell
source /app/.env && \
CREATED_QUEUE_ARN=$(cat /app/build/orders-created-arn.txt) && \
CANCELLED_QUEUE_ARN=$(cat /app/build/orders-cancelled-arn.txt) && \
aws events put-targets --rule on-order-created --event-bus-name order-events \
  --targets "Id=1,Arn=$CREATED_QUEUE_ARN" --output json && \
aws events put-targets --rule on-order-cancelled --event-bus-name order-events \
  --targets "Id=1,Arn=$CANCELLED_QUEUE_ARN" --output json && \
echo "targets attached"
output
{
    "FailedEntryCount": 0,
    "FailedEntries": []
}
{
    "FailedEntryCount": 0,
    "FailedEntries": []
}
targets attached

[stdout]
{
    "FailedEntryCount": 0,
    "FailedEntries": []
}
{
    "FailedEntryCount": 0,
    "FailedEntries": []
}
targets attached
  • Create KMS CMK with alias outbox-cmk
  • Create DynamoDB tables (orders, outbox with stream, processed_events with TTL)
  • Create SQS DLQs and main queues
  • Create EventBridge bus and rules
  • Create IAM roles with least privilege
  • Write and deploy order-api lambda
  • Write and deploy outbox-relay lambda with stream ESM
  • End-to-end smoke test

Create IAM roles with lambda trust

shell
cat > /app/policies/lambda-trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF
source /app/.env && \
aws iam create-role --role-name order-api-role \
  --assume-role-policy-document file:///app/policies/lambda-trust.json --output json > /app/build/role-api.json && \
aws iam create-role --role-name outbox-relay-role \
  --assume-role-policy-document file:///app/policies/lambda-trust.json --output json > /app/build/role-relay.json && \
echo "roles created"
output
roles created

[stdout]
roles created

Attach least-privilege inline policies to lambda roles

shell
source /app/.env && \
KEY_ARN=$(cat /app/build/kms-key-arn.txt) && \
ORDERS_ARN=$(cat /app/build/orders-arn.txt) && \
OUTBOX_ARN=$(cat /app/build/outbox-arn.txt) && \
OUTBOX_STREAM_ARN=$(cat /app/build/outbox-stream-arn.txt) && \
PROCESSED_ARN=$(cat /app/build/processed-arn.txt) && \
BUS_ARN=$(cat /app/build/bus-arn.txt) && \
# order-api-role inline policy: Transact write to orders+outbox, KMS encrypt/decrypt on CMK, logs
cat > /app/policies/order-api-policy.json <<EOF
{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"WriteOrdersAndOutboxAtomically",
      "Effect":"Allow",
      "Action":["dynamodb:PutItem","dynamodb:UpdateItem","dynamodb:TransactWriteItems"],
      "Resource":["$ORDERS_ARN","$OUTBOX_ARN"]
    },
    {
      "Sid":"UseCMKForDDB",
      "Effect":"Allow",
      "Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],
      "Resource":"$KEY_ARN"
    },
    {
      "Sid":"Logs",
      "Effect":"Allow",
      "Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],
      "Resource":"arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/order-api*"
    }
  ]
}
EOF
# outbox-relay-role inline policy: read outbox stream only, write processed_events, putevents to bus, kms, logs
cat > /app/policies/outbox-relay-policy.json <<EOF
{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"ReadOutboxStreamOnly",
      "Effect":"Allow",
      "Action":["dynamodb:DescribeStream","dynamodb:GetRecords","dynamodb:GetShardIterator","dynamodb:ListStreams"],
      "Resource":"$OUTBOX_STREAM_ARN"
    },
    {
      "Sid":"DedupTableWrites",
      "Effect":"Allow",
      "Action":["dynamodb:PutItem"],
      "Resource":"$PROCESSED_ARN"
    },
    {
      "Sid":"PublishToOrderEventsBus",
      "Effect":"Allow",
      "Action":["events:PutEvents"],
      "Resource":"$BUS_ARN"
    },
    {
      "Sid":"UseCMKForStreamAndBus",
      "Effect":"Allow",
      "Action":["kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],
      "Resource":"$KEY_ARN"
    },
    {
      "Sid":"Logs",
      "Effect":"Allow",
      "Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],
      "Resource":"arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/outbox-relay*"
    }
  ]
}
EOF
aws iam put-role-policy --role-name order-api-role --policy-name order-api-inline \
  --policy-document file:///app/policies/order-api-policy.json && \
aws iam put-role-policy --role-name outbox-relay-role --policy-name outbox-relay-inline \
  --policy-document file:///app/policies/outbox-relay-policy.json && \
echo "policies attached"
output
policies attached

[stdout]
policies attached
  • Create KMS CMK with alias outbox-cmk
  • Create DynamoDB tables (orders, outbox with stream, processed_events with TTL)
  • Create SQS DLQs and main queues
  • Create EventBridge bus and rules
  • Create IAM roles with least privilege
  • Write and deploy order-api lambda
  • Write and deploy outbox-relay lambda with stream ESM
  • End-to-end smoke test
write · /app/src/order_api.py
import json
import os
import time
import uuid

import boto3

ENDPOINT = os.environ["AWS_ENDPOINT_URL"]
ORDERS_TABLE = os.environ["ORDERS_TABLE"]
OUTBOX_TABLE = os.environ["OUTBOX_TABLE"]

ddb = boto3.client("dynamodb", endpoint_url=ENDPOINT)

VALID_KINDS = {"OrderCreated", "OrderCancelled"}


def _to_dynamo_str(v):
    if isinstance(v, bool):
        return {"BOOL": v}
    if isinstance(v, (int, float)):
        return {"N": str(v)}
    if v is None:
        return {"NULL": True}
    return {"S": str(v)}


def handler(event, _context):
    order_id = event["order_id"]
    kind = event["kind"]
    if kind not in VALID_KINDS:
        raise ValueError(f"unsupported kind: {kind}")

    event_id = str(uuid.uuid4())
    now = int(time.time())

    payload = {k: v for k, v in event.items() if k not in ("order_id", "kind")}

    order_item = {
        "order_id": {"S": order_id},
        "kind": {"S": kind},
        "updated_at": {"N": str(now)},
    }
    for k, v in payload.items():
        order_item[k] = _to_dynamo_str(v)

    outbox_item = {
        "event_id": {"S": event_id},
        "order_id": {"S": order_id},
        "kind": {"S": kind},
        "created_at": {"N": str(now)},
        "payload": {"S": json.dumps(event)},
    }

    ddb.transact_write_items(
        TransactItems=[
            {
                "Put": {
                    "TableName": ORDERS_TABLE,
                    "Item": order_item,
                }
            },
            {
                "Put": {
                    "TableName": OUTBOX_TABLE,
                    "Item": outbox_item,
                    "ConditionExpression": "attribute_not_exists(event_id)",
                }
            },
        ]
    )

    return {"order_id": order_id, "event_id": event_id, "kind": kind}
write · /app/src/outbox_relay.py
import json
import logging
import os
import time

import boto3
from botocore.exceptions import ClientError

ENDPOINT = os.environ["AWS_ENDPOINT_URL"]
EVENT_BUS = os.environ["EVENT_BUS"]
DEDUP_TABLE = os.environ["DEDUP_TABLE"]
DEDUP_TTL_SECONDS = int(os.environ.get("DEDUP_TTL_SECONDS", "604800"))

ddb = boto3.client("dynamodb", endpoint_url=ENDPOINT)
events = boto3.client("events", endpoint_url=ENDPOINT)

log = logging.getLogger()
log.setLevel(logging.INFO)


def _unwrap(image):
    out = {}
    for k, v in (image or {}).items():
        if "S" in v:
            out[k] = v["S"]
        elif "N" in v:
            n = v["N"]
            out[k] = int(n) if n.isdigit() else float(n)
        elif "BOOL" in v:
            out[k] = v["BOOL"]
        elif "NULL" in v:
            out[k] = None
        else:
            out[k] = v
    return out


def _claim(event_id):
    try:
        ddb.put_item(
            TableName=DEDUP_TABLE,
            Item={
                "event_id": {"S": event_id},
                "ttl": {"N": str(int(time.time()) + DEDUP_TTL_SECONDS)},
                "processed_at": {"N": str(int(time.time()))},
            },
            ConditionExpression="attribute_not_exists(event_id)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise


def handler(event, _context):
    failures = []
    for record in event.get("Records", []):
        seq = record.get("dynamodb", {}).get("SequenceNumber")
        try:
            if record.get("eventName") != "INSERT":
                continue
            new_image = record["dynamodb"].get("NewImage") or {}
            old_image = record["dynamodb"].get("OldImage")
            row = _unwrap(new_image)
            event_id = row.get("event_id")
            kind = row.get("kind")
            if not event_id or kind not in ("OrderCreated", "OrderCancelled"):
                log.warning("skipping record with bad shape seq=%s row=%s", seq, row)
                continue

            if not _claim(event_id):
                log.info("duplicate event_id=%s seq=%s , skipping", event_id, seq)
                continue

            detail = {
                "event_id": event_id,
                "order_id": row.get("order_id"),
                "kind": kind,
                "new_image": row,
                "old_image": _unwrap(old_image) if old_image else None,
            }
            resp = events.put_events(
                Entries=[
                    {
                        "EventBusName": EVENT_BUS,
                        "Source": "outbox.relay",
                        "DetailType": kind,
                        "Detail": json.dumps(detail),
                    }
                ]
            )
            if resp.get("FailedEntryCount", 0) > 0:
                raise RuntimeError(f"PutEvents failed: {resp}")
        except Exception:
            log.exception("failed to relay seq=%s", seq)
            if seq:
                failures.append({"itemIdentifier": seq})

    return {"batchItemFailures": failures}

Zip lambda code

shell
cd /app/src && zip -j /app/build/order-api.zip order_api.py && zip -j /app/build/outbox-relay.zip outbox_relay.py && ls -la /app/build/*.zip
output
adding: order_api.py (deflated 63%)
  adding: outbox_relay.py (deflated 63%)
-rw-r--r-- 1 root root  842 Apr 28 14:50 /app/build/order-api.zip
-rw-r--r-- 1 root root 1350 Apr 28 14:50 /app/build/outbox-relay.zip

[stdout]
  adding: order_api.py (deflated 63%)
  adding: outbox_relay.py (deflated 63%)
-rw-r--r-- 1 root root  842 Apr 28 14:50 /app/build/order-api.zip
-rw-r--r-- 1 root root 1350 Apr 28 14:50 /app/build/outbox-relay.zip

Create order-api lambda

shell
source /app/.env && \
API_ROLE_ARN=$(jq -r '.Role.Arn' /app/build/role-api.json) && \
RELAY_ROLE_ARN=$(jq -r '.Role.Arn' /app/build/role-relay.json) && \
KEY_ARN=$(cat /app/build/kms-key-arn.txt) && \
# Use localstack-internal endpoint inside lambda containers
aws lambda create-function --function-name order-api \
  --runtime python3.11 --handler order_api.handler \
  --role $API_ROLE_ARN \
  --zip-file fileb:///app/build/order-api.zip \
  --timeout 15 \
  --kms-key-arn $KEY_ARN \
  --environment "Variables={AWS_ENDPOINT_URL=http://localstack:4566,ORDERS_TABLE=orders,OUTBOX_TABLE=outbox}" \
  --output json > /app/build/lambda-api.json && \
echo "order-api created"
output
order-api created

[stdout]
order-api created

Create outbox-relay lambda

shell
source /app/.env && \
RELAY_ROLE_ARN=$(jq -r '.Role.Arn' /app/build/role-relay.json) && \
KEY_ARN=$(cat /app/build/kms-key-arn.txt) && \
aws lambda create-function --function-name outbox-relay \
  --runtime python3.11 --handler outbox_relay.handler \
  --role $RELAY_ROLE_ARN \
  --zip-file fileb:///app/build/outbox-relay.zip \
  --timeout 30 \
  --kms-key-arn $KEY_ARN \
  --environment "Variables={AWS_ENDPOINT_URL=http://localstack:4566,EVENT_BUS=order-events,DEDUP_TABLE=processed_events,DEDUP_TTL_SECONDS=604800}" \
  --output json > /app/build/lambda-relay.json && \
echo "outbox-relay created"
output
outbox-relay created

[stdout]
outbox-relay created

Wait for lambdas to become active

shell
source /app/.env && \
# Wait for lambdas to be ready
for fn in order-api outbox-relay; do
  for i in 1 2 3 4 5 6 7 8 9 10; do
    state=$(aws lambda get-function --function-name $fn --query 'Configuration.State' --output text 2>/dev/null || echo "Pending")
    if [ "$state" = "Active" ]; then echo "$fn Active"; break; fi
    sleep 1
  done
done
output
order-api Active
outbox-relay Active

[stdout]
order-api Active
outbox-relay Active

Create stream event source mapping with ReportBatchItemFailures

shell
source /app/.env && \
OUTBOX_STREAM_ARN=$(cat /app/build/outbox-stream-arn.txt) && \
aws lambda create-event-source-mapping \
  --function-name outbox-relay \
  --event-source-arn $OUTBOX_STREAM_ARN \
  --starting-position TRIM_HORIZON \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 1 \
  --function-response-types ReportBatchItemFailures \
  --output json > /app/build/esm.json && \
cat /app/build/esm.json | jq '{UUID, State, FunctionResponseTypes}'
output
{
  "UUID": "8cc67d68-4784-4dc7-b1bd-24dd63d7adb1",
  "State": "Creating",
  "FunctionResponseTypes": [
    "ReportBatchItemFailures"
  ]
}

[stdout]
{
  "UUID": "8cc67d68-4784-4dc7-b1bd-24dd63d7adb1",
  "State": "Creating",
  "FunctionResponseTypes": [
    "ReportBatchItemFailures"
  ]
}
  • Create KMS CMK with alias outbox-cmk
  • Create DynamoDB tables (orders, outbox with stream, processed_events with TTL)
  • Create SQS DLQs and main queues
  • Create EventBridge bus and rules
  • Create IAM roles with least privilege
  • Write and deploy order-api lambda
  • Write and deploy outbox-relay lambda with stream ESM
  • End-to-end smoke test

Wait for ESM enabled

shell
source /app/.env && ESM_UUID=$(jq -r '.UUID' /app/build/esm.json) && \
for i in $(seq 1 20); do
  state=$(aws lambda get-event-source-mapping --uuid $ESM_UUID --query 'State' --output text 2>/dev/null)
  echo "ESM state=$state"
  if [ "$state" = "Enabled" ]; then break; fi
  sleep 1
done
output
ESM state=Enabled

[stdout]
ESM state=Enabled

Invoke order-api with OrderCreated

shell
source /app/.env && \
aws lambda invoke --function-name order-api \
  --payload "$(echo -n '{"order_id":"o-1","kind":"OrderCreated","sku":"abc","qty":2}' | base64)" \
  /app/build/invoke-1.json --output json && \
echo "---" && cat /app/build/invoke-1.json && echo
output
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
---
{"order_id": "o-1", "event_id": "34b52730-3277-46bf-8437-dd1dd3cf3ee1", "kind": "OrderCreated"}

[stdout]
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
---
{"order_id": "o-1", "event_id": "34b52730-3277-46bf-8437-dd1dd3cf3ee1", "kind": "OrderCreated"}

Check first event landed correctly

shell
source /app/.env && \
sleep 8 && \
CREATED_URL=$(cat /app/build/orders-created-url.txt) && \
CANCELLED_URL=$(cat /app/build/orders-cancelled-url.txt) && \
echo "== orders-created ==" && aws sqs receive-message --queue-url $CREATED_URL --max-number-of-messages 10 --wait-time-seconds 2 --visibility-timeout 0 --output json | jq '.Messages | length' && \
echo "== orders-cancelled ==" && aws sqs receive-message --queue-url $CANCELLED_URL --max-number-of-messages 10 --wait-time-seconds 2 --visibility-timeout 0 --output json | jq '.Messages | length' && \
echo "== processed_events ==" && aws dynamodb scan --table-name processed_events --output json | jq '.Items | length'
output
== orders-created ==
1
== orders-cancelled ==
== processed_events ==
1

[stdout]
== orders-created ==
1
== orders-cancelled ==
== processed_events ==
1

Invoke OrderCancelled and verify

shell
source /app/.env && \
aws lambda invoke --function-name order-api \
  --payload "$(echo -n '{"order_id":"o-2","kind":"OrderCancelled","reason":"customer requested"}' | base64)" \
  /app/build/invoke-2.json --output json && cat /app/build/invoke-2.json && echo && \
sleep 8 && \
CREATED_URL=$(cat /app/build/orders-created-url.txt) && \
CANCELLED_URL=$(cat /app/build/orders-cancelled-url.txt) && \
echo "== orders-created (should still be 0 new since first msg pending) ==" && \
aws sqs get-queue-attributes --queue-url $CREATED_URL --attribute-names ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible --output json | jq && \
echo "== orders-cancelled ==" && \
aws sqs get-queue-attributes --queue-url $CANCELLED_URL --attribute-names ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible --output json | jq && \
echo "== processed_events ==" && aws dynamodb scan --table-name processed_events --output json | jq '.Items | length'
output
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
{"order_id": "o-2", "event_id": "f0fd4478-9ac2-45b9-b3c7-521148f288eb", "kind": "OrderCancelled"}
== orders-created (should still be 0 new since first msg pending) ==

aws: [ERROR]: An error occurred (InvalidAttributeName) when calling the GetQueueAttributes operation: Unknown Attribute ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible.
== orders-cancelled ==

aws: [ERROR]: An error occurred (InvalidAttributeName) when calling the GetQueueAttributes operation: Unknown Attribute ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible.
== processed_events ==
2

[stdout]
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
{"order_id": "o-2", "event_id": "f0fd4478-9ac2-45b9-b3c7-521148f288eb", "kind": "OrderCancelled"}
== orders-created (should still be 0 new since first msg pending) ==

aws: [ERROR]: An error occurred (InvalidAttributeName) when calling the GetQueueAttributes operation: Unknown Attribute ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible.
== orders-cancelled ==

aws: [ERROR]: An error occurred (InvalidAttributeName) when calling the GetQueueAttributes operation: Unknown Attribute ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible.
== processed_events ==
2

Get queue attribute counts (separate args)

shell
source /app/.env && \
CREATED_URL=$(cat /app/build/orders-created-url.txt) && \
CANCELLED_URL=$(cat /app/build/orders-cancelled-url.txt) && \
echo "== orders-created ==" && aws sqs get-queue-attributes --queue-url $CREATED_URL --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible --output json && \
echo "== orders-cancelled ==" && aws sqs get-queue-attributes --queue-url $CANCELLED_URL --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible --output json
output
== orders-created ==
{
    "Attributes": {
        "ApproximateNumberOfMessages": "1",
        "ApproximateNumberOfMessagesNotVisible": "0"
    }
}
== orders-cancelled ==
{
    "Attributes": {
        "ApproximateNumberOfMessages": "1",
        "ApproximateNumberOfMessagesNotVisible": "0"
    }
}

[stdout]
== orders-created ==
{
    "Attributes": {
        "ApproximateNumberOfMessages": "1",
        "ApproximateNumberOfMessagesNotVisible": "0"
    }
}
== orders-cancelled ==
{
    "Attributes": {
        "ApproximateNumberOfMessages": "1",
        "ApproximateNumberOfMessagesNotVisible": "0"
    }
}

Inspect queue message shape

shell
source /app/.env && \
CREATED_URL=$(cat /app/build/orders-created-url.txt) && \
CANCELLED_URL=$(cat /app/build/orders-cancelled-url.txt) && \
echo "== sample orders-created ==" && \
aws sqs receive-message --queue-url $CREATED_URL --max-number-of-messages 1 --visibility-timeout 0 --wait-time-seconds 1 --output json | jq -r '.Messages[0].Body' | jq '{source,"detail-type":."detail-type",detail:.detail}' && \
echo "== sample orders-cancelled ==" && \
aws sqs receive-message --queue-url $CANCELLED_URL --max-number-of-messages 1 --visibility-timeout 0 --wait-time-seconds 1 --output json | jq -r '.Messages[0].Body' | jq '{source,"detail-type":."detail-type",detail:.detail}'
output
== sample orders-created ==
{
  "source": "outbox.relay",
  "detail-type": "OrderCreated",
  "detail": {
    "event_id": "34b52730-3277-46bf-8437-dd1dd3cf3ee1",
    "order_id": "o-1",
    "kind": "OrderCreated",
    "new_image": {
      "event_id": "34b52730-3277-46bf-8437-dd1dd3cf3ee1",
      "order_id": "o-1",
      "kind": "OrderCreated",
      "created_at": 1777387870,
      "payload": "{\"order_id\": \"o-1\", \"kind\": \"OrderCreated\", \"sku\": \"abc\", \"qty\": 2}"
    },
    "old_image": null
  }
}
== sample orders-cancelled ==
{
  "source": "outbox.relay",
  "detail-type": "OrderCancelled",
  "detail": {
    "event_id": "f0fd4478-9ac2-45b9-b3c7-521148f288eb",
    "order_id": "o-2",
    "kind": "OrderCancelled",
    "new_image": {
      "event_id": "f0fd4478-9ac2-45b9-b3c7-521148f288eb",
      "order_id": "o-2",
      "kind": "OrderCancelled",
      "created_at": 1777387899,
      "payload": "{\"order_id\": \"o-2\", \"kind\": \"OrderCancelled\", \"reason\": \"customer requested\"}"
    },
    "old_image": null
  }
}

[stdout]
== sample orders-created ==
{
  "source": "outbox.relay",
  "detail-type": "OrderCreated",
  "detail": {
    "event_id": "34b52730-3277-46bf-8437-dd1dd3cf3ee1",
    "order_id": "o-1",
    "kind": "OrderCreated",
    "new_image": {
      "event_id": "34b52730-3277-46bf-8437-dd1dd3cf3ee1",
      "order_id": "o-1",
      "kind": "OrderCreated",
      "created_at": 1777387870,
      "payload": "{\"order_id\": \"o-1\", \"kind\": \"OrderCreated\", \"sku\": \"abc\", \"qty\": 2}"
    },
    "old_image": null
  }
}
== sample orders-cancelled ==
{
  "source": "outbox.relay",
  "detail-type": "OrderCancelled",
  "detail": {
    "event_id": "f0fd4478-9ac2-45b9-b3c7-521148f288eb",
    "order_id": "o-2",
    "kind": "OrderCancelled",
    "new_image": {
      "event_id": "f0fd4478-9ac2-45b9-b3c7-521148f288eb",
      "order_id": "o-2",
      "kind": "OrderCancelled",
      "created_at": 1777387899,
      "payload": "{\"order_id\": \"o-2\", \"kind\": \"OrderCancelled\", \"reason\": \"customer requested\"}"
    },
    "old_image": null
  }
}

Verify spec compliance points

shell
source /app/.env && \
echo "== outbox stream view type ==" && \
aws dynamodb describe-table --table-name outbox --query 'Table.StreamSpecification' --output json && \
echo "== orders has no stream (least privilege check) ==" && \
aws dynamodb describe-table --table-name orders --query 'Table.{StreamArn:LatestStreamArn,Spec:StreamSpecification}' --output json && \
echo "== processed_events TTL ==" && \
aws dynamodb describe-time-to-live --table-name processed_events --output json && \
echo "== KMS alias resolves to CMK ==" && \
aws kms describe-key --key-id alias/outbox-cmk --query 'KeyMetadata.{KeyManager:KeyManager,Arn:Arn}' --output json && \
echo "== ESM has ReportBatchItemFailures ==" && \
aws lambda list-event-source-mappings --function-name outbox-relay --query 'EventSourceMappings[].{State:State,FRT:FunctionResponseTypes,EventSrc:EventSourceArn}' --output json
output
== outbox stream view type ==
{
    "StreamEnabled": true,
    "StreamViewType": "NEW_AND_OLD_IMAGES"
}
== orders has no stream (least privilege check) ==
{
    "StreamArn": null,
    "Spec": null
}
== processed_events TTL ==
{
    "TimeToLiveDescription": {
        "TimeToLiveStatus": "ENABLED",
        "AttributeName": "ttl"
    }
}
== KMS alias resolves to CMK ==
{
    "KeyManager": "CUSTOMER",
    "Arn": "arn:aws:kms:us-east-1:000000000000:key/0bbb32cd-65fd-40dc-af71-e284aeee1f8c"
}
== ESM has ReportBatchItemFailures ==
[
    {
        "State": "Enabled",
        "FRT": [
            "ReportBatchItemFailures"
        ],
        "EventSrc": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:48:06.432"
    }
]

[stdout]
== outbox stream view type ==
{
    "StreamEnabled": true,
    "StreamViewType": "NEW_AND_OLD_IMAGES"
}
== orders has no stream (least privilege check) ==
{
    "StreamArn": null,
    "Spec": null
}
== processed_events TTL ==
{
    "TimeToLiveDescription": {
        "TimeToLiveStatus": "ENABLED",
        "AttributeName": "ttl"
    }
}
== KMS alias resolves to CMK ==
{
    "KeyManager": "CUSTOMER",
    "Arn": "arn:aws:kms:us-east-1:000000000000:key/0bbb32cd-65fd-40dc-af71-e284aeee1f8c"
}
== ESM has ReportBatchItemFailures ==
[
    {
        "State": "Enabled",
        "FRT": [
            "ReportBatchItemFailures"
        ],
        "EventSrc": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:48:06.432"
    }
]

Verify role policy docs are tight

shell
source /app/.env && \
echo "== order-api inline policies ==" && \
aws iam get-role-policy --role-name order-api-role --policy-name order-api-inline --output json | jq '.PolicyDocument' && \
echo && \
echo "== outbox-relay inline policies ==" && \
aws iam get-role-policy --role-name outbox-relay-role --policy-name outbox-relay-inline --output json | jq '.PolicyDocument'
output
== order-api inline policies ==
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteOrdersAndOutboxAtomically",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:TransactWriteItems"
      ],
      "Resource": [
        "arn:aws:dynamodb:us-east-1:000000000000:table/orders",
        "arn:aws:dynamodb:us-east-1:000000000000:table/outbox"
      ]
    },
    {
      "Sid": "UseCMKForDDB",
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/0bbb32cd-65fd-40dc-af71-e284aeee1f8c"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/order-api*"
    }
  ]
}

== outbox-relay inline policies ==
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOutboxStreamOnly",
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:48:06.432"
    },
    {
      "Sid": "DedupTableWrites",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/processed_events"
    },
    {
      "Sid": "PublishToOrderEventsBus",
      "Effect": "Allow",
      "Action": [
        "events:PutEvents"
      ],
      "Resource": "arn:aws:events:us-east-1:000000000000:event-bus/order-events"
    },
    {
      "Sid": "UseCMKForStreamAndBus",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/0bbb32cd-65fd-40dc-af71-e284aeee1f8c"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/outbox-relay*"
    }
  ]
}

[stdout]
== order-api inline policies ==
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteOrdersAndOutboxAtomically",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:TransactWriteItems"
      ],
      "Resource": [
        "arn:aws:dynamodb:us-east-1:000000000000:table/orders",
        "arn:aws:dynamodb:us-east-1:000000000000:table/outbox"
      ]
    },
    {
      "Sid": "UseCMKForDDB",
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/0bbb32cd-65fd-40dc-af71-e284aeee1f8c"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/order-api*"
    }
  ]
}

== outbox-relay inline policies ==
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOutboxStreamOnly",
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:48:06.432"
    },
    {
      "Sid": "DedupTableWrites",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/processed_events"
    },
    {
      "Sid": "PublishToOrderEventsBus",
      "Effect": "Allow",
      "Action": [
        "events:PutEvents"
      ],
      "Resource": "arn:aws:events:us-east-1:000000000000:event-bus/order-events"
    },
    {
      "Sid": "UseCMKForStreamAndBus",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/0bbb32cd-65fd-40dc-af71-e284aeee1f8c"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/outbox-relay*"
    }
  ]
}
  • Create KMS CMK with alias outbox-cmk
  • Create DynamoDB tables (orders, outbox with stream, processed_events with TTL)
  • Create SQS DLQs and main queues
  • Create EventBridge bus and rules
  • Create IAM roles with least privilege
  • Write and deploy order-api lambda
  • Write and deploy outbox-relay lambda with stream ESM
  • End-to-end smoke test
End-to-end working on localstack 4.13.1. Summary of what got stood up: **Resources** - `alias/outbox-cmk` , customer-managed KMS key; encrypts both DDB tables, both SQS queues + DLQs, the event bus, and the lambdas. - `orders` (pk `order_id`/S), `outbox` (pk `event_id`/S, stream `NEW_AND_OLD_IMAGES`), `processed_events` (pk `event_id`/S, TTL on `ttl` attribute). - `orders-created`/`orders-cancelled` SQS queues, each with its own DLQ (`*-dlq`) via redrive policy, max receive count 5. - `order-events` event bus + `on-order-created` / `on-order-cancelled` rules pattern-matched on `source=outbox.relay` and `detail-type` of `OrderCreated` / `OrderCancelled`. **Lambdas (python3.11)** - `order-api` → `transact_write_items` writes orders row + outbox row (with generated `event_id`) atomically. - `outbox-relay` → consumes the outbox stream (ESM with `ReportBatchItemFailures`), conditionally claims `event_id` in `processed_events` (idempotency, with TTL), publishes to `order-events` with `source=outbox.relay` and `detail-type=kind`. Returns `{batchItemFailures: [{itemIdentifier: <seq#>}, ...]}` on per-record failures. **Smoke test** - Invoke `{order_id:"o-1", kind:"OrderCreated", ...}` → `orders-created` got 1 message, `orders-cancelled` got 0, `processed_events` got the row. - Invoke `{order_id:"o-2", kind:"OrderCancelled", ...}` → `orders-cancelled` got 1, `orders-created` unchanged. **Least privilege (verified by reading the inline docs)** - `order-api-role`: TransactWriteItems/PutItem/UpdateItem on orders+outbox table ARNs only; KMS Encrypt/Decrypt/GenerateDataKey/DescribeKey on the CMK ARN only; logs scoped to `/aws/lambda/order-api*`. No SQS, no events. - `outbox-relay-role`: stream actions scoped to the outbox stream ARN (not orders); PutItem only on `processed_events`; PutEvents only on the `order-events` bus ARN; KMS Decrypt/GenerateDataKey/DescribeKey on CMK only; logs scoped to `/aws/lambda/outbox-relay*`. No write access to outbox or orders tables. Source at `/app/src/`, build artifacts and policy JSON at `/app/build/` and `/app/policies/`.
Final diff , what the agent changed

No file changes, the agent left the workspace untouched.

Verifier output , the proof behind the reward
tests/test.sh
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/WEt2Hy4zwGxARMHQ875Yd/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 20 items

../tests/test_outputs.py::test_01_cmk_alias_exists_and_enabled PASSED    [  5%]
../tests/test_outputs.py::test_02_orders_table_exists PASSED             [ 10%]
../tests/test_outputs.py::test_03_outbox_table_exists PASSED             [ 15%]
../tests/test_outputs.py::test_04_processed_events_table_exists PASSED   [ 20%]
../tests/test_outputs.py::test_05_custom_event_bus_exists PASSED         [ 25%]
../tests/test_outputs.py::test_06_orders_table_encrypted_with_cmk PASSED [ 30%]
../tests/test_outputs.py::test_07_outbox_table_encrypted_with_cmk PASSED [ 35%]
../tests/test_outputs.py::test_08_outbox_stream_is_new_and_old_images PASSED [ 40%]
../tests/test_outputs.py::test_09_processed_events_ttl_enabled PASSED    [ 45%]
../tests/test_outputs.py::test_10_both_rule_patterns_use_array_wrap_and_disjoint_types PASSED [ 50%]
../tests/test_outputs.py::test_11_each_rule_targets_only_its_intended_queue PASSED [ 55%]
../tests/test_outputs.py::test_12_both_main_queues_encrypted_with_cmk PASSED [ 60%]
../tests/test_outputs.py::test_13_main_queues_have_redrive_to_distinct_dlqs PASSED [ 65%]
../tests/test_outputs.py::test_14_queue_policies_scope_events_with_source_arn PASSED [ 70%]
../tests/test_outputs.py::test_15_relay_esm_has_batch_item_failures_on_outbox_stream PASSED [ 75%]
../tests/test_outputs.py::test_16_relay_role_is_scoped_not_wildcard PASSED [ 80%]
../tests/test_outputs.py::test_17_order_api_role_scoped_to_orders_and_outbox_only PASSED [ 85%]
../tests/test_outputs.py::test_18_e2e_order_created_routes_to_created_queue_only PASSED [ 90%]
../tests/test_outputs.py::test_19_e2e_order_cancelled_routes_to_cancelled_queue_only PASSED [ 95%]
../tests/test_outputs.py::test_20_e2e_idempotent_duplicate_does_not_double_fanout PASSED [100%]

=============================== warnings summary ===============================
test_outputs.py: 89 warnings
  /root/.cache/uv/archive-v0/WEt2Hy4zwGxARMHQ875Yd/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_outputs.py::test_01_cmk_alias_exists_and_enabled
PASSED ../tests/test_outputs.py::test_02_orders_table_exists
PASSED ../tests/test_outputs.py::test_03_outbox_table_exists
PASSED ../tests/test_outputs.py::test_04_processed_events_table_exists
PASSED ../tests/test_outputs.py::test_05_custom_event_bus_exists
PASSED ../tests/test_outputs.py::test_06_orders_table_encrypted_with_cmk
PASSED ../tests/test_outputs.py::test_07_outbox_table_encrypted_with_cmk
PASSED ../tests/test_outputs.py::test_08_outbox_stream_is_new_and_old_images
PASSED ../tests/test_outputs.py::test_09_processed_events_ttl_enabled
PASSED ../tests/test_outputs.py::test_10_both_rule_patterns_use_array_wrap_and_disjoint_types
PASSED ../tests/test_outputs.py::test_11_each_rule_targets_only_its_intended_queue
PASSED ../tests/test_outputs.py::test_12_both_main_queues_encrypted_with_cmk
PASSED ../tests/test_outputs.py::test_13_main_queues_have_redrive_to_distinct_dlqs
PASSED ../tests/test_outputs.py::test_14_queue_policies_scope_events_with_source_arn
PASSED ../tests/test_outputs.py::test_15_relay_esm_has_batch_item_failures_on_outbox_stream
PASSED ../tests/test_outputs.py::test_16_relay_role_is_scoped_not_wildcard
PASSED ../tests/test_outputs.py::test_17_order_api_role_scoped_to_orders_and_outbox_only
PASSED ../tests/test_outputs.py::test_18_e2e_order_created_routes_to_created_queue_only
PASSED ../tests/test_outputs.py::test_19_e2e_order_cancelled_routes_to_cancelled_queue_only
PASSED ../tests/test_outputs.py::test_20_e2e_idempotent_duplicate_does_not_double_fanout
======================= 20 passed, 89 warnings in 22.50s =======================

Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_d00fd2066aa0460f. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.

Trial trial_d00fd2066aa0460f · verifier authoritative; classifier explanatory.