SyncValsverifier → artifact → classifier → verdict
SyncVals · Trajectory

ddb-outbox-eventbridge-fanout

claude-code claude-opus-4-7 ✗ failed GOOD_FAILURE ↑ View task
Solved from the instruction alone, tests/ and solution/ were withheld from the agent's workspace and restored only for grading.
Reward = tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.
Classification , post-hoc; cannot change the reward
GOOD_FAILUREHonest miss, the agent ran correctly but couldn't solve it. Expected for a hard task; the task is sound.
SubtypeIncomplete Understanding
EvidenceTest `test_14_queue_policies_scope_events_with_source_arn` fails with: "orders-created resource policy must include an Allow for events.amazonaws.com scoped by aws:SourceArn containing rule 'on-order-created' - confused-deputy protection". Agent passed 19/20 tests including all e2e functional tests. The e2e tests in steps 46-48 of trajectory show the implementation works end-to-end (order routing to correct queues, idempotency working, least-privilege roles correctly scoped). Missing only the aws:SourceArn Condition clause in SQS queue resource policies to prevent confused-deputy attacks.
Root causeThe agent implemented a largely correct DynamoDB outbox + EventBridge fanout pattern with proper tables, streams, rules, lambda functions, and IAM roles. However, it missed adding the `aws:SourceArn` Condition to SQS queue resource policies when allowing EventBridge's `events.amazonaws.com` principal to send messages. This AWS security pattern (confused-deputy protection) is not explicitly stated in the instruction, only hinted at through phrases like "least privilege" and "the verifier reads the policy docs directly."
RecommendationN/A - task is fine
Trajectory
Tool-by-tool agent trajectory
32 tool calls · 4 tool types · 44 steps
ok the checkout service has been dropping events again. every time the orders table gets a new row, *something* downstream needs to know , pricing, fulfillment, the analytics folks, all of them. right now we're just hoping two sequential putitems stick. they don't. last week we had a row written and no event fired because the lambda crashed between the two writes. nobody noticed for 6 hours. so: outbox pattern. one transaction, two rows, and let the stream do the fanout. build it on localstack , `http://localstack:4566`, creds already exported (`AWS_ACCESS_KEY_ID=test`, same for secret, region `us-east-1`). you've got `aws`, `python3`, `boto3`, `jq`, `zip`. build the whole thing from zero. shape of it: - an "order api" lambda is the only thing that writes orders. it takes `{order_id, kind, ...}` where `kind` is either `OrderCreated` or `OrderCancelled`. it writes the business row **and** the outbox row in one atomic step. no half-states allowed. - the outbox table has a stream , view type `NEW_AND_OLD_IMAGES` (the relay needs to see both the new and old image of the row, not just keys, so downstream subscribers can react on diffs and not just inserts). a relay lambda reads the stream via an event source mapping and republishes onto a custom eventbridge bus. - the relay's event source mapping must use `ReportBatchItemFailures` so a single bad record can fail without retrying the whole batch , the relay returns `{"batchItemFailures": [{"itemIdentifier": "<seq#>"}, ...]}` for the records it couldn't republish. - two rules on that bus route by kind: `OrderCreated` goes to one sqs queue, `OrderCancelled` goes to another. each queue has its own dlq. no shared dlq. - the relay has to be idempotent , if the stream redelivers the same record, we don't want the downstream queue to see the message twice. keep a small ddb table for dedup, ttl it so it doesn't grow forever. - everything is encrypted with a customer-managed kms key. not `aws/dynamodb`, not `aws/sqs`. ours. done looks like this: invoke `order-api` with `{order_id: "o-1", kind: "OrderCreated", ...}`, and within 30s: - `orders-created` sqs queue has exactly 1 message - `orders-cancelled` sqs queue has 0 messages - `processed_events` table has a row for the corresponding outbox event_id then invoke again with `kind: "OrderCancelled"` and a different `order_id`: - `orders-cancelled` picks it up, `orders-created` stays quiet least privilege: every role scoped to its own resources. the outbox-relay role should not be able to touch the orders table's stream; the order-api role should not be able to touch sqs; nobody gets `*` on kms. the verifier reads the policy docs directly, not just "does it work end-to-end". names the verifier grabs by hand, use them exactly: | thing | name | |-|-| | kms alias | `alias/outbox-cmk` | | orders table | `orders` (pk `order_id` / String) | | outbox table | `outbox` (pk `event_id` / String) | | dedup table | `processed_events` (pk `event_id` / String) | | api lambda | `order-api` | | api lambda role | `order-api-role` | | relay lambda | `outbox-relay` | | relay lambda role | `outbox-relay-role` | | event bus | `order-events` | | rule for created | `on-order-created` | | rule for cancelled | `on-order-cancelled` | | created queue | `orders-created` | | cancelled queue | `orders-cancelled` | | created dlq | `orders-created-dlq` | | cancelled dlq | `orders-cancelled-dlq` | eventbridge detail shape: the relay republishes with `source = "outbox.relay"` and `detail-type` matching `kind` exactly , `OrderCreated` or `OrderCancelled`. rule patterns match by those. python3.11 for both lambdas. they run on the shared compose network so `http://localstack:4566` is reachable from inside them. localstack community 4.13.1. outbox rows need an `event_id` so the relay has a stable dedup key; generate one however you like, just make sure it's the same id when the row gets read off the stream.

Check working directory and AWS env

shell
ls /app && echo "---" && env | grep -E "AWS|LOCAL" | sort
output
---
AWS_ACCESS_KEY_ID=test
AWS_DEFAULT_REGION=us-east-1
AWS_ENDPOINT_URL=http://localstack:4566
AWS_PAGER=
AWS_SECRET_ACCESS_KEY=test

[stdout]
---
AWS_ACCESS_KEY_ID=test
AWS_DEFAULT_REGION=us-east-1
AWS_ENDPOINT_URL=http://localstack:4566
AWS_PAGER=
AWS_SECRET_ACCESS_KEY=test

Verify AWS connectivity and tooling

shell
aws sts get-caller-identity && echo "---" && aws --version && echo "---" && python3 --version && which zip jq
output
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}
---
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24
---
Python 3.12.3
/usr/bin/zip
/usr/bin/jq

[stdout]
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}
---
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24
---
Python 3.12.3
/usr/bin/zip
/usr/bin/jq

Set up build directory

shell
mkdir -p /app/build && ls -la
output
total 12
drwxr-xr-x 1 root root 4096 Apr 28 14:43 .
drwxr-xr-x 1 root root 4096 Apr 28 14:41 ..
drwxr-xr-x 2 root root 4096 Apr 28 14:43 build

[stdout]
total 12
drwxr-xr-x 1 root root 4096 Apr 28 14:43 .
drwxr-xr-x 1 root root 4096 Apr 28 14:41 ..
drwxr-xr-x 2 root root 4096 Apr 28 14:43 build
Let me check current LocalStack state then plan the build.

Check LocalStack state

shell
aws --endpoint-url=http://localstack:4566 dynamodb list-tables 2>&1 | head -20 && echo "---" && aws --endpoint-url=http://localstack:4566 lambda list-functions --query 'Functions[].FunctionName' 2>&1 | head -20 && echo "---" && aws --endpoint-url=http://localstack:4566 sqs list-queues 2>&1 | head -20 && echo "---" && aws --endpoint-url=http://localstack:4566 events list-event-buses --query 'EventBuses[].Name' 2>&1 | head
output
{
    "TableNames": []
}
---
[]
---
---
[
    "default"
]

[stdout]
{
    "TableNames": []
}
---
[]
---
---
[
    "default"
]
Clean state. I'll lay out the project, write the lambdas, and a single deploy script.

Create project directories

shell
mkdir -p /app/lambdas /app/policies /app/build
output
(Bash completed with no output)
write · /app/lambdas/order_api.py
import json
import os
import time
import uuid

import boto3
from botocore.exceptions import ClientError

ENDPOINT = os.environ.get("AWS_ENDPOINT_URL")
ORDERS_TABLE = os.environ["ORDERS_TABLE"]
OUTBOX_TABLE = os.environ["OUTBOX_TABLE"]

_ddb = boto3.client("dynamodb", endpoint_url=ENDPOINT)

ALLOWED_KINDS = {"OrderCreated", "OrderCancelled"}


def _ddb_value(v):
    if isinstance(v, bool):
        return {"BOOL": v}
    if isinstance(v, (int, float)):
        return {"N": str(v)}
    if isinstance(v, str):
        return {"S": v}
    if v is None:
        return {"NULL": True}
    return {"S": json.dumps(v)}


def _to_item(d):
    return {k: _ddb_value(v) for k, v in d.items()}


def handler(event, context):
    body = event
    if isinstance(event, dict) and "body" in event and isinstance(event["body"], str):
        body = json.loads(event["body"])

    order_id = body.get("order_id")
    kind = body.get("kind")

    if not order_id or not isinstance(order_id, str):
        return {"statusCode": 400, "body": json.dumps({"error": "order_id required"})}
    if kind not in ALLOWED_KINDS:
        return {"statusCode": 400, "body": json.dumps({"error": f"kind must be one of {sorted(ALLOWED_KINDS)}"})}

    event_id = str(uuid.uuid4())
    now = int(time.time())

    order_item = {"order_id": order_id, "kind": kind, "updated_at": now}
    for k, v in body.items():
        if k in ("order_id", "kind"):
            continue
        order_item[k] = v

    payload = {k: v for k, v in body.items() if k != "kind"}

    outbox_item = {
        "event_id": event_id,
        "order_id": order_id,
        "kind": kind,
        "payload": json.dumps(payload),
        "created_at": now,
    }

    try:
        _ddb.transact_write_items(
            TransactItems=[
                {"Put": {"TableName": ORDERS_TABLE, "Item": _to_item(order_item)}},
                {"Put": {"TableName": OUTBOX_TABLE, "Item": _to_item(outbox_item)}},
            ]
        )
    except ClientError as e:
        return {"statusCode": 500, "body": json.dumps({"error": str(e)})}

    return {
        "statusCode": 200,
        "body": json.dumps({"order_id": order_id, "event_id": event_id, "kind": kind}),
    }
write · /app/lambdas/outbox_relay.py
import json
import os
import time

import boto3
from botocore.exceptions import ClientError

ENDPOINT = os.environ.get("AWS_ENDPOINT_URL")
EVENT_BUS = os.environ["EVENT_BUS"]
DEDUP_TABLE = os.environ["DEDUP_TABLE"]
DEDUP_TTL_SECONDS = int(os.environ.get("DEDUP_TTL_SECONDS", "604800"))

_events = boto3.client("events", endpoint_url=ENDPOINT)
_ddb = boto3.client("dynamodb", endpoint_url=ENDPOINT)


def _from_ddb(image):
    out = {}
    for k, v in image.items():
        if "S" in v:
            out[k] = v["S"]
        elif "N" in v:
            n = v["N"]
            out[k] = int(n) if "." not in n else float(n)
        elif "BOOL" in v:
            out[k] = v["BOOL"]
        elif "NULL" in v:
            out[k] = None
        else:
            out[k] = v
    return out


def _already_processed(event_id):
    now = int(time.time())
    expires = now + DEDUP_TTL_SECONDS
    try:
        _ddb.put_item(
            TableName=DEDUP_TABLE,
            Item={
                "event_id": {"S": event_id},
                "processed_at": {"N": str(now)},
                "expires_at": {"N": str(expires)},
            },
            ConditionExpression="attribute_not_exists(event_id)",
        )
        return False
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return True
        raise


def _unmark(event_id):
    try:
        _ddb.delete_item(TableName=DEDUP_TABLE, Key={"event_id": {"S": event_id}})
    except ClientError:
        pass


def handler(event, context):
    failures = []

    for record in event.get("Records", []):
        seq = record.get("dynamodb", {}).get("SequenceNumber")
        try:
            new_image = record.get("dynamodb", {}).get("NewImage")
            if not new_image:
                continue

            item = _from_ddb(new_image)
            event_id = item.get("event_id")
            kind = item.get("kind")
            if not event_id or not kind:
                continue

            if _already_processed(event_id):
                continue

            detail = {
                "event_id": event_id,
                "order_id": item.get("order_id"),
                "kind": kind,
                "payload": item.get("payload"),
                "created_at": item.get("created_at"),
            }

            try:
                resp = _events.put_events(
                    Entries=[
                        {
                            "Source": "outbox.relay",
                            "DetailType": kind,
                            "Detail": json.dumps(detail),
                            "EventBusName": EVENT_BUS,
                        }
                    ]
                )
                if resp.get("FailedEntryCount", 0) > 0:
                    _unmark(event_id)
                    failures.append({"itemIdentifier": seq})
            except ClientError:
                _unmark(event_id)
                failures.append({"itemIdentifier": seq})

        except Exception:
            if seq:
                failures.append({"itemIdentifier": seq})

    return {"batchItemFailures": failures}
Now the IAM policy documents , scoped tightly per the brief.
write · /app/policies/lambda-trust.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}
ToolSearch
{
  "query": "select:TodoWrite",
  "max_results": 1
}

, output, 
{"type": "tool_reference", "tool_name": "TodoWrite"}
  • Create KMS CMK + alias
  • Create DynamoDB tables (orders, outbox w/ stream, processed_events w/ TTL)
  • Create SQS queues + DLQs encrypted with CMK
  • Create EventBridge bus + rules
  • Build IAM roles with least-privilege policies
  • Package and deploy lambdas
  • Wire stream event source mapping with ReportBatchItemFailures
  • End-to-end smoke test (created + cancelled paths)
Now I'll create the deploy script. Building it in stages so I can verify each piece.
write · /app/policies/kms-key-policy.json
{
  "Version": "2012-10-17",
  "Id": "outbox-cmk-policy",
  "Statement": [
    {
      "Sid": "EnableRootAccount",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::000000000000:root"},
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "AllowEventBridgeForSqsEncryption",
      "Effect": "Allow",
      "Principal": {"Service": "events.amazonaws.com"},
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowSqsService",
      "Effect": "Allow",
      "Principal": {"Service": "sqs.amazonaws.com"},
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowDynamoDBService",
      "Effect": "Allow",
      "Principal": {"Service": "dynamodb.amazonaws.com"},
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:CreateGrant"
      ],
      "Resource": "*"
    }
  ]
}

Create KMS CMK and alias/outbox-cmk

shell
set -e
cd /app

KEY_OUT=$(aws --endpoint-url=http://localstack:4566 kms create-key \
  --description "outbox CMK" \
  --policy "file:///app/policies/kms-key-policy.json")
echo "$KEY_OUT" | jq .
KEY_ID=$(echo "$KEY_OUT" | jq -r '.KeyMetadata.KeyId')
KEY_ARN=$(echo "$KEY_OUT" | jq -r '.KeyMetadata.Arn')
echo "KEY_ID=$KEY_ID"
echo "KEY_ARN=$KEY_ARN"
echo "$KEY_ID" > /app/build/kms_key_id
echo "$KEY_ARN" > /app/build/kms_key_arn

aws --endpoint-url=http://localstack:4566 kms create-alias \
  --alias-name alias/outbox-cmk \
  --target-key-id "$KEY_ID"

aws --endpoint-url=http://localstack:4566 kms list-aliases --query 'Aliases[?AliasName==`alias/outbox-cmk`]'
output
{
  "KeyMetadata": {
    "AWSAccountId": "000000000000",
    "KeyId": "7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
    "Arn": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
    "CreationDate": "2026-04-28T14:45:18.846487+00:00",
    "Enabled": true,
    "Description": "outbox CMK",
    "KeyUsage": "ENCRYPT_DECRYPT",
    "KeyState": "Enabled",
    "Origin": "AWS_KMS",
    "KeyManager": "CUSTOMER",
    "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
    "KeySpec": "SYMMETRIC_DEFAULT",
    "EncryptionAlgorithms": [
      "SYMMETRIC_DEFAULT"
    ],
    "MultiRegion": false,
    "CurrentKeyMaterialId": "90ada03499395321b80d2fee74589bd490ada03499395321b80d2fee74589bd4"
  }
}
KEY_ID=7c1f56dc-9476-4b76-ba8c-61d9285b06e9
KEY_ARN=arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9
[
    {
        "AliasName": "alias/outbox-cmk",
        "AliasArn": "arn:aws:kms:us-east-1:000000000000:alias/outbox-cmk",
        "TargetKeyId": "7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
        "CreationDate": "2026-04-28T14:45:19.254621+00:00"
    }
]

[stdout]
{
  "KeyMetadata": {
    "AWSAccountId": "000000000000",
    "KeyId": "7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
    "Arn": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
    "CreationDate": "2026-04-28T14:45:18.846487+00:00",
    "Enabled": true,
    "Description": "outbox CMK",
    "KeyUsage": "ENCRYPT_DECRYPT",
    "KeyState": "Enabled",
    "Origin": "AWS_KMS",
    "KeyManager": "CUSTOMER",
    "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
    "KeySpec": "SYMMETRIC_DEFAULT",
    "EncryptionAlgorithms": [
      "SYMMETRIC_DEFAULT"
    ],
    "MultiRegion": false,
    "CurrentKeyMaterialId": "90ada03499395321b80d2fee74589bd490ada03499395321b80d2fee74589bd4"
  }
}
KEY_ID=7c1f56dc-9476-4b76-ba8c-61d9285b06e9
KEY_ARN=arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9
[
    {
        "AliasName": "alias/outbox-cmk",
        "AliasArn": "arn:aws:kms:us-east-1:000000000000:alias/outbox-cmk",
        "TargetKeyId": "7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
        "CreationDate": "2026-04-28T14:45:19.254621+00:00"
    }
]
  • Create KMS CMK + alias
  • Create DynamoDB tables (orders, outbox w/ stream, processed_events w/ TTL)
  • Create SQS queues + DLQs encrypted with CMK
  • Create EventBridge bus + rules
  • Build IAM roles with least-privilege policies
  • Package and deploy lambdas
  • Wire stream event source mapping with ReportBatchItemFailures
  • End-to-end smoke test (created + cancelled paths)

Create DynamoDB tables with CMK encryption

shell
set -e
KEY_ARN=$(cat /app/build/kms_key_arn)
EP=http://localstack:4566

aws --endpoint-url=$EP dynamodb create-table \
  --table-name orders \
  --attribute-definitions AttributeName=order_id,AttributeType=S \
  --key-schema AttributeName=order_id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --sse-specification "Enabled=true,SSEType=KMS,KMSMasterKeyId=${KEY_ARN}" \
  >/dev/null

aws --endpoint-url=$EP dynamodb create-table \
  --table-name outbox \
  --attribute-definitions AttributeName=event_id,AttributeType=S \
  --key-schema AttributeName=event_id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --sse-specification "Enabled=true,SSEType=KMS,KMSMasterKeyId=${KEY_ARN}" \
  --stream-specification "StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES" \
  >/dev/null

aws --endpoint-url=$EP dynamodb create-table \
  --table-name processed_events \
  --attribute-definitions AttributeName=event_id,AttributeType=S \
  --key-schema AttributeName=event_id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --sse-specification "Enabled=true,SSEType=KMS,KMSMasterKeyId=${KEY_ARN}" \
  >/dev/null

# Wait active
for t in orders outbox processed_events; do
  for _ in 1 2 3 4 5 6 7 8 9 10; do
    s=$(aws --endpoint-url=$EP dynamodb describe-table --table-name $t --query 'Table.TableStatus' --output text)
    [ "$s" = "ACTIVE" ] && break
    sleep 1
  done
done

aws --endpoint-url=$EP dynamodb update-time-to-live \
  --table-name processed_events \
  --time-to-live-specification "Enabled=true,AttributeName=expires_at" \
  >/dev/null

OUTBOX_STREAM_ARN=$(aws --endpoint-url=$EP dynamodb describe-table --table-name outbox --query 'Table.LatestStreamArn' --output text)
ORDERS_TABLE_ARN=$(aws --endpoint-url=$EP dynamodb describe-table --table-name orders --query 'Table.TableArn' --output text)
OUTBOX_TABLE_ARN=$(aws --endpoint-url=$EP dynamodb describe-table --table-name outbox --query 'Table.TableArn' --output text)
DEDUP_TABLE_ARN=$(aws --endpoint-url=$EP dynamodb describe-table --table-name processed_events --query 'Table.TableArn' --output text)

echo "$OUTBOX_STREAM_ARN" > /app/build/outbox_stream_arn
echo "$ORDERS_TABLE_ARN" > /app/build/orders_table_arn
echo "$OUTBOX_TABLE_ARN" > /app/build/outbox_table_arn
echo "$DEDUP_TABLE_ARN" > /app/build/dedup_table_arn

echo "OUTBOX_STREAM_ARN=$OUTBOX_STREAM_ARN"
echo "ORDERS_TABLE_ARN=$ORDERS_TABLE_ARN"
echo "OUTBOX_TABLE_ARN=$OUTBOX_TABLE_ARN"
echo "DEDUP_TABLE_ARN=$DEDUP_TABLE_ARN"

echo "---"
aws --endpoint-url=$EP dynamodb describe-table --table-name outbox --query 'Table.{Name:TableName,Stream:StreamSpecification,SSE:SSEDescription}'
echo "---"
aws --endpoint-url=$EP dynamodb describe-time-to-live --table-name processed_events
output
OUTBOX_STREAM_ARN=arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187
ORDERS_TABLE_ARN=arn:aws:dynamodb:us-east-1:000000000000:table/orders
OUTBOX_TABLE_ARN=arn:aws:dynamodb:us-east-1:000000000000:table/outbox
DEDUP_TABLE_ARN=arn:aws:dynamodb:us-east-1:000000000000:table/processed_events
---
{
    "Name": "outbox",
    "Stream": {
        "StreamEnabled": true,
        "StreamViewType": "NEW_AND_OLD_IMAGES"
    },
    "SSE": {
        "Status": "ENABLED",
        "SSEType": "KMS",
        "KMSMasterKeyArn": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
    }
}
---
{
    "TimeToLiveDescription": {
        "TimeToLiveStatus": "ENABLED",
        "AttributeName": "expires_at"
    }
}

[stdout]
OUTBOX_STREAM_ARN=arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187
ORDERS_TABLE_ARN=arn:aws:dynamodb:us-east-1:000000000000:table/orders
OUTBOX_TABLE_ARN=arn:aws:dynamodb:us-east-1:000000000000:table/outbox
DEDUP_TABLE_ARN=arn:aws:dynamodb:us-east-1:000000000000:table/processed_events
---
{
    "Name": "outbox",
    "Stream": {
        "StreamEnabled": true,
        "StreamViewType": "NEW_AND_OLD_IMAGES"
    },
    "SSE": {
        "Status": "ENABLED",
        "SSEType": "KMS",
        "KMSMasterKeyArn": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
    }
}
---
{
    "TimeToLiveDescription": {
        "TimeToLiveStatus": "ENABLED",
        "AttributeName": "expires_at"
    }
}
Tables look right , outbox has the stream + CMK SSE, processed_events has TTL on `expires_at`.
  • Create KMS CMK + alias
  • Create DynamoDB tables (orders, outbox w/ stream, processed_events w/ TTL)
  • Create SQS queues + DLQs encrypted with CMK
  • Create EventBridge bus + rules
  • Build IAM roles with least-privilege policies
  • Package and deploy lambdas
  • Wire stream event source mapping with ReportBatchItemFailures
  • End-to-end smoke test (created + cancelled paths)

Create SQS queues + DLQs encrypted with CMK

shell
set -e
KEY_ID=$(cat /app/build/kms_key_id)
EP=http://localstack:4566

# DLQs first (so main queues can reference them in redrive)
for q in orders-created-dlq orders-cancelled-dlq; do
  aws --endpoint-url=$EP sqs create-queue \
    --queue-name "$q" \
    --attributes "{\"KmsMasterKeyId\":\"$KEY_ID\"}" \
    --query QueueUrl --output text > /app/build/${q//-/_}_url
done

CREATED_DLQ_URL=$(cat /app/build/orders_created_dlq_url)
CANCELLED_DLQ_URL=$(cat /app/build/orders_cancelled_dlq_url)
CREATED_DLQ_ARN=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_DLQ_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
CANCELLED_DLQ_ARN=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_DLQ_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
echo "$CREATED_DLQ_ARN" > /app/build/orders_created_dlq_arn
echo "$CANCELLED_DLQ_ARN" > /app/build/orders_cancelled_dlq_arn

# Main queues with redrive policy + CMK encryption
CREATED_REDRIVE=$(jq -nc --arg arn "$CREATED_DLQ_ARN" '{deadLetterTargetArn:$arn, maxReceiveCount:"5"}')
CANCELLED_REDRIVE=$(jq -nc --arg arn "$CANCELLED_DLQ_ARN" '{deadLetterTargetArn:$arn, maxReceiveCount:"5"}')

aws --endpoint-url=$EP sqs create-queue \
  --queue-name orders-created \
  --attributes "$(jq -nc --arg k "$KEY_ID" --arg r "$CREATED_REDRIVE" '{KmsMasterKeyId:$k, RedrivePolicy:$r}')" \
  --query QueueUrl --output text > /app/build/orders_created_url

aws --endpoint-url=$EP sqs create-queue \
  --queue-name orders-cancelled \
  --attributes "$(jq -nc --arg k "$KEY_ID" --arg r "$CANCELLED_REDRIVE" '{KmsMasterKeyId:$k, RedrivePolicy:$r}')" \
  --query QueueUrl --output text > /app/build/orders_cancelled_url

CREATED_URL=$(cat /app/build/orders_created_url)
CANCELLED_URL=$(cat /app/build/orders_cancelled_url)
CREATED_ARN=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
CANCELLED_ARN=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
echo "$CREATED_ARN" > /app/build/orders_created_arn
echo "$CANCELLED_ARN" > /app/build/orders_cancelled_arn

echo "CREATED_URL=$CREATED_URL ARN=$CREATED_ARN DLQ=$CREATED_DLQ_ARN"
echo "CANCELLED_URL=$CANCELLED_URL ARN=$CANCELLED_ARN DLQ=$CANCELLED_DLQ_ARN"

# Verify encryption + redrive on a sample
echo "---"
aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names KmsMasterKeyId RedrivePolicy
output
CREATED_URL=http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/orders-created ARN=arn:aws:sqs:us-east-1:000000000000:orders-created DLQ=arn:aws:sqs:us-east-1:000000000000:orders-created-dlq
CANCELLED_URL=http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/orders-cancelled ARN=arn:aws:sqs:us-east-1:000000000000:orders-cancelled DLQ=arn:aws:sqs:us-east-1:000000000000:orders-cancelled-dlq
---
{
    "Attributes": {
        "KmsMasterKeyId": "7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
        "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:000000000000:orders-created-dlq\",\"maxReceiveCount\":\"5\"}"
    }
}

[stdout]
CREATED_URL=http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/orders-created ARN=arn:aws:sqs:us-east-1:000000000000:orders-created DLQ=arn:aws:sqs:us-east-1:000000000000:orders-created-dlq
CANCELLED_URL=http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/orders-cancelled ARN=arn:aws:sqs:us-east-1:000000000000:orders-cancelled DLQ=arn:aws:sqs:us-east-1:000000000000:orders-cancelled-dlq
---
{
    "Attributes": {
        "KmsMasterKeyId": "7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
        "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:000000000000:orders-created-dlq\",\"maxReceiveCount\":\"5\"}"
    }
}
  • Create KMS CMK + alias
  • Create DynamoDB tables (orders, outbox w/ stream, processed_events w/ TTL)
  • Create SQS queues + DLQs encrypted with CMK
  • Create EventBridge bus + rules
  • Build IAM roles with least-privilege policies
  • Package and deploy lambdas
  • Wire stream event source mapping with ReportBatchItemFailures
  • End-to-end smoke test (created + cancelled paths)

Create EventBridge bus + rules + queue policies

shell
set -e
EP=http://localstack:4566

aws --endpoint-url=$EP events create-event-bus --name order-events --query 'EventBusArn' --output text > /app/build/bus_arn
BUS_ARN=$(cat /app/build/bus_arn)
echo "BUS_ARN=$BUS_ARN"

CREATED_ARN=$(cat /app/build/orders_created_arn)
CANCELLED_ARN=$(cat /app/build/orders_cancelled_arn)

# Rules
aws --endpoint-url=$EP events put-rule \
  --name on-order-created \
  --event-bus-name order-events \
  --event-pattern '{"source":["outbox.relay"],"detail-type":["OrderCreated"]}' \
  >/dev/null

aws --endpoint-url=$EP events put-rule \
  --name on-order-cancelled \
  --event-bus-name order-events \
  --event-pattern '{"source":["outbox.relay"],"detail-type":["OrderCancelled"]}' \
  >/dev/null

# Targets
aws --endpoint-url=$EP events put-targets \
  --rule on-order-created \
  --event-bus-name order-events \
  --targets "Id=created-queue,Arn=$CREATED_ARN" >/dev/null

aws --endpoint-url=$EP events put-targets \
  --rule on-order-cancelled \
  --event-bus-name order-events \
  --targets "Id=cancelled-queue,Arn=$CANCELLED_ARN" >/dev/null

# Allow EventBridge to send to each queue (queue policies)
for q_url_file in orders_created_url orders_cancelled_url; do
  Q_URL=$(cat /app/build/$q_url_file)
  Q_ARN=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$Q_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
  POLICY=$(jq -nc --arg arn "$Q_ARN" '{
    Version:"2012-10-17",
    Statement:[
      {Sid:"AllowEventBridge", Effect:"Allow", Principal:{Service:"events.amazonaws.com"}, Action:"sqs:SendMessage", Resource:$arn}
    ]
  }')
  aws --endpoint-url=$EP sqs set-queue-attributes \
    --queue-url "$Q_URL" \
    --attributes "$(jq -nc --arg p "$POLICY" '{Policy:$p}')" >/dev/null
done

aws --endpoint-url=$EP events list-rules --event-bus-name order-events
echo "---"
aws --endpoint-url=$EP events list-targets-by-rule --rule on-order-created --event-bus-name order-events
aws --endpoint-url=$EP events list-targets-by-rule --rule on-order-cancelled --event-bus-name order-events
output
BUS_ARN=arn:aws:events:us-east-1:000000000000:event-bus/order-events
{
    "Rules": [
        {
            "Name": "on-order-created",
            "Arn": "arn:aws:events:us-east-1:000000000000:rule/order-events/on-order-created",
            "EventPattern": "{\"source\":[\"outbox.relay\"],\"detail-type\":[\"OrderCreated\"]}",
            "State": "ENABLED",
            "EventBusName": "order-events"
        },
        {
            "Name": "on-order-cancelled",
            "Arn": "arn:aws:events:us-east-1:000000000000:rule/order-events/on-order-cancelled",
            "EventPattern": "{\"source\":[\"outbox.relay\"],\"detail-type\":[\"OrderCancelled\"]}",
            "State": "ENABLED",
            "EventBusName": "order-events"
        }
    ]
}
---
{
    "Targets": [
        {
            "Id": "created-queue",
            "Arn": "arn:aws:sqs:us-east-1:000000000000:orders-created"
        }
    ]
}
{
    "Targets": [
        {
            "Id": "cancelled-queue",
            "Arn": "arn:aws:sqs:us-east-1:000000000000:orders-cancelled"
        }
    ]
}

[stdout]
BUS_ARN=arn:aws:events:us-east-1:000000000000:event-bus/order-events
{
    "Rules": [
        {
            "Name": "on-order-created",
            "Arn": "arn:aws:events:us-east-1:000000000000:rule/order-events/on-order-created",
            "EventPattern": "{\"source\":[\"outbox.relay\"],\"detail-type\":[\"OrderCreated\"]}",
            "State": "ENABLED",
            "EventBusName": "order-events"
        },
        {
            "Name": "on-order-cancelled",
            "Arn": "arn:aws:events:us-east-1:000000000000:rule/order-events/on-order-cancelled",
            "EventPattern": "{\"source\":[\"outbox.relay\"],\"detail-type\":[\"OrderCancelled\"]}",
            "State": "ENABLED",
            "EventBusName": "order-events"
        }
    ]
}
---
{
    "Targets": [
        {
            "Id": "created-queue",
            "Arn": "arn:aws:sqs:us-east-1:000000000000:orders-created"
        }
    ]
}
{
    "Targets": [
        {
            "Id": "cancelled-queue",
            "Arn": "arn:aws:sqs:us-east-1:000000000000:orders-cancelled"
        }
    ]
}
  • Create KMS CMK + alias
  • Create DynamoDB tables (orders, outbox w/ stream, processed_events w/ TTL)
  • Create SQS queues + DLQs encrypted with CMK
  • Create EventBridge bus + rules
  • Build IAM roles with least-privilege policies
  • Package and deploy lambdas
  • Wire stream event source mapping with ReportBatchItemFailures
  • End-to-end smoke test (created + cancelled paths)

Create IAM roles + scoped inline policies

shell
set -e
EP=http://localstack:4566

KEY_ARN=$(cat /app/build/kms_key_arn)
ORDERS_TABLE_ARN=$(cat /app/build/orders_table_arn)
OUTBOX_TABLE_ARN=$(cat /app/build/outbox_table_arn)
DEDUP_TABLE_ARN=$(cat /app/build/dedup_table_arn)
OUTBOX_STREAM_ARN=$(cat /app/build/outbox_stream_arn)
BUS_ARN=$(cat /app/build/bus_arn)

# order-api-role policy
jq -n \
  --arg orders "$ORDERS_TABLE_ARN" \
  --arg outbox "$OUTBOX_TABLE_ARN" \
  --arg key "$KEY_ARN" \
  '{
    Version:"2012-10-17",
    Statement:[
      {
        Sid:"TransactWriteOrdersAndOutbox",
        Effect:"Allow",
        Action:["dynamodb:PutItem","dynamodb:UpdateItem","dynamodb:DescribeTable","dynamodb:TransactWriteItems"],
        Resource:[$orders, $outbox]
      },
      {
        Sid:"KmsForDynamoDb",
        Effect:"Allow",
        Action:["kms:GenerateDataKey","kms:Decrypt","kms:DescribeKey"],
        Resource:$key,
        Condition:{StringEquals:{"kms:ViaService":"dynamodb.us-east-1.amazonaws.com"}}
      },
      {
        Sid:"Logs",
        Effect:"Allow",
        Action:["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],
        Resource:"*"
      }
    ]
  }' > /app/policies/order-api-policy.json

# outbox-relay-role policy
jq -n \
  --arg stream "$OUTBOX_STREAM_ARN" \
  --arg dedup "$DEDUP_TABLE_ARN" \
  --arg bus "$BUS_ARN" \
  --arg key "$KEY_ARN" \
  '{
    Version:"2012-10-17",
    Statement:[
      {
        Sid:"ReadOutboxStream",
        Effect:"Allow",
        Action:["dynamodb:DescribeStream","dynamodb:GetRecords","dynamodb:GetShardIterator","dynamodb:ListStreams"],
        Resource:$stream
      },
      {
        Sid:"DedupTable",
        Effect:"Allow",
        Action:["dynamodb:PutItem","dynamodb:GetItem","dynamodb:DeleteItem"],
        Resource:$dedup
      },
      {
        Sid:"PutEventsToBus",
        Effect:"Allow",
        Action:"events:PutEvents",
        Resource:$bus
      },
      {
        Sid:"KmsDecryptForCmk",
        Effect:"Allow",
        Action:["kms:Decrypt","kms:DescribeKey","kms:GenerateDataKey"],
        Resource:$key
      },
      {
        Sid:"Logs",
        Effect:"Allow",
        Action:["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],
        Resource:"*"
      }
    ]
  }' > /app/policies/outbox-relay-policy.json

# Roles
aws --endpoint-url=$EP iam create-role \
  --role-name order-api-role \
  --assume-role-policy-document file:///app/policies/lambda-trust.json \
  --query 'Role.Arn' --output text > /app/build/order_api_role_arn

aws --endpoint-url=$EP iam create-role \
  --role-name outbox-relay-role \
  --assume-role-policy-document file:///app/policies/lambda-trust.json \
  --query 'Role.Arn' --output text > /app/build/outbox_relay_role_arn

# Inline policies
aws --endpoint-url=$EP iam put-role-policy \
  --role-name order-api-role \
  --policy-name order-api-inline \
  --policy-document file:///app/policies/order-api-policy.json

aws --endpoint-url=$EP iam put-role-policy \
  --role-name outbox-relay-role \
  --policy-name outbox-relay-inline \
  --policy-document file:///app/policies/outbox-relay-policy.json

echo "order-api-role: $(cat /app/build/order_api_role_arn)"
echo "outbox-relay-role: $(cat /app/build/outbox_relay_role_arn)"

echo "--- order-api-policy.json ---"
cat /app/policies/order-api-policy.json | jq .
echo "--- outbox-relay-policy.json ---"
cat /app/policies/outbox-relay-policy.json | jq .
output
order-api-role: arn:aws:iam::000000000000:role/order-api-role
outbox-relay-role: arn:aws:iam::000000000000:role/outbox-relay-role
--- order-api-policy.json ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TransactWriteOrdersAndOutbox",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DescribeTable",
        "dynamodb:TransactWriteItems"
      ],
      "Resource": [
        "arn:aws:dynamodb:us-east-1:000000000000:table/orders",
        "arn:aws:dynamodb:us-east-1:000000000000:table/outbox"
      ]
    },
    {
      "Sid": "KmsForDynamoDb",
      "Effect": "Allow",
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "dynamodb.us-east-1.amazonaws.com"
        }
      }
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
--- outbox-relay-policy.json ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOutboxStream",
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187"
    },
    {
      "Sid": "DedupTable",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:GetItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/processed_events"
    },
    {
      "Sid": "PutEventsToBus",
      "Effect": "Allow",
      "Action": "events:PutEvents",
      "Resource": "arn:aws:events:us-east-1:000000000000:event-bus/order-events"
    },
    {
      "Sid": "KmsDecryptForCmk",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:GenerateDataKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

[stdout]
order-api-role: arn:aws:iam::000000000000:role/order-api-role
outbox-relay-role: arn:aws:iam::000000000000:role/outbox-relay-role
--- order-api-policy.json ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TransactWriteOrdersAndOutbox",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DescribeTable",
        "dynamodb:TransactWriteItems"
      ],
      "Resource": [
        "arn:aws:dynamodb:us-east-1:000000000000:table/orders",
        "arn:aws:dynamodb:us-east-1:000000000000:table/outbox"
      ]
    },
    {
      "Sid": "KmsForDynamoDb",
      "Effect": "Allow",
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "dynamodb.us-east-1.amazonaws.com"
        }
      }
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
--- outbox-relay-policy.json ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOutboxStream",
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187"
    },
    {
      "Sid": "DedupTable",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:GetItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/processed_events"
    },
    {
      "Sid": "PutEventsToBus",
      "Effect": "Allow",
      "Action": "events:PutEvents",
      "Resource": "arn:aws:events:us-east-1:000000000000:event-bus/order-events"
    },
    {
      "Sid": "KmsDecryptForCmk",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:GenerateDataKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
  • Create KMS CMK + alias
  • Create DynamoDB tables (orders, outbox w/ stream, processed_events w/ TTL)
  • Create SQS queues + DLQs encrypted with CMK
  • Create EventBridge bus + rules
  • Build IAM roles with least-privilege policies
  • Package and deploy lambdas
  • Wire stream event source mapping with ReportBatchItemFailures
  • End-to-end smoke test (created + cancelled paths)

Zip lambda code

shell
set -e
cd /app/lambdas
zip -j /app/build/order_api.zip order_api.py >/dev/null
zip -j /app/build/outbox_relay.zip outbox_relay.py >/dev/null
ls -la /app/build/*.zip
output
-rw-r--r-- 1 root root  986 Apr 28 14:47 /app/build/order_api.zip
-rw-r--r-- 1 root root 1208 Apr 28 14:47 /app/build/outbox_relay.zip

[stdout]
-rw-r--r-- 1 root root  986 Apr 28 14:47 /app/build/order_api.zip
-rw-r--r-- 1 root root 1208 Apr 28 14:47 /app/build/outbox_relay.zip

Deploy both lambdas

shell
set -e
EP=http://localstack:4566

ORDER_ROLE=$(cat /app/build/order_api_role_arn)
RELAY_ROLE=$(cat /app/build/outbox_relay_role_arn)
KEY_ARN=$(cat /app/build/kms_key_arn)

# Create order-api lambda
aws --endpoint-url=$EP lambda create-function \
  --function-name order-api \
  --runtime python3.11 \
  --role "$ORDER_ROLE" \
  --handler order_api.handler \
  --zip-file fileb:///app/build/order_api.zip \
  --timeout 15 \
  --kms-key-arn "$KEY_ARN" \
  --environment "Variables={AWS_ENDPOINT_URL=http://localstack:4566,ORDERS_TABLE=orders,OUTBOX_TABLE=outbox}" \
  --query 'FunctionArn' --output text > /app/build/order_api_arn

aws --endpoint-url=$EP lambda create-function \
  --function-name outbox-relay \
  --runtime python3.11 \
  --role "$RELAY_ROLE" \
  --handler outbox_relay.handler \
  --zip-file fileb:///app/build/outbox_relay.zip \
  --timeout 30 \
  --kms-key-arn "$KEY_ARN" \
  --environment "Variables={AWS_ENDPOINT_URL=http://localstack:4566,EVENT_BUS=order-events,DEDUP_TABLE=processed_events,DEDUP_TTL_SECONDS=604800}" \
  --query 'FunctionArn' --output text > /app/build/outbox_relay_arn

# wait active
for fn in order-api outbox-relay; do
  for _ in $(seq 1 30); do
    s=$(aws --endpoint-url=$EP lambda get-function-configuration --function-name $fn --query 'State' --output text 2>/dev/null || echo Pending)
    [ "$s" = "Active" ] && break
    sleep 1
  done
  echo "$fn state: $s"
done

cat /app/build/order_api_arn
cat /app/build/outbox_relay_arn
output
order-api state: Active
outbox-relay state: Active
arn:aws:lambda:us-east-1:000000000000:function:order-api
arn:aws:lambda:us-east-1:000000000000:function:outbox-relay

[stdout]
order-api state: Active
outbox-relay state: Active
arn:aws:lambda:us-east-1:000000000000:function:order-api
arn:aws:lambda:us-east-1:000000000000:function:outbox-relay

Create event source mapping with ReportBatchItemFailures

shell
set -e
EP=http://localstack:4566
STREAM_ARN=$(cat /app/build/outbox_stream_arn)

ESM_OUT=$(aws --endpoint-url=$EP lambda create-event-source-mapping \
  --function-name outbox-relay \
  --event-source-arn "$STREAM_ARN" \
  --starting-position TRIM_HORIZON \
  --batch-size 10 \
  --maximum-retry-attempts 3 \
  --function-response-types ReportBatchItemFailures)
echo "$ESM_OUT" | jq .
ESM_UUID=$(echo "$ESM_OUT" | jq -r '.UUID')

# wait for it to enable
for _ in $(seq 1 30); do
  s=$(aws --endpoint-url=$EP lambda get-event-source-mapping --uuid "$ESM_UUID" --query 'State' --output text)
  echo "  ESM state: $s"
  [ "$s" = "Enabled" ] && break
  sleep 1
done

aws --endpoint-url=$EP lambda get-event-source-mapping --uuid "$ESM_UUID" --query '{State:State,FRT:FunctionResponseTypes,EventSource:EventSourceArn,FunctionArn:FunctionArn}'
output
{
  "UUID": "461e2b56-dcf2-405b-a8b7-2416509bdc1f",
  "StartingPosition": "TRIM_HORIZON",
  "BatchSize": 10,
  "MaximumBatchingWindowInSeconds": 0,
  "ParallelizationFactor": 1,
  "EventSourceArn": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187",
  "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:outbox-relay",
  "LastModified": "2026-04-28T14:48:04.308436+00:00",
  "LastProcessingResult": "No records processed",
  "State": "Creating",
  "StateTransitionReason": "User action",
  "DestinationConfig": {
    "OnFailure": {}
  },
  "MaximumRecordAgeInSeconds": -1,
  "BisectBatchOnFunctionError": false,
  "MaximumRetryAttempts": 3,
  "TumblingWindowInSeconds": 0,
  "FunctionResponseTypes": [
    "ReportBatchItemFailures"
  ],
  "EventSourceMappingArn": "arn:aws:lambda:us-east-1:000000000000:event-source-mapping:461e2b56-dcf2-405b-a8b7-2416509bdc1f"
}
  ESM state: Enabled
{
    "State": "Enabled",
    "FRT": [
        "ReportBatchItemFailures"
    ],
    "EventSource": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187",
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:outbox-relay"
}

[stdout]
{
  "UUID": "461e2b56-dcf2-405b-a8b7-2416509bdc1f",
  "StartingPosition": "TRIM_HORIZON",
  "BatchSize": 10,
  "MaximumBatchingWindowInSeconds": 0,
  "ParallelizationFactor": 1,
  "EventSourceArn": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187",
  "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:outbox-relay",
  "LastModified": "2026-04-28T14:48:04.308436+00:00",
  "LastProcessingResult": "No records processed",
  "State": "Creating",
  "StateTransitionReason": "User action",
  "DestinationConfig": {
    "OnFailure": {}
  },
  "MaximumRecordAgeInSeconds": -1,
  "BisectBatchOnFunctionError": false,
  "MaximumRetryAttempts": 3,
  "TumblingWindowInSeconds": 0,
  "FunctionResponseTypes": [
    "ReportBatchItemFailures"
  ],
  "EventSourceMappingArn": "arn:aws:lambda:us-east-1:000000000000:event-source-mapping:461e2b56-dcf2-405b-a8b7-2416509bdc1f"
}
  ESM state: Enabled
{
    "State": "Enabled",
    "FRT": [
        "ReportBatchItemFailures"
    ],
    "EventSource": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187",
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:outbox-relay"
}
  • Create KMS CMK + alias
  • Create DynamoDB tables (orders, outbox w/ stream, processed_events w/ TTL)
  • Create SQS queues + DLQs encrypted with CMK
  • Create EventBridge bus + rules
  • Build IAM roles with least-privilege policies
  • Package and deploy lambdas
  • Wire stream event source mapping with ReportBatchItemFailures
  • End-to-end smoke test (created + cancelled paths)
Now the end-to-end smoke test.

Smoke test: OrderCreated path

shell
set -e
EP=http://localstack:4566
CREATED_URL=$(cat /app/build/orders_created_url)
CANCELLED_URL=$(cat /app/build/orders_cancelled_url)

# Drain queues just in case
aws --endpoint-url=$EP sqs purge-queue --queue-url "$CREATED_URL" 2>/dev/null || true
aws --endpoint-url=$EP sqs purge-queue --queue-url "$CANCELLED_URL" 2>/dev/null || true

# Invoke order-api with OrderCreated
PAYLOAD1=$(echo '{"order_id":"o-1","kind":"OrderCreated","amount":4200,"sku":"sku-A"}' | base64 -w0)
aws --endpoint-url=$EP lambda invoke \
  --function-name order-api \
  --payload "$PAYLOAD1" \
  --cli-binary-format raw-in-base64-out \
  /app/build/order_api_resp1.json
cat /app/build/order_api_resp1.json
echo
echo "--- Wait for stream/relay to deliver ---"

# Poll until message arrives in orders-created (up to 30s)
DEADLINE=$(( $(date +%s) + 30 ))
GOT=0
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
  N=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
  echo "  orders-created depth: $N"
  if [ "$N" -gt 0 ]; then GOT=1; break; fi
  sleep 2
done
[ $GOT -eq 1 ] || { echo "TIMEOUT: no message in orders-created"; exit 1; }

echo "--- Message in orders-created ---"
aws --endpoint-url=$EP sqs receive-message --queue-url "$CREATED_URL" --max-number-of-messages 10 --wait-time-seconds 1 --visibility-timeout 1 | tee /app/build/created_msg.json | jq .
echo "--- Empty? orders-cancelled depth ---"
aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names ApproximateNumberOfMessages

echo "--- Outbox / dedup table snapshot ---"
aws --endpoint-url=$EP dynamodb scan --table-name outbox --query 'Count'
aws --endpoint-url=$EP dynamodb scan --table-name processed_events
output
Exit code 1
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}
{"errorMessage": "Unable to unmarshal input: Expecting value: line 1 column 1 (char 0)", "errorType": "Runtime.UnmarshalError", "requestId": "896b67c9-125b-4d75-94b1-7feaf621c76d", "stackTrace": []}
--- Wait for stream/relay to deliver ---
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
  orders-created depth: 0
TIMEOUT: no message in orders-created

[error] tool reported failure
Payload encoding mismatch. Let me pass it as a file with `--cli-binary-format raw-in-base64-out` (raw input):

Re-run OrderCreated path with raw payload

shell
set -e
EP=http://localstack:4566
CREATED_URL=$(cat /app/build/orders_created_url)
CANCELLED_URL=$(cat /app/build/orders_cancelled_url)

# Drain queues
aws --endpoint-url=$EP sqs purge-queue --queue-url "$CREATED_URL" 2>/dev/null || true
aws --endpoint-url=$EP sqs purge-queue --queue-url "$CANCELLED_URL" 2>/dev/null || true

cat > /app/build/p1.json <<'EOF'
{"order_id":"o-1","kind":"OrderCreated","amount":4200,"sku":"sku-A"}
EOF

aws --endpoint-url=$EP lambda invoke \
  --function-name order-api \
  --payload fileb:///app/build/p1.json \
  --cli-binary-format raw-in-base64-out \
  /app/build/order_api_resp1.json
echo "Response body:"
cat /app/build/order_api_resp1.json
echo

DEADLINE=$(( $(date +%s) + 30 ))
GOT=0
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
  N=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
  echo "  orders-created depth: $N"
  if [ "$N" -gt 0 ]; then GOT=1; break; fi
  sleep 2
done
[ $GOT -eq 1 ] || { echo "TIMEOUT"; exit 1; }

echo "--- Message body ---"
aws --endpoint-url=$EP sqs receive-message --queue-url "$CREATED_URL" --max-number-of-messages 10 --wait-time-seconds 1 --visibility-timeout 1 | jq -r '.Messages[0].Body' | jq .

echo "--- orders-cancelled should be 0 ---"
aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names ApproximateNumberOfMessages

echo "--- processed_events ---"
aws --endpoint-url=$EP dynamodb scan --table-name processed_events --query 'Items'
output
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
Response body:
{"statusCode": 200, "body": "{\"order_id\": \"o-1\", \"event_id\": \"53c91c06-bacf-40af-ad0d-3c9f7d85c3a1\", \"kind\": \"OrderCreated\"}"}
  orders-created depth: 0
  orders-created depth: 1
--- Message body ---
{
  "version": "0",
  "id": "baf9f2d7-c624-44d2-8c46-6672c4ef0212",
  "detail-type": "OrderCreated",
  "source": "outbox.relay",
  "account": "000000000000",
  "time": "2026-04-28T14:49:17Z",
  "region": "us-east-1",
  "resources": [],
  "detail": {
    "event_id": "53c91c06-bacf-40af-ad0d-3c9f7d85c3a1",
    "order_id": "o-1",
    "kind": "OrderCreated",
    "payload": "{\"order_id\": \"o-1\", \"amount\": 4200, \"sku\": \"sku-A\"}",
    "created_at": 1777387755
  }
}
--- orders-cancelled should be 0 ---
{
    "Attributes": {
        "ApproximateNumberOfMessages": "0"
    }
}
--- processed_events ---
[
    {
        "processed_at": {
            "N": "1777387757"
        },
        "event_id": {
            "S": "53c91c06-bacf-40af-ad0d-3c9f7d85c3a1"
        },
        "expires_at": {
            "N": "1777992557"
        }
    }
]

[stdout]
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
Response body:
{"statusCode": 200, "body": "{\"order_id\": \"o-1\", \"event_id\": \"53c91c06-bacf-40af-ad0d-3c9f7d85c3a1\", \"kind\": \"OrderCreated\"}"}
  orders-created depth: 0
  orders-created depth: 1
--- Message body ---
{
  "version": "0",
  "id": "baf9f2d7-c624-44d2-8c46-6672c4ef0212",
  "detail-type": "OrderCreated",
  "source": "outbox.relay",
  "account": "000000000000",
  "time": "2026-04-28T14:49:17Z",
  "region": "us-east-1",
  "resources": [],
  "detail": {
    "event_id": "53c91c06-bacf-40af-ad0d-3c9f7d85c3a1",
    "order_id": "o-1",
    "kind": "OrderCreated",
    "payload": "{\"order_id\": \"o-1\", \"amount\": 4200, \"sku\": \"sku-A\"}",
    "created_at": 1777387755
  }
}
--- orders-cancelled should be 0 ---
{
    "Attributes": {
        "ApproximateNumberOfMessages": "0"
    }
}
--- processed_events ---
[
    {
        "processed_at": {
            "N": "1777387757"
        },
        "event_id": {
            "S": "53c91c06-bacf-40af-ad0d-3c9f7d85c3a1"
        },
        "expires_at": {
            "N": "1777992557"
        }
    }
]
Round-trip works. Same `event_id` flowed end to end. Now the cancelled path:

Smoke test OrderCancelled path

shell
set -e
EP=http://localstack:4566
CREATED_URL=$(cat /app/build/orders_created_url)
CANCELLED_URL=$(cat /app/build/orders_cancelled_url)

cat > /app/build/p2.json <<'EOF'
{"order_id":"o-2","kind":"OrderCancelled","reason":"customer_request"}
EOF

aws --endpoint-url=$EP lambda invoke \
  --function-name order-api \
  --payload fileb:///app/build/p2.json \
  --cli-binary-format raw-in-base64-out \
  /app/build/order_api_resp2.json
cat /app/build/order_api_resp2.json
echo

DEADLINE=$(( $(date +%s) + 30 ))
GOT=0
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
  N=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
  echo "  orders-cancelled depth: $N"
  if [ "$N" -gt 0 ]; then GOT=1; break; fi
  sleep 2
done
[ $GOT -eq 1 ] || { echo "TIMEOUT"; exit 1; }

echo "--- orders-cancelled body ---"
aws --endpoint-url=$EP sqs receive-message --queue-url "$CANCELLED_URL" --max-number-of-messages 10 --wait-time-seconds 1 --visibility-timeout 1 | jq -r '.Messages[0].Body' | jq '{detail_type:."detail-type", source, event_id:.detail.event_id, order_id:.detail.order_id, kind:.detail.kind}'

echo "--- orders-created stays at 0 ---"
aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names ApproximateNumberOfMessages

echo "--- both event_ids in processed_events ---"
aws --endpoint-url=$EP dynamodb scan --table-name processed_events --query 'Items[].event_id.S'
output
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
{"statusCode": 200, "body": "{\"order_id\": \"o-2\", \"event_id\": \"37c251d4-b6d2-431b-af39-d9b962cfc301\", \"kind\": \"OrderCancelled\"}"}
  orders-cancelled depth: 0
  orders-cancelled depth: 1
--- orders-cancelled body ---
{
  "detail_type": "OrderCancelled",
  "source": "outbox.relay",
  "event_id": "37c251d4-b6d2-431b-af39-d9b962cfc301",
  "order_id": "o-2",
  "kind": "OrderCancelled"
}
--- orders-created stays at 0 ---
{
    "Attributes": {
        "ApproximateNumberOfMessages": "1"
    }
}
--- both event_ids in processed_events ---
[
    "37c251d4-b6d2-431b-af39-d9b962cfc301",
    "53c91c06-bacf-40af-ad0d-3c9f7d85c3a1"
]

[stdout]
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
{"statusCode": 200, "body": "{\"order_id\": \"o-2\", \"event_id\": \"37c251d4-b6d2-431b-af39-d9b962cfc301\", \"kind\": \"OrderCancelled\"}"}
  orders-cancelled depth: 0
  orders-cancelled depth: 1
--- orders-cancelled body ---
{
  "detail_type": "OrderCancelled",
  "source": "outbox.relay",
  "event_id": "37c251d4-b6d2-431b-af39-d9b962cfc301",
  "order_id": "o-2",
  "kind": "OrderCancelled"
}
--- orders-created stays at 0 ---
{
    "Attributes": {
        "ApproximateNumberOfMessages": "1"
    }
}
--- both event_ids in processed_events ---
[
    "37c251d4-b6d2-431b-af39-d9b962cfc301",
    "53c91c06-bacf-40af-ad0d-3c9f7d85c3a1"
]
The "1" in orders-created is the prior message that became visible again after my receive with visibility-timeout=1. Let me run the verifier-shaped sequence cleanly.

End-to-end verifier-shaped test

shell
set -e
EP=http://localstack:4566
CREATED_URL=$(cat /app/build/orders_created_url)
CANCELLED_URL=$(cat /app/build/orders_cancelled_url)

# Purge both
aws --endpoint-url=$EP sqs purge-queue --queue-url "$CREATED_URL"
aws --endpoint-url=$EP sqs purge-queue --queue-url "$CANCELLED_URL"
sleep 1

# Wipe outbox + dedup so event_ids start fresh-ish (purely for clean assert)
echo "before: dedup count = $(aws --endpoint-url=$EP dynamodb scan --table-name processed_events --query 'Count')"

cat > /app/build/p_a.json <<'EOF'
{"order_id":"o-1","kind":"OrderCreated","amount":4200}
EOF
aws --endpoint-url=$EP lambda invoke --function-name order-api \
  --payload fileb:///app/build/p_a.json --cli-binary-format raw-in-base64-out \
  /tmp/r1.json >/dev/null
EVENT_ID_A=$(jq -r '.body|fromjson|.event_id' /tmp/r1.json)
echo "Created event_id: $EVENT_ID_A"

# wait orders-created == 1
DEADLINE=$(( $(date +%s) + 30 ))
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
  N=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
  [ "$N" = "1" ] && break
  sleep 1
done
N_CREATED=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
N_CANCELLED=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
echo "After OrderCreated: orders-created=$N_CREATED, orders-cancelled=$N_CANCELLED"
[ "$N_CREATED" = "1" ] || { echo "FAIL: created should be 1"; exit 1; }
[ "$N_CANCELLED" = "0" ] || { echo "FAIL: cancelled should be 0"; exit 1; }

# dedup row exists for $EVENT_ID_A
DEDUP_ROW=$(aws --endpoint-url=$EP dynamodb get-item --table-name processed_events --key "{\"event_id\":{\"S\":\"$EVENT_ID_A\"}}" --query 'Item' --output json)
echo "dedup row for $EVENT_ID_A: $DEDUP_ROW"
[ "$DEDUP_ROW" != "null" ] || { echo "FAIL: no dedup row"; exit 1; }

# Now OrderCancelled with different order_id
cat > /app/build/p_b.json <<'EOF'
{"order_id":"o-2","kind":"OrderCancelled","reason":"customer_request"}
EOF
aws --endpoint-url=$EP lambda invoke --function-name order-api \
  --payload fileb:///app/build/p_b.json --cli-binary-format raw-in-base64-out \
  /tmp/r2.json >/dev/null
EVENT_ID_B=$(jq -r '.body|fromjson|.event_id' /tmp/r2.json)
echo "Cancelled event_id: $EVENT_ID_B"

DEADLINE=$(( $(date +%s) + 30 ))
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
  N=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
  [ "$N" = "1" ] && break
  sleep 1
done

N_CREATED=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
N_CANCELLED=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
echo "After OrderCancelled: orders-created=$N_CREATED, orders-cancelled=$N_CANCELLED"
[ "$N_CREATED" = "1" ] || { echo "FAIL: created should still be 1 (untouched)"; exit 1; }
[ "$N_CANCELLED" = "1" ] || { echo "FAIL: cancelled should be 1"; exit 1; }

echo "PASS"
output
before: dedup count = 2
Created event_id: 0a727360-d32c-4463-a40e-cfd387d8e1fe
After OrderCreated: orders-created=1, orders-cancelled=0
dedup row for 0a727360-d32c-4463-a40e-cfd387d8e1fe: {
    "processed_at": {
        "N": "1777387802"
    },
    "event_id": {
        "S": "0a727360-d32c-4463-a40e-cfd387d8e1fe"
    },
    "expires_at": {
        "N": "1777992602"
    }
}
Cancelled event_id: 48be0162-cba2-41ee-b66f-d825fae5ea1d
After OrderCancelled: orders-created=1, orders-cancelled=1
PASS

[stdout]
before: dedup count = 2
Created event_id: 0a727360-d32c-4463-a40e-cfd387d8e1fe
After OrderCreated: orders-created=1, orders-cancelled=0
dedup row for 0a727360-d32c-4463-a40e-cfd387d8e1fe: {
    "processed_at": {
        "N": "1777387802"
    },
    "event_id": {
        "S": "0a727360-d32c-4463-a40e-cfd387d8e1fe"
    },
    "expires_at": {
        "N": "1777992602"
    }
}
Cancelled event_id: 48be0162-cba2-41ee-b66f-d825fae5ea1d
After OrderCancelled: orders-created=1, orders-cancelled=1
PASS
Both scenarios pass. Let me also verify the redelivery / idempotency path and least-privilege isolation:

Verify dedup suppresses redelivery

shell
set -e
EP=http://localstack:4566
CREATED_URL=$(cat /app/build/orders_created_url)
CANCELLED_URL=$(cat /app/build/orders_cancelled_url)

# Re-invoke relay directly with the same outbox row to simulate a stream redelivery
# Pull most recent outbox row via scan
EVENT_ID=$(aws --endpoint-url=$EP dynamodb scan --table-name outbox --query 'Items[0].event_id.S' --output text)
ORDER_ID=$(aws --endpoint-url=$EP dynamodb scan --table-name outbox --query 'Items[0].order_id.S' --output text)
KIND=$(aws --endpoint-url=$EP dynamodb scan --table-name outbox --query 'Items[0].kind.S' --output text)
echo "Replaying event_id=$EVENT_ID order_id=$ORDER_ID kind=$KIND"

cat > /app/build/replay.json <<EOF
{"Records":[{"eventID":"replay-1","eventName":"INSERT","dynamodb":{"SequenceNumber":"replay-seq-1","NewImage":{"event_id":{"S":"$EVENT_ID"},"order_id":{"S":"$ORDER_ID"},"kind":{"S":"$KIND"},"payload":{"S":"{}"},"created_at":{"N":"1"}}}}]}
EOF

DEPTH_BEFORE_C=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
DEPTH_BEFORE_X=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)

aws --endpoint-url=$EP lambda invoke \
  --function-name outbox-relay \
  --payload fileb:///app/build/replay.json \
  --cli-binary-format raw-in-base64-out \
  /tmp/relay_replay.json >/dev/null
echo "Relay replay response: $(cat /tmp/relay_replay.json)"
sleep 3

DEPTH_AFTER_C=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CREATED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
DEPTH_AFTER_X=$(aws --endpoint-url=$EP sqs get-queue-attributes --queue-url "$CANCELLED_URL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
echo "before: created=$DEPTH_BEFORE_C cancelled=$DEPTH_BEFORE_X"
echo "after:  created=$DEPTH_AFTER_C cancelled=$DEPTH_AFTER_X"
[ "$DEPTH_BEFORE_C" = "$DEPTH_AFTER_C" ] && [ "$DEPTH_BEFORE_X" = "$DEPTH_AFTER_X" ] && echo "PASS: redelivery suppressed"
output
Replaying event_id=48be0162-cba2-41ee-b66f-d825fae5ea1d order_id=o-2 kind=OrderCancelled
Relay replay response: {"batchItemFailures": []}
before: created=1 cancelled=1
after:  created=1 cancelled=1
PASS: redelivery suppressed

[stdout]
Replaying event_id=48be0162-cba2-41ee-b66f-d825fae5ea1d order_id=o-2 kind=OrderCancelled
Relay replay response: {"batchItemFailures": []}
before: created=1 cancelled=1
after:  created=1 cancelled=1
PASS: redelivery suppressed

Verify least-privilege policy isolation

shell
set -e
EP=http://localstack:4566
echo "=== order-api role policy actions ==="
aws --endpoint-url=$EP iam get-role-policy --role-name order-api-role --policy-name order-api-inline --query 'PolicyDocument.Statement[].{Sid:Sid,Action:Action,Resource:Resource}'
echo "=== outbox-relay role policy actions ==="
aws --endpoint-url=$EP iam get-role-policy --role-name outbox-relay-role --policy-name outbox-relay-inline --query 'PolicyDocument.Statement[].{Sid:Sid,Action:Action,Resource:Resource}'

echo "=== Spot check: neither role uses '*' on KMS ==="
for r in order-api-role outbox-relay-role; do
  STAR=$(aws --endpoint-url=$EP iam get-role-policy --role-name $r --policy-name ${r%-role}-inline --query "PolicyDocument.Statement[?contains(to_string(Action), 'kms')].Resource" --output json 2>/dev/null || true)
  echo "$r kms resources: $STAR"
done

echo "=== order-api role: has SQS perm? ==="
aws --endpoint-url=$EP iam get-role-policy --role-name order-api-role --policy-name order-api-inline --query "PolicyDocument.Statement[?contains(to_string(Action), 'sqs')]"
echo "=== outbox-relay role: has orders-table perm? ==="
aws --endpoint-url=$EP iam get-role-policy --role-name outbox-relay-role --policy-name outbox-relay-inline --query "PolicyDocument.Statement[?contains(to_string(Resource), 'table/orders')]"
output
=== order-api role policy actions ===
[
    {
        "Sid": "TransactWriteOrdersAndOutbox",
        "Action": [
            "dynamodb:PutItem",
            "dynamodb:UpdateItem",
            "dynamodb:DescribeTable",
            "dynamodb:TransactWriteItems"
        ],
        "Resource": [
            "arn:aws:dynamodb:us-east-1:000000000000:table/orders",
            "arn:aws:dynamodb:us-east-1:000000000000:table/outbox"
        ]
    },
    {
        "Sid": "KmsForDynamoDb",
        "Action": [
            "kms:GenerateDataKey",
            "kms:Decrypt",
            "kms:DescribeKey"
        ],
        "Resource": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
    },
    {
        "Sid": "Logs",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": "*"
    }
]
=== outbox-relay role policy actions ===
[
    {
        "Sid": "ReadOutboxStream",
        "Action": [
            "dynamodb:DescribeStream",
            "dynamodb:GetRecords",
            "dynamodb:GetShardIterator",
            "dynamodb:ListStreams"
        ],
        "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187"
    },
    {
        "Sid": "DedupTable",
        "Action": [
            "dynamodb:PutItem",
            "dynamodb:GetItem",
            "dynamodb:DeleteItem"
        ],
        "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/processed_events"
    },
    {
        "Sid": "PutEventsToBus",
        "Action": "events:PutEvents",
        "Resource": "arn:aws:events:us-east-1:000000000000:event-bus/order-events"
    },
    {
        "Sid": "KmsDecryptForCmk",
        "Action": [
            "kms:Decrypt",
            "kms:DescribeKey",
            "kms:GenerateDataKey"
        ],
        "Resource": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
    },
    {
        "Sid": "Logs",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": "*"
    }
]
=== Spot check: neither role uses '*' on KMS ===
order-api-role kms resources: [
    "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
]
outbox-relay-role kms resources: [
    "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
]
=== order-api role: has SQS perm? ===
[]
=== outbox-relay role: has orders-table perm? ===
[]

[stdout]
=== order-api role policy actions ===
[
    {
        "Sid": "TransactWriteOrdersAndOutbox",
        "Action": [
            "dynamodb:PutItem",
            "dynamodb:UpdateItem",
            "dynamodb:DescribeTable",
            "dynamodb:TransactWriteItems"
        ],
        "Resource": [
            "arn:aws:dynamodb:us-east-1:000000000000:table/orders",
            "arn:aws:dynamodb:us-east-1:000000000000:table/outbox"
        ]
    },
    {
        "Sid": "KmsForDynamoDb",
        "Action": [
            "kms:GenerateDataKey",
            "kms:Decrypt",
            "kms:DescribeKey"
        ],
        "Resource": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
    },
    {
        "Sid": "Logs",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": "*"
    }
]
=== outbox-relay role policy actions ===
[
    {
        "Sid": "ReadOutboxStream",
        "Action": [
            "dynamodb:DescribeStream",
            "dynamodb:GetRecords",
            "dynamodb:GetShardIterator",
            "dynamodb:ListStreams"
        ],
        "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/outbox/stream/2026-04-28T14:45:43.187"
    },
    {
        "Sid": "DedupTable",
        "Action": [
            "dynamodb:PutItem",
            "dynamodb:GetItem",
            "dynamodb:DeleteItem"
        ],
        "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/processed_events"
    },
    {
        "Sid": "PutEventsToBus",
        "Action": "events:PutEvents",
        "Resource": "arn:aws:events:us-east-1:000000000000:event-bus/order-events"
    },
    {
        "Sid": "KmsDecryptForCmk",
        "Action": [
            "kms:Decrypt",
            "kms:DescribeKey",
            "kms:GenerateDataKey"
        ],
        "Resource": "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
    },
    {
        "Sid": "Logs",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": "*"
    }
]
=== Spot check: neither role uses '*' on KMS ===
order-api-role kms resources: [
    "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
]
outbox-relay-role kms resources: [
    "arn:aws:kms:us-east-1:000000000000:key/7c1f56dc-9476-4b76-ba8c-61d9285b06e9"
]
=== order-api role: has SQS perm? ===
[]
=== outbox-relay role: has orders-table perm? ===
[]
  • Create KMS CMK + alias
  • Create DynamoDB tables (orders, outbox w/ stream, processed_events w/ TTL)
  • Create SQS queues + DLQs encrypted with CMK
  • Create EventBridge bus + rules
  • Build IAM roles with least-privilege policies
  • Package and deploy lambdas
  • Wire stream event source mapping with ReportBatchItemFailures
  • End-to-end smoke test (created + cancelled paths)
All wired up and passing. Summary of what's deployed on `http://localstack:4566`: **Infra** - KMS CMK with `alias/outbox-cmk`; key policy allows root + EventBridge/SQS/DDB services for service-side encryption. - DynamoDB: `orders` (pk `order_id`), `outbox` (pk `event_id`, stream `NEW_AND_OLD_IMAGES`), `processed_events` (pk `event_id`, TTL on `expires_at`). All three encrypted with the CMK. - SQS: `orders-created` / `orders-cancelled` with their own DLQs `orders-created-dlq` / `orders-cancelled-dlq`. Each queue uses the CMK; main queues redrive to their own DLQ at `maxReceiveCount=5`. Queue policies allow `events.amazonaws.com` to `sqs:SendMessage`. - EventBridge bus `order-events`, with rules `on-order-created` / `on-order-cancelled` matching `source = "outbox.relay"` and `detail-type` of `OrderCreated` / `OrderCancelled` respectively. **Lambdas (python3.11, AWS_ENDPOINT_URL=http://localstack:4566)** - `order-api` (`/app/lambdas/order_api.py`): validates `kind`, generates `event_id`, single `TransactWriteItems` writing to both tables. - `outbox-relay` (`/app/lambdas/outbox_relay.py`): conditional `PutItem` on `processed_events` for dedup, `PutEvents` to the bus with `Source=outbox.relay` and `DetailType=kind`. On publish failure it deletes the dedup row and returns `{batchItemFailures:[{itemIdentifier: <SequenceNumber>}]}`. - Event source mapping: outbox stream → `outbox-relay` with `FunctionResponseTypes=[ReportBatchItemFailures]`. **Least privilege (verified by reading policy docs)** - `order-api-role` only: `dynamodb:PutItem|UpdateItem|DescribeTable|TransactWriteItems` on the two table ARNs, `kms:GenerateDataKey|Decrypt|DescribeKey` on the CMK ARN with `kms:ViaService = dynamodb.us-east-1.amazonaws.com`, plus logs. No SQS, no streams, no events. - `outbox-relay-role` only: stream actions on the **outbox stream ARN** (not orders), put/get/delete on `processed_events`, `events:PutEvents` on the named bus, KMS scoped to the CMK ARN, plus logs. No access to the `orders` table or its stream. **Smoke test results** - `OrderCreated` for `o-1` → `orders-created` got 1 message, `orders-cancelled` 0, `processed_events` row present with the same `event_id` returned by `order-api`. - `OrderCancelled` for `o-2` → `orders-cancelled` got 1 message, `orders-created` untouched. - Replaying the same stream record → relay returned `{"batchItemFailures": []}` and queue depths did not change (dedup suppressed it).
Final diff , what the agent changed

No file changes, the agent left the workspace untouched.

Verifier output , the proof behind the reward
tests/test.sh
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/jGQG4BAookO5wB-x_iLzk/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 20 items

../tests/test_outputs.py::test_01_cmk_alias_exists_and_enabled PASSED    [  5%]
../tests/test_outputs.py::test_02_orders_table_exists PASSED             [ 10%]
../tests/test_outputs.py::test_03_outbox_table_exists PASSED             [ 15%]
../tests/test_outputs.py::test_04_processed_events_table_exists PASSED   [ 20%]
../tests/test_outputs.py::test_05_custom_event_bus_exists PASSED         [ 25%]
../tests/test_outputs.py::test_06_orders_table_encrypted_with_cmk PASSED [ 30%]
../tests/test_outputs.py::test_07_outbox_table_encrypted_with_cmk PASSED [ 35%]
../tests/test_outputs.py::test_08_outbox_stream_is_new_and_old_images PASSED [ 40%]
../tests/test_outputs.py::test_09_processed_events_ttl_enabled PASSED    [ 45%]
../tests/test_outputs.py::test_10_both_rule_patterns_use_array_wrap_and_disjoint_types PASSED [ 50%]
../tests/test_outputs.py::test_11_each_rule_targets_only_its_intended_queue PASSED [ 55%]
../tests/test_outputs.py::test_12_both_main_queues_encrypted_with_cmk PASSED [ 60%]
../tests/test_outputs.py::test_13_main_queues_have_redrive_to_distinct_dlqs PASSED [ 65%]
../tests/test_outputs.py::test_14_queue_policies_scope_events_with_source_arn FAILED [ 70%]
../tests/test_outputs.py::test_15_relay_esm_has_batch_item_failures_on_outbox_stream PASSED [ 75%]
../tests/test_outputs.py::test_16_relay_role_is_scoped_not_wildcard PASSED [ 80%]
../tests/test_outputs.py::test_17_order_api_role_scoped_to_orders_and_outbox_only PASSED [ 85%]
../tests/test_outputs.py::test_18_e2e_order_created_routes_to_created_queue_only PASSED [ 90%]
../tests/test_outputs.py::test_19_e2e_order_cancelled_routes_to_cancelled_queue_only PASSED [ 95%]
../tests/test_outputs.py::test_20_e2e_idempotent_duplicate_does_not_double_fanout PASSED [100%]

=================================== FAILURES ===================================
_____________ test_14_queue_policies_scope_events_with_source_arn ______________

    def test_14_queue_policies_scope_events_with_source_arn():
        """Each main queue's resource policy must allow events.amazonaws.com
        principal ONLY when aws:SourceArn matches the rule ARN for that queue.
        This is the classic 'confused-deputy' protection that LLMs typically
        either omit or wildcard."""
        sqs = _client("sqs")
        mapping = [
            (QUEUE_CREATED, RULE_CREATED),
            (QUEUE_CANCELLED, RULE_CANCELLED),
        ]
        for qname, rule_name in mapping:
            url = _queue_url(qname)
            attrs = sqs.get_queue_attributes(QueueUrl=url, AttributeNames=["Policy"])[
                "Attributes"
            ]
            policy_str = attrs.get("Policy")
            assert policy_str, f"{qname} missing resource policy"
            policy = json.loads(policy_str)
            stmts = policy.get("Statement", [])
            if isinstance(stmts, dict):
                stmts = [stmts]
    
            matched = False
            for s in stmts:
                if s.get("Effect") != "Allow":
                    continue
                principal = s.get("Principal", {})
                svc = principal.get("Service") if isinstance(principal, dict) else None
                svc_list = _as_list(svc)
                if "events.amazonaws.com" not in svc_list:
                    continue
                actions = _as_list(s.get("Action"))
                if not any("SendMessage" in a or a == "sqs:*" for a in actions):
                    continue
                cond = s.get("Condition", {})
                arn_like = cond.get("ArnEquals") or cond.get("ArnLike") or {}
                src_arn = arn_like.get("aws:SourceArn")
                src_list = _as_list(src_arn)
                if any(rule_name in v for v in src_list):
                    matched = True
                    break
>           assert matched, (
                f"{qname} resource policy must include an Allow for "
                f"events.amazonaws.com scoped by aws:SourceArn containing "
                f"rule '{rule_name}' - confused-deputy protection"
            )
E           AssertionError: orders-created resource policy must include an Allow for events.amazonaws.com scoped by aws:SourceArn containing rule 'on-order-created' - confused-deputy protection
E           assert False

/tests/test_outputs.py:395: AssertionError
=============================== warnings summary ===============================
test_outputs.py: 88 warnings
  /root/.cache/uv/archive-v0/jGQG4BAookO5wB-x_iLzk/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_outputs.py::test_01_cmk_alias_exists_and_enabled
PASSED ../tests/test_outputs.py::test_02_orders_table_exists
PASSED ../tests/test_outputs.py::test_03_outbox_table_exists
PASSED ../tests/test_outputs.py::test_04_processed_events_table_exists
PASSED ../tests/test_outputs.py::test_05_custom_event_bus_exists
PASSED ../tests/test_outputs.py::test_06_orders_table_encrypted_with_cmk
PASSED ../tests/test_outputs.py::test_07_outbox_table_encrypted_with_cmk
PASSED ../tests/test_outputs.py::test_08_outbox_stream_is_new_and_old_images
PASSED ../tests/test_outputs.py::test_09_processed_events_ttl_enabled
PASSED ../tests/test_outputs.py::test_10_both_rule_patterns_use_array_wrap_and_disjoint_types
PASSED ../tests/test_outputs.py::test_11_each_rule_targets_only_its_intended_queue
PASSED ../tests/test_outputs.py::test_12_both_main_queues_encrypted_with_cmk
PASSED ../tests/test_outputs.py::test_13_main_queues_have_redrive_to_distinct_dlqs
PASSED ../tests/test_outputs.py::test_15_relay_esm_has_batch_item_failures_on_outbox_stream
PASSED ../tests/test_outputs.py::test_16_relay_role_is_scoped_not_wildcard
PASSED ../tests/test_outputs.py::test_17_order_api_role_scoped_to_orders_and_outbox_only
PASSED ../tests/test_outputs.py::test_18_e2e_order_created_routes_to_created_queue_only
PASSED ../tests/test_outputs.py::test_19_e2e_order_cancelled_routes_to_cancelled_queue_only
PASSED ../tests/test_outputs.py::test_20_e2e_idempotent_duplicate_does_not_double_fanout
FAILED ../tests/test_outputs.py::test_14_queue_policies_scope_events_with_source_arn
================== 1 failed, 19 passed, 88 warnings in 27.25s ==================

Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_3908dea9d2524166. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.

Trial trial_3908dea9d2524166 · verifier authoritative; classifier explanatory.