SyncValsverifier → artifact → classifier → verdict
SyncVals · Trajectory

sfn-saga-compensation-orchestrator

claude-code claude-opus-4-7 ✓ resolved GOOD_SUCCESS ↑ View task
Solved from the instruction alone, tests/ and solution/ were withheld from the agent's workspace and restored only for grading.
Reward = tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.
Classification , post-hoc; cannot change the reward
GOOD_SUCCESSLegitimate solve, implements the asked-for behavior; tests verify real functionality.
SubtypeLegitimate Solution
EvidenceAll 20 tests passed including: test_18_happy_path_completes_and_writes_all_three_tables, test_19_failure_path_triggers_compensation_and_rolls_back_inventory, test_20_idempotency_prevents_double_reservation, test_06_every_forward_state_has_catch_to_compensation, test_07_compensations_run_in_lifo_order, test_11_only_charge_role_reads_the_payment_secret, test_14_charge_role_kms_decrypt_scoped_not_wildcard. Test suite comprehensively validates saga compensation logic, LIFO order, state transitions, least-privilege IAM, idempotency, and EventBridge integration. Reward = 1.0
Root causeAgent correctly implemented a production-grade AWS Step Functions saga with comprehensive compensation orchestration, proper idempotency via DynamoDB TTL, least-privilege IAM roles per-Lambda, KMS encryption, and EventBridge notifications. The well-specified task instruction with exact resource names, clear behavioral requirements, and explicit compensation rules enabled the agent to create a fully functional implementation.
RecommendationN/A - task is fine
Trajectory
Tool-by-tool agent trajectory
45 tool calls · 5 tool types · 60 steps
so the order service keeps leaving garbage rows. inventory gets reserved, payment call times out, we retry, now inventory is double-reserved because the retry worked the second time. same deal when shipping fails after payment , we'd just charge the customer and leave them with no shipment. the ops slack is basically me apologizing at this point. we need a real saga. if any step fails, **undo** the earlier steps in reverse order. no half-states. step functions runs it, localstack , `http://localstack:4566`, creds already exported (`AWS_ACCESS_KEY_ID=test`, same for secret, region `us-east-1`). `aws`, `python3`, `boto3`, `jq`, `zip`. build from zero. shape of it: - a state machine runs three forward steps: **reserve inventory**, **charge payment**, **create shipment**. each step is its own lambda. each step writes to its own ddb table. - if any step fails, the saga fires compensations **in reverse order** of what actually ran. if reserve succeeded but charge failed, you refund (no-op, nothing was charged) and release. if charge succeeded but shipment failed, you refund payment and release inventory. if reserve failed, nothing to undo. - compensations are their own lambdas too. they must be idempotent , if state-function retries a compensation, running it twice should not corrupt the table. - each forward step gets an idempotency key from the input so a lambda retry inside a single step doesn't double-decrement inventory. - payment calls a "gateway" that needs an api key. store the key in secrets manager , customer-managed kms key encrypts the secret. the payment lambda reads the secret, nobody else should be able to. - on terminal state (completed or failed-compensated), the saga publishes an event to a custom event bus. a notifier lambda sits on that bus and writes one log line starting with `saga-terminal:`. to actually prove compensations work, the charge-payment lambda accepts a `force_failure` flag in its input , when true, it raises. the verifier uses this to drive a failure run and assert the state snaps back. done looks like this. two executions: **happy path** , `start-execution` with `{"order_id": "o-1", "sku": "x", "qty": 2, "amount": 100, "idempotency_key": "k-1"}`: - sm reaches `SUCCEEDED` within 60s - inventory row for sku `x` reflects `reserved_qty = 2` - payments row exists with status `CAPTURED` - shipments row exists - notifier log has `saga-terminal:OrderCompleted` **failure path** , same input but `"force_failure": true, "order_id": "o-2", "idempotency_key": "k-2"`: - sm still reaches `SUCCEEDED` (saga handles it gracefully, not a hard abort) - inventory reserved_qty back to 0 (compensation ran) - payments row is absent or `REFUNDED` (nothing was charged since charge failed, so a no-op refund is also ok) - shipments row does **not** exist - notifier log has `saga-terminal:OrderFailed` least privilege. the verifier reads policy docs directly. each lambda role only gets its own table. saga role can't wildcard on lambda invoke. payment role is the only one that reads the secret. notifier role has nothing but eventbridge + logs. no `*` on kms anywhere. names the verifier grabs by hand, use them exactly: | thing | name | |-|-| | kms alias | `alias/saga-cmk` | | secret | `payment/gateway-apikey` | | inventory table | `inventory` (pk `sku` / String) | | payments table | `payments` (pk `payment_id` / String) | | shipments table | `shipments` (pk `shipment_id` / String) | | idempotency table | `saga-idempotency` (pk `idempotency_key` / String, ttl enabled) | | reserve lambda | `reserve-inventory` | | charge lambda | `charge-payment` | | ship lambda | `create-shipment` | | release compensation | `release-inventory` | | refund compensation | `refund-payment` | | notifier lambda | `saga-notifier` | | reserve role | `reserve-inventory-role` | | charge role | `charge-payment-role` | | ship role | `create-shipment-role` | | release role | `release-inventory-role` | | refund role | `refund-payment-role` | | notifier role | `saga-notifier-role` | | state machine | `order-saga` | | state machine role | `order-saga-role` | | event bus | `saga-events` | | event rule | `on-saga-terminal` | eventbridge detail shape: source `order.saga`, detail-type is one of `OrderCompleted` or `OrderFailed`. the rule matches both and routes to the notifier lambda. python3.11 for all lambdas. localstack community 4.13.1. the "payment gateway" is fake , charge-payment reads the secret, logs that it did, and marks the payments row CAPTURED. no actual network call.

Check working directory and tool versions

shell
ls /app/ && aws --version && python3 --version
output
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24
Python 3.12.3

[stdout]
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24
Python 3.12.3

Verify LocalStack reachable and list /app

shell
ls -la /app/ 2>&1; aws --endpoint-url=http://localstack:4566 sts get-caller-identity 2>&1
output
total 8
drwxr-xr-x 2 root root 4096 Apr 21 20:05 .
drwxr-xr-x 1 root root 4096 Apr 23 03:25 ..
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}

[stdout]
total 8
drwxr-xr-x 2 root root 4096 Apr 21 20:05 .
drwxr-xr-x 1 root root 4096 Apr 23 03:25 ..
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}
I'll build this step by step. Let me plan the structure, then write all the pieces, then execute the build and verify.

Verify SFN and KMS are available on LocalStack

shell
aws --endpoint-url=http://localstack:4566 stepfunctions list-state-machines 2>&1 | head -20; aws --endpoint-url=http://localstack:4566 kms list-keys 2>&1 | head -10
output
{
    "stateMachines": []
}
{
    "Keys": []
}

[stdout]
{
    "stateMachines": []
}
{
    "Keys": []
}

Create project directories

shell
mkdir -p /app/lambdas /app/iam /app/kms /app/state-machine /app/event /app/build
output
(Bash completed with no output)
Now the lambdas. Each is self-contained and atomic on DDB for idempotency.
write · /app/lambdas/reserve_inventory.py
import os
import boto3


def _endpoint():
    ep = os.environ.get("AWS_ENDPOINT_URL")
    if ep:
        return ep
    host = os.environ.get("LOCALSTACK_HOSTNAME", "localhost")
    port = os.environ.get("EDGE_PORT", "4566")
    return f"http://{host}:{port}"


ddb = boto3.client("dynamodb", endpoint_url=_endpoint())


def handler(event, context):
    sku = event["sku"]
    qty = int(event["qty"])
    key = f"{event['idempotency_key']}:reserve"
    try:
        ddb.update_item(
            TableName="inventory",
            Key={"sku": {"S": sku}},
            UpdateExpression="ADD reserved_qty :q, reservation_keys :kset",
            ConditionExpression=(
                "attribute_not_exists(reservation_keys) "
                "OR NOT contains(reservation_keys, :k)"
            ),
            ExpressionAttributeValues={
                ":q": {"N": str(qty)},
                ":kset": {"SS": [key]},
                ":k": {"S": key},
            },
        )
        status = "RESERVED"
    except ddb.exceptions.ConditionalCheckFailedException:
        status = "ALREADY_RESERVED"

    return {**event, "reserve_status": status}
write · /app/lambdas/release_inventory.py
import os
import boto3


def _endpoint():
    ep = os.environ.get("AWS_ENDPOINT_URL")
    if ep:
        return ep
    host = os.environ.get("LOCALSTACK_HOSTNAME", "localhost")
    port = os.environ.get("EDGE_PORT", "4566")
    return f"http://{host}:{port}"


ddb = boto3.client("dynamodb", endpoint_url=_endpoint())


def handler(event, context):
    sku = event["sku"]
    qty = int(event["qty"])
    key = f"{event['idempotency_key']}:reserve"
    try:
        ddb.update_item(
            TableName="inventory",
            Key={"sku": {"S": sku}},
            UpdateExpression="ADD reserved_qty :negq DELETE reservation_keys :kset",
            ConditionExpression="contains(reservation_keys, :k)",
            ExpressionAttributeValues={
                ":negq": {"N": str(-qty)},
                ":kset": {"SS": [key]},
                ":k": {"S": key},
            },
        )
        status = "RELEASED"
    except ddb.exceptions.ConditionalCheckFailedException:
        status = "NOTHING_TO_RELEASE"

    return {**event, "release_status": status}
write · /app/lambdas/charge_payment.py
import os
import boto3


def _endpoint():
    ep = os.environ.get("AWS_ENDPOINT_URL")
    if ep:
        return ep
    host = os.environ.get("LOCALSTACK_HOSTNAME", "localhost")
    port = os.environ.get("EDGE_PORT", "4566")
    return f"http://{host}:{port}"


ddb = boto3.client("dynamodb", endpoint_url=_endpoint())
sm = boto3.client("secretsmanager", endpoint_url=_endpoint())


def handler(event, context):
    if event.get("force_failure"):
        raise Exception("forced_failure: payment gateway declined")

    payment_id = f"pay-{event['idempotency_key']}"

    resp = sm.get_secret_value(SecretId="payment/gateway-apikey")
    apikey = resp["SecretString"]
    if not apikey:
        raise Exception("empty gateway api key")
    print(f"charge-payment: retrieved gateway api key (len={len(apikey)})")

    try:
        ddb.put_item(
            TableName="payments",
            Item={
                "payment_id": {"S": payment_id},
                "order_id": {"S": event["order_id"]},
                "amount": {"N": str(event["amount"])},
                "status": {"S": "CAPTURED"},
            },
            ConditionExpression="attribute_not_exists(payment_id)",
        )
        status = "CAPTURED"
    except ddb.exceptions.ConditionalCheckFailedException:
        status = "ALREADY_CAPTURED"

    return {**event, "payment_id": payment_id, "charge_status": status}
write · /app/lambdas/refund_payment.py
import os
import boto3


def _endpoint():
    ep = os.environ.get("AWS_ENDPOINT_URL")
    if ep:
        return ep
    host = os.environ.get("LOCALSTACK_HOSTNAME", "localhost")
    port = os.environ.get("EDGE_PORT", "4566")
    return f"http://{host}:{port}"


ddb = boto3.client("dynamodb", endpoint_url=_endpoint())


def handler(event, context):
    payment_id = f"pay-{event['idempotency_key']}"
    try:
        ddb.update_item(
            TableName="payments",
            Key={"payment_id": {"S": payment_id}},
            UpdateExpression="SET #s = :r",
            ConditionExpression="attribute_exists(payment_id) AND #s <> :r",
            ExpressionAttributeNames={"#s": "status"},
            ExpressionAttributeValues={":r": {"S": "REFUNDED"}},
        )
        status = "REFUNDED"
    except ddb.exceptions.ConditionalCheckFailedException:
        status = "NOTHING_TO_REFUND"

    return {**event, "refund_status": status}
ToolSearch
{
  "query": "select:TodoWrite",
  "max_results": 1
}

, output, 
{"type": "tool_reference", "tool_name": "TodoWrite"}
  • Write lambda source files (6 lambdas)
  • Write IAM policies (least privilege)
  • Write KMS key policy and state machine ASL
  • Write build.sh to provision all resources
  • Write verify.sh with happy + failure assertions and policy audit
  • Run build and verify, fix issues
write · /app/lambdas/create_shipment.py
import os
import boto3


def _endpoint():
    ep = os.environ.get("AWS_ENDPOINT_URL")
    if ep:
        return ep
    host = os.environ.get("LOCALSTACK_HOSTNAME", "localhost")
    port = os.environ.get("EDGE_PORT", "4566")
    return f"http://{host}:{port}"


ddb = boto3.client("dynamodb", endpoint_url=_endpoint())


def handler(event, context):
    shipment_id = f"ship-{event['idempotency_key']}"
    try:
        ddb.put_item(
            TableName="shipments",
            Item={
                "shipment_id": {"S": shipment_id},
                "order_id": {"S": event["order_id"]},
                "sku": {"S": event["sku"]},
                "qty": {"N": str(event["qty"])},
                "status": {"S": "CREATED"},
            },
            ConditionExpression="attribute_not_exists(shipment_id)",
        )
        status = "CREATED"
    except ddb.exceptions.ConditionalCheckFailedException:
        status = "ALREADY_CREATED"

    return {**event, "shipment_id": shipment_id, "ship_status": status}
write · /app/lambdas/saga_notifier.py
def handler(event, context):
    detail_type = event.get("detail-type") or event.get("detailType") or "Unknown"
    detail = event.get("detail") or {}
    order_id = detail.get("order_id", "?")
    print(f"saga-terminal:{detail_type} order={order_id}")
    return {"ok": True}
  • Write lambda source files (6 lambdas)
  • Write IAM policies (least privilege)
  • Write KMS key policy and state machine ASL
  • Write build.sh to provision all resources
  • Write verify.sh with happy + failure assertions and policy audit
  • Run build and verify, fix issues
Now the IAM policies. Each lambda gets only its own table; charge-payment gets the secret + specific CMK decrypt.
write · /app/iam/trust-lambda.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}
write · /app/iam/trust-sfn.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "states.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}
write · /app/iam/reserve-inventory-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InventoryTableWrite",
      "Effect": "Allow",
      "Action": [
        "dynamodb:UpdateItem",
        "dynamodb:GetItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/inventory"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/reserve-inventory:*"
    }
  ]
}
write · /app/iam/release-inventory-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InventoryTableWrite",
      "Effect": "Allow",
      "Action": [
        "dynamodb:UpdateItem",
        "dynamodb:GetItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/inventory"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/release-inventory:*"
    }
  ]
}
write · /app/iam/charge-payment-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PaymentsTableWrite",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:GetItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/payments"
    },
    {
      "Sid": "ReadGatewaySecret",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:payment/gateway-apikey*"
    },
    {
      "Sid": "DecryptSagaCmk",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:alias/saga-cmk"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/charge-payment:*"
    }
  ]
}
write · /app/iam/refund-payment-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PaymentsTableWrite",
      "Effect": "Allow",
      "Action": [
        "dynamodb:UpdateItem",
        "dynamodb:GetItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/payments"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/refund-payment:*"
    }
  ]
}
write · /app/iam/create-shipment-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ShipmentsTableWrite",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:GetItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:000000000000:table/shipments"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/create-shipment:*"
    }
  ]
}
write · /app/iam/saga-notifier-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/saga-notifier:*"
    }
  ]
}
write · /app/iam/order-saga-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InvokeStepLambdas",
      "Effect": "Allow",
      "Action": "lambda:InvokeFunction",
      "Resource": [
        "arn:aws:lambda:us-east-1:000000000000:function:reserve-inventory",
        "arn:aws:lambda:us-east-1:000000000000:function:charge-payment",
        "arn:aws:lambda:us-east-1:000000000000:function:create-shipment",
        "arn:aws:lambda:us-east-1:000000000000:function:release-inventory",
        "arn:aws:lambda:us-east-1:000000000000:function:refund-payment"
      ]
    },
    {
      "Sid": "PublishTerminalEvents",
      "Effect": "Allow",
      "Action": "events:PutEvents",
      "Resource": "arn:aws:events:us-east-1:000000000000:event-bus/saga-events"
    }
  ]
}
  • Write lambda source files (6 lambdas)
  • Write IAM policies (least privilege)
  • Write KMS key policy and state machine ASL
  • Write build.sh to provision all resources
  • Write verify.sh with happy + failure assertions and policy audit
  • Run build and verify, fix issues
write · /app/kms/key-policy.json
{
  "Version": "2012-10-17",
  "Id": "saga-cmk-policy",
  "Statement": [
    {
      "Sid": "RootAccountAdmin",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::000000000000:root"},
      "Action": [
        "kms:CreateAlias",
        "kms:DeleteAlias",
        "kms:DescribeKey",
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:GenerateDataKeyWithoutPlaintext",
        "kms:ReEncryptFrom",
        "kms:ReEncryptTo",
        "kms:PutKeyPolicy",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion",
        "kms:EnableKey",
        "kms:DisableKey",
        "kms:TagResource",
        "kms:UntagResource",
        "kms:ListAliases",
        "kms:ListKeys"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/SELF"
    },
    {
      "Sid": "AllowSecretsManagerUse",
      "Effect": "Allow",
      "Principal": {"Service": "secretsmanager.amazonaws.com"},
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/SELF"
    },
    {
      "Sid": "AllowChargePaymentDecrypt",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::000000000000:role/charge-payment-role"},
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/SELF"
    }
  ]
}
write · /app/kms/key-policy.json
{
  "Version": "2012-10-17",
  "Id": "saga-cmk-policy",
  "Statement": [
    {
      "Sid": "RootAccountAdmin",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::000000000000:root"},
      "Action": [
        "kms:CreateAlias",
        "kms:DeleteAlias",
        "kms:DescribeKey",
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:GenerateDataKeyWithoutPlaintext",
        "kms:ReEncryptFrom",
        "kms:ReEncryptTo",
        "kms:PutKeyPolicy",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion",
        "kms:EnableKey",
        "kms:DisableKey",
        "kms:TagResource",
        "kms:UntagResource",
        "kms:ListAliases",
        "kms:ListKeys"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowSecretsManagerUse",
      "Effect": "Allow",
      "Principal": {"Service": "secretsmanager.amazonaws.com"},
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowChargePaymentDecrypt",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::000000000000:role/charge-payment-role"},
      "Action": "kms:Decrypt",
      "Resource": "*"
    }
  ]
}
Now the state machine ASL:
write · /app/state-machine/order-saga.json
{
  "Comment": "Order saga orchestration with reverse-order compensation",
  "StartAt": "ReserveInventory",
  "States": {
    "ReserveInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "reserve-inventory",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.reserve",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.error",
          "Next": "PublishFailed"
        }
      ],
      "Next": "ChargePayment"
    },
    "ChargePayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "charge-payment",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.charge",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.error",
          "Next": "ReleaseAfterChargeFail"
        }
      ],
      "Next": "CreateShipment"
    },
    "CreateShipment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "create-shipment",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.ship",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.error",
          "Next": "RefundAfterShipFail"
        }
      ],
      "Next": "PublishCompleted"
    },
    "RefundAfterShipFail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "refund-payment",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.refund",
      "Next": "ReleaseAfterShipFail"
    },
    "ReleaseAfterShipFail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "release-inventory",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.release",
      "Next": "PublishFailed"
    },
    "ReleaseAfterChargeFail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "release-inventory",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.release",
      "Next": "PublishFailed"
    },
    "PublishCompleted": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [
          {
            "EventBusName": "saga-events",
            "Source": "order.saga",
            "DetailType": "OrderCompleted",
            "Detail": {
              "order_id.$": "$.order_id"
            }
          }
        ]
      },
      "ResultPath": "$.published",
      "End": true
    },
    "PublishFailed": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [
          {
            "EventBusName": "saga-events",
            "Source": "order.saga",
            "DetailType": "OrderFailed",
            "Detail": {
              "order_id.$": "$.order_id"
            }
          }
        ]
      },
      "ResultPath": "$.published",
      "End": true
    }
  }
}
write · /app/event/rule-pattern.json
{
  "source": ["order.saga"],
  "detail-type": ["OrderCompleted", "OrderFailed"]
}
  • Write lambda source files (6 lambdas)
  • Write IAM policies (least privilege)
  • Write KMS key policy and state machine ASL
  • Write build.sh to provision all resources
  • Write verify.sh with happy + failure assertions and policy audit
  • Run build and verify, fix issues
Now build.sh. It needs to be idempotent in case we re-run.
write · /app/build.sh
#!/usr/bin/env bash
set -euo pipefail

EP="http://localstack:4566"
REGION="us-east-1"
ACCOUNT="000000000000"

AWS() { aws --endpoint-url="$EP" --region "$REGION" "$@"; }

cd /app

# ---------- DynamoDB tables ----------
create_table() {
  local name="$1" pk_name="$2" pk_type="$3"
  if AWS dynamodb describe-table --table-name "$name" >/dev/null 2>&1; then
    echo "ddb table $name: exists"
    return
  fi
  AWS dynamodb create-table \
    --table-name "$name" \
    --attribute-definitions "AttributeName=$pk_name,AttributeType=$pk_type" \
    --key-schema "AttributeName=$pk_name,KeyType=HASH" \
    --billing-mode PAY_PER_REQUEST >/dev/null
  AWS dynamodb wait table-exists --table-name "$name"
  echo "ddb table $name: created"
}

create_table inventory         sku             S
create_table payments          payment_id      S
create_table shipments         shipment_id     S
create_table saga-idempotency  idempotency_key S

# Enable TTL on saga-idempotency (attribute name: expire_at)
ttl_status=$(AWS dynamodb describe-time-to-live --table-name saga-idempotency --query 'TimeToLiveDescription.TimeToLiveStatus' --output text 2>/dev/null || echo NONE)
if [[ "$ttl_status" != "ENABLED" && "$ttl_status" != "ENABLING" ]]; then
  AWS dynamodb update-time-to-live \
    --table-name saga-idempotency \
    --time-to-live-specification "Enabled=true,AttributeName=expire_at" >/dev/null
  echo "ddb ttl saga-idempotency: enabled"
else
  echo "ddb ttl saga-idempotency: $ttl_status"
fi

# ---------- KMS CMK + alias ----------
existing_alias=$(AWS kms list-aliases --query "Aliases[?AliasName=='alias/saga-cmk'].TargetKeyId" --output text 2>/dev/null || true)
if [[ -z "$existing_alias" || "$existing_alias" == "None" ]]; then
  KEY_ID=$(AWS kms create-key \
    --description "saga CMK for payment gateway secret" \
    --policy file:///app/kms/key-policy.json \
    --query 'KeyMetadata.KeyId' --output text)
  AWS kms create-alias --alias-name alias/saga-cmk --target-key-id "$KEY_ID" >/dev/null
  echo "kms saga-cmk: created key=$KEY_ID alias=alias/saga-cmk"
else
  KEY_ID="$existing_alias"
  AWS kms put-key-policy --key-id "$KEY_ID" --policy-name default --policy file:///app/kms/key-policy.json >/dev/null
  echo "kms saga-cmk: exists key=$KEY_ID (policy refreshed)"
fi
KEY_ARN="arn:aws:kms:${REGION}:${ACCOUNT}:key/${KEY_ID}"

# ---------- Secret ----------
if AWS secretsmanager describe-secret --secret-id payment/gateway-apikey >/dev/null 2>&1; then
  echo "secret payment/gateway-apikey: exists"
else
  AWS secretsmanager create-secret \
    --name payment/gateway-apikey \
    --kms-key-id "alias/saga-cmk" \
    --secret-string "sk_live_fakekey_$(date +%s)" >/dev/null
  echo "secret payment/gateway-apikey: created with CMK"
fi

# ---------- IAM roles ----------
create_role() {
  local role="$1" trust="$2" policy="$3" policy_name="$4"
  if AWS iam get-role --role-name "$role" >/dev/null 2>&1; then
    echo "iam role $role: exists"
  else
    AWS iam create-role --role-name "$role" \
      --assume-role-policy-document "file://$trust" >/dev/null
    echo "iam role $role: created"
  fi
  AWS iam put-role-policy \
    --role-name "$role" \
    --policy-name "$policy_name" \
    --policy-document "file://$policy" >/dev/null
}

create_role reserve-inventory-role /app/iam/trust-lambda.json /app/iam/reserve-inventory-policy.json reserve-inventory-policy
create_role release-inventory-role /app/iam/trust-lambda.json /app/iam/release-inventory-policy.json release-inventory-policy
create_role charge-payment-role    /app/iam/trust-lambda.json /app/iam/charge-payment-policy.json    charge-payment-policy
create_role refund-payment-role    /app/iam/trust-lambda.json /app/iam/refund-payment-policy.json    refund-payment-policy
create_role create-shipment-role   /app/iam/trust-lambda.json /app/iam/create-shipment-policy.json   create-shipment-policy
create_role saga-notifier-role     /app/iam/trust-lambda.json /app/iam/saga-notifier-policy.json     saga-notifier-policy
create_role order-saga-role        /app/iam/trust-sfn.json    /app/iam/order-saga-policy.json        order-saga-policy

# ---------- Lambdas ----------
BUILD_DIR=/app/build
rm -rf "$BUILD_DIR"; mkdir -p "$BUILD_DIR"

package_and_deploy() {
  local name="$1" src="$2" handler="$3" role="$4"
  local zip="$BUILD_DIR/${name}.zip"
  ( cd "$(dirname "$src")" && zip -q "$zip" "$(basename "$src")" )

  if AWS lambda get-function --function-name "$name" >/dev/null 2>&1; then
    AWS lambda update-function-code \
      --function-name "$name" \
      --zip-file "fileb://$zip" >/dev/null
    AWS lambda wait function-updated --function-name "$name"
    echo "lambda $name: code updated"
  else
    AWS lambda create-function \
      --function-name "$name" \
      --runtime python3.11 \
      --role "arn:aws:iam::${ACCOUNT}:role/${role}" \
      --handler "$handler" \
      --zip-file "fileb://$zip" \
      --timeout 30 >/dev/null
    AWS lambda wait function-active --function-name "$name"
    echo "lambda $name: created"
  fi
}

package_and_deploy reserve-inventory /app/lambdas/reserve_inventory.py reserve_inventory.handler reserve-inventory-role
package_and_deploy release-inventory /app/lambdas/release_inventory.py release_inventory.handler release-inventory-role
package_and_deploy charge-payment    /app/lambdas/charge_payment.py    charge_payment.handler    charge-payment-role
package_and_deploy refund-payment    /app/lambdas/refund_payment.py    refund_payment.handler    refund-payment-role
package_and_deploy create-shipment   /app/lambdas/create_shipment.py   create_shipment.handler   create-shipment-role
package_and_deploy saga-notifier     /app/lambdas/saga_notifier.py     saga_notifier.handler     saga-notifier-role

# ---------- EventBridge bus + rule + target ----------
if AWS events describe-event-bus --name saga-events >/dev/null 2>&1; then
  echo "event bus saga-events: exists"
else
  AWS events create-event-bus --name saga-events >/dev/null
  echo "event bus saga-events: created"
fi

AWS events put-rule \
  --name on-saga-terminal \
  --event-bus-name saga-events \
  --event-pattern file:///app/event/rule-pattern.json \
  --state ENABLED >/dev/null
echo "event rule on-saga-terminal: put"

NOTIFIER_ARN="arn:aws:lambda:${REGION}:${ACCOUNT}:function:saga-notifier"
RULE_ARN="arn:aws:events:${REGION}:${ACCOUNT}:rule/saga-events/on-saga-terminal"

# Give EventBridge permission to invoke the notifier
AWS lambda remove-permission --function-name saga-notifier --statement-id allow-eventbridge >/dev/null 2>&1 || true
AWS lambda add-permission \
  --function-name saga-notifier \
  --statement-id allow-eventbridge \
  --action lambda:InvokeFunction \
  --principal events.amazonaws.com \
  --source-arn "$RULE_ARN" >/dev/null
echo "lambda permission: eventbridge -> saga-notifier"

AWS events put-targets \
  --rule on-saga-terminal \
  --event-bus-name saga-events \
  --targets "Id=1,Arn=${NOTIFIER_ARN}" >/dev/null
echo "event target: on-saga-terminal -> saga-notifier"

# ---------- Step Functions state machine ----------
SM_ARN="arn:aws:states:${REGION}:${ACCOUNT}:stateMachine:order-saga"
if AWS stepfunctions describe-state-machine --state-machine-arn "$SM_ARN" >/dev/null 2>&1; then
  AWS stepfunctions update-state-machine \
    --state-machine-arn "$SM_ARN" \
    --definition file:///app/state-machine/order-saga.json \
    --role-arn "arn:aws:iam::${ACCOUNT}:role/order-saga-role" >/dev/null
  echo "state machine order-saga: updated"
else
  AWS stepfunctions create-state-machine \
    --name order-saga \
    --definition file:///app/state-machine/order-saga.json \
    --role-arn "arn:aws:iam::${ACCOUNT}:role/order-saga-role" \
    --type STANDARD >/dev/null
  echo "state machine order-saga: created"
fi

echo
echo "build: done"

Make build.sh executable

shell
chmod +x /app/build.sh
output
(Bash completed with no output)
  • Write lambda source files (6 lambdas)
  • Write IAM policies (least privilege)
  • Write KMS key policy and state machine ASL
  • Write build.sh to provision all resources
  • Write verify.sh with happy + failure assertions and policy audit
  • Run build and verify, fix issues
Now let me run build first to catch any issues before writing the verifier.

Run build script end-to-end

shell
bash build.sh 2>&1
output
ddb table inventory: created
ddb table payments: created
ddb table shipments: created
ddb table saga-idempotency: created
ddb ttl saga-idempotency: enabled
kms saga-cmk: created key=9212acfb-1fb9-471a-b465-ba89de50ec04 alias=alias/saga-cmk
secret payment/gateway-apikey: created with CMK
iam role reserve-inventory-role: created
iam role release-inventory-role: created
iam role charge-payment-role: created
iam role refund-payment-role: created
iam role create-shipment-role: created
iam role saga-notifier-role: created
iam role order-saga-role: created
lambda reserve-inventory: created
lambda release-inventory: created
lambda charge-payment: created
lambda refund-payment: created
lambda create-shipment: created
lambda saga-notifier: created
event bus saga-events: created
event rule on-saga-terminal: put
lambda permission: eventbridge -> saga-notifier
event target: on-saga-terminal -> saga-notifier
state machine order-saga: created

build: done

[stdout]
ddb table inventory: created
ddb table payments: created
ddb table shipments: created
ddb table saga-idempotency: created
ddb ttl saga-idempotency: enabled
kms saga-cmk: created key=9212acfb-1fb9-471a-b465-ba89de50ec04 alias=alias/saga-cmk
secret payment/gateway-apikey: created with CMK
iam role reserve-inventory-role: created
iam role release-inventory-role: created
iam role charge-payment-role: created
iam role refund-payment-role: created
iam role create-shipment-role: created
iam role saga-notifier-role: created
iam role order-saga-role: created
lambda reserve-inventory: created
lambda release-inventory: created
lambda charge-payment: created
lambda refund-payment: created
lambda create-shipment: created
lambda saga-notifier: created
event bus saga-events: created
event rule on-saga-terminal: put
lambda permission: eventbridge -> saga-notifier
event target: on-saga-terminal -> saga-notifier
state machine order-saga: created

build: done
Build succeeded cleanly. Let me do a quick smoke test of the happy path before writing the verifier.

Smoke-test happy path

shell
EP=http://localstack:4566
SM_ARN="arn:aws:states:us-east-1:000000000000:stateMachine:order-saga"
EXEC=$(aws --endpoint-url=$EP stepfunctions start-execution \
  --state-machine-arn "$SM_ARN" \
  --input '{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke"}' \
  --query executionArn --output text)
echo "Started: $EXEC"
for i in $(seq 1 30); do
  STATUS=$(aws --endpoint-url=$EP stepfunctions describe-execution --execution-arn "$EXEC" --query status --output text)
  echo "t=$i status=$STATUS"
  [ "$STATUS" != "RUNNING" ] && break
  sleep 2
done
aws --endpoint-url=$EP stepfunctions describe-execution --execution-arn "$EXEC" --query '[status,output]' --output text
output
Started: arn:aws:states:us-east-1:000000000000:execution:order-saga:e46d4c8f-f76c-4eb7-8b33-a2669982e557
t=1 status=RUNNING
t=2 status=RUNNING
t=3 status=SUCCEEDED
SUCCEEDED	{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve_status":"RESERVED"}},"charge":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve_status":"RESERVED"}},"payment_id":"pay-k-smoke","charge_status":"CAPTURED"}},"ship":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve_status":"RESERVED"}},"charge":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve_status":"RESERVED"}},"payment_id":"pay-k-smoke","charge_status":"CAPTURED"}},"shipment_id":"ship-k-smoke","ship_status":"CREATED"}},"published":{"FailedEntryCount":0,"Entries":[{"EventId":"0cadb450-be6f-416d-b2d2-d8c8bc126d7c"}]}}

[stdout]
Started: arn:aws:states:us-east-1:000000000000:execution:order-saga:e46d4c8f-f76c-4eb7-8b33-a2669982e557
t=1 status=RUNNING
t=2 status=RUNNING
t=3 status=SUCCEEDED
SUCCEEDED	{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve_status":"RESERVED"}},"charge":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve_status":"RESERVED"}},"payment_id":"pay-k-smoke","charge_status":"CAPTURED"}},"ship":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve_status":"RESERVED"}},"charge":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve":{"payload":{"order_id":"o-smoke","sku":"x","qty":2,"amount":100,"idempotency_key":"k-smoke","reserve_status":"RESERVED"}},"payment_id":"pay-k-smoke","charge_status":"CAPTURED"}},"shipment_id":"ship-k-smoke","ship_status":"CREATED"}},"published":{"FailedEntryCount":0,"Entries":[{"EventId":"0cadb450-be6f-416d-b2d2-d8c8bc126d7c"}]}}
Happy path runs end-to-end. Let me verify DDB state and the notifier log fired.

Check DDB state and notifier log

shell
EP=http://localstack:4566
echo "--- inventory ---"
aws --endpoint-url=$EP dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}'
echo "--- payments ---"
aws --endpoint-url=$EP dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-k-smoke"}}'
echo "--- shipments ---"
aws --endpoint-url=$EP dynamodb get-item --table-name shipments --key '{"shipment_id":{"S":"ship-k-smoke"}}'
echo "--- notifier logs ---"
aws --endpoint-url=$EP logs describe-log-groups --log-group-name-prefix /aws/lambda/saga-notifier --query 'logGroups[*].logGroupName' --output text
GROUP=/aws/lambda/saga-notifier
sleep 2
aws --endpoint-url=$EP logs filter-log-events --log-group-name $GROUP --filter-pattern 'saga-terminal' --query 'events[*].message' --output text 2>&1 | head -20
output
--- inventory ---
{
    "Item": {
        "sku": {
            "S": "x"
        },
        "reservation_keys": {
            "SS": [
                "k-smoke:reserve"
            ]
        },
        "reserved_qty": {
            "N": "2"
        }
    }
}
--- payments ---
{
    "Item": {
        "amount": {
            "N": "100"
        },
        "order_id": {
            "S": "o-smoke"
        },
        "payment_id": {
            "S": "pay-k-smoke"
        },
        "status": {
            "S": "CAPTURED"
        }
    }
}
--- shipments ---
{
    "Item": {
        "shipment_id": {
            "S": "ship-k-smoke"
        },
        "sku": {
            "S": "x"
        },
        "order_id": {
            "S": "o-smoke"
        },
        "qty": {
            "N": "2"
        },
        "status": {
            "S": "CREATED"
        }
    }
}
--- notifier logs ---
/aws/lambda/saga-notifier
START RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342 Version: $LATEST
	saga-terminal:OrderCompleted order=o-smoke
	END RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342
	REPORT RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342	Duration: 7.98 ms	Billed Duration: 8 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

[stdout]
--- inventory ---
{
    "Item": {
        "sku": {
            "S": "x"
        },
        "reservation_keys": {
            "SS": [
                "k-smoke:reserve"
            ]
        },
        "reserved_qty": {
            "N": "2"
        }
    }
}
--- payments ---
{
    "Item": {
        "amount": {
            "N": "100"
        },
        "order_id": {
            "S": "o-smoke"
        },
        "payment_id": {
            "S": "pay-k-smoke"
        },
        "status": {
            "S": "CAPTURED"
        }
    }
}
--- shipments ---
{
    "Item": {
        "shipment_id": {
            "S": "ship-k-smoke"
        },
        "sku": {
            "S": "x"
        },
        "order_id": {
            "S": "o-smoke"
        },
        "qty": {
            "N": "2"
        },
        "status": {
            "S": "CREATED"
        }
    }
}
--- notifier logs ---
/aws/lambda/saga-notifier
START RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342 Version: $LATEST
	saga-terminal:OrderCompleted order=o-smoke
	END RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342
	REPORT RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342	Duration: 7.98 ms	Billed Duration: 8 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
Excellent , happy path fully validated, including notifier log. Now smoke-test the failure path.

Smoke-test failure path

shell
EP=http://localstack:4566
SM_ARN="arn:aws:states:us-east-1:000000000000:stateMachine:order-saga"
EXEC=$(aws --endpoint-url=$EP stepfunctions start-execution \
  --state-machine-arn "$SM_ARN" \
  --input '{"order_id":"o-smokefail","sku":"y","qty":3,"amount":50,"idempotency_key":"k-smokefail","force_failure":true}' \
  --query executionArn --output text)
echo "Started: $EXEC"
for i in $(seq 1 30); do
  STATUS=$(aws --endpoint-url=$EP stepfunctions describe-execution --execution-arn "$EXEC" --query status --output text)
  echo "t=$i status=$STATUS"
  [ "$STATUS" != "RUNNING" ] && break
  sleep 2
done
echo "--- inventory (y) should be 0 ---"
aws --endpoint-url=$EP dynamodb get-item --table-name inventory --key '{"sku":{"S":"y"}}'
echo "--- payments (pay-k-smokefail) should be absent ---"
aws --endpoint-url=$EP dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-k-smokefail"}}'
echo "--- shipments (ship-k-smokefail) should be absent ---"
aws --endpoint-url=$EP dynamodb get-item --table-name shipments --key '{"shipment_id":{"S":"ship-k-smokefail"}}'
sleep 2
echo "--- notifier logs ---"
aws --endpoint-url=$EP logs filter-log-events --log-group-name /aws/lambda/saga-notifier --filter-pattern 'saga-terminal' --query 'events[*].message' --output text 2>&1 | tail -10
output
Started: arn:aws:states:us-east-1:000000000000:execution:order-saga:36195bb2-57b6-4190-b3c8-ebd2881a8c19
t=1 status=RUNNING
t=2 status=RUNNING
t=3 status=SUCCEEDED
--- inventory (y) should be 0 ---
{
    "Item": {
        "sku": {
            "S": "y"
        },
        "reserved_qty": {
            "N": "0"
        }
    }
}
--- payments (pay-k-smokefail) should be absent ---
--- shipments (ship-k-smokefail) should be absent ---
--- notifier logs ---
START RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342 Version: $LATEST
	saga-terminal:OrderCompleted order=o-smoke
	END RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342
	REPORT RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342	Duration: 7.98 ms	Billed Duration: 8 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
	START RequestId: 7d4204d3-64fe-42f0-8162-0c3174c717d3 Version: $LATEST
	saga-terminal:OrderFailed order=o-smokefail
	END RequestId: 7d4204d3-64fe-42f0-8162-0c3174c717d3
	REPORT RequestId: 7d4204d3-64fe-42f0-8162-0c3174c717d3	Duration: 9.53 ms	Billed Duration: 10 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

[stdout]
Started: arn:aws:states:us-east-1:000000000000:execution:order-saga:36195bb2-57b6-4190-b3c8-ebd2881a8c19
t=1 status=RUNNING
t=2 status=RUNNING
t=3 status=SUCCEEDED
--- inventory (y) should be 0 ---
{
    "Item": {
        "sku": {
            "S": "y"
        },
        "reserved_qty": {
            "N": "0"
        }
    }
}
--- payments (pay-k-smokefail) should be absent ---
--- shipments (ship-k-smokefail) should be absent ---
--- notifier logs ---
START RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342 Version: $LATEST
	saga-terminal:OrderCompleted order=o-smoke
	END RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342
	REPORT RequestId: 6dae9c75-dd5c-47be-9884-568b1354c342	Duration: 7.98 ms	Billed Duration: 8 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
	START RequestId: 7d4204d3-64fe-42f0-8162-0c3174c717d3 Version: $LATEST
	saga-terminal:OrderFailed order=o-smokefail
	END RequestId: 7d4204d3-64fe-42f0-8162-0c3174c717d3
	REPORT RequestId: 7d4204d3-64fe-42f0-8162-0c3174c717d3	Duration: 9.53 ms	Billed Duration: 10 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
All smoke tests green. Now the formal verifier with clean-state + assertions + policy audit.
write · /app/verify.sh
#!/usr/bin/env bash
set -uo pipefail

EP="http://localstack:4566"
REGION="us-east-1"
ACCOUNT="000000000000"
SM_ARN="arn:aws:states:${REGION}:${ACCOUNT}:stateMachine:order-saga"

AWS() { aws --endpoint-url="$EP" --region "$REGION" "$@"; }

PASS=0
FAIL=0
check() {
  local name="$1"; shift
  if "$@" >/dev/null 2>&1; then
    echo "PASS  $name"
    PASS=$((PASS+1))
  else
    echo "FAIL  $name"
    FAIL=$((FAIL+1))
  fi
}

assert_eq() {
  local name="$1" expected="$2" actual="$3"
  if [[ "$expected" == "$actual" ]]; then
    echo "PASS  $name"
    PASS=$((PASS+1))
  else
    echo "FAIL  $name (expected=$expected actual=$actual)"
    FAIL=$((FAIL+1))
  fi
}

reset_row() {
  local tbl="$1" keyname="$2" keyval="$3"
  AWS dynamodb delete-item --table-name "$tbl" \
    --key "{\"$keyname\":{\"S\":\"$keyval\"}}" >/dev/null 2>&1 || true
}

run_execution() {
  local input="$1"
  local arn
  arn=$(AWS stepfunctions start-execution \
    --state-machine-arn "$SM_ARN" \
    --input "$input" \
    --query executionArn --output text)
  local status=RUNNING
  for _ in $(seq 1 30); do
    status=$(AWS stepfunctions describe-execution --execution-arn "$arn" --query status --output text)
    [[ "$status" != "RUNNING" ]] && break
    sleep 2
  done
  echo "$status"
}

##############################################################################
# Pre-clean: remove prior test rows so the verifier is idempotent
##############################################################################
echo "=== pre-clean ==="
reset_row inventory sku x
reset_row inventory sku y
reset_row payments payment_id pay-k-1
reset_row payments payment_id pay-k-2
reset_row shipments shipment_id ship-k-1
reset_row shipments shipment_id ship-k-2
echo "pre-clean: done"
echo

# Log group for the notifier , record starting offset to filter only new logs
NOTIFIER_LG=/aws/lambda/saga-notifier
START_TS=$(( ($(date +%s) - 5) * 1000 ))

##############################################################################
# Happy path
##############################################################################
echo "=== happy path: o-1 k-1 ==="
HAPPY_INPUT='{"order_id":"o-1","sku":"x","qty":2,"amount":100,"idempotency_key":"k-1"}'
HAPPY_STATUS=$(run_execution "$HAPPY_INPUT")
assert_eq "happy: state machine SUCCEEDED" "SUCCEEDED" "$HAPPY_STATUS"

# inventory[x].reserved_qty == 2
QTY=$(AWS dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}' \
  --query 'Item.reserved_qty.N' --output text 2>/dev/null)
assert_eq "happy: inventory x reserved_qty=2" "2" "$QTY"

# payments.pay-k-1 status == CAPTURED
PAY_STATUS=$(AWS dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-k-1"}}' \
  --query 'Item.status.S' --output text 2>/dev/null)
assert_eq "happy: payments CAPTURED" "CAPTURED" "$PAY_STATUS"

# shipments.ship-k-1 exists
SHIP_STATUS=$(AWS dynamodb get-item --table-name shipments --key '{"shipment_id":{"S":"ship-k-1"}}' \
  --query 'Item.status.S' --output text 2>/dev/null)
assert_eq "happy: shipments row exists (CREATED)" "CREATED" "$SHIP_STATUS"

# notifier log contains saga-terminal:OrderCompleted
sleep 2
LOG_HIT=$(AWS logs filter-log-events --log-group-name "$NOTIFIER_LG" \
  --start-time "$START_TS" \
  --filter-pattern 'saga-terminal:OrderCompleted' \
  --query 'events[?contains(message, `o-1`)] | length(@)' --output text 2>/dev/null)
if [[ "$LOG_HIT" != "0" && -n "$LOG_HIT" && "$LOG_HIT" != "None" ]]; then
  echo "PASS  happy: notifier logged saga-terminal:OrderCompleted (order o-1)"
  PASS=$((PASS+1))
else
  echo "FAIL  happy: notifier did not log OrderCompleted for o-1 (hits=$LOG_HIT)"
  FAIL=$((FAIL+1))
fi

echo

##############################################################################
# Failure path (compensations must run in reverse order)
##############################################################################
echo "=== failure path: o-2 k-2 force_failure=true ==="
FAIL_INPUT='{"order_id":"o-2","sku":"x","qty":1,"amount":77,"idempotency_key":"k-2","force_failure":true}'
# Capture reserved_qty before so we can confirm the net delta is 0
PRE_QTY=$(AWS dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}' \
  --query 'Item.reserved_qty.N' --output text 2>/dev/null)

FAIL_STATUS=$(run_execution "$FAIL_INPUT")
assert_eq "failure: state machine SUCCEEDED (saga handled gracefully)" "SUCCEEDED" "$FAIL_STATUS"

# inventory net unchanged: reserve happened, release compensated it
POST_QTY=$(AWS dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}' \
  --query 'Item.reserved_qty.N' --output text 2>/dev/null)
assert_eq "failure: inventory x reserved_qty unchanged (=$PRE_QTY)" "$PRE_QTY" "$POST_QTY"

# The reservation_keys set must NOT contain k-2:reserve (released)
HAS_K2=$(AWS dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}' \
  --query 'Item.reservation_keys.SS' --output json 2>/dev/null)
if echo "$HAS_K2" | grep -q "k-2:reserve"; then
  echo "FAIL  failure: reservation_keys still contains k-2:reserve , release compensation did not run"
  FAIL=$((FAIL+1))
else
  echo "PASS  failure: reservation_keys does NOT contain k-2:reserve (release ran)"
  PASS=$((PASS+1))
fi

# payments row absent OR REFUNDED
PAY2=$(AWS dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-k-2"}}' \
  --query 'Item.status.S' --output text 2>/dev/null)
if [[ -z "$PAY2" || "$PAY2" == "None" || "$PAY2" == "REFUNDED" ]]; then
  echo "PASS  failure: payments row absent or REFUNDED (got: '${PAY2:-absent}')"
  PASS=$((PASS+1))
else
  echo "FAIL  failure: payments row has status='$PAY2' (expected absent or REFUNDED)"
  FAIL=$((FAIL+1))
fi

# shipments row must NOT exist
SHIP2=$(AWS dynamodb get-item --table-name shipments --key '{"shipment_id":{"S":"ship-k-2"}}' \
  --query 'Item.shipment_id.S' --output text 2>/dev/null)
if [[ -z "$SHIP2" || "$SHIP2" == "None" ]]; then
  echo "PASS  failure: shipments row does NOT exist for ship-k-2"
  PASS=$((PASS+1))
else
  echo "FAIL  failure: shipments row EXISTS for ship-k-2 (shipment must not have been created)"
  FAIL=$((FAIL+1))
fi

# notifier log contains saga-terminal:OrderFailed for o-2
sleep 2
LOG_HIT2=$(AWS logs filter-log-events --log-group-name "$NOTIFIER_LG" \
  --start-time "$START_TS" \
  --filter-pattern 'saga-terminal:OrderFailed' \
  --query 'events[?contains(message, `o-2`)] | length(@)' --output text 2>/dev/null)
if [[ "$LOG_HIT2" != "0" && -n "$LOG_HIT2" && "$LOG_HIT2" != "None" ]]; then
  echo "PASS  failure: notifier logged saga-terminal:OrderFailed (order o-2)"
  PASS=$((PASS+1))
else
  echo "FAIL  failure: notifier did not log OrderFailed for o-2 (hits=$LOG_HIT2)"
  FAIL=$((FAIL+1))
fi

echo

##############################################################################
# Least-privilege policy audit (reads JSON files directly)
##############################################################################
echo "=== policy audit ==="

policy_has() {
  # returns 0 if the policy JSON contains the substring, 1 otherwise
  grep -q "$2" "$1"
}

# Each lambda role must not name business tables other than its own
check "reserve-inventory-policy: no 'payments' table"   ! policy_has /app/iam/reserve-inventory-policy.json "table/payments"
check "reserve-inventory-policy: no 'shipments' table"  ! policy_has /app/iam/reserve-inventory-policy.json "table/shipments"
check "release-inventory-policy: no 'payments' table"   ! policy_has /app/iam/release-inventory-policy.json "table/payments"
check "release-inventory-policy: no 'shipments' table"  ! policy_has /app/iam/release-inventory-policy.json "table/shipments"
check "charge-payment-policy: no 'inventory' table"     ! policy_has /app/iam/charge-payment-policy.json "table/inventory"
check "charge-payment-policy: no 'shipments' table"     ! policy_has /app/iam/charge-payment-policy.json "table/shipments"
check "refund-payment-policy: no 'inventory' table"     ! policy_has /app/iam/refund-payment-policy.json "table/inventory"
check "refund-payment-policy: no 'shipments' table"     ! policy_has /app/iam/refund-payment-policy.json "table/shipments"
check "create-shipment-policy: no 'inventory' table"    ! policy_has /app/iam/create-shipment-policy.json "table/inventory"
check "create-shipment-policy: no 'payments' table"     ! policy_has /app/iam/create-shipment-policy.json "table/payments"

# Only charge-payment may read the gateway secret
for p in reserve-inventory release-inventory refund-payment create-shipment saga-notifier; do
  check "${p}-policy: no secretsmanager access" \
    ! policy_has "/app/iam/${p}-policy.json" "secretsmanager"
done
check "charge-payment-policy: HAS secretsmanager:GetSecretValue on payment/gateway-apikey" \
  bash -c "grep -q 'secretsmanager:GetSecretValue' /app/iam/charge-payment-policy.json && grep -q 'payment/gateway-apikey' /app/iam/charge-payment-policy.json"

# No lambda role policy may use Resource:"*" on any kms: action
# (simple check: any role policy file that mentions kms: must not contain "Resource": "*")
audit_kms_wildcard() {
  local f="$1"
  if grep -q 'kms:' "$f"; then
    if grep -E '"Resource"\s*:\s*"\*"' "$f" >/dev/null; then
      return 1
    fi
  fi
  return 0
}
for f in /app/iam/*-policy.json; do
  check "$(basename "$f"): no wildcard Resource on kms actions" audit_kms_wildcard "$f"
done

# Saga role: no wildcard on lambda:InvokeFunction
if grep -q '"lambda:InvokeFunction"' /app/iam/order-saga-policy.json; then
  if grep -E '"Resource"\s*:\s*"\*"' /app/iam/order-saga-policy.json >/dev/null; then
    echo "FAIL  order-saga-policy: wildcard Resource on lambda:InvokeFunction"
    FAIL=$((FAIL+1))
  else
    echo "PASS  order-saga-policy: lambda:InvokeFunction scoped to specific ARNs"
    PASS=$((PASS+1))
  fi
fi

# Notifier role has nothing but logs
if grep -q -E 'dynamodb|kms|secretsmanager|lambda:Invoke|events:PutEvents' /app/iam/saga-notifier-policy.json; then
  echo "FAIL  saga-notifier-policy: has non-logs permissions"
  FAIL=$((FAIL+1))
else
  echo "PASS  saga-notifier-policy: logs-only"
  PASS=$((PASS+1))
fi

echo
echo "=== summary: PASS=$PASS FAIL=$FAIL ==="
[[ $FAIL -eq 0 ]]
  • Write lambda source files (6 lambdas)
  • Write IAM policies (least privilege)
  • Write KMS key policy and state machine ASL
  • Write build.sh to provision all resources
  • Write verify.sh with happy + failure assertions and policy audit
  • Run build and verify, fix issues

Run the verifier

shell
chmod +x /app/verify.sh && bash /app/verify.sh 2>&1
output
Exit code 1
=== pre-clean ===
pre-clean: done

=== happy path: o-1 k-1 ===
PASS  happy: state machine SUCCEEDED
PASS  happy: inventory x reserved_qty=2
PASS  happy: payments CAPTURED
PASS  happy: shipments row exists (CREATED)
PASS  happy: notifier logged saga-terminal:OrderCompleted (order o-1)

=== failure path: o-2 k-2 force_failure=true ===
PASS  failure: state machine SUCCEEDED (saga handled gracefully)
PASS  failure: inventory x reserved_qty unchanged (=2)
PASS  failure: reservation_keys does NOT contain k-2:reserve (release ran)
PASS  failure: payments row absent or REFUNDED (got: 'None')
PASS  failure: shipments row does NOT exist for ship-k-2
PASS  failure: notifier logged saga-terminal:OrderFailed (order o-2)

=== policy audit ===
FAIL  reserve-inventory-policy: no 'payments' table
FAIL  reserve-inventory-policy: no 'shipments' table
FAIL  release-inventory-policy: no 'payments' table
FAIL  release-inventory-policy: no 'shipments' table
FAIL  charge-payment-policy: no 'inventory' table
FAIL  charge-payment-policy: no 'shipments' table
FAIL  refund-payment-policy: no 'inventory' table
FAIL  refund-payment-policy: no 'shipments' table
FAIL  create-shipment-policy: no 'inventory' table
FAIL  create-shipment-policy: no 'payments' table
FAIL  reserve-inventory-policy: no secretsmanager access
FAIL  release-inventory-policy: no secretsmanager access
FAIL  refund-payment-policy: no secretsmanager access
FAIL  create-shipment-policy: no secretsmanager access
FAIL  saga-notifier-policy: no secretsmanager access
PASS  charge-payment-policy: HAS secretsmanager:GetSecretValue on payment/gateway-apikey
PASS  charge-payment-policy.json: no wildcard Resource on kms actions
PASS  create-shipment-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy.json: no wildcard Resource on kms actions
PASS  refund-payment-policy.json: no wildcard Resource on kms actions
PASS  release-inventory-policy.json: no wildcard Resource on kms actions
PASS  reserve-inventory-policy.json: no wildcard Resource on kms actions
PASS  saga-notifier-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy: lambda:InvokeFunction scoped to specific ARNs
PASS  saga-notifier-policy: logs-only

=== summary: PASS=21 FAIL=15 ===

[error] tool reported failure
The functional tests all pass. The policy audit checks are buggy: passing `! policy_has ...` through `"$@"` doesn't work because `!` is a bash keyword, not a command. Let me fix the audit helpers.

/app/verify.sh

− removed
policy_has() {
  # returns 0 if the policy JSON contains the substring, 1 otherwise
  grep -q "$2" "$1"
}

# Each lambda role must not name business tables other than its own
check "reserve-inventory-policy: no 'payments' table"   ! policy_has /app/iam/reserve-inventory-policy.json "table/payments"
check "reserve-inventory-policy: no 'shipments' table"  ! policy_has /app/iam/reserve-inventory-policy.json "table/shipments"
check "release-inventory-policy: no 'payments' table"   ! policy_has /app/iam/release-inventory-policy.json "table/payments"
check "release-inventory-policy: no 'shipments' table"  ! policy_has /app/iam/release-inventory-policy.json "table/shipments"
check "charge-payment-policy: no 'inventory' table"     ! policy_has /app/iam/charge-payment-policy.json "table/inventory"
check "charge-payment-policy: no 'shipments' table"     ! policy_has /app/iam/charge-payment-policy.json "table/shipments"
check "refund-payment-policy: no 'inventory' table"     ! policy_has /app/iam/refund-payment-policy.json "table/inventory"
check "refund-payment-policy: no 'shipments' table"     ! policy_has /app/iam/refund-payment-policy.json "table/shipments"
check "create-shipment-policy: no 'inventory' table"    ! policy_has /app/iam/create-shipment-policy.json "table/inventory"
check "create-shipment-policy: no 'payments' table"     ! policy_has /app/iam/create-shipment-policy.json "table/payments"

# Only charge-payment may read the gateway secret
for p in reserve-inventory release-inventory refund-payment create-shipment saga-notifier; do
  check "${p}-policy: no secretsmanager access" \
    ! policy_has "/app/iam/${p}-policy.json" "secretsmanager"
done
check "charge-payment-policy: HAS secretsmanager:GetSecretValue on payment/gateway-apikey" \
  bash -c "grep -q 'secretsmanager:GetSecretValue' /app/iam/charge-payment-policy.json && grep -q 'payment/gateway-apikey' /app/iam/charge-payment-policy.json"
+ added
assert_absent() {
  # PASS if $2 is NOT found in file $3
  local name="$1" needle="$2" file="$3"
  if grep -q -- "$needle" "$file"; then
    echo "FAIL  $name (found '$needle' in $file)"
    FAIL=$((FAIL+1))
  else
    echo "PASS  $name"
    PASS=$((PASS+1))
  fi
}

assert_present() {
  local name="$1" needle="$2" file="$3"
  if grep -q -- "$needle" "$file"; then
    echo "PASS  $name"
    PASS=$((PASS+1))
  else
    echo "FAIL  $name (missing '$needle' in $file)"
    FAIL=$((FAIL+1))
  fi
}

# Each lambda role must not name business tables other than its own
assert_absent "reserve-inventory-policy: no 'payments' table"   "table/payments"   /app/iam/reserve-inventory-policy.json
assert_absent "reserve-inventory-policy: no 'shipments' table"  "table/shipments"  /app/iam/reserve-inventory-policy.json
assert_absent "release-inventory-policy: no 'payments' table"   "table/payments"   /app/iam/release-inventory-policy.json
assert_absent "release-inventory-policy: no 'shipments' table"  "table/shipments"  /app/iam/release-inventory-policy.json
assert_absent "charge-payment-policy: no 'inventory' table"     "table/inventory"  /app/iam/charge-payment-policy.json
assert_absent "charge-payment-policy: no 'shipments' table"     "table/shipments"  /app/iam/charge-payment-policy.json
assert_absent "refund-payment-policy: no 'inventory' table"     "table/inventory"  /app/iam/refund-payment-policy.json
assert_absent "refund-payment-policy: no 'shipments' table"     "table/shipments"  /app/iam/refund-payment-policy.json
assert_absent "create-shipment-policy: no 'inventory' table"    "table/inventory"  /app/iam/create-shipment-policy.json
assert_absent "create-shipment-policy: no 'payments' table"     "table/payments"   /app/iam/create-shipment-policy.json

# Only charge-payment may read the gateway secret
for p in reserve-inventory release-inventory refund-payment create-shipment saga-notifier; do
  assert_absent "${p}-policy: no secretsmanager access" "secretsmanager" "/app/iam/${p}-policy.json"
done
assert_present "charge-payment-policy: HAS secretsmanager:GetSecretValue" "secretsmanager:GetSecretValue" /app/iam/charge-payment-policy.json
assert_present "charge-payment-policy: scoped to payment/gateway-apikey"  "payment/gateway-apikey"        /app/iam/charge-payment-policy.json

Re-run verifier with fixed audit helpers

shell
bash /app/verify.sh 2>&1
output
=== pre-clean ===
pre-clean: done

=== happy path: o-1 k-1 ===
PASS  happy: state machine SUCCEEDED
PASS  happy: inventory x reserved_qty=2
PASS  happy: payments CAPTURED
PASS  happy: shipments row exists (CREATED)
PASS  happy: notifier logged saga-terminal:OrderCompleted (order o-1)

=== failure path: o-2 k-2 force_failure=true ===
PASS  failure: state machine SUCCEEDED (saga handled gracefully)
PASS  failure: inventory x reserved_qty unchanged (=2)
PASS  failure: reservation_keys does NOT contain k-2:reserve (release ran)
PASS  failure: payments row absent or REFUNDED (got: 'None')
PASS  failure: shipments row does NOT exist for ship-k-2
PASS  failure: notifier logged saga-terminal:OrderFailed (order o-2)

=== policy audit ===
PASS  reserve-inventory-policy: no 'payments' table
PASS  reserve-inventory-policy: no 'shipments' table
PASS  release-inventory-policy: no 'payments' table
PASS  release-inventory-policy: no 'shipments' table
PASS  charge-payment-policy: no 'inventory' table
PASS  charge-payment-policy: no 'shipments' table
PASS  refund-payment-policy: no 'inventory' table
PASS  refund-payment-policy: no 'shipments' table
PASS  create-shipment-policy: no 'inventory' table
PASS  create-shipment-policy: no 'payments' table
PASS  reserve-inventory-policy: no secretsmanager access
PASS  release-inventory-policy: no secretsmanager access
PASS  refund-payment-policy: no secretsmanager access
PASS  create-shipment-policy: no secretsmanager access
PASS  saga-notifier-policy: no secretsmanager access
PASS  charge-payment-policy: HAS secretsmanager:GetSecretValue
PASS  charge-payment-policy: scoped to payment/gateway-apikey
PASS  charge-payment-policy.json: no wildcard Resource on kms actions
PASS  create-shipment-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy.json: no wildcard Resource on kms actions
PASS  refund-payment-policy.json: no wildcard Resource on kms actions
PASS  release-inventory-policy.json: no wildcard Resource on kms actions
PASS  reserve-inventory-policy.json: no wildcard Resource on kms actions
PASS  saga-notifier-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy: lambda:InvokeFunction scoped to specific ARNs
PASS  saga-notifier-policy: logs-only

=== summary: PASS=37 FAIL=0 ===

[stdout]
=== pre-clean ===
pre-clean: done

=== happy path: o-1 k-1 ===
PASS  happy: state machine SUCCEEDED
PASS  happy: inventory x reserved_qty=2
PASS  happy: payments CAPTURED
PASS  happy: shipments row exists (CREATED)
PASS  happy: notifier logged saga-terminal:OrderCompleted (order o-1)

=== failure path: o-2 k-2 force_failure=true ===
PASS  failure: state machine SUCCEEDED (saga handled gracefully)
PASS  failure: inventory x reserved_qty unchanged (=2)
PASS  failure: reservation_keys does NOT contain k-2:reserve (release ran)
PASS  failure: payments row absent or REFUNDED (got: 'None')
PASS  failure: shipments row does NOT exist for ship-k-2
PASS  failure: notifier logged saga-terminal:OrderFailed (order o-2)

=== policy audit ===
PASS  reserve-inventory-policy: no 'payments' table
PASS  reserve-inventory-policy: no 'shipments' table
PASS  release-inventory-policy: no 'payments' table
PASS  release-inventory-policy: no 'shipments' table
PASS  charge-payment-policy: no 'inventory' table
PASS  charge-payment-policy: no 'shipments' table
PASS  refund-payment-policy: no 'inventory' table
PASS  refund-payment-policy: no 'shipments' table
PASS  create-shipment-policy: no 'inventory' table
PASS  create-shipment-policy: no 'payments' table
PASS  reserve-inventory-policy: no secretsmanager access
PASS  release-inventory-policy: no secretsmanager access
PASS  refund-payment-policy: no secretsmanager access
PASS  create-shipment-policy: no secretsmanager access
PASS  saga-notifier-policy: no secretsmanager access
PASS  charge-payment-policy: HAS secretsmanager:GetSecretValue
PASS  charge-payment-policy: scoped to payment/gateway-apikey
PASS  charge-payment-policy.json: no wildcard Resource on kms actions
PASS  create-shipment-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy.json: no wildcard Resource on kms actions
PASS  refund-payment-policy.json: no wildcard Resource on kms actions
PASS  release-inventory-policy.json: no wildcard Resource on kms actions
PASS  reserve-inventory-policy.json: no wildcard Resource on kms actions
PASS  saga-notifier-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy: lambda:InvokeFunction scoped to specific ARNs
PASS  saga-notifier-policy: logs-only

=== summary: PASS=37 FAIL=0 ===
All 37 assertions green. Let me add a targeted idempotency test since the user called it out explicitly , invoke a forward lambda and a compensation lambda twice with the same key, confirm no double-counting.

/app/verify.sh

− removed
echo
echo "=== summary: PASS=$PASS FAIL=$FAIL ==="
+ added
echo

##############################################################################
# Idempotency: directly invoke lambdas twice with the same key
##############################################################################
echo "=== idempotency checks (direct invoke, same key twice) ==="
IDEMP_SKU="idem-sku"
IDEMP_KEY="idem-k"
reset_row inventory sku "$IDEMP_SKU"

invoke_lambda() {
  local fn="$1" payload="$2" out
  out=$(mktemp)
  AWS lambda invoke --function-name "$fn" \
    --payload "$(echo -n "$payload" | base64 -w0)" \
    --cli-binary-format raw-in-base64-out \
    "$out" >/dev/null 2>&1
  cat "$out"; rm -f "$out"
}

PAYLOAD="{\"order_id\":\"idem-o\",\"sku\":\"$IDEMP_SKU\",\"qty\":5,\"amount\":1,\"idempotency_key\":\"$IDEMP_KEY\"}"
invoke_lambda reserve-inventory "$PAYLOAD" >/dev/null
invoke_lambda reserve-inventory "$PAYLOAD" >/dev/null
IDEM_QTY=$(AWS dynamodb get-item --table-name inventory --key "{\"sku\":{\"S\":\"$IDEMP_SKU\"}}" \
  --query 'Item.reserved_qty.N' --output text 2>/dev/null)
assert_eq "idempotency: reserve-inventory twice -> qty stays 5 (not 10)" "5" "$IDEM_QTY"

invoke_lambda release-inventory "$PAYLOAD" >/dev/null
invoke_lambda release-inventory "$PAYLOAD" >/dev/null
IDEM_QTY2=$(AWS dynamodb get-item --table-name inventory --key "{\"sku\":{\"S\":\"$IDEMP_SKU\"}}" \
  --query 'Item.reserved_qty.N' --output text 2>/dev/null)
assert_eq "idempotency: release-inventory twice -> qty back to 0 (not -5)" "0" "$IDEM_QTY2"

# refund-payment is idempotent: second call on a REFUNDED row is a no-op
reset_row payments payment_id "pay-$IDEMP_KEY"
AWS dynamodb put-item --table-name payments --item \
  "{\"payment_id\":{\"S\":\"pay-$IDEMP_KEY\"},\"order_id\":{\"S\":\"idem-o\"},\"amount\":{\"N\":\"1\"},\"status\":{\"S\":\"CAPTURED\"}}" >/dev/null
invoke_lambda refund-payment "$PAYLOAD" >/dev/null
invoke_lambda refund-payment "$PAYLOAD" >/dev/null
REF_STATUS=$(AWS dynamodb get-item --table-name payments --key "{\"payment_id\":{\"S\":\"pay-$IDEMP_KEY\"}}" \
  --query 'Item.status.S' --output text 2>/dev/null)
assert_eq "idempotency: refund-payment twice -> status REFUNDED" "REFUNDED" "$REF_STATUS"

# cleanup
reset_row inventory sku "$IDEMP_SKU"
reset_row payments payment_id "pay-$IDEMP_KEY"

echo
echo "=== summary: PASS=$PASS FAIL=$FAIL ==="

Final verifier run

shell
bash /app/verify.sh 2>&1 | tail -50
output
=== happy path: o-1 k-1 ===
PASS  happy: state machine SUCCEEDED
PASS  happy: inventory x reserved_qty=2
PASS  happy: payments CAPTURED
PASS  happy: shipments row exists (CREATED)
PASS  happy: notifier logged saga-terminal:OrderCompleted (order o-1)

=== failure path: o-2 k-2 force_failure=true ===
PASS  failure: state machine SUCCEEDED (saga handled gracefully)
PASS  failure: inventory x reserved_qty unchanged (=2)
PASS  failure: reservation_keys does NOT contain k-2:reserve (release ran)
PASS  failure: payments row absent or REFUNDED (got: 'None')
PASS  failure: shipments row does NOT exist for ship-k-2
PASS  failure: notifier logged saga-terminal:OrderFailed (order o-2)

=== policy audit ===
PASS  reserve-inventory-policy: no 'payments' table
PASS  reserve-inventory-policy: no 'shipments' table
PASS  release-inventory-policy: no 'payments' table
PASS  release-inventory-policy: no 'shipments' table
PASS  charge-payment-policy: no 'inventory' table
PASS  charge-payment-policy: no 'shipments' table
PASS  refund-payment-policy: no 'inventory' table
PASS  refund-payment-policy: no 'shipments' table
PASS  create-shipment-policy: no 'inventory' table
PASS  create-shipment-policy: no 'payments' table
PASS  reserve-inventory-policy: no secretsmanager access
PASS  release-inventory-policy: no secretsmanager access
PASS  refund-payment-policy: no secretsmanager access
PASS  create-shipment-policy: no secretsmanager access
PASS  saga-notifier-policy: no secretsmanager access
PASS  charge-payment-policy: HAS secretsmanager:GetSecretValue
PASS  charge-payment-policy: scoped to payment/gateway-apikey
PASS  charge-payment-policy.json: no wildcard Resource on kms actions
PASS  create-shipment-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy.json: no wildcard Resource on kms actions
PASS  refund-payment-policy.json: no wildcard Resource on kms actions
PASS  release-inventory-policy.json: no wildcard Resource on kms actions
PASS  reserve-inventory-policy.json: no wildcard Resource on kms actions
PASS  saga-notifier-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy: lambda:InvokeFunction scoped to specific ARNs
PASS  saga-notifier-policy: logs-only

=== idempotency checks (direct invoke, same key twice) ===
FAIL  idempotency: reserve-inventory twice -> qty stays 5 (not 10) (expected=5 actual=None)
FAIL  idempotency: release-inventory twice -> qty back to 0 (not -5) (expected=0 actual=None)
FAIL  idempotency: refund-payment twice -> status REFUNDED (expected=REFUNDED actual=CAPTURED)

=== summary: PASS=37 FAIL=3 ===

[stdout]
=== happy path: o-1 k-1 ===
PASS  happy: state machine SUCCEEDED
PASS  happy: inventory x reserved_qty=2
PASS  happy: payments CAPTURED
PASS  happy: shipments row exists (CREATED)
PASS  happy: notifier logged saga-terminal:OrderCompleted (order o-1)

=== failure path: o-2 k-2 force_failure=true ===
PASS  failure: state machine SUCCEEDED (saga handled gracefully)
PASS  failure: inventory x reserved_qty unchanged (=2)
PASS  failure: reservation_keys does NOT contain k-2:reserve (release ran)
PASS  failure: payments row absent or REFUNDED (got: 'None')
PASS  failure: shipments row does NOT exist for ship-k-2
PASS  failure: notifier logged saga-terminal:OrderFailed (order o-2)

=== policy audit ===
PASS  reserve-inventory-policy: no 'payments' table
PASS  reserve-inventory-policy: no 'shipments' table
PASS  release-inventory-policy: no 'payments' table
PASS  release-inventory-policy: no 'shipments' table
PASS  charge-payment-policy: no 'inventory' table
PASS  charge-payment-policy: no 'shipments' table
PASS  refund-payment-policy: no 'inventory' table
PASS  refund-payment-policy: no 'shipments' table
PASS  create-shipment-policy: no 'inventory' table
PASS  create-shipment-policy: no 'payments' table
PASS  reserve-inventory-policy: no secretsmanager access
PASS  release-inventory-policy: no secretsmanager access
PASS  refund-payment-policy: no secretsmanager access
PASS  create-shipment-policy: no secretsmanager access
PASS  saga-notifier-policy: no secretsmanager access
PASS  charge-payment-policy: HAS secretsmanager:GetSecretValue
PASS  charge-payment-policy: scoped to payment/gateway-apikey
PASS  charge-payment-policy.json: no wildcard Resource on kms actions
PASS  create-shipment-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy.json: no wildcard Resource on kms actions
PASS  refund-payment-policy.json: no wildcard Resource on kms actions
PASS  release-inventory-policy.json: no wildcard Resource on kms actions
PASS  reserve-inventory-policy.json: no wildcard Resource on kms actions
PASS  saga-notifier-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy: lambda:InvokeFunction scoped to specific ARNs
PASS  saga-notifier-policy: logs-only

=== idempotency checks (direct invoke, same key twice) ===
FAIL  idempotency: reserve-inventory twice -> qty stays 5 (not 10) (expected=5 actual=None)
FAIL  idempotency: release-inventory twice -> qty back to 0 (not -5) (expected=0 actual=None)
FAIL  idempotency: refund-payment twice -> status REFUNDED (expected=REFUNDED actual=CAPTURED)

=== summary: PASS=37 FAIL=3 ===
Payload encoding bug. `--cli-binary-format raw-in-base64-out` takes raw input but I was also base64'ing it. Fix:

/app/verify.sh

− removed
invoke_lambda() {
  local fn="$1" payload="$2" out
  out=$(mktemp)
  AWS lambda invoke --function-name "$fn" \
    --payload "$(echo -n "$payload" | base64 -w0)" \
    --cli-binary-format raw-in-base64-out \
    "$out" >/dev/null 2>&1
  cat "$out"; rm -f "$out"
}
+ added
invoke_lambda() {
  local fn="$1" payload="$2" out
  out=$(mktemp)
  AWS lambda invoke --function-name "$fn" \
    --cli-binary-format raw-in-base64-out \
    --payload "$payload" \
    "$out" >/dev/null 2>&1
  cat "$out"; rm -f "$out"
}

Re-run verifier after payload fix

shell
bash /app/verify.sh 2>&1 | tail -15
output
PASS  create-shipment-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy.json: no wildcard Resource on kms actions
PASS  refund-payment-policy.json: no wildcard Resource on kms actions
PASS  release-inventory-policy.json: no wildcard Resource on kms actions
PASS  reserve-inventory-policy.json: no wildcard Resource on kms actions
PASS  saga-notifier-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy: lambda:InvokeFunction scoped to specific ARNs
PASS  saga-notifier-policy: logs-only

=== idempotency checks (direct invoke, same key twice) ===
PASS  idempotency: reserve-inventory twice -> qty stays 5 (not 10)
PASS  idempotency: release-inventory twice -> qty back to 0 (not -5)
PASS  idempotency: refund-payment twice -> status REFUNDED

=== summary: PASS=40 FAIL=0 ===

[stdout]
PASS  create-shipment-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy.json: no wildcard Resource on kms actions
PASS  refund-payment-policy.json: no wildcard Resource on kms actions
PASS  release-inventory-policy.json: no wildcard Resource on kms actions
PASS  reserve-inventory-policy.json: no wildcard Resource on kms actions
PASS  saga-notifier-policy.json: no wildcard Resource on kms actions
PASS  order-saga-policy: lambda:InvokeFunction scoped to specific ARNs
PASS  saga-notifier-policy: logs-only

=== idempotency checks (direct invoke, same key twice) ===
PASS  idempotency: reserve-inventory twice -> qty stays 5 (not 10)
PASS  idempotency: release-inventory twice -> qty back to 0 (not -5)
PASS  idempotency: refund-payment twice -> status REFUNDED

=== summary: PASS=40 FAIL=0 ===
  • Write lambda source files (6 lambdas)
  • Write IAM policies (least privilege)
  • Write KMS key policy and state machine ASL
  • Write build.sh to provision all resources
  • Write verify.sh with happy + failure assertions and policy audit
  • Run build and verify, fix issues
All 40 assertions pass. Here's what got built: **Saga orchestration** , `order-saga` state machine runs `ReserveInventory → ChargePayment → CreateShipment`. Each forward step has a `Catch` that branches into compensations in **reverse order** of what ran: - Reserve fails → no compensation - Charge fails → `ReleaseAfterChargeFail` - Ship fails → `RefundAfterShipFail → ReleaseAfterShipFail` Both terminal states (`PublishCompleted` / `PublishFailed`) use SFN's `events:putEvents` direct integration to push `OrderCompleted` / `OrderFailed` with source `order.saga` onto the `saga-events` bus. **Idempotency** , atomic DDB conditional writes tied to `idempotency_key`: - `reserve-inventory`: `ADD reserved_qty :q, reservation_keys :kset` with `NOT contains(reservation_keys, :k)` - `release-inventory`: `ADD :negq DELETE :kset` with `contains(...)` , safe to retry - `charge-payment` / `create-shipment`: conditional `put_item` on `attribute_not_exists(pk)` - `refund-payment`: conditional update with `#s <> :r` **Secrets** , customer-managed KMS CMK `alias/saga-cmk` encrypts the `payment/gateway-apikey` secret. Only `charge-payment-role` has `secretsmanager:GetSecretValue` on that specific secret and `kms:Decrypt` on the specific CMK alias. **Least privilege** , each lambda role scopes DDB to its one business table; `saga-notifier-role` is logs-only; `order-saga-role` enumerates the 5 invoked lambda ARNs (no `*`). **Files:** - `/app/lambdas/*.py` , 6 handlers - `/app/iam/*.json` , 7 policies + 2 trust docs - `/app/kms/key-policy.json` , CMK policy - `/app/state-machine/order-saga.json` , ASL - `/app/event/rule-pattern.json` , EventBridge pattern - `/app/build.sh` , idempotent provisioning - `/app/verify.sh` , **PASS=40 FAIL=0** Run `bash /app/build.sh && bash /app/verify.sh` to reproduce.
Final diff , what the agent changed

No file changes, the agent left the workspace untouched.

Verifier output , the proof behind the reward
tests/test.sh
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/Gq8y36yJZny4rxO0OqKTO/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 20 items

../tests/test_outputs.py::test_01_cmk_alias_and_secret_exist PASSED      [  5%]
../tests/test_outputs.py::test_02_all_four_tables_exist PASSED           [ 10%]
../tests/test_outputs.py::test_03_all_six_lambdas_exist PASSED           [ 15%]
../tests/test_outputs.py::test_04_state_machine_and_bus_exist PASSED     [ 20%]
../tests/test_outputs.py::test_05_terminal_rule_exists_on_saga_bus PASSED [ 25%]
../tests/test_outputs.py::test_06_every_forward_state_has_catch_to_compensation PASSED [ 30%]
../tests/test_outputs.py::test_07_compensations_run_in_lifo_order PASSED [ 35%]
../tests/test_outputs.py::test_08_compensation_states_preserve_error_via_result_path PASSED [ 40%]
../tests/test_outputs.py::test_09_idempotency_table_has_ttl_and_correct_pk PASSED [ 45%]
../tests/test_outputs.py::test_10_each_lambda_role_scoped_to_its_own_table PASSED [ 50%]
../tests/test_outputs.py::test_11_only_charge_role_reads_the_payment_secret PASSED [ 55%]
../tests/test_outputs.py::test_12_saga_role_lambda_invoke_not_wildcarded PASSED [ 60%]
../tests/test_outputs.py::test_13_rule_pattern_matches_both_terminal_types_with_array_wrap PASSED [ 65%]
../tests/test_outputs.py::test_14_charge_role_kms_decrypt_scoped_not_wildcard PASSED [ 70%]
../tests/test_outputs.py::test_15_rule_targets_notifier_lambda PASSED    [ 75%]
../tests/test_outputs.py::test_16_notifier_role_is_least_privilege_no_ddb_kms_secrets PASSED [ 80%]
../tests/test_outputs.py::test_17_notifier_lambda_resource_policy_allows_events_with_source_arn PASSED [ 85%]
../tests/test_outputs.py::test_18_happy_path_completes_and_writes_all_three_tables PASSED [ 90%]
../tests/test_outputs.py::test_19_failure_path_triggers_compensation_and_rolls_back_inventory PASSED [ 95%]
../tests/test_outputs.py::test_20_idempotency_prevents_double_reservation PASSED [100%]

=============================== warnings summary ===============================
test_outputs.py: 134 warnings
  /root/.cache/uv/archive-v0/Gq8y36yJZny4rxO0OqKTO/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_outputs.py::test_01_cmk_alias_and_secret_exist
PASSED ../tests/test_outputs.py::test_02_all_four_tables_exist
PASSED ../tests/test_outputs.py::test_03_all_six_lambdas_exist
PASSED ../tests/test_outputs.py::test_04_state_machine_and_bus_exist
PASSED ../tests/test_outputs.py::test_05_terminal_rule_exists_on_saga_bus
PASSED ../tests/test_outputs.py::test_06_every_forward_state_has_catch_to_compensation
PASSED ../tests/test_outputs.py::test_07_compensations_run_in_lifo_order
PASSED ../tests/test_outputs.py::test_08_compensation_states_preserve_error_via_result_path
PASSED ../tests/test_outputs.py::test_09_idempotency_table_has_ttl_and_correct_pk
PASSED ../tests/test_outputs.py::test_10_each_lambda_role_scoped_to_its_own_table
PASSED ../tests/test_outputs.py::test_11_only_charge_role_reads_the_payment_secret
PASSED ../tests/test_outputs.py::test_12_saga_role_lambda_invoke_not_wildcarded
PASSED ../tests/test_outputs.py::test_13_rule_pattern_matches_both_terminal_types_with_array_wrap
PASSED ../tests/test_outputs.py::test_14_charge_role_kms_decrypt_scoped_not_wildcard
PASSED ../tests/test_outputs.py::test_15_rule_targets_notifier_lambda
PASSED ../tests/test_outputs.py::test_16_notifier_role_is_least_privilege_no_ddb_kms_secrets
PASSED ../tests/test_outputs.py::test_17_notifier_lambda_resource_policy_allows_events_with_source_arn
PASSED ../tests/test_outputs.py::test_18_happy_path_completes_and_writes_all_three_tables
PASSED ../tests/test_outputs.py::test_19_failure_path_triggers_compensation_and_rolls_back_inventory
PASSED ../tests/test_outputs.py::test_20_idempotency_prevents_double_reservation
====================== 20 passed, 134 warnings in 20.93s =======================

Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_dab3af4409c84c70. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.

Trial trial_dab3af4409c84c70 · verifier authoritative; classifier explanatory.