SyncValsverifier → artifact → classifier → verdict
SyncVals · Trajectory

sfn-saga-compensation-orchestrator

claude-code claude-opus-4-7 ✗ failed BAD_FAILURE ↑ View task
Solved from the instruction alone, tests/ and solution/ were withheld from the agent's workspace and restored only for grading.
Reward = tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.
Classification , post-hoc; cannot change the reward
BAD_FAILUREThe task is at fault, underspecified/contradictory instruction, brittle/flaky tests, or tests demanding undiscoverable behavior.
SubtypeRigid/Brittle Tests - Specification Contradiction
EvidenceTest `test_16_notifier_role_is_least_privilege_no_ddb_kms_secrets` fails with: 'saga-notifier-role has forbidden action events:PutEvents - only logs are needed'. However, instruction.md line 32 explicitly states 'notifier role has nothing but eventbridge + logs'. The instruction's lines 12 and 61 make clear the notifier lambda must receive events from EventBridge (via the event bus) and write logs. The test forbids 'events:PutEvents' which contradicts the instruction's requirement for eventbridge permissions. Agent's implementation passes 19/20 tests including all functional behavior tests (happy path, failure compensation, idempotency, etc.).
Root causeThe test specification contradicts the instruction specification. The instruction requires the notifier role to have EventBridge permissions ('eventbridge + logs'), but the test forbids events:PutEvents. The agent correctly implemented what the instruction specified, resulting in a brittle test failure that masks correct functional implementation.
RecommendationRemove 'events:PutEvents' from the forbidden_prefixes list in test_16, or clarify the specification. Either: (1) the notifier role should have EventBridge permissions to receive/respond to events (per instruction), or (2) clarify that it should ONLY have CloudWatch Logs and rely on a resource-based policy for EventBridge invocation (without IAM role permissions). The current test and instruction contradict each other. The agent's implementation is correct per the stated instruction."
Trajectory
Tool-by-tool agent trajectory
34 tool calls · 4 tool types · 42 steps
so the order service keeps leaving garbage rows. inventory gets reserved, payment call times out, we retry, now inventory is double-reserved because the retry worked the second time. same deal when shipping fails after payment , we'd just charge the customer and leave them with no shipment. the ops slack is basically me apologizing at this point. we need a real saga. if any step fails, **undo** the earlier steps in reverse order. no half-states. step functions runs it, localstack , `http://localstack:4566`, creds already exported (`AWS_ACCESS_KEY_ID=test`, same for secret, region `us-east-1`). `aws`, `python3`, `boto3`, `jq`, `zip`. build from zero. shape of it: - a state machine runs three forward steps: **reserve inventory**, **charge payment**, **create shipment**. each step is its own lambda. each step writes to its own ddb table. - if any step fails, the saga fires compensations **in reverse order** of what actually ran. if reserve succeeded but charge failed, you refund (no-op, nothing was charged) and release. if charge succeeded but shipment failed, you refund payment and release inventory. if reserve failed, nothing to undo. - compensations are their own lambdas too. they must be idempotent , if state-function retries a compensation, running it twice should not corrupt the table. - each forward step gets an idempotency key from the input so a lambda retry inside a single step doesn't double-decrement inventory. - payment calls a "gateway" that needs an api key. store the key in secrets manager , customer-managed kms key encrypts the secret. the payment lambda reads the secret, nobody else should be able to. - on terminal state (completed or failed-compensated), the saga publishes an event to a custom event bus. a notifier lambda sits on that bus and writes one log line starting with `saga-terminal:`. to actually prove compensations work, the charge-payment lambda accepts a `force_failure` flag in its input , when true, it raises. the verifier uses this to drive a failure run and assert the state snaps back. done looks like this. two executions: **happy path** , `start-execution` with `{"order_id": "o-1", "sku": "x", "qty": 2, "amount": 100, "idempotency_key": "k-1"}`: - sm reaches `SUCCEEDED` within 60s - inventory row for sku `x` reflects `reserved_qty = 2` - payments row exists with status `CAPTURED` - shipments row exists - notifier log has `saga-terminal:OrderCompleted` **failure path** , same input but `"force_failure": true, "order_id": "o-2", "idempotency_key": "k-2"`: - sm still reaches `SUCCEEDED` (saga handles it gracefully, not a hard abort) - inventory reserved_qty back to 0 (compensation ran) - payments row is absent or `REFUNDED` (nothing was charged since charge failed, so a no-op refund is also ok) - shipments row does **not** exist - notifier log has `saga-terminal:OrderFailed` least privilege. the verifier reads policy docs directly. each lambda role only gets its own table. saga role can't wildcard on lambda invoke. payment role is the only one that reads the secret. notifier role has nothing but eventbridge + logs. no `*` on kms anywhere. names the verifier grabs by hand, use them exactly: | thing | name | |-|-| | kms alias | `alias/saga-cmk` | | secret | `payment/gateway-apikey` | | inventory table | `inventory` (pk `sku` / String) | | payments table | `payments` (pk `payment_id` / String) | | shipments table | `shipments` (pk `shipment_id` / String) | | idempotency table | `saga-idempotency` (pk `idempotency_key` / String, ttl enabled) | | reserve lambda | `reserve-inventory` | | charge lambda | `charge-payment` | | ship lambda | `create-shipment` | | release compensation | `release-inventory` | | refund compensation | `refund-payment` | | notifier lambda | `saga-notifier` | | reserve role | `reserve-inventory-role` | | charge role | `charge-payment-role` | | ship role | `create-shipment-role` | | release role | `release-inventory-role` | | refund role | `refund-payment-role` | | notifier role | `saga-notifier-role` | | state machine | `order-saga` | | state machine role | `order-saga-role` | | event bus | `saga-events` | | event rule | `on-saga-terminal` | eventbridge detail shape: source `order.saga`, detail-type is one of `OrderCompleted` or `OrderFailed`. the rule matches both and routes to the notifier lambda. python3.11 for all lambdas. localstack community 4.13.1. the "payment gateway" is fake , charge-payment reads the secret, logs that it did, and marks the payments row CAPTURED. no actual network call.

List app directory contents

shell
ls /app && pwd
output
/app

[stdout]
/app

Verify localstack connectivity

shell
aws --endpoint-url=http://localstack:4566 sts get-caller-identity --region us-east-1
output
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}

[stdout]
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}
I'll build this end-to-end. Let me first set up a working directory and plan the artifacts.

Check existing lambdas

shell
aws --endpoint-url=http://localstack:4566 --region us-east-1 lambda list-functions --query 'Functions[].FunctionName' 2>&1 | head -40
output
[]

[stdout]
[]

Create working directories

shell
mkdir -p /app/lambdas /app/iam /app/sm && ls /app
output
iam
lambdas
sm

[stdout]
iam
lambdas
sm
Plan: I'll lay down lambda code, IAM policies, the state machine, a setup script, and a verify script. Let me write all files in parallel where possible.
write · /app/lambdas/reserve_inventory.py
import os
import time
import boto3
from botocore.exceptions import ClientError


def _endpoint():
    if os.environ.get("LOCALSTACK_HOSTNAME"):
        return f"http://{os.environ['LOCALSTACK_HOSTNAME']}:{os.environ.get('EDGE_PORT', '4566')}"
    return os.environ.get("AWS_ENDPOINT_URL")


_EP = _endpoint()
_ddb = boto3.resource("dynamodb", endpoint_url=_EP) if _EP else boto3.resource("dynamodb")
_inventory = _ddb.Table("inventory")
_idem = _ddb.Table("saga-idempotency")

_TTL = 3600


def _claim(key, action):
    full = f"{key}:{action}"
    try:
        _idem.put_item(
            Item={"idempotency_key": full, "expires_at": int(time.time()) + _TTL, "action": action},
            ConditionExpression="attribute_not_exists(idempotency_key)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise


def lambda_handler(event, context):
    key = event["idempotency_key"]
    sku = event["sku"]
    qty = int(event["qty"])
    if _claim(key, "reserve"):
        _inventory.update_item(
            Key={"sku": sku},
            UpdateExpression="ADD reserved_qty :q",
            ExpressionAttributeValues={":q": qty},
        )
    return {"status": "reserved", "sku": sku, "qty": qty}
write · /app/lambdas/charge_payment.py
import os
import time
import boto3
from botocore.exceptions import ClientError


def _endpoint():
    if os.environ.get("LOCALSTACK_HOSTNAME"):
        return f"http://{os.environ['LOCALSTACK_HOSTNAME']}:{os.environ.get('EDGE_PORT', '4566')}"
    return os.environ.get("AWS_ENDPOINT_URL")


_EP = _endpoint()
_ddb = boto3.resource("dynamodb", endpoint_url=_EP) if _EP else boto3.resource("dynamodb")
_sm = boto3.client("secretsmanager", endpoint_url=_EP) if _EP else boto3.client("secretsmanager")
_payments = _ddb.Table("payments")
_idem = _ddb.Table("saga-idempotency")

_SECRET_NAME = os.environ.get("SECRET_NAME", "payment/gateway-apikey")
_TTL = 3600


def _claim(key, action):
    full = f"{key}:{action}"
    try:
        _idem.put_item(
            Item={"idempotency_key": full, "expires_at": int(time.time()) + _TTL, "action": action},
            ConditionExpression="attribute_not_exists(idempotency_key)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise


def lambda_handler(event, context):
    if event.get("force_failure"):
        raise RuntimeError("forced payment gateway failure")

    key = event["idempotency_key"]
    payment_id = f"pay-{event['order_id']}"

    secret = _sm.get_secret_value(SecretId=_SECRET_NAME)
    print(f"loaded payment gateway secret bytes={len(secret['SecretString'])}")

    if _claim(key, "charge"):
        _payments.put_item(Item={
            "payment_id": payment_id,
            "order_id": event["order_id"],
            "amount": int(event["amount"]),
            "status": "CAPTURED",
        })
    return {"status": "charged", "payment_id": payment_id}
write · /app/lambdas/create_shipment.py
import os
import time
import boto3
from botocore.exceptions import ClientError


def _endpoint():
    if os.environ.get("LOCALSTACK_HOSTNAME"):
        return f"http://{os.environ['LOCALSTACK_HOSTNAME']}:{os.environ.get('EDGE_PORT', '4566')}"
    return os.environ.get("AWS_ENDPOINT_URL")


_EP = _endpoint()
_ddb = boto3.resource("dynamodb", endpoint_url=_EP) if _EP else boto3.resource("dynamodb")
_shipments = _ddb.Table("shipments")
_idem = _ddb.Table("saga-idempotency")

_TTL = 3600


def _claim(key, action):
    full = f"{key}:{action}"
    try:
        _idem.put_item(
            Item={"idempotency_key": full, "expires_at": int(time.time()) + _TTL, "action": action},
            ConditionExpression="attribute_not_exists(idempotency_key)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise


def lambda_handler(event, context):
    if event.get("force_shipment_failure"):
        raise RuntimeError("forced shipment failure")

    key = event["idempotency_key"]
    shipment_id = f"ship-{event['order_id']}"
    if _claim(key, "ship"):
        _shipments.put_item(Item={
            "shipment_id": shipment_id,
            "order_id": event["order_id"],
            "sku": event["sku"],
            "qty": int(event["qty"]),
            "status": "CREATED",
        })
    return {"status": "shipped", "shipment_id": shipment_id}
write · /app/lambdas/release_inventory.py
import os
import time
import boto3
from botocore.exceptions import ClientError


def _endpoint():
    if os.environ.get("LOCALSTACK_HOSTNAME"):
        return f"http://{os.environ['LOCALSTACK_HOSTNAME']}:{os.environ.get('EDGE_PORT', '4566')}"
    return os.environ.get("AWS_ENDPOINT_URL")


_EP = _endpoint()
_ddb = boto3.resource("dynamodb", endpoint_url=_EP) if _EP else boto3.resource("dynamodb")
_inventory = _ddb.Table("inventory")
_idem = _ddb.Table("saga-idempotency")

_TTL = 3600


def _claim(key, action):
    full = f"{key}:{action}"
    try:
        _idem.put_item(
            Item={"idempotency_key": full, "expires_at": int(time.time()) + _TTL, "action": action},
            ConditionExpression="attribute_not_exists(idempotency_key)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise


def lambda_handler(event, context):
    key = event["idempotency_key"]
    sku = event["sku"]
    qty = int(event["qty"])
    if _claim(key, "release"):
        reserve_marker = _idem.get_item(Key={"idempotency_key": f"{key}:reserve"})
        if "Item" in reserve_marker:
            _inventory.update_item(
                Key={"sku": sku},
                UpdateExpression="ADD reserved_qty :q",
                ExpressionAttributeValues={":q": -qty},
            )
    return {"status": "released", "sku": sku, "qty": qty}
write · /app/lambdas/refund_payment.py
import os
import time
import boto3
from botocore.exceptions import ClientError


def _endpoint():
    if os.environ.get("LOCALSTACK_HOSTNAME"):
        return f"http://{os.environ['LOCALSTACK_HOSTNAME']}:{os.environ.get('EDGE_PORT', '4566')}"
    return os.environ.get("AWS_ENDPOINT_URL")


_EP = _endpoint()
_ddb = boto3.resource("dynamodb", endpoint_url=_EP) if _EP else boto3.resource("dynamodb")
_payments = _ddb.Table("payments")
_idem = _ddb.Table("saga-idempotency")

_TTL = 3600


def _claim(key, action):
    full = f"{key}:{action}"
    try:
        _idem.put_item(
            Item={"idempotency_key": full, "expires_at": int(time.time()) + _TTL, "action": action},
            ConditionExpression="attribute_not_exists(idempotency_key)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise


def lambda_handler(event, context):
    key = event["idempotency_key"]
    payment_id = f"pay-{event['order_id']}"
    if _claim(key, "refund"):
        try:
            _payments.update_item(
                Key={"payment_id": payment_id},
                UpdateExpression="SET #s = :r",
                ConditionExpression="attribute_exists(payment_id) AND #s = :c",
                ExpressionAttributeNames={"#s": "status"},
                ExpressionAttributeValues={":r": "REFUNDED", ":c": "CAPTURED"},
            )
        except ClientError as e:
            if e.response["Error"]["Code"] != "ConditionalCheckFailedException":
                raise
    return {"status": "refunded", "payment_id": payment_id}
write · /app/lambdas/saga_notifier.py
def lambda_handler(event, context):
    detail_type = event.get("detail-type", "Unknown")
    print(f"saga-terminal:{detail_type}")
    return {"logged": True, "detail_type": detail_type}
write · /app/iam/lambda-trust.json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "lambda.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
write · /app/iam/states-trust.json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "states.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
write · /app/iam/kms-key-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RootAdminExplicit",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::000000000000:root"},
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncryptFrom",
        "kms:ReEncryptTo",
        "kms:GenerateDataKey",
        "kms:GenerateDataKeyWithoutPlaintext",
        "kms:DescribeKey",
        "kms:GetKeyPolicy",
        "kms:PutKeyPolicy",
        "kms:CreateAlias",
        "kms:DeleteAlias",
        "kms:UpdateAlias",
        "kms:TagResource",
        "kms:UntagResource",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion",
        "kms:EnableKey",
        "kms:DisableKey",
        "kms:ListGrants",
        "kms:CreateGrant",
        "kms:RevokeGrant",
        "kms:RetireGrant",
        "kms:GetKeyRotationStatus",
        "kms:EnableKeyRotation",
        "kms:DisableKeyRotation"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowSecretsManagerService",
      "Effect": "Allow",
      "Principal": {"Service": "secretsmanager.amazonaws.com"},
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:ReEncryptFrom",
        "kms:ReEncryptTo",
        "kms:DescribeKey",
        "kms:CreateGrant"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {"kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"}
      }
    },
    {
      "Sid": "AllowChargePaymentRoleDecrypt",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::000000000000:role/charge-payment-role"},
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {"kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"}
      }
    }
  ]
}
write · /app/iam/policy-reserve.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["dynamodb:UpdateItem", "dynamodb:GetItem"],
      "Resource": "__INV_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["dynamodb:PutItem", "dynamodb:GetItem"],
      "Resource": "__IDEM_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:__REG__:__ACCT__:log-group:/aws/lambda/reserve-inventory:*"
    }
  ]
}
write · /app/iam/policy-charge.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["dynamodb:PutItem", "dynamodb:GetItem"],
      "Resource": "__PAY_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["dynamodb:PutItem", "dynamodb:GetItem"],
      "Resource": "__IDEM_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "__SECRET_ARN__"
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "__KMS_ARN__"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:__REG__:__ACCT__:log-group:/aws/lambda/charge-payment:*"
    }
  ]
}
write · /app/iam/policy-ship.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["dynamodb:PutItem", "dynamodb:GetItem"],
      "Resource": "__SHIP_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["dynamodb:PutItem", "dynamodb:GetItem"],
      "Resource": "__IDEM_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:__REG__:__ACCT__:log-group:/aws/lambda/create-shipment:*"
    }
  ]
}
write · /app/iam/policy-release.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["dynamodb:UpdateItem", "dynamodb:GetItem"],
      "Resource": "__INV_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["dynamodb:PutItem", "dynamodb:GetItem"],
      "Resource": "__IDEM_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:__REG__:__ACCT__:log-group:/aws/lambda/release-inventory:*"
    }
  ]
}
write · /app/iam/policy-refund.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["dynamodb:UpdateItem", "dynamodb:GetItem"],
      "Resource": "__PAY_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["dynamodb:PutItem", "dynamodb:GetItem"],
      "Resource": "__IDEM_TABLE__"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:__REG__:__ACCT__:log-group:/aws/lambda/refund-payment:*"
    }
  ]
}
write · /app/iam/policy-notifier.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["events:PutEvents"],
      "Resource": "__BUS_ARN__"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:__REG__:__ACCT__:log-group:/aws/lambda/saga-notifier:*"
    }
  ]
}
write · /app/iam/policy-saga.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["lambda:InvokeFunction"],
      "Resource": [
        "arn:aws:lambda:__REG__:__ACCT__:function:reserve-inventory",
        "arn:aws:lambda:__REG__:__ACCT__:function:charge-payment",
        "arn:aws:lambda:__REG__:__ACCT__:function:create-shipment",
        "arn:aws:lambda:__REG__:__ACCT__:function:release-inventory",
        "arn:aws:lambda:__REG__:__ACCT__:function:refund-payment"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["events:PutEvents"],
      "Resource": "__BUS_ARN__"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:CreateLogDelivery",
        "logs:GetLogDelivery",
        "logs:UpdateLogDelivery",
        "logs:DeleteLogDelivery",
        "logs:ListLogDeliveries",
        "logs:DescribeLogGroups",
        "logs:DescribeResourcePolicies",
        "logs:PutResourcePolicy"
      ],
      "Resource": "arn:aws:logs:__REG__:__ACCT__:log-group:/aws/states/order-saga*"
    }
  ]
}
write · /app/sm/order-saga.json
{
  "Comment": "Order saga: reserve inventory -> charge payment -> create shipment, with reverse-order compensations on failure.",
  "StartAt": "ReserveInventory",
  "States": {
    "ReserveInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "reserve-inventory",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.reserve_result",
      "Retry": [{
        "ErrorEquals": ["Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.SdkClientException", "Lambda.TooManyRequestsException"],
        "IntervalSeconds": 1,
        "MaxAttempts": 2,
        "BackoffRate": 2
      }],
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "ResultPath": "$.error",
        "Next": "PublishOrderFailedNoCompensation"
      }],
      "Next": "ChargePayment"
    },
    "ChargePayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "charge-payment",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.charge_result",
      "Retry": [{
        "ErrorEquals": ["Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.SdkClientException", "Lambda.TooManyRequestsException"],
        "IntervalSeconds": 1,
        "MaxAttempts": 2,
        "BackoffRate": 2
      }],
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "ResultPath": "$.error",
        "Next": "ReleaseInventoryAfterChargeFail"
      }],
      "Next": "CreateShipment"
    },
    "CreateShipment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "create-shipment",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.shipment_result",
      "Retry": [{
        "ErrorEquals": ["Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.SdkClientException", "Lambda.TooManyRequestsException"],
        "IntervalSeconds": 1,
        "MaxAttempts": 2,
        "BackoffRate": 2
      }],
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "ResultPath": "$.error",
        "Next": "RefundPaymentAfterShipFail"
      }],
      "Next": "PublishOrderCompleted"
    },
    "ReleaseInventoryAfterChargeFail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "release-inventory",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.release_result",
      "Next": "PublishOrderFailed"
    },
    "RefundPaymentAfterShipFail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "refund-payment",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.refund_result",
      "Next": "ReleaseInventoryAfterShipFail"
    },
    "ReleaseInventoryAfterShipFail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "release-inventory",
        "Payload.$": "$"
      },
      "ResultSelector": {"payload.$": "$.Payload"},
      "ResultPath": "$.release_result",
      "Next": "PublishOrderFailed"
    },
    "PublishOrderCompleted": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [{
          "Source": "order.saga",
          "DetailType": "OrderCompleted",
          "EventBusName": "saga-events",
          "Detail.$": "States.JsonToString($)"
        }]
      },
      "End": true
    },
    "PublishOrderFailed": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [{
          "Source": "order.saga",
          "DetailType": "OrderFailed",
          "EventBusName": "saga-events",
          "Detail.$": "States.JsonToString($)"
        }]
      },
      "End": true
    },
    "PublishOrderFailedNoCompensation": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [{
          "Source": "order.saga",
          "DetailType": "OrderFailed",
          "EventBusName": "saga-events",
          "Detail.$": "States.JsonToString($)"
        }]
      },
      "End": true
    }
  }
}
ToolSearch
{
  "query": "select:TodoWrite",
  "max_results": 1
}

, output, 
{"type": "tool_reference", "tool_name": "TodoWrite"}
  • Write setup script and deploy infra
  • Run happy path test
  • Run failure path test
write · /app/setup.sh
#!/usr/bin/env bash
set -euo pipefail

EP=http://localstack:4566
REG=us-east-1
ACCT=000000000000
AWS="aws --endpoint-url=$EP --region $REG"

LDIR=/app/lambdas
IDIR=/app/iam
SDIR=/app/sm
TMP=/tmp/saga-build
mkdir -p "$TMP/policies"

echo "=== Roles ==="
for r in reserve-inventory charge-payment create-shipment release-inventory refund-payment saga-notifier; do
  $AWS iam create-role --role-name "${r}-role" \
    --assume-role-policy-document "file://$IDIR/lambda-trust.json" >/dev/null 2>&1 || true
done
$AWS iam create-role --role-name order-saga-role \
  --assume-role-policy-document "file://$IDIR/states-trust.json" >/dev/null 2>&1 || true

echo "=== KMS key + alias ==="
EXISTING_ALIAS=$($AWS kms list-aliases --query "Aliases[?AliasName=='alias/saga-cmk'].TargetKeyId | [0]" --output text 2>/dev/null || echo "None")
if [ "$EXISTING_ALIAS" = "None" ] || [ -z "$EXISTING_ALIAS" ]; then
  KEY_ID=$($AWS kms create-key --description "saga CMK" \
    --policy "file://$IDIR/kms-key-policy.json" \
    --query 'KeyMetadata.KeyId' --output text)
  $AWS kms create-alias --alias-name alias/saga-cmk --target-key-id "$KEY_ID" >/dev/null
else
  KEY_ID="$EXISTING_ALIAS"
  $AWS kms put-key-policy --key-id "$KEY_ID" --policy-name default \
    --policy "file://$IDIR/kms-key-policy.json" >/dev/null
fi
KEY_ARN="arn:aws:kms:${REG}:${ACCT}:key/${KEY_ID}"
echo "KEY_ARN=$KEY_ARN"

echo "=== Secret ==="
$AWS secretsmanager create-secret --name payment/gateway-apikey \
  --secret-string '{"api_key":"sk_test_fake_gateway_apikey"}' \
  --kms-key-id alias/saga-cmk >/dev/null 2>&1 || true
SECRET_ARN=$($AWS secretsmanager describe-secret --secret-id payment/gateway-apikey --query ARN --output text)
echo "SECRET_ARN=$SECRET_ARN"

echo "=== DynamoDB tables ==="
$AWS dynamodb create-table --table-name inventory \
  --attribute-definitions AttributeName=sku,AttributeType=S \
  --key-schema AttributeName=sku,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST >/dev/null 2>&1 || true
$AWS dynamodb create-table --table-name payments \
  --attribute-definitions AttributeName=payment_id,AttributeType=S \
  --key-schema AttributeName=payment_id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST >/dev/null 2>&1 || true
$AWS dynamodb create-table --table-name shipments \
  --attribute-definitions AttributeName=shipment_id,AttributeType=S \
  --key-schema AttributeName=shipment_id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST >/dev/null 2>&1 || true
$AWS dynamodb create-table --table-name saga-idempotency \
  --attribute-definitions AttributeName=idempotency_key,AttributeType=S \
  --key-schema AttributeName=idempotency_key,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST >/dev/null 2>&1 || true
$AWS dynamodb update-time-to-live --table-name saga-idempotency \
  --time-to-live-specification "Enabled=true,AttributeName=expires_at" >/dev/null 2>&1 || true

echo "=== Event bus ==="
$AWS events create-event-bus --name saga-events >/dev/null 2>&1 || true

echo "=== Render & attach IAM permission policies ==="
INV_TABLE="arn:aws:dynamodb:${REG}:${ACCT}:table/inventory"
PAY_TABLE="arn:aws:dynamodb:${REG}:${ACCT}:table/payments"
SHIP_TABLE="arn:aws:dynamodb:${REG}:${ACCT}:table/shipments"
IDEM_TABLE="arn:aws:dynamodb:${REG}:${ACCT}:table/saga-idempotency"
BUS_ARN="arn:aws:events:${REG}:${ACCT}:event-bus/saga-events"

render() {
  sed \
    -e "s|__KMS_ARN__|$KEY_ARN|g" \
    -e "s|__SECRET_ARN__|$SECRET_ARN|g" \
    -e "s|__INV_TABLE__|$INV_TABLE|g" \
    -e "s|__PAY_TABLE__|$PAY_TABLE|g" \
    -e "s|__SHIP_TABLE__|$SHIP_TABLE|g" \
    -e "s|__IDEM_TABLE__|$IDEM_TABLE|g" \
    -e "s|__BUS_ARN__|$BUS_ARN|g" \
    -e "s|__ACCT__|$ACCT|g" \
    -e "s|__REG__|$REG|g" \
    "$1"
}

for p in reserve charge ship release refund notifier saga; do
  render "$IDIR/policy-${p}.json" > "$TMP/policies/policy-${p}.json"
done

attach() {
  local role=$1; local policy=$2
  $AWS iam put-role-policy --role-name "$role" --policy-name "${role}-policy" \
    --policy-document "file://$TMP/policies/policy-${policy}.json" >/dev/null
}
attach reserve-inventory-role  reserve
attach charge-payment-role     charge
attach create-shipment-role    ship
attach release-inventory-role  release
attach refund-payment-role     refund
attach saga-notifier-role      notifier
attach order-saga-role         saga

echo "=== Build & deploy lambdas ==="
build_zip() {
  local name=$1; local file=$2
  rm -f "$TMP/${name}.zip"
  (cd "$LDIR" && zip -q -j "$TMP/${name}.zip" "${file}")
}

deploy() {
  local fn=$1; local file=$2; local role=$3
  build_zip "$fn" "$file"
  $AWS lambda delete-function --function-name "$fn" >/dev/null 2>&1 || true
  $AWS lambda create-function \
    --function-name "$fn" \
    --runtime python3.11 \
    --role "arn:aws:iam::${ACCT}:role/${role}" \
    --handler "$(basename "$file" .py).lambda_handler" \
    --zip-file "fileb://$TMP/${fn}.zip" \
    --timeout 30 \
    --environment "Variables={SECRET_NAME=payment/gateway-apikey,AWS_ENDPOINT_URL=$EP}" \
    >/dev/null
}

deploy reserve-inventory  reserve_inventory.py  reserve-inventory-role
deploy charge-payment     charge_payment.py     charge-payment-role
deploy create-shipment    create_shipment.py    create-shipment-role
deploy release-inventory  release_inventory.py  release-inventory-role
deploy refund-payment     refund_payment.py     refund-payment-role
deploy saga-notifier      saga_notifier.py      saga-notifier-role

for fn in reserve-inventory charge-payment create-shipment release-inventory refund-payment saga-notifier; do
  for i in $(seq 1 60); do
    state=$($AWS lambda get-function --function-name "$fn" --query 'Configuration.State' --output text 2>/dev/null || echo "Pending")
    if [ "$state" = "Active" ]; then break; fi
    sleep 1
  done
  echo "  $fn -> $state"
done

echo "=== EventBridge rule + notifier target ==="
$AWS events put-rule \
  --name on-saga-terminal \
  --event-bus-name saga-events \
  --event-pattern '{"source":["order.saga"],"detail-type":["OrderCompleted","OrderFailed"]}' >/dev/null
NOTIFIER_ARN="arn:aws:lambda:${REG}:${ACCT}:function:saga-notifier"
$AWS events put-targets --rule on-saga-terminal --event-bus-name saga-events \
  --targets "Id=1,Arn=$NOTIFIER_ARN" >/dev/null

RULE_ARN="arn:aws:events:${REG}:${ACCT}:rule/saga-events/on-saga-terminal"
$AWS lambda remove-permission --function-name saga-notifier --statement-id allow-eventbridge >/dev/null 2>&1 || true
$AWS lambda add-permission \
  --function-name saga-notifier \
  --statement-id allow-eventbridge \
  --action lambda:InvokeFunction \
  --principal events.amazonaws.com \
  --source-arn "$RULE_ARN" >/dev/null

echo "=== State machine ==="
SM_ARN_EXISTING=$($AWS stepfunctions list-state-machines --query "stateMachines[?name=='order-saga'].stateMachineArn | [0]" --output text 2>/dev/null || echo "None")
if [ "$SM_ARN_EXISTING" != "None" ] && [ -n "$SM_ARN_EXISTING" ]; then
  $AWS stepfunctions delete-state-machine --state-machine-arn "$SM_ARN_EXISTING" >/dev/null
  sleep 1
fi
SM_ARN=$($AWS stepfunctions create-state-machine \
  --name order-saga \
  --definition "file://$SDIR/order-saga.json" \
  --role-arn "arn:aws:iam::${ACCT}:role/order-saga-role" \
  --query stateMachineArn --output text)
echo "SM_ARN=$SM_ARN"
echo "$SM_ARN" > /app/.sm_arn
echo "$KEY_ARN" > /app/.kms_arn
echo "$SECRET_ARN" > /app/.secret_arn
echo "DONE"

Run setup script

shell
chmod +x /app/setup.sh && bash /app/setup.sh 2>&1 | tail -80
output
=== Roles ===
=== KMS key + alias ===
KEY_ARN=arn:aws:kms:us-east-1:000000000000:key/baa15dbe-4c66-4f0f-8539-c347e51b228e
=== Secret ===
SECRET_ARN=arn:aws:secretsmanager:us-east-1:000000000000:secret:payment/gateway-apikey-alERUT
=== DynamoDB tables ===
=== Event bus ===
=== Render & attach IAM permission policies ===
=== Build & deploy lambdas ===
  reserve-inventory -> Active
  charge-payment -> Active
  create-shipment -> Active
  release-inventory -> Active
  refund-payment -> Active
  saga-notifier -> Active
=== EventBridge rule + notifier target ===
=== State machine ===
SM_ARN=arn:aws:states:us-east-1:000000000000:stateMachine:order-saga
DONE

[stdout]
=== Roles ===
=== KMS key + alias ===
KEY_ARN=arn:aws:kms:us-east-1:000000000000:key/baa15dbe-4c66-4f0f-8539-c347e51b228e
=== Secret ===
SECRET_ARN=arn:aws:secretsmanager:us-east-1:000000000000:secret:payment/gateway-apikey-alERUT
=== DynamoDB tables ===
=== Event bus ===
=== Render & attach IAM permission policies ===
=== Build & deploy lambdas ===
  reserve-inventory -> Active
  charge-payment -> Active
  create-shipment -> Active
  release-inventory -> Active
  refund-payment -> Active
  saga-notifier -> Active
=== EventBridge rule + notifier target ===
=== State machine ===
SM_ARN=arn:aws:states:us-east-1:000000000000:stateMachine:order-saga
DONE
Infra is up. Now let me run the happy path execution.

Run happy path execution

shell
SM_ARN=$(cat /app/.sm_arn)
AWS="aws --endpoint-url=http://localstack:4566 --region us-east-1"
EXE_ARN=$($AWS stepfunctions start-execution \
  --state-machine-arn "$SM_ARN" \
  --name "happy-$(date +%s)" \
  --input '{"order_id":"o-1","sku":"x","qty":2,"amount":100,"idempotency_key":"k-1"}' \
  --query executionArn --output text)
echo "EXE=$EXE_ARN"
for i in $(seq 1 60); do
  status=$($AWS stepfunctions describe-execution --execution-arn "$EXE_ARN" --query status --output text)
  echo "  status=$status"
  if [ "$status" != "RUNNING" ]; then break; fi
  sleep 1
done
$AWS stepfunctions describe-execution --execution-arn "$EXE_ARN" --query '{status:status,output:output}'
output
EXE=arn:aws:states:us-east-1:000000000000:execution:order-saga:happy-1776919385
  status=RUNNING
  status=RUNNING
  status=RUNNING
  status=SUCCEEDED
{
    "status": "SUCCEEDED",
    "output": "{\"FailedEntryCount\":0,\"Entries\":[{\"EventId\":\"ee72f7d1-35f4-482f-ab5b-e3ffa616f33c\"}]}"
}

[stdout]
EXE=arn:aws:states:us-east-1:000000000000:execution:order-saga:happy-1776919385
  status=RUNNING
  status=RUNNING
  status=RUNNING
  status=SUCCEEDED
{
    "status": "SUCCEEDED",
    "output": "{\"FailedEntryCount\":0,\"Entries\":[{\"EventId\":\"ee72f7d1-35f4-482f-ab5b-e3ffa616f33c\"}]}"
}

Verify happy path state

shell
AWS="aws --endpoint-url=http://localstack:4566 --region us-east-1"
echo "--- inventory(x) ---"
$AWS dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}'
echo "--- payments(pay-o-1) ---"
$AWS dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-o-1"}}'
echo "--- shipments(ship-o-1) ---"
$AWS dynamodb get-item --table-name shipments --key '{"shipment_id":{"S":"ship-o-1"}}'
echo "--- notifier logs ---"
LG=$($AWS logs describe-log-groups --log-group-name-prefix /aws/lambda/saga-notifier --query 'logGroups[0].logGroupName' --output text)
echo "log group: $LG"
STREAMS=$($AWS logs describe-log-streams --log-group-name "$LG" --order-by LastEventTime --descending --max-items 1 --query 'logStreams[].logStreamName' --output text)
for s in $STREAMS; do
  $AWS logs get-log-events --log-group-name "$LG" --log-stream-name "$s" --query 'events[].message' --output text
done
output
--- inventory(x) ---
{
    "Item": {
        "sku": {
            "S": "x"
        },
        "reserved_qty": {
            "N": "2"
        }
    }
}
--- payments(pay-o-1) ---
{
    "Item": {
        "amount": {
            "N": "100"
        },
        "order_id": {
            "S": "o-1"
        },
        "payment_id": {
            "S": "pay-o-1"
        },
        "status": {
            "S": "CAPTURED"
        }
    }
}
--- shipments(ship-o-1) ---
{
    "Item": {
        "shipment_id": {
            "S": "ship-o-1"
        },
        "sku": {
            "S": "x"
        },
        "order_id": {
            "S": "o-1"
        },
        "qty": {
            "N": "2"
        },
        "status": {
            "S": "CREATED"
        }
    }
}
--- notifier logs ---
log group: /aws/lambda/saga-notifier
START RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969 Version: $LATEST
	saga-terminal:OrderCompleted
	END RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969
	REPORT RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969	Duration: 8.83 ms	Billed Duration: 9 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

[stdout]
--- inventory(x) ---
{
    "Item": {
        "sku": {
            "S": "x"
        },
        "reserved_qty": {
            "N": "2"
        }
    }
}
--- payments(pay-o-1) ---
{
    "Item": {
        "amount": {
            "N": "100"
        },
        "order_id": {
            "S": "o-1"
        },
        "payment_id": {
            "S": "pay-o-1"
        },
        "status": {
            "S": "CAPTURED"
        }
    }
}
--- shipments(ship-o-1) ---
{
    "Item": {
        "shipment_id": {
            "S": "ship-o-1"
        },
        "sku": {
            "S": "x"
        },
        "order_id": {
            "S": "o-1"
        },
        "qty": {
            "N": "2"
        },
        "status": {
            "S": "CREATED"
        }
    }
}
--- notifier logs ---
log group: /aws/lambda/saga-notifier
START RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969 Version: $LATEST
	saga-terminal:OrderCompleted
	END RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969
	REPORT RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969	Duration: 8.83 ms	Billed Duration: 9 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
Happy path passes: `reserved_qty=2`, payment `CAPTURED`, shipment `CREATED`, log line `saga-terminal:OrderCompleted`.
  • Write setup script and deploy infra
  • Run happy path test
  • Run failure path test

Run failure path execution

shell
SM_ARN=$(cat /app/.sm_arn)
AWS="aws --endpoint-url=http://localstack:4566 --region us-east-1"
EXE_ARN=$($AWS stepfunctions start-execution \
  --state-machine-arn "$SM_ARN" \
  --name "fail-$(date +%s)" \
  --input '{"order_id":"o-2","sku":"x","qty":2,"amount":100,"idempotency_key":"k-2","force_failure":true}' \
  --query executionArn --output text)
echo "EXE=$EXE_ARN"
for i in $(seq 1 60); do
  status=$($AWS stepfunctions describe-execution --execution-arn "$EXE_ARN" --query status --output text)
  echo "  status=$status"
  if [ "$status" != "RUNNING" ]; then break; fi
  sleep 1
done
$AWS stepfunctions describe-execution --execution-arn "$EXE_ARN" --query '{status:status,output:output}'
output
EXE=arn:aws:states:us-east-1:000000000000:execution:order-saga:fail-1776919412
  status=RUNNING
  status=RUNNING
  status=RUNNING
  status=SUCCEEDED
{
    "status": "SUCCEEDED",
    "output": "{\"FailedEntryCount\":0,\"Entries\":[{\"EventId\":\"282648bb-0f5c-422d-a54d-39949f376074\"}]}"
}

[stdout]
EXE=arn:aws:states:us-east-1:000000000000:execution:order-saga:fail-1776919412
  status=RUNNING
  status=RUNNING
  status=RUNNING
  status=SUCCEEDED
{
    "status": "SUCCEEDED",
    "output": "{\"FailedEntryCount\":0,\"Entries\":[{\"EventId\":\"282648bb-0f5c-422d-a54d-39949f376074\"}]}"
}

Verify failure path state

shell
AWS="aws --endpoint-url=http://localstack:4566 --region us-east-1"
echo "--- inventory(x) ---"
$AWS dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}'
echo "--- payments(pay-o-2) ---"
$AWS dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-o-2"}}'
echo "--- shipments(ship-o-2) ---"
$AWS dynamodb get-item --table-name shipments --key '{"shipment_id":{"S":"ship-o-2"}}'
echo "--- recent notifier logs ---"
LG=/aws/lambda/saga-notifier
STREAMS=$($AWS logs describe-log-streams --log-group-name "$LG" --order-by LastEventTime --descending --max-items 3 --query 'logStreams[].logStreamName' --output text)
for s in $STREAMS; do
  echo "  stream: $s"
  $AWS logs get-log-events --log-group-name "$LG" --log-stream-name "$s" --query 'events[].message' --output text
done
output
--- inventory(x) ---
{
    "Item": {
        "sku": {
            "S": "x"
        },
        "reserved_qty": {
            "N": "2"
        }
    }
}
--- payments(pay-o-2) ---
--- shipments(ship-o-2) ---
--- recent notifier logs ---
  stream: 2026/04/23/[$LATEST]d070f9de1ebed0772761c1013ab1fbd2
START RequestId: 7faa47e6-3fa8-497f-9acf-d90e34a18e5e Version: $LATEST
	saga-terminal:OrderFailed
	END RequestId: 7faa47e6-3fa8-497f-9acf-d90e34a18e5e
	REPORT RequestId: 7faa47e6-3fa8-497f-9acf-d90e34a18e5e	Duration: 9.93 ms	Billed Duration: 10 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	

  stream: 2026/04/23/[$LATEST]82776cee18c911b96f8c9b3923c17614
START RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969 Version: $LATEST
	saga-terminal:OrderCompleted
	END RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969
	REPORT RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969	Duration: 8.83 ms	Billed Duration: 9 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

[stdout]
--- inventory(x) ---
{
    "Item": {
        "sku": {
            "S": "x"
        },
        "reserved_qty": {
            "N": "2"
        }
    }
}
--- payments(pay-o-2) ---
--- shipments(ship-o-2) ---
--- recent notifier logs ---
  stream: 2026/04/23/[$LATEST]d070f9de1ebed0772761c1013ab1fbd2
START RequestId: 7faa47e6-3fa8-497f-9acf-d90e34a18e5e Version: $LATEST
	saga-terminal:OrderFailed
	END RequestId: 7faa47e6-3fa8-497f-9acf-d90e34a18e5e
	REPORT RequestId: 7faa47e6-3fa8-497f-9acf-d90e34a18e5e	Duration: 9.93 ms	Billed Duration: 10 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	

  stream: 2026/04/23/[$LATEST]82776cee18c911b96f8c9b3923c17614
START RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969 Version: $LATEST
	saga-terminal:OrderCompleted
	END RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969
	REPORT RequestId: 6f815dba-2441-4b68-8a15-ab81beecc969	Duration: 8.83 ms	Billed Duration: 9 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
Let me verify the compensation actually ran (the inventory count of 2 is happy_reserve(+2) + fail_reserve(+2) + fail_release(-2) = 2 , net zero for the failure path).

Inspect idempotency records and execution history

shell
AWS="aws --endpoint-url=http://localstack:4566 --region us-east-1"
echo "--- idempotency markers for k-2 ---"
for action in reserve charge ship release refund; do
  $AWS dynamodb get-item --table-name saga-idempotency --key "{\"idempotency_key\":{\"S\":\"k-2:${action}\"}}" --query 'Item.action.S' --output text 2>/dev/null
done
echo "--- execution history for the fail run ---"
EXE=$($AWS stepfunctions list-executions --state-machine-arn $(cat /app/.sm_arn) --query "executions[?starts_with(name, 'fail-')].executionArn | [0]" --output text)
$AWS stepfunctions get-execution-history --execution-arn "$EXE" --query 'events[].{t:type,n:stateEnteredEventDetails.name,err:stateExitedEventDetails.name}' --output table | head -50
output
--- idempotency markers for k-2 ---
reserve
None
None
release
None
--- execution history for the fail run ---
----------------------------------------------------------------------------------------------
|                                     GetExecutionHistory                                    |
+----------------------------------+-----------------------------------+---------------------+
|                err               |                 n                 |          t          |
+----------------------------------+-----------------------------------+---------------------+
|  None                            |  None                             |  ExecutionStarted   |
|  None                            |  ReserveInventory                 |  TaskStateEntered   |
|  None                            |  None                             |  TaskScheduled      |
|  None                            |  None                             |  TaskStarted        |
|  None                            |  None                             |  TaskSucceeded      |
|  ReserveInventory                |  None                             |  TaskStateExited    |
|  None                            |  ChargePayment                    |  TaskStateEntered   |
|  None                            |  None                             |  TaskScheduled      |
|  None                            |  None                             |  TaskStarted        |
|  None                            |  None                             |  TaskFailed         |
|  ChargePayment                   |  None                             |  TaskStateExited    |
|  None                            |  ReleaseInventoryAfterChargeFail  |  TaskStateEntered   |
|  None                            |  None                             |  TaskScheduled      |
|  None                            |  None                             |  TaskStarted        |
|  None                            |  None                             |  TaskSucceeded      |
|  ReleaseInventoryAfterChargeFail |  None                             |  TaskStateExited    |
|  None                            |  PublishOrderFailed               |  TaskStateEntered   |
|  None                            |  None                             |  TaskScheduled      |
|  None                            |  None                             |  TaskStarted        |
|  None                            |  None                             |  TaskSucceeded      |
|  PublishOrderFailed              |  None                             |  TaskStateExited    |
|  None                            |  None                             |  ExecutionSucceeded |
+----------------------------------+-----------------------------------+---------------------+

[stdout]
--- idempotency markers for k-2 ---
reserve
None
None
release
None
--- execution history for the fail run ---
----------------------------------------------------------------------------------------------
|                                     GetExecutionHistory                                    |
+----------------------------------+-----------------------------------+---------------------+
|                err               |                 n                 |          t          |
+----------------------------------+-----------------------------------+---------------------+
|  None                            |  None                             |  ExecutionStarted   |
|  None                            |  ReserveInventory                 |  TaskStateEntered   |
|  None                            |  None                             |  TaskScheduled      |
|  None                            |  None                             |  TaskStarted        |
|  None                            |  None                             |  TaskSucceeded      |
|  ReserveInventory                |  None                             |  TaskStateExited    |
|  None                            |  ChargePayment                    |  TaskStateEntered   |
|  None                            |  None                             |  TaskScheduled      |
|  None                            |  None                             |  TaskStarted        |
|  None                            |  None                             |  TaskFailed         |
|  ChargePayment                   |  None                             |  TaskStateExited    |
|  None                            |  ReleaseInventoryAfterChargeFail  |  TaskStateEntered   |
|  None                            |  None                             |  TaskScheduled      |
|  None                            |  None                             |  TaskStarted        |
|  None                            |  None                             |  TaskSucceeded      |
|  ReleaseInventoryAfterChargeFail |  None                             |  TaskStateExited    |
|  None                            |  PublishOrderFailed               |  TaskStateEntered   |
|  None                            |  None                             |  TaskScheduled      |
|  None                            |  None                             |  TaskStarted        |
|  None                            |  None                             |  TaskSucceeded      |
|  PublishOrderFailed              |  None                             |  TaskStateExited    |
|  None                            |  None                             |  ExecutionSucceeded |
+----------------------------------+-----------------------------------+---------------------+
The state machine runs the full forward + compensation flow correctly. The current value of `reserved_qty=2` is `happy(+2) + fail(+2) - fail_release(-2)` , the failure-path delta is exactly 0, which is the actual compensation guarantee. Let me write a `verify.sh` that resets state between runs so each assertion holds in absolute terms.
write · /app/verify.sh
#!/usr/bin/env bash
set -euo pipefail

EP=http://localstack:4566
REG=us-east-1
ACCT=000000000000
AWS="aws --endpoint-url=$EP --region $REG"
SM_ARN="arn:aws:states:${REG}:${ACCT}:stateMachine:order-saga"

pass() { echo "  PASS: $1"; }
fail() { echo "  FAIL: $1"; FAILED=1; }

reset_state() {
  $AWS dynamodb delete-item --table-name inventory --key '{"sku":{"S":"x"}}' >/dev/null 2>&1 || true
  for pid in pay-o-1 pay-o-2; do
    $AWS dynamodb delete-item --table-name payments --key "{\"payment_id\":{\"S\":\"$pid\"}}" >/dev/null 2>&1 || true
  done
  for sid in ship-o-1 ship-o-2; do
    $AWS dynamodb delete-item --table-name shipments --key "{\"shipment_id\":{\"S\":\"$sid\"}}" >/dev/null 2>&1 || true
  done
  for k in k-1 k-2; do
    for a in reserve charge ship release refund; do
      $AWS dynamodb delete-item --table-name saga-idempotency \
        --key "{\"idempotency_key\":{\"S\":\"${k}:${a}\"}}" >/dev/null 2>&1 || true
    done
  done
}

run_execution() {
  local name=$1; local input=$2
  local exe
  exe=$($AWS stepfunctions start-execution --state-machine-arn "$SM_ARN" \
    --name "$name" --input "$input" --query executionArn --output text)
  for i in $(seq 1 60); do
    s=$($AWS stepfunctions describe-execution --execution-arn "$exe" --query status --output text)
    [ "$s" != "RUNNING" ] && break
    sleep 1
  done
  echo "$s"
}

notifier_has() {
  local needle=$1
  local lg=/aws/lambda/saga-notifier
  local found=0
  for stream in $($AWS logs describe-log-streams --log-group-name "$lg" \
      --order-by LastEventTime --descending --max-items 5 \
      --query 'logStreams[].logStreamName' --output text); do
    if $AWS logs get-log-events --log-group-name "$lg" --log-stream-name "$stream" \
        --query 'events[].message' --output text | grep -F -q "$needle"; then
      found=1; break
    fi
  done
  [ "$found" -eq 1 ]
}

FAILED=0

echo "=== Happy path ==="
reset_state
status=$(run_execution "verify-happy-$(date +%s)" '{"order_id":"o-1","sku":"x","qty":2,"amount":100,"idempotency_key":"k-1"}')
[ "$status" = "SUCCEEDED" ] && pass "state machine SUCCEEDED" || fail "state machine status=$status"

inv=$($AWS dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}' \
  --query 'Item.reserved_qty.N' --output text 2>/dev/null)
[ "$inv" = "2" ] && pass "inventory.x.reserved_qty == 2" || fail "inventory.x.reserved_qty=$inv (want 2)"

pay=$($AWS dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-o-1"}}' \
  --query 'Item.status.S' --output text 2>/dev/null)
[ "$pay" = "CAPTURED" ] && pass "payments[pay-o-1].status == CAPTURED" || fail "payments[pay-o-1].status=$pay"

ship=$($AWS dynamodb get-item --table-name shipments --key '{"shipment_id":{"S":"ship-o-1"}}' \
  --query 'Item.shipment_id.S' --output text 2>/dev/null)
[ "$ship" = "ship-o-1" ] && pass "shipments[ship-o-1] exists" || fail "shipments[ship-o-1] missing"

sleep 2
notifier_has "saga-terminal:OrderCompleted" \
  && pass "notifier log: saga-terminal:OrderCompleted" \
  || fail "notifier log missing OrderCompleted"

echo
echo "=== Failure path ==="
reset_state
status=$(run_execution "verify-fail-$(date +%s)" '{"order_id":"o-2","sku":"x","qty":2,"amount":100,"idempotency_key":"k-2","force_failure":true}')
[ "$status" = "SUCCEEDED" ] && pass "state machine SUCCEEDED (graceful)" || fail "state machine status=$status"

inv=$($AWS dynamodb get-item --table-name inventory --key '{"sku":{"S":"x"}}' \
  --query 'Item.reserved_qty.N' --output text 2>/dev/null)
inv=${inv:-0}; [ "$inv" = "None" ] && inv=0
[ "$inv" = "0" ] && pass "inventory.x.reserved_qty back to 0 (compensation ran)" \
  || fail "inventory.x.reserved_qty=$inv (want 0)"

pay=$($AWS dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-o-2"}}' \
  --query 'Item' --output text 2>/dev/null)
if [ -z "$pay" ] || [ "$pay" = "None" ]; then
  pass "payments[pay-o-2] absent (no charge happened)"
else
  st=$($AWS dynamodb get-item --table-name payments --key '{"payment_id":{"S":"pay-o-2"}}' \
    --query 'Item.status.S' --output text)
  [ "$st" = "REFUNDED" ] && pass "payments[pay-o-2].status == REFUNDED (no-op refund)" \
    || fail "payments[pay-o-2] present with status=$st"
fi

ship=$($AWS dynamodb get-item --table-name shipments --key '{"shipment_id":{"S":"ship-o-2"}}' \
  --query 'Item' --output text 2>/dev/null)
if [ -z "$ship" ] || [ "$ship" = "None" ]; then
  pass "shipments[ship-o-2] absent"
else
  fail "shipments[ship-o-2] present"
fi

sleep 2
notifier_has "saga-terminal:OrderFailed" \
  && pass "notifier log: saga-terminal:OrderFailed" \
  || fail "notifier log missing OrderFailed"

echo
if [ "$FAILED" -eq 0 ]; then
  echo "=== ALL CHECKS PASSED ==="
else
  echo "=== SOME CHECKS FAILED ==="
  exit 1
fi

Run end-to-end verifier

shell
chmod +x /app/verify.sh && bash /app/verify.sh 2>&1 | tail -40
output
=== Happy path ===
  PASS: state machine SUCCEEDED
  PASS: inventory.x.reserved_qty == 2
  PASS: payments[pay-o-1].status == CAPTURED
  PASS: shipments[ship-o-1] exists
  PASS: notifier log: saga-terminal:OrderCompleted

=== Failure path ===
  PASS: state machine SUCCEEDED (graceful)
  PASS: inventory.x.reserved_qty back to 0 (compensation ran)
  PASS: payments[pay-o-2] absent (no charge happened)
  PASS: shipments[ship-o-2] absent
  PASS: notifier log: saga-terminal:OrderFailed

=== ALL CHECKS PASSED ===

[stdout]
=== Happy path ===
  PASS: state machine SUCCEEDED
  PASS: inventory.x.reserved_qty == 2
  PASS: payments[pay-o-1].status == CAPTURED
  PASS: shipments[ship-o-1] exists
  PASS: notifier log: saga-terminal:OrderCompleted

=== Failure path ===
  PASS: state machine SUCCEEDED (graceful)
  PASS: inventory.x.reserved_qty back to 0 (compensation ran)
  PASS: payments[pay-o-2] absent (no charge happened)
  PASS: shipments[ship-o-2] absent
  PASS: notifier log: saga-terminal:OrderFailed

=== ALL CHECKS PASSED ===
  • Write setup script and deploy infra
  • Run happy path test
  • Run failure path test
All checks pass on both runs. Saga is deployed and working end-to-end. **What's in `/app`:** - `lambdas/` , 6 python handlers; each forward step + each compensation claims a scoped idempotency key (`<key>:reserve`, `<key>:charge`, `<key>:ship`, `<key>:release`, `<key>:refund`) in `saga-idempotency` so retries are no-ops. - `iam/` , per-role least-privilege policies + KMS key policy. No `kms:*`. Each lambda role gets only its own table; `charge-payment-role` is the sole role with `kms:Decrypt` and `secretsmanager:GetSecretValue` on `payment/gateway-apikey` (gated by `kms:ViaService=secretsmanager.us-east-1.amazonaws.com`). `order-saga-role` lists the five worker lambda ARNs explicitly , no wildcard. `saga-notifier-role` has only `events:PutEvents` on the bus + log-group-scoped logs. - `sm/order-saga.json` , `Reserve → Charge → Ship`. Catches per state route to the **reverse-order** compensation chain: charge fail → release; ship fail → refund → release. Both compensation chains end at `PublishOrderFailed` (PutEvents to `saga-events` with detail-type `OrderFailed`), and a clean run hits `PublishOrderCompleted`. Saga always ends `SUCCEEDED`. - `setup.sh` , idempotent provisioning (KMS+alias, secret, 4 tables w/ TTL on idempotency, event bus, roles+policies, lambdas, EventBridge rule + lambda resource policy for invocation, state machine). - `verify.sh` , runs both executions on clean state. **Verifier output:** ``` === Happy path === PASS: state machine SUCCEEDED PASS: inventory.x.reserved_qty == 2 PASS: payments[pay-o-1].status == CAPTURED PASS: shipments[ship-o-1] exists PASS: notifier log: saga-terminal:OrderCompleted === Failure path === PASS: state machine SUCCEEDED (graceful) PASS: inventory.x.reserved_qty back to 0 (compensation ran) PASS: payments[pay-o-2] absent (no charge happened) PASS: shipments[ship-o-2] absent PASS: notifier log: saga-terminal:OrderFailed ```
Final diff , what the agent changed

No file changes, the agent left the workspace untouched.

Verifier output , the proof behind the reward
tests/test.sh
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/sPS3_eXX143GGyXnVjZ9L/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 20 items

../tests/test_outputs.py::test_01_cmk_alias_and_secret_exist PASSED      [  5%]
../tests/test_outputs.py::test_02_all_four_tables_exist PASSED           [ 10%]
../tests/test_outputs.py::test_03_all_six_lambdas_exist PASSED           [ 15%]
../tests/test_outputs.py::test_04_state_machine_and_bus_exist PASSED     [ 20%]
../tests/test_outputs.py::test_05_terminal_rule_exists_on_saga_bus PASSED [ 25%]
../tests/test_outputs.py::test_06_every_forward_state_has_catch_to_compensation PASSED [ 30%]
../tests/test_outputs.py::test_07_compensations_run_in_lifo_order PASSED [ 35%]
../tests/test_outputs.py::test_08_compensation_states_preserve_error_via_result_path PASSED [ 40%]
../tests/test_outputs.py::test_09_idempotency_table_has_ttl_and_correct_pk PASSED [ 45%]
../tests/test_outputs.py::test_10_each_lambda_role_scoped_to_its_own_table PASSED [ 50%]
../tests/test_outputs.py::test_11_only_charge_role_reads_the_payment_secret PASSED [ 55%]
../tests/test_outputs.py::test_12_saga_role_lambda_invoke_not_wildcarded PASSED [ 60%]
../tests/test_outputs.py::test_13_rule_pattern_matches_both_terminal_types_with_array_wrap PASSED [ 65%]
../tests/test_outputs.py::test_14_charge_role_kms_decrypt_scoped_not_wildcard PASSED [ 70%]
../tests/test_outputs.py::test_15_rule_targets_notifier_lambda PASSED    [ 75%]
../tests/test_outputs.py::test_16_notifier_role_is_least_privilege_no_ddb_kms_secrets FAILED [ 80%]
../tests/test_outputs.py::test_17_notifier_lambda_resource_policy_allows_events_with_source_arn PASSED [ 85%]
../tests/test_outputs.py::test_18_happy_path_completes_and_writes_all_three_tables PASSED [ 90%]
../tests/test_outputs.py::test_19_failure_path_triggers_compensation_and_rolls_back_inventory PASSED [ 95%]
../tests/test_outputs.py::test_20_idempotency_prevents_double_reservation PASSED [100%]

=================================== FAILURES ===================================
_________ test_16_notifier_role_is_least_privilege_no_ddb_kms_secrets __________

    def test_16_notifier_role_is_least_privilege_no_ddb_kms_secrets():
        """saga-notifier-role must NOT have dynamodb, kms, or secretsmanager
        permissions. It only publishes logs. Least privilege."""
        stmts = _statements(_get_role_policies(ROLE_NOTIFY))
        forbidden_prefixes = ("dynamodb:", "kms:", "secretsmanager:", "sqs:", "events:PutEvents")
        for s in stmts:
            if s.get("Effect") != "Allow":
                continue
            for a in _as_list(s.get("Action")):
                if a == "*":
                    pytest.fail(
                        "saga-notifier-role has Action='*' - violates least "
                        "privilege (should only need logs)"
                    )
                for fp in forbidden_prefixes:
>                   assert not a.startswith(fp), (
                        f"saga-notifier-role has forbidden action '{a}' - "
                        "only logs are needed"
                    )
E                   AssertionError: saga-notifier-role has forbidden action 'events:PutEvents' - only logs are needed
E                   assert not True
E                    +  where True = <built-in method startswith of str object at 0xffffb6ba2470>('events:PutEvents')
E                    +    where <built-in method startswith of str object at 0xffffb6ba2470> = 'events:PutEvents'.startswith

/tests/test_outputs.py:655: AssertionError
=============================== warnings summary ===============================
test_outputs.py: 134 warnings
  /root/.cache/uv/archive-v0/sPS3_eXX143GGyXnVjZ9L/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_outputs.py::test_01_cmk_alias_and_secret_exist
PASSED ../tests/test_outputs.py::test_02_all_four_tables_exist
PASSED ../tests/test_outputs.py::test_03_all_six_lambdas_exist
PASSED ../tests/test_outputs.py::test_04_state_machine_and_bus_exist
PASSED ../tests/test_outputs.py::test_05_terminal_rule_exists_on_saga_bus
PASSED ../tests/test_outputs.py::test_06_every_forward_state_has_catch_to_compensation
PASSED ../tests/test_outputs.py::test_07_compensations_run_in_lifo_order
PASSED ../tests/test_outputs.py::test_08_compensation_states_preserve_error_via_result_path
PASSED ../tests/test_outputs.py::test_09_idempotency_table_has_ttl_and_correct_pk
PASSED ../tests/test_outputs.py::test_10_each_lambda_role_scoped_to_its_own_table
PASSED ../tests/test_outputs.py::test_11_only_charge_role_reads_the_payment_secret
PASSED ../tests/test_outputs.py::test_12_saga_role_lambda_invoke_not_wildcarded
PASSED ../tests/test_outputs.py::test_13_rule_pattern_matches_both_terminal_types_with_array_wrap
PASSED ../tests/test_outputs.py::test_14_charge_role_kms_decrypt_scoped_not_wildcard
PASSED ../tests/test_outputs.py::test_15_rule_targets_notifier_lambda
PASSED ../tests/test_outputs.py::test_17_notifier_lambda_resource_policy_allows_events_with_source_arn
PASSED ../tests/test_outputs.py::test_18_happy_path_completes_and_writes_all_three_tables
PASSED ../tests/test_outputs.py::test_19_failure_path_triggers_compensation_and_rolls_back_inventory
PASSED ../tests/test_outputs.py::test_20_idempotency_prevents_double_reservation
FAILED ../tests/test_outputs.py::test_16_notifier_role_is_least_privilege_no_ddb_kms_secrets
================= 1 failed, 19 passed, 134 warnings in 20.95s ==================

Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_d0d8cf3f063641fa. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.

Trial trial_d0d8cf3f063641fa · verifier authoritative; classifier explanatory.