tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.Inspect working directory and tool versions
ls /app && echo "---" && aws --version && python3 --version
--- aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24 Python 3.12.3 [stdout] --- aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24 Python 3.12.3
Check localstack health
curl -s http://localstack:4566/_localstack/health | head -c 2000
{"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "disabled", "dynamodbstreams": "disabled", "ec2": "disabled", "es": "disabled", "events": "available", "firehose": "disabled", "iam": "available", "kinesis": "disabled", "kms": "available", "lambda": "available", "logs": "available", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "available", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "available", "ses": "disabled", "sns": "disabled", "sqs": "disabled", "ssm": "disabled", "stepfunctions": "available", "sts": "available", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "4.13.1"}
[stdout]
{"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "disabled", "dynamodbstreams": "disabled", "ec2": "disabled", "es": "disabled", "events": "available", "firehose": "disabled", "iam": "available", "kinesis": "disabled", "kms": "available", "lambda": "available", "logs": "available", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "available", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "available", "ses": "disabled", "sns": "disabled", "sqs": "disabled", "ssm": "disabled", "stepfunctions": "available", "sts": "available", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "4.13.1"}Verify localstack credentials work
mkdir -p /app/build && echo "export AWS_ENDPOINT_URL=http://localstack:4566" > /app/build/env.sh && aws --endpoint-url=http://localstack:4566 sts get-caller-identity
{
"UserId": "AKIAIOSFODNN7EXAMPLE",
"Account": "000000000000",
"Arn": "arn:aws:iam::000000000000:root"
}
[stdout]
{
"UserId": "AKIAIOSFODNN7EXAMPLE",
"Account": "000000000000",
"Arn": "arn:aws:iam::000000000000:root"
}Inventory existing resources
export AWS_ENDPOINT_URL=http://localstack:4566 # Inventory what exists echo "=== KMS ===" aws kms list-aliases --query "Aliases[?AliasName=='alias/app-rotation-key']" 2>&1 | head -20 echo "=== Secrets ===" aws secretsmanager list-secrets --query "SecretList[].Name" 2>&1 | head -20 echo "=== Lambdas ===" aws lambda list-functions --query "Functions[].FunctionName" 2>&1 | head -20 echo "=== State machines ===" aws stepfunctions list-state-machines 2>&1 | head -40 echo "=== Event buses ===" aws events list-event-buses --query "EventBuses[].Name" 2>&1 | head -20 echo "=== Roles ===" aws iam list-roles --query "Roles[].RoleName" 2>&1 | head -40
=== KMS ===
[]
=== Secrets ===
[]
=== Lambdas ===
[]
=== State machines ===
{
"stateMachines": []
}
=== Event buses ===
[
"default"
]
=== Roles ===
[]
[stdout]
=== KMS ===
[]
=== Secrets ===
[]
=== Lambdas ===
[]
=== State machines ===
{
"stateMachines": []
}
=== Event buses ===
[
"default"
]
=== Roles ===
[]Create build directory structure
mkdir -p /app/build/lambdas/rotator /app/build/lambdas/notifier /app/build/policies /app/build/sm
(Bash completed with no output)
import os
import json
import logging
import secrets
import string
import boto3
logger = logging.getLogger()
logger.setLevel(logging.INFO)
ENDPOINT = os.environ.get("AWS_ENDPOINT_URL", "http://localstack:4566")
sm = boto3.client("secretsmanager", endpoint_url=ENDPOINT)
def _generate_password(length: int = 32) -> str:
alphabet = string.ascii_letters + string.digits + "-_~."
return "".join(secrets.choice(alphabet) for _ in range(length))
def lambda_handler(event, context):
logger.info("rotator event: %s", json.dumps(event))
secret_id = event["SecretId"]
token = event["ClientRequestToken"]
step = event["Step"]
meta = sm.describe_secret(SecretId=secret_id)
versions = meta.get("VersionIdsToStages", {})
if token not in versions:
raise ValueError(f"version {token} not staged for {secret_id}")
if step == "createSecret":
return _create_secret(secret_id, token)
if step == "setSecret":
return _set_secret(secret_id, token)
if step == "testSecret":
return _test_secret(secret_id, token)
if step == "finishSecret":
return _finish_secret(secret_id, token, versions)
raise ValueError(f"unknown step {step}")
def _create_secret(secret_id: str, token: str):
try:
sm.get_secret_value(SecretId=secret_id, VersionId=token, VersionStage="AWSPENDING")
logger.info("AWSPENDING already exists for %s", token)
return
except sm.exceptions.ResourceNotFoundException:
pass
current = sm.get_secret_value(SecretId=secret_id, VersionStage="AWSCURRENT")
try:
payload = json.loads(current["SecretString"])
except (ValueError, TypeError):
payload = {"password": current.get("SecretString", "")}
new_password = _generate_password()
# Guarantee different string on consecutive rotations.
while new_password == payload.get("password"):
new_password = _generate_password()
payload["password"] = new_password
sm.put_secret_value(
SecretId=secret_id,
ClientRequestToken=token,
SecretString=json.dumps(payload),
VersionStages=["AWSPENDING"],
)
logger.info("staged AWSPENDING version %s", token)
def _set_secret(secret_id: str, token: str):
# No external database to update in this pipeline; the rotation is a no-op
# on the credential consumer side. Keeping the step explicit per the
# Secrets Manager rotation contract.
logger.info("setSecret no-op for %s version %s", secret_id, token)
def _test_secret(secret_id: str, token: str):
value = sm.get_secret_value(SecretId=secret_id, VersionId=token, VersionStage="AWSPENDING")
payload = json.loads(value["SecretString"])
if not payload.get("password"):
raise ValueError("rotated password is empty")
logger.info("testSecret verified AWSPENDING version %s", token)
def _finish_secret(secret_id: str, token: str, versions):
current_version = None
for version_id, stages in versions.items():
if "AWSCURRENT" in stages:
current_version = version_id
break
if current_version == token:
logger.info("AWSCURRENT already at %s", token)
return
sm.update_secret_version_stage(
SecretId=secret_id,
VersionStage="AWSCURRENT",
MoveToVersionId=token,
RemoveFromVersionId=current_version,
)
logger.info("promoted %s to AWSCURRENT (from %s)", token, current_version)
import json
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
detail = event.get("detail", {})
secret_id = detail.get("secretId", "unknown")
version_id = detail.get("versionId", "unknown")
logger.info("notified: rotation complete secret=%s version=%s", secret_id, version_id)
print(f"notified: rotation complete secret={secret_id} version={version_id}")
return {"status": "ok"}
Package lambda zips
cd /app/build/lambdas/rotator && zip -q /app/build/rotator.zip lambda_function.py && cd /app/build/lambdas/notifier && zip -q /app/build/notifier.zip lambda_function.py && ls -la /app/build/*.zip
-rw-r--r-- 1 root root 417 Apr 23 06:03 /app/build/notifier.zip -rw-r--r-- 1 root root 1376 Apr 23 06:03 /app/build/rotator.zip [stdout] -rw-r--r-- 1 root root 417 Apr 23 06:03 /app/build/notifier.zip -rw-r--r-- 1 root root 1376 Apr 23 06:03 /app/build/rotator.zip
Create KMS key, alias, and event bus
export AWS_ENDPOINT_URL=http://localstack:4566 # 1. Create CMK KMS key KMS_KEY_ID=$(aws kms create-key --description "Customer-managed key for app/db/password rotation" --query 'KeyMetadata.KeyId' --output text) echo "KMS_KEY_ID=$KMS_KEY_ID" aws kms create-alias --alias-name alias/app-rotation-key --target-key-id "$KMS_KEY_ID" KMS_KEY_ARN=$(aws kms describe-key --key-id "$KMS_KEY_ID" --query 'KeyMetadata.Arn' --output text) echo "KMS_KEY_ARN=$KMS_KEY_ARN" # 2. Create custom event bus aws events create-event-bus --name rotation-events --query 'EventBusArn' --output text # Store for later echo "$KMS_KEY_ID" > /app/build/kms_key_id echo "$KMS_KEY_ARN" > /app/build/kms_key_arn
KMS_KEY_ID=d021b380-23c5-4157-a347-9017027be7e2 KMS_KEY_ARN=arn:aws:kms:us-east-1:000000000000:key/d021b380-23c5-4157-a347-9017027be7e2 arn:aws:events:us-east-1:000000000000:event-bus/rotation-events [stdout] KMS_KEY_ID=d021b380-23c5-4157-a347-9017027be7e2 KMS_KEY_ARN=arn:aws:kms:us-east-1:000000000000:key/d021b380-23c5-4157-a347-9017027be7e2 arn:aws:events:us-east-1:000000000000:event-bus/rotation-events
{
"query": "select:TodoWrite",
"max_results": 1
}
, output,
{"type": "tool_reference", "tool_name": "TodoWrite"}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "lambda.amazonaws.com"},
"Action": "sts:AssumeRole"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:us-east-1:000000000000:*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/notification-lambda:*"
}
]
}
Create notification lambda + role
export AWS_ENDPOINT_URL=http://localstack:4566 # Role for notification lambda aws iam create-role \ --role-name notification-lambda-role \ --assume-role-policy-document file:///app/build/policies/lambda_trust.json \ --query 'Role.Arn' --output text aws iam put-role-policy \ --role-name notification-lambda-role \ --policy-name notification-lambda-logs \ --policy-document file:///app/build/policies/notification_lambda_policy.json # Create notification lambda aws lambda create-function \ --function-name notification-lambda \ --runtime python3.11 \ --role arn:aws:iam::000000000000:role/notification-lambda-role \ --handler lambda_function.lambda_handler \ --zip-file fileb:///app/build/notifier.zip \ --timeout 15 \ --query 'FunctionArn' --output text
arn:aws:iam::000000000000:role/notification-lambda-role arn:aws:lambda:us-east-1:000000000000:function:notification-lambda [stdout] arn:aws:iam::000000000000:role/notification-lambda-role arn:aws:lambda:us-east-1:000000000000:function:notification-lambda
Wait for notification lambda to be Active
export AWS_ENDPOINT_URL=http://localstack:4566 # Wait briefly for lambda to be ready (localstack sometimes needs a moment) for i in 1 2 3 4 5; do state=$(aws lambda get-function --function-name notification-lambda --query 'Configuration.State' --output text 2>/dev/null) echo "notification-lambda state: $state" [ "$state" = "Active" ] && break sleep 1 done
notification-lambda state: Active [stdout] notification-lambda state: Active
{
"source": ["rotation.pipeline"],
"detail-type": ["RotationComplete"]
}
Create EventBridge rule with notifier target
export AWS_ENDPOINT_URL=http://localstack:4566 # Create rule on custom bus aws events put-rule \ --name on-rotation-success \ --event-bus-name rotation-events \ --event-pattern file:///app/build/policies/event_pattern.json \ --state ENABLED \ --query 'RuleArn' --output text RULE_ARN=$(aws events describe-rule --name on-rotation-success --event-bus-name rotation-events --query 'Arn' --output text) echo "RULE_ARN=$RULE_ARN" # Grant EventBridge permission to invoke notification lambda via this specific rule aws lambda add-permission \ --function-name notification-lambda \ --statement-id allow-eventbridge-on-rotation-success \ --action lambda:InvokeFunction \ --principal events.amazonaws.com \ --source-arn "$RULE_ARN" \ --query 'Statement' --output text >/dev/null # Add the lambda as a rule target aws events put-targets \ --rule on-rotation-success \ --event-bus-name rotation-events \ --targets "Id=notif,Arn=arn:aws:lambda:us-east-1:000000000000:function:notification-lambda"
arn:aws:events:us-east-1:000000000000:rule/rotation-events/on-rotation-success
RULE_ARN=arn:aws:events:us-east-1:000000000000:rule/rotation-events/on-rotation-success
{
"FailedEntryCount": 0,
"FailedEntries": []
}
[stdout]
arn:aws:events:us-east-1:000000000000:rule/rotation-events/on-rotation-success
RULE_ARN=arn:aws:events:us-east-1:000000000000:rule/rotation-events/on-rotation-success
{
"FailedEntryCount": 0,
"FailedEntries": []
}{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Logs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/secrets-rotator:*"
},
{
"Sid": "SecretRotationOps",
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
],
"Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-*"
},
{
"Sid": "KmsForSecret",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "__KMS_KEY_ARN__"
}
]
}
Create rotator lambda with scoped IAM policy
export AWS_ENDPOINT_URL=http://localstack:4566
KMS_KEY_ARN=$(cat /app/build/kms_key_arn)
# Inline the actual KMS ARN into the policy
python3 -c "
import json
p = json.load(open('/app/build/policies/rotator_policy.json'))
for s in p['Statement']:
if s.get('Sid') == 'KmsForSecret':
s['Resource'] = '$KMS_KEY_ARN'
json.dump(p, open('/app/build/policies/rotator_policy.json','w'), indent=2)
"
cat /app/build/policies/rotator_policy.json
# Create rotator role and attach policy
aws iam create-role \
--role-name secrets-rotator-role \
--assume-role-policy-document file:///app/build/policies/lambda_trust.json \
--query 'Role.Arn' --output text
aws iam put-role-policy \
--role-name secrets-rotator-role \
--policy-name secrets-rotator-inline \
--policy-document file:///app/build/policies/rotator_policy.json
# Create rotator lambda - pass endpoint so boto3 inside lambda hits localstack
aws lambda create-function \
--function-name secrets-rotator \
--runtime python3.11 \
--role arn:aws:iam::000000000000:role/secrets-rotator-role \
--handler lambda_function.lambda_handler \
--zip-file fileb:///app/build/rotator.zip \
--timeout 30 \
--environment 'Variables={AWS_ENDPOINT_URL=http://localstack:4566}' \
--query 'FunctionArn' --output text
# Allow SecretsManager to invoke the rotator
aws lambda add-permission \
--function-name secrets-rotator \
--statement-id allow-secretsmanager-invoke \
--action lambda:InvokeFunction \
--principal secretsmanager.amazonaws.com \
--query 'Statement' --output text >/dev/null
for i in 1 2 3 4 5; do
state=$(aws lambda get-function --function-name secrets-rotator --query 'Configuration.State' --output text 2>/dev/null)
echo "secrets-rotator state: $state"
[ "$state" = "Active" ] && break
sleep 1
done{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Logs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/secrets-rotator:*"
},
{
"Sid": "SecretRotationOps",
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
],
"Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-*"
},
{
"Sid": "KmsForSecret",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:us-east-1:000000000000:key/d021b380-23c5-4157-a347-9017027be7e2"
}
]
}arn:aws:iam::000000000000:role/secrets-rotator-role
arn:aws:lambda:us-east-1:000000000000:function:secrets-rotator
secrets-rotator state: Active
[stdout]
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Logs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/secrets-rotator:*"
},
{
"Sid": "SecretRotationOps",
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
],
"Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-*"
},
{
"Sid": "KmsForSecret",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:us-east-1:000000000000:key/d021b380-23c5-4157-a347-9017027be7e2"
}
]
}arn:aws:iam::000000000000:role/secrets-rotator-role
arn:aws:lambda:us-east-1:000000000000:function:secrets-rotator
secrets-rotator state: ActiveCreate CMK-encrypted secret and configure rotation
export AWS_ENDPOINT_URL=http://localstack:4566
KMS_KEY_ID=$(cat /app/build/kms_key_id)
# Create the secret encrypted with our CMK
aws secretsmanager create-secret \
--name app/db/password \
--description "Rotatable DB password for app" \
--kms-key-id "$KMS_KEY_ID" \
--secret-string '{"username":"app","password":"initial-password-0"}' \
--query 'ARN' --output text
# Attach rotation via the rotator lambda. Disable automatic scheduled rotation
# (still allows on-demand RotateSecret).
aws secretsmanager rotate-secret \
--secret-id app/db/password \
--rotation-lambda-arn arn:aws:lambda:us-east-1:000000000000:function:secrets-rotator \
--rotation-rules '{"AutomaticallyAfterDays": 365}' \
--query 'ARN' --output text 2>&1 | head -20
sleep 2
aws secretsmanager describe-secret --secret-id app/db/password --query '{RotationEnabled:RotationEnabled, KmsKeyId:KmsKeyId, VersionIdsToStages:VersionIdsToStages, RotationLambdaARN:RotationLambdaARN}'arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-vNAQay
arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-vNAQay
{
"RotationEnabled": true,
"KmsKeyId": "d021b380-23c5-4157-a347-9017027be7e2",
"VersionIdsToStages": {
"444ac165-2e90-4a19-a287-cf43d703cd09": [
"AWSPREVIOUS"
],
"c6228cb8-d23a-48e7-8e44-fdcb57c815fa": [
"AWSPENDING",
"AWSCURRENT"
]
},
"RotationLambdaARN": "arn:aws:lambda:us-east-1:000000000000:function:secrets-rotator"
}
[stdout]
arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-vNAQay
arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-vNAQay
{
"RotationEnabled": true,
"KmsKeyId": "d021b380-23c5-4157-a347-9017027be7e2",
"VersionIdsToStages": {
"444ac165-2e90-4a19-a287-cf43d703cd09": [
"AWSPREVIOUS"
],
"c6228cb8-d23a-48e7-8e44-fdcb57c815fa": [
"AWSPENDING",
"AWSCURRENT"
]
},
"RotationLambdaARN": "arn:aws:lambda:us-east-1:000000000000:function:secrets-rotator"
}Check if rotator ran and what current value is
export AWS_ENDPOINT_URL=http://localstack:4566 # Check if my rotator lambda was invoked (or if localstack did its own swap) aws logs describe-log-streams --log-group-name /aws/lambda/secrets-rotator 2>/dev/null --query 'logStreams[].logStreamName' --output text echo "---" # Check current secret value vs what we set aws secretsmanager get-secret-value --secret-id app/db/password --version-stage AWSCURRENT --query SecretString --output text
2026/04/23/[$LATEST]0535271212ee4b0bb3e8df1c6eda3edc 2026/04/23/[$LATEST]271412de694edba3455c2473893f47fe 2026/04/23/[$LATEST]790822bec676856092a0e531bddd36fd 2026/04/23/[$LATEST]7f24aaace389fe0701566318b6cbc87b
---
{"username": "app", "password": "zbSPBlkjGHjMyorPILAjW_Ex1mT1dNzk"}
[stdout]
2026/04/23/[$LATEST]0535271212ee4b0bb3e8df1c6eda3edc 2026/04/23/[$LATEST]271412de694edba3455c2473893f47fe 2026/04/23/[$LATEST]790822bec676856092a0e531bddd36fd 2026/04/23/[$LATEST]7f24aaace389fe0701566318b6cbc87b
---
{"username": "app", "password": "zbSPBlkjGHjMyorPILAjW_Ex1mT1dNzk"}{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "states.amazonaws.com"},
"Action": "sts:AssumeRole"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "RotateTheSecret",
"Effect": "Allow",
"Action": "secretsmanager:RotateSecret",
"Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-*"
},
{
"Sid": "EmitRotationEvent",
"Effect": "Allow",
"Action": "events:PutEvents",
"Resource": "arn:aws:events:us-east-1:000000000000:event-bus/rotation-events"
}
]
}
{
"Comment": "Rotate app/db/password, then emit RotationComplete to the rotation-events bus",
"StartAt": "RotateSecret",
"States": {
"RotateSecret": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:secretsmanager:rotateSecret",
"Parameters": {
"SecretId": "app/db/password"
},
"ResultPath": "$.rotation",
"Next": "EmitRotationComplete"
},
"EmitRotationComplete": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:eventbridge:putEvents",
"Parameters": {
"Entries": [
{
"EventBusName": "rotation-events",
"Source": "rotation.pipeline",
"DetailType": "RotationComplete",
"Detail.$": "States.JsonToString($.rotation)"
}
]
},
"End": true
}
}
}
Create rotation-child SM and role
export AWS_ENDPOINT_URL=http://localstack:4566 aws iam create-role \ --role-name rotation-child-role \ --assume-role-policy-document file:///app/build/policies/states_trust.json \ --query 'Role.Arn' --output text aws iam put-role-policy \ --role-name rotation-child-role \ --policy-name rotation-child-inline \ --policy-document file:///app/build/policies/rotation_child_policy.json aws stepfunctions create-state-machine \ --name rotation-child \ --definition file:///app/build/sm/child.asl.json \ --role-arn arn:aws:iam::000000000000:role/rotation-child-role \ --type STANDARD \ --query 'stateMachineArn' --output text
arn:aws:iam::000000000000:role/rotation-child-role arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child [stdout] arn:aws:iam::000000000000:role/rotation-child-role arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child
Smoke-test child state machine
export AWS_ENDPOINT_URL=http://localstack:4566
# Quick smoke test: execute child SM directly and see if it runs
EXEC_ARN=$(aws stepfunctions start-execution \
--state-machine-arn arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child \
--input '{}' \
--query 'executionArn' --output text)
echo "exec=$EXEC_ARN"
for i in 1 2 3 4 5 6 7 8 9 10; do
status=$(aws stepfunctions describe-execution --execution-arn "$EXEC_ARN" --query 'status' --output text)
echo "status=$status"
case "$status" in RUNNING) sleep 1 ;; *) break ;; esac
done
aws stepfunctions describe-execution --execution-arn "$EXEC_ARN" --query '{status:status,output:output,cause:cause,error:error}'
echo "--- events ---"
aws stepfunctions get-execution-history --execution-arn "$EXEC_ARN" --query 'events[].{t:type,d:executionFailedEventDetails,f:taskFailedEventDetails}' --output json 2>&1 | head -80exec=arn:aws:states:us-east-1:000000000000:execution:rotation-child:2167196e-93cd-4e7a-bc62-84943e80d310
status=RUNNING
status=RUNNING
status=RUNNING
status=RUNNING
status=FAILED
{
"status": "FAILED",
"output": null,
"cause": "An error occurred while executing the state 'EmitRotationComplete' (entered at the event id #9). Unknown service: 'eventbridge'. Valid service names are: accessanalyzer, account, acm, acm-pca, aiops, amp, amplify, amplifybackend, amplifyuibuilder, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, appconfigdata, appfabric, appflow, appintegrations, application-autoscaling, application-insights, application-signals, applicationcostprofiler, appmesh, apprunner, appstream, appsync, arc-region-switch, arc-zonal-shift, artifact, athena, auditmanager, autoscaling, autoscaling-plans, b2bi, backup, backup-gateway, backupsearch, batch, bcm-dashboards, bcm-data-exports, bcm-pricing-calculator, bcm-recommended-actions, bedrock, bedrock-agent, bedrock-agent-runtime, bedrock-agentcore, bedrock-agentcore-control, bedrock-data-automation, bedrock-data-automation-runtime, bedrock-runtime, billing, billingconductor, braket, budgets, ce, chatbot, chime, chime-sdk-identity, chime-sdk-media-pipelines, chime-sdk-meetings, chime-sdk-messaging, chime-sdk-voice, cleanrooms, cleanroomsml, cloud9, cloudcontrol, clouddirectory, cloudformation, cloudfront, cloudfront-keyvaluestore, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudtrail-data, cloudwatch, codeartifact, codebuild, codecatalyst, codecommit, codeconnections, codedeploy, codeguru-reviewer, codeguru-security, codeguruprofiler, codepipeline, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, compute-optimizer-automation, config, connect, connect-contact-lens, connectcampaigns, connectcampaignsv2, connectcases, connectparticipant, controlcatalog, controltower, cost-optimization-hub, cur, customer-profiles, databrew, dataexchange, datapipeline, datasync, datazone, dax, deadline, detective, devicefarm, devops-guru, directconnect, discovery, dlm, dms, docdb, docdb-elastic, drs, ds, ds-data, dsql, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecr-public, ecs, efs, eks, eks-auth, elasticache, elasticbeanstalk, elb, elbv2, emr, emr-containers, emr-serverless, entityresolution, es, events, evidently, evs, finspace, finspace-data, firehose, fis, fms, forecast, forecastquery, frauddetector, freetier, fsx, gamelift, gameliftstreams, geo-maps, geo-places, geo-routes, glacier, globalaccelerator, glue, grafana, greengrass, greengrassv2, groundstation, guardduty, health, healthlake, iam, identitystore, imagebuilder, importexport, inspector, inspector-scan, inspector2, internetmonitor, invoicing, iot, iot-data, iot-jobs-data, iot-managed-integrations, iotanalytics, iotdeviceadvisor, iotevents, iotevents-data, iotfleetwise, iotsecuretunneling, iotsitewise, iotthingsgraph, iottwinmaker, iotwireless, ivs, ivs-realtime, ivschat, kafka, kafkaconnect, kendra, kendra-ranking, keyspaces, keyspacesstreams, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesis-video-webrtc-storage, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, launch-wizard, lex-models, lex-runtime, lexv2-models, lexv2-runtime, license-manager, license-manager-linux-subscriptions, license-manager-user-subscriptions, lightsail, location, logs, lookoutequipment, m2, machinelearning, macie2, mailmanager, managedblockchain, managedblockchain-query, marketplace-agreement, marketplace-catalog, marketplace-deployment, marketplace-entitlement, marketplace-reporting, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediapackagev2, mediastore, mediastore-data, mediatailor, medical-imaging, memorydb, meteringmarketplace, mgh, mgn, migration-hub-refactor-spaces, migrationhub-config, migrationhuborchestrator, migrationhubstrategy, mpa, mq, mturk, mwaa, mwaa-serverless, neptune, neptune-graph, neptunedata, network-firewall, networkflowmonitor, networkmanager, networkmonitor, notifications, notificationscontacts, nova-act, oam, observabilityadmin, odb, omics, opensearch, opensearchserverless, organizations, osis, outposts, panorama, partnercentral-account, partnercentral-benefits, partnercentral-channel, partnercentral-selling, payment-cryptography, payment-cryptography-data, pca-connector-ad, pca-connector-scep, pcs, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, pinpoint-sms-voice-v2, pipes, polly, pricing, proton, qapps, qbusiness, qconnect, quicksight, ram, rbin, rds, rds-data, redshift, redshift-data, redshift-serverless, rekognition, repostspace, resiliencehub, resource-explorer-2, resource-groups, resourcegroupstaggingapi, rolesanywhere, route53, route53-recovery-cluster, route53-recovery-control-config, route53-recovery-readiness, route53domains, route53globalresolver, route53profiles, route53resolver, rtbfabric, rum, s3, s3control, s3outposts, s3tables, s3vectors, sagemaker, sagemaker-a2i-runtime, sagemaker-edge, sagemaker-featurestore-runtime, sagemaker-geospatial, sagemaker-metrics, sagemaker-runtime, savingsplans, scheduler, schemas, sdb, secretsmanager, security-ir, securityhub, securitylake, serverlessrepo, service-quotas, servicecatalog, servicecatalog-appregistry, servicediscovery, ses, sesv2, shield, signer, signin, simspaceweaver, sms-voice, snow-device-management, snowball, sns, socialmessaging, sqs, sqs-query, ssm, ssm-contacts, ssm-guiconnect, ssm-incidents, ssm-quicksetup, ssm-sap, sso, sso-admin, sso-oidc, stepfunctions, storagegateway, sts, supplychain, support, support-app, swf, synthetics, taxsettings, textract, timestream-influxdb, timestream-query, timestream-write, tnb, transcribe, transfer, translate, trustedadvisor, verifiedpermissions, voice-id, vpc-lattice, waf, waf-regional, wafv2, wellarchitected, wickr, wisdom, workdocs, workmail, workmailmessageflow, workspaces, workspaces-instances, workspaces-thin-client, workspaces-web, xray",
"error": "States.Runtime"
}
--- events ---
[
{
"t": "ExecutionStarted",
"d": null,
"f": null
},
{
"t": "TaskStateEntered",
"d": null,
"f": null
},
{
"t": "TaskScheduled",
"d": null,
"f": null
},
{
"t": "TaskStarted",
"d": null,
"f": null
},
{
"t": "TaskSucceeded",
"d": null,
"f": null
},
{
"t": "TaskStateExited",
"d": null,
"f": null
},
{
"t": "TaskStateEntered",
"d": null,
"f": null
},
{
"t": "TaskScheduled",
"d": null,
"f": null
},
{
"t": "TaskStarted",
"d": null,
"f": null
},
{
"t": "TaskFailed",
"d": null,
"f": {
"error": "States.Runtime",
"cause": "Unknown service: 'eventbridge'. Valid service names are: accessanalyzer, account, acm, acm-pca, aiops, amp, amplify, amplifybackend, amplifyuibuilder, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, appconfigdata, appfabric, appflow, appintegrations, application-autoscaling, application-insights, application-signals, applicationcostprofiler, appmesh, apprunner, appstream, appsync, arc-region-switch, arc-zonal-shift, artifact, athena, auditmanager, autoscaling, autoscaling-plans, b2bi, backup, backup-gateway, backupsearch, batch, bcm-dashboards, bcm-data-exports, bcm-pricing-calculator, bcm-recommended-actions, bedrock, bedrock-agent, bedrock-agent-runtime, bedrock-agentcore, bedrock-agentcore-control, bedrock-data-automation, bedrock-data-automation-runtime, bedrock-runtime, billing, billingconductor, braket, budgets, ce, chatbot, chime, chime-sdk-identity, chime-sdk-media-pipelines, chime-sdk-meetings, chime-sdk-messaging, chime-sdk-voice, cleanrooms, cleanroomsml, cloud9, cloudcontrol, clouddirectory, cloudformation, cloudfront, cloudfront-keyvaluestore, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudtrail-data, cloudwatch, codeartifact, codebuild, codecatalyst, codecommit, codeconnections, codedeploy, codeguru-reviewer, codeguru-security, codeguruprofiler, codepipeline, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, compute-optimizer-automation, config, connect, connect-contact-lens, connectcampaigns, connectcampaignsv2, connectcases, connectparticipant, controlcatalog, controltower, cost-optimization-hub, cur, customer-profiles, databrew, dataexchange, datapipeline, datasync, datazone, dax, deadline, detective, devicefarm, devops-guru, directconnect, discovery, dlm, dms, docdb, docdb-elastic, drs, ds, ds-data, dsql, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecr-public, ecs, efs, eks, eks-auth, elasticache, elasticbeanstalk, elb, elbv2, emr, emr-containers, emr-serverless, entityresolution, es, events, evidently, evs, finspace, finspace-data, firehose, fis, fms, forecast, forecastquery, frauddetector, freetier, fsx, gamelift, gameliftstreams, geo-maps, geo-places, geo-routes, glacier, globalaccelerator, glue, grafana, greengrass, greengrassv2, groundstation, guardduty, health, healthlake, iam, identitystore, imagebuilder, importexport, inspector, inspector-scan, inspector2, internetmonitor, invoicing, iot, iot-data, iot-jobs-data, iot-managed-integrations, iotanalytics, iotdeviceadvisor, iotevents, iotevents-data, iotfleetwise, iotsecuretunneling, iotsitewise, iotthingsgraph, iottwinmaker, iotwireless, ivs, ivs-realtime, ivschat, kafka, kafkaconnect, kendra, kendra-ranking, keyspaces, keyspacesstreams, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesis-video-webrtc-storage, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, launch-wizard, lex-models, lex-runtime, lexv2-models, lexv2-runtime, license-manager, license-manager-linux-subscriptions, license-manager-user-subscriptions, lightsail, location, logs, lookoutequipment, m2, machinelearning, macie2, mailmanager, managedblockchain, managedblockchain-query, marketplace-agreement, marketplace-catalog, marketplace-deployment, marketplace-entitlement, marketplace-reporting, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediapackagev2, mediastore, mediastore-data, mediatailor, medical-imaging, memorydb, meteringmarketplace, mgh, mgn, migration-hub-refactor-spaces, migrationhub-config, migrationhuborchestrator, migrationhubstrategy, mpa, mq, mturk, mwaa, mwaa-serverless, neptune, neptune-graph, neptunedata, network-firewall, networkflowmonitor, networkmanager, networkmonitor, notifications, notificationscontacts, nova-act, oam, observabilityadmin, odb, omics, opensearch, opensearchserverless, organizations, osis, outposts, panorama, partnercentral-account, partnercentral-benefits, partnercentral-channel, partnercentral-selling, payment-cryptography, payment-cryptography-data, pca-connector-ad, pca-connector-scep, pcs, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, pinpoint-sms-voice-v2, pipes, polly, pricing, proton, qapps, qbusiness, qconnect, quicksight, ram, rbin, rds, rds-data, redshift, redshift-data, redshift-serverless, rekognition, repostspace, resiliencehub, resource-explorer-2, resource-groups, resourcegroupstaggingapi, rolesanywhere, route53, route53-recovery-cluster, route53-recovery-control-config, route53-recovery-readiness, route53domains, route53globalresolver, route53profiles, route53resolver, rtbfabric, rum, s3, s3control, s3outposts, s3tables, s3vectors, sagemaker, sagemaker-a2i-runtime, sagemaker-edge, sagemaker-featurestore-runtime, sagemaker-geospatial, sagemaker-metrics, sagemaker-runtime, savingsplans, scheduler, schemas, sdb, secretsmanager, security-ir, securityhub, securitylake, serverlessrepo, service-quotas, servicecatalog, servicecatalog-appregistry, servicediscovery, ses, sesv2, shield, signer, signin, simspaceweaver, sms-voice, snow-device-management, snowball, sns, socialmessaging, sqs, sqs-query, ssm, ssm-contacts, ssm-guiconnect, ssm-incidents, ssm-quicksetup, ssm-sap, sso, sso-admin, sso-oidc, stepfunctions, storagegateway, sts, supplychain, support, support-app, swf, synthetics, taxsettings, textract, timestream-influxdb, timestream-query, timestream-write, tnb, transcribe, transfer, translate, trustedadvisor, verifiedpermissions, voice-id, vpc-lattice, waf, waf-regional, wafv2, wellarchitected, wickr, wisdom, workdocs, workmail, workmailmessageflow, workspaces, workspaces-instances, workspaces-thin-client, workspaces-web, xray"
}
},
{
"t": "ExecutionFailed",
"d": {
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'EmitRotationComplete' (entered at the event id #9). Unknown service: 'eventbridge'. Valid service names are: accessanalyzer, account, acm, acm-pca, aiops, amp, amplify, amplifybackend, amplifyuibuilder, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, appconfigdata, appfabric, appflow, appintegrations, application-autoscaling, application-insights, application-signals, applicationcostprofiler, appmesh, apprunner, appstream, appsync, arc-region-switch, arc-zonal-shift, artifact, athena, auditmanager, autoscaling, autoscaling-plans, b2bi, backup, backup-gateway, backupsearch, batch, bcm-dashboards, bcm-data-exports, bcm-pricing-calculator, bcm-recommended-actions, bedrock, bedrock-agent, bedrock-agent-runtime, bedrock-agentcore, bedrock-agentcore-control, bedrock-data-automation, bedrock-data-automation-runtime, bedrock-runtime, billing, billingconductor, braket, budgets, ce, chatbot, chime, chime-sdk-identity, chime-sdk-media-pipelines, chime-sdk-meetings, chime-sdk-messaging, chime-sdk-voice, cleanrooms, cleanroomsml, cloud9, cloudcontrol, clouddirectory, cloudformation, cloudfront, cloudfront-keyvaluestore, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudtrail-data, cloudwatch, codeartifact, codebuild, codecatalyst, codecommit, codeconnections, codedeploy, codeguru-reviewer, codeguru-security, codeguruprofiler, codepipeline, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, compute-optimizer-automation, config, connect, connect-contact-lens, connectcampaigns, connectcampaignsv2, connectcases, connectparticipant, controlcatalog, controltower, cost-optimization-hub, cur, customer-profiles, databrew, dataexchange, datapipeline, datasync, datazone, dax, deadline, detective, devicefarm, devops-guru, directconnect, discovery, dlm, dms, docdb, docdb-elastic, drs, ds, ds-data, dsql, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecr-public, ecs, efs, eks, eks-auth, elasticache, elasticbeanstalk, elb, elbv2, emr, emr-containers, emr-serverless, entityresolution, es, events, evidently, evs, finspace, finspace-data, firehose, fis, fms, forecast, forecastquery, frauddetector, freetier, fsx, gamelift, gameliftstreams, geo-maps, geo-places, geo-routes, glacier, globalaccelerator, glue, grafana, greengrass, greengrassv2, groundstation, guardduty, health, healthlake, iam, identitystore, imagebuilder, importexport, inspector, inspector-scan, inspector2, internetmonitor, invoicing, iot, iot-data, iot-jobs-data, iot-managed-integrations, iotanalytics, iotdeviceadvisor, iotevents, iotevents-data, iotfleetwise, iotsecuretunneling, iotsitewise, iotthingsgraph, iottwinmaker, iotwireless, ivs, ivs-realtime, ivschat, kafka, kafkaconnect, kendra, kendra-ranking, keyspaces, keyspacesstreams, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesis-video-webrtc-storage, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, launch-wizard, lex-models, lex-runtime, lexv2-models, lexv2-runtime, license-manager, license-manager-linux-subscriptions, license-manager-user-subscriptions, lightsail, location, logs, lookoutequipment, m2, machinelearning, macie2, mailmanager, managedblockchain, managedblockchain-query, marketplace-agreement, marketplace-catalog, marketplace-deployment, marketplace-entitlement, marketplace-reporting, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediapackagev2, mediastore, mediastore-data, mediatailor, medical-imaging, memorydb, meteringmarketplace, mgh, mgn, migration-hub-refactor-spaces, migrationhub-config, migrationhuborchestrator, migrationhubstrategy, mpa, mq, mturk, mwaa, mwaa-serverless, neptune, neptune-graph, neptunedata, network-firewall, networkflowmonitor, networkmanager, networkmonitor, notifications, notificationscontacts, nova-act, oam, observabilityadmin, odb, omics, opensearch, opensearchserverless, organizations, osis, outposts, panorama, partnercentral-account, partnercentral-benefits, partnercentral-channel, partnercentral-selling, payment-cryptography, payment-cryptography-data, pca-connector-ad, pca-connector-scep, pcs, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, pinpoint-sms-voice-v2, pipes, polly, pricing, proton, qapps, qbusiness, qconnect, quicksight, ram, rbin, rds, rds-data, redshift, redshift-data, redshift-serverless, rekognition, repostspace, resiliencehub, resource-explorer-2, resource-groups, resourcegroupstaggingapi, rolesanywhere, route53, route53-recovery-cluster, route53-recovery-control-config, route53-recovery-readiness, route53domains, route53globalresolver, route53profiles, route53resolver, rtbfabric, rum, s3, s3control, s3outposts, s3tables, s3vectors, sagemaker, sagemaker-a2i-runtime, sagemaker-edge, sagemaker-featurestore-runtime, sagemaker-geospatial, sagemaker-metrics, sagemaker-runtime, savingsplans, scheduler, schemas, sdb, secretsmanager, security-ir, securityhub, securitylake, serverlessrepo, service-quotas, servicecatalog, servicecatalog-appregistry, servicediscovery, ses, sesv2, shield, signer, signin, simspaceweaver, sms-voice, snow-device-management, snowball, sns, socialmessaging, sqs, sqs-query, ssm, ssm-contacts, ssm-guiconnect, ssm-incidents, ssm-quicksetup, ssm-sap, sso, sso-admin, sso-oidc, stepfunctions, storagegateway, sts, supplychain, support, support-app, swf, synthetics, taxsettings, textract, timestream-influxdb, timestream-query, timestream-write, tnb, transcribe, transfer, translate, trustedadvisor, verifiedpermissions, voice-id, vpc-lattice, waf, waf-regional, wafv2, wellarchitected, wickr, wisdom, workdocs, workmail, workmailmessageflow, workspaces, workspaces-instances, workspaces-thin-client, workspaces-web, xray"
},
"f": null
}
]
[stdout]
exec=arn:aws:states:us-east-1:000000000000:execution:rotation-child:2167196e-93cd-4e7a-bc62-84943e80d310
status=RUNNING
status=RUNNING
status=RUNNING
status=RUNNING
status=FAILED
{
"status": "FAILED",
"output": null,
"cause": "An error occurred while executing the state 'EmitRotationComplete' (entered at the event id #9). Unknown service: 'eventbridge'. Valid service names are: accessanalyzer, account, acm, acm-pca, aiops, amp, amplify, amplifybackend, amplifyuibuilder, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, appconfigdata, appfabric, appflow, appintegrations, application-autoscaling, application-insights, application-signals, applicationcostprofiler, appmesh, apprunner, appstream, appsync, arc-region-switch, arc-zonal-shift, artifact, athena, auditmanager, autoscaling, autoscaling-plans, b2bi, backup, backup-gateway, backupsearch, batch, bcm-dashboards, bcm-data-exports, bcm-pricing-calculator, bcm-recommended-actions, bedrock, bedrock-agent, bedrock-agent-runtime, bedrock-agentcore, bedrock-agentcore-control, bedrock-data-automation, bedrock-data-automation-runtime, bedrock-runtime, billing, billingconductor, braket, budgets, ce, chatbot, chime, chime-sdk-identity, chime-sdk-media-pipelines, chime-sdk-meetings, chime-sdk-messaging, chime-sdk-voice, cleanrooms, cleanroomsml, cloud9, cloudcontrol, clouddirectory, cloudformation, cloudfront, cloudfront-keyvaluestore, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudtrail-data, cloudwatch, codeartifact, codebuild, codecatalyst, codecommit, codeconnections, codedeploy, codeguru-reviewer, codeguru-security, codeguruprofiler, codepipeline, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, compute-optimizer-automation, config, connect, connect-contact-lens, connectcampaigns, connectcampaignsv2, connectcases, connectparticipant, controlcatalog, controltower, cost-optimization-hub, cur, customer-profiles, databrew, dataexchange, datapipeline, datasync, datazone, dax, deadline, detective, devicefarm, devops-guru, directconnect, discovery, dlm, dms, docdb, docdb-elastic, drs, ds, ds-data, dsql, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecr-public, ecs, efs, eks, eks-auth, elasticache, elasticbeanstalk, elb, elbv2, emr, emr-containers, emr-serverless, entityresolution, es, events, evidently, evs, finspace, finspace-data, firehose, fis, fms, forecast, forecastquery, frauddetector, freetier, fsx, gamelift, gameliftstreams, geo-maps, geo-places, geo-routes, glacier, globalaccelerator, glue, grafana, greengrass, greengrassv2, groundstation, guardduty, health, healthlake, iam, identitystore, imagebuilder, importexport, inspector, inspector-scan, inspector2, internetmonitor, invoicing, iot, iot-data, iot-jobs-data, iot-managed-integrations, iotanalytics, iotdeviceadvisor, iotevents, iotevents-data, iotfleetwise, iotsecuretunneling, iotsitewise, iotthingsgraph, iottwinmaker, iotwireless, ivs, ivs-realtime, ivschat, kafka, kafkaconnect, kendra, kendra-ranking, keyspaces, keyspacesstreams, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesis-video-webrtc-storage, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, launch-wizard, lex-models, lex-runtime, lexv2-models, lexv2-runtime, license-manager, license-manager-linux-subscriptions, license-manager-user-subscriptions, lightsail, location, logs, lookoutequipment, m2, machinelearning, macie2, mailmanager, managedblockchain, managedblockchain-query, marketplace-agreement, marketplace-catalog, marketplace-deployment, marketplace-entitlement, marketplace-reporting, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediapackagev2, mediastore, mediastore-data, mediatailor, medical-imaging, memorydb, meteringmarketplace, mgh, mgn, migration-hub-refactor-spaces, migrationhub-config, migrationhuborchestrator, migrationhubstrategy, mpa, mq, mturk, mwaa, mwaa-serverless, neptune, neptune-graph, neptunedata, network-firewall, networkflowmonitor, networkmanager, networkmonitor, notifications, notificationscontacts, nova-act, oam, observabilityadmin, odb, omics, opensearch, opensearchserverless, organizations, osis, outposts, panorama, partnercentral-account, partnercentral-benefits, partnercentral-channel, partnercentral-selling, payment-cryptography, payment-cryptography-data, pca-connector-ad, pca-connector-scep, pcs, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, pinpoint-sms-voice-v2, pipes, polly, pricing, proton, qapps, qbusiness, qconnect, quicksight, ram, rbin, rds, rds-data, redshift, redshift-data, redshift-serverless, rekognition, repostspace, resiliencehub, resource-explorer-2, resource-groups, resourcegroupstaggingapi, rolesanywhere, route53, route53-recovery-cluster, route53-recovery-control-config, route53-recovery-readiness, route53domains, route53globalresolver, route53profiles, route53resolver, rtbfabric, rum, s3, s3control, s3outposts, s3tables, s3vectors, sagemaker, sagemaker-a2i-runtime, sagemaker-edge, sagemaker-featurestore-runtime, sagemaker-geospatial, sagemaker-metrics, sagemaker-runtime, savingsplans, scheduler, schemas, sdb, secretsmanager, security-ir, securityhub, securitylake, serverlessrepo, service-quotas, servicecatalog, servicecatalog-appregistry, servicediscovery, ses, sesv2, shield, signer, signin, simspaceweaver, sms-voice, snow-device-management, snowball, sns, socialmessaging, sqs, sqs-query, ssm, ssm-contacts, ssm-guiconnect, ssm-incidents, ssm-quicksetup, ssm-sap, sso, sso-admin, sso-oidc, stepfunctions, storagegateway, sts, supplychain, support, support-app, swf, synthetics, taxsettings, textract, timestream-influxdb, timestream-query, timestream-write, tnb, transcribe, transfer, translate, trustedadvisor, verifiedpermissions, voice-id, vpc-lattice, waf, waf-regional, wafv2, wellarchitected, wickr, wisdom, workdocs, workmail, workmailmessageflow, workspaces, workspaces-instances, workspaces-thin-client, workspaces-web, xray",
"error": "States.Runtime"
}
--- events ---
[
{
"t": "ExecutionStarted",
"d": null,
"f": null
},
{
"t": "TaskStateEntered",
"d": null,
"f": null
},
{
"t": "TaskScheduled",
"d": null,
"f": null
},
{
"t": "TaskStarted",
"d": null,
"f": null
},
{
"t": "TaskSucceeded",
"d": null,
"f": null
},
{
"t": "TaskStateExited",
"d": null,
"f": null
},
{
"t": "TaskStateEntered",
"d": null,
"f": null
},
{
"t": "TaskScheduled",
"d": null,
"f": null
},
{
"t": "TaskStarted",
"d": null,
"f": null
},
{
"t": "TaskFailed",
"d": null,
"f": {
"error": "States.Runtime",
"cause": "Unknown service: 'eventbridge'. Valid service names are: accessanalyzer, account, acm, acm-pca, aiops, amp, amplify, amplifybackend, amplifyuibuilder, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, appconfigdata, appfabric, appflow, appintegrations, application-autoscaling, application-insights, application-signals, applicationcostprofiler, appmesh, apprunner, appstream, appsync, arc-region-switch, arc-zonal-shift, artifact, athena, auditmanager, autoscaling, autoscaling-plans, b2bi, backup, backup-gateway, backupsearch, batch, bcm-dashboards, bcm-data-exports, bcm-pricing-calculator, bcm-recommended-actions, bedrock, bedrock-agent, bedrock-agent-runtime, bedrock-agentcore, bedrock-agentcore-control, bedrock-data-automation, bedrock-data-automation-runtime, bedrock-runtime, billing, billingconductor, braket, budgets, ce, chatbot, chime, chime-sdk-identity, chime-sdk-media-pipelines, chime-sdk-meetings, chime-sdk-messaging, chime-sdk-voice, cleanrooms, cleanroomsml, cloud9, cloudcontrol, clouddirectory, cloudformation, cloudfront, cloudfront-keyvaluestore, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudtrail-data, cloudwatch, codeartifact, codebuild, codecatalyst, codecommit, codeconnections, codedeploy, codeguru-reviewer, codeguru-security, codeguruprofiler, codepipeline, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, compute-optimizer-automation, config, connect, connect-contact-lens, connectcampaigns, connectcampaignsv2, connectcases, connectparticipant, controlcatalog, controltower, cost-optimization-hub, cur, customer-profiles, databrew, dataexchange, datapipeline, datasync, datazone, dax, deadline, detective, devicefarm, devops-guru, directconnect, discovery, dlm, dms, docdb, docdb-elastic, drs, ds, ds-data, dsql, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecr-public, ecs, efs, eks, eks-auth, elasticache, elasticbeanstalk, elb, elbv2, emr, emr-containers, emr-serverless, entityresolution, es, events, evidently, evs, finspace, finspace-data, firehose, fis, fms, forecast, forecastquery, frauddetector, freetier, fsx, gamelift, gameliftstreams, geo-maps, geo-places, geo-routes, glacier, globalaccelerator, glue, grafana, greengrass, greengrassv2, groundstation, guardduty, health, healthlake, iam, identitystore, imagebuilder, importexport, inspector, inspector-scan, inspector2, internetmonitor, invoicing, iot, iot-data, iot-jobs-data, iot-managed-integrations, iotanalytics, iotdeviceadvisor, iotevents, iotevents-data, iotfleetwise, iotsecuretunneling, iotsitewise, iotthingsgraph, iottwinmaker, iotwireless, ivs, ivs-realtime, ivschat, kafka, kafkaconnect, kendra, kendra-ranking, keyspaces, keyspacesstreams, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesis-video-webrtc-storage, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, launch-wizard, lex-models, lex-runtime, lexv2-models, lexv2-runtime, license-manager, license-manager-linux-subscriptions, license-manager-user-subscriptions, lightsail, location, logs, lookoutequipment, m2, machinelearning, macie2, mailmanager, managedblockchain, managedblockchain-query, marketplace-agreement, marketplace-catalog, marketplace-deployment, marketplace-entitlement, marketplace-reporting, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediapackagev2, mediastore, mediastore-data, mediatailor, medical-imaging, memorydb, meteringmarketplace, mgh, mgn, migration-hub-refactor-spaces, migrationhub-config, migrationhuborchestrator, migrationhubstrategy, mpa, mq, mturk, mwaa, mwaa-serverless, neptune, neptune-graph, neptunedata, network-firewall, networkflowmonitor, networkmanager, networkmonitor, notifications, notificationscontacts, nova-act, oam, observabilityadmin, odb, omics, opensearch, opensearchserverless, organizations, osis, outposts, panorama, partnercentral-account, partnercentral-benefits, partnercentral-channel, partnercentral-selling, payment-cryptography, payment-cryptography-data, pca-connector-ad, pca-connector-scep, pcs, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, pinpoint-sms-voice-v2, pipes, polly, pricing, proton, qapps, qbusiness, qconnect, quicksight, ram, rbin, rds, rds-data, redshift, redshift-data, redshift-serverless, rekognition, repostspace, resiliencehub, resource-explorer-2, resource-groups, resourcegroupstaggingapi, rolesanywhere, route53, route53-recovery-cluster, route53-recovery-control-config, route53-recovery-readiness, route53domains, route53globalresolver, route53profiles, route53resolver, rtbfabric, rum, s3, s3control, s3outposts, s3tables, s3vectors, sagemaker, sagemaker-a2i-runtime, sagemaker-edge, sagemaker-featurestore-runtime, sagemaker-geospatial, sagemaker-metrics, sagemaker-runtime, savingsplans, scheduler, schemas, sdb, secretsmanager, security-ir, securityhub, securitylake, serverlessrepo, service-quotas, servicecatalog, servicecatalog-appregistry, servicediscovery, ses, sesv2, shield, signer, signin, simspaceweaver, sms-voice, snow-device-management, snowball, sns, socialmessaging, sqs, sqs-query, ssm, ssm-contacts, ssm-guiconnect, ssm-incidents, ssm-quicksetup, ssm-sap, sso, sso-admin, sso-oidc, stepfunctions, storagegateway, sts, supplychain, support, support-app, swf, synthetics, taxsettings, textract, timestream-influxdb, timestream-query, timestream-write, tnb, transcribe, transfer, translate, trustedadvisor, verifiedpermissions, voice-id, vpc-lattice, waf, waf-regional, wafv2, wellarchitected, wickr, wisdom, workdocs, workmail, workmailmessageflow, workspaces, workspaces-instances, workspaces-thin-client, workspaces-web, xray"
}
},
{
"t": "ExecutionFailed",
"d": {
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'EmitRotationComplete' (entered at the event id #9). Unknown service: 'eventbridge'. Valid service names are: accessanalyzer, account, acm, acm-pca, aiops, amp, amplify, amplifybackend, amplifyuibuilder, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, appconfigdata, appfabric, appflow, appintegrations, application-autoscaling, application-insights, application-signals, applicationcostprofiler, appmesh, apprunner, appstream, appsync, arc-region-switch, arc-zonal-shift, artifact, athena, auditmanager, autoscaling, autoscaling-plans, b2bi, backup, backup-gateway, backupsearch, batch, bcm-dashboards, bcm-data-exports, bcm-pricing-calculator, bcm-recommended-actions, bedrock, bedrock-agent, bedrock-agent-runtime, bedrock-agentcore, bedrock-agentcore-control, bedrock-data-automation, bedrock-data-automation-runtime, bedrock-runtime, billing, billingconductor, braket, budgets, ce, chatbot, chime, chime-sdk-identity, chime-sdk-media-pipelines, chime-sdk-meetings, chime-sdk-messaging, chime-sdk-voice, cleanrooms, cleanroomsml, cloud9, cloudcontrol, clouddirectory, cloudformation, cloudfront, cloudfront-keyvaluestore, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudtrail-data, cloudwatch, codeartifact, codebuild, codecatalyst, codecommit, codeconnections, codedeploy, codeguru-reviewer, codeguru-security, codeguruprofiler, codepipeline, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, compute-optimizer-automation, config, connect, connect-contact-lens, connectcampaigns, connectcampaignsv2, connectcases, connectparticipant, controlcatalog, controltower, cost-optimization-hub, cur, customer-profiles, databrew, dataexchange, datapipeline, datasync, datazone, dax, deadline, detective, devicefarm, devops-guru, directconnect, discovery, dlm, dms, docdb, docdb-elastic, drs, ds, ds-data, dsql, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecr-public, ecs, efs, eks, eks-auth, elasticache, elasticbeanstalk, elb, elbv2, emr, emr-containers, emr-serverless, entityresolution, es, events, evidently, evs, finspace, finspace-data, firehose, fis, fms, forecast, forecastquery, frauddetector, freetier, fsx, gamelift, gameliftstreams, geo-maps, geo-places, geo-routes, glacier, globalaccelerator, glue, grafana, greengrass, greengrassv2, groundstation, guardduty, health, healthlake, iam, identitystore, imagebuilder, importexport, inspector, inspector-scan, inspector2, internetmonitor, invoicing, iot, iot-data, iot-jobs-data, iot-managed-integrations, iotanalytics, iotdeviceadvisor, iotevents, iotevents-data, iotfleetwise, iotsecuretunneling, iotsitewise, iotthingsgraph, iottwinmaker, iotwireless, ivs, ivs-realtime, ivschat, kafka, kafkaconnect, kendra, kendra-ranking, keyspaces, keyspacesstreams, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesis-video-webrtc-storage, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, launch-wizard, lex-models, lex-runtime, lexv2-models, lexv2-runtime, license-manager, license-manager-linux-subscriptions, license-manager-user-subscriptions, lightsail, location, logs, lookoutequipment, m2, machinelearning, macie2, mailmanager, managedblockchain, managedblockchain-query, marketplace-agreement, marketplace-catalog, marketplace-deployment, marketplace-entitlement, marketplace-reporting, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediapackagev2, mediastore, mediastore-data, mediatailor, medical-imaging, memorydb, meteringmarketplace, mgh, mgn, migration-hub-refactor-spaces, migrationhub-config, migrationhuborchestrator, migrationhubstrategy, mpa, mq, mturk, mwaa, mwaa-serverless, neptune, neptune-graph, neptunedata, network-firewall, networkflowmonitor, networkmanager, networkmonitor, notifications, notificationscontacts, nova-act, oam, observabilityadmin, odb, omics, opensearch, opensearchserverless, organizations, osis, outposts, panorama, partnercentral-account, partnercentral-benefits, partnercentral-channel, partnercentral-selling, payment-cryptography, payment-cryptography-data, pca-connector-ad, pca-connector-scep, pcs, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, pinpoint-sms-voice-v2, pipes, polly, pricing, proton, qapps, qbusiness, qconnect, quicksight, ram, rbin, rds, rds-data, redshift, redshift-data, redshift-serverless, rekognition, repostspace, resiliencehub, resource-explorer-2, resource-groups, resourcegroupstaggingapi, rolesanywhere, route53, route53-recovery-cluster, route53-recovery-control-config, route53-recovery-readiness, route53domains, route53globalresolver, route53profiles, route53resolver, rtbfabric, rum, s3, s3control, s3outposts, s3tables, s3vectors, sagemaker, sagemaker-a2i-runtime, sagemaker-edge, sagemaker-featurestore-runtime, sagemaker-geospatial, sagemaker-metrics, sagemaker-runtime, savingsplans, scheduler, schemas, sdb, secretsmanager, security-ir, securityhub, securitylake, serverlessrepo, service-quotas, servicecatalog, servicecatalog-appregistry, servicediscovery, ses, sesv2, shield, signer, signin, simspaceweaver, sms-voice, snow-device-management, snowball, sns, socialmessaging, sqs, sqs-query, ssm, ssm-contacts, ssm-guiconnect, ssm-incidents, ssm-quicksetup, ssm-sap, sso, sso-admin, sso-oidc, stepfunctions, storagegateway, sts, supplychain, support, support-app, swf, synthetics, taxsettings, textract, timestream-influxdb, timestream-query, timestream-write, tnb, transcribe, transfer, translate, trustedadvisor, verifiedpermissions, voice-id, vpc-lattice, waf, waf-regional, wafv2, wellarchitected, wickr, wisdom, workdocs, workmail, workmailmessageflow, workspaces, workspaces-instances, workspaces-thin-client, workspaces-web, xray"
},
"f": null
}
]/app/build/sm/child.asl.json
"Resource": "arn:aws:states:::aws-sdk:eventbridge:putEvents",
"Resource": "arn:aws:states:::aws-sdk:events:putEvents",
Update and re-test child SM
export AWS_ENDPOINT_URL=http://localstack:4566
aws stepfunctions update-state-machine \
--state-machine-arn arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child \
--definition file:///app/build/sm/child.asl.json --query 'updateDate' --output text
# Re-test
EXEC_ARN=$(aws stepfunctions start-execution \
--state-machine-arn arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child \
--input '{}' --query 'executionArn' --output text)
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
status=$(aws stepfunctions describe-execution --execution-arn "$EXEC_ARN" --query 'status' --output text)
echo "child status=$status"
case "$status" in RUNNING) sleep 1 ;; *) break ;; esac
done
aws stepfunctions describe-execution --execution-arn "$EXEC_ARN" --query '{status:status,error:error,cause:cause}'2026-04-23T06:06:28.712591+00:00
child status=RUNNING
child status=RUNNING
child status=RUNNING
child status=RUNNING
child status=SUCCEEDED
{
"status": "SUCCEEDED",
"error": null,
"cause": null
}
[stdout]
2026-04-23T06:06:28.712591+00:00
child status=RUNNING
child status=RUNNING
child status=RUNNING
child status=RUNNING
child status=SUCCEEDED
{
"status": "SUCCEEDED",
"error": null,
"cause": null
}Confirm notifier saw the event
export AWS_ENDPOINT_URL=http://localstack:4566 # Did the notifier log it? sleep 2 aws logs describe-log-streams --log-group-name /aws/lambda/notification-lambda --query 'logStreams[-1].logStreamName' --output text 2>/dev/null echo "---" LS=$(aws logs describe-log-streams --log-group-name /aws/lambda/notification-lambda --query 'logStreams[-1].logStreamName' --output text 2>/dev/null) aws logs get-log-events --log-group-name /aws/lambda/notification-lambda --log-stream-name "$LS" --query 'events[].message' --output text 2>&1 | head -20 echo "--- current secret ---" aws secretsmanager get-secret-value --secret-id app/db/password --version-stage AWSCURRENT --query SecretString --output text
2026/04/23/[$LATEST]8aa2bcea71cfac0205e260f0c65f3e8b
---
START RequestId: 2a832f53-f996-4fc2-af8a-ec575c57cfd9 Version: $LATEST
[INFO] 2026-04-23T06:06:34.793Z 2a832f53-f996-4fc2-af8a-ec575c57cfd9 notified: rotation complete secret=unknown version=unknown
notified: rotation complete secret=unknown version=unknown
END RequestId: 2a832f53-f996-4fc2-af8a-ec575c57cfd9
REPORT RequestId: 2a832f53-f996-4fc2-af8a-ec575c57cfd9 Duration: 9.94 ms Billed Duration: 10 ms Memory Size: 128 MB Max Memory Used: 128 MB
--- current secret ---
{"username": "app", "password": "pz0X_s3f16Z_Wu9kyWh_Se1z0P.L2Xtl"}
[stdout]
2026/04/23/[$LATEST]8aa2bcea71cfac0205e260f0c65f3e8b
---
START RequestId: 2a832f53-f996-4fc2-af8a-ec575c57cfd9 Version: $LATEST
[INFO] 2026-04-23T06:06:34.793Z 2a832f53-f996-4fc2-af8a-ec575c57cfd9 notified: rotation complete secret=unknown version=unknown
notified: rotation complete secret=unknown version=unknown
END RequestId: 2a832f53-f996-4fc2-af8a-ec575c57cfd9
REPORT RequestId: 2a832f53-f996-4fc2-af8a-ec575c57cfd9 Duration: 9.94 ms Billed Duration: 10 ms Memory Size: 128 MB Max Memory Used: 128 MB
--- current secret ---
{"username": "app", "password": "pz0X_s3f16Z_Wu9kyWh_Se1z0P.L2Xtl"}{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "InvokeChild",
"Effect": "Allow",
"Action": "states:StartExecution",
"Resource": "arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child"
},
{
"Sid": "ObserveChildExecution",
"Effect": "Allow",
"Action": [
"states:DescribeExecution",
"states:StopExecution"
],
"Resource": "arn:aws:states:us-east-1:000000000000:execution:rotation-child:*"
},
{
"Sid": "Sync2ManagedRule",
"Effect": "Allow",
"Action": [
"events:PutRule",
"events:PutTargets",
"events:DescribeRule"
],
"Resource": "arn:aws:events:us-east-1:000000000000:rule/StepFunctionsGetEventsForStepFunctionsExecutionRule"
}
]
}
{
"Comment": "Entry point for on-demand rotation. Delegates to rotation-child and waits.",
"StartAt": "InvokeChild",
"States": {
"InvokeChild": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution.sync:2",
"Parameters": {
"StateMachineArn": "arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child",
"Input": {}
},
"End": true
}
}
}
Create parent SM and role
export AWS_ENDPOINT_URL=http://localstack:4566 aws iam create-role \ --role-name rotation-orchestrator-role \ --assume-role-policy-document file:///app/build/policies/states_trust.json \ --query 'Role.Arn' --output text aws iam put-role-policy \ --role-name rotation-orchestrator-role \ --policy-name rotation-orchestrator-inline \ --policy-document file:///app/build/policies/rotation_orchestrator_policy.json aws stepfunctions create-state-machine \ --name rotation-orchestrator \ --definition file:///app/build/sm/parent.asl.json \ --role-arn arn:aws:iam::000000000000:role/rotation-orchestrator-role \ --type STANDARD \ --query 'stateMachineArn' --output text
arn:aws:iam::000000000000:role/rotation-orchestrator-role arn:aws:states:us-east-1:000000000000:stateMachine:rotation-orchestrator [stdout] arn:aws:iam::000000000000:role/rotation-orchestrator-role arn:aws:states:us-east-1:000000000000:stateMachine:rotation-orchestrator
Execute parent SM and wait for completion
export AWS_ENDPOINT_URL=http://localstack:4566
# Capture pre-rotation state
BEFORE_CURRENT=$(aws secretsmanager get-secret-value --secret-id app/db/password --version-stage AWSCURRENT --query '{ver:VersionId, str:SecretString}' --output json)
echo "BEFORE: $BEFORE_CURRENT"
START=$(date +%s)
EXEC_ARN=$(aws stepfunctions start-execution \
--state-machine-arn arn:aws:states:us-east-1:000000000000:stateMachine:rotation-orchestrator \
--query 'executionArn' --output text)
echo "parent exec=$EXEC_ARN"
for i in $(seq 1 90); do
status=$(aws stepfunctions describe-execution --execution-arn "$EXEC_ARN" --query 'status' --output text)
echo "[$i] parent status=$status"
case "$status" in RUNNING) sleep 1 ;; *) break ;; esac
done
END=$(date +%s)
echo "elapsed=$((END - START))s"
aws stepfunctions describe-execution --execution-arn "$EXEC_ARN" --query '{status:status,error:error,cause:cause,output:output}'BEFORE: {
"ver": "7df0764d-d914-49ec-93ad-b97143d4be76",
"str": "{\"username\": \"app\", \"password\": \"pz0X_s3f16Z_Wu9kyWh_Se1z0P.L2Xtl\"}"
}
parent exec=arn:aws:states:us-east-1:000000000000:execution:rotation-orchestrator:3fcb12b8-2fdf-47fb-b5cc-6deb947dfaa4
[1] parent status=RUNNING
[2] parent status=RUNNING
[3] parent status=RUNNING
[4] parent status=RUNNING
[5] parent status=SUCCEEDED
elapsed=7s
{
"status": "SUCCEEDED",
"error": null,
"cause": null,
"output": "{\"ExecutionArn\":\"arn:aws:states:us-east-1:000000000000:execution:rotation-child:8ba14de1-cb9e-4784-a473-3b866d90d350\",\"StateMachineArn\":\"arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child\",\"Name\":\"8ba14de1-cb9e-4784-a473-3b866d90d350\",\"Status\":\"SUCCEEDED\",\"StartDate\":\"2026-04-23T06:07:26.949225+00:00\",\"StopDate\":\"2026-04-23T06:07:31.879373+00:00\",\"Input\":{},\"InputDetails\":{\"Included\":true},\"Output\":{\"FailedEntryCount\":0,\"Entries\":[{\"EventId\":\"1b5763c6-2911-4307-96d2-a1f9eb843875\"}]},\"OutputDetails\":{\"Included\":true}}"
}
[stdout]
BEFORE: {
"ver": "7df0764d-d914-49ec-93ad-b97143d4be76",
"str": "{\"username\": \"app\", \"password\": \"pz0X_s3f16Z_Wu9kyWh_Se1z0P.L2Xtl\"}"
}
parent exec=arn:aws:states:us-east-1:000000000000:execution:rotation-orchestrator:3fcb12b8-2fdf-47fb-b5cc-6deb947dfaa4
[1] parent status=RUNNING
[2] parent status=RUNNING
[3] parent status=RUNNING
[4] parent status=RUNNING
[5] parent status=SUCCEEDED
elapsed=7s
{
"status": "SUCCEEDED",
"error": null,
"cause": null,
"output": "{\"ExecutionArn\":\"arn:aws:states:us-east-1:000000000000:execution:rotation-child:8ba14de1-cb9e-4784-a473-3b866d90d350\",\"StateMachineArn\":\"arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child\",\"Name\":\"8ba14de1-cb9e-4784-a473-3b866d90d350\",\"Status\":\"SUCCEEDED\",\"StartDate\":\"2026-04-23T06:07:26.949225+00:00\",\"StopDate\":\"2026-04-23T06:07:31.879373+00:00\",\"Input\":{},\"InputDetails\":{\"Included\":true},\"Output\":{\"FailedEntryCount\":0,\"Entries\":[{\"EventId\":\"1b5763c6-2911-4307-96d2-a1f9eb843875\"}]},\"OutputDetails\":{\"Included\":true}}"
}Verify all success criteria
export AWS_ENDPOINT_URL=http://localstack:4566
echo "=== 1. Parent SUCCEEDED ==="
aws stepfunctions list-executions --state-machine-arn arn:aws:states:us-east-1:000000000000:stateMachine:rotation-orchestrator --max-results 1 --query 'executions[0].{status:status,name:name}'
echo "=== 2. Secret version stages ==="
aws secretsmanager describe-secret --secret-id app/db/password --query 'VersionIdsToStages'
echo "=== 3. Secret AWSCURRENT value ==="
aws secretsmanager get-secret-value --secret-id app/db/password --version-stage AWSCURRENT --query '{ver:VersionId, str:SecretString}' --output json
echo "=== 4. Secret AWSPREVIOUS value ==="
aws secretsmanager get-secret-value --secret-id app/db/password --version-stage AWSPREVIOUS --query '{ver:VersionId, str:SecretString}' --output json 2>&1
echo "=== 5. Notifier logs ==="
sleep 2
LS=$(aws logs describe-log-streams --log-group-name /aws/lambda/notification-lambda --query 'logStreams[-1].logStreamName' --output text)
aws logs get-log-events --log-group-name /aws/lambda/notification-lambda --log-stream-name "$LS" --query 'events[].message' --output text | grep -i 'notified:' | head -5=== 1. Parent SUCCEEDED ===
{
"status": "SUCCEEDED",
"name": "3fcb12b8-2fdf-47fb-b5cc-6deb947dfaa4"
}
=== 2. Secret version stages ===
{
"7df0764d-d914-49ec-93ad-b97143d4be76": [
"AWSPREVIOUS"
],
"0b7f1f90-0fba-4a59-89e7-0e6d76b7e95e": [
"AWSPENDING",
"AWSCURRENT"
]
}
=== 3. Secret AWSCURRENT value ===
{
"ver": "0b7f1f90-0fba-4a59-89e7-0e6d76b7e95e",
"str": "{\"username\": \"app\", \"password\": \"L~MaQy5oq1QSXG6RGoIWICSOX~VZGXeK\"}"
}
=== 4. Secret AWSPREVIOUS value ===
{
"ver": "7df0764d-d914-49ec-93ad-b97143d4be76",
"str": "{\"username\": \"app\", \"password\": \"pz0X_s3f16Z_Wu9kyWh_Se1z0P.L2Xtl\"}"
}
=== 5. Notifier logs ===
[INFO] 2026-04-23T06:07:32.474Z 5b1fd326-df8c-4b68-a03d-ca15b1fccf7c notified: rotation complete secret=unknown version=unknown
notified: rotation complete secret=unknown version=unknown
[stdout]
=== 1. Parent SUCCEEDED ===
{
"status": "SUCCEEDED",
"name": "3fcb12b8-2fdf-47fb-b5cc-6deb947dfaa4"
}
=== 2. Secret version stages ===
{
"7df0764d-d914-49ec-93ad-b97143d4be76": [
"AWSPREVIOUS"
],
"0b7f1f90-0fba-4a59-89e7-0e6d76b7e95e": [
"AWSPENDING",
"AWSCURRENT"
]
}
=== 3. Secret AWSCURRENT value ===
{
"ver": "0b7f1f90-0fba-4a59-89e7-0e6d76b7e95e",
"str": "{\"username\": \"app\", \"password\": \"L~MaQy5oq1QSXG6RGoIWICSOX~VZGXeK\"}"
}
=== 4. Secret AWSPREVIOUS value ===
{
"ver": "7df0764d-d914-49ec-93ad-b97143d4be76",
"str": "{\"username\": \"app\", \"password\": \"pz0X_s3f16Z_Wu9kyWh_Se1z0P.L2Xtl\"}"
}
=== 5. Notifier logs ===
[INFO] 2026-04-23T06:07:32.474Z 5b1fd326-df8c-4b68-a03d-ca15b1fccf7c notified: rotation complete secret=unknown version=unknown
notified: rotation complete secret=unknown version=unknownConfirm consecutive rotations differ
export AWS_ENDPOINT_URL=http://localstack:4566 P1=$(aws secretsmanager get-secret-value --secret-id app/db/password --version-stage AWSCURRENT --query SecretString --output text) EXEC=$(aws stepfunctions start-execution --state-machine-arn arn:aws:states:us-east-1:000000000000:stateMachine:rotation-orchestrator --query 'executionArn' --output text) for i in $(seq 1 30); do s=$(aws stepfunctions describe-execution --execution-arn "$EXEC" --query 'status' --output text) [ "$s" != "RUNNING" ] && break sleep 1 done echo "second-run status=$s" P2=$(aws secretsmanager get-secret-value --secret-id app/db/password --version-stage AWSCURRENT --query SecretString --output text) echo "P1=$P1" echo "P2=$P2" [ "$P1" != "$P2" ] && echo "DIFFERENT: ok" || echo "SAME: PROBLEM"
second-run status=SUCCEEDED
P1={"username": "app", "password": "L~MaQy5oq1QSXG6RGoIWICSOX~VZGXeK"}
P2={"username": "app", "password": "zgtY9y0msEl6ct2ZS~3vJd1tKSBwrig_"}
DIFFERENT: ok
[stdout]
second-run status=SUCCEEDED
P1={"username": "app", "password": "L~MaQy5oq1QSXG6RGoIWICSOX~VZGXeK"}
P2={"username": "app", "password": "zgtY9y0msEl6ct2ZS~3vJd1tKSBwrig_"}
DIFFERENT: okFinal audit of policies and wiring
export AWS_ENDPOINT_URL=http://localstack:4566
echo "=== rotation-orchestrator-role inline ==="
aws iam get-role-policy --role-name rotation-orchestrator-role --policy-name rotation-orchestrator-inline --query PolicyDocument
echo "=== rotation-child-role inline ==="
aws iam get-role-policy --role-name rotation-child-role --policy-name rotation-child-inline --query PolicyDocument
echo "=== secrets-rotator-role inline ==="
aws iam get-role-policy --role-name secrets-rotator-role --policy-name secrets-rotator-inline --query PolicyDocument
echo "=== notification-lambda-role inline ==="
aws iam get-role-policy --role-name notification-lambda-role --policy-name notification-lambda-logs --query PolicyDocument
echo "=== Event rule pattern ==="
aws events describe-rule --name on-rotation-success --event-bus-name rotation-events --query '{bus:EventBusName, pat:EventPattern, state:State}'
echo "=== Event rule targets ==="
aws events list-targets-by-rule --rule on-rotation-success --event-bus-name rotation-events --query 'Targets[].Arn'
echo "=== Secret config ==="
aws secretsmanager describe-secret --secret-id app/db/password --query '{KmsKeyId:KmsKeyId, RotationEnabled:RotationEnabled, RotationLambdaARN:RotationLambdaARN}'=== rotation-orchestrator-role inline ===
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "InvokeChild",
"Effect": "Allow",
"Action": "states:StartExecution",
"Resource": "arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child"
},
{
"Sid": "ObserveChildExecution",
"Effect": "Allow",
"Action": [
"states:DescribeExecution",
"states:StopExecution"
],
"Resource": "arn:aws:states:us-east-1:000000000000:execution:rotation-child:*"
},
{
"Sid": "Sync2ManagedRule",
"Effect": "Allow",
"Action": [
"events:PutRule",
"events:PutTargets",
"events:DescribeRule"
],
"Resource": "arn:aws:events:us-east-1:000000000000:rule/StepFunctionsGetEventsForStepFunctionsExecutionRule"
}
]
}
=== rotation-child-role inline ===
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "RotateTheSecret",
"Effect": "Allow",
"Action": "secretsmanager:RotateSecret",
"Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-*"
},
{
"Sid": "EmitRotationEvent",
"Effect": "Allow",
"Action": "events:PutEvents",
"Resource": "arn:aws:events:us-east-1:000000000000:event-bus/rotation-events"
}
]
}
=== secrets-rotator-role inline ===
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Logs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/secrets-rotator:*"
},
{
"Sid": "SecretRotationOps",
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
],
"Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-*"
},
{
"Sid": "KmsForSecret",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:us-east-1:000000000000:key/d021b380-23c5-4157-a347-9017027be7e2"
}
]
}
=== notification-lambda-role inline ===
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:us-east-1:000000000000:*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/notification-lambda:*"
}
]
}
=== Event rule pattern ===
{
"bus": "rotation-events",
"pat": "{\n \"source\": [\"rotation.pipeline\"],\n \"detail-type\": [\"RotationComplete\"]\n}\n",
"state": "ENABLED"
}
=== Event rule targets ===
[
"arn:aws:lambda:us-east-1:000000000000:function:notification-lambda"
]
=== Secret config ===
{
"KmsKeyId": "d021b380-23c5-4157-a347-9017027be7e2",
"RotationEnabled": true,
"RotationLambdaARN": "arn:aws:lambda:us-east-1:000000000000:function:secrets-rotator"
}
[stdout]
=== rotation-orchestrator-role inline ===
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "InvokeChild",
"Effect": "Allow",
"Action": "states:StartExecution",
"Resource": "arn:aws:states:us-east-1:000000000000:stateMachine:rotation-child"
},
{
"Sid": "ObserveChildExecution",
"Effect": "Allow",
"Action": [
"states:DescribeExecution",
"states:StopExecution"
],
"Resource": "arn:aws:states:us-east-1:000000000000:execution:rotation-child:*"
},
{
"Sid": "Sync2ManagedRule",
"Effect": "Allow",
"Action": [
"events:PutRule",
"events:PutTargets",
"events:DescribeRule"
],
"Resource": "arn:aws:events:us-east-1:000000000000:rule/StepFunctionsGetEventsForStepFunctionsExecutionRule"
}
]
}
=== rotation-child-role inline ===
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "RotateTheSecret",
"Effect": "Allow",
"Action": "secretsmanager:RotateSecret",
"Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-*"
},
{
"Sid": "EmitRotationEvent",
"Effect": "Allow",
"Action": "events:PutEvents",
"Resource": "arn:aws:events:us-east-1:000000000000:event-bus/rotation-events"
}
]
}
=== secrets-rotator-role inline ===
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Logs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/secrets-rotator:*"
},
{
"Sid": "SecretRotationOps",
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
],
"Resource": "arn:aws:secretsmanager:us-east-1:000000000000:secret:app/db/password-*"
},
{
"Sid": "KmsForSecret",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:us-east-1:000000000000:key/d021b380-23c5-4157-a347-9017027be7e2"
}
]
}
=== notification-lambda-role inline ===
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:us-east-1:000000000000:*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/notification-lambda:*"
}
]
}
=== Event rule pattern ===
{
"bus": "rotation-events",
"pat": "{\n \"source\": [\"rotation.pipeline\"],\n \"detail-type\": [\"RotationComplete\"]\n}\n",
"state": "ENABLED"
}
=== Event rule targets ===
[
"arn:aws:lambda:us-east-1:000000000000:function:notification-lambda"
]
=== Secret config ===
{
"KmsKeyId": "d021b380-23c5-4157-a347-9017027be7e2",
"RotationEnabled": true,
"RotationLambdaARN": "arn:aws:lambda:us-east-1:000000000000:function:secrets-rotator"
}No file changes, the agent left the workspace untouched.
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/fTqqGgipPYWeNaqK3uROA/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 15 items
../tests/test_state.py::test_aws_cli_binary_is_real PASSED [ 6%]
../tests/test_state.py::test_localstack_reachable PASSED [ 13%]
../tests/test_state.py::test_secret_exists_and_uses_customer_cmk PASSED [ 20%]
../tests/test_state.py::test_rotation_lambda_exists PASSED [ 26%]
../tests/test_state.py::test_notification_lambda_exists PASSED [ 33%]
../tests/test_state.py::test_parent_and_child_state_machines_exist PASSED [ 40%]
../tests/test_state.py::test_event_bus_and_rule_exist PASSED [ 46%]
../tests/test_state.py::test_rule_target_points_at_notification_lambda PASSED [ 53%]
../tests/test_state.py::test_rotator_role_grants_kms_actions PASSED [ 60%]
../tests/test_state.py::test_kms_key_policy_grants_rotator_role FAILED [ 66%]
../tests/test_state.py::test_rotator_lambda_allows_secretsmanager_invoke PASSED [ 73%]
../tests/test_state.py::test_parent_role_grants_sync2_managed_rule_perms FAILED [ 80%]
../tests/test_state.py::test_child_role_can_rotate_and_publish PASSED [ 86%]
../tests/test_state.py::test_notification_lambda_allows_eventbridge_invoke PASSED [ 93%]
../tests/test_state.py::test_end_to_end_parent_execution_rotates_and_notifies PASSED [100%]
=================================== FAILURES ===================================
___________________ test_kms_key_policy_grants_rotator_role ____________________
iam = <botocore.client.IAM object at 0xffffb1bfdf40>
kms = <botocore.client.KMS object at 0xffffb2148560>
def test_kms_key_policy_grants_rotator_role(iam, kms):
role_arn = iam.get_role(RoleName=ROTATOR_ROLE)["Role"]["Arn"]
policy_str = kms.get_key_policy(KeyId=KEY_ALIAS, PolicyName="default")[
"Policy"
]
policy = json.loads(policy_str)
match = False
for st in policy.get("Statement", []):
if st.get("Effect") != "Allow":
continue
principal = st.get("Principal") or {}
if not isinstance(principal, dict):
continue
arns = set(_normalise_list(principal.get("AWS")))
if role_arn not in arns:
continue
actions = set(_normalise_list(st.get("Action")))
if _actions_cover(actions, REQUIRED_KMS_ACTIONS):
match = True
break
> assert match, (
f"KMS key policy on {KEY_ALIAS} has no Allow statement whose "
f"Principal.AWS includes {role_arn} and whose Action covers "
f"{sorted(REQUIRED_KMS_ACTIONS)}. Key policy: {policy_str}"
)
E AssertionError: KMS key policy on alias/app-rotation-key has no Allow statement whose Principal.AWS includes arn:aws:iam::000000000000:role/secrets-rotator-role and whose Action covers ['kms:Decrypt', 'kms:GenerateDataKey']. Key policy: {"Version": "2012-10-17", "Id": "key-default-1", "Statement": [{"Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::000000000000:root"}, "Action": "kms:*", "Resource": "*"}]}
E assert False
/tests/test_state.py:322: AssertionError
_______________ test_parent_role_grants_sync2_managed_rule_perms _______________
iam = <botocore.client.IAM object at 0xffffb1bfdf40>
def test_parent_role_grants_sync2_managed_rule_perms(iam):
"""The .sync:2 integration registers an EventBridge-managed callback
rule. Parent role needs events:PutRule/PutTargets/DescribeRule plus
full states:* on the child."""
found_events = False
found_states = False
for _pname, st in _inline_statements(iam, PARENT_ROLE):
actions = set(_normalise_list(st.get("Action")))
if _actions_cover(actions, REQUIRED_PARENT_EVENTS_ACTIONS):
found_events = True
if _actions_cover(actions, REQUIRED_PARENT_STATES_ACTIONS):
found_states = True
assert found_events, (
f"{PARENT_ROLE} missing events:PutRule/PutTargets/DescribeRule. "
f".sync:2 uses an EventBridge-managed callback rule; without "
f"these the parent cannot set up the callback."
)
> assert found_states, (
f"{PARENT_ROLE} missing states:StartExecution/DescribeExecution/"
f"StopExecution. .sync:2 requires all three on the child SM."
)
E AssertionError: rotation-orchestrator-role missing states:StartExecution/DescribeExecution/StopExecution. .sync:2 requires all three on the child SM.
E assert False
/tests/test_state.py:377: AssertionError
=============================== warnings summary ===============================
test_state.py: 35 warnings
/root/.cache/uv/archive-v0/fTqqGgipPYWeNaqK3uROA/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
datetime_now = datetime.datetime.utcnow()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_state.py::test_aws_cli_binary_is_real
PASSED ../tests/test_state.py::test_localstack_reachable
PASSED ../tests/test_state.py::test_secret_exists_and_uses_customer_cmk
PASSED ../tests/test_state.py::test_rotation_lambda_exists
PASSED ../tests/test_state.py::test_notification_lambda_exists
PASSED ../tests/test_state.py::test_parent_and_child_state_machines_exist
PASSED ../tests/test_state.py::test_event_bus_and_rule_exist
PASSED ../tests/test_state.py::test_rule_target_points_at_notification_lambda
PASSED ../tests/test_state.py::test_rotator_role_grants_kms_actions
PASSED ../tests/test_state.py::test_rotator_lambda_allows_secretsmanager_invoke
PASSED ../tests/test_state.py::test_child_role_can_rotate_and_publish
PASSED ../tests/test_state.py::test_notification_lambda_allows_eventbridge_invoke
PASSED ../tests/test_state.py::test_end_to_end_parent_execution_rotates_and_notifies
FAILED ../tests/test_state.py::test_kms_key_policy_grants_rotator_role - Asse...
FAILED ../tests/test_state.py::test_parent_role_grants_sync2_managed_rule_perms
================== 2 failed, 13 passed, 35 warnings in 11.27s ==================Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_d7a928460bc84af6. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.
Trial trial_d7a928460bc84af6 · verifier authoritative; classifier explanatory.