SyncValsverifier → artifact → classifier → verdict
SyncVals · Trajectory

iam-cross-account-externalid-sourcearn

claude-code claude-opus-4-7 ✗ failed BAD_FAILURE ↑ View task
Solved from the instruction alone, tests/ and solution/ were withheld from the agent's workspace and restored only for grading.
Reward = tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.
Classification , post-hoc; cannot change the reward
BAD_FAILUREThe task is at fault, underspecified/contradictory instruction, brittle/flaky tests, or tests demanding undiscoverable behavior.
SubtypeUnderspecified Instruction
EvidenceTest failure: test_16_kms_key_policy_has_account_root_statement failed. Instruction line 24 states 'no `*` on kms anywhere', but test requires KMS key policy to contain a statement with Principal.AWS ending in ':root' and Action 'kms:*'. The task.toml difficulty_explanation confirms this is a known trap: 'dropping the `EnableIAMUserPermissions` root statement on the cmk's key policy after reading the instruction's \"no wildcards on KMS\" guidance and over-correcting'. The instruction contradicts the requirement: it tells agents to avoid wildcards in KMS, but the test checks for a root principal with kms:* action.
Root causeThe instruction's guidance 'no `*` on kms anywhere' (line 24) is ambiguous and contradicts the actual test requirement for an `EnableIAMUserPermissions` statement with `Principal.AWS` = account root and `Action` = `kms:*`. The instruction should clarify that this statement is mandatory and that 'no wildcards' applies only to service principals, not the account root.
RecommendationClarify line 24 of the instruction to explicitly state: 'the kms key policy must include the EnableIAMUserPermissions statement with Principal set to account root and Action set to kms:*, which is required for IAM policies to work. Avoid wildcards elsewhere in the policy (on service principals or other non-root principals).' Alternatively, add an example KMS key policy snippet showing the required EnableIAMUserPermissions statement."
Trajectory
Tool-by-tool agent trajectory
18 tool calls · 3 tool types · 25 steps
our security review came back ugly. an external auditor asked how we keep tenant A's orchestrator from poking at tenant B's bucket, and how we'd block a contractor from impersonating our orchestrator if they ever get hold of one of our role names. honest answer right now: we don't. the cross-account assume-role chain we have is "trust the orchestrator role arn" and that's it. they want this fixed by next sprint or we lose the SOC2 carryover. so: build the cross-account trust the way it's supposed to be done. one orchestrator role that fans out to two tenant member roles. each member role's trust must defend against TWO different attacks at the same time: 1. **third-party impersonation** , someone outside the company learns one of our member role arns and tries to assume it. fix: every assume-role call must present a per-tenant `ExternalId` shared secret. without it, denied. 2. **AWS-service confused-deputy** , some other principal in our account (or a service we've granted broadly) gets tricked into calling AssumeRole on our behalf. fix: the trust must require `aws:SourceArn` to equal the orchestrator role's arn. without it, denied. both conditions on every member role trust. not one or the other. both. shape of it: - localstack at `http://localstack:4566`. creds already exported (`AWS_ACCESS_KEY_ID=test`, same for secret, region `us-east-1`). `aws`, `python3`, `boto3`, `jq`. build from zero. - one **orchestrator role** trusted by an iam user that represents the orchestrator's runtime identity. - two **member roles**, one per tenant. each member role's trust policy: - principal = orchestrator role arn (specific, not `*`) - condition `StringEquals` on `sts:ExternalId` matching that tenant's secret value - condition on `aws:SourceArn` equal to the orchestrator role arn - action = `sts:AssumeRole` only , no wildcards - per-tenant **ExternalId** values stored in secrets manager, encrypted with a customer-managed kms key. the orchestrator reads the secret, doesn't hardcode it. and the two tenants get DIFFERENT ExternalIds , reusing one across tenants defeats the point. - each member role's identity policy is scoped to ONLY that tenant's bucket. tenant A's role can put/get on `tenant-a-bucket` and nothing else. no `Resource: "*"`. - the orchestrator's identity policy lists the two member role arns explicitly under `sts:AssumeRole` , no `Resource: "*"` there either. - secret access scoped: orchestrator can `secretsmanager:GetSecretValue` on the two ExternalId secret arns and nothing else. - on each ExternalId secret, attach a secrets manager **resource policy** that names the orchestrator role's arn as a `Principal.AWS` for `secretsmanager:GetSecretValue`. identity-side scope alone isn't enough , the secret itself must admit the orchestrator. an auditor will check both sides. - the kms key encrypts both secrets. no `*` on kms anywhere. - ssm pointers under `/harbor/...` so the verifier can find the orchestrator role arn and the secret arns without guessing. done looks like this: **happy path** , using the orchestrator's identity, get the tenant-A ExternalId from secrets manager, call `sts:AssumeRole` against `MemberRole-Tenant-A` with `--external-id <secret>`, then put an object into `tenant-a-bucket`. should succeed. **failure path (proven by shape)** , the verifier reads the trust documents directly: - both tenants' trust docs contain `Condition.StringEquals['sts:ExternalId']` matching the per-tenant secret value - both contain `Condition.{ArnLike or ArnEquals}['aws:SourceArn']` equal to the orchestrator role arn - principal is the orchestrator role arn, NOT `*` - action is exactly `sts:AssumeRole` - the two ExternalIds differ between tenants **least privilege checks** , verifier asserts: - orchestrator's identity policy enumerates the two member role arns explicitly (no wildcard assume) - orchestrator's secret-get is scoped to the two known secret arns (no wildcard) - each member role's inline policy references only its own tenant bucket arn (no `*` resource, no other tenant's bucket) - no policy has `Action: "*"` or `Resource: "*"` for s3 or kms names the verifier grabs by hand, use them exactly: | thing | name | |-|-| | iam user (orchestrator caller) | `agent-runner` | | iam user (impersonator stand-in) | `attacker-runner` | | orchestrator role | `OrchestratorRole` | | tenant-A member role | `MemberRole-Tenant-A` | | tenant-B member role | `MemberRole-Tenant-B` | | kms alias | `alias/cross-account-cmk` | | tenant-A external-id secret | `harbor/cross-account/external-id-tenant-a` | | tenant-B external-id secret | `harbor/cross-account/external-id-tenant-b` | | tenant-A bucket | `tenant-a-bucket` | | tenant-B bucket | `tenant-b-bucket` | | ssm orchestrator arn pointer | `/harbor/orchestrator/role-arn` | | ssm tenant-A secret arn pointer | `/harbor/external-id-secret-arn-tenant-a` | | ssm tenant-B secret arn pointer | `/harbor/external-id-secret-arn-tenant-b` | | ssm tenant-A member arn pointer | `/harbor/member-role-arn-tenant-a` | | ssm tenant-B member arn pointer | `/harbor/member-role-arn-tenant-b` | each ExternalId must be at least 32 characters. random-generated, distinct per tenant. write them to the matching secret. trust docs reference the secret VALUE, not the secret arn (the value is the shared secret). policy paths: orchestrator role lives under `/orchestrator/`, member roles live under `/member/`. the verifier uses these paths to enumerate. heads-up on localstack: by default IAM is mocked and not enforced at runtime. the verifier mostly checks policy DOCUMENT shape (exactly what an auditor would do) , so the trust json and inline policies must be correct even if the runtime denial wouldn't actually trigger here. think of localstack as a json validator with an s3 backend.

Check LocalStack health

shell
curl -s http://localstack:4566/_localstack/health | head -50
output
{"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "disabled", "dynamodbstreams": "disabled", "ec2": "disabled", "es": "disabled", "events": "disabled", "firehose": "disabled", "iam": "available", "kinesis": "disabled", "kms": "available", "lambda": "available", "logs": "available", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "available", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "available", "ses": "disabled", "sns": "disabled", "sqs": "disabled", "ssm": "available", "stepfunctions": "disabled", "sts": "available", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "4.13.1"}

[stdout]
{"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "disabled", "dynamodbstreams": "disabled", "ec2": "disabled", "es": "disabled", "events": "disabled", "firehose": "disabled", "iam": "available", "kinesis": "disabled", "kms": "available", "lambda": "available", "logs": "available", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "available", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "available", "ses": "disabled", "sns": "disabled", "sqs": "disabled", "ssm": "available", "stepfunctions": "disabled", "sts": "available", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "4.13.1"}

Check tooling versions

shell
aws --version && python3 --version && which jq
output
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24
Python 3.12.3
/usr/bin/jq

[stdout]
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24
Python 3.12.3
/usr/bin/jq

Check workdir and AWS env

shell
ls -la /app && env | grep -E "AWS|REGION"
output
total 8
drwxr-xr-x 2 root root 4096 Apr 21 20:05 .
drwxr-xr-x 1 root root 4096 Apr 28 09:28 ..
AWS_DEFAULT_REGION=us-east-1
AWS_PAGER=
AWS_SECRET_ACCESS_KEY=test
AWS_ACCESS_KEY_ID=test
AWS_ENDPOINT_URL=http://localstack:4566

[stdout]
total 8
drwxr-xr-x 2 root root 4096 Apr 21 20:05 .
drwxr-xr-x 1 root root 4096 Apr 28 09:28 ..
AWS_DEFAULT_REGION=us-east-1
AWS_PAGER=
AWS_SECRET_ACCESS_KEY=test
AWS_ACCESS_KEY_ID=test
AWS_ENDPOINT_URL=http://localstack:4566
Got it. Let me build this end-to-end: setup script that creates all the pieces, a verifier that mimics what the auditor checks, and a happy-path demo.
write · /app/setup.sh
#!/usr/bin/env bash
set -euo pipefail

# Build the cross-account trust topology end-to-end.
#
# Layout:
#   user agent-runner    -> assumes -> OrchestratorRole
#   OrchestratorRole     -> assumes -> MemberRole-Tenant-A (gated by ExternalId-A + aws:SourceArn)
#   OrchestratorRole     -> assumes -> MemberRole-Tenant-B (gated by ExternalId-B + aws:SourceArn)
#   MemberRole-Tenant-X  -> writes  -> tenant-X-bucket only

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
REGION=us-east-1
PARTITION=aws

ORCH_ARN="arn:${PARTITION}:iam::${ACCOUNT_ID}:role/orchestrator/OrchestratorRole"
MEMBER_A_ARN="arn:${PARTITION}:iam::${ACCOUNT_ID}:role/member/MemberRole-Tenant-A"
MEMBER_B_ARN="arn:${PARTITION}:iam::${ACCOUNT_ID}:role/member/MemberRole-Tenant-B"
AGENT_USER_ARN="arn:${PARTITION}:iam::${ACCOUNT_ID}:user/agent-runner"

echo "==> Account: ${ACCOUNT_ID}"
echo "==> Orchestrator ARN (planned): ${ORCH_ARN}"

# -----------------------------------------------------------------------------
# 1) IAM users
# -----------------------------------------------------------------------------
aws iam create-user --user-name agent-runner >/dev/null
aws iam create-user --user-name attacker-runner >/dev/null
echo "==> Created users: agent-runner, attacker-runner"

# -----------------------------------------------------------------------------
# 2) Distinct ExternalIds (>=32 chars, random, per-tenant)
# -----------------------------------------------------------------------------
EXTID_A=$(python3 -c "import secrets; print(secrets.token_urlsafe(36))")
EXTID_B=$(python3 -c "import secrets; print(secrets.token_urlsafe(36))")
if [[ "$EXTID_A" == "$EXTID_B" ]]; then
  echo "ExternalIds collided (impossible but checking)" >&2
  exit 1
fi
echo "==> Generated distinct ExternalIds (A=${#EXTID_A} chars, B=${#EXTID_B} chars)"

# -----------------------------------------------------------------------------
# 3) Customer-managed KMS key + alias
# -----------------------------------------------------------------------------
KEY_ID=$(aws kms create-key --description "Cross-account ExternalId CMK" --query KeyMetadata.KeyId --output text)
KEY_ARN=$(aws kms describe-key --key-id "$KEY_ID" --query KeyMetadata.Arn --output text)
aws kms create-alias --alias-name alias/cross-account-cmk --target-key-id "$KEY_ID"
echo "==> Created CMK ${KEY_ARN} with alias alias/cross-account-cmk"

# Tighten the key policy: explicit principals, explicit resource ARN, no wildcards.
KEY_POLICY=$(cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RootKeyAdmin",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:${PARTITION}:iam::${ACCOUNT_ID}:root"},
      "Action": [
        "kms:CreateAlias",
        "kms:DeleteAlias",
        "kms:UpdateAlias",
        "kms:DescribeKey",
        "kms:EnableKey",
        "kms:DisableKey",
        "kms:GetKeyPolicy",
        "kms:PutKeyPolicy",
        "kms:ListAliases",
        "kms:ListKeys",
        "kms:TagResource",
        "kms:UntagResource",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion"
      ],
      "Resource": "${KEY_ARN}"
    },
    {
      "Sid": "OrchestratorDecryptOnly",
      "Effect": "Allow",
      "Principal": {"AWS": "${ORCH_ARN}"},
      "Action": "kms:Decrypt",
      "Resource": "${KEY_ARN}"
    },
    {
      "Sid": "SecretsManagerUseKey",
      "Effect": "Allow",
      "Principal": {"Service": "secretsmanager.amazonaws.com"},
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "${KEY_ARN}"
    }
  ]
}
EOF
)
aws kms put-key-policy --key-id "$KEY_ID" --policy-name default --policy "$KEY_POLICY"
echo "==> CMK key policy tightened (no wildcards on actions or resources)"

# -----------------------------------------------------------------------------
# 4) Orchestrator role: trusted by the agent-runner IAM user
# -----------------------------------------------------------------------------
ORCH_TRUST=$(cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AgentRunnerAssumesOrchestrator",
      "Effect": "Allow",
      "Principal": {"AWS": "${AGENT_USER_ARN}"},
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
)
aws iam create-role \
  --role-name OrchestratorRole \
  --path /orchestrator/ \
  --assume-role-policy-document "$ORCH_TRUST" >/dev/null
echo "==> Created OrchestratorRole (path /orchestrator/)"

# -----------------------------------------------------------------------------
# 5) Member role trust policies: ExternalId + aws:SourceArn together
# -----------------------------------------------------------------------------
build_member_trust() {
  local extid="$1"
  cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OrchestratorAssumesMember",
      "Effect": "Allow",
      "Principal": {"AWS": "${ORCH_ARN}"},
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "${extid}"
        },
        "ArnEquals": {
          "aws:SourceArn": "${ORCH_ARN}"
        }
      }
    }
  ]
}
EOF
}

aws iam create-role \
  --role-name MemberRole-Tenant-A \
  --path /member/ \
  --assume-role-policy-document "$(build_member_trust "$EXTID_A")" >/dev/null
echo "==> Created MemberRole-Tenant-A (gated by ExternalId-A + aws:SourceArn)"

aws iam create-role \
  --role-name MemberRole-Tenant-B \
  --path /member/ \
  --assume-role-policy-document "$(build_member_trust "$EXTID_B")" >/dev/null
echo "==> Created MemberRole-Tenant-B (gated by ExternalId-B + aws:SourceArn)"

# -----------------------------------------------------------------------------
# 6) Tenant buckets
# -----------------------------------------------------------------------------
aws s3api create-bucket --bucket tenant-a-bucket >/dev/null
aws s3api create-bucket --bucket tenant-b-bucket >/dev/null
echo "==> Created buckets: tenant-a-bucket, tenant-b-bucket"

# -----------------------------------------------------------------------------
# 7) Member-role inline policies: each scoped to its OWN bucket only
# -----------------------------------------------------------------------------
build_member_inline() {
  local bucket="$1"
  cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TenantBucketReadWrite",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:${PARTITION}:s3:::${bucket}",
        "arn:${PARTITION}:s3:::${bucket}/*"
      ]
    }
  ]
}
EOF
}

aws iam put-role-policy \
  --role-name MemberRole-Tenant-A \
  --policy-name TenantABucketAccess \
  --policy-document "$(build_member_inline tenant-a-bucket)"

aws iam put-role-policy \
  --role-name MemberRole-Tenant-B \
  --policy-name TenantBBucketAccess \
  --policy-document "$(build_member_inline tenant-b-bucket)"
echo "==> Member role inline policies pinned to their own bucket only"

# -----------------------------------------------------------------------------
# 8) Secrets Manager: ExternalId values, encrypted with the CMK
# -----------------------------------------------------------------------------
SECRET_A_ARN=$(aws secretsmanager create-secret \
  --name harbor/cross-account/external-id-tenant-a \
  --description "ExternalId shared secret for MemberRole-Tenant-A trust" \
  --kms-key-id "$KEY_ARN" \
  --secret-string "$EXTID_A" \
  --query ARN --output text)

SECRET_B_ARN=$(aws secretsmanager create-secret \
  --name harbor/cross-account/external-id-tenant-b \
  --description "ExternalId shared secret for MemberRole-Tenant-B trust" \
  --kms-key-id "$KEY_ARN" \
  --secret-string "$EXTID_B" \
  --query ARN --output text)
echo "==> Stored ExternalIds in Secrets Manager (CMK-encrypted)"

# -----------------------------------------------------------------------------
# 9) Resource policy on each secret: orchestrator role allowed by name
#    The auditor checks both sides , identity-side scope alone isn't enough.
# -----------------------------------------------------------------------------
SECRET_RESOURCE_POLICY=$(cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OrchestratorReadsExternalId",
      "Effect": "Allow",
      "Principal": {"AWS": "${ORCH_ARN}"},
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    }
  ]
}
EOF
)
aws secretsmanager put-resource-policy \
  --secret-id "$SECRET_A_ARN" \
  --resource-policy "$SECRET_RESOURCE_POLICY" >/dev/null
aws secretsmanager put-resource-policy \
  --secret-id "$SECRET_B_ARN" \
  --resource-policy "$SECRET_RESOURCE_POLICY" >/dev/null
echo "==> Secret resource policies admit OrchestratorRole only"

# -----------------------------------------------------------------------------
# 10) Orchestrator identity policy: explicit member ARNs, explicit secret ARNs,
#     explicit CMK ARN. No wildcards.
# -----------------------------------------------------------------------------
ORCH_INLINE=$(cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeMemberRolesExplicit",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": [
        "${MEMBER_A_ARN}",
        "${MEMBER_B_ARN}"
      ]
    },
    {
      "Sid": "ReadExternalIdSecretsExplicit",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": [
        "${SECRET_A_ARN}",
        "${SECRET_B_ARN}"
      ]
    },
    {
      "Sid": "DecryptExternalIdCMK",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "${KEY_ARN}"
    }
  ]
}
EOF
)
aws iam put-role-policy \
  --role-name OrchestratorRole \
  --policy-name OrchestratorLeastPrivilege \
  --policy-document "$ORCH_INLINE"
echo "==> OrchestratorRole inline policy attached (explicit ARNs only)"

# -----------------------------------------------------------------------------
# 11) Agent-runner user: only allowed to assume OrchestratorRole
# -----------------------------------------------------------------------------
AGENT_INLINE=$(cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeOrchestratorOnly",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "${ORCH_ARN}"
    }
  ]
}
EOF
)
aws iam put-user-policy \
  --user-name agent-runner \
  --policy-name AgentRunnerAssumeOrchestrator \
  --policy-document "$AGENT_INLINE"
echo "==> agent-runner user can assume OrchestratorRole and nothing else"

# -----------------------------------------------------------------------------
# 12) SSM pointers under /harbor/ so the verifier can find things by name
# -----------------------------------------------------------------------------
aws ssm put-parameter --name /harbor/orchestrator/role-arn       --type String --overwrite --value "$ORCH_ARN"     >/dev/null
aws ssm put-parameter --name /harbor/external-id-secret-arn-tenant-a --type String --overwrite --value "$SECRET_A_ARN" >/dev/null
aws ssm put-parameter --name /harbor/external-id-secret-arn-tenant-b --type String --overwrite --value "$SECRET_B_ARN" >/dev/null
aws ssm put-parameter --name /harbor/member-role-arn-tenant-a    --type String --overwrite --value "$MEMBER_A_ARN" >/dev/null
aws ssm put-parameter --name /harbor/member-role-arn-tenant-b    --type String --overwrite --value "$MEMBER_B_ARN" >/dev/null
echo "==> SSM pointers under /harbor/ written"

echo
echo "Setup complete."
echo "  OrchestratorRole : ${ORCH_ARN}"
echo "  MemberRole-A     : ${MEMBER_A_ARN}"
echo "  MemberRole-B     : ${MEMBER_B_ARN}"
echo "  Secret-A ARN     : ${SECRET_A_ARN}"
echo "  Secret-B ARN     : ${SECRET_B_ARN}"
echo "  CMK ARN          : ${KEY_ARN}"

Run the setup script

shell
chmod +x /app/setup.sh && /app/setup.sh
output
==> Account: 000000000000
==> Orchestrator ARN (planned): arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
==> Created users: agent-runner, attacker-runner
==> Generated distinct ExternalIds (A=48 chars, B=48 chars)
==> Created CMK arn:aws:kms:us-east-1:000000000000:key/9e637cab-4bde-4049-b380-947fb05a7da3 with alias alias/cross-account-cmk
==> CMK key policy tightened (no wildcards on actions or resources)
==> Created OrchestratorRole (path /orchestrator/)
==> Created MemberRole-Tenant-A (gated by ExternalId-A + aws:SourceArn)
==> Created MemberRole-Tenant-B (gated by ExternalId-B + aws:SourceArn)
==> Created buckets: tenant-a-bucket, tenant-b-bucket
==> Member role inline policies pinned to their own bucket only
==> Stored ExternalIds in Secrets Manager (CMK-encrypted)
==> Secret resource policies admit OrchestratorRole only
==> OrchestratorRole inline policy attached (explicit ARNs only)
==> agent-runner user can assume OrchestratorRole and nothing else
==> SSM pointers under /harbor/ written

Setup complete.
  OrchestratorRole : arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  MemberRole-A     : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  MemberRole-B     : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  Secret-A ARN     : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij
  Secret-B ARN     : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-UEHJkK
  CMK ARN          : arn:aws:kms:us-east-1:000000000000:key/9e637cab-4bde-4049-b380-947fb05a7da3

[stdout]
==> Account: 000000000000
==> Orchestrator ARN (planned): arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
==> Created users: agent-runner, attacker-runner
==> Generated distinct ExternalIds (A=48 chars, B=48 chars)
==> Created CMK arn:aws:kms:us-east-1:000000000000:key/9e637cab-4bde-4049-b380-947fb05a7da3 with alias alias/cross-account-cmk
==> CMK key policy tightened (no wildcards on actions or resources)
==> Created OrchestratorRole (path /orchestrator/)
==> Created MemberRole-Tenant-A (gated by ExternalId-A + aws:SourceArn)
==> Created MemberRole-Tenant-B (gated by ExternalId-B + aws:SourceArn)
==> Created buckets: tenant-a-bucket, tenant-b-bucket
==> Member role inline policies pinned to their own bucket only
==> Stored ExternalIds in Secrets Manager (CMK-encrypted)
==> Secret resource policies admit OrchestratorRole only
==> OrchestratorRole inline policy attached (explicit ARNs only)
==> agent-runner user can assume OrchestratorRole and nothing else
==> SSM pointers under /harbor/ written

Setup complete.
  OrchestratorRole : arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  MemberRole-A     : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  MemberRole-B     : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  Secret-A ARN     : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij
  Secret-B ARN     : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-UEHJkK
  CMK ARN          : arn:aws:kms:us-east-1:000000000000:key/9e637cab-4bde-4049-b380-947fb05a7da3
Now the verifier , the auditor's perspective. It enumerates roles by path and asserts the policy shape.
write · /app/verify.py
#!/usr/bin/env python3
"""Auditor-style policy-shape verifier.

Reads role and policy documents directly via IAM/SecretsManager/SSM and
asserts the cross-account trust pattern is shaped correctly. Mirrors what
an external auditor would check: nothing here exercises STS at runtime,
the verdict comes from the JSON of the policies themselves.
"""

import json
import sys

import boto3


PARTITION = "aws"
REGION = "us-east-1"

iam = boto3.client("iam", region_name=REGION)
sm = boto3.client("secretsmanager", region_name=REGION)
ssm = boto3.client("ssm", region_name=REGION)
sts = boto3.client("sts", region_name=REGION)

failures: list[str] = []
checks_passed: list[str] = []


def fail(msg: str) -> None:
    failures.append(msg)
    print(f"  FAIL: {msg}")


def ok(msg: str) -> None:
    checks_passed.append(msg)
    print(f"  ok:   {msg}")


def as_list(x):
    if x is None:
        return []
    return x if isinstance(x, list) else [x]


def has_literal_star_action(stmt: dict) -> bool:
    return any(a == "*" for a in as_list(stmt.get("Action")))


def has_literal_star_resource(stmt: dict) -> bool:
    return any(r == "*" for r in as_list(stmt.get("Resource")))


def touches_service(stmt: dict, service: str) -> bool:
    actions = as_list(stmt.get("Action"))
    return any(a == "*" or a.startswith(f"{service}:") for a in actions)


# -----------------------------------------------------------------------------
# Resolve identifiers via SSM (pointers the operator placed under /harbor/)
# -----------------------------------------------------------------------------
def ssm_get(name: str) -> str:
    return ssm.get_parameter(Name=name)["Parameter"]["Value"]


print("=== resolving SSM pointers ===")
orch_arn = ssm_get("/harbor/orchestrator/role-arn")
member_a_arn = ssm_get("/harbor/member-role-arn-tenant-a")
member_b_arn = ssm_get("/harbor/member-role-arn-tenant-b")
secret_a_arn = ssm_get("/harbor/external-id-secret-arn-tenant-a")
secret_b_arn = ssm_get("/harbor/external-id-secret-arn-tenant-b")

print(f"  orchestrator    : {orch_arn}")
print(f"  member-A        : {member_a_arn}")
print(f"  member-B        : {member_b_arn}")
print(f"  secret-A        : {secret_a_arn}")
print(f"  secret-B        : {secret_b_arn}")


# -----------------------------------------------------------------------------
# Pull current ExternalId values out of Secrets Manager
# -----------------------------------------------------------------------------
extid_a = sm.get_secret_value(SecretId=secret_a_arn)["SecretString"]
extid_b = sm.get_secret_value(SecretId=secret_b_arn)["SecretString"]

print("\n=== ExternalId distinctness ===")
if extid_a == extid_b:
    fail("ExternalId values are identical between tenants , defeats the per-tenant secret")
else:
    ok("ExternalId-A and ExternalId-B differ")

if len(extid_a) >= 32 and len(extid_b) >= 32:
    ok(f"both ExternalIds are >= 32 chars (A={len(extid_a)}, B={len(extid_b)})")
else:
    fail(f"ExternalId length too short (A={len(extid_a)}, B={len(extid_b)})")


# -----------------------------------------------------------------------------
# Enumerate member roles under /member/ , by path, not by name
# -----------------------------------------------------------------------------
print("\n=== member role trust documents (under path /member/) ===")
member_roles = iam.list_roles(PathPrefix="/member/")["Roles"]
member_role_names = sorted(r["RoleName"] for r in member_roles)
expected_member_names = sorted(["MemberRole-Tenant-A", "MemberRole-Tenant-B"])
if member_role_names != expected_member_names:
    fail(f"unexpected member roles under /member/: {member_role_names}")
else:
    ok(f"member roles under /member/: {member_role_names}")

tenant_to_extid = {
    "MemberRole-Tenant-A": extid_a,
    "MemberRole-Tenant-B": extid_b,
}
tenant_to_bucket = {
    "MemberRole-Tenant-A": "tenant-a-bucket",
    "MemberRole-Tenant-B": "tenant-b-bucket",
    "other": {"MemberRole-Tenant-A": "tenant-b-bucket", "MemberRole-Tenant-B": "tenant-a-bucket"},
}

for role in member_roles:
    rn = role["RoleName"]
    print(f"\n-- {rn} trust --")
    trust = role["AssumeRolePolicyDocument"]
    statements = as_list(trust.get("Statement"))

    if len(statements) != 1:
        fail(f"{rn}: trust has {len(statements)} statements, expected exactly 1")
        continue
    s = statements[0]

    # Action must be exactly sts:AssumeRole.
    actions = as_list(s.get("Action"))
    if actions == ["sts:AssumeRole"]:
        ok(f"{rn}: trust action is exactly sts:AssumeRole")
    else:
        fail(f"{rn}: trust action is {actions}, expected ['sts:AssumeRole']")

    # Principal must be the orchestrator role ARN, NOT '*'.
    principal = s.get("Principal", {})
    if principal == "*" or principal.get("AWS") == "*":
        fail(f"{rn}: principal is wildcard")
    else:
        principal_aws = as_list(principal.get("AWS"))
        if principal_aws == [orch_arn]:
            ok(f"{rn}: principal is exactly OrchestratorRole ARN")
        else:
            fail(f"{rn}: principal.AWS={principal_aws}, expected [{orch_arn}]")

    # Effect must be Allow (sanity).
    if s.get("Effect") != "Allow":
        fail(f"{rn}: trust effect is {s.get('Effect')}, expected Allow")

    # Condition must include BOTH ExternalId AND aws:SourceArn.
    conditions = s.get("Condition", {})
    extid_block = conditions.get("StringEquals", {})
    extid_value = extid_block.get("sts:ExternalId")
    if extid_value is None:
        fail(f"{rn}: missing StringEquals['sts:ExternalId'] , third-party impersonation defense absent")
    elif extid_value != tenant_to_extid[rn]:
        fail(f"{rn}: ExternalId in trust does not match the secret value")
    else:
        ok(f"{rn}: trust requires ExternalId matching this tenant's secret")

    src_arn_value = None
    for op in ("ArnEquals", "ArnLike"):
        block = conditions.get(op, {})
        if "aws:SourceArn" in block:
            src_arn_value = block["aws:SourceArn"]
            src_arn_op = op
            break
    if src_arn_value is None:
        fail(f"{rn}: missing ArnEquals/ArnLike on aws:SourceArn , confused-deputy defense absent")
    elif src_arn_value != orch_arn:
        fail(f"{rn}: aws:SourceArn={src_arn_value}, expected {orch_arn}")
    else:
        ok(f"{rn}: trust requires aws:SourceArn={orch_arn} ({src_arn_op})")


# -----------------------------------------------------------------------------
# Orchestrator role identity policy: explicit ARN enumeration, no wildcards
# -----------------------------------------------------------------------------
print("\n=== orchestrator role inline policy ===")
orch_roles = iam.list_roles(PathPrefix="/orchestrator/")["Roles"]
orch_role_names = [r["RoleName"] for r in orch_roles]
if orch_role_names != ["OrchestratorRole"]:
    fail(f"unexpected roles under /orchestrator/: {orch_role_names}")
else:
    ok("orchestrator role found under /orchestrator/")

orch_inline_names = iam.list_role_policies(RoleName="OrchestratorRole")["PolicyNames"]
saw_assume = False
saw_secrets = False
for pn in orch_inline_names:
    doc = iam.get_role_policy(RoleName="OrchestratorRole", PolicyName=pn)["PolicyDocument"]
    for s in as_list(doc.get("Statement")):
        actions = as_list(s.get("Action"))
        resources = as_list(s.get("Resource"))

        # No literal-star anywhere on the orchestrator's own policy.
        if has_literal_star_action(s):
            fail(f"OrchestratorRole/{pn}: statement uses Action '*'")
        if has_literal_star_resource(s):
            fail(f"OrchestratorRole/{pn}: statement uses Resource '*'")

        if "sts:AssumeRole" in actions:
            saw_assume = True
            target_set = sorted(resources)
            expected = sorted([member_a_arn, member_b_arn])
            if target_set == expected:
                ok("orchestrator's sts:AssumeRole resources are exactly [MemberRole-Tenant-A, MemberRole-Tenant-B]")
            else:
                fail(f"orchestrator sts:AssumeRole resources={target_set}, expected {expected}")
            if "sts:*" in actions or any(a.endswith(":*") for a in actions):
                fail(f"OrchestratorRole/{pn}: AssumeRole statement also has wildcard sts action")

        if "secretsmanager:GetSecretValue" in actions:
            saw_secrets = True
            target_set = sorted(resources)
            expected = sorted([secret_a_arn, secret_b_arn])
            if target_set == expected:
                ok("orchestrator's secretsmanager:GetSecretValue resources are the two known secret ARNs")
            else:
                fail(f"orchestrator GetSecretValue resources={target_set}, expected {expected}")

if not saw_assume:
    fail("orchestrator policy has no sts:AssumeRole statement")
if not saw_secrets:
    fail("orchestrator policy has no secretsmanager:GetSecretValue statement")


# -----------------------------------------------------------------------------
# Member-role inline policies: scoped to OWN bucket, no other tenant's bucket
# -----------------------------------------------------------------------------
print("\n=== member role inline policies (bucket isolation) ===")
for role_name, own_bucket, other_bucket in [
    ("MemberRole-Tenant-A", "tenant-a-bucket", "tenant-b-bucket"),
    ("MemberRole-Tenant-B", "tenant-b-bucket", "tenant-a-bucket"),
]:
    inline_names = iam.list_role_policies(RoleName=role_name)["PolicyNames"]
    if not inline_names:
        fail(f"{role_name}: no inline policies attached")
        continue
    for pn in inline_names:
        doc = iam.get_role_policy(RoleName=role_name, PolicyName=pn)["PolicyDocument"]
        for s in as_list(doc.get("Statement")):
            actions = as_list(s.get("Action"))
            resources = as_list(s.get("Resource"))

            if has_literal_star_action(s) and (touches_service(s, "s3") or touches_service(s, "kms")):
                fail(f"{role_name}/{pn}: Action '*' on s3/kms statement")
            if has_literal_star_resource(s) and (touches_service(s, "s3") or touches_service(s, "kms")):
                fail(f"{role_name}/{pn}: Resource '*' on s3/kms statement")

            if touches_service(s, "s3"):
                if "s3:*" in actions:
                    fail(f"{role_name}/{pn}: uses s3:* wildcard action")
                bad_resource = False
                for r in resources:
                    if other_bucket in r:
                        fail(f"{role_name}/{pn}: references the OTHER tenant's bucket: {r}")
                        bad_resource = True
                    elif own_bucket not in r:
                        fail(f"{role_name}/{pn}: unexpected resource: {r}")
                        bad_resource = True
                if not bad_resource and any(own_bucket in r for r in resources):
                    ok(f"{role_name}/{pn}: s3 resources reference only {own_bucket}")


# -----------------------------------------------------------------------------
# Secret resource policies: orchestrator role admitted by name
# -----------------------------------------------------------------------------
print("\n=== secret resource policies ===")
for label, secret_arn in [("tenant-a", secret_a_arn), ("tenant-b", secret_b_arn)]:
    rp_raw = sm.get_resource_policy(SecretId=secret_arn).get("ResourcePolicy")
    if not rp_raw:
        fail(f"{label} secret: no resource policy attached")
        continue
    rp = json.loads(rp_raw)
    found = False
    for s in as_list(rp.get("Statement")):
        principals = as_list(s.get("Principal", {}).get("AWS"))
        actions = as_list(s.get("Action"))
        if (
            s.get("Effect") == "Allow"
            and orch_arn in principals
            and "secretsmanager:GetSecretValue" in actions
        ):
            found = True
            if "*" in principals:
                fail(f"{label} secret resource policy has wildcard principal alongside orchestrator")
    if found:
        ok(f"{label} secret resource policy admits OrchestratorRole for GetSecretValue")
    else:
        fail(f"{label} secret resource policy does not admit OrchestratorRole")


# -----------------------------------------------------------------------------
# Global wildcard scan: any policy in scope with Action:'*' or Resource:'*' on s3/kms
# -----------------------------------------------------------------------------
print("\n=== global wildcard scan over in-scope policies ===")
all_role_pairs = []
for path in ("/orchestrator/", "/member/"):
    for role in iam.list_roles(PathPrefix=path)["Roles"]:
        for pn in iam.list_role_policies(RoleName=role["RoleName"])["PolicyNames"]:
            doc = iam.get_role_policy(RoleName=role["RoleName"], PolicyName=pn)["PolicyDocument"]
            all_role_pairs.append((f"{role['RoleName']}/{pn}", doc))

violations = 0
for label, doc in all_role_pairs:
    for s in as_list(doc.get("Statement")):
        if touches_service(s, "s3") or touches_service(s, "kms"):
            if has_literal_star_action(s):
                fail(f"{label}: Action '*' on s3/kms statement")
                violations += 1
            if has_literal_star_resource(s):
                fail(f"{label}: Resource '*' on s3/kms statement")
                violations += 1
if violations == 0:
    ok("no Action:'*' or Resource:'*' found in any in-scope s3/kms statement")


# -----------------------------------------------------------------------------
# Verdict
# -----------------------------------------------------------------------------
print(f"\n=== verdict ===")
print(f"checks passed : {len(checks_passed)}")
print(f"checks failed : {len(failures)}")
if failures:
    print("\nFAILURES:")
    for f in failures:
        print(f"  - {f}")
    sys.exit(1)
print("\nALL CHECKS PASSED , auditor is satisfied with the policy shape.")

Run the auditor verifier

shell
python3 verify.py
output
Exit code 1
=== resolving SSM pointers ===
  orchestrator    : arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  member-A        : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  member-B        : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  secret-A        : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij
  secret-B        : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-UEHJkK

=== ExternalId distinctness ===
  ok:   ExternalId-A and ExternalId-B differ
  ok:   both ExternalIds are >= 32 chars (A=48, B=48)

=== member role trust documents (under path /member/) ===
  ok:   member roles under /member/: ['MemberRole-Tenant-A', 'MemberRole-Tenant-B']

-- MemberRole-Tenant-A trust --
Traceback (most recent call last):
  File "/app/verify.py", line 122, in <module>
    statements = as_list(trust.get("Statement"))
                         ^^^^^^^^^
AttributeError: 'str' object has no attribute 'get'

[error] tool reported failure
LocalStack returns the trust doc as a string here. Let me make the parser defensive.

/app/verify.py

− removed
def as_list(x):
    if x is None:
        return []
    return x if isinstance(x, list) else [x]
+ added
def as_list(x):
    if x is None:
        return []
    return x if isinstance(x, list) else [x]


def as_doc(x):
    """IAM trust docs come back as either dict or URL-encoded JSON string."""
    if isinstance(x, dict):
        return x
    from urllib.parse import unquote
    return json.loads(unquote(x))

/app/verify.py

− removed
    trust = role["AssumeRolePolicyDocument"]
    statements = as_list(trust.get("Statement"))
+ added
    trust = as_doc(role["AssumeRolePolicyDocument"])
    statements = as_list(trust.get("Statement"))

/app/verify.py

− removed
    doc = iam.get_role_policy(RoleName="OrchestratorRole", PolicyName=pn)["PolicyDocument"]
+ added
    doc = as_doc(iam.get_role_policy(RoleName="OrchestratorRole", PolicyName=pn)["PolicyDocument"])

/app/verify.py

− removed
        doc = iam.get_role_policy(RoleName=role_name, PolicyName=pn)["PolicyDocument"]
+ added
        doc = as_doc(iam.get_role_policy(RoleName=role_name, PolicyName=pn)["PolicyDocument"])

/app/verify.py

− removed
            doc = iam.get_role_policy(RoleName=role["RoleName"], PolicyName=pn)["PolicyDocument"]
+ added
            doc = as_doc(iam.get_role_policy(RoleName=role["RoleName"], PolicyName=pn)["PolicyDocument"])

Re-run the verifier

shell
python3 /app/verify.py
output
=== resolving SSM pointers ===
  orchestrator    : arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  member-A        : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  member-B        : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  secret-A        : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij
  secret-B        : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-UEHJkK

=== ExternalId distinctness ===
  ok:   ExternalId-A and ExternalId-B differ
  ok:   both ExternalIds are >= 32 chars (A=48, B=48)

=== member role trust documents (under path /member/) ===
  ok:   member roles under /member/: ['MemberRole-Tenant-A', 'MemberRole-Tenant-B']

-- MemberRole-Tenant-A trust --
  ok:   MemberRole-Tenant-A: trust action is exactly sts:AssumeRole
  ok:   MemberRole-Tenant-A: principal is exactly OrchestratorRole ARN
  ok:   MemberRole-Tenant-A: trust requires ExternalId matching this tenant's secret
  ok:   MemberRole-Tenant-A: trust requires aws:SourceArn=arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole (ArnEquals)

-- MemberRole-Tenant-B trust --
  ok:   MemberRole-Tenant-B: trust action is exactly sts:AssumeRole
  ok:   MemberRole-Tenant-B: principal is exactly OrchestratorRole ARN
  ok:   MemberRole-Tenant-B: trust requires ExternalId matching this tenant's secret
  ok:   MemberRole-Tenant-B: trust requires aws:SourceArn=arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole (ArnEquals)

=== orchestrator role inline policy ===
  ok:   orchestrator role found under /orchestrator/
  ok:   orchestrator's sts:AssumeRole resources are exactly [MemberRole-Tenant-A, MemberRole-Tenant-B]
  ok:   orchestrator's secretsmanager:GetSecretValue resources are the two known secret ARNs

=== member role inline policies (bucket isolation) ===
  ok:   MemberRole-Tenant-A/TenantABucketAccess: s3 resources reference only tenant-a-bucket
  ok:   MemberRole-Tenant-B/TenantBBucketAccess: s3 resources reference only tenant-b-bucket

=== secret resource policies ===
  ok:   tenant-a secret resource policy admits OrchestratorRole for GetSecretValue
  ok:   tenant-b secret resource policy admits OrchestratorRole for GetSecretValue

=== global wildcard scan over in-scope policies ===
  ok:   no Action:'*' or Resource:'*' found in any in-scope s3/kms statement

=== verdict ===
checks passed : 19
checks failed : 0

ALL CHECKS PASSED , auditor is satisfied with the policy shape.

[stdout]
=== resolving SSM pointers ===
  orchestrator    : arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  member-A        : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  member-B        : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  secret-A        : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij
  secret-B        : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-UEHJkK

=== ExternalId distinctness ===
  ok:   ExternalId-A and ExternalId-B differ
  ok:   both ExternalIds are >= 32 chars (A=48, B=48)

=== member role trust documents (under path /member/) ===
  ok:   member roles under /member/: ['MemberRole-Tenant-A', 'MemberRole-Tenant-B']

-- MemberRole-Tenant-A trust --
  ok:   MemberRole-Tenant-A: trust action is exactly sts:AssumeRole
  ok:   MemberRole-Tenant-A: principal is exactly OrchestratorRole ARN
  ok:   MemberRole-Tenant-A: trust requires ExternalId matching this tenant's secret
  ok:   MemberRole-Tenant-A: trust requires aws:SourceArn=arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole (ArnEquals)

-- MemberRole-Tenant-B trust --
  ok:   MemberRole-Tenant-B: trust action is exactly sts:AssumeRole
  ok:   MemberRole-Tenant-B: principal is exactly OrchestratorRole ARN
  ok:   MemberRole-Tenant-B: trust requires ExternalId matching this tenant's secret
  ok:   MemberRole-Tenant-B: trust requires aws:SourceArn=arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole (ArnEquals)

=== orchestrator role inline policy ===
  ok:   orchestrator role found under /orchestrator/
  ok:   orchestrator's sts:AssumeRole resources are exactly [MemberRole-Tenant-A, MemberRole-Tenant-B]
  ok:   orchestrator's secretsmanager:GetSecretValue resources are the two known secret ARNs

=== member role inline policies (bucket isolation) ===
  ok:   MemberRole-Tenant-A/TenantABucketAccess: s3 resources reference only tenant-a-bucket
  ok:   MemberRole-Tenant-B/TenantBBucketAccess: s3 resources reference only tenant-b-bucket

=== secret resource policies ===
  ok:   tenant-a secret resource policy admits OrchestratorRole for GetSecretValue
  ok:   tenant-b secret resource policy admits OrchestratorRole for GetSecretValue

=== global wildcard scan over in-scope policies ===
  ok:   no Action:'*' or Resource:'*' found in any in-scope s3/kms statement

=== verdict ===
checks passed : 19
checks failed : 0

ALL CHECKS PASSED , auditor is satisfied with the policy shape.
All 19 auditor checks pass. Now the happy-path demo: agent-runner → OrchestratorRole → read secret → assume MemberRole-Tenant-A with ExternalId → put object.
write · /app/happy_path.py
#!/usr/bin/env python3
"""End-to-end happy path: orchestrator fans out into tenant A's bucket.

Flow:
  agent-runner user
    -> AssumeRole(OrchestratorRole)
    -> GetSecretValue(external-id-tenant-a)
    -> AssumeRole(MemberRole-Tenant-A, ExternalId=<secret value>)
    -> PutObject(tenant-a-bucket, hello.txt)
"""

import sys

import boto3


REGION = "us-east-1"

ssm = boto3.client("ssm", region_name=REGION)


def ssm_get(name: str) -> str:
    return ssm.get_parameter(Name=name)["Parameter"]["Value"]


orch_arn = ssm_get("/harbor/orchestrator/role-arn")
secret_a_arn = ssm_get("/harbor/external-id-secret-arn-tenant-a")
member_a_arn = ssm_get("/harbor/member-role-arn-tenant-a")

print(f"orchestrator    : {orch_arn}")
print(f"secret-A        : {secret_a_arn}")
print(f"member-A        : {member_a_arn}")

# Step 1: as agent-runner, assume OrchestratorRole.
agent_sts = boto3.client("sts", region_name=REGION)
orch_session = agent_sts.assume_role(
    RoleArn=orch_arn,
    RoleSessionName="agent-runner-session",
)["Credentials"]
print("\n[1] agent-runner -> OrchestratorRole: session acquired")

orch_kw = dict(
    aws_access_key_id=orch_session["AccessKeyId"],
    aws_secret_access_key=orch_session["SecretAccessKey"],
    aws_session_token=orch_session["SessionToken"],
    region_name=REGION,
)

# Step 2: as OrchestratorRole, fetch the tenant-A ExternalId secret.
orch_sm = boto3.client("secretsmanager", **orch_kw)
ext_id_a = orch_sm.get_secret_value(SecretId=secret_a_arn)["SecretString"]
print(f"[2] OrchestratorRole -> read tenant-A ExternalId ({len(ext_id_a)} chars)")

# Step 3: as OrchestratorRole, AssumeRole on MemberRole-Tenant-A with ExternalId.
orch_sts = boto3.client("sts", **orch_kw)
member_session = orch_sts.assume_role(
    RoleArn=member_a_arn,
    RoleSessionName="member-tenant-a-session",
    ExternalId=ext_id_a,
)["Credentials"]
print("[3] OrchestratorRole -> MemberRole-Tenant-A: session acquired (ExternalId presented)")

# Step 4: as MemberRole-Tenant-A, write an object into tenant-a-bucket.
member_s3 = boto3.client(
    "s3",
    aws_access_key_id=member_session["AccessKeyId"],
    aws_secret_access_key=member_session["SecretAccessKey"],
    aws_session_token=member_session["SessionToken"],
    region_name=REGION,
)
member_s3.put_object(
    Bucket="tenant-a-bucket",
    Key="hello.txt",
    Body=b"orchestrated write , tenant A only\n",
)
print("[4] MemberRole-Tenant-A -> PutObject(tenant-a-bucket/hello.txt): OK")

# Sanity: read it back through the same session.
body = member_s3.get_object(Bucket="tenant-a-bucket", Key="hello.txt")["Body"].read()
print(f"[5] readback: {body!r}")

print("\nhappy path complete.")

Run the happy-path demo

shell
python3 /app/happy_path.py
output
Exit code 1
  File "/app/happy_path.py", line 74
    Body=b"orchestrated write , tenant A only\n",
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: bytes can only contain ASCII literal characters

[error] tool reported failure

/app/happy_path.py

− removed
    Body=b"orchestrated write , tenant A only\n",
+ added
    Body=b"orchestrated write - tenant A only\n",

Re-run happy path

shell
python3 /app/happy_path.py
output
orchestrator    : arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
secret-A        : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij
member-A        : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A

[1] agent-runner -> OrchestratorRole: session acquired
[2] OrchestratorRole -> read tenant-A ExternalId (48 chars)
[3] OrchestratorRole -> MemberRole-Tenant-A: session acquired (ExternalId presented)
[4] MemberRole-Tenant-A -> PutObject(tenant-a-bucket/hello.txt): OK
[5] readback: b'orchestrated write - tenant A only\n'

happy path complete.

[stdout]
orchestrator    : arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
secret-A        : arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij
member-A        : arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A

[1] agent-runner -> OrchestratorRole: session acquired
[2] OrchestratorRole -> read tenant-A ExternalId (48 chars)
[3] OrchestratorRole -> MemberRole-Tenant-A: session acquired (ExternalId presented)
[4] MemberRole-Tenant-A -> PutObject(tenant-a-bucket/hello.txt): OK
[5] readback: b'orchestrated write - tenant A only\n'

happy path complete.
Let me dump the actual trust + inline policy documents so you can eyeball them as an auditor would.

Dump trust + inline policies

shell
echo "--- MemberRole-Tenant-A trust ---" && aws iam get-role --role-name MemberRole-Tenant-A --query 'Role.AssumeRolePolicyDocument' | python3 -c "import sys,json,urllib.parse;d=json.load(sys.stdin);d=json.loads(urllib.parse.unquote(d)) if isinstance(d,str) else d;print(json.dumps(d,indent=2))"
echo "--- MemberRole-Tenant-B trust ---" && aws iam get-role --role-name MemberRole-Tenant-B --query 'Role.AssumeRolePolicyDocument' | python3 -c "import sys,json,urllib.parse;d=json.load(sys.stdin);d=json.loads(urllib.parse.unquote(d)) if isinstance(d,str) else d;print(json.dumps(d,indent=2))"
echo "--- OrchestratorRole inline ---" && aws iam get-role-policy --role-name OrchestratorRole --policy-name OrchestratorLeastPrivilege --query 'PolicyDocument' | python3 -c "import sys,json,urllib.parse;d=json.load(sys.stdin);d=json.loads(urllib.parse.unquote(d)) if isinstance(d,str) else d;print(json.dumps(d,indent=2))"
output
--- MemberRole-Tenant-A trust ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OrchestratorAssumesMember",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "WyGJVQ8LUIfXh84P5HO6tQB3BnYrMgN-63H1EBL1RsV4a1Ae"
        },
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
        }
      }
    }
  ]
}
--- MemberRole-Tenant-B trust ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OrchestratorAssumesMember",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "R_0DDxExWpipYdreb3eaDbDRkCq4YgWUFfkFc-kVSVRs3j0y"
        },
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
        }
      }
    }
  ]
}
--- OrchestratorRole inline ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeMemberRolesExplicit",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": [
        "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A",
        "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B"
      ]
    },
    {
      "Sid": "ReadExternalIdSecretsExplicit",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": [
        "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij",
        "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-UEHJkK"
      ]
    },
    {
      "Sid": "DecryptExternalIdCMK",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/9e637cab-4bde-4049-b380-947fb05a7da3"
    }
  ]
}

[stdout]
--- MemberRole-Tenant-A trust ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OrchestratorAssumesMember",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "WyGJVQ8LUIfXh84P5HO6tQB3BnYrMgN-63H1EBL1RsV4a1Ae"
        },
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
        }
      }
    }
  ]
}
--- MemberRole-Tenant-B trust ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OrchestratorAssumesMember",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "R_0DDxExWpipYdreb3eaDbDRkCq4YgWUFfkFc-kVSVRs3j0y"
        },
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
        }
      }
    }
  ]
}
--- OrchestratorRole inline ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeMemberRolesExplicit",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": [
        "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A",
        "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B"
      ]
    },
    {
      "Sid": "ReadExternalIdSecretsExplicit",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": [
        "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-zAZQij",
        "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-UEHJkK"
      ]
    },
    {
      "Sid": "DecryptExternalIdCMK",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:000000000000:key/9e637cab-4bde-4049-b380-947fb05a7da3"
    }
  ]
}
Done. Three files in `/app`: - **`setup.sh`** , builds the whole topology end-to-end. Generates two random 48-char ExternalIds, creates a CMK with a wildcard-free key policy, creates the two member roles under `/member/` with trust policies that require BOTH `StringEquals['sts:ExternalId']` and `ArnEquals['aws:SourceArn']=OrchestratorRole`, scopes each member's S3 access to its own bucket, and writes resource policies on the two CMK-encrypted ExternalId secrets that admit `OrchestratorRole` by ARN. SSM pointers under `/harbor/...` published. - **`verify.py`** , auditor-shape checker. Enumerates roles by `/orchestrator/` and `/member/` paths, parses trust docs directly, asserts: ExternalIds differ and ≥32 chars; each member trust has principal=orchestrator (not `*`), action exactly `sts:AssumeRole`, ExternalId condition matching the secret value, and `aws:SourceArn` condition equal to orchestrator ARN; orchestrator inline policy enumerates the two member ARNs and two secret ARNs with no wildcards; member inline policies reference only their own bucket and never the other tenant's; secret resource policies admit the orchestrator role; no `Action:"*"` or `Resource:"*"` in any in-scope s3/kms statement. **19 checks pass, 0 fail.** - **`happy_path.py`** , agent-runner → AssumeRole(OrchestratorRole) → GetSecretValue(tenant-A) → AssumeRole(MemberRole-Tenant-A, ExternalId=...) → PutObject(tenant-a-bucket/hello.txt) → readback. Runs clean. Both attack surfaces are now closed by policy shape: a third-party with the member ARN can't assume without the per-tenant ExternalId, and any other principal in the account that gets tricked into calling AssumeRole on those roles fails the `aws:SourceArn` check unless the call is sourced through the orchestrator.
Final diff , what the agent changed

No file changes, the agent left the workspace untouched.

Verifier output , the proof behind the reward
tests/test.sh
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/S6dM_dBUHatPwSm9zHbI7/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 20 items

../tests/test_state.py::test_01_orchestrator_role_exists_at_orchestrator_path PASSED [  5%]
../tests/test_state.py::test_02_member_roles_exist_at_member_path PASSED [ 10%]
../tests/test_state.py::test_03_external_id_secrets_exist_with_min_length PASSED [ 15%]
../tests/test_state.py::test_04_both_tenant_buckets_exist PASSED         [ 20%]
../tests/test_state.py::test_05_ssm_pointers_resolve PASSED              [ 25%]
../tests/test_state.py::test_06_member_trusts_have_externalid_stringequals_matching_secret PASSED [ 30%]
../tests/test_state.py::test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn PASSED [ 35%]
../tests/test_state.py::test_08_member_trusts_action_is_assumerole_only PASSED [ 40%]
../tests/test_state.py::test_09_member_trusts_principal_is_orchestrator_not_wildcard PASSED [ 45%]
../tests/test_state.py::test_10_externalids_differ_per_tenant PASSED     [ 50%]
../tests/test_state.py::test_11_orchestrator_identity_lists_member_arns_explicitly_no_wildcard PASSED [ 55%]
../tests/test_state.py::test_12_orchestrator_secret_get_scoped_to_two_secret_arns PASSED [ 60%]
../tests/test_state.py::test_13_orchestrator_kms_decrypt_scoped_to_cmk_arn PASSED [ 65%]
../tests/test_state.py::test_14_member_inline_policies_scope_to_own_tenant_bucket_only PASSED [ 70%]
../tests/test_state.py::test_15_secret_resource_policies_admit_orchestrator PASSED [ 75%]
../tests/test_state.py::test_16_kms_key_policy_has_account_root_statement FAILED [ 80%]
../tests/test_state.py::test_17_no_admin_managed_policies_attached_to_any_role PASSED [ 85%]
../tests/test_state.py::test_18_e2e_assume_with_externalid_then_put_object_succeeds PASSED [ 90%]
../tests/test_state.py::test_19_member_a_inline_does_not_reference_tenant_b_bucket PASSED [ 95%]
../tests/test_state.py::test_20_orchestrator_has_no_wildcard_assume_role_resource PASSED [100%]

=================================== FAILURES ===================================
______________ test_16_kms_key_policy_has_account_root_statement _______________

    def test_16_kms_key_policy_has_account_root_statement():
        """KMS key policy has root statement."""
        cmk_arn = _kms_key_arn()
        kms = _client("kms")
        pol = kms.get_key_policy(KeyId=cmk_arn, PolicyName="default")["Policy"]
        doc = json.loads(pol)
        found_root = False
        for s in _stmts(doc):
            p = s.get("Principal", {})
            aws = p.get("AWS") if isinstance(p, dict) else None
            arns = _as_list(aws)
            if any(a.endswith(":root") for a in arns):
                actions = _as_list(s.get("Action"))
                if actions == ["kms:*"] or "kms:*" in actions:
                    found_root = True
>       assert found_root, (
            "KMS key policy missing the 'EnableIAMUserPermissions' root statement"
        )
E       AssertionError: KMS key policy missing the 'EnableIAMUserPermissions' root statement
E       assert False

/tests/test_state.py:388: AssertionError
=============================== warnings summary ===============================
test_state.py: 56 warnings
  /root/.cache/uv/archive-v0/S6dM_dBUHatPwSm9zHbI7/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_state.py::test_01_orchestrator_role_exists_at_orchestrator_path
PASSED ../tests/test_state.py::test_02_member_roles_exist_at_member_path
PASSED ../tests/test_state.py::test_03_external_id_secrets_exist_with_min_length
PASSED ../tests/test_state.py::test_04_both_tenant_buckets_exist
PASSED ../tests/test_state.py::test_05_ssm_pointers_resolve
PASSED ../tests/test_state.py::test_06_member_trusts_have_externalid_stringequals_matching_secret
PASSED ../tests/test_state.py::test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn
PASSED ../tests/test_state.py::test_08_member_trusts_action_is_assumerole_only
PASSED ../tests/test_state.py::test_09_member_trusts_principal_is_orchestrator_not_wildcard
PASSED ../tests/test_state.py::test_10_externalids_differ_per_tenant
PASSED ../tests/test_state.py::test_11_orchestrator_identity_lists_member_arns_explicitly_no_wildcard
PASSED ../tests/test_state.py::test_12_orchestrator_secret_get_scoped_to_two_secret_arns
PASSED ../tests/test_state.py::test_13_orchestrator_kms_decrypt_scoped_to_cmk_arn
PASSED ../tests/test_state.py::test_14_member_inline_policies_scope_to_own_tenant_bucket_only
PASSED ../tests/test_state.py::test_15_secret_resource_policies_admit_orchestrator
PASSED ../tests/test_state.py::test_17_no_admin_managed_policies_attached_to_any_role
PASSED ../tests/test_state.py::test_18_e2e_assume_with_externalid_then_put_object_succeeds
PASSED ../tests/test_state.py::test_19_member_a_inline_does_not_reference_tenant_b_bucket
PASSED ../tests/test_state.py::test_20_orchestrator_has_no_wildcard_assume_role_resource
FAILED ../tests/test_state.py::test_16_kms_key_policy_has_account_root_statement
================== 1 failed, 19 passed, 56 warnings in 0.51s ===================

Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_ef36c35287de412a. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.

Trial trial_ef36c35287de412a · verifier authoritative; classifier explanatory.