SyncValsverifier → artifact → classifier → verdict
SyncVals · Trajectory

iam-cross-account-externalid-sourcearn

claude-code claude-opus-4-7 ✗ failed BAD_FAILURE ↑ View task
Solved from the instruction alone, tests/ and solution/ were withheld from the agent's workspace and restored only for grading.
Reward = tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.
Classification , post-hoc; cannot change the reward
BAD_FAILUREThe task is at fault, underspecified/contradictory instruction, brittle/flaky tests, or tests demanding undiscoverable behavior.
SubtypeUnderspecified Instruction
EvidenceTest 16 `test_16_kms_key_policy_has_account_root_statement` failed with: "KMS key policy missing the 'EnableIAMUserPermissions' root statement". The test requires a statement with Principal.AWS as the account root and Action=kms:*. The instruction states 'the kms key encrypts both secrets. no `*` on kms anywhere' which conflicts with the required AWS best practice of having EnableIAMUserPermissions on CMK policies. The task.toml difficulty_explanation even acknowledges this as a known failure pattern: 'dropping the `EnableIAMUserPermissions` root statement on the cmk's key policy after reading the instruction's 'no wildcards on KMS' guidance and over-correcting'."
Root causeThe instruction's guidance "no `*` on kms anywhere" is ambiguous and contradicts AWS best practices for CMK policies, which REQUIRE an EnableIAMUserPermissions root account statement with kms:* action. The agent interpreted this as "avoid all wildcards on KMS" rather than "don't grant wildcard KMS permissions to user roles," causing them to omit the required root statement.
RecommendationClarify the instruction to explicitly require the EnableIAMUserPermissions root statement on the CMK key policy: 'The KMS key policy must include an EnableIAMUserPermissions statement with Principal as the AWS account root and Action: kms:* (this is standard AWS best practice for CMKs). The 'no wildcards' guidance applies to user-facing identity policies and secret-access scoping, not to the account root statement in the key policy.'
Trajectory
Tool-by-tool agent trajectory
36 tool calls · 5 tool types · 46 steps
our security review came back ugly. an external auditor asked how we keep tenant A's orchestrator from poking at tenant B's bucket, and how we'd block a contractor from impersonating our orchestrator if they ever get hold of one of our role names. honest answer right now: we don't. the cross-account assume-role chain we have is "trust the orchestrator role arn" and that's it. they want this fixed by next sprint or we lose the SOC2 carryover. so: build the cross-account trust the way it's supposed to be done. one orchestrator role that fans out to two tenant member roles. each member role's trust must defend against TWO different attacks at the same time: 1. **third-party impersonation** , someone outside the company learns one of our member role arns and tries to assume it. fix: every assume-role call must present a per-tenant `ExternalId` shared secret. without it, denied. 2. **AWS-service confused-deputy** , some other principal in our account (or a service we've granted broadly) gets tricked into calling AssumeRole on our behalf. fix: the trust must require `aws:SourceArn` to equal the orchestrator role's arn. without it, denied. both conditions on every member role trust. not one or the other. both. shape of it: - localstack at `http://localstack:4566`. creds already exported (`AWS_ACCESS_KEY_ID=test`, same for secret, region `us-east-1`). `aws`, `python3`, `boto3`, `jq`. build from zero. - one **orchestrator role** trusted by an iam user that represents the orchestrator's runtime identity. - two **member roles**, one per tenant. each member role's trust policy: - principal = orchestrator role arn (specific, not `*`) - condition `StringEquals` on `sts:ExternalId` matching that tenant's secret value - condition on `aws:SourceArn` equal to the orchestrator role arn - action = `sts:AssumeRole` only , no wildcards - per-tenant **ExternalId** values stored in secrets manager, encrypted with a customer-managed kms key. the orchestrator reads the secret, doesn't hardcode it. and the two tenants get DIFFERENT ExternalIds , reusing one across tenants defeats the point. - each member role's identity policy is scoped to ONLY that tenant's bucket. tenant A's role can put/get on `tenant-a-bucket` and nothing else. no `Resource: "*"`. - the orchestrator's identity policy lists the two member role arns explicitly under `sts:AssumeRole` , no `Resource: "*"` there either. - secret access scoped: orchestrator can `secretsmanager:GetSecretValue` on the two ExternalId secret arns and nothing else. - on each ExternalId secret, attach a secrets manager **resource policy** that names the orchestrator role's arn as a `Principal.AWS` for `secretsmanager:GetSecretValue`. identity-side scope alone isn't enough , the secret itself must admit the orchestrator. an auditor will check both sides. - the kms key encrypts both secrets. no `*` on kms anywhere. - ssm pointers under `/harbor/...` so the verifier can find the orchestrator role arn and the secret arns without guessing. done looks like this: **happy path** , using the orchestrator's identity, get the tenant-A ExternalId from secrets manager, call `sts:AssumeRole` against `MemberRole-Tenant-A` with `--external-id <secret>`, then put an object into `tenant-a-bucket`. should succeed. **failure path (proven by shape)** , the verifier reads the trust documents directly: - both tenants' trust docs contain `Condition.StringEquals['sts:ExternalId']` matching the per-tenant secret value - both contain `Condition.{ArnLike or ArnEquals}['aws:SourceArn']` equal to the orchestrator role arn - principal is the orchestrator role arn, NOT `*` - action is exactly `sts:AssumeRole` - the two ExternalIds differ between tenants **least privilege checks** , verifier asserts: - orchestrator's identity policy enumerates the two member role arns explicitly (no wildcard assume) - orchestrator's secret-get is scoped to the two known secret arns (no wildcard) - each member role's inline policy references only its own tenant bucket arn (no `*` resource, no other tenant's bucket) - no policy has `Action: "*"` or `Resource: "*"` for s3 or kms names the verifier grabs by hand, use them exactly: | thing | name | |-|-| | iam user (orchestrator caller) | `agent-runner` | | iam user (impersonator stand-in) | `attacker-runner` | | orchestrator role | `OrchestratorRole` | | tenant-A member role | `MemberRole-Tenant-A` | | tenant-B member role | `MemberRole-Tenant-B` | | kms alias | `alias/cross-account-cmk` | | tenant-A external-id secret | `harbor/cross-account/external-id-tenant-a` | | tenant-B external-id secret | `harbor/cross-account/external-id-tenant-b` | | tenant-A bucket | `tenant-a-bucket` | | tenant-B bucket | `tenant-b-bucket` | | ssm orchestrator arn pointer | `/harbor/orchestrator/role-arn` | | ssm tenant-A secret arn pointer | `/harbor/external-id-secret-arn-tenant-a` | | ssm tenant-B secret arn pointer | `/harbor/external-id-secret-arn-tenant-b` | | ssm tenant-A member arn pointer | `/harbor/member-role-arn-tenant-a` | | ssm tenant-B member arn pointer | `/harbor/member-role-arn-tenant-b` | each ExternalId must be at least 32 characters. random-generated, distinct per tenant. write them to the matching secret. trust docs reference the secret VALUE, not the secret arn (the value is the shared secret). policy paths: orchestrator role lives under `/orchestrator/`, member roles live under `/member/`. the verifier uses these paths to enumerate. heads-up on localstack: by default IAM is mocked and not enforced at runtime. the verifier mostly checks policy DOCUMENT shape (exactly what an auditor would do) , so the trust json and inline policies must be correct even if the runtime denial wouldn't actually trigger here. think of localstack as a json validator with an s3 backend.
ToolSearch
{
  "query": "select:TodoWrite",
  "max_results": 1
}

, output, 
{"type": "tool_reference", "tool_name": "TodoWrite"}
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks

Check localstack connectivity and identity

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && aws sts get-caller-identity && aws --version
output
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24

[stdout]
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24

Create working directory

shell
mkdir -p /app/harbor && cd /app/harbor && ls -la
output
total 8
drwxr-xr-x 2 root root 4096 Apr 28 09:53 .
drwxr-xr-x 1 root root 4096 Apr 28 09:53 ..

[stdout]
total 8
drwxr-xr-x 2 root root 4096 Apr 28 09:53 .
drwxr-xr-x 1 root root 4096 Apr 28 09:53 ..

Create IAM users agent-runner and attacker-runner

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
aws iam create-user --user-name agent-runner --query 'User.Arn' --output text && \
aws iam create-user --user-name attacker-runner --query 'User.Arn' --output text
output
arn:aws:iam::000000000000:user/agent-runner
arn:aws:iam::000000000000:user/attacker-runner

[stdout]
arn:aws:iam::000000000000:user/agent-runner
arn:aws:iam::000000000000:user/attacker-runner
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks
Now I'll create the KMS customer-managed key with a strict policy.
write · /app/harbor/kms-key-policy-bootstrap.json
{
  "Version": "2012-10-17",
  "Id": "harbor-cross-account-cmk-bootstrap",
  "Statement": [
    {
      "Sid": "RootAccountAdmin",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::000000000000:root" },
      "Action": [
        "kms:Create*",
        "kms:Describe*",
        "kms:Enable*",
        "kms:List*",
        "kms:Put*",
        "kms:Update*",
        "kms:Revoke*",
        "kms:Disable*",
        "kms:Get*",
        "kms:Delete*",
        "kms:TagResource",
        "kms:UntagResource",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion"
      ],
      "Resource": "*"
    }
  ]
}

Create KMS CMK and alias

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
KEY_ID=$(aws kms create-key \
  --description "harbor cross-account CMK for ExternalId secrets" \
  --key-usage ENCRYPT_DECRYPT \
  --key-spec SYMMETRIC_DEFAULT \
  --policy file:///app/harbor/kms-key-policy-bootstrap.json \
  --query 'KeyMetadata.KeyId' --output text) && \
echo "KEY_ID=$KEY_ID" && \
KEY_ARN=$(aws kms describe-key --key-id "$KEY_ID" --query 'KeyMetadata.Arn' --output text) && \
echo "KEY_ARN=$KEY_ARN" && \
aws kms create-alias --alias-name alias/cross-account-cmk --target-key-id "$KEY_ID" && \
echo "$KEY_ID" > /app/harbor/.kms-key-id && \
echo "$KEY_ARN" > /app/harbor/.kms-key-arn && \
aws kms list-aliases --query 'Aliases[?AliasName==`alias/cross-account-cmk`]' --output json
output
KEY_ID=222eceb8-8c4d-42c3-9090-d3048224c2c9
KEY_ARN=arn:aws:kms:us-east-1:000000000000:key/222eceb8-8c4d-42c3-9090-d3048224c2c9
[
    {
        "AliasName": "alias/cross-account-cmk",
        "AliasArn": "arn:aws:kms:us-east-1:000000000000:alias/cross-account-cmk",
        "TargetKeyId": "222eceb8-8c4d-42c3-9090-d3048224c2c9",
        "CreationDate": "2026-04-28T09:54:43.161234+00:00"
    }
]

[stdout]
KEY_ID=222eceb8-8c4d-42c3-9090-d3048224c2c9
KEY_ARN=arn:aws:kms:us-east-1:000000000000:key/222eceb8-8c4d-42c3-9090-d3048224c2c9
[
    {
        "AliasName": "alias/cross-account-cmk",
        "AliasArn": "arn:aws:kms:us-east-1:000000000000:alias/cross-account-cmk",
        "TargetKeyId": "222eceb8-8c4d-42c3-9090-d3048224c2c9",
        "CreationDate": "2026-04-28T09:54:43.161234+00:00"
    }
]
Now I'll generate per-tenant ExternalIds and set up shell variables.

Generate per-tenant ExternalIds

shell
\
python3 -c "import secrets; print(secrets.token_urlsafe(48))" > .extid-tenant-a && \
python3 -c "import secrets; print(secrets.token_urlsafe(48))" > .extid-tenant-b && \
EXTID_A=$(cat .extid-tenant-a) && \
EXTID_B=$(cat .extid-tenant-b) && \
echo "len A: ${#EXTID_A}" && \
echo "len B: ${#EXTID_B}" && \
[ "$EXTID_A" != "$EXTID_B" ] && echo "distinct: ok"
output
len A: 64
len B: 64
distinct: ok

[stdout]
len A: 64
len B: 64
distinct: ok

Locate the generated extid files

shell
ls -la /app/.extid-tenant-* 2>/dev/null; ls -la /app/harbor/.extid-tenant-* 2>/dev/null; pwd
output
-rw-r--r-- 1 root root 65 Apr 28 09:54 /app/harbor/.extid-tenant-a
-rw-r--r-- 1 root root 65 Apr 28 09:54 /app/harbor/.extid-tenant-b
/app/harbor

[stdout]
-rw-r--r-- 1 root root 65 Apr 28 09:54 /app/harbor/.extid-tenant-a
-rw-r--r-- 1 root root 65 Apr 28 09:54 /app/harbor/.extid-tenant-b
/app/harbor
Now I'll create the OrchestratorRole, trusted by the agent-runner IAM user.
write · /app/harbor/orchestrator-trust.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AgentRunnerAssumesOrchestrator",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::000000000000:user/agent-runner"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create OrchestratorRole

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
ORCH_ARN=$(aws iam create-role \
  --role-name OrchestratorRole \
  --path /orchestrator/ \
  --assume-role-policy-document file:///app/harbor/orchestrator-trust.json \
  --description "harbor cross-account orchestrator (assumed by agent-runner user)" \
  --query 'Role.Arn' --output text) && \
echo "ORCH_ARN=$ORCH_ARN" && \
echo "$ORCH_ARN" > /app/harbor/.orch-arn
output
ORCH_ARN=arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole

[stdout]
ORCH_ARN=arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks

Create both ExternalId secrets encrypted with the CMK

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
KEY_ID=$(cat /app/harbor/.kms-key-id) && \
SECRET_A_ARN=$(aws secretsmanager create-secret \
  --name harbor/cross-account/external-id-tenant-a \
  --description "Per-tenant ExternalId shared secret for Tenant A AssumeRole challenge" \
  --kms-key-id "$KEY_ID" \
  --secret-string "$(cat /app/harbor/.extid-tenant-a)" \
  --query 'ARN' --output text) && \
SECRET_B_ARN=$(aws secretsmanager create-secret \
  --name harbor/cross-account/external-id-tenant-b \
  --description "Per-tenant ExternalId shared secret for Tenant B AssumeRole challenge" \
  --kms-key-id "$KEY_ID" \
  --secret-string "$(cat /app/harbor/.extid-tenant-b)" \
  --query 'ARN' --output text) && \
echo "SECRET_A_ARN=$SECRET_A_ARN" && \
echo "SECRET_B_ARN=$SECRET_B_ARN" && \
echo "$SECRET_A_ARN" > /app/harbor/.secret-a-arn && \
echo "$SECRET_B_ARN" > /app/harbor/.secret-b-arn
output
SECRET_A_ARN=arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl
SECRET_B_ARN=arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-AMbvIw

[stdout]
SECRET_A_ARN=arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl
SECRET_B_ARN=arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-AMbvIw

Generate member-role trust documents

shell
EXTID_A=$(cat /app/harbor/.extid-tenant-a) && \
EXTID_B=$(cat /app/harbor/.extid-tenant-b) && \
ORCH_ARN=$(cat /app/harbor/.orch-arn) && \
python3 - <<PY
import json, pathlib
extid_a = open("/app/harbor/.extid-tenant-a").read().strip()
extid_b = open("/app/harbor/.extid-tenant-b").read().strip()
orch_arn = open("/app/harbor/.orch-arn").read().strip()

def trust(extid, sid_suffix):
    return {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "OrchestratorAssumesMember" + sid_suffix,
          "Effect": "Allow",
          "Principal": { "AWS": orch_arn },
          "Action": "sts:AssumeRole",
          "Condition": {
            "StringEquals": {
              "sts:ExternalId": extid
            },
            "ArnEquals": {
              "aws:SourceArn": orch_arn
            }
          }
        }
      ]
    }

pathlib.Path("/app/harbor/member-trust-tenant-a.json").write_text(json.dumps(trust(extid_a, "TenantA"), indent=2))
pathlib.Path("/app/harbor/member-trust-tenant-b.json").write_text(json.dumps(trust(extid_b, "TenantB"), indent=2))
print("written")
PY
ls -la /app/harbor/member-trust-tenant-*.json
output
written
-rw-r--r-- 1 root root 563 Apr 28 09:55 /app/harbor/member-trust-tenant-a.json
-rw-r--r-- 1 root root 563 Apr 28 09:55 /app/harbor/member-trust-tenant-b.json

[stdout]
written
-rw-r--r-- 1 root root 563 Apr 28 09:55 /app/harbor/member-trust-tenant-a.json
-rw-r--r-- 1 root root 563 Apr 28 09:55 /app/harbor/member-trust-tenant-b.json

/app/harbor/member-trust-tenant-a.json

contents
1	{
2	  "Version": "2012-10-17",
3	  "Statement": [
4	    {
5	      "Sid": "OrchestratorAssumesMemberTenantA",
6	      "Effect": "Allow",
7	      "Principal": {
8	        "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
9	      },
10	      "Action": "sts:AssumeRole",
11	      "Condition": {
12	        "StringEquals": {
13	          "sts:ExternalId": "k9q7LeUjU_2QX4HSao1t5iqqBkygRfgdslSTmS7TLiuh8E69SuErNnH8XFBkPpzx"
14	        },
15	        "ArnEquals": {
16	          "aws:SourceArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
17	        }
18	      }
19	    }
20	  ]
21	}

Create both tenant member roles

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
MEM_A_ARN=$(aws iam create-role \
  --role-name MemberRole-Tenant-A \
  --path /member/ \
  --assume-role-policy-document file:///app/harbor/member-trust-tenant-a.json \
  --description "Tenant A member role; assumed only by OrchestratorRole with per-tenant ExternalId" \
  --query 'Role.Arn' --output text) && \
MEM_B_ARN=$(aws iam create-role \
  --role-name MemberRole-Tenant-B \
  --path /member/ \
  --assume-role-policy-document file:///app/harbor/member-trust-tenant-b.json \
  --description "Tenant B member role; assumed only by OrchestratorRole with per-tenant ExternalId" \
  --query 'Role.Arn' --output text) && \
echo "MEM_A_ARN=$MEM_A_ARN" && \
echo "MEM_B_ARN=$MEM_B_ARN" && \
echo "$MEM_A_ARN" > /app/harbor/.mem-a-arn && \
echo "$MEM_B_ARN" > /app/harbor/.mem-b-arn
output
MEM_A_ARN=arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
MEM_B_ARN=arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B

[stdout]
MEM_A_ARN=arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
MEM_B_ARN=arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks
Now I'll write the inline identity policies. Orchestrator gets explicit AssumeRole on the two member ARNs and explicit GetSecretValue on the two secret ARNs. Each member role gets s3 actions on its own bucket only.

Generate identity policy documents

shell
python3 - <<'PY'
import json, pathlib
mem_a = open("/app/harbor/.mem-a-arn").read().strip()
mem_b = open("/app/harbor/.mem-b-arn").read().strip()
sec_a = open("/app/harbor/.secret-a-arn").read().strip()
sec_b = open("/app/harbor/.secret-b-arn").read().strip()
key_arn = open("/app/harbor/.kms-key-arn").read().strip()

orch_policy = {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeMemberRolesExplicit",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": [ mem_a, mem_b ]
    },
    {
      "Sid": "ReadExternalIdSecretsExplicit",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": [ sec_a, sec_b ]
    },
    {
      "Sid": "DecryptCrossAccountCmkForSecrets",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": key_arn,
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

def member_policy(bucket):
    bucket_arn = f"arn:aws:s3:::{bucket}"
    return {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "ScopedBucketObjects",
          "Effect": "Allow",
          "Action": [
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject"
          ],
          "Resource": f"{bucket_arn}/*"
        },
        {
          "Sid": "ScopedBucketList",
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation"
          ],
          "Resource": bucket_arn
        }
      ]
    }

pathlib.Path("/app/harbor/orchestrator-inline.json").write_text(json.dumps(orch_policy, indent=2))
pathlib.Path("/app/harbor/member-inline-tenant-a.json").write_text(json.dumps(member_policy("tenant-a-bucket"), indent=2))
pathlib.Path("/app/harbor/member-inline-tenant-b.json").write_text(json.dumps(member_policy("tenant-b-bucket"), indent=2))
print("orchestrator + member identity policies written")
PY
ls -la /app/harbor/*-inline*.json
output
orchestrator + member identity policies written
-rw-r--r-- 1 root root  479 Apr 28 09:56 /app/harbor/member-inline-tenant-a.json
-rw-r--r-- 1 root root  479 Apr 28 09:56 /app/harbor/member-inline-tenant-b.json
-rw-r--r-- 1 root root 1064 Apr 28 09:56 /app/harbor/orchestrator-inline.json

[stdout]
orchestrator + member identity policies written
-rw-r--r-- 1 root root  479 Apr 28 09:56 /app/harbor/member-inline-tenant-a.json
-rw-r--r-- 1 root root  479 Apr 28 09:56 /app/harbor/member-inline-tenant-b.json
-rw-r--r-- 1 root root 1064 Apr 28 09:56 /app/harbor/orchestrator-inline.json

/app/harbor/orchestrator-inline.json

contents
1	{
2	  "Version": "2012-10-17",
3	  "Statement": [
4	    {
5	      "Sid": "AssumeMemberRolesExplicit",
6	      "Effect": "Allow",
7	      "Action": "sts:AssumeRole",
8	      "Resource": [
9	        "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A",
10	        "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B"
11	      ]
12	    },
13	    {
14	      "Sid": "ReadExternalIdSecretsExplicit",
15	      "Effect": "Allow",
16	      "Action": "secretsmanager:GetSecretValue",
17	      "Resource": [
18	        "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl",
19	        "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-AMbvIw"
20	      ]
21	    },
22	    {
23	      "Sid": "DecryptCrossAccountCmkForSecrets",
24	      "Effect": "Allow",
25	      "Action": "kms:Decrypt",
26	      "Resource": "arn:aws:kms:us-east-1:000000000000:key/222eceb8-8c4d-42c3-9090-d3048224c2c9",
27	      "Condition": {
28	        "StringEquals": {
29	          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
30	        }
31	      }
32	    }
33	  ]
34	}

/app/harbor/member-inline-tenant-a.json

contents
1	{
2	  "Version": "2012-10-17",
3	  "Statement": [
4	    {
5	      "Sid": "ScopedBucketObjects",
6	      "Effect": "Allow",
7	      "Action": [
8	        "s3:PutObject",
9	        "s3:GetObject",
10	        "s3:DeleteObject"
11	      ],
12	      "Resource": "arn:aws:s3:::tenant-a-bucket/*"
13	    },
14	    {
15	      "Sid": "ScopedBucketList",
16	      "Effect": "Allow",
17	      "Action": [
18	        "s3:ListBucket",
19	        "s3:GetBucketLocation"
20	      ],
21	      "Resource": "arn:aws:s3:::tenant-a-bucket"
22	    }
23	  ]
24	}

Attach inline identity policies

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
aws iam put-role-policy \
  --role-name OrchestratorRole \
  --policy-name OrchestratorAssumeAndSecretsAccess \
  --policy-document file:///app/harbor/orchestrator-inline.json && \
aws iam put-role-policy \
  --role-name MemberRole-Tenant-A \
  --policy-name TenantABucketAccess \
  --policy-document file:///app/harbor/member-inline-tenant-a.json && \
aws iam put-role-policy \
  --role-name MemberRole-Tenant-B \
  --policy-name TenantBBucketAccess \
  --policy-document file:///app/harbor/member-inline-tenant-b.json && \
echo "OK -- inline policies attached" && \
aws iam list-role-policies --role-name OrchestratorRole && \
aws iam list-role-policies --role-name MemberRole-Tenant-A && \
aws iam list-role-policies --role-name MemberRole-Tenant-B
output
OK -- inline policies attached
{
    "PolicyNames": [
        "OrchestratorAssumeAndSecretsAccess"
    ]
}
{
    "PolicyNames": [
        "TenantABucketAccess"
    ]
}
{
    "PolicyNames": [
        "TenantBBucketAccess"
    ]
}

[stdout]
OK -- inline policies attached
{
    "PolicyNames": [
        "OrchestratorAssumeAndSecretsAccess"
    ]
}
{
    "PolicyNames": [
        "TenantABucketAccess"
    ]
}
{
    "PolicyNames": [
        "TenantBBucketAccess"
    ]
}
I'll also attach a tight policy to the `agent-runner` user so the user-side AssumeRole grant is symmetric with the role's trust.
write · /app/harbor/agent-runner-inline.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeOrchestratorOnly",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
    }
  ]
}

Attach agent-runner inline policy

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
aws iam put-user-policy \
  --user-name agent-runner \
  --policy-name AssumeOrchestratorOnly \
  --policy-document file:///app/harbor/agent-runner-inline.json && \
aws iam list-user-policies --user-name agent-runner
output
{
    "PolicyNames": [
        "AssumeOrchestratorOnly"
    ]
}

[stdout]
{
    "PolicyNames": [
        "AssumeOrchestratorOnly"
    ]
}
Now I'll attach resource policies to the two ExternalId secrets so the secret itself names the orchestrator role as a permitted principal.

Generate Secrets Manager resource policy docs

shell
python3 - <<'PY'
import json, pathlib
orch_arn = open("/app/harbor/.orch-arn").read().strip()

def secret_resource_policy(secret_name):
    return {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "OrchestratorMayReadExternalId",
          "Effect": "Allow",
          "Principal": { "AWS": orch_arn },
          "Action": "secretsmanager:GetSecretValue",
          "Resource": "*"
        },
        {
          "Sid": "DenyEveryoneElse",
          "Effect": "Deny",
          "Principal": "*",
          "Action": "secretsmanager:GetSecretValue",
          "Resource": "*",
          "Condition": {
            "StringNotEquals": {
              "aws:PrincipalArn": orch_arn
            }
          }
        }
      ]
    }

pathlib.Path("/app/harbor/secret-resource-policy-a.json").write_text(json.dumps(secret_resource_policy("a"), indent=2))
pathlib.Path("/app/harbor/secret-resource-policy-b.json").write_text(json.dumps(secret_resource_policy("b"), indent=2))
print("written")
PY
ls -la /app/harbor/secret-resource-policy-*.json
output
written
-rw-r--r-- 1 root root 640 Apr 28 09:56 /app/harbor/secret-resource-policy-a.json
-rw-r--r-- 1 root root 640 Apr 28 09:56 /app/harbor/secret-resource-policy-b.json

[stdout]
written
-rw-r--r-- 1 root root 640 Apr 28 09:56 /app/harbor/secret-resource-policy-a.json
-rw-r--r-- 1 root root 640 Apr 28 09:56 /app/harbor/secret-resource-policy-b.json
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks

Attach resource policies to both secrets

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
SECRET_A_ARN=$(cat /app/harbor/.secret-a-arn) && \
SECRET_B_ARN=$(cat /app/harbor/.secret-b-arn) && \
aws secretsmanager put-resource-policy \
  --secret-id "$SECRET_A_ARN" \
  --resource-policy file:///app/harbor/secret-resource-policy-a.json \
  --block-public-policy && \
aws secretsmanager put-resource-policy \
  --secret-id "$SECRET_B_ARN" \
  --resource-policy file:///app/harbor/secret-resource-policy-b.json \
  --block-public-policy && \
echo "---- secret A resource policy ----" && \
aws secretsmanager get-resource-policy --secret-id "$SECRET_A_ARN" --query 'ResourcePolicy' --output text | python3 -m json.tool && \
echo "---- secret B resource policy ----" && \
aws secretsmanager get-resource-policy --secret-id "$SECRET_B_ARN" --query 'ResourcePolicy' --output text | python3 -m json.tool
output
{
    "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl",
    "Name": "harbor/cross-account/external-id-tenant-a"
}
{
    "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-AMbvIw",
    "Name": "harbor/cross-account/external-id-tenant-b"
}
---- secret A resource policy ----
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "OrchestratorMayReadExternalId",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
            },
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*"
        },
        {
            "Sid": "DenyEveryoneElse",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*",
            "Condition": {
                "StringNotEquals": {
                    "aws:PrincipalArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
                }
            }
        }
    ]
}
---- secret B resource policy ----
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "OrchestratorMayReadExternalId",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
            },
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*"
        },
        {
            "Sid": "DenyEveryoneElse",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*",
            "Condition": {
                "StringNotEquals": {
                    "aws:PrincipalArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
                }
            }
        }
    ]
}

[stdout]
{
    "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl",
    "Name": "harbor/cross-account/external-id-tenant-a"
}
{
    "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-AMbvIw",
    "Name": "harbor/cross-account/external-id-tenant-b"
}
---- secret A resource policy ----
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "OrchestratorMayReadExternalId",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
            },
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*"
        },
        {
            "Sid": "DenyEveryoneElse",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*",
            "Condition": {
                "StringNotEquals": {
                    "aws:PrincipalArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
                }
            }
        }
    ]
}
---- secret B resource policy ----
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "OrchestratorMayReadExternalId",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
            },
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*"
        },
        {
            "Sid": "DenyEveryoneElse",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*",
            "Condition": {
                "StringNotEquals": {
                    "aws:PrincipalArn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole"
                }
            }
        }
    ]
}
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks

Create tenant buckets and write SSM pointers

shell
export AWS_ENDPOINT_URL=http://localstack:4566 && \
aws s3api create-bucket --bucket tenant-a-bucket --region us-east-1 && \
aws s3api create-bucket --bucket tenant-b-bucket --region us-east-1 && \
ORCH_ARN=$(cat /app/harbor/.orch-arn) && \
SECRET_A_ARN=$(cat /app/harbor/.secret-a-arn) && \
SECRET_B_ARN=$(cat /app/harbor/.secret-b-arn) && \
MEM_A_ARN=$(cat /app/harbor/.mem-a-arn) && \
MEM_B_ARN=$(cat /app/harbor/.mem-b-arn) && \
aws ssm put-parameter --name /harbor/orchestrator/role-arn        --type String --value "$ORCH_ARN"     --overwrite && \
aws ssm put-parameter --name /harbor/external-id-secret-arn-tenant-a --type String --value "$SECRET_A_ARN" --overwrite && \
aws ssm put-parameter --name /harbor/external-id-secret-arn-tenant-b --type String --value "$SECRET_B_ARN" --overwrite && \
aws ssm put-parameter --name /harbor/member-role-arn-tenant-a     --type String --value "$MEM_A_ARN"    --overwrite && \
aws ssm put-parameter --name /harbor/member-role-arn-tenant-b     --type String --value "$MEM_B_ARN"    --overwrite && \
echo "---- ssm tree ----" && \
aws ssm get-parameters-by-path --path /harbor --recursive --query 'Parameters[*].[Name,Value]' --output table
output
{
    "Location": "/tenant-a-bucket"
}
{
    "Location": "/tenant-b-bucket"
}
{
    "Version": 1,
    "Tier": "Standard"
}
{
    "Version": 1,
    "Tier": "Standard"
}
{
    "Version": 1,
    "Tier": "Standard"
}
{
    "Version": 1,
    "Tier": "Standard"
}
{
    "Version": 1,
    "Tier": "Standard"
}
---- ssm tree ----
------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                 GetParametersByPath                                                                |
+------------------------------------------+---------------------------------------------------------------------------------------------------------+
|  /harbor/orchestrator/role-arn           |  arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole                                           |
|  /harbor/external-id-secret-arn-tenant-a |  arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl  |
|  /harbor/external-id-secret-arn-tenant-b |  arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-AMbvIw  |
|  /harbor/member-role-arn-tenant-a        |  arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A                                              |
|  /harbor/member-role-arn-tenant-b        |  arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B                                              |
+------------------------------------------+---------------------------------------------------------------------------------------------------------+

[stdout]
{
    "Location": "/tenant-a-bucket"
}
{
    "Location": "/tenant-b-bucket"
}
{
    "Version": 1,
    "Tier": "Standard"
}
{
    "Version": 1,
    "Tier": "Standard"
}
{
    "Version": 1,
    "Tier": "Standard"
}
{
    "Version": 1,
    "Tier": "Standard"
}
{
    "Version": 1,
    "Tier": "Standard"
}
---- ssm tree ----
------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                 GetParametersByPath                                                                |
+------------------------------------------+---------------------------------------------------------------------------------------------------------+
|  /harbor/orchestrator/role-arn           |  arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole                                           |
|  /harbor/external-id-secret-arn-tenant-a |  arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl  |
|  /harbor/external-id-secret-arn-tenant-b |  arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-AMbvIw  |
|  /harbor/member-role-arn-tenant-a        |  arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A                                              |
|  /harbor/member-role-arn-tenant-b        |  arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B                                              |
+------------------------------------------+---------------------------------------------------------------------------------------------------------+
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks
Now I'll write a happy-path script that walks through the full chain: agent-runner → assume Orchestrator → read tenant-A secret → assume MemberRole-Tenant-A with that ExternalId → put an object.
write · /app/harbor/happy_path.py
#!/usr/bin/env python3
"""Happy path: orchestrator -> Tenant A ExternalId -> MemberRole-Tenant-A -> S3 PutObject.

Walks each AWS principal change explicitly so each step is auditable on its own.
"""
import json
import os
import sys
import boto3

ENDPOINT = os.environ.get("AWS_ENDPOINT_URL", "http://localstack:4566")
REGION = os.environ.get("AWS_REGION", "us-east-1")


def client(service, **creds):
    return boto3.client(service, endpoint_url=ENDPOINT, region_name=REGION, **creds)


def section(title):
    print(f"\n=== {title} ===")


def main():
    section("0. Discover topology via SSM (no hardcoded ARNs)")
    ssm = client("ssm")
    params = {
        p["Name"]: p["Value"]
        for p in ssm.get_parameters_by_path(Path="/harbor", Recursive=True)["Parameters"]
    }
    orch_arn = params["/harbor/orchestrator/role-arn"]
    secret_a_arn = params["/harbor/external-id-secret-arn-tenant-a"]
    member_a_arn = params["/harbor/member-role-arn-tenant-a"]
    print(f"orchestrator role  = {orch_arn}")
    print(f"tenant-A secret    = {secret_a_arn}")
    print(f"tenant-A member    = {member_a_arn}")

    section("1. Create access keys for agent-runner (orchestrator's runtime identity)")
    iam_root = client("iam")
    keys = iam_root.create_access_key(UserName="agent-runner")["AccessKey"]
    runner_ak = keys["AccessKeyId"]
    runner_sk = keys["SecretAccessKey"]
    print(f"agent-runner access key id = {runner_ak}")

    section("2. agent-runner assumes OrchestratorRole")
    runner_sts = client(
        "sts",
        aws_access_key_id=runner_ak,
        aws_secret_access_key=runner_sk,
    )
    orch_creds = runner_sts.assume_role(
        RoleArn=orch_arn,
        RoleSessionName="orchestrator-runtime",
    )["Credentials"]
    print(f"orchestrator session OK; expires {orch_creds['Expiration']}")

    section("3. As OrchestratorRole, read tenant-A ExternalId from Secrets Manager")
    sm = client(
        "secretsmanager",
        aws_access_key_id=orch_creds["AccessKeyId"],
        aws_secret_access_key=orch_creds["SecretAccessKey"],
        aws_session_token=orch_creds["SessionToken"],
    )
    extid_a = sm.get_secret_value(SecretId=secret_a_arn)["SecretString"]
    print(f"tenant-A ExternalId len = {len(extid_a)} (value redacted)")

    section("4. As OrchestratorRole, assume MemberRole-Tenant-A with ExternalId")
    orch_sts = client(
        "sts",
        aws_access_key_id=orch_creds["AccessKeyId"],
        aws_secret_access_key=orch_creds["SecretAccessKey"],
        aws_session_token=orch_creds["SessionToken"],
    )
    mem_creds = orch_sts.assume_role(
        RoleArn=member_a_arn,
        RoleSessionName="orchestrator-as-tenant-a",
        ExternalId=extid_a,
    )["Credentials"]
    print(f"member session OK; expires {mem_creds['Expiration']}")

    section("5. As MemberRole-Tenant-A, PutObject in tenant-a-bucket")
    s3 = client(
        "s3",
        aws_access_key_id=mem_creds["AccessKeyId"],
        aws_secret_access_key=mem_creds["SecretAccessKey"],
        aws_session_token=mem_creds["SessionToken"],
    )
    s3.put_object(
        Bucket="tenant-a-bucket",
        Key="harbor/happy-path/probe.txt",
        Body=b"orchestrator wrote here through MemberRole-Tenant-A\n",
    )
    listed = s3.list_objects_v2(Bucket="tenant-a-bucket")
    keys_seen = [o["Key"] for o in listed.get("Contents", [])]
    print(f"objects in tenant-a-bucket: {keys_seen}")

    section("HAPPY PATH OK")
    print(json.dumps({
        "orchestrator": orch_arn,
        "member_role_used": member_a_arn,
        "external_id_source": secret_a_arn,
        "wrote_object": "s3://tenant-a-bucket/harbor/happy-path/probe.txt",
    }, indent=2))


if __name__ == "__main__":
    try:
        main()
    except Exception as e:
        print(f"FAIL: {e}", file=sys.stderr)
        sys.exit(1)

Run the happy-path script

shell
\
export AWS_ENDPOINT_URL=http://localstack:4566 AWS_REGION=us-east-1 && \
python3 happy_path.py
output
=== 0. Discover topology via SSM (no hardcoded ARNs) ===
orchestrator role  = arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
tenant-A secret    = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl
tenant-A member    = arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A

=== 1. Create access keys for agent-runner (orchestrator's runtime identity) ===
agent-runner access key id = LKIAQAAAAAAAL2KHLOL3

=== 2. agent-runner assumes OrchestratorRole ===
orchestrator session OK; expires 2026-04-28 10:57:54.865359+00:00

=== 3. As OrchestratorRole, read tenant-A ExternalId from Secrets Manager ===
tenant-A ExternalId len = 64 (value redacted)

=== 4. As OrchestratorRole, assume MemberRole-Tenant-A with ExternalId ===
member session OK; expires 2026-04-28 10:57:54.873454+00:00

=== 5. As MemberRole-Tenant-A, PutObject in tenant-a-bucket ===
objects in tenant-a-bucket: ['harbor/happy-path/probe.txt']

=== HAPPY PATH OK ===
{
  "orchestrator": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole",
  "member_role_used": "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A",
  "external_id_source": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl",
  "wrote_object": "s3://tenant-a-bucket/harbor/happy-path/probe.txt"
}

[stdout]
=== 0. Discover topology via SSM (no hardcoded ARNs) ===
orchestrator role  = arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
tenant-A secret    = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl
tenant-A member    = arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A

=== 1. Create access keys for agent-runner (orchestrator's runtime identity) ===
agent-runner access key id = LKIAQAAAAAAAL2KHLOL3

=== 2. agent-runner assumes OrchestratorRole ===
orchestrator session OK; expires 2026-04-28 10:57:54.865359+00:00

=== 3. As OrchestratorRole, read tenant-A ExternalId from Secrets Manager ===
tenant-A ExternalId len = 64 (value redacted)

=== 4. As OrchestratorRole, assume MemberRole-Tenant-A with ExternalId ===
member session OK; expires 2026-04-28 10:57:54.873454+00:00

=== 5. As MemberRole-Tenant-A, PutObject in tenant-a-bucket ===
objects in tenant-a-bucket: ['harbor/happy-path/probe.txt']

=== HAPPY PATH OK ===
{
  "orchestrator": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole",
  "member_role_used": "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A",
  "external_id_source": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-TGWiYl",
  "wrote_object": "s3://tenant-a-bucket/harbor/happy-path/probe.txt"
}
Happy path passes end-to-end. Now I'll build the verifier , the part the auditor will actually run.
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks
write · /app/harbor/verify.py
#!/usr/bin/env python3
"""Auditor-shaped verifier for the harbor cross-account trust model.

Reads the IAM/Secrets Manager state directly from LocalStack and asserts:

  Trust shape (per member role):
    - principal is OrchestratorRole ARN, NOT "*"
    - action is exactly "sts:AssumeRole"
    - StringEquals on sts:ExternalId is set and matches the per-tenant secret value
    - ArnEquals/ArnLike on aws:SourceArn equals the OrchestratorRole ARN
    - the two tenants' ExternalIds are different

  Least-privilege shape:
    - orchestrator's identity policy enumerates the two member ARNs explicitly
    - orchestrator's secret-get is scoped to the two known secret ARNs
    - each member role's inline policy references only its own bucket ARN
    - no policy in the system uses Action:"*" or Resource:"*" for s3 or kms

  Secret resource policy:
    - names OrchestratorRole as Principal.AWS for secretsmanager:GetSecretValue

A non-zero exit code on any failure.
"""
import json
import os
import sys
import boto3

ENDPOINT = os.environ.get("AWS_ENDPOINT_URL", "http://localstack:4566")
REGION = os.environ.get("AWS_REGION", "us-east-1")

ORCH_PATH = "/orchestrator/"
MEMBER_PATH = "/member/"
ORCH_NAME = "OrchestratorRole"
MEMBER_A = "MemberRole-Tenant-A"
MEMBER_B = "MemberRole-Tenant-B"
BUCKET_A = "tenant-a-bucket"
BUCKET_B = "tenant-b-bucket"


def aws(service):
    return boto3.client(service, endpoint_url=ENDPOINT, region_name=REGION)


def as_list(x):
    if x is None:
        return []
    return x if isinstance(x, list) else [x]


class Report:
    def __init__(self):
        self.passes = []
        self.fails = []

    def check(self, name, ok, detail=""):
        if ok:
            self.passes.append(name)
            print(f"  PASS  {name}")
        else:
            self.fails.append((name, detail))
            print(f"  FAIL  {name}\n        {detail}")

    def section(self, title):
        print(f"\n[{title}]")

    def done(self):
        print(f"\n----\n{len(self.passes)} passed, {len(self.fails)} failed")
        for n, d in self.fails:
            print(f"  - {n}: {d}")
        return 0 if not self.fails else 1


def get_role(iam, name, path):
    r = iam.get_role(RoleName=name)["Role"]
    if r.get("Path") != path:
        raise SystemExit(f"role {name} path={r.get('Path')!r}, expected {path!r}")
    return r


def get_inline(iam, name):
    out = {}
    for pn in iam.list_role_policies(RoleName=name)["PolicyNames"]:
        out[pn] = iam.get_role_policy(RoleName=name, PolicyName=pn)["PolicyDocument"]
    return out


def secret_value(sm, secret_arn):
    return sm.get_secret_value(SecretId=secret_arn)["SecretString"]


def secret_resource_policy(sm, secret_arn):
    raw = sm.get_resource_policy(SecretId=secret_arn).get("ResourcePolicy")
    return json.loads(raw) if raw else None


def has_starwild_resource_or_action(doc, services):
    """Return list of (sid, reason) for statements using Action:* or Resource:* on
    the given service namespaces.
    """
    findings = []
    for stmt in as_list(doc.get("Statement")):
        sid = stmt.get("Sid", "<no-sid>")
        actions = as_list(stmt.get("Action"))
        resources = as_list(stmt.get("Resource"))
        touches = any(
            a == "*"
            or any(a.startswith(svc + ":") for svc in services)
            or (resources and any(svc + ":" in r for svc in services) for r in [])
            for a in actions
        )
        for a in actions:
            if a == "*":
                findings.append((sid, "Action:'*'"))
            elif any(a.startswith(svc + ":") for svc in services):
                if "*" in resources:
                    findings.append((sid, f"Resource:'*' for service {a}"))
        for r in resources:
            if isinstance(r, str) and r != "*" and any(s + ":" in r for s in services):
                pass
    return findings


def main():
    iam = aws("iam")
    sm = aws("secretsmanager")
    ssm = aws("ssm")

    rep = Report()

    rep.section("0. Topology discovery via SSM")
    params = {
        p["Name"]: p["Value"]
        for p in ssm.get_parameters_by_path(Path="/harbor", Recursive=True)["Parameters"]
    }
    needed = [
        "/harbor/orchestrator/role-arn",
        "/harbor/external-id-secret-arn-tenant-a",
        "/harbor/external-id-secret-arn-tenant-b",
        "/harbor/member-role-arn-tenant-a",
        "/harbor/member-role-arn-tenant-b",
    ]
    for n in needed:
        rep.check(f"ssm has {n}", n in params, detail=f"missing")
    orch_arn = params["/harbor/orchestrator/role-arn"]
    secret_a = params["/harbor/external-id-secret-arn-tenant-a"]
    secret_b = params["/harbor/external-id-secret-arn-tenant-b"]
    member_a_arn = params["/harbor/member-role-arn-tenant-a"]
    member_b_arn = params["/harbor/member-role-arn-tenant-b"]

    rep.section("1. Roles exist on the expected paths")
    orch = get_role(iam, ORCH_NAME, ORCH_PATH)
    rep.check("OrchestratorRole exists under /orchestrator/", orch["Arn"] == orch_arn,
              detail=f"{orch['Arn']} vs {orch_arn}")
    mem_a = get_role(iam, MEMBER_A, MEMBER_PATH)
    mem_b = get_role(iam, MEMBER_B, MEMBER_PATH)
    rep.check("MemberRole-Tenant-A exists under /member/", mem_a["Arn"] == member_a_arn)
    rep.check("MemberRole-Tenant-B exists under /member/", mem_b["Arn"] == member_b_arn)

    rep.section("2. Member-role trust policies , anti-impersonation + anti-confused-deputy")
    extid_a = secret_value(sm, secret_a)
    extid_b = secret_value(sm, secret_b)
    rep.check("ExternalId tenant-a length >= 32", len(extid_a) >= 32, detail=f"len={len(extid_a)}")
    rep.check("ExternalId tenant-b length >= 32", len(extid_b) >= 32, detail=f"len={len(extid_b)}")
    rep.check("ExternalIds differ between tenants", extid_a != extid_b)

    for label, role, expected_extid in [
        ("Tenant-A", mem_a, extid_a),
        ("Tenant-B", mem_b, extid_b),
    ]:
        trust = role["AssumeRolePolicyDocument"]
        stmts = as_list(trust.get("Statement"))
        rep.check(f"{label} trust has exactly 1 statement", len(stmts) == 1,
                  detail=f"got {len(stmts)}")
        s = stmts[0] if stmts else {}
        principal = s.get("Principal", {})
        aws_principal = principal.get("AWS") if isinstance(principal, dict) else principal
        aws_principal_list = as_list(aws_principal)
        rep.check(
            f"{label} trust Principal.AWS == OrchestratorRole ARN (NOT '*')",
            aws_principal_list == [orch_arn],
            detail=f"got {aws_principal!r}",
        )
        rep.check(
            f"{label} trust Principal does NOT include '*'",
            "*" not in aws_principal_list and principal != "*",
            detail=f"got {principal!r}",
        )
        actions = as_list(s.get("Action"))
        rep.check(
            f"{label} trust Action is exactly ['sts:AssumeRole']",
            actions == ["sts:AssumeRole"],
            detail=f"got {actions!r}",
        )
        rep.check(f"{label} trust Effect == Allow", s.get("Effect") == "Allow")
        cond = s.get("Condition") or {}
        ext_eq = (cond.get("StringEquals") or {}).get("sts:ExternalId")
        rep.check(
            f"{label} trust StringEquals.sts:ExternalId matches the secret value",
            ext_eq == expected_extid,
            detail=f"trust value differs from secret value (or missing)",
        )
        src_eq = (
            (cond.get("ArnEquals") or {}).get("aws:SourceArn")
            or (cond.get("ArnLike") or {}).get("aws:SourceArn")
        )
        rep.check(
            f"{label} trust Arn(Like|Equals).aws:SourceArn == OrchestratorRole ARN",
            src_eq == orch_arn,
            detail=f"got {src_eq!r}",
        )

    rep.section("3. Orchestrator identity policy , explicit, no wildcards")
    orch_inline = get_inline(iam, ORCH_NAME)
    rep.check("OrchestratorRole has at least one inline policy", bool(orch_inline))
    all_actions_resources = []
    assume_resources = set()
    secret_get_resources = set()
    for pname, pdoc in orch_inline.items():
        for s in as_list(pdoc.get("Statement")):
            actions = as_list(s.get("Action"))
            resources = as_list(s.get("Resource"))
            all_actions_resources.append((pname, s.get("Sid"), actions, resources))
            if "sts:AssumeRole" in actions:
                assume_resources.update(resources)
            if "secretsmanager:GetSecretValue" in actions:
                secret_get_resources.update(resources)
    rep.check(
        "Orchestrator AssumeRole resources == both member ARNs (set equality)",
        assume_resources == {member_a_arn, member_b_arn},
        detail=f"got {assume_resources}",
    )
    rep.check(
        "Orchestrator AssumeRole has NO wildcard resource",
        "*" not in assume_resources,
        detail=f"got {assume_resources}",
    )
    rep.check(
        "Orchestrator GetSecretValue resources == both secret ARNs",
        secret_get_resources == {secret_a, secret_b},
        detail=f"got {secret_get_resources}",
    )
    rep.check(
        "Orchestrator GetSecretValue has NO wildcard resource",
        "*" not in secret_get_resources,
        detail=f"got {secret_get_resources}",
    )

    rep.section("4. Member identity policies , only own bucket")
    for label, role_name, own_bucket, other_bucket in [
        ("Tenant-A", MEMBER_A, BUCKET_A, BUCKET_B),
        ("Tenant-B", MEMBER_B, BUCKET_B, BUCKET_A),
    ]:
        inlines = get_inline(iam, role_name)
        rep.check(f"{label} member has at least one inline policy", bool(inlines))
        own_arn = f"arn:aws:s3:::{own_bucket}"
        own_arn_objs = f"{own_arn}/*"
        other_arn = f"arn:aws:s3:::{other_bucket}"
        seen_resources = set()
        wildcards = []
        cross_tenant = []
        for pname, pdoc in inlines.items():
            for s in as_list(pdoc.get("Statement")):
                resources = as_list(s.get("Resource"))
                actions = as_list(s.get("Action"))
                for r in resources:
                    seen_resources.add(r)
                    if r == "*":
                        wildcards.append((pname, s.get("Sid"), "Resource:'*'"))
                    if r.startswith(other_arn):
                        cross_tenant.append((pname, s.get("Sid"), r))
                for a in actions:
                    if a == "*":
                        wildcards.append((pname, s.get("Sid"), "Action:'*'"))
        allowed = {own_arn, own_arn_objs}
        rep.check(
            f"{label} member resources are subset of own-bucket only",
            seen_resources.issubset(allowed),
            detail=f"unexpected resources: {seen_resources - allowed}",
        )
        rep.check(f"{label} member has no Resource/Action '*'", not wildcards,
                  detail=f"{wildcards}")
        rep.check(f"{label} member has no reference to the other tenant's bucket",
                  not cross_tenant, detail=f"{cross_tenant}")

    rep.section("5. No Action:'*' / Resource:'*' on s3 or kms anywhere")
    all_role_inlines = {
        ORCH_NAME: orch_inline,
        MEMBER_A: get_inline(iam, MEMBER_A),
        MEMBER_B: get_inline(iam, MEMBER_B),
    }
    s3_kms_violations = []
    for role_name, policies in all_role_inlines.items():
        for pname, pdoc in policies.items():
            for s in as_list(pdoc.get("Statement")):
                actions = as_list(s.get("Action"))
                resources = as_list(s.get("Resource"))
                if "*" in actions:
                    s3_kms_violations.append((role_name, pname, s.get("Sid"), "Action:'*'"))
                touches_s3_kms = any(
                    a.startswith("s3:") or a.startswith("kms:") or a == "*" for a in actions
                )
                if touches_s3_kms and "*" in resources:
                    s3_kms_violations.append((role_name, pname, s.get("Sid"),
                                              "Resource:'*' on s3/kms"))
    rep.check("No Action:'*' or Resource:'*' for s3/kms in any inline policy",
              not s3_kms_violations, detail=f"{s3_kms_violations}")

    rep.section("6. Secret resource policies admit OrchestratorRole")
    for label, secret_arn in [("tenant-A", secret_a), ("tenant-B", secret_b)]:
        rp = secret_resource_policy(sm, secret_arn)
        rep.check(f"{label} secret has a resource policy", rp is not None)
        if rp is None:
            continue
        admits_orch = False
        for s in as_list(rp.get("Statement")):
            if s.get("Effect") != "Allow":
                continue
            principals = s.get("Principal", {})
            aws_p = principals.get("AWS") if isinstance(principals, dict) else None
            actions = as_list(s.get("Action"))
            if (orch_arn in as_list(aws_p)) and ("secretsmanager:GetSecretValue" in actions):
                admits_orch = True
                break
        rep.check(
            f"{label} secret resource policy names OrchestratorRole as Principal.AWS for GetSecretValue",
            admits_orch,
        )

    rep.section("7. Sanity: agent-runner identity policy and orchestrator trust")
    agent_inlines = {
        pn: iam.get_user_policy(UserName="agent-runner", PolicyName=pn)["PolicyDocument"]
        for pn in iam.list_user_policies(UserName="agent-runner")["PolicyNames"]
    }
    agent_assume_resources = set()
    for pdoc in agent_inlines.values():
        for s in as_list(pdoc.get("Statement")):
            if "sts:AssumeRole" in as_list(s.get("Action")):
                agent_assume_resources.update(as_list(s.get("Resource")))
    rep.check(
        "agent-runner can AssumeRole only on OrchestratorRole",
        agent_assume_resources == {orch_arn},
        detail=f"got {agent_assume_resources}",
    )
    orch_trust = orch["AssumeRolePolicyDocument"]
    orch_trust_principals = set()
    for s in as_list(orch_trust.get("Statement")):
        p = s.get("Principal", {})
        orch_trust_principals.update(as_list(p.get("AWS") if isinstance(p, dict) else p))
    rep.check(
        "OrchestratorRole trust principal is exactly the agent-runner user",
        orch_trust_principals == {"arn:aws:iam::000000000000:user/agent-runner"},
        detail=f"got {orch_trust_principals}",
    )

    code = rep.done()
    sys.exit(code)


if __name__ == "__main__":
    main()

Run the verifier

shell
export AWS_ENDPOINT_URL=http://localstack:4566 AWS_REGION=us-east-1 && \
python3 verify.py
output
[0. Topology discovery via SSM]
  PASS  ssm has /harbor/orchestrator/role-arn
  PASS  ssm has /harbor/external-id-secret-arn-tenant-a
  PASS  ssm has /harbor/external-id-secret-arn-tenant-b
  PASS  ssm has /harbor/member-role-arn-tenant-a
  PASS  ssm has /harbor/member-role-arn-tenant-b

[1. Roles exist on the expected paths]
  PASS  OrchestratorRole exists under /orchestrator/
  PASS  MemberRole-Tenant-A exists under /member/
  PASS  MemberRole-Tenant-B exists under /member/

[2. Member-role trust policies , anti-impersonation + anti-confused-deputy]
  PASS  ExternalId tenant-a length >= 32
  PASS  ExternalId tenant-b length >= 32
  PASS  ExternalIds differ between tenants
  PASS  Tenant-A trust has exactly 1 statement
  PASS  Tenant-A trust Principal.AWS == OrchestratorRole ARN (NOT '*')
  PASS  Tenant-A trust Principal does NOT include '*'
  PASS  Tenant-A trust Action is exactly ['sts:AssumeRole']
  PASS  Tenant-A trust Effect == Allow
  PASS  Tenant-A trust StringEquals.sts:ExternalId matches the secret value
  PASS  Tenant-A trust Arn(Like|Equals).aws:SourceArn == OrchestratorRole ARN
  PASS  Tenant-B trust has exactly 1 statement
  PASS  Tenant-B trust Principal.AWS == OrchestratorRole ARN (NOT '*')
  PASS  Tenant-B trust Principal does NOT include '*'
  PASS  Tenant-B trust Action is exactly ['sts:AssumeRole']
  PASS  Tenant-B trust Effect == Allow
  PASS  Tenant-B trust StringEquals.sts:ExternalId matches the secret value
  PASS  Tenant-B trust Arn(Like|Equals).aws:SourceArn == OrchestratorRole ARN

[3. Orchestrator identity policy , explicit, no wildcards]
  PASS  OrchestratorRole has at least one inline policy
  PASS  Orchestrator AssumeRole resources == both member ARNs (set equality)
  PASS  Orchestrator AssumeRole has NO wildcard resource
  PASS  Orchestrator GetSecretValue resources == both secret ARNs
  PASS  Orchestrator GetSecretValue has NO wildcard resource

[4. Member identity policies , only own bucket]
  PASS  Tenant-A member has at least one inline policy
  PASS  Tenant-A member resources are subset of own-bucket only
  PASS  Tenant-A member has no Resource/Action '*'
  PASS  Tenant-A member has no reference to the other tenant's bucket
  PASS  Tenant-B member has at least one inline policy
  PASS  Tenant-B member resources are subset of own-bucket only
  PASS  Tenant-B member has no Resource/Action '*'
  PASS  Tenant-B member has no reference to the other tenant's bucket

[5. No Action:'*' / Resource:'*' on s3 or kms anywhere]
  PASS  No Action:'*' or Resource:'*' for s3/kms in any inline policy

[6. Secret resource policies admit OrchestratorRole]
  PASS  tenant-A secret has a resource policy
  PASS  tenant-A secret resource policy names OrchestratorRole as Principal.AWS for GetSecretValue
  PASS  tenant-B secret has a resource policy
  PASS  tenant-B secret resource policy names OrchestratorRole as Principal.AWS for GetSecretValue

[7. Sanity: agent-runner identity policy and orchestrator trust]
  PASS  agent-runner can AssumeRole only on OrchestratorRole
  PASS  OrchestratorRole trust principal is exactly the agent-runner user

----
45 passed, 0 failed

[stdout]
[0. Topology discovery via SSM]
  PASS  ssm has /harbor/orchestrator/role-arn
  PASS  ssm has /harbor/external-id-secret-arn-tenant-a
  PASS  ssm has /harbor/external-id-secret-arn-tenant-b
  PASS  ssm has /harbor/member-role-arn-tenant-a
  PASS  ssm has /harbor/member-role-arn-tenant-b

[1. Roles exist on the expected paths]
  PASS  OrchestratorRole exists under /orchestrator/
  PASS  MemberRole-Tenant-A exists under /member/
  PASS  MemberRole-Tenant-B exists under /member/

[2. Member-role trust policies , anti-impersonation + anti-confused-deputy]
  PASS  ExternalId tenant-a length >= 32
  PASS  ExternalId tenant-b length >= 32
  PASS  ExternalIds differ between tenants
  PASS  Tenant-A trust has exactly 1 statement
  PASS  Tenant-A trust Principal.AWS == OrchestratorRole ARN (NOT '*')
  PASS  Tenant-A trust Principal does NOT include '*'
  PASS  Tenant-A trust Action is exactly ['sts:AssumeRole']
  PASS  Tenant-A trust Effect == Allow
  PASS  Tenant-A trust StringEquals.sts:ExternalId matches the secret value
  PASS  Tenant-A trust Arn(Like|Equals).aws:SourceArn == OrchestratorRole ARN
  PASS  Tenant-B trust has exactly 1 statement
  PASS  Tenant-B trust Principal.AWS == OrchestratorRole ARN (NOT '*')
  PASS  Tenant-B trust Principal does NOT include '*'
  PASS  Tenant-B trust Action is exactly ['sts:AssumeRole']
  PASS  Tenant-B trust Effect == Allow
  PASS  Tenant-B trust StringEquals.sts:ExternalId matches the secret value
  PASS  Tenant-B trust Arn(Like|Equals).aws:SourceArn == OrchestratorRole ARN

[3. Orchestrator identity policy , explicit, no wildcards]
  PASS  OrchestratorRole has at least one inline policy
  PASS  Orchestrator AssumeRole resources == both member ARNs (set equality)
  PASS  Orchestrator AssumeRole has NO wildcard resource
  PASS  Orchestrator GetSecretValue resources == both secret ARNs
  PASS  Orchestrator GetSecretValue has NO wildcard resource

[4. Member identity policies , only own bucket]
  PASS  Tenant-A member has at least one inline policy
  PASS  Tenant-A member resources are subset of own-bucket only
  PASS  Tenant-A member has no Resource/Action '*'
  PASS  Tenant-A member has no reference to the other tenant's bucket
  PASS  Tenant-B member has at least one inline policy
  PASS  Tenant-B member resources are subset of own-bucket only
  PASS  Tenant-B member has no Resource/Action '*'
  PASS  Tenant-B member has no reference to the other tenant's bucket

[5. No Action:'*' / Resource:'*' on s3 or kms anywhere]
  PASS  No Action:'*' or Resource:'*' for s3/kms in any inline policy

[6. Secret resource policies admit OrchestratorRole]
  PASS  tenant-A secret has a resource policy
  PASS  tenant-A secret resource policy names OrchestratorRole as Principal.AWS for GetSecretValue
  PASS  tenant-B secret has a resource policy
  PASS  tenant-B secret resource policy names OrchestratorRole as Principal.AWS for GetSecretValue

[7. Sanity: agent-runner identity policy and orchestrator trust]
  PASS  agent-runner can AssumeRole only on OrchestratorRole
  PASS  OrchestratorRole trust principal is exactly the agent-runner user

----
45 passed, 0 failed
  • Verify localstack connectivity and capture account ID
  • Create IAM users (agent-runner, attacker-runner)
  • Create KMS CMK with alias/cross-account-cmk
  • Generate per-tenant ExternalIds and create Secrets Manager secrets
  • Create OrchestratorRole trusted by agent-runner
  • Create MemberRole-Tenant-A and MemberRole-Tenant-B with strict trust
  • Attach inline identity policies (orchestrator, members)
  • Attach resource policies on the two ExternalId secrets
  • Create tenant buckets and SSM pointers
  • Run happy-path: orchestrator -> read secret -> assume Member-A -> put object
  • Build verifier script asserting all shape/least-privilege checks
All 45 verifier checks pass and the happy path writes through every principal change. **What got built (`/app/harbor/`)** - `agent-runner` and `attacker-runner` IAM users - `OrchestratorRole` (path `/orchestrator/`), trust = `agent-runner` user only; identity policy is explicit-resource only , `sts:AssumeRole` on the two member ARNs, `secretsmanager:GetSecretValue` on the two secret ARNs, and `kms:Decrypt` on the CMK gated by `kms:ViaService=secretsmanager.us-east-1.amazonaws.com` - `MemberRole-Tenant-A` / `MemberRole-Tenant-B` (path `/member/`), each with the auditor-required trust: - `Principal.AWS = OrchestratorRole ARN` (no `*`) - `Action = "sts:AssumeRole"` (single string, no wildcards) - `StringEquals.sts:ExternalId` = that tenant's 64-char secret value (different per tenant) - `ArnEquals.aws:SourceArn` = OrchestratorRole ARN - KMS CMK `alias/cross-account-cmk` encrypts both ExternalId secrets - Secrets Manager: `harbor/cross-account/external-id-tenant-{a,b}` , each carries a resource policy that names `OrchestratorRole` as `Principal.AWS` for `GetSecretValue` and an explicit Deny for everyone else - Member identity policies scoped to their own bucket ARN only (no `*`, no cross-tenant reference) - SSM pointers under `/harbor/...` for orchestrator ARN, both secret ARNs, both member ARNs **Files**: `happy_path.py` (end-to-end runtime walk), `verify.py` (shape audit, exit 0/1), plus the trust/identity/resource policy JSONs and the dotfile ARN cache. **Run them again** ``` export AWS_ENDPOINT_URL=http://localstack:4566 AWS_REGION=us-east-1 python3 /app/harbor/verify.py # 45/45 PASS python3 /app/harbor/happy_path.py # writes s3://tenant-a-bucket/harbor/happy-path/probe.txt ``` Caveat the spec already called out: LocalStack doesn't enforce IAM denies at runtime, so the negative tests (impersonator without ExternalId, confused-deputy without SourceArn) would still succeed at the API layer here. The verifier exercises the same checks an auditor would , it reads the trust documents and asserts the conditions are present and correctly bound , which is the meaningful evidence for the SOC2 carryover.
Final diff , what the agent changed

No file changes, the agent left the workspace untouched.

Verifier output , the proof behind the reward
tests/test.sh
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/A3p9YClXugtNfuNulZWWv/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 20 items

../tests/test_state.py::test_01_orchestrator_role_exists_at_orchestrator_path PASSED [  5%]
../tests/test_state.py::test_02_member_roles_exist_at_member_path PASSED [ 10%]
../tests/test_state.py::test_03_external_id_secrets_exist_with_min_length PASSED [ 15%]
../tests/test_state.py::test_04_both_tenant_buckets_exist PASSED         [ 20%]
../tests/test_state.py::test_05_ssm_pointers_resolve PASSED              [ 25%]
../tests/test_state.py::test_06_member_trusts_have_externalid_stringequals_matching_secret PASSED [ 30%]
../tests/test_state.py::test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn PASSED [ 35%]
../tests/test_state.py::test_08_member_trusts_action_is_assumerole_only PASSED [ 40%]
../tests/test_state.py::test_09_member_trusts_principal_is_orchestrator_not_wildcard PASSED [ 45%]
../tests/test_state.py::test_10_externalids_differ_per_tenant PASSED     [ 50%]
../tests/test_state.py::test_11_orchestrator_identity_lists_member_arns_explicitly_no_wildcard PASSED [ 55%]
../tests/test_state.py::test_12_orchestrator_secret_get_scoped_to_two_secret_arns PASSED [ 60%]
../tests/test_state.py::test_13_orchestrator_kms_decrypt_scoped_to_cmk_arn PASSED [ 65%]
../tests/test_state.py::test_14_member_inline_policies_scope_to_own_tenant_bucket_only PASSED [ 70%]
../tests/test_state.py::test_15_secret_resource_policies_admit_orchestrator PASSED [ 75%]
../tests/test_state.py::test_16_kms_key_policy_has_account_root_statement FAILED [ 80%]
../tests/test_state.py::test_17_no_admin_managed_policies_attached_to_any_role PASSED [ 85%]
../tests/test_state.py::test_18_e2e_assume_with_externalid_then_put_object_succeeds PASSED [ 90%]
../tests/test_state.py::test_19_member_a_inline_does_not_reference_tenant_b_bucket PASSED [ 95%]
../tests/test_state.py::test_20_orchestrator_has_no_wildcard_assume_role_resource PASSED [100%]

=================================== FAILURES ===================================
______________ test_16_kms_key_policy_has_account_root_statement _______________

    def test_16_kms_key_policy_has_account_root_statement():
        """KMS key policy has root statement."""
        cmk_arn = _kms_key_arn()
        kms = _client("kms")
        pol = kms.get_key_policy(KeyId=cmk_arn, PolicyName="default")["Policy"]
        doc = json.loads(pol)
        found_root = False
        for s in _stmts(doc):
            p = s.get("Principal", {})
            aws = p.get("AWS") if isinstance(p, dict) else None
            arns = _as_list(aws)
            if any(a.endswith(":root") for a in arns):
                actions = _as_list(s.get("Action"))
                if actions == ["kms:*"] or "kms:*" in actions:
                    found_root = True
>       assert found_root, (
            "KMS key policy missing the 'EnableIAMUserPermissions' root statement"
        )
E       AssertionError: KMS key policy missing the 'EnableIAMUserPermissions' root statement
E       assert False

/tests/test_state.py:388: AssertionError
=============================== warnings summary ===============================
test_state.py: 56 warnings
  /root/.cache/uv/archive-v0/A3p9YClXugtNfuNulZWWv/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_state.py::test_01_orchestrator_role_exists_at_orchestrator_path
PASSED ../tests/test_state.py::test_02_member_roles_exist_at_member_path
PASSED ../tests/test_state.py::test_03_external_id_secrets_exist_with_min_length
PASSED ../tests/test_state.py::test_04_both_tenant_buckets_exist
PASSED ../tests/test_state.py::test_05_ssm_pointers_resolve
PASSED ../tests/test_state.py::test_06_member_trusts_have_externalid_stringequals_matching_secret
PASSED ../tests/test_state.py::test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn
PASSED ../tests/test_state.py::test_08_member_trusts_action_is_assumerole_only
PASSED ../tests/test_state.py::test_09_member_trusts_principal_is_orchestrator_not_wildcard
PASSED ../tests/test_state.py::test_10_externalids_differ_per_tenant
PASSED ../tests/test_state.py::test_11_orchestrator_identity_lists_member_arns_explicitly_no_wildcard
PASSED ../tests/test_state.py::test_12_orchestrator_secret_get_scoped_to_two_secret_arns
PASSED ../tests/test_state.py::test_13_orchestrator_kms_decrypt_scoped_to_cmk_arn
PASSED ../tests/test_state.py::test_14_member_inline_policies_scope_to_own_tenant_bucket_only
PASSED ../tests/test_state.py::test_15_secret_resource_policies_admit_orchestrator
PASSED ../tests/test_state.py::test_17_no_admin_managed_policies_attached_to_any_role
PASSED ../tests/test_state.py::test_18_e2e_assume_with_externalid_then_put_object_succeeds
PASSED ../tests/test_state.py::test_19_member_a_inline_does_not_reference_tenant_b_bucket
PASSED ../tests/test_state.py::test_20_orchestrator_has_no_wildcard_assume_role_resource
FAILED ../tests/test_state.py::test_16_kms_key_policy_has_account_root_statement
================== 1 failed, 19 passed, 56 warnings in 0.51s ===================

Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_bbb4bbf9a94d4998. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.

Trial trial_bbb4bbf9a94d4998 · verifier authoritative; classifier explanatory.