SyncValsverifier → artifact → classifier → verdict
SyncVals · Trajectory

iam-cross-account-externalid-sourcearn

claude-code claude-opus-4-7 ✗ failed HARNESS_ERROR ↑ View task
Solved from the instruction alone, tests/ and solution/ were withheld from the agent's workspace and restored only for grading.
Reward = tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.
Classification , post-hoc; cannot change the reward
HARNESS_ERRORInfrastructure failure, the agent never ran properly. Not a signal about agent or task.
SubtypeClassification Failed
EvidenceClaude Code classification failed: (classifier unavailable for this trial)
Root causeCould not analyze trial with Claude Code
RecommendationReview trial manually or check authentication
Trajectory
Tool-by-tool agent trajectory
11 tool calls · 2 tool types · 16 steps
our security review came back ugly. an external auditor asked how we keep tenant A's orchestrator from poking at tenant B's bucket, and how we'd block a contractor from impersonating our orchestrator if they ever get hold of one of our role names. honest answer right now: we don't. the cross-account assume-role chain we have is "trust the orchestrator role arn" and that's it. they want this fixed by next sprint or we lose the SOC2 carryover. so: build the cross-account trust the way it's supposed to be done. one orchestrator role that fans out to two tenant member roles. each member role's trust must defend against TWO different attacks at the same time: 1. **third-party impersonation** , someone outside the company learns one of our member role arns and tries to assume it. fix: every assume-role call must present a per-tenant `ExternalId` shared secret. without it, denied. 2. **AWS-service confused-deputy** , some other principal in our account (or a service we've granted broadly) gets tricked into calling AssumeRole on our behalf. fix: the trust must require `aws:SourceArn` to equal the orchestrator role's arn. without it, denied. both conditions on every member role trust. not one or the other. both. shape of it: - localstack at `http://localstack:4566`. creds already exported (`AWS_ACCESS_KEY_ID=test`, same for secret, region `us-east-1`). `aws`, `python3`, `boto3`, `jq`. build from zero. - one **orchestrator role** trusted by an iam user that represents the orchestrator's runtime identity. - two **member roles**, one per tenant. each member role's trust policy: - principal = orchestrator role arn (specific, not `*`) - condition `StringEquals` on `sts:ExternalId` matching that tenant's secret value - condition on `aws:SourceArn` equal to the orchestrator role arn - action = `sts:AssumeRole` only , no wildcards - per-tenant **ExternalId** values stored in secrets manager, encrypted with a customer-managed kms key. the orchestrator reads the secret, doesn't hardcode it. and the two tenants get DIFFERENT ExternalIds , reusing one across tenants defeats the point. - each member role's identity policy is scoped to ONLY that tenant's bucket. tenant A's role can put/get on `tenant-a-bucket` and nothing else. no `Resource: "*"`. - the orchestrator's identity policy lists the two member role arns explicitly under `sts:AssumeRole` , no `Resource: "*"` there either. - secret access scoped: orchestrator can `secretsmanager:GetSecretValue` on the two ExternalId secret arns and nothing else. - on each ExternalId secret, attach a secrets manager **resource policy** that names the orchestrator role's arn as a `Principal.AWS` for `secretsmanager:GetSecretValue`. identity-side scope alone isn't enough , the secret itself must admit the orchestrator. an auditor will check both sides. - the kms key encrypts both secrets. no `*` on kms anywhere. - ssm pointers under `/harbor/...` so the verifier can find the orchestrator role arn and the secret arns without guessing. done looks like this: **happy path** , using the orchestrator's identity, get the tenant-A ExternalId from secrets manager, call `sts:AssumeRole` against `MemberRole-Tenant-A` with `--external-id <secret>`, then put an object into `tenant-a-bucket`. should succeed. **failure path (proven by shape)** , the verifier reads the trust documents directly: - both tenants' trust docs contain `Condition.StringEquals['sts:ExternalId']` matching the per-tenant secret value - both contain `Condition.{ArnLike or ArnEquals}['aws:SourceArn']` equal to the orchestrator role arn - principal is the orchestrator role arn, NOT `*` - action is exactly `sts:AssumeRole` - the two ExternalIds differ between tenants **least privilege checks** , verifier asserts: - orchestrator's identity policy enumerates the two member role arns explicitly (no wildcard assume) - orchestrator's secret-get is scoped to the two known secret arns (no wildcard) - each member role's inline policy references only its own tenant bucket arn (no `*` resource, no other tenant's bucket) - no policy has `Action: "*"` or `Resource: "*"` for s3 or kms names the verifier grabs by hand, use them exactly: | thing | name | |-|-| | iam user (orchestrator caller) | `agent-runner` | | iam user (impersonator stand-in) | `attacker-runner` | | orchestrator role | `OrchestratorRole` | | tenant-A member role | `MemberRole-Tenant-A` | | tenant-B member role | `MemberRole-Tenant-B` | | kms alias | `alias/cross-account-cmk` | | tenant-A external-id secret | `harbor/cross-account/external-id-tenant-a` | | tenant-B external-id secret | `harbor/cross-account/external-id-tenant-b` | | tenant-A bucket | `tenant-a-bucket` | | tenant-B bucket | `tenant-b-bucket` | | ssm orchestrator arn pointer | `/harbor/orchestrator/role-arn` | | ssm tenant-A secret arn pointer | `/harbor/external-id-secret-arn-tenant-a` | | ssm tenant-B secret arn pointer | `/harbor/external-id-secret-arn-tenant-b` | | ssm tenant-A member arn pointer | `/harbor/member-role-arn-tenant-a` | | ssm tenant-B member arn pointer | `/harbor/member-role-arn-tenant-b` | each ExternalId must be at least 32 characters. random-generated, distinct per tenant. write them to the matching secret. trust docs reference the secret VALUE, not the secret arn (the value is the shared secret). policy paths: orchestrator role lives under `/orchestrator/`, member roles live under `/member/`. the verifier uses these paths to enumerate. heads-up on localstack: by default IAM is mocked and not enforced at runtime. the verifier mostly checks policy DOCUMENT shape (exactly what an auditor would do) , so the trust json and inline policies must be correct even if the runtime denial wouldn't actually trigger here. think of localstack as a json validator with an s3 backend.

Verify environment and tools

shell
aws --version && python3 --version && which jq && env | grep AWS
output
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24
Python 3.12.3
/usr/bin/jq
AWS_DEFAULT_REGION=us-east-1
AWS_PAGER=
AWS_SECRET_ACCESS_KEY=test
AWS_ACCESS_KEY_ID=test
AWS_ENDPOINT_URL=http://localstack:4566

[stdout]
aws-cli/2.34.33 Python/3.14.4 Linux/6.10.14-linuxkit exe/aarch64.ubuntu.24
Python 3.12.3
/usr/bin/jq
AWS_DEFAULT_REGION=us-east-1
AWS_PAGER=
AWS_SECRET_ACCESS_KEY=test
AWS_ACCESS_KEY_ID=test
AWS_ENDPOINT_URL=http://localstack:4566

Check localstack health

shell
curl -s http://localstack:4566/_localstack/health | head -c 2000
output
{"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "disabled", "dynamodbstreams": "disabled", "ec2": "disabled", "es": "disabled", "events": "disabled", "firehose": "disabled", "iam": "available", "kinesis": "disabled", "kms": "available", "lambda": "available", "logs": "available", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "available", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "available", "ses": "disabled", "sns": "disabled", "sqs": "disabled", "ssm": "available", "stepfunctions": "disabled", "sts": "available", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "4.13.1"}

[stdout]
{"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "disabled", "dynamodbstreams": "disabled", "ec2": "disabled", "es": "disabled", "events": "disabled", "firehose": "disabled", "iam": "available", "kinesis": "disabled", "kms": "available", "lambda": "available", "logs": "available", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "available", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "available", "ses": "disabled", "sns": "disabled", "sqs": "disabled", "ssm": "available", "stepfunctions": "disabled", "sts": "available", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "4.13.1"}

List app directory

shell
ls /app
output
(Bash completed with no output)
Got it , I'll build this in three files: `build.py` to provision everything, `happy_path.py` to prove the end-to-end chain works, and `verify.py` to check policy document shape (the auditor's lens). Let me confirm a few discovery details and then build.

Get account ID for ARN construction

shell
aws sts get-caller-identity
output
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}

[stdout]
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}

Confirm boto3 version

shell
python3 -c "import boto3; print(boto3.__version__)"
output
1.35.99

[stdout]
1.35.99
Account is `000000000000`. Writing the build script now.
write · /app/build.py
#!/usr/bin/env python3
"""Provision the cross-account trust chain in localstack.

Order matters because of inter-resource references:
  1. Generate two distinct ExternalId values (>=32 chars each)
  2. Create IAM users (agent-runner, attacker-runner)
  3. Create CMK + alias (default policy initially; we replace it after orch role exists)
  4. Create OrchestratorRole (trust = agent-runner user) so we know its ARN
  5. Replace KMS key policy: root admin (specific actions + key arn) + orchestrator decrypt via secretsmanager
  6. Create both ExternalId secrets, encrypted with the CMK
  7. Attach resource policies to both secrets naming OrchestratorRole as principal
  8. Create both member roles with strict trust (orch principal + ExternalId + SourceArn)
  9. Inline policies on member roles: scoped to that tenant's bucket
 10. Inline policies on OrchestratorRole: AssumeRole on the two member arns; GetSecretValue on the two secret arns; kms:Decrypt on the CMK via secretsmanager
 11. Create the two tenant buckets
 12. Publish SSM pointers under /harbor/...
"""
import boto3
import json
import os
import secrets

REGION = "us-east-1"
ENDPOINT = os.environ.get("AWS_ENDPOINT_URL", "http://localstack:4566")

iam = boto3.client("iam", region_name=REGION, endpoint_url=ENDPOINT)
kms = boto3.client("kms", region_name=REGION, endpoint_url=ENDPOINT)
sm = boto3.client("secretsmanager", region_name=REGION, endpoint_url=ENDPOINT)
ssm = boto3.client("ssm", region_name=REGION, endpoint_url=ENDPOINT)
s3 = boto3.client("s3", region_name=REGION, endpoint_url=ENDPOINT)
sts = boto3.client("sts", region_name=REGION, endpoint_url=ENDPOINT)

ACCOUNT = sts.get_caller_identity()["Account"]


def log(msg):
    print(f"[build] {msg}", flush=True)


# 1) ExternalIds , distinct, random, >=32 chars
ext_a = secrets.token_urlsafe(48)
ext_b = secrets.token_urlsafe(48)
assert ext_a != ext_b
assert len(ext_a) >= 32 and len(ext_b) >= 32

# 2) IAM users
for u in ["agent-runner", "attacker-runner"]:
    try:
        iam.create_user(UserName=u)
        log(f"created user {u}")
    except iam.exceptions.EntityAlreadyExistsException:
        log(f"user {u} already exists")

agent_runner_arn = f"arn:aws:iam::{ACCOUNT}:user/agent-runner"

# 3) KMS key + alias (default policy on creation; replaced once we know orchestrator arn)
existing_alias = next(
    (a for a in kms.list_aliases().get("Aliases", []) if a["AliasName"] == "alias/cross-account-cmk"),
    None,
)
if existing_alias:
    key_id = existing_alias["TargetKeyId"]
    log(f"reusing kms key {key_id}")
else:
    res = kms.create_key(Description="cross-account external-id encryption")
    key_id = res["KeyMetadata"]["KeyId"]
    kms.create_alias(AliasName="alias/cross-account-cmk", TargetKeyId=key_id)
    log(f"created kms key {key_id} alias alias/cross-account-cmk")

key_arn = f"arn:aws:kms:{REGION}:{ACCOUNT}:key/{key_id}"

# 4) OrchestratorRole , trust the agent-runner IAM user
orch_trust = {
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {"AWS": agent_runner_arn},
        "Action": "sts:AssumeRole",
    }],
}
try:
    iam.create_role(
        RoleName="OrchestratorRole",
        Path="/orchestrator/",
        AssumeRolePolicyDocument=json.dumps(orch_trust),
    )
    log("created OrchestratorRole")
except iam.exceptions.EntityAlreadyExistsException:
    iam.update_assume_role_policy(
        RoleName="OrchestratorRole",
        PolicyDocument=json.dumps(orch_trust),
    )
    log("updated OrchestratorRole trust")

orch_arn = iam.get_role(RoleName="OrchestratorRole")["Role"]["Arn"]
log(f"orchestrator arn = {orch_arn}")

# 5) Replace KMS key policy with explicit principals + specific key arn
key_policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "RootAccountAdmin",
            "Effect": "Allow",
            "Principal": {"AWS": f"arn:aws:iam::{ACCOUNT}:root"},
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncryptFrom",
                "kms:ReEncryptTo",
                "kms:GenerateDataKey",
                "kms:GenerateDataKeyWithoutPlaintext",
                "kms:DescribeKey",
                "kms:GetKeyPolicy",
                "kms:PutKeyPolicy",
                "kms:CreateAlias",
                "kms:DeleteAlias",
                "kms:UpdateAlias",
                "kms:CreateGrant",
                "kms:RetireGrant",
                "kms:RevokeGrant",
                "kms:ListGrants",
                "kms:EnableKey",
                "kms:DisableKey",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion",
                "kms:TagResource",
                "kms:UntagResource",
            ],
            "Resource": key_arn,
        },
        {
            "Sid": "OrchestratorDecryptViaSecretsManager",
            "Effect": "Allow",
            "Principal": {"AWS": orch_arn},
            "Action": ["kms:Decrypt", "kms:DescribeKey"],
            "Resource": key_arn,
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": f"secretsmanager.{REGION}.amazonaws.com"
                }
            },
        },
    ],
}
kms.put_key_policy(KeyId=key_id, PolicyName="default", Policy=json.dumps(key_policy))
log("kms key policy installed (specific principals, specific key arn, no wildcards)")


# 6) Secrets, encrypted with the CMK
def upsert_secret(name, value):
    try:
        res = sm.create_secret(Name=name, SecretString=value, KmsKeyId=key_id)
        return res["ARN"]
    except sm.exceptions.ResourceExistsException:
        sm.update_secret(SecretId=name, SecretString=value, KmsKeyId=key_id)
        return sm.describe_secret(SecretId=name)["ARN"]


secret_a_arn = upsert_secret("harbor/cross-account/external-id-tenant-a", ext_a)
secret_b_arn = upsert_secret("harbor/cross-account/external-id-tenant-b", ext_b)
log(f"secret A = {secret_a_arn}")
log(f"secret B = {secret_b_arn}")


# 7) Resource policies on secrets , orchestrator role admitted explicitly
def secret_resource_policy(secret_arn):
    return {
        "Version": "2012-10-17",
        "Statement": [{
            "Sid": "AllowOrchestratorRead",
            "Effect": "Allow",
            "Principal": {"AWS": orch_arn},
            "Action": "secretsmanager:GetSecretValue",
            "Resource": secret_arn,
        }],
    }


sm.put_resource_policy(
    SecretId="harbor/cross-account/external-id-tenant-a",
    ResourcePolicy=json.dumps(secret_resource_policy(secret_a_arn)),
)
sm.put_resource_policy(
    SecretId="harbor/cross-account/external-id-tenant-b",
    ResourcePolicy=json.dumps(secret_resource_policy(secret_b_arn)),
)
log("attached resource policies to both secrets")


# 8) Member role trust documents , both conditions on every trust
def member_trust(ext_id):
    return {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {"AWS": orch_arn},
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {"sts:ExternalId": ext_id},
                "ArnEquals": {"aws:SourceArn": orch_arn},
            },
        }],
    }


def upsert_member_role(name, ext_id):
    trust = json.dumps(member_trust(ext_id))
    try:
        iam.create_role(
            RoleName=name, Path="/member/", AssumeRolePolicyDocument=trust
        )
        log(f"created {name}")
    except iam.exceptions.EntityAlreadyExistsException:
        iam.update_assume_role_policy(RoleName=name, PolicyDocument=trust)
        log(f"updated trust on {name}")


upsert_member_role("MemberRole-Tenant-A", ext_a)
upsert_member_role("MemberRole-Tenant-B", ext_b)

member_a_arn = iam.get_role(RoleName="MemberRole-Tenant-A")["Role"]["Arn"]
member_b_arn = iam.get_role(RoleName="MemberRole-Tenant-B")["Role"]["Arn"]


# 9) Member identity policies , scoped to that tenant's bucket only
def bucket_only_policy(bucket):
    return {
        "Version": "2012-10-17",
        "Statement": [{
            "Sid": "OwnTenantBucketAccess",
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:GetObject"],
            "Resource": [f"arn:aws:s3:::{bucket}/*"],
        }],
    }


iam.put_role_policy(
    RoleName="MemberRole-Tenant-A",
    PolicyName="TenantABucketAccess",
    PolicyDocument=json.dumps(bucket_only_policy("tenant-a-bucket")),
)
iam.put_role_policy(
    RoleName="MemberRole-Tenant-B",
    PolicyName="TenantBBucketAccess",
    PolicyDocument=json.dumps(bucket_only_policy("tenant-b-bucket")),
)
log("attached member identity policies (per-tenant bucket only)")

# 10) OrchestratorRole inline policies , explicit member arns + explicit secret arns
orch_assume_policy = {
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "AssumeMemberRolesOnly",
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": [member_a_arn, member_b_arn],
    }],
}
iam.put_role_policy(
    RoleName="OrchestratorRole",
    PolicyName="AssumeMemberRoles",
    PolicyDocument=json.dumps(orch_assume_policy),
)

orch_secret_policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReadExternalIdSecrets",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": [secret_a_arn, secret_b_arn],
        },
        {
            "Sid": "DecryptExternalIdSecretsViaSecretsManager",
            "Effect": "Allow",
            "Action": ["kms:Decrypt", "kms:DescribeKey"],
            "Resource": key_arn,
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": f"secretsmanager.{REGION}.amazonaws.com"
                }
            },
        },
    ],
}
iam.put_role_policy(
    RoleName="OrchestratorRole",
    PolicyName="ReadExternalIdSecrets",
    PolicyDocument=json.dumps(orch_secret_policy),
)
log("attached orchestrator identity policies")

# 11) Tenant buckets
for b in ["tenant-a-bucket", "tenant-b-bucket"]:
    try:
        s3.create_bucket(Bucket=b)
        log(f"created bucket {b}")
    except s3.exceptions.BucketAlreadyOwnedByYou:
        log(f"bucket {b} already exists")
    except s3.exceptions.BucketAlreadyExists:
        log(f"bucket {b} already exists (other-owned)")

# 12) SSM pointers
ssm_params = [
    ("/harbor/orchestrator/role-arn", orch_arn),
    ("/harbor/external-id-secret-arn-tenant-a", secret_a_arn),
    ("/harbor/external-id-secret-arn-tenant-b", secret_b_arn),
    ("/harbor/member-role-arn-tenant-a", member_a_arn),
    ("/harbor/member-role-arn-tenant-b", member_b_arn),
]
for name, val in ssm_params:
    ssm.put_parameter(Name=name, Type="String", Value=val, Overwrite=True)
log("ssm pointers published under /harbor/")

print(json.dumps({
    "account": ACCOUNT,
    "orchestrator_role_arn": orch_arn,
    "member_a_arn": member_a_arn,
    "member_b_arn": member_b_arn,
    "secret_a_arn": secret_a_arn,
    "secret_b_arn": secret_b_arn,
    "kms_key_arn": key_arn,
    "agent_runner_arn": agent_runner_arn,
}, indent=2))
write · /app/happy_path.py
#!/usr/bin/env python3
"""End-to-end happy path. Discovers everything via SSM.

Chain: agent-runner creds -> assume OrchestratorRole -> read tenant-A
ExternalId from Secrets Manager -> assume MemberRole-Tenant-A with
--external-id <secret> -> PutObject into tenant-a-bucket -> GetObject
roundtrip.
"""
import boto3
import os
import sys

REGION = "us-east-1"
ENDPOINT = os.environ.get("AWS_ENDPOINT_URL", "http://localstack:4566")

ssm = boto3.client("ssm", region_name=REGION, endpoint_url=ENDPOINT)
sts = boto3.client("sts", region_name=REGION, endpoint_url=ENDPOINT)


def get_param(name):
    return ssm.get_parameter(Name=name)["Parameter"]["Value"]


orch_arn = get_param("/harbor/orchestrator/role-arn")
secret_a_arn = get_param("/harbor/external-id-secret-arn-tenant-a")
member_a_arn = get_param("/harbor/member-role-arn-tenant-a")

print(f"[happy] orchestrator    = {orch_arn}")
print(f"[happy] member tenant-A = {member_a_arn}")
print(f"[happy] secret  tenant-A= {secret_a_arn}")

# Step 1: assume OrchestratorRole (as agent-runner would)
orch_creds = sts.assume_role(
    RoleArn=orch_arn, RoleSessionName="orchestrator-session"
)["Credentials"]
print("[happy] step 1: assumed OrchestratorRole")

# Step 2: orchestrator reads the tenant-A external id from Secrets Manager
sm = boto3.client(
    "secretsmanager",
    region_name=REGION,
    endpoint_url=ENDPOINT,
    aws_access_key_id=orch_creds["AccessKeyId"],
    aws_secret_access_key=orch_creds["SecretAccessKey"],
    aws_session_token=orch_creds["SessionToken"],
)
ext_id = sm.get_secret_value(SecretId=secret_a_arn)["SecretString"]
print(f"[happy] step 2: read tenant-A ExternalId ({len(ext_id)} chars)")

# Step 3: orchestrator -> assume MemberRole-Tenant-A with --external-id
sts_orch = boto3.client(
    "sts",
    region_name=REGION,
    endpoint_url=ENDPOINT,
    aws_access_key_id=orch_creds["AccessKeyId"],
    aws_secret_access_key=orch_creds["SecretAccessKey"],
    aws_session_token=orch_creds["SessionToken"],
)
member_creds = sts_orch.assume_role(
    RoleArn=member_a_arn,
    RoleSessionName="member-a-session",
    ExternalId=ext_id,
)["Credentials"]
print("[happy] step 3: assumed MemberRole-Tenant-A with ExternalId")

# Step 4: PutObject into tenant-a-bucket using member-role creds
s3 = boto3.client(
    "s3",
    region_name=REGION,
    endpoint_url=ENDPOINT,
    aws_access_key_id=member_creds["AccessKeyId"],
    aws_secret_access_key=member_creds["SecretAccessKey"],
    aws_session_token=member_creds["SessionToken"],
)
s3.put_object(Bucket="tenant-a-bucket", Key="hello.txt", Body=b"happy path proof")
print("[happy] step 4: PutObject -> s3://tenant-a-bucket/hello.txt")

got = s3.get_object(Bucket="tenant-a-bucket", Key="hello.txt")["Body"].read()
if got != b"happy path proof":
    print(f"[happy] roundtrip mismatch: {got!r}")
    sys.exit(1)
print("[happy] step 5: GetObject roundtrip confirmed")
print("[happy] OK")
write · /app/verify.py
#!/usr/bin/env python3
"""Auditor-style verifier , checks policy DOCUMENT shape, not runtime behavior.

Reads SSM pointers, then asserts:
  * each member role's trust contains BOTH StringEquals[sts:ExternalId]
    matching the per-tenant secret value AND ArnEquals/ArnLike
    [aws:SourceArn] equal to the orchestrator role arn
  * trust principal is the orchestrator role arn (not '*')
  * trust action is exactly sts:AssumeRole
  * the two ExternalIds differ between tenants and are >=32 chars
  * orchestrator identity policy enumerates the two member arns and
    the two secret arns explicitly (no wildcards)
  * each member identity policy references only its own tenant bucket
  * each secret has a resource policy admitting the orchestrator role
  * no policy has Action:'*', s3:'*', or kms:'*'; no Resource:'*' for
    s3- or kms-actioned statements
"""
import boto3
import json
import os
import sys

REGION = "us-east-1"
ENDPOINT = os.environ.get("AWS_ENDPOINT_URL", "http://localstack:4566")

iam = boto3.client("iam", region_name=REGION, endpoint_url=ENDPOINT)
ssm = boto3.client("ssm", region_name=REGION, endpoint_url=ENDPOINT)
sm = boto3.client("secretsmanager", region_name=REGION, endpoint_url=ENDPOINT)
kms = boto3.client("kms", region_name=REGION, endpoint_url=ENDPOINT)

failures = []


def check(cond, msg):
    if cond:
        print(f"  PASS {msg}")
    else:
        print(f"  FAIL {msg}")
        failures.append(msg)


def get_param(name):
    return ssm.get_parameter(Name=name)["Parameter"]["Value"]


def stmts(policy):
    s = policy.get("Statement", [])
    return s if isinstance(s, list) else [s]


def as_list(x):
    if x is None:
        return []
    return x if isinstance(x, list) else [x]


def actions_of(s):
    return as_list(s.get("Action"))


def resources_of(s):
    return as_list(s.get("Resource"))


def principal_aws_of(s):
    p = s.get("Principal", {})
    if not isinstance(p, dict):
        return []
    return as_list(p.get("AWS"))


print("== ssm pointers ==")
orch_arn = get_param("/harbor/orchestrator/role-arn")
member_a_arn = get_param("/harbor/member-role-arn-tenant-a")
member_b_arn = get_param("/harbor/member-role-arn-tenant-b")
secret_a_arn = get_param("/harbor/external-id-secret-arn-tenant-a")
secret_b_arn = get_param("/harbor/external-id-secret-arn-tenant-b")
print(f"  orchestrator: {orch_arn}")
print(f"  member-A:     {member_a_arn}")
print(f"  member-B:     {member_b_arn}")
print(f"  secret-A:     {secret_a_arn}")
print(f"  secret-B:     {secret_b_arn}")

print("\n== role enumeration via path ==")
orch_roles = iam.list_roles(PathPrefix="/orchestrator/")["Roles"]
check(
    [r["RoleName"] for r in orch_roles] == ["OrchestratorRole"],
    "/orchestrator/ contains exactly OrchestratorRole",
)
member_roles = iam.list_roles(PathPrefix="/member/")["Roles"]
member_names = sorted(r["RoleName"] for r in member_roles)
check(
    member_names == ["MemberRole-Tenant-A", "MemberRole-Tenant-B"],
    f"/member/ contains exactly the two member roles (got {member_names})",
)


def check_member_trust(role_name, ext_id_value, label):
    print(f"\n== {role_name} trust ==")
    role = iam.get_role(RoleName=role_name)["Role"]
    doc = role["AssumeRolePolicyDocument"]
    if isinstance(doc, str):
        doc = json.loads(doc)
    sts_list = stmts(doc)
    check(len(sts_list) >= 1, f"{label}: trust has at least one Statement")
    s = sts_list[0]

    check(s.get("Effect") == "Allow", f"{label}: Effect == Allow")

    principals = principal_aws_of(s)
    check("*" not in principals, f"{label}: Principal.AWS is NOT '*'")
    check(orch_arn in principals, f"{label}: Principal.AWS includes orchestrator role arn")

    acts = actions_of(s)
    check(acts == ["sts:AssumeRole"], f"{label}: Action is exactly ['sts:AssumeRole'] (got {acts})")

    cond = s.get("Condition", {})
    str_eq_ext = cond.get("StringEquals", {}).get("sts:ExternalId")
    check(
        str_eq_ext == ext_id_value,
        f"{label}: Condition.StringEquals.sts:ExternalId matches the secret VALUE",
    )

    src_arn = None
    cond_op_used = None
    for op in ("ArnEquals", "ArnLike"):
        v = cond.get(op, {}).get("aws:SourceArn")
        if v is not None:
            src_arn = v
            cond_op_used = op
            break
    check(
        src_arn == orch_arn,
        f"{label}: Condition.{{ArnEquals|ArnLike}}.aws:SourceArn == orchestrator arn"
        + (f" (via {cond_op_used})" if cond_op_used else ""),
    )


ext_a = sm.get_secret_value(SecretId=secret_a_arn)["SecretString"]
ext_b = sm.get_secret_value(SecretId=secret_b_arn)["SecretString"]
check_member_trust("MemberRole-Tenant-A", ext_a, "tenant-A")
check_member_trust("MemberRole-Tenant-B", ext_b, "tenant-B")

print("\n== external id properties ==")
check(len(ext_a) >= 32, f"tenant-A ExternalId length >= 32 (got {len(ext_a)})")
check(len(ext_b) >= 32, f"tenant-B ExternalId length >= 32 (got {len(ext_b)})")
check(ext_a != ext_b, "tenant-A and tenant-B ExternalIds differ")


def collect_inline_statements(role_name):
    out = []
    for pn in iam.list_role_policies(RoleName=role_name)["PolicyNames"]:
        doc = iam.get_role_policy(RoleName=role_name, PolicyName=pn)["PolicyDocument"]
        if isinstance(doc, str):
            doc = json.loads(doc)
        for s in stmts(doc):
            out.append((pn, s))
    return out


print("\n== orchestrator identity policies ==")
orch_pairs = collect_inline_statements("OrchestratorRole")
assume_resources = []
for pn, s in orch_pairs:
    if s.get("Effect") != "Allow":
        continue
    if "sts:AssumeRole" in actions_of(s):
        assume_resources.extend(resources_of(s))
check(
    set(assume_resources) == {member_a_arn, member_b_arn},
    f"orchestrator sts:AssumeRole resources are exactly the two member arns (got {assume_resources})",
)
check(
    "*" not in assume_resources,
    "orchestrator sts:AssumeRole resource list contains no '*'",
)

sec_resources = []
for pn, s in orch_pairs:
    if s.get("Effect") != "Allow":
        continue
    if "secretsmanager:GetSecretValue" in actions_of(s):
        sec_resources.extend(resources_of(s))
check(
    set(sec_resources) == {secret_a_arn, secret_b_arn},
    f"orchestrator GetSecretValue resources are exactly the two secret arns (got {sec_resources})",
)
check("*" not in sec_resources, "orchestrator GetSecretValue resource list contains no '*'")


def check_member_identity(role_name, own_bucket, other_bucket, label):
    print(f"\n== {role_name} identity policy ==")
    own_prefix = f"arn:aws:s3:::{own_bucket}"
    other_prefix = f"arn:aws:s3:::{other_bucket}"
    pairs = collect_inline_statements(role_name)
    all_resources, all_actions = [], []
    for pn, s in pairs:
        if s.get("Effect") != "Allow":
            continue
        all_resources.extend(resources_of(s))
        all_actions.extend(actions_of(s))
    check(
        len(all_resources) > 0 and all(r.startswith(own_prefix) for r in all_resources),
        f"{label}: every resource references {own_bucket} (got {all_resources})",
    )
    check(
        not any(r.startswith(other_prefix) for r in all_resources),
        f"{label}: no resource references {other_bucket}",
    )
    check("*" not in all_resources, f"{label}: no Resource '*'")
    check("*" not in all_actions, f"{label}: no Action '*'")
    check("s3:*" not in all_actions, f"{label}: no Action 's3:*'")


check_member_identity("MemberRole-Tenant-A", "tenant-a-bucket", "tenant-b-bucket", "tenant-A")
check_member_identity("MemberRole-Tenant-B", "tenant-b-bucket", "tenant-a-bucket", "tenant-B")


def check_secret_resource_policy(secret_id, label):
    print(f"\n== {label} secret resource policy ==")
    rp = sm.get_resource_policy(SecretId=secret_id)
    rp_str = rp.get("ResourcePolicy")
    check(bool(rp_str), f"{label}: resource policy attached")
    if not rp_str:
        return
    pol = json.loads(rp_str)
    found = False
    for s in stmts(pol):
        if s.get("Effect") != "Allow":
            continue
        if "secretsmanager:GetSecretValue" not in actions_of(s):
            continue
        if orch_arn in principal_aws_of(s):
            found = True
            break
    check(found, f"{label}: resource policy admits orchestrator role for GetSecretValue")


check_secret_resource_policy("harbor/cross-account/external-id-tenant-a", "tenant-A")
check_secret_resource_policy("harbor/cross-account/external-id-tenant-b", "tenant-B")

print("\n== wildcard scan (s3 / kms) over all role inline policies ==")
WILDCARD_ACTIONS = {"*", "s3:*", "kms:*"}
for role_name in ["OrchestratorRole", "MemberRole-Tenant-A", "MemberRole-Tenant-B"]:
    for pn, s in collect_inline_statements(role_name):
        if s.get("Effect") != "Allow":
            continue
        acts = actions_of(s)
        rsrcs = resources_of(s)
        for a in acts:
            if a in WILDCARD_ACTIONS:
                msg = f"{role_name}/{pn}: forbidden wildcard action {a!r}"
                print(f"  FAIL {msg}")
                failures.append(msg)
        if "*" in rsrcs:
            for a in acts:
                svc = a.split(":")[0] if ":" in a else None
                if svc in ("s3", "kms"):
                    msg = f"{role_name}/{pn}: Resource '*' on {svc} action {a!r}"
                    print(f"  FAIL {msg}")
                    failures.append(msg)
print("  (scan done)")

print("\n== kms key policy ==")
key_id = kms.describe_key(KeyId="alias/cross-account-cmk")["KeyMetadata"]["KeyId"]
kp_str = kms.get_key_policy(KeyId=key_id, PolicyName="default")["Policy"]
kp = json.loads(kp_str)
for s in stmts(kp):
    if s.get("Effect") != "Allow":
        continue
    principals = principal_aws_of(s)
    check(
        "*" not in principals,
        f"kms key policy stmt {s.get('Sid', '?')!r}: Principal.AWS is not '*'",
    )
    acts = actions_of(s)
    check(
        "*" not in acts and "kms:*" not in acts,
        f"kms key policy stmt {s.get('Sid', '?')!r}: no wildcard kms action",
    )
    rsrcs = resources_of(s)
    check(
        "*" not in rsrcs,
        f"kms key policy stmt {s.get('Sid', '?')!r}: Resource is not '*'",
    )

print()
if failures:
    print(f"== VERIFY FAILED ({len(failures)} issues) ==")
    for f in failures:
        print(f"  - {f}")
    sys.exit(1)
print("== VERIFY OK ==")
Now running build, happy path, and verifier in sequence.

Run the build script

shell
python3 build.py
output
[build] created user agent-runner
[build] created user attacker-runner
[build] created kms key 00829f8d-6b6f-4269-b0a3-0657172bb5dc alias alias/cross-account-cmk
[build] created OrchestratorRole
[build] orchestrator arn = arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
[build] kms key policy installed (specific principals, specific key arn, no wildcards)
[build] secret A = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi
[build] secret B = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-CxgtmV
[build] attached resource policies to both secrets
[build] created MemberRole-Tenant-A
[build] created MemberRole-Tenant-B
[build] attached member identity policies (per-tenant bucket only)
[build] attached orchestrator identity policies
[build] created bucket tenant-a-bucket
[build] created bucket tenant-b-bucket
[build] ssm pointers published under /harbor/
{
  "account": "000000000000",
  "orchestrator_role_arn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole",
  "member_a_arn": "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A",
  "member_b_arn": "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B",
  "secret_a_arn": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi",
  "secret_b_arn": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-CxgtmV",
  "kms_key_arn": "arn:aws:kms:us-east-1:000000000000:key/00829f8d-6b6f-4269-b0a3-0657172bb5dc",
  "agent_runner_arn": "arn:aws:iam::000000000000:user/agent-runner"
}

[stdout]
[build] created user agent-runner
[build] created user attacker-runner
[build] created kms key 00829f8d-6b6f-4269-b0a3-0657172bb5dc alias alias/cross-account-cmk
[build] created OrchestratorRole
[build] orchestrator arn = arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
[build] kms key policy installed (specific principals, specific key arn, no wildcards)
[build] secret A = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi
[build] secret B = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-CxgtmV
[build] attached resource policies to both secrets
[build] created MemberRole-Tenant-A
[build] created MemberRole-Tenant-B
[build] attached member identity policies (per-tenant bucket only)
[build] attached orchestrator identity policies
[build] created bucket tenant-a-bucket
[build] created bucket tenant-b-bucket
[build] ssm pointers published under /harbor/
{
  "account": "000000000000",
  "orchestrator_role_arn": "arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole",
  "member_a_arn": "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A",
  "member_b_arn": "arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B",
  "secret_a_arn": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi",
  "secret_b_arn": "arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-CxgtmV",
  "kms_key_arn": "arn:aws:kms:us-east-1:000000000000:key/00829f8d-6b6f-4269-b0a3-0657172bb5dc",
  "agent_runner_arn": "arn:aws:iam::000000000000:user/agent-runner"
}

Run the happy path test

shell
python3 happy_path.py
output
[happy] orchestrator    = arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
[happy] member tenant-A = arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
[happy] secret  tenant-A= arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi
[happy] step 1: assumed OrchestratorRole
[happy] step 2: read tenant-A ExternalId (64 chars)
[happy] step 3: assumed MemberRole-Tenant-A with ExternalId
[happy] step 4: PutObject -> s3://tenant-a-bucket/hello.txt
[happy] step 5: GetObject roundtrip confirmed
[happy] OK

[stdout]
[happy] orchestrator    = arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
[happy] member tenant-A = arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
[happy] secret  tenant-A= arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi
[happy] step 1: assumed OrchestratorRole
[happy] step 2: read tenant-A ExternalId (64 chars)
[happy] step 3: assumed MemberRole-Tenant-A with ExternalId
[happy] step 4: PutObject -> s3://tenant-a-bucket/hello.txt
[happy] step 5: GetObject roundtrip confirmed
[happy] OK

Run the verifier

shell
python3 verify.py
output
== ssm pointers ==
  orchestrator: arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  member-A:     arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  member-B:     arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  secret-A:     arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi
  secret-B:     arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-CxgtmV

== role enumeration via path ==
  PASS /orchestrator/ contains exactly OrchestratorRole
  PASS /member/ contains exactly the two member roles (got ['MemberRole-Tenant-A', 'MemberRole-Tenant-B'])

== MemberRole-Tenant-A trust ==
  PASS tenant-A: trust has at least one Statement
  PASS tenant-A: Effect == Allow
  PASS tenant-A: Principal.AWS is NOT '*'
  PASS tenant-A: Principal.AWS includes orchestrator role arn
  PASS tenant-A: Action is exactly ['sts:AssumeRole'] (got ['sts:AssumeRole'])
  PASS tenant-A: Condition.StringEquals.sts:ExternalId matches the secret VALUE
  PASS tenant-A: Condition.{ArnEquals|ArnLike}.aws:SourceArn == orchestrator arn (via ArnEquals)

== MemberRole-Tenant-B trust ==
  PASS tenant-B: trust has at least one Statement
  PASS tenant-B: Effect == Allow
  PASS tenant-B: Principal.AWS is NOT '*'
  PASS tenant-B: Principal.AWS includes orchestrator role arn
  PASS tenant-B: Action is exactly ['sts:AssumeRole'] (got ['sts:AssumeRole'])
  PASS tenant-B: Condition.StringEquals.sts:ExternalId matches the secret VALUE
  PASS tenant-B: Condition.{ArnEquals|ArnLike}.aws:SourceArn == orchestrator arn (via ArnEquals)

== external id properties ==
  PASS tenant-A ExternalId length >= 32 (got 64)
  PASS tenant-B ExternalId length >= 32 (got 64)
  PASS tenant-A and tenant-B ExternalIds differ

== orchestrator identity policies ==
  PASS orchestrator sts:AssumeRole resources are exactly the two member arns (got ['arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A', 'arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B'])
  PASS orchestrator sts:AssumeRole resource list contains no '*'
  PASS orchestrator GetSecretValue resources are exactly the two secret arns (got ['arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi', 'arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-CxgtmV'])
  PASS orchestrator GetSecretValue resource list contains no '*'

== MemberRole-Tenant-A identity policy ==
  PASS tenant-A: every resource references tenant-a-bucket (got ['arn:aws:s3:::tenant-a-bucket/*'])
  PASS tenant-A: no resource references tenant-b-bucket
  PASS tenant-A: no Resource '*'
  PASS tenant-A: no Action '*'
  PASS tenant-A: no Action 's3:*'

== MemberRole-Tenant-B identity policy ==
  PASS tenant-B: every resource references tenant-b-bucket (got ['arn:aws:s3:::tenant-b-bucket/*'])
  PASS tenant-B: no resource references tenant-a-bucket
  PASS tenant-B: no Resource '*'
  PASS tenant-B: no Action '*'
  PASS tenant-B: no Action 's3:*'

== tenant-A secret resource policy ==
  PASS tenant-A: resource policy attached
  PASS tenant-A: resource policy admits orchestrator role for GetSecretValue

== tenant-B secret resource policy ==
  PASS tenant-B: resource policy attached
  PASS tenant-B: resource policy admits orchestrator role for GetSecretValue

== wildcard scan (s3 / kms) over all role inline policies ==
  (scan done)

== kms key policy ==
  PASS kms key policy stmt 'RootAccountAdmin': Principal.AWS is not '*'
  PASS kms key policy stmt 'RootAccountAdmin': no wildcard kms action
  PASS kms key policy stmt 'RootAccountAdmin': Resource is not '*'
  PASS kms key policy stmt 'OrchestratorDecryptViaSecretsManager': Principal.AWS is not '*'
  PASS kms key policy stmt 'OrchestratorDecryptViaSecretsManager': no wildcard kms action
  PASS kms key policy stmt 'OrchestratorDecryptViaSecretsManager': Resource is not '*'

== VERIFY OK ==

[stdout]
== ssm pointers ==
  orchestrator: arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  member-A:     arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  member-B:     arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  secret-A:     arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi
  secret-B:     arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-CxgtmV

== role enumeration via path ==
  PASS /orchestrator/ contains exactly OrchestratorRole
  PASS /member/ contains exactly the two member roles (got ['MemberRole-Tenant-A', 'MemberRole-Tenant-B'])

== MemberRole-Tenant-A trust ==
  PASS tenant-A: trust has at least one Statement
  PASS tenant-A: Effect == Allow
  PASS tenant-A: Principal.AWS is NOT '*'
  PASS tenant-A: Principal.AWS includes orchestrator role arn
  PASS tenant-A: Action is exactly ['sts:AssumeRole'] (got ['sts:AssumeRole'])
  PASS tenant-A: Condition.StringEquals.sts:ExternalId matches the secret VALUE
  PASS tenant-A: Condition.{ArnEquals|ArnLike}.aws:SourceArn == orchestrator arn (via ArnEquals)

== MemberRole-Tenant-B trust ==
  PASS tenant-B: trust has at least one Statement
  PASS tenant-B: Effect == Allow
  PASS tenant-B: Principal.AWS is NOT '*'
  PASS tenant-B: Principal.AWS includes orchestrator role arn
  PASS tenant-B: Action is exactly ['sts:AssumeRole'] (got ['sts:AssumeRole'])
  PASS tenant-B: Condition.StringEquals.sts:ExternalId matches the secret VALUE
  PASS tenant-B: Condition.{ArnEquals|ArnLike}.aws:SourceArn == orchestrator arn (via ArnEquals)

== external id properties ==
  PASS tenant-A ExternalId length >= 32 (got 64)
  PASS tenant-B ExternalId length >= 32 (got 64)
  PASS tenant-A and tenant-B ExternalIds differ

== orchestrator identity policies ==
  PASS orchestrator sts:AssumeRole resources are exactly the two member arns (got ['arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A', 'arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B'])
  PASS orchestrator sts:AssumeRole resource list contains no '*'
  PASS orchestrator GetSecretValue resources are exactly the two secret arns (got ['arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZskPfi', 'arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-CxgtmV'])
  PASS orchestrator GetSecretValue resource list contains no '*'

== MemberRole-Tenant-A identity policy ==
  PASS tenant-A: every resource references tenant-a-bucket (got ['arn:aws:s3:::tenant-a-bucket/*'])
  PASS tenant-A: no resource references tenant-b-bucket
  PASS tenant-A: no Resource '*'
  PASS tenant-A: no Action '*'
  PASS tenant-A: no Action 's3:*'

== MemberRole-Tenant-B identity policy ==
  PASS tenant-B: every resource references tenant-b-bucket (got ['arn:aws:s3:::tenant-b-bucket/*'])
  PASS tenant-B: no resource references tenant-a-bucket
  PASS tenant-B: no Resource '*'
  PASS tenant-B: no Action '*'
  PASS tenant-B: no Action 's3:*'

== tenant-A secret resource policy ==
  PASS tenant-A: resource policy attached
  PASS tenant-A: resource policy admits orchestrator role for GetSecretValue

== tenant-B secret resource policy ==
  PASS tenant-B: resource policy attached
  PASS tenant-B: resource policy admits orchestrator role for GetSecretValue

== wildcard scan (s3 / kms) over all role inline policies ==
  (scan done)

== kms key policy ==
  PASS kms key policy stmt 'RootAccountAdmin': Principal.AWS is not '*'
  PASS kms key policy stmt 'RootAccountAdmin': no wildcard kms action
  PASS kms key policy stmt 'RootAccountAdmin': Resource is not '*'
  PASS kms key policy stmt 'OrchestratorDecryptViaSecretsManager': Principal.AWS is not '*'
  PASS kms key policy stmt 'OrchestratorDecryptViaSecretsManager': no wildcard kms action
  PASS kms key policy stmt 'OrchestratorDecryptViaSecretsManager': Resource is not '*'

== VERIFY OK ==
Build, happy path, and verifier all pass. Summary: **What got built** (`build.py` , idempotent): - IAM users `agent-runner` and `attacker-runner` - KMS CMK with alias `alias/cross-account-cmk`; key policy uses specific principals (root + OrchestratorRole) and the specific key ARN , no `*` - `OrchestratorRole` under `/orchestrator/`, trusted by the `agent-runner` IAM user - Two ExternalId secrets (64-char URL-safe random, distinct per tenant), encrypted with the CMK, each with a resource policy admitting the orchestrator role - `MemberRole-Tenant-A` and `MemberRole-Tenant-B` under `/member/`, each trust enforcing **both** `StringEquals[sts:ExternalId]` and `ArnEquals[aws:SourceArn] == orchestrator arn`, principal pinned to the orchestrator arn (no `*`), action exactly `sts:AssumeRole` - Member identity policies scoped to that tenant's bucket only (object ARN, no wildcard, no other tenant) - Orchestrator identity policies enumerating the two member arns and the two secret arns explicitly, plus `kms:Decrypt` scoped to the CMK with `kms:ViaService=secretsmanager.us-east-1.amazonaws.com` - `tenant-a-bucket` and `tenant-b-bucket` - SSM pointers under `/harbor/...` **Happy path** (`happy_path.py`): agent-runner -> AssumeRole(OrchestratorRole) -> GetSecretValue(tenant-A) -> AssumeRole(MemberRole-Tenant-A, ExternalId=...) -> PutObject + GetObject roundtrip on `tenant-a-bucket`. Pass. **Verifier** (`verify.py`): every shape assertion the auditor cares about passes , both trust conditions on each member role, ExternalId values match the secrets and differ between tenants, no wildcards in identity or KMS policies, secret resource policies admit the orchestrator role explicitly. Pass.
Final diff , what the agent changed

No file changes, the agent left the workspace untouched.

Verifier output , the proof behind the reward
tests/test.sh
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/x07pTSFstGkU6puWsFmL9/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 20 items

../tests/test_state.py::test_01_orchestrator_role_exists_at_orchestrator_path PASSED [  5%]
../tests/test_state.py::test_02_member_roles_exist_at_member_path PASSED [ 10%]
../tests/test_state.py::test_03_external_id_secrets_exist_with_min_length PASSED [ 15%]
../tests/test_state.py::test_04_both_tenant_buckets_exist PASSED         [ 20%]
../tests/test_state.py::test_05_ssm_pointers_resolve PASSED              [ 25%]
../tests/test_state.py::test_06_member_trusts_have_externalid_stringequals_matching_secret PASSED [ 30%]
../tests/test_state.py::test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn PASSED [ 35%]
../tests/test_state.py::test_08_member_trusts_action_is_assumerole_only PASSED [ 40%]
../tests/test_state.py::test_09_member_trusts_principal_is_orchestrator_not_wildcard PASSED [ 45%]
../tests/test_state.py::test_10_externalids_differ_per_tenant PASSED     [ 50%]
../tests/test_state.py::test_11_orchestrator_identity_lists_member_arns_explicitly_no_wildcard PASSED [ 55%]
../tests/test_state.py::test_12_orchestrator_secret_get_scoped_to_two_secret_arns PASSED [ 60%]
../tests/test_state.py::test_13_orchestrator_kms_decrypt_scoped_to_cmk_arn PASSED [ 65%]
../tests/test_state.py::test_14_member_inline_policies_scope_to_own_tenant_bucket_only PASSED [ 70%]
../tests/test_state.py::test_15_secret_resource_policies_admit_orchestrator PASSED [ 75%]
../tests/test_state.py::test_16_kms_key_policy_has_account_root_statement FAILED [ 80%]
../tests/test_state.py::test_17_no_admin_managed_policies_attached_to_any_role PASSED [ 85%]
../tests/test_state.py::test_18_e2e_assume_with_externalid_then_put_object_succeeds PASSED [ 90%]
../tests/test_state.py::test_19_member_a_inline_does_not_reference_tenant_b_bucket PASSED [ 95%]
../tests/test_state.py::test_20_orchestrator_has_no_wildcard_assume_role_resource PASSED [100%]

=================================== FAILURES ===================================
______________ test_16_kms_key_policy_has_account_root_statement _______________

    def test_16_kms_key_policy_has_account_root_statement():
        """KMS key policy has root statement."""
        cmk_arn = _kms_key_arn()
        kms = _client("kms")
        pol = kms.get_key_policy(KeyId=cmk_arn, PolicyName="default")["Policy"]
        doc = json.loads(pol)
        found_root = False
        for s in _stmts(doc):
            p = s.get("Principal", {})
            aws = p.get("AWS") if isinstance(p, dict) else None
            arns = _as_list(aws)
            if any(a.endswith(":root") for a in arns):
                actions = _as_list(s.get("Action"))
                if actions == ["kms:*"] or "kms:*" in actions:
                    found_root = True
>       assert found_root, (
            "KMS key policy missing the 'EnableIAMUserPermissions' root statement"
        )
E       AssertionError: KMS key policy missing the 'EnableIAMUserPermissions' root statement
E       assert False

/tests/test_state.py:388: AssertionError
=============================== warnings summary ===============================
test_state.py: 60 warnings
  /root/.cache/uv/archive-v0/x07pTSFstGkU6puWsFmL9/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_state.py::test_01_orchestrator_role_exists_at_orchestrator_path
PASSED ../tests/test_state.py::test_02_member_roles_exist_at_member_path
PASSED ../tests/test_state.py::test_03_external_id_secrets_exist_with_min_length
PASSED ../tests/test_state.py::test_04_both_tenant_buckets_exist
PASSED ../tests/test_state.py::test_05_ssm_pointers_resolve
PASSED ../tests/test_state.py::test_06_member_trusts_have_externalid_stringequals_matching_secret
PASSED ../tests/test_state.py::test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn
PASSED ../tests/test_state.py::test_08_member_trusts_action_is_assumerole_only
PASSED ../tests/test_state.py::test_09_member_trusts_principal_is_orchestrator_not_wildcard
PASSED ../tests/test_state.py::test_10_externalids_differ_per_tenant
PASSED ../tests/test_state.py::test_11_orchestrator_identity_lists_member_arns_explicitly_no_wildcard
PASSED ../tests/test_state.py::test_12_orchestrator_secret_get_scoped_to_two_secret_arns
PASSED ../tests/test_state.py::test_13_orchestrator_kms_decrypt_scoped_to_cmk_arn
PASSED ../tests/test_state.py::test_14_member_inline_policies_scope_to_own_tenant_bucket_only
PASSED ../tests/test_state.py::test_15_secret_resource_policies_admit_orchestrator
PASSED ../tests/test_state.py::test_17_no_admin_managed_policies_attached_to_any_role
PASSED ../tests/test_state.py::test_18_e2e_assume_with_externalid_then_put_object_succeeds
PASSED ../tests/test_state.py::test_19_member_a_inline_does_not_reference_tenant_b_bucket
PASSED ../tests/test_state.py::test_20_orchestrator_has_no_wildcard_assume_role_resource
FAILED ../tests/test_state.py::test_16_kms_key_policy_has_account_root_statement
================== 1 failed, 19 passed, 60 warnings in 0.54s ===================

Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_d1e39deba97e4071. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.

Trial trial_d1e39deba97e4071 · verifier authoritative; classifier explanatory.