SyncValsverifier → artifact → classifier → verdict
SyncVals · Trajectory

iam-cross-account-externalid-sourcearn

claude-code claude-opus-4-7 ✓ resolved GOOD_SUCCESS ↑ View task
Solved from the instruction alone, tests/ and solution/ were withheld from the agent's workspace and restored only for grading.
Reward = tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.
Classification , post-hoc; cannot change the reward
GOOD_SUCCESSLegitimate solve, implements the asked-for behavior; tests verify real functionality.
SubtypeLegitimate Solution
EvidenceAll 20 tests passed. Agent correctly built: (1) OrchestratorRole under /orchestrator/ trusted by agent-runner; (2) MemberRole-Tenant-A/B under /member/ with trust policies requiring BOTH StringEquals on sts:ExternalId (per-tenant secret value) AND ArnEquals on aws:SourceArn (orchestrator ARN); (3) distinct 48-char random ExternalIds stored in KMS-encrypted secrets with resource policies; (4) identity policies with explicit member-role enumeration (no wildcards), secret-ARN scoping (no wildcards), tenant-bucket-scoped member policies (no cross-tenant ARN references); (5) happy-path e2e verification: assume orchestrator → fetch ExternalId → assume member with --external-id → S3 put/get. Test output: 'test_06_member_trusts_have_externalid_stringequals_matching_secret PASSED', 'test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn PASSED', 'test_15_secret_resource_policies_admit_orchestrator PASSED', 'test_18_e2e_assume_with_externalid_then_put_object_succeeds PASSED', 'test_19_member_a_inline_does_not_reference_tenant_b_bucket PASSED'.
Root causeThe agent correctly understood the complex SOC2-grade cross-account trust requirement and implemented all mandatory defenses: third-party impersonation protection (ExternalId) and confused-deputy protection (SourceArn) on every member role trust, plus least-privilege identity and resource policies with proper KMS/secrets encryption and scoping.
RecommendationN/A - task is fine
Trajectory
Tool-by-tool agent trajectory
6 tool calls · 2 tool types · 11 steps
our security review came back ugly. an external auditor asked how we keep tenant A's orchestrator from poking at tenant B's bucket, and how we'd block a contractor from impersonating our orchestrator if they ever get hold of one of our role names. honest answer right now: we don't. the cross-account assume-role chain we have is "trust the orchestrator role arn" and that's it. they want this fixed by next sprint or we lose the SOC2 carryover. so: build the cross-account trust the way it's supposed to be done. one orchestrator role that fans out to two tenant member roles. each member role's trust must defend against TWO different attacks at the same time: 1. **third-party impersonation** , someone outside the company learns one of our member role arns and tries to assume it. fix: every assume-role call must present a per-tenant `ExternalId` shared secret. without it, denied. 2. **AWS-service confused-deputy** , some other principal in our account (or a service we've granted broadly) gets tricked into calling AssumeRole on our behalf. fix: the trust must require `aws:SourceArn` to equal the orchestrator role's arn. without it, denied. both conditions on every member role trust. not one or the other. both. shape of it: - localstack at `http://localstack:4566`. creds already exported (`AWS_ACCESS_KEY_ID=test`, same for secret, region `us-east-1`). `aws`, `python3`, `boto3`, `jq`. build from zero. - one **orchestrator role** trusted by an iam user that represents the orchestrator's runtime identity. - two **member roles**, one per tenant. each member role's trust policy: - principal = orchestrator role arn (specific, not `*`) - condition `StringEquals` on `sts:ExternalId` matching that tenant's secret value - condition on `aws:SourceArn` equal to the orchestrator role arn - action = `sts:AssumeRole` only , no wildcards - per-tenant **ExternalId** values stored in secrets manager, encrypted with a customer-managed kms key. the orchestrator reads the secret, doesn't hardcode it. and the two tenants get DIFFERENT ExternalIds , reusing one across tenants defeats the point. - each member role's identity policy is scoped to ONLY that tenant's bucket. tenant A's role can put/get on `tenant-a-bucket` and nothing else. no `Resource: "*"`. - the orchestrator's identity policy lists the two member role arns explicitly under `sts:AssumeRole` , no `Resource: "*"` there either. - secret access scoped: orchestrator can `secretsmanager:GetSecretValue` on the two ExternalId secret arns and nothing else. - on each ExternalId secret, attach a secrets manager **resource policy** that names the orchestrator role's arn as a `Principal.AWS` for `secretsmanager:GetSecretValue`. identity-side scope alone isn't enough , the secret itself must admit the orchestrator. an auditor will check both sides. - the kms key encrypts both secrets. no `*` on kms anywhere. - ssm pointers under `/harbor/...` so the verifier can find the orchestrator role arn and the secret arns without guessing. done looks like this: **happy path** , using the orchestrator's identity, get the tenant-A ExternalId from secrets manager, call `sts:AssumeRole` against `MemberRole-Tenant-A` with `--external-id <secret>`, then put an object into `tenant-a-bucket`. should succeed. **failure path (proven by shape)** , the verifier reads the trust documents directly: - both tenants' trust docs contain `Condition.StringEquals['sts:ExternalId']` matching the per-tenant secret value - both contain `Condition.{ArnLike or ArnEquals}['aws:SourceArn']` equal to the orchestrator role arn - principal is the orchestrator role arn, NOT `*` - action is exactly `sts:AssumeRole` - the two ExternalIds differ between tenants **least privilege checks** , verifier asserts: - orchestrator's identity policy enumerates the two member role arns explicitly (no wildcard assume) - orchestrator's secret-get is scoped to the two known secret arns (no wildcard) - each member role's inline policy references only its own tenant bucket arn (no `*` resource, no other tenant's bucket) - no policy has `Action: "*"` or `Resource: "*"` for s3 or kms names the verifier grabs by hand, use them exactly: | thing | name | |-|-| | iam user (orchestrator caller) | `agent-runner` | | iam user (impersonator stand-in) | `attacker-runner` | | orchestrator role | `OrchestratorRole` | | tenant-A member role | `MemberRole-Tenant-A` | | tenant-B member role | `MemberRole-Tenant-B` | | kms alias | `alias/cross-account-cmk` | | tenant-A external-id secret | `harbor/cross-account/external-id-tenant-a` | | tenant-B external-id secret | `harbor/cross-account/external-id-tenant-b` | | tenant-A bucket | `tenant-a-bucket` | | tenant-B bucket | `tenant-b-bucket` | | ssm orchestrator arn pointer | `/harbor/orchestrator/role-arn` | | ssm tenant-A secret arn pointer | `/harbor/external-id-secret-arn-tenant-a` | | ssm tenant-B secret arn pointer | `/harbor/external-id-secret-arn-tenant-b` | | ssm tenant-A member arn pointer | `/harbor/member-role-arn-tenant-a` | | ssm tenant-B member arn pointer | `/harbor/member-role-arn-tenant-b` | each ExternalId must be at least 32 characters. random-generated, distinct per tenant. write them to the matching secret. trust docs reference the secret VALUE, not the secret arn (the value is the shared secret). policy paths: orchestrator role lives under `/orchestrator/`, member roles live under `/member/`. the verifier uses these paths to enumerate. heads-up on localstack: by default IAM is mocked and not enforced at runtime. the verifier mostly checks policy DOCUMENT shape (exactly what an auditor would do) , so the trust json and inline policies must be correct even if the runtime denial wouldn't actually trigger here. think of localstack as a json validator with an s3 backend.
I'll build the cross-account trust setup end-to-end. Let me first verify the environment, then construct the resources in dependency order.

Check working dir and verify localstack reachable

shell
ls /app && which aws python3 jq && aws --endpoint-url=http://localstack:4566 sts get-caller-identity
output
/usr/local/bin/aws
/opt/venv/bin/python3
/usr/bin/jq
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}

[stdout]
/usr/local/bin/aws
/opt/venv/bin/python3
/usr/bin/jq
{
    "UserId": "AKIAIOSFODNN7EXAMPLE",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:root"
}
Account is `000000000000`. /app is empty. I'll build a setup script (creates everything) plus a verifier that runs the happy path and validates policy doc shape.

Confirm /app is empty

shell
ls -la /app/
output
total 8
drwxr-xr-x 2 root root 4096 Apr 21 20:05 .
drwxr-xr-x 1 root root 4096 Apr 28 10:00 ..

[stdout]
total 8
drwxr-xr-x 2 root root 4096 Apr 21 20:05 .
drwxr-xr-x 1 root root 4096 Apr 28 10:00 ..
write · /app/setup.py
"""Build the cross-account trust topology end-to-end on LocalStack.

Resources created (in dependency order):
  - IAM users: agent-runner (orchestrator caller), attacker-runner (impersonator stand-in)
  - KMS CMK + alias/cross-account-cmk
  - Two ExternalId secrets (KMS-encrypted) with resource policies
  - OrchestratorRole (trusted by agent-runner) under path /orchestrator/
  - MemberRole-Tenant-A and MemberRole-Tenant-B under path /member/
      trust = orchestrator role arn AS PRINCIPAL
            + sts:ExternalId == per-tenant secret value
            + aws:SourceArn  == orchestrator role arn
  - Two tenant S3 buckets
  - SSM /harbor/... pointers so the verifier can find things by name
"""

import json
import secrets
import string

import boto3

ENDPOINT = "http://localstack:4566"
REGION = "us-east-1"
ACCOUNT_ID = "000000000000"

iam = boto3.client("iam", endpoint_url=ENDPOINT, region_name=REGION)
kms = boto3.client("kms", endpoint_url=ENDPOINT, region_name=REGION)
sm = boto3.client("secretsmanager", endpoint_url=ENDPOINT, region_name=REGION)
s3 = boto3.client("s3", endpoint_url=ENDPOINT, region_name=REGION)
ssm = boto3.client("ssm", endpoint_url=ENDPOINT, region_name=REGION)


def gen_external_id() -> str:
    alphabet = string.ascii_letters + string.digits + "_-"
    return "".join(secrets.choice(alphabet) for _ in range(48))


def main() -> None:
    ext_id_a = gen_external_id()
    ext_id_b = gen_external_id()
    assert ext_id_a != ext_id_b, "ExternalIds must differ between tenants"
    assert len(ext_id_a) >= 32 and len(ext_id_b) >= 32

    # ── pre-compute ARNs we'll cross-reference ───────────────────────────────
    orch_arn = f"arn:aws:iam::{ACCOUNT_ID}:role/orchestrator/OrchestratorRole"
    mem_a_arn = f"arn:aws:iam::{ACCOUNT_ID}:role/member/MemberRole-Tenant-A"
    mem_b_arn = f"arn:aws:iam::{ACCOUNT_ID}:role/member/MemberRole-Tenant-B"
    agent_arn = f"arn:aws:iam::{ACCOUNT_ID}:user/agent-runner"

    # ── IAM users ────────────────────────────────────────────────────────────
    iam.create_user(UserName="agent-runner")
    iam.create_user(UserName="attacker-runner")

    # ── KMS key for ExternalId secret encryption ─────────────────────────────
    key = kms.create_key(Description="cross-account ExternalId CMK")
    key_id = key["KeyMetadata"]["KeyId"]
    key_arn = key["KeyMetadata"]["Arn"]
    kms.create_alias(AliasName="alias/cross-account-cmk", TargetKeyId=key_id)

    # ── Secrets (KMS-encrypted) holding the per-tenant ExternalId values ────
    sec_a = sm.create_secret(
        Name="harbor/cross-account/external-id-tenant-a",
        KmsKeyId=key_arn,
        SecretString=ext_id_a,
    )
    sec_b = sm.create_secret(
        Name="harbor/cross-account/external-id-tenant-b",
        KmsKeyId=key_arn,
        SecretString=ext_id_b,
    )
    sec_a_arn = sec_a["ARN"]
    sec_b_arn = sec_b["ARN"]

    # ── OrchestratorRole , trusted by the agent-runner IAM user ─────────────
    orch_trust = {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {"AWS": agent_arn},
                "Action": "sts:AssumeRole",
            }
        ],
    }
    iam.create_role(
        RoleName="OrchestratorRole",
        Path="/orchestrator/",
        AssumeRolePolicyDocument=json.dumps(orch_trust),
    )

    # ── Member roles , trust requires ExternalId AND SourceArn ──────────────
    def member_trust(ext_id: str) -> dict:
        return {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {"AWS": orch_arn},
                    "Action": "sts:AssumeRole",
                    "Condition": {
                        "StringEquals": {"sts:ExternalId": ext_id},
                        "ArnEquals": {"aws:SourceArn": orch_arn},
                    },
                }
            ],
        }

    iam.create_role(
        RoleName="MemberRole-Tenant-A",
        Path="/member/",
        AssumeRolePolicyDocument=json.dumps(member_trust(ext_id_a)),
    )
    iam.create_role(
        RoleName="MemberRole-Tenant-B",
        Path="/member/",
        AssumeRolePolicyDocument=json.dumps(member_trust(ext_id_b)),
    )

    # ── Resource policies on each secret , orchestrator must be admitted ────
    def secret_resource_policy(secret_arn: str) -> dict:
        return {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "AllowOrchestratorRead",
                    "Effect": "Allow",
                    "Principal": {"AWS": orch_arn},
                    "Action": "secretsmanager:GetSecretValue",
                    "Resource": secret_arn,
                }
            ],
        }

    sm.put_resource_policy(
        SecretId=sec_a_arn,
        ResourcePolicy=json.dumps(secret_resource_policy(sec_a_arn)),
    )
    sm.put_resource_policy(
        SecretId=sec_b_arn,
        ResourcePolicy=json.dumps(secret_resource_policy(sec_b_arn)),
    )

    # ── Tenant buckets ──────────────────────────────────────────────────────
    s3.create_bucket(Bucket="tenant-a-bucket")
    s3.create_bucket(Bucket="tenant-b-bucket")

    # ── Identity policies (least privilege) ─────────────────────────────────
    orch_identity = {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AssumeMemberRoles",
                "Effect": "Allow",
                "Action": "sts:AssumeRole",
                "Resource": [mem_a_arn, mem_b_arn],
            },
            {
                "Sid": "ReadExternalIdSecrets",
                "Effect": "Allow",
                "Action": "secretsmanager:GetSecretValue",
                "Resource": [sec_a_arn, sec_b_arn],
            },
            {
                "Sid": "DecryptExternalIdSecrets",
                "Effect": "Allow",
                "Action": "kms:Decrypt",
                "Resource": key_arn,
            },
        ],
    }
    iam.put_role_policy(
        RoleName="OrchestratorRole",
        PolicyName="OrchestratorIdentityPolicy",
        PolicyDocument=json.dumps(orch_identity),
    )

    def member_identity(bucket: str) -> dict:
        return {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "TenantBucketObjectRW",
                    "Effect": "Allow",
                    "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
                    "Resource": f"arn:aws:s3:::{bucket}/*",
                },
                {
                    "Sid": "TenantBucketList",
                    "Effect": "Allow",
                    "Action": "s3:ListBucket",
                    "Resource": f"arn:aws:s3:::{bucket}",
                },
            ],
        }

    iam.put_role_policy(
        RoleName="MemberRole-Tenant-A",
        PolicyName="TenantABucketPolicy",
        PolicyDocument=json.dumps(member_identity("tenant-a-bucket")),
    )
    iam.put_role_policy(
        RoleName="MemberRole-Tenant-B",
        PolicyName="TenantBBucketPolicy",
        PolicyDocument=json.dumps(member_identity("tenant-b-bucket")),
    )

    # agent-runner needs identity-side allow to assume OrchestratorRole
    iam.put_user_policy(
        UserName="agent-runner",
        PolicyName="AssumeOrchestrator",
        PolicyDocument=json.dumps(
            {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Action": "sts:AssumeRole",
                        "Resource": orch_arn,
                    }
                ],
            }
        ),
    )

    # ── SSM pointers ────────────────────────────────────────────────────────
    for name, val in [
        ("/harbor/orchestrator/role-arn", orch_arn),
        ("/harbor/external-id-secret-arn-tenant-a", sec_a_arn),
        ("/harbor/external-id-secret-arn-tenant-b", sec_b_arn),
        ("/harbor/member-role-arn-tenant-a", mem_a_arn),
        ("/harbor/member-role-arn-tenant-b", mem_b_arn),
    ]:
        ssm.put_parameter(Name=name, Value=val, Type="String", Overwrite=True)

    print("setup complete")
    print(f"  OrchestratorRole       = {orch_arn}")
    print(f"  MemberRole-Tenant-A    = {mem_a_arn}")
    print(f"  MemberRole-Tenant-B    = {mem_b_arn}")
    print(f"  external-id secret A   = {sec_a_arn}")
    print(f"  external-id secret B   = {sec_b_arn}")
    print(f"  KMS CMK                = {key_arn}  (alias/cross-account-cmk)")


if __name__ == "__main__":
    main()

Run setup script

shell
python3 setup.py
output
setup complete
  OrchestratorRole       = arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  MemberRole-Tenant-A    = arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  MemberRole-Tenant-B    = arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  external-id secret A   = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZjFuYK
  external-id secret B   = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-QdKzye
  KMS CMK                = arn:aws:kms:us-east-1:000000000000:key/d3d5e268-2470-42e7-94af-794761d4cdbd  (alias/cross-account-cmk)

[stdout]
setup complete
  OrchestratorRole       = arn:aws:iam::000000000000:role/orchestrator/OrchestratorRole
  MemberRole-Tenant-A    = arn:aws:iam::000000000000:role/member/MemberRole-Tenant-A
  MemberRole-Tenant-B    = arn:aws:iam::000000000000:role/member/MemberRole-Tenant-B
  external-id secret A   = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-a-ZjFuYK
  external-id secret B   = arn:aws:secretsmanager:us-east-1:000000000000:secret:harbor/cross-account/external-id-tenant-b-QdKzye
  KMS CMK                = arn:aws:kms:us-east-1:000000000000:key/d3d5e268-2470-42e7-94af-794761d4cdbd  (alias/cross-account-cmk)
Setup ran clean. Now the verifier , does the happy path (assume → get secret → assume member → S3 put) and the policy-shape audit checks.
write · /app/verify.py
"""End-to-end verifier for the cross-account trust topology.

Two phases:
  1. happy path  , assume OrchestratorRole, fetch tenant-A ExternalId from
                   Secrets Manager, assume MemberRole-Tenant-A with that
                   ExternalId, then put an object into tenant-a-bucket.
  2. shape audit , read the trust + identity policy documents directly
                   and assert the SOC2 conditions an auditor would check.
"""

import json
import sys

import boto3

ENDPOINT = "http://localstack:4566"
REGION = "us-east-1"
ACCOUNT_ID = "000000000000"


def base_client(service: str, **overrides):
    kwargs = {"endpoint_url": ENDPOINT, "region_name": REGION, **overrides}
    return boto3.client(service, **kwargs)


def assumed_clients(creds: dict, services):
    out = {}
    for svc in services:
        out[svc] = base_client(
            svc,
            aws_access_key_id=creds["AccessKeyId"],
            aws_secret_access_key=creds["SecretAccessKey"],
            aws_session_token=creds["SessionToken"],
        )
    return out


# ── helpers for shape checks ─────────────────────────────────────────────────
def fail(msg: str):
    print(f"  FAIL: {msg}")
    fail.count += 1


fail.count = 0


def ok(msg: str):
    print(f"  ok:   {msg}")


def as_list(x):
    return x if isinstance(x, list) else [x]


def has_wildcard(value) -> bool:
    if isinstance(value, str):
        return value == "*"
    if isinstance(value, list):
        return any(v == "*" for v in value)
    return False


# ── phase 1: happy path ──────────────────────────────────────────────────────
def happy_path(ssm, iam):
    print("== phase 1: happy path ==")

    orch_arn = ssm.get_parameter(Name="/harbor/orchestrator/role-arn")["Parameter"]["Value"]
    sec_a_arn = ssm.get_parameter(Name="/harbor/external-id-secret-arn-tenant-a")["Parameter"]["Value"]
    mem_a_arn = ssm.get_parameter(Name="/harbor/member-role-arn-tenant-a")["Parameter"]["Value"]

    # Create access keys for agent-runner (the orchestrator's runtime identity).
    key = iam.create_access_key(UserName="agent-runner")["AccessKey"]
    user_sts = base_client(
        "sts",
        aws_access_key_id=key["AccessKeyId"],
        aws_secret_access_key=key["SecretAccessKey"],
    )

    # 1) agent-runner assumes OrchestratorRole.
    orch_creds = user_sts.assume_role(
        RoleArn=orch_arn, RoleSessionName="orchestrator-session"
    )["Credentials"]
    print(f"  assumed OrchestratorRole → AKID {orch_creds['AccessKeyId'][:8]}…")

    # 2) Orchestrator pulls the tenant-A ExternalId from Secrets Manager.
    orch = assumed_clients(orch_creds, ["secretsmanager", "sts"])
    ext_id = orch["secretsmanager"].get_secret_value(SecretId=sec_a_arn)["SecretString"]
    print(f"  fetched tenant-A ExternalId  (len={len(ext_id)})")

    # 3) Orchestrator assumes MemberRole-Tenant-A with the ExternalId.
    member_creds = orch["sts"].assume_role(
        RoleArn=mem_a_arn,
        RoleSessionName="tenant-a-session",
        ExternalId=ext_id,
    )["Credentials"]
    print(f"  assumed MemberRole-Tenant-A → AKID {member_creds['AccessKeyId'][:8]}…")

    # 4) Member writes an object to tenant-a-bucket.
    member = assumed_clients(member_creds, ["s3"])
    member["s3"].put_object(
        Bucket="tenant-a-bucket",
        Key="proof-of-life.txt",
        Body=b"happy-path-ok",
    )
    body = member["s3"].get_object(Bucket="tenant-a-bucket", Key="proof-of-life.txt")["Body"].read()
    assert body == b"happy-path-ok", f"unexpected body: {body!r}"
    print("  put + get on tenant-a-bucket → ok")
    print()


# ── phase 2: shape audit ─────────────────────────────────────────────────────
def shape_audit(iam, sm):
    print("== phase 2: policy-shape audit ==")

    orch_role = iam.get_role(RoleName="OrchestratorRole")["Role"]
    orch_arn = orch_role["Arn"]
    mem_a = iam.get_role(RoleName="MemberRole-Tenant-A")["Role"]
    mem_b = iam.get_role(RoleName="MemberRole-Tenant-B")["Role"]
    mem_a_arn = mem_a["Arn"]
    mem_b_arn = mem_b["Arn"]

    sec_a = sm.describe_secret(SecretId="harbor/cross-account/external-id-tenant-a")
    sec_b = sm.describe_secret(SecretId="harbor/cross-account/external-id-tenant-b")
    sec_a_arn = sec_a["ARN"]
    sec_b_arn = sec_b["ARN"]
    ext_a = sm.get_secret_value(SecretId=sec_a_arn)["SecretString"]
    ext_b = sm.get_secret_value(SecretId=sec_b_arn)["SecretString"]

    # ── path scoping ────────────────────────────────────────────────────────
    if orch_role["Path"] == "/orchestrator/":
        ok("OrchestratorRole sits under /orchestrator/")
    else:
        fail(f"OrchestratorRole path is {orch_role['Path']!r}, expected /orchestrator/")
    for r, name in ((mem_a, "MemberRole-Tenant-A"), (mem_b, "MemberRole-Tenant-B")):
        if r["Path"] == "/member/":
            ok(f"{name} sits under /member/")
        else:
            fail(f"{name} path is {r['Path']!r}, expected /member/")

    # ── ExternalIds differ ──────────────────────────────────────────────────
    if ext_a != ext_b:
        ok("tenant-A and tenant-B ExternalIds differ")
    else:
        fail("ExternalIds are identical across tenants , defeats the purpose")
    if len(ext_a) >= 32 and len(ext_b) >= 32:
        ok(f"both ExternalIds ≥ 32 chars (a={len(ext_a)}, b={len(ext_b)})")
    else:
        fail(f"ExternalId too short (a={len(ext_a)}, b={len(ext_b)})")

    # ── trust documents ─────────────────────────────────────────────────────
    def assert_member_trust(role, expected_ext_id, label):
        doc = role["AssumeRolePolicyDocument"]
        if isinstance(doc, str):
            doc = json.loads(doc)
        statements = as_list(doc["Statement"])
        if len(statements) != 1:
            fail(f"{label}: expected exactly 1 trust statement, got {len(statements)}")
            return
        stmt = statements[0]

        # principal = orchestrator role arn (not wildcard)
        principal = stmt.get("Principal")
        if principal == "*" or principal == {"AWS": "*"}:
            fail(f"{label}: trust principal is wildcard")
        elif isinstance(principal, dict) and principal.get("AWS") == orch_arn:
            ok(f"{label}: principal is OrchestratorRole arn (not *)")
        else:
            fail(f"{label}: principal is {principal!r}, expected {{'AWS': '{orch_arn}'}}")

        # action is exactly sts:AssumeRole, no wildcards
        action = stmt.get("Action")
        if action == "sts:AssumeRole" or action == ["sts:AssumeRole"]:
            ok(f"{label}: action is exactly sts:AssumeRole")
        else:
            fail(f"{label}: action is {action!r}, expected exactly sts:AssumeRole")

        cond = stmt.get("Condition") or {}

        # ExternalId guard against third-party impersonation
        ext = (cond.get("StringEquals") or {}).get("sts:ExternalId")
        if ext == expected_ext_id:
            ok(f"{label}: StringEquals[sts:ExternalId] matches secret value")
        else:
            fail(f"{label}: sts:ExternalId condition missing or wrong (got {ext!r})")

        # SourceArn guard against confused-deputy
        src = None
        for op in ("ArnEquals", "ArnLike"):
            v = (cond.get(op) or {}).get("aws:SourceArn")
            if v is not None:
                src = (op, v)
                break
        if src and src[1] == orch_arn:
            ok(f"{label}: {src[0]}[aws:SourceArn] equals OrchestratorRole arn")
        else:
            fail(f"{label}: aws:SourceArn condition missing or not equal to orchestrator arn (got {src!r})")

    assert_member_trust(mem_a, ext_a, "MemberRole-Tenant-A trust")
    assert_member_trust(mem_b, ext_b, "MemberRole-Tenant-B trust")

    # ── orchestrator identity policy: explicit member arns, no wildcards ───
    orch_id = json.loads(
        iam.get_role_policy(RoleName="OrchestratorRole", PolicyName="OrchestratorIdentityPolicy")[
            "PolicyDocument"
        ]
        if isinstance(
            iam.get_role_policy(RoleName="OrchestratorRole", PolicyName="OrchestratorIdentityPolicy")[
                "PolicyDocument"
            ],
            str,
        )
        else json.dumps(
            iam.get_role_policy(RoleName="OrchestratorRole", PolicyName="OrchestratorIdentityPolicy")[
                "PolicyDocument"
            ]
        )
    )
    assume_stmt = next(
        (s for s in as_list(orch_id["Statement"]) if "sts:AssumeRole" in as_list(s["Action"])),
        None,
    )
    if assume_stmt is None:
        fail("orchestrator identity: no sts:AssumeRole statement")
    else:
        res = as_list(assume_stmt["Resource"])
        if has_wildcard(res):
            fail(f"orchestrator identity: sts:AssumeRole has wildcard resource ({res!r})")
        elif set(res) == {mem_a_arn, mem_b_arn}:
            ok("orchestrator identity: sts:AssumeRole enumerates exactly the two member arns")
        else:
            fail(f"orchestrator identity: AssumeRole resources are {res!r}, expected {{{mem_a_arn}, {mem_b_arn}}}")

    secret_stmt = next(
        (
            s
            for s in as_list(orch_id["Statement"])
            if "secretsmanager:GetSecretValue" in as_list(s["Action"])
        ),
        None,
    )
    if secret_stmt is None:
        fail("orchestrator identity: no secretsmanager:GetSecretValue statement")
    else:
        res = as_list(secret_stmt["Resource"])
        if has_wildcard(res):
            fail("orchestrator identity: GetSecretValue has wildcard resource")
        elif set(res) == {sec_a_arn, sec_b_arn}:
            ok("orchestrator identity: GetSecretValue scoped to the two known secret arns")
        else:
            fail(f"orchestrator identity: GetSecretValue resources are {res!r}")

    # ── member identity policies: scoped to own bucket only ────────────────
    def assert_member_identity(role_name, policy_name, own_bucket, other_bucket):
        doc = iam.get_role_policy(RoleName=role_name, PolicyName=policy_name)["PolicyDocument"]
        if isinstance(doc, str):
            doc = json.loads(doc)
        own_arns = {f"arn:aws:s3:::{own_bucket}", f"arn:aws:s3:::{own_bucket}/*"}
        seen = set()
        for stmt in as_list(doc["Statement"]):
            actions = as_list(stmt["Action"])
            resources = as_list(stmt["Resource"])
            if has_wildcard(actions):
                fail(f"{role_name}: Action wildcard present")
            if has_wildcard(resources):
                fail(f"{role_name}: Resource wildcard present")
            for r in resources:
                if other_bucket in r:
                    fail(f"{role_name}: references other tenant's bucket: {r}")
                seen.add(r)
        if seen <= own_arns:
            ok(f"{role_name}: identity policy references only {own_bucket} arns")
        else:
            fail(f"{role_name}: unexpected resources {seen - own_arns}")

    assert_member_identity("MemberRole-Tenant-A", "TenantABucketPolicy", "tenant-a-bucket", "tenant-b-bucket")
    assert_member_identity("MemberRole-Tenant-B", "TenantBBucketPolicy", "tenant-b-bucket", "tenant-a-bucket")

    # ── secret resource policies admit the orchestrator ────────────────────
    def assert_secret_resource_policy(secret_arn, label):
        rp = sm.get_resource_policy(SecretId=secret_arn).get("ResourcePolicy")
        if not rp:
            fail(f"{label}: no resource policy attached")
            return
        doc = json.loads(rp) if isinstance(rp, str) else rp
        for stmt in as_list(doc["Statement"]):
            principal = stmt.get("Principal", {})
            principals = as_list(principal.get("AWS", [])) if isinstance(principal, dict) else []
            if (
                stmt.get("Effect") == "Allow"
                and "secretsmanager:GetSecretValue" in as_list(stmt["Action"])
                and orch_arn in principals
            ):
                ok(f"{label}: resource policy admits OrchestratorRole for GetSecretValue")
                return
        fail(f"{label}: resource policy does not name OrchestratorRole as principal for GetSecretValue")

    assert_secret_resource_policy(sec_a_arn, "tenant-A secret")
    assert_secret_resource_policy(sec_b_arn, "tenant-B secret")

    # ── KMS scoping in orchestrator policy ─────────────────────────────────
    kms_stmt = next(
        (s for s in as_list(orch_id["Statement"]) if any(a.startswith("kms:") for a in as_list(s["Action"]))),
        None,
    )
    if kms_stmt is None:
        fail("orchestrator identity: no kms statement (needed to decrypt secrets)")
    else:
        if has_wildcard(as_list(kms_stmt["Resource"])):
            fail("orchestrator identity: kms statement has wildcard resource")
        elif has_wildcard(as_list(kms_stmt["Action"])):
            fail("orchestrator identity: kms statement has wildcard action")
        else:
            ok("orchestrator identity: kms statement is scoped (no * action, no * resource)")

    print()


def main():
    ssm = base_client("ssm")
    iam = base_client("iam")
    sm = base_client("secretsmanager")

    happy_path(ssm, iam)
    shape_audit(iam, sm)

    if fail.count:
        print(f"VERIFIER: {fail.count} failure(s)")
        sys.exit(1)
    print("VERIFIER: all checks passed")


if __name__ == "__main__":
    main()

Run verifier (happy path + shape audit)

shell
python3 verify.py
output
== phase 1: happy path ==
  assumed OrchestratorRole → AKID LSIAQAAA…
  fetched tenant-A ExternalId  (len=48)
  assumed MemberRole-Tenant-A → AKID LSIAQAAA…
  put + get on tenant-a-bucket → ok

== phase 2: policy-shape audit ==
  ok:   OrchestratorRole sits under /orchestrator/
  ok:   MemberRole-Tenant-A sits under /member/
  ok:   MemberRole-Tenant-B sits under /member/
  ok:   tenant-A and tenant-B ExternalIds differ
  ok:   both ExternalIds ≥ 32 chars (a=48, b=48)
  ok:   MemberRole-Tenant-A trust: principal is OrchestratorRole arn (not *)
  ok:   MemberRole-Tenant-A trust: action is exactly sts:AssumeRole
  ok:   MemberRole-Tenant-A trust: StringEquals[sts:ExternalId] matches secret value
  ok:   MemberRole-Tenant-A trust: ArnEquals[aws:SourceArn] equals OrchestratorRole arn
  ok:   MemberRole-Tenant-B trust: principal is OrchestratorRole arn (not *)
  ok:   MemberRole-Tenant-B trust: action is exactly sts:AssumeRole
  ok:   MemberRole-Tenant-B trust: StringEquals[sts:ExternalId] matches secret value
  ok:   MemberRole-Tenant-B trust: ArnEquals[aws:SourceArn] equals OrchestratorRole arn
  ok:   orchestrator identity: sts:AssumeRole enumerates exactly the two member arns
  ok:   orchestrator identity: GetSecretValue scoped to the two known secret arns
  ok:   MemberRole-Tenant-A: identity policy references only tenant-a-bucket arns
  ok:   MemberRole-Tenant-B: identity policy references only tenant-b-bucket arns
  ok:   tenant-A secret: resource policy admits OrchestratorRole for GetSecretValue
  ok:   tenant-B secret: resource policy admits OrchestratorRole for GetSecretValue
  ok:   orchestrator identity: kms statement is scoped (no * action, no * resource)

VERIFIER: all checks passed

[stdout]
== phase 1: happy path ==
  assumed OrchestratorRole → AKID LSIAQAAA…
  fetched tenant-A ExternalId  (len=48)
  assumed MemberRole-Tenant-A → AKID LSIAQAAA…
  put + get on tenant-a-bucket → ok

== phase 2: policy-shape audit ==
  ok:   OrchestratorRole sits under /orchestrator/
  ok:   MemberRole-Tenant-A sits under /member/
  ok:   MemberRole-Tenant-B sits under /member/
  ok:   tenant-A and tenant-B ExternalIds differ
  ok:   both ExternalIds ≥ 32 chars (a=48, b=48)
  ok:   MemberRole-Tenant-A trust: principal is OrchestratorRole arn (not *)
  ok:   MemberRole-Tenant-A trust: action is exactly sts:AssumeRole
  ok:   MemberRole-Tenant-A trust: StringEquals[sts:ExternalId] matches secret value
  ok:   MemberRole-Tenant-A trust: ArnEquals[aws:SourceArn] equals OrchestratorRole arn
  ok:   MemberRole-Tenant-B trust: principal is OrchestratorRole arn (not *)
  ok:   MemberRole-Tenant-B trust: action is exactly sts:AssumeRole
  ok:   MemberRole-Tenant-B trust: StringEquals[sts:ExternalId] matches secret value
  ok:   MemberRole-Tenant-B trust: ArnEquals[aws:SourceArn] equals OrchestratorRole arn
  ok:   orchestrator identity: sts:AssumeRole enumerates exactly the two member arns
  ok:   orchestrator identity: GetSecretValue scoped to the two known secret arns
  ok:   MemberRole-Tenant-A: identity policy references only tenant-a-bucket arns
  ok:   MemberRole-Tenant-B: identity policy references only tenant-b-bucket arns
  ok:   tenant-A secret: resource policy admits OrchestratorRole for GetSecretValue
  ok:   tenant-B secret: resource policy admits OrchestratorRole for GetSecretValue
  ok:   orchestrator identity: kms statement is scoped (no * action, no * resource)

VERIFIER: all checks passed
Both phases are green. **Built**: - `setup.py` , provisions IAM users, KMS CMK + alias, two KMS-encrypted ExternalId secrets (distinct 48-char random values), `OrchestratorRole` (path `/orchestrator/`, trusted by `agent-runner`), `MemberRole-Tenant-A/B` (path `/member/`), tenant buckets, identity policies, secret resource policies, and SSM `/harbor/...` pointers. - `verify.py` , runs the happy path (assume orchestrator → fetch ExternalId → assume member with `--external-id` → S3 put/get) then audits policy-document shape. **Defenses in place on each member trust**: - `Principal.AWS = OrchestratorRole arn` (not `*`) - `Action = sts:AssumeRole` only - `StringEquals` on `sts:ExternalId` = per-tenant secret value (third-party impersonation block) - `ArnEquals` on `aws:SourceArn` = OrchestratorRole arn (confused-deputy block) **Least-privilege confirmed**: - Orchestrator's `sts:AssumeRole` enumerates the two member ARNs; `GetSecretValue` scoped to the two secret ARNs; `kms:Decrypt` scoped to the CMK. - Each member role's S3 policy references only its own bucket's two ARNs (no `*` action, no `*` resource, no cross-tenant arn). - Both secrets carry resource policies naming OrchestratorRole as `Principal.AWS` for `GetSecretValue`. - Tenant ExternalIds differ and are ≥ 32 chars.
Final diff , what the agent changed

No file changes, the agent left the workspace untouched.

Verifier output , the proof behind the reward
tests/test.sh
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /root/.cache/uv/archive-v0/AFnzOgs8fbvLVcTuPVst8/bin/python
cachedir: .pytest_cache
rootdir: /tests
plugins: json-ctrf-0.3.5
collecting ... collected 20 items

../tests/test_state.py::test_01_orchestrator_role_exists_at_orchestrator_path PASSED [  5%]
../tests/test_state.py::test_02_member_roles_exist_at_member_path PASSED [ 10%]
../tests/test_state.py::test_03_external_id_secrets_exist_with_min_length PASSED [ 15%]
../tests/test_state.py::test_04_both_tenant_buckets_exist PASSED         [ 20%]
../tests/test_state.py::test_05_ssm_pointers_resolve PASSED              [ 25%]
../tests/test_state.py::test_06_member_trusts_have_externalid_stringequals_matching_secret PASSED [ 30%]
../tests/test_state.py::test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn PASSED [ 35%]
../tests/test_state.py::test_08_member_trusts_action_is_assumerole_only PASSED [ 40%]
../tests/test_state.py::test_09_member_trusts_principal_is_orchestrator_not_wildcard PASSED [ 45%]
../tests/test_state.py::test_10_externalids_differ_per_tenant PASSED     [ 50%]
../tests/test_state.py::test_11_orchestrator_identity_lists_member_arns_explicitly_no_wildcard PASSED [ 55%]
../tests/test_state.py::test_12_orchestrator_secret_get_scoped_to_two_secret_arns PASSED [ 60%]
../tests/test_state.py::test_13_orchestrator_kms_decrypt_scoped_to_cmk_arn PASSED [ 65%]
../tests/test_state.py::test_14_member_inline_policies_scope_to_own_tenant_bucket_only PASSED [ 70%]
../tests/test_state.py::test_15_secret_resource_policies_admit_orchestrator PASSED [ 75%]
../tests/test_state.py::test_16_kms_key_policy_has_account_root_statement PASSED [ 80%]
../tests/test_state.py::test_17_no_admin_managed_policies_attached_to_any_role PASSED [ 85%]
../tests/test_state.py::test_18_e2e_assume_with_externalid_then_put_object_succeeds PASSED [ 90%]
../tests/test_state.py::test_19_member_a_inline_does_not_reference_tenant_b_bucket PASSED [ 95%]
../tests/test_state.py::test_20_orchestrator_has_no_wildcard_assume_role_resource PASSED [100%]

=============================== warnings summary ===============================
test_state.py: 56 warnings
  /root/.cache/uv/archive-v0/AFnzOgs8fbvLVcTuPVst8/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== PASSES ====================================
=========================== short test summary info ============================
PASSED ../tests/test_state.py::test_01_orchestrator_role_exists_at_orchestrator_path
PASSED ../tests/test_state.py::test_02_member_roles_exist_at_member_path
PASSED ../tests/test_state.py::test_03_external_id_secrets_exist_with_min_length
PASSED ../tests/test_state.py::test_04_both_tenant_buckets_exist
PASSED ../tests/test_state.py::test_05_ssm_pointers_resolve
PASSED ../tests/test_state.py::test_06_member_trusts_have_externalid_stringequals_matching_secret
PASSED ../tests/test_state.py::test_07_member_trusts_have_sourcearn_equal_to_orchestrator_arn
PASSED ../tests/test_state.py::test_08_member_trusts_action_is_assumerole_only
PASSED ../tests/test_state.py::test_09_member_trusts_principal_is_orchestrator_not_wildcard
PASSED ../tests/test_state.py::test_10_externalids_differ_per_tenant
PASSED ../tests/test_state.py::test_11_orchestrator_identity_lists_member_arns_explicitly_no_wildcard
PASSED ../tests/test_state.py::test_12_orchestrator_secret_get_scoped_to_two_secret_arns
PASSED ../tests/test_state.py::test_13_orchestrator_kms_decrypt_scoped_to_cmk_arn
PASSED ../tests/test_state.py::test_14_member_inline_policies_scope_to_own_tenant_bucket_only
PASSED ../tests/test_state.py::test_15_secret_resource_policies_admit_orchestrator
PASSED ../tests/test_state.py::test_16_kms_key_policy_has_account_root_statement
PASSED ../tests/test_state.py::test_17_no_admin_managed_policies_attached_to_any_role
PASSED ../tests/test_state.py::test_18_e2e_assume_with_externalid_then_put_object_succeeds
PASSED ../tests/test_state.py::test_19_member_a_inline_does_not_reference_tenant_b_bucket
PASSED ../tests/test_state.py::test_20_orchestrator_has_no_wildcard_assume_role_resource
======================= 20 passed, 56 warnings in 0.39s ========================

Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_50aae8d21ad14722. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.

Trial trial_50aae8d21ad14722 · verifier authoritative; classifier explanatory.