ci(sandbox): run the egressDeny guard weekly instead of never

[ItamarZand88]

Summary

We publish egressDeny as an enforced property, and its only automated check is #[ignore]d and
named by no workflow — so it runs when someone remembers. This adds a weekly job that builds the
probe image, deploys it, runs that check against real AWS, and removes everything afterwards.

Step-by-step flow when the schedule fires:

1. Build the aarch64 agent, write an image bundle, and render the probe stack with the shipped
   CloudFormation emitter — so the guard tests the template deployments actually get.
2. Deploy, then start two MicroVMs from one image: one with the egress connector, one without.
   Both try to reach a public address. ← the check lives here
3. Tear down the stack, and a second job removes anything a cancelled run left behind.

This PR changes a security property verified by hand into one verified on a schedule.

What was broken

a_denied_sandbox_cannot_reach_the_internet_and_an_open_one_can asserts the right thing and never
executes: it is #[ignore]d, and PROBE_IMAGE_NAME / PROBE_CONNECTOR_ARN appear in no workflow.
If AWS changed how a network connector governs egress, nothing would tell us and we would keep
reporting egressDeny: true.

What I did

• Added a weekly workflow that provisions the probe image and connector, runs the check, and tears
  down. Weekly rather than per-PR: it needs a real account and guards against an AWS-side change,
  not a change in this repository.
• Kept the test #[ignore]d and named it exactly from the workflow, with --no-tests=fail. A gate
  that skipped when the image was missing would report a provisioning failure as a pass, which is
  the hole this exists to close; with the filter, a name that matches nothing fails the job.
• Made the assertions name egressDeny, so an on-call reader knows what regressed without opening
  the test.
• Added two teardown tests: one reclaims the probe image's sessions and versions, one asserts the
  image is gone once its stack is. Both are best-effort per resource and report everything they
  could not clear, so one stubborn MicroVM cannot abandon the rest.
• Put the cleanup in a dependent job as well as a trailing step. A cancelled or timed-out job gives
  always() only whatever window the runner allows, which does not reliably cover deleting a VPC.

Files touched

• .github/workflows/egress-deny-guard.yml — the weekly job and the cleanup job.
• crates/alien-aws-clients/src/aws/lambda_microvms.rs — assertion wording, plus
  reclaim_the_probe_image and the_probe_image_is_gone.
• crates/alien-build/examples/sandbox-bundle.rs — a CLI over sandbox_bundle::write_bundle, so
  the bundle comes from the shipped packaging code rather than a Dockerfile pasted into YAML.
• scripts/egress-deny-guard-teardown.sh — reclaim, delete, and verify the stack is gone.
• scripts/README.md — an entry for it, and the two preconditions that live outside this repo.

The teardown is the part worth reviewing

The check itself is short. Nearly all the risk is in not leaking a VPC, and in never reporting a
clean teardown over something that survived. Three things shape it:

• A failed reclaim keeps the stack. A delete on a held image is accepted and removes nothing
  (delete_images_versions_first), so deleting the stack anyway would reach DELETE_COMPLETE
  while MicroVMs kept billing — and every survival check keys on the stack existing, so nothing
  would look again. The stack is the handle back to the image; it stays up, loudly, and the next
  run retries.
• An inconclusive AWS call is never read as "absent". describe-stacks failing on expired
  credentials or a throttle exits 1 rather than reporting nothing to clean, matching how
  cleanup-aws-e2e-resources.sh draws the same distinction.
• The image is resolved two ways. Outputs are absent in exactly the stack states this clears,
  so it falls back to list-stack-resources. Only a stack that genuinely declares no image may
  leave that empty.

How I tested

• Rendered the probe stack locally with the workflow's exact input and read the result: its
  AWS::Lambda::NetworkConnector resolves SecurityGroupIds to ProbeEgressSecurityGroup, whose
  only egress rule is CidrIp: 127.0.0.1/32. The other 0.0.0.0/0 hits are two route-table
  default routes and a separate workload security group the connector does not reference.
• Drove the teardown through a stubbed aws/cargo on PATH across nine paths: absent stack,
  inconclusive describe-stacks, happy path via outputs, empty outputs falling back to
  list-stack-resources, reclaim failure, image-still-present failure, delete failure, outputs
  carrying no sandbox resource, and both queries returning None. No path exits 0 with a resource
  surviving; none exits non-zero when everything is clean.
• Built and ran the bundle example, unzipped the result, and docker build --platform linux/arm64 on amazonlinux:2023 succeeded — uid 60000 exists, /sandbox is drwx------, and
  /usr/bin/curl (which the probe invokes) is present.
• shellcheck clean, actionlint clean apart from the depot-* runner label it does not know,
  cargo clippy -p alien-aws-clients --lib --tests and cargo fmt --check clean.
• Anything I couldn't test: none of it has run against AWS. Two first-run risks, both failing
  red rather than green: the_probe_image_is_gone assumes a deleted image answers 404, and its
  expect_err says to widen the check if AWS returns a terminal state instead; and
  public.ecr.aws/amazonlinux/amazonlinux:2023 was validated against Docker Hub's image, since
  public.ecr.aws was unreachable from here.

I also ran a security review on the diff. What it checked:

• Can a secret reach the public Actions log? No — every secret is interpolated verbatim, never
  transformed, so GitHub's literal auto-mask covers it.
• Can the target account id leak? No — ::add-mask:: is registered before any step that can print
  it, including CloudFormation's ValidationError and waiter-failure paths, and the cleanup job
  sets mask-aws-account-id.
• Can the session endpoint leak? It could: --no-capture streamed a passing run's output live, and
  a transport error carries the full exec URL. Dropped the flag — nextest prints failing output
  anyway.
• Does the second credential assume inherit an expired ambient session? No — it sets
  unset-current-credentials, and role-duration-seconds covers the job budget rather than
  expiring inside it.
• Can empty credentials be mistaken for configured ones? No — a workflow step reading an
  unresolved expression passes "", so client() now rejects empty as unset rather than signing
  with it and surfacing a 403.

Nothing turned up.

Before this can run

An operator has to create the egress-deny-guard environment, set EGRESS_GUARD_AWS_ROLE_ARN,
EGRESS_GUARD_ARTIFACT_BUCKET and EGRESS_GUARD_AWS_REGION, and grant the role the MicroVMs
actions including lambda:ListMicrovms. The role needs MaxSessionDuration of at least 5400
seconds. A lifecycle rule expiring the bucket's egress-deny-guard/ prefix is what reclaims a
bundle from a cancelled run — nothing in this diff can.

[greptile-apps]

Greptile Summary

The PR adds a scheduled AWS guard for sandbox egress denial and a defensive cleanup path for the temporary MicroVM infrastructure.

• Builds and deploys a probe image through the shipped packaging and CloudFormation paths.
• Compares denied and unrestricted MicroVM internet access.
• Reclaims MicroVMs and image versions before deleting the stack, with an independent cleanup job for interrupted runs.
• Pins all third-party workflow actions to full commit SHAs.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/workflows/egress-deny-guard.yml	Adds the pinned, weekly AWS deployment, validation, and independent cleanup workflow.
crates/alien-aws-clients/src/aws/lambda_microvms.rs	Adds the live egress assertion and resource-reclamation checks, including polling asynchronous MicroVM termination before version deletion.
crates/alien-build/examples/sandbox-bundle.rs	Adds a small CLI wrapper around the shipped sandbox bundle writer.
scripts/egress-deny-guard-teardown.sh	Adds fail-closed teardown that reclaims image dependencies, deletes the stack, and verifies absence.
scripts/README.md	Documents the guard teardown script and its external AWS prerequisites.

Sequence Diagram

sequenceDiagram
    participant Schedule as Weekly schedule
    participant Build as Build and render
    participant AWS as AWS CloudFormation/MicroVMs
    participant Guard as egressDeny test
    participant Cleanup as Teardown job
    Schedule->>Build: Build agent and image bundle
    Build->>AWS: Deploy probe stack
    AWS-->>Guard: Image and connector identifiers
    Guard->>AWS: Start denied and open MicroVMs
    Guard->>Guard: Compare internet reachability
    Guard->>Cleanup: Run teardown
    Cleanup->>AWS: Terminate MicroVMs
    Cleanup->>AWS: Delete image versions and stack
    Cleanup->>AWS: Verify image and stack are absent

Loading

_{Reviews (2): Last reviewed commit: "ci(sandbox): run the egressDeny guard we..." | Re-trigger Greptile}