[Bug] 2-networking-a-fedramp: the NVA cluster runs on Spot VMs with automatic_restart disabled, so a preemption removes the organization's only internet egress path — the IL5 stage runs the equivalent appliances on-demand

[JohnHales]

Bug Description

The FedRAMP High / Moderate networking stage provisions its Network Virtual Appliance cluster on Spot instances:

• nva.tf:83-84 — the nva-template options block sets spot = true and termination_action = "STOP" (machine type n2d-standard-2, nva.tf:57).
• Through modules/compute-vm/main.tf:294-298, that resolves on the instance template to provisioning_model = "SPOT", preemptible = true, instance_termination_action = "STOP" and — because the module computes automatic_restart = !var.options.spot — automatic_restart = false.
• nva.tf:107-118 — nva-mig, target_size = 2, autohealing on a TCP:22 health check with initial_delay_sec = 30.
• variables.tf:161-168 — regions is typed object({ primary = string }), so the for_each = var.regions on the template and the MIG (nva.tf:52,108) resolves to exactly one element. The cluster is two Spot instances in one regional MIG in every deployment, not only small ones.

Those two instances are the organization's only route to the internet, and that stays true however large the deployment gets. Workload subnets are free-form in region — subnets is typed map(list(object({ ... region = string ... }))) (variables.tf:240-262) — but every part of the egress path is pinned to the single primary region: the NVA template, the NVA MIG and both internal load balancers key off var.regions (nva.tf:52,108,127,156), and Cloud NAT is a single dmz-nat-primary at var.regions.primary (net-vdss.tf:122-130). Both ILB forwarding rules set global_access = true (nva.tf:135,164), so subnets outside the primary region reach the primary ILB and hairpin their egress cross-region into those same two appliances.

nva.tf:184-190 then creates default-route-nva — dest_range = "0.0.0.0/0", next_hop_ilb pointing at the primary region's vdss ILB — so a multi-region deployment and a single-region one have the identical choke point. While the NVAs are down the route still resolves to the ILB and traffic blackholes: no internet egress anywhere in the organization, in any region, and no inspected path. The larger the deployment, the more it is concentrated on two reclaimable instances.

Spot capacity is reclaimed on 30 seconds' notice whenever Compute Engine needs it back, and reclamation is driven by regional capacity pressure rather than by anything per-instance — so the two instances are not the independent failure domains that target_size = 2 suggests. With termination_action = "STOP" and automatic_restart = false, a preempted NVA stays stopped; recovery depends entirely on MIG autohealing repairing it, and that repair has to obtain Spot capacity too. If the region is short of it, the cluster cannot come back, and the outage lasts as long as the shortage does.

The IL5 option makes the opposite choice for the same role — ngfw.tf:203-206:

scheduling {
  automatic_restart   = true
  on_host_maintenance = "MIGRATE"
}

On-demand, restarted automatically, live-migrated on host maintenance. Its MIG also rolls updates with max_unavailable = 0 and min_ready_sec = 300 (ngfw.tf:288-296) — that stage treats the appliance tier as something that must not lose capacity, while the FedRAMP High stage puts the same tier on instances that can be reclaimed at any moment.

Nothing in the repository records the choice. Neither the stage README nor docs/tdd.md mentions Spot, and git log -S 'spot' -- fast/stages-aw/2-networking-a-fedramp-high/nva.tf returns a single commit — an unrelated upgrade of the project module — so the setting has never been introduced or changed by a commit that explains it. An operator reading the stage has no way to tell whether it is deliberate.

Environment and Deployment Context

• Stellar Engine Version/Commit: v2.13.0 (8f5b67a6); unchanged at v3.0.0 (f64ce6cd), where the directory is renamed 2-networking-a-fedramp and the same spot = true / termination_action = "STOP" pair sits at nva.tf:83-84. Because v3.0.0 folds FedRAMP Moderate into this stage, the Spot NVA cluster is now the shipped pattern for both regimes.
• Deployment Type:
  • US Region Restricted (e.g., Access Policy constraint)
  • FedRAMP Medium
  • FedRAMP High
  • FedRAMP Moderate
  • DoD IL4
  • DoD IL5
  • Stand-alone / Custom
• FAST Stage (if applicable): Stage 2 (Networking) — the a-fedramp option
  • Stage 0 (Bootstrap)
  • Stage 1 (Resource Management)
  • Stage 2 (Networking)
  • Stage 3 (Security)

Steps to Reproduce

1. Deploy fast/stages-aw/2-networking-a-fedramp (FedRAMP High or FedRAMP Moderate).
2. Read the provisioning model off the running appliances:
   gcloud compute instances list --project <prefix>-net-vdss-host --filter='name~nva' --format='table(name,zone,status,scheduling.provisioningModel,scheduling.instanceTerminationAction,scheduling.automaticRestart)'
   Every NVA reports SPOT / STOP / False.
3. Run the same command against a 2-networking-b-il5-ngfw deployment for comparison: its appliances report STANDARD / True.
4. To observe the impact without waiting for a preemption, stop both NVA instances and attempt any outbound connection from a spoke VM. DNS still resolves (the metadata resolver needs no egress); all real egress times out until the instances are running again.

Expected Behavior

The appliance tier that carries the organization's only egress and inspection path runs on standard (on-demand) instances by default, as the IL5 option's does — or, if Spot is a deliberate cost decision, it is exposed as a stage variable that defaults to standard, and the stage README states the availability trade-off so an operator can make the choice knowingly.

Actual Behavior

spot = true is hardcoded in the instance template with no variable and no documentation, and the derived automatic_restart = false means a preempted appliance is not brought back by Compute Engine at all. Verified in a built FedRAMP High organization: the NVA instances report provisioningModel: SPOT, and the organization's billing for 2026-08-14 → 2026-08-20 charges them under the Spot N2D core and RAM SKUs.

Relevant Logs and Errors

None. The apply succeeds and emits no warning — which is the substance of the report: nothing in the plan, the apply output or the stage documentation surfaces that the egress path is preemptible.

Additional Context

We have not observed a preemption-caused outage in our own deployment; this is reported from the configuration and the failure mode it implies, not from an incident. That is deliberate — the point is that the exposure exists by default and is invisible to whoever inherits it.

Two further multi-region observations, recorded here rather than filed separately. First, the NVA's trusted-side return routes are generated from a hardcoded "${var.regions.primary}/..." subnet lookup (nva.tf:30-32), so a workload subnet outside the primary region receives no return route on the appliance — the failure mode reported in #146, along its region dimension rather than its subnet-index one. Second, the for_each = var.regions idiom reads as though written for a multi-region map: were the regions type widened to add a secondary, the NVA template, the MIG and both ILBs would fan out on their own, but dmz-nat-primary and google_compute_route.default are pinned to var.regions.primary and would not follow — the second region's appliances would be built and never used.

Related: the same file's KMS gap is #193. Both are cases where the FedRAMP High networking stage and the IL5 stage diverge on the appliance tier without the divergence being recorded anywhere.