Skip to content

fix(sandbox): harden sandbox control plane - #676

Merged
xiaods merged 2 commits into
mainfrom
fix/sandbox-security-hardening
Nov 16, 2026
Merged

xiaods merged 2 commits into
mainfrom
fix/sandbox-security-hardening

Conversation

@xiaods

@xiaods xiaods commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Summary

Applies the security hardening from docs/security/sandbox-security-review.md (added in this PR). All changes are backward compatible: existing API keys are unbound and keep the legacy shared-credential behaviour.

Findings fixed

ID Severity Fix
F1 High Opt-in tenant isolation. apikey.Record gains an optional tenant. The gateway derives the caller's tenant from the client-cert CommonName (API key name) and enforces it on every session-scoped RPC. The E2B control plane enforces the same binding.
F2 High Certificate revocation is persisted (sandbox-revoked.json), loaded on start, and reconciled against the active key set on every load — including the first load after a restart (a key deleted while the gateway was down is now revoked).
F3 High Login now checks revocation, so a revoked certificate cannot re-enroll and mint a fresh one.
F4 Medium The rate limiter keys buckets on the authenticated identity (cert CN / token hash) instead of the client-supplied x-sandbox-tenant header; loopback gets its own bucket.
F5 Medium Warm-pool pods now deny ingress except kubelet TCP probes (the "deny-all" policy only declared egress, leaving sandboxd :2024 reachable from any pod).
F6 Medium Session / sub-agent / background-run / approval IDs are crypto-random instead of timestamp-derived.

Tenant isolation details

  • Server.apiKeyTenants is reloaded from the Secret with the keys and swapped atomically.
  • The auth interceptor stashes the caller's tenant in the request context.
  • getSession / getPodIP return NotFound on a tenant mismatch (not PermissionDenied) so a cross-tenant caller cannot learn that a session exists.
  • listSessions filters; createSessionWithTTL stamps the bound tenant or rejects an explicit mismatch.
  • Expose registry (ExposeService fast path, UnexposeService, ListExposed) and ConfirmAction are tenant-checked.
  • E2B: SecretKeySet.Tenants feeds ReplaceAPIKeysWithTenants; create stamps the tenant, findLive/handleConnect/listLiveSessions scope to it, reacquireByKey refuses another tenant's metadata.name.

Usage

# bind a key to a tenant (existing keys are unbound until recreated)
k8e sandbox-apikey create acme-agent --tenant acme
k8e sandbox-apikey list          # shows the tenant

# clients then pass --tenant acme (or K8E_SANDBOX_TENANT=acme)

Tests

New/updated tests: pkg/sandbox/apikey/store_test.go, pkg/sandboxmatrix/grpc/authz_test.go, pkg/sandboxmatrix/grpc/cert_store_test.go, pkg/sandboxmatrix/grpc/cert_test.go, pkg/sandboxmatrix/grpc/orchestrator_test.go, pkg/sandbox/e2b/authz_test.go, pkg/sandbox/e2b/protocol_test.go, pkg/sandboxmatrix/ratelimit/ratelimit_test.go.

Validation

go build ./...
go vet   ./pkg/sandboxmatrix/... ./pkg/sandbox/... ./pkg/server/...
go test  ./pkg/sandboxmatrix/... ./pkg/sandbox/... ./pkg/sandboxmcp/... \
         ./pkg/sandboxcli/... ./pkg/server/... -count=1

All pass.

Follow-ups (documented in the report, not in this PR)

  • F7: secret_refs is still a name-based deny-list; make it a tenant-labelled allowlist.
  • F8: require a credential on loopback if the gateway host is shared.
  • F9: add AutomountServiceAccountToken: false to SandboxPodSpec.
  • F10/F11: tighten sanitizePipPackage; require an explicit K8E_E2B_SIGNING_SECRET.
  • Scope snapshots per tenant.

Address the findings in docs/security/sandbox-security-review.md.

- Tenant isolation (opt-in): apikey.Record gains an optional tenant. The
  gateway derives the caller's tenant from the client-cert CommonName (the API
  key name) and enforces it on every session-scoped RPC: getSession/getPodIP
  return NotFound on mismatch (so existence is not leaked), list filters, and
  create stamps the bound tenant or rejects an explicit mismatch. The E2B
  control plane enforces the same binding (create/find/list/reacquire). Keys
  without a tenant keep the legacy shared-credential behaviour, so existing
  single-tenant installs are unaffected.
- Certificate revocation is now persisted (sandbox-revoked.json), loaded on
  start, reconciled against the active key set on every load (including the
  first load after a restart), and checked at Login so a revoked certificate
  cannot re-enroll.
- The rate limiter keys buckets on the authenticated identity (cert CN /
  token hash) instead of the client-supplied x-sandbox-tenant header, and gives
  loopback its own bucket.
- Session, sub-agent, background-run and approval identifiers are
  crypto-random instead of timestamp-derived.
- Warm-pool pods now deny ingress except kubelet TCP probes; previously the
  "deny-all" policy only declared egress, leaving sandboxd reachable from any
  pod.

Adds regression tests for each path and a security review report.
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

1 059 tests  +13   1 032 ✅ +13   7m 3s ⏱️ -3s
  121 suites ± 0      27 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 416cdba. ± Comparison against base commit d06b5f0.

This pull request removes 4 and adds 17 tests. Note that renamed tests count towards both.
github.com/xiaods/k8e/pkg/sandboxmatrix/ratelimit ‑ TestExtractTenant/api_key_prefix
github.com/xiaods/k8e/pkg/sandboxmatrix/ratelimit ‑ TestExtractTenant/authorization_without_bearer
github.com/xiaods/k8e/pkg/sandboxmatrix/ratelimit ‑ TestExtractTenant/short_api_key
github.com/xiaods/k8e/pkg/sandboxmatrix/ratelimit ‑ TestExtractTenant/tenant_header_wins
github.com/xiaods/k8e/pkg/sandbox/apikey ‑ TestTenantOmittedWhenEmpty
github.com/xiaods/k8e/pkg/sandbox/apikey ‑ TestTenantRoundTripsThroughV2
github.com/xiaods/k8e/pkg/sandbox/e2b ‑ TestControlPlaneTenantIsolation
github.com/xiaods/k8e/pkg/sandbox/e2b ‑ TestControlPlaneTenantNameConflict
github.com/xiaods/k8e/pkg/sandboxmatrix/grpc ‑ TestCallerTenantFromCertificate
github.com/xiaods/k8e/pkg/sandboxmatrix/grpc ‑ TestConfirmActionApprovalIDIsRandom
github.com/xiaods/k8e/pkg/sandboxmatrix/grpc ‑ TestCreateSessionTenantBinding
github.com/xiaods/k8e/pkg/sandboxmatrix/grpc ‑ TestExposeServiceTenantIsolation
github.com/xiaods/k8e/pkg/sandboxmatrix/grpc ‑ TestGetSessionTenantIsolation
github.com/xiaods/k8e/pkg/sandboxmatrix/grpc ‑ TestListSessionsTenantFilter
…

♻️ This comment has been updated with latest results.

DeepSource flags a nil body passed to http.NewRequest as an antipattern
(GO-... / antipattern). Use http.NoBody, which is what the stdlib uses
internally for a bodyless request.
@sonarqubecloud

Copy link
Copy Markdown

@xiaods
xiaods merged commit a737c24 into main Nov 16, 2026
9 checks passed
@xiaods
xiaods deleted the fix/sandbox-security-hardening branch November 16, 2026 02:40
xiaods added a commit that referenced this pull request Nov 16, 2026
A session could reference any non-reserved Secret in the sandbox namespace and
have the gateway inject it into the exec environment, so one tenant could
exfiltrate another tenant's provider key by naming it. This was independent of
the tenant-isolation work in #676: even a caller bound to its own tenant could
name any Secret.

A Secret may now carry the label sandbox.k8e.io/tenant: <tenant>.
A labelled Secret is readable only by a session whose spec.tenantID matches.
Unlabelled Secrets stay legacy-compatible for the shared default tenant, so
existing single-tenant installs keep working; a non-default tenant may not
reference them. The gateway-internal deny-list is unchanged and still applies
(defence in depth), and the error never includes the Secret value.

Enforcement is at exec resolution (readSecretKey) — the last point before the
value enters the sandbox environment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant