Enterprise-level testing: the suite is green while the deployed product is unusable (a test read the placeholder client_id and passed)

[TortoiseWolfe]

Every test suite was green while no human could sign up on production (#287). That's not a missing test — it's a class of testing the repo doesn't do. If ScriptHammer is going to claim enterprise-ready (#280), the test strategy has to be able to catch "the deployed product is unusable."

The proof: a test looked straight at the bug and passed

tests/e2e/security/oauth-csrf.spec.ts:186 reads the real OAuth client_id off the authorize redirect and asserts:

expect(url.searchParams.get('client_id')).toBeTruthy();

The live value was "placeholder_google_client_id" — a non-empty string. The assertion passed. The test had the defect in its hand and let it through, because it asserted presence, not validity. Same for GitHub.

Why 369 users existed and zero humans could sign up

tests/e2e/utils/test-user-factory.ts:159-162:

const { data, error } = await client.auth.admin.createUser({
  ...
  email_confirm: true, // Auto-confirm email
});

Every test user is minted through the service-role admin API, which bypasses: the real signup form, the email confirmation mailer, and OAuth entirely. The suite exercises a path no human can take, and never touches the three paths every human must take. Result: 368 E2E users in prod, and the only real account (admin@scripthammer.com) had never signed in.

The gaps, concretely

Gap	Evidence	What #287 proved it misses
Config validity never asserted	oauth-csrf.spec.ts:186 — toBeTruthy() on client_id	Literal placeholder_*_client_id in prod
Real signup path untested	all users via admin.createUser(email_confirm:true)	Email signup returns 429 (no SMTP, 2/hr cap)
No production smoke test	nothing in tests/ targets scripthammer.com	site_url was http://localhost:3000, allow-list empty
Deployed config drift undetected	auth-config.json only diffed the 4 keys it listed	site_url/uri_allow_list drifted invisibly for the project's lifetime
OAuth "integration" tests don't fail on broken providers	oauth-flow.test.ts asserts data.url contains 'authorize'	A placeholder client still produces an authorize URL

Proposed work

1. Assert config validity, not presence (cheapest, would've caught #287 outright)

• client_id must not match /^placeholder|^changeme|^your[-_]/i and must look like a real client (Google: *.apps.googleusercontent.com; GitHub: Iv1.*/Ov23*).
• Fail if site_url contains localhost while targeting a non-local environment.
• Fail if uri_allow_list is empty.

2. A production smoke suite (@smoke, runs post-deploy against the real origin)

• /auth/v1/settings reports the providers we expect enabled.
• The Google + GitHub authorize redirects carry valid-shaped, non-placeholder client IDs (this is the Production OAuth broken since the Supabase project migration: client IDs are placeholders (recurrence of #85, closed-but-never-fixed) #287 detector).
• Sign-in page loads with zero console errors (would also catch the Invalid or unexpected token defect in Production OAuth broken since the Supabase project migration: client IDs are placeholders (recurrence of #85, closed-but-never-fixed) #287).
• Canonical origin resolves as expected (apex; www/github.io → 301).

3. Test the path a human actually takes (at least once)

• One E2E that signs up through the real form — not admin.createUser — and confirms via a mail-catcher/inbox API. Gate on SMTP landing (Production OAuth broken since the Supabase project migration: client IDs are placeholders (recurrence of #85, closed-but-never-fixed) #287).
• Keep admin.createUser for the other 300+ fixtures (it's correct for speed/isolation) — but stop letting it be the only signup path ever exercised.

4. Config-as-code, drift-checked in CI

• scripts/supabase/auth-config.json now declares site_url + uri_allow_list (done 2026-07-15 — previously my live fixes were untracked and a re-apply would have silently reverted them).
• Run pnpm supabase:auth-config (dry-run) in CI and fail on drift, so prod config can't diverge from the repo again.
• Consider extending the desired-state file to assert OAuth *_enabled + SMTP presence.

The principle

The suite tested the code. Nobody tested the deployed product. Enterprise-ready means the tests can fail when the thing a customer touches is broken — even when every unit, integration, and E2E test is green. #287 is the existence proof that today they can't.

Tracked under #280 (criterion 3: production operations posture). Directly enables closing #287.