Skip to content

fix(ci): run changed criteria and standalone e2e on PRs (#2341, #2342) - #2345

Merged
cjimti merged 1 commit into
mainfrom
fix/2341-2342-pr-acceptance-e2e-standalone
Oct 27, 2026
Merged

cjimti merged 1 commit into
mainfrom
fix/2341-2342-pr-acceptance-e2e-standalone

Conversation

@cjimti

@cjimti cjimti commented Oct 27, 2026

Copy link
Copy Markdown
Member

Changes .github/workflows/ci.yml, .github/workflows/acceptance-run.yml, Makefile, scripts/acceptance-pr-paths.py, scripts/acceptance-env.py, scripts/release-acceptance.sh, test/e2e/helpers/services.go and three test/e2e tests. Closes two gaps that let #2336's nightly failures merge with every pre-merge gate green, and one defect that kept make verify from finishing.

Why

The E2E Nightly on c0eb937 failed two ways (#2336, fixed by #2344):

  1. TestIssue2179_* passed on the author's machine because the encryption key came from a gitignored .env. CI runners have no .env, and nothing ran those criteria on a CI machine before merge.
  2. TestAdminAPI_AuthEnforcement still expected 401 after Portal: a signed-in non-admin who opens Knowledge > Changesets is stuck in a sign-in loop (admin API answers 401 instead of 403) #2337 deliberately changed the answer to 403. It needs no service, but test/e2e ran only in the nightly's Real-dependency E2E job.

What changes

  1. Changed criteria run on every PR (dev: the stack starts with no field-encryption key unless an untracked .env sets one #2341). A new CI job, Acceptance (changed criteria), runs the criteria of every test/acceptance/issue_<n>_test.go a pull request changes. It runs on a fresh Linux stack with no .env, through the same acceptance-run.yml the nightly uses.
    • scripts/acceptance-pr-paths.py --tickets reads the diff and prints the ticket list. It keeps only files that still exist, so a deleted criterion names no ticket.
    • scripts/acceptance-env.py flags --tickets turns the list into -run/-skip flags for the plain stack. It refuses a malformed list, and a named environment.
    • make acceptance and scripts/release-acceptance.sh take ACCEPTANCE_TICKETS. With it set, release-acceptance.sh skips the browser suite.
    • The job is skipped when a PR changes no criterion.
  2. Standalone e2e in verify and CI (verify: run the test/e2e tests that need no running service #2342). make e2e-standalone runs every test/e2e test with E2E_STANDALONE=1, in about 30 seconds.
    • Tests that need the make e2e-up services call helpers.NeedsServices and are skipped there: TestKnowledgeLifecycle, TestTrinoToCatalogEnrichment and TestEnrichment_RealAssembledServer. The nightly still runs them.
    • make verify runs it right after vet-tags, and CI's test job runs it after the unit tests.
    • A new e2e test runs before merge unless it calls NeedsServices.
  3. make release-check no longer hangs on a machine with a GitLab or Gitea token. With GITLAB_TOKEN or GITEA_TOKEN in the shell, GoReleaser picks that forge's client, and its homebrew step blocks with no CPU and no output. make verify then never finished; one run sat there for 25 minutes. The target now runs GoReleaser with both unset, which matches CI's runners. The dry run publishes nothing.
  4. Gate tests. test/gates/acceptance_changed_test.go pins:
    • the ticket list (dedupe, order, deleted and non-criteria paths)
    • the --tickets flags and their three refusals
    • the CI wiring strings
    • e2e-standalone's place in verify, between vet-tags and the lanes
  5. Docs. README.md, CLAUDE.md and test/e2e/README.md describe both checks.

Verification

  1. CI's changed-criteria path, run locally with .env moved aside: ACCEPTANCE_TICKETS=2179,2228 bash scripts/release-acceptance.sh.
    • It built the stack, ran only those tickets (2179: 4 pass, 2228: 3 pass), skipped the browser suite and tore the stack down.
    • A ticket whose criteria all belong to another environment runs zero tests and exits 0.
  2. make e2e-standalone: 6 pass, 3 skip. With the old 401 expectation put back temporarily, it fails, so it would have caught E2E Nightly is failing #2336's admin failure before merge.
  3. make release-check: exit 0 in 2m14s with GITLAB_TOKEN in the shell. Before the change it hung at the homebrew formula step.
  4. make verify: passed on the committed tree ("All checks passed").

Limits

  1. This PR changes no issue_<n>_test.go, so Acceptance (changed criteria) is skipped here. It first runs on the next PR that changes a criterion. The shared workflow is still exercised on this PR by the existing worker-off acceptance job, which runs because the Makefile and scripts changed.
  2. Whether the new job blocks merging depends on branch protection's required checks, which this PR does not change.

Closes #2341
Closes #2342

1. CI job "Acceptance (changed criteria)" (.github/workflows/ci.yml) runs the criteria of every test/acceptance/issue_<n>_test.go a pull request changes, on a fresh Linux stack with no .env, through the shared .github/workflows/acceptance-run.yml (new tickets input). The ticket list comes from scripts/acceptance-pr-paths.py --tickets; scripts/acceptance-env.py flags --tickets selects them on the plain stack and refuses a malformed list or a named environment; make acceptance and scripts/release-acceptance.sh take ACCEPTANCE_TICKETS, and the latter skips the browser suite when it is set.
2. make e2e-standalone runs every test/e2e test except those calling helpers.NeedsServices (TestKnowledgeLifecycle, TestTrinoToCatalogEnrichment, TestEnrichment_RealAssembledServer), with E2E_STANDALONE=1. make verify runs it right after vet-tags, and CI's test job runs it after the unit tests.
3. test/gates/acceptance_changed_test.go pins the ticket list, the flags and their refusals, the CI wiring, and the target's place in verify. README.md, CLAUDE.md and test/e2e/README.md describe both.
4. make release-check runs GoReleaser with GITLAB_TOKEN and GITEA_TOKEN unset: with either in the shell, its homebrew step blocked with no output, so make verify never finished on such a machine. CI's runners carry neither.

Closes #2341
Closes #2342
@codecov

codecov Bot commented Oct 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.81%. Comparing base (36f5e29) to head (cdb1cc1).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2345   +/-   ##
=======================================
  Coverage   94.80%   94.81%           
=======================================
  Files        1443     1443           
  Lines      117111   117111           
=======================================
+ Hits       111029   111035    +6     
+ Misses       6062     6056    -6     
  Partials       20       20           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cjimti
cjimti merged commit 82fe75f into main Oct 27, 2026
13 checks passed
@cjimti
cjimti deleted the fix/2341-2342-pr-acceptance-e2e-standalone branch October 27, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

verify: run the test/e2e tests that need no running service dev: the stack starts with no field-encryption key unless an untracked .env sets one

1 participant