Repository navigation
fix(ci): run changed criteria and standalone e2e on PRs (#2341, #2342) - #2345
Merged
Merged
Conversation
1. CI job "Acceptance (changed criteria)" (.github/workflows/ci.yml) runs the criteria of every test/acceptance/issue_<n>_test.go a pull request changes, on a fresh Linux stack with no .env, through the shared .github/workflows/acceptance-run.yml (new tickets input). The ticket list comes from scripts/acceptance-pr-paths.py --tickets; scripts/acceptance-env.py flags --tickets selects them on the plain stack and refuses a malformed list or a named environment; make acceptance and scripts/release-acceptance.sh take ACCEPTANCE_TICKETS, and the latter skips the browser suite when it is set. 2. make e2e-standalone runs every test/e2e test except those calling helpers.NeedsServices (TestKnowledgeLifecycle, TestTrinoToCatalogEnrichment, TestEnrichment_RealAssembledServer), with E2E_STANDALONE=1. make verify runs it right after vet-tags, and CI's test job runs it after the unit tests. 3. test/gates/acceptance_changed_test.go pins the ticket list, the flags and their refusals, the CI wiring, and the target's place in verify. README.md, CLAUDE.md and test/e2e/README.md describe both. 4. make release-check runs GoReleaser with GITLAB_TOKEN and GITEA_TOKEN unset: with either in the shell, its homebrew step blocked with no output, so make verify never finished on such a machine. CI's runners carry neither. Closes #2341 Closes #2342
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2345 +/- ##
=======================================
Coverage 94.80% 94.81%
=======================================
Files 1443 1443
Lines 117111 117111
=======================================
+ Hits 111029 111035 +6
+ Misses 6062 6056 -6
Partials 20 20 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
.github/workflows/ci.yml,.github/workflows/acceptance-run.yml,Makefile,scripts/acceptance-pr-paths.py,scripts/acceptance-env.py,scripts/release-acceptance.sh,test/e2e/helpers/services.goand threetest/e2etests. Closes two gaps that let #2336's nightly failures merge with every pre-merge gate green, and one defect that keptmake verifyfrom finishing.Why
The E2E Nightly on c0eb937 failed two ways (#2336, fixed by #2344):
TestIssue2179_*passed on the author's machine because the encryption key came from a gitignored.env. CI runners have no.env, and nothing ran those criteria on a CI machine before merge.TestAdminAPI_AuthEnforcementstill expected 401 after Portal: a signed-in non-admin who opens Knowledge > Changesets is stuck in a sign-in loop (admin API answers 401 instead of 403) #2337 deliberately changed the answer to 403. It needs no service, buttest/e2eran only in the nightly's Real-dependency E2E job.What changes
Acceptance (changed criteria), runs the criteria of everytest/acceptance/issue_<n>_test.goa pull request changes. It runs on a fresh Linux stack with no.env, through the sameacceptance-run.ymlthe nightly uses.scripts/acceptance-pr-paths.py --ticketsreads the diff and prints the ticket list. It keeps only files that still exist, so a deleted criterion names no ticket.scripts/acceptance-env.py flags --ticketsturns the list into-run/-skipflags for the plain stack. It refuses a malformed list, and a named environment.make acceptanceandscripts/release-acceptance.shtakeACCEPTANCE_TICKETS. With it set,release-acceptance.shskips the browser suite.make e2e-standaloneruns everytest/e2etest withE2E_STANDALONE=1, in about 30 seconds.make e2e-upservices callhelpers.NeedsServicesand are skipped there:TestKnowledgeLifecycle,TestTrinoToCatalogEnrichmentandTestEnrichment_RealAssembledServer. The nightly still runs them.make verifyruns it right aftervet-tags, and CI's test job runs it after the unit tests.NeedsServices.make release-checkno longer hangs on a machine with a GitLab or Gitea token. WithGITLAB_TOKENorGITEA_TOKENin the shell, GoReleaser picks that forge's client, and its homebrew step blocks with no CPU and no output.make verifythen never finished; one run sat there for 25 minutes. The target now runs GoReleaser with both unset, which matches CI's runners. The dry run publishes nothing.test/gates/acceptance_changed_test.gopins:--ticketsflags and their three refusalse2e-standalone's place in verify, betweenvet-tagsand the lanesREADME.md,CLAUDE.mdandtest/e2e/README.mddescribe both checks.Verification
.envmoved aside:ACCEPTANCE_TICKETS=2179,2228 bash scripts/release-acceptance.sh.make e2e-standalone: 6 pass, 3 skip. With the old 401 expectation put back temporarily, it fails, so it would have caught E2E Nightly is failing #2336's admin failure before merge.make release-check: exit 0 in 2m14s withGITLAB_TOKENin the shell. Before the change it hung at the homebrew formula step.make verify: passed on the committed tree ("All checks passed").Limits
issue_<n>_test.go, soAcceptance (changed criteria)is skipped here. It first runs on the next PR that changes a criterion. The shared workflow is still exercised on this PR by the existing worker-off acceptance job, which runs because the Makefile and scripts changed.Closes #2341
Closes #2342