SRE-1060: Let turbo's task graph drive pruning and CI selection - #9754
TimDiekmann wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9754 +/- ##
=======================================
Coverage 66.28% 66.28%
=======================================
Files 1775 1775
Lines 191861 191861
Branches 7853 7853
=======================================
Hits 127171 127171
Misses 63209 63209
Partials 1481 1481
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 15.93%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | bit_matrix/dense/iter_row[64] |
137.2 ns | 166.4 ns | -17.53% |
| ❌ | bit_matrix/dense/iter_row[200] |
174.7 ns | 203.9 ns | -14.31% |
as_constant |
< 1 ns | < 1 ns | N/A | |
constant_equal |
< 1 ns | < 1 ns | N/A | |
constant_not_equal |
< 1 ns | < 1 ns | N/A | |
access |
< 1 ns | < 1 ns | N/A | |
runtime_equal |
< 1 ns | < 1 ns | N/A | |
runtime_not_equal |
< 1 ns | < 1 ns | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing t/sre-1060-let-turbos-task-graph-drive-pruning-and-ci-selection (76a7648) with main (5af427b)1
Footnotes
Pins turbo to 2.10.14-canary.3 and enables `affectedUsingTaskInputs`, which makes `turbo prune` retain the packages the retained packages' tasks depend on, across toolchains. The Dockerfiles pass the image's package alone instead of the closure `prune-scopes.sh` computed over the package and task graphs. `affected-packages.sh` selects by the `command` turbo resolves for a task, which covers `package.json` scripts as well as a `command` in a `turbo.json`, and is `null` for a task that only orders others. Two edges the flag exposes as missing are declared: `@tests/hash-playwright#test:integration` drives the frontend and the API, and `@local/petrinaut-arch-docs#doc:architecture` reads annotations in the Petrinaut packages' sources; `^manifest` carries those sources into their hashes. `@rust/hash-graph-postgres-store` loses its `dev` task. Nothing ran it, it pointed at the graph's old `migrate` while the crate's tests use the migrations CLI, and through it prune retained `@apps/hash-graph` for the crate's test job, whose unfiltered `start:test` then ran both migrations.
3441cb8 to
76a7648
Compare
🌟 What is the purpose of this PR?
turbo 2.10.14 lets the task graph drive two things we compute by hand today:
turbo prunefollows task dependencies across toolchains behindfutureFlags.affectedUsingTaskInputs, andturbo queryreports thecommandit resolves for a task. This PR adopts both, removes the workaround script, and declares the two edges the flag exposed as missing.The flag also narrows
affectedTasksto the declared task graph: a package no longer selects its dependents' tasks unless a task edge carries the change. Compared over the recentmainhistory, the selection differs from today's only where the dependent never reads the changed package — the Playwright type checks on frontend changes,lint:ruffon TypeScript changes, and the Rust crates onyarn.lock-only bumps.🔗 Related links
affectedPackagesandprune), vercel/turborepo#13889 (commandinturbo query), discussion🚫 Blocked by
2.10.14-canary.3and switches to the release before merging🔍 What does this change?
2.10.14-canary.3in the mise config and lockfileaffectedUsingTaskInputsin the rootturbo.json.github/scripts/prune-scopes.shand theyqit needed inpetrinaut-optare removed.github/scripts/affected-packages.shselects bycommandinstead ofscript, so a task defined by acommandin aturbo.jsoncounts like apackage.jsonscript@tests/hash-playwright#test:integrationdepends on^manifest: it drives the frontend and the API, and its override had replaced the root's^manifestwith the health checks alone@local/petrinaut-arch-docs#doc:architecturedepends on^manifest: it reads annotations in the Petrinaut packages' sources; thecache: falsecomment now names the reason that still holds@rust/hash-graph-postgres-storeloses itsdevtask: nothing ran it, and its edge to@apps/hash-graph#start:test:migratemade prune retain the graph for the crate's test job, whose unfilteredstart:testthen applied the graph's migrations next to the migrations CLI the tests runPre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
turbo.json's have been updated to reflect thisturbo prunewith the flag also follows the rootdoc:dependency-diagram → @rust/hash-repo-chores#buildedge, so thepetrinaut-optimage carrieshash-repo-choresand its three crates as build context. Nothing builds or starts them; excluding them would mean moving that task definition out of the root, which is not worth it for build context..github/actions/prune-repository/prune.pystill computes the package-and-task closure itself before callingturbo prune. With the flag that computation is redundant; its extra rules (thehashql-compiletestcycle, thedarwin-kperffamily, test-data crates, the atlas fixtures for the frontend, non-workspace paths) are not, so trimming it is a follow-up.package.jsondependencies,affectedByrecords only packages beyond those, and the removeddevtask had no script.🐾 Next steps
prune.pyto its extra rules once the flag carries the closurepackage.jsonscripts intoturbo.jsoncommands (experimentalTaskCommand), which thecommandcriterion inaffected-packages.shprepares forprune.pystill generate forturbo prune🛡 What tests cover this?
turbo pruneoutput for all eight images against theprune-scopes.shclosure,affectedTasksover the recentmainhistory with the flag off and on, and the CI prune for@rust/hash-graph-postgres-storefollowed byturbo run start:test --dry=jsonin the pruned tree.❓ How to test this?
mise install --locked npm:turboturbo prune '@apps/hash-api' --docker --out-dir /tmp/pruneand compare the retained packages withgit show origin/main:.github/scripts/prune-scopes.sh | bash -s '@apps/hash-api'onmain.github/scripts/affected-packages.sh test:integrationon a branch that changesapps/hash-frontendselects@tests/hash-playwright