Skip to content

SRE-1060: Let turbo's task graph drive pruning and CI selection - #9754

Draft
TimDiekmann wants to merge 1 commit into
mainfrom
t/sre-1060-let-turbos-task-graph-drive-pruning-and-ci-selection
Draft

TimDiekmann wants to merge 1 commit into
mainfrom
t/sre-1060-let-turbos-task-graph-drive-pruning-and-ci-selection

Conversation

@TimDiekmann

@TimDiekmann TimDiekmann commented Sep 16, 2026

Copy link
Copy Markdown
Member

🌟 What is the purpose of this PR?

turbo 2.10.14 lets the task graph drive two things we compute by hand today: turbo prune follows task dependencies across toolchains behind futureFlags.affectedUsingTaskInputs, and turbo query reports the command it resolves for a task. This PR adopts both, removes the workaround script, and declares the two edges the flag exposed as missing.

The flag also narrows affectedTasks to the declared task graph: a package no longer selects its dependents' tasks unless a task edge carries the change. Compared over the recent main history, the selection differs from today's only where the dependent never reads the changed package — the Playwright type checks on frontend changes, lint:ruff on TypeScript changes, and the Rust crates on yarn.lock-only bumps.

🔗 Related links

🚫 Blocked by

  • turbo 2.10.14 release — the branch pins 2.10.14-canary.3 and switches to the release before merging

🔍 What does this change?

  • Pins turbo to 2.10.14-canary.3 in the mise config and lockfile
  • Enables affectedUsingTaskInputs in the root turbo.json
  • The eight Dockerfiles prune the image's package alone; .github/scripts/prune-scopes.sh and the yq it needed in petrinaut-opt are removed
  • .github/scripts/affected-packages.sh selects by command instead of script, so a task defined by a command in a turbo.json counts like a package.json script
  • @tests/hash-playwright#test:integration depends on ^manifest: it drives the frontend and the API, and its override had replaced the root's ^manifest with the health checks alone
  • @local/petrinaut-arch-docs#doc:architecture depends on ^manifest: it reads annotations in the Petrinaut packages' sources; the cache: false comment now names the reason that still holds
  • @rust/hash-graph-postgres-store loses its dev task: nothing ran it, and its edge to @apps/hash-graph#start:test:migrate made prune retain the graph for the crate's test job, whose unfiltered start:test then applied the graph's migrations next to the migrations CLI the tests run

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • affected the execution graph, and the turbo.json's have been updated to reflect this

⚠️ Known issues

  • turbo prune with the flag also follows the root doc:dependency-diagram → @rust/hash-repo-chores#build edge, so the petrinaut-opt image carries hash-repo-chores and its three crates as build context. Nothing builds or starts them; excluding them would mean moving that task definition out of the root, which is not worth it for build context.
  • .github/actions/prune-repository/prune.py still computes the package-and-task closure itself before calling turbo prune. With the flag that computation is redundant; its extra rules (the hashql-compiletest cycle, the darwin-kperf family, test-data crates, the atlas fixtures for the frontend, non-workspace paths) are not, so trimming it is a follow-up.
  • The task documents do not change: the two new edges point at packages that are already package.json dependencies, affectedBy records only packages beyond those, and the removed dev task had no script.

🐾 Next steps

  • Trim prune.py to its extra rules once the flag carries the closure
  • Move the crates' package.json scripts into turbo.json commands (experimentalTaskCommand), which the command criterion in affected-packages.sh prepares for
  • SRE-984: native Cargo workspaces, which also removes the dummy crate manifests the Dockerfiles and prune.py still generate for turbo prune

🛡 What tests cover this?

  • None automated. Verified locally: turbo prune output for all eight images against the prune-scopes.sh closure, affectedTasks over the recent main history with the flag off and on, and the CI prune for @rust/hash-graph-postgres-store followed by turbo run start:test --dry=json in the pruned tree.

❓ How to test this?

  1. Checkout the branch, mise install --locked npm:turbo
  2. turbo prune '@apps/hash-api' --docker --out-dir /tmp/prune and compare the retained packages with git show origin/main:.github/scripts/prune-scopes.sh | bash -s '@apps/hash-api' on main
  3. .github/scripts/affected-packages.sh test:integration on a branch that changes apps/hash-frontend selects @tests/hash-playwright

@TimDiekmann TimDiekmann self-assigned this Sep 16, 2026
@vercel

vercel Bot commented Sep 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Sep 16, 2026 11:20am UTC
hashdotdesign-tokens Ready Ready Preview Sep 16, 2026 11:20am UTC
petrinaut Ready Ready Preview Sep 16, 2026 11:20am UTC
petrinaut-docs Ready Ready Preview Sep 16, 2026 11:20am UTC

Request Review

@github-actions github-actions Bot added area/apps > hash* Affects HASH (a `hash-*` app) area/infra Relates to version control, CI, CD or IaC (area) area/apps > hash-api Affects the HASH API (app) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team type/eng > backend Owned by the @backend team area/tests New or updated tests area/tests > playwright New or updated Playwright tests area/apps area/apps > hash-graph labels Sep 16, 2026
@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.28%. Comparing base (5af427b) to head (76a7648).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9754   +/-   ##
=======================================
  Coverage   66.28%   66.28%           
=======================================
  Files        1775     1775           
  Lines      191861   191861           
  Branches     7853     7853           
=======================================
  Hits       127171   127171           
  Misses      63209    63209           
  Partials     1481     1481           
Flag Coverage Δ
apps.hash-ai-worker-ts 1.99% <ø> (ø)
apps.hash-api 15.35% <ø> (ø)
apps.hash-graph 12.54% <ø> (ø)
local.hash-backend-utils 3.27% <ø> (ø)
local.hash-graph-sdk 10.02% <ø> (ø)
local.hash-isomorphic-utils 12.22% <ø> (ø)
rust.hash-graph-api 19.71% <ø> (ø)
rust.hash-graph-atlas 80.36% <ø> (ø)
rust.hash-graph-postgres-store 32.15% <ø> (ø)
rust.hashql-ast 89.63% <ø> (ø)
rust.hashql-compiletest 28.39% <ø> (ø)
rust.hashql-eval 79.82% <ø> (ø)
rust.hashql-hir 89.09% <ø> (ø)
rust.hashql-mir 87.92% <ø> (ø)
rust.hashql-syntax-jexpr 94.04% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed

codspeed Bot commented Sep 16, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 15.93%

⚠️ 6 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 2 regressed benchmarks
✅ 96 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
bit_matrix/dense/iter_row[64] 137.2 ns 166.4 ns -17.53%
bit_matrix/dense/iter_row[200] 174.7 ns 203.9 ns -14.31%
⚠️ as_constant < 1 ns < 1 ns N/A
⚠️ constant_equal < 1 ns < 1 ns N/A
⚠️ constant_not_equal < 1 ns < 1 ns N/A
⚠️ access < 1 ns < 1 ns N/A
⚠️ runtime_equal < 1 ns < 1 ns N/A
⚠️ runtime_not_equal < 1 ns < 1 ns N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing t/sre-1060-let-turbos-task-graph-drive-pruning-and-ci-selection (76a7648) with main (5af427b)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (c3c24bd) during the generation of this report, so 5af427b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Pins turbo to 2.10.14-canary.3 and enables `affectedUsingTaskInputs`, which makes `turbo prune` retain the packages the retained packages' tasks depend on, across toolchains. The Dockerfiles pass the image's package alone instead of the closure `prune-scopes.sh` computed over the package and task graphs.

`affected-packages.sh` selects by the `command` turbo resolves for a task, which covers `package.json` scripts as well as a `command` in a `turbo.json`, and is `null` for a task that only orders others.

Two edges the flag exposes as missing are declared: `@tests/hash-playwright#test:integration` drives the frontend and the API, and `@local/petrinaut-arch-docs#doc:architecture` reads annotations in the Petrinaut packages' sources; `^manifest` carries those sources into their hashes.

`@rust/hash-graph-postgres-store` loses its `dev` task. Nothing ran it, it pointed at the graph's old `migrate` while the crate's tests use the migrations CLI, and through it prune retained `@apps/hash-graph` for the crate's test job, whose unfiltered `start:test` then ran both migrations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash* Affects HASH (a `hash-*` app) area/apps > hash-api Affects the HASH API (app) area/apps > hash-graph area/apps area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) area/tests > playwright New or updated Playwright tests area/tests New or updated tests type/eng > backend Owned by the @backend team type/eng > frontend Owned by the @frontend team

Development

Successfully merging this pull request may close these issues.

1 participant