Skip to content

BE-846: atlas: delta feed, placement and projector - #9710

Open
indietyp wants to merge 2 commits into
bm/be-845-atlas-world-epoch-and-schedules-of-the-serving-layerfrom
bm/be-846-atlas-delta-feed-placement-and-projector
Open

indietyp wants to merge 2 commits into
bm/be-845-atlas-world-epoch-and-schedules-of-the-serving-layerfrom
bm/be-846-atlas-delta-feed-placement-and-projector

Conversation

@indietyp

@indietyp indietyp commented Sep 14, 2026

Copy link
Copy Markdown
Member

🌟 What is the purpose of this PR?

Add the delta pipeline that keeps a generation's overlay live, from the feed through placement and projection to the task that runs them. A delta feed bound to one process-wide atlas and a single epoch cannot run per generation, and a second generation has no feed of its own. Here a generation's feed runs against its own World, and each published revision is a new Epoch.

The module stays dead code under the lib.rs attribute until the read API consumes it (BE-815).

For review, the properties this stands on:

  • The feed reads arrivals from the store and the placement task positions them, after which the projector maps their embeddings onto the layout. The task lifecycle runs all of it and publishes each revision as a new Epoch.
  • The feed starts, stops and joins on its own.
  • A walk delivers a captured schedule with the rows an epoch refuses removed, and every bucket run stays contiguous.

🔍 What does this change?

  • serve/delta/feed/: DeltaReader and the pending queue.
  • serve/delta/placement/: DeltaPlacementTask and its pending set.
  • serve/delta/projector/: DeltaProjector over the EmbeddingWorkflow.
  • serve/delta/task/: DeltaTask with its options, the lifecycle that runs the three and publishes.
  • serve/walk/: Walk over a captured schedule.
  • Each module carries its tests beside it.

❓ How to test this?

  1. cargo nextest run -p hash-graph-atlas --all-features serve::delta:: runs the feed, placement, projector and task tests, including the start, stop and join cases.
  2. Reading order: serve/delta/task/mod.rs is the lifecycle, and the other three are what it drives.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

@vercel

vercel Bot commented Sep 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Sep 14, 2026 9:17pm UTC
petrinaut Ready Ready Preview Sep 14, 2026 9:17pm UTC
petrinaut-docs Ready Ready Preview Sep 14, 2026 9:17pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Sep 14, 2026 9:17pm UTC

Request Review

@vercel
vercel Bot temporarily deployed to Preview – petrinaut September 14, 2026 13:40 Inactive
@vercel
vercel Bot temporarily deployed to Preview – petrinaut-docs September 14, 2026 13:40 Inactive
@github-actions github-actions Bot added area/libs Relates to first-party libraries/crates/packages (area) type/eng > backend Owned by the @backend team labels Sep 14, 2026
@indietyp
indietyp deployed to pull-request September 14, 2026 13:41 — with GitHub Actions Active
@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.51292% with 363 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.84%. Comparing base (192ab0c) to head (c999864).

Files with missing lines Patch % Lines
...local/graph/atlas/src/serve/delta/placement/mod.rs 43.36% 122 Missing and 6 partials ⚠️
...ibs/@local/graph/atlas/src/serve/delta/feed/mod.rs 58.02% 98 Missing and 4 partials ⚠️
...local/graph/atlas/src/serve/delta/projector/mod.rs 57.43% 81 Missing and 2 partials ⚠️
...ibs/@local/graph/atlas/src/serve/delta/task/mod.rs 80.58% 19 Missing and 1 partial ⚠️
libs/@local/graph/atlas/src/serve/walk/mod.rs 57.89% 16 Missing ⚠️
...al/graph/atlas/src/serve/delta/feed/pending/mod.rs 93.03% 8 Missing and 6 partials ⚠️
Additional details and impacted files
@@                                        Coverage Diff                                         @@
##           bm/be-845-atlas-world-epoch-and-schedules-of-the-serving-layer    #9710      +/-   ##
==================================================================================================
+ Coverage                                                           68.72%   68.84%   +0.12%     
==================================================================================================
  Files                                                                1114     1121       +7     
  Lines                                                              101466   102550    +1084     
  Branches                                                             5000     5073      +73     
==================================================================================================
+ Hits                                                                69729    70602     +873     
- Misses                                                              30888    31079     +191     
- Partials                                                              849      869      +20     
Flag Coverage Δ
apps.hash-ai-worker-ts 1.99% <ø> (ø)
apps.hash-api 15.35% <ø> (ø)
apps.hash-graph 13.62% <ø> (ø)
rust.hash-graph-atlas 80.80% <66.51%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread libs/@local/graph/atlas/src/serve/delta/mod.rs Fixed
Comment thread libs/@local/graph/atlas/src/serve/delta/mod.rs Fixed
Comment thread libs/@local/graph/atlas/src/serve/delta/mod.rs Fixed
Comment thread libs/@local/graph/atlas/src/serve/delta/mod.rs Fixed
Comment thread libs/@local/graph/atlas/src/serve/delta/mod.rs Fixed
Comment thread libs/@local/graph/atlas/src/serve/delta/mod.rs Fixed
Comment thread libs/@local/graph/atlas/src/serve/delta/mod.rs Fixed
@indietyp
indietyp added this pull request to stack #9714 September 14, 2026 13:56
@indietyp indietyp self-assigned this Sep 14, 2026
@indietyp indietyp changed the title feat: add the atlas delta pipeline (feed, placement, projector, task) BE-846: atlas: delta feed, placement and projector Sep 14, 2026
@indietyp
indietyp force-pushed the bm/be-846-atlas-delta-feed-placement-and-projector branch from 3dd45c1 to 3177d2c Compare September 14, 2026 14:44
@indietyp
indietyp force-pushed the bm/be-846-atlas-delta-feed-placement-and-projector branch from 3177d2c to 5f1e185 Compare September 14, 2026 15:39
@indietyp
indietyp force-pushed the bm/be-846-atlas-delta-feed-placement-and-projector branch from 5f1e185 to 91767d7 Compare September 14, 2026 15:55
@indietyp
indietyp marked this pull request as ready for review September 14, 2026 15:58
Copilot AI balanced review requested due to automatic review settings September 14, 2026 15:58
@cursor

cursor Bot commented Sep 14, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
New concurrent replay path over Postgres, placement, and optional Temporal workflows affects how live entities enter the layout, though it remains unwired from the public API until BE-815.

Overview
Introduces the Atlas live-delta pipeline so a generation's overlay can track Postgres entity arrivals: poll and coalesce events, classify nodes/edges, resolve displays, request wire positions, and publish each revision as a new Epoch via DeltaReader.

DeltaFeedTask replays safety-lagged entity events, stages per-entity work in Pending (overlap by transaction time, withdrawals, version-checked placement completions), and exchanges the working Delta through a coalescing Publication loop.

DeltaPlacementTask admits placement requests, polls DB embeddings with retry budgets, optionally starts Temporal embedding workflows, projects batches on Rayon through DeltaProjector, and returns fitted-world positions or terminal **PlacementError**s.

DeltaTask opens the projector (checkpoint + roundtrip sample when present), wires feed ↔ placement channels, and runs feed, publication, and placement workers until shutdown.

Extensive unit tests cover pending/feed/placement/projector edge cases. Unrelated test fixtures in salt/fit/tests.rs rename the sample reviewer to Max Mustermann.

Reviewed by Cursor Bugbot for commit c999864. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c999864. Configure here.

else {
return true;
};
delta.update_edge(entity, legend.clone(), Some([source, target]))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type change keeps both identities live

Medium Severity

apply writes a node or edge from the latest classification without withdrawing the other identity. After an entity's type flips between link and non-link, classify follows the new kind, and both the old row and the new one stay visible in the published overlay.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit c999864. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

3 participants