Repository navigation

feat(runtime): Implement pull based arroyo - #560

Merged

tryangul merged 41 commits into

mainfrom

rbroughan/pull-based-arroyo-part-2

Oct 20, 2026

Merged

feat(runtime): Implement pull based arroyo#560
tryangul merged 41 commits into
mainfrom
rbroughan/pull-based-arroyo-part-2

Commits on Aug 31, 2026
- Full stage based implementation of pull streaming
  tryangul
  committed
  319e885
- Handle N:1 and N:0 use cases. Refactor StageResult, handlers and organize for clarity.
  tryangul
  committed
  da3d724
- Support concurrent stage application.
  tryangul
  committed
  da211d7
- Actually commit offsets.
  tryangul
  committed
  448488a
- Add graceful termination aka cancellation support.
  tryangul
  committed
  ada8559
- Comment tweaks.
  tryangul
  committed
  7acfc20
- handle fatal kafka errors, preserve dlq headers, init coarsetime updater, move offset tracker.
  tryangul
  committed
  a855d06
- Basic rebalance handling.
  tryangul
  committed
  e65d4ac
- Factor out PipelineRunner. Update examples.
  tryangul
  committed
  5a36b93
- Formatting and comment cleanup.
  tryangul
  committed
  543b7df
- Fix stage name lifetimes.
  tryangul
  committed
  6a6d6b9
- Use singleton updater to prevent leak.
  tryangul
  committed
  44387cd
- Synchronize drain on rebalance.
  tryangul
  committed
  cf646b1
- Drain regardless of outcome
  tryangul
  committed
  3f89896
- Revert drain on rebalance due to potential threading deadlock.
  tryangul
  committed
  2e3d449
- Add BatchStage
  tryangul
  committed
  ad8cbcc
- Break out stream collector.
  tryangul
  committed
  840632a
- Add flush timer
  tryangul
  committed
  bace648
- Buffer now has an Out type
  tryangul
  committed
  36533f3
- Formatting
  tryangul
  committed
  4f3c5a7
- Add Pipeline trait
  tryangul
  committed
  bff165d
- Fmt.
  tryangul
  committed
  15e873a
- Doc test import
  tryangul
  committed
  e7af45f
- Cleanup types with aliases. Convert kafka stats logging to metrics.
  tryangul
  committed
  c2c4b09
- Cargo fmt.
  tryangul
  committed
  f79f05b
- ref(pull): remove unnecessary Arc around raw payload
  tryangul
  committed
  4b03c33
- ref(pull): use coarsetime for stage durations.
  tryangul
  committed
  4d1c367
- ref(pull): use parking_lot mutex for batch stage to avoid mutex syscall.
  tryangul
  committed
  43fd7ed
- ref(pull): boxed dyn producer for dlq handler.
  tryangul
  committed
  13a872d
- ref(pull): owned streams.
  tryangul
  committed
  498b419
- ref(pull): Map_emit.
  tryangul
  committed
  47e5e8c
- ref(pull): Fix-up pipeline runner.
  tryangul
  committed
  47e1a08
- ref(pull): Add batch time metric with flush reason for parity with push.
  tryangul
  committed
  b274934
- ref(pull): Fmt.
  tryangul
  committed
  1bf5057
Commits on Sep 16, 2026
- ref(pull): gate pull/stream API behind `pull` feature flag
  tryangul
  committed
  ee52fc4
Commits on Oct 1, 2026
Commits on Oct 20, 2026
- Merge branch 'main' into rbroughan/pull-based-arroyo-part-2
  tryangul
  committed
  61e4b6c

Commits

Commits on Aug 31, 2026

Commits on Sep 16, 2026

Commits on Oct 1, 2026

Commits on Oct 20, 2026