Phase 8 scheduling contract

[streaky]

This pull request introduces Phase 8 of the scheduler, bringing priority-aware, fair, and resumable workload scheduling to the project. It adds a new versioned mixed-workload proof and supporting infrastructure, updates documentation to reflect the new scheduler and its capabilities, and refines the codebase to support trusted scheduling metadata and diagnostics.

Scheduler and Workload Scheduling Enhancements:

• Implements Phase 8 scheduler with priority-aware deficit round-robin, per-principal/class fairness, FIFO ordering, monotonic age promotion, resumable execution sessions, and trusted scheduling metadata. Suspended sessions now release slot occupancy but retain model references, and cancellation/deadline/shutdown callbacks remain registered appropriately (README.md, AGENTS.md, crates/server/src/lib.rs). [1] [2] [3] [4] [5]
• Adds SchedulerPolicyConfig and related validation, supporting configuration of weights, refill, promotion rounds, and diagnostic capacity (crates/server/src/lib.rs).
• Introduces new SchedulingMetadata, SchedulingClass, and PrioritySource types, and updates InferRequest and EngineRequest to support trusted scheduling metadata (crates/server/src/lib.rs). [1] [2] [3]

Testing and Proof Infrastructure:

• Adds a new versioned mixed-workload proof for Phase 8, including config/phase8-workload.json and a corresponding phase8-proof service in compose.test.yaml, along with a shell script to run the scheduler report and collect diagnostic output (config/phase8-workload.json, compose.test.yaml, README.md). [1] [2] [3]

Documentation Updates:

• Updates README.md and AGENTS.md to document the completion of Phase 8, new scheduler features, and revised future goals. This includes instructions for running the Phase 8 scheduler report and clarifies the current state and next steps for compatibility and production hardening (README.md, AGENTS.md). [1] [2] [3] [4] [5]

Codebase Improvements:

• Refactors EngineRequest to be cloneable, use owned types, and carry scheduling metadata and prefill chunk size. Makes RequestControl methods public to support scheduler integration (crates/server/src/lib.rs). [1] [2] [3] [4]
• Marks GreedySampler as Send to support safe transfer between scheduler threads (crates/executor/src/lib.rs).
• Minor dependency update: adds serde to workspace dependencies in CLI crate (crates/cli/Cargo.toml).

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (5)

crates/server/src/lib.rs:2128

• The scheduler metadata is assigned a new UUID here, but http_debug_middleware independently creates another request ID and overwrites the response x-request-id header (lib.rs:1912,1968-1971). As a result, the transport correlation ID in scheduler decisions cannot be joined to the client-visible/header or HTTP diagnostic records, contrary to the single-correlation-ID contract. Generate the ID once at the middleware boundary and pass that same value through the handler into SchedulingMetadata.

    let id = Uuid::new_v4().to_string();
    let correlation_id = id.clone();

crates/cli/src/main.rs:283

• Wrapping the entire ResidentEngine in one WorkloadScheduler introduces a single synchronous scheduler worker for every resident model. run_scheduler executes one selected quantum at a time, so requests for otherwise independent resident model executors now serialize globally, contradicting the documented guarantee that distinct resident models execute independently. Scheduling/execution needs to be sharded per resident model/slot, or selected quanta must be dispatched concurrently while retaining per-slot serialization.

            let engine = WorkloadScheduler::new(
                engine,
                SchedulerPolicyConfig {

crates/cli/src/main.rs:560

• The proof labels this as the per-quantum latency gate, but mixed_quantums is built from inter-token wall-clock waits. Those values include queueing and other requests' quanta, and omit preparation/prefill measurements that do not emit tokens, even though actual quantum durations are already recorded as quantum_duration_ns in scheduler decisions. Derive both baseline and mixed quantum percentiles from the attributed decision records so the claimed acceptance gate measures the intended quantity.

    let mixed_quantums = mixed
        .iter()
        .flat_map(|result| result.token_wait_ms.iter().copied())
        .collect::<Vec<_>>();
    let p95_first_event_ms = percentile(&mixed_first, 95);
    let p95_token_wait_ms = percentile(&mixed_quantums, 95);

crates/server/src/scheduler.rs:709

• SchedulerMetrics::admitted is exposed in status but is never incremented, so every runtime and proof artifact reports zero admitted requests regardless of workload. Increment this counter when a submission is accepted into the jobs/policy queues.

            policy.enqueue(id, flow, admitted_round, admitted_at);

crates/server/src/scheduler.rs:462

• This drain target only includes records already counted as emitted and never retries flush_loss_summary. If the final scheduler decision overflows the channel, its pending loss summary remains buffered forever once the workload becomes idle, even after sink capacity returns. The flush path should repeatedly enqueue any pending summary and include it in the drain target, so the documented later loss summary is guaranteed at quiescence.

    pub fn flush_diagnostics(&self, timeout: Duration) -> bool {
        let target = self.diagnostics.emitted.load(Ordering::Acquire);
        let started = Instant::now();
        while self.diagnostics.delivered.load(Ordering::Acquire) < target {