Skip to content

test: stabilize provider deadline verification - #318

Draft
EvanProgramming wants to merge 1 commit into
Evan/v3-native-plan-guardfrom
Evan/v3-deadline-test-stability
Draft

EvanProgramming wants to merge 1 commit into
Evan/v3-native-plan-guardfrom
Evan/v3-deadline-test-stability

Conversation

@EvanProgramming

@EvanProgramming EvanProgramming commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Stacked on PR #317; supports the production-readiness checks in #268.

The macOS Python 3.12 matrix failed the streaming deadline test at 0.100039 seconds against a strict 0.100000-second wall-clock assertion. That assertion includes scheduler and test overhead, so it can fail before checking whether late provider chunks were discarded. The matching non-streaming test used the same pattern.

Both tests now hold the provider behind an event and require the timeout to return while the provider remains blocked; after release they wait for the worker to finish and verify the timeout result and absence of late chunks. This checks the behavior directly without assuming the test process will be scheduled within a 100 ms wall-clock window. It does not assert a 100 ms end-to-end latency service level. No runtime code or dependency changes.

Both tests passed in 20 consecutive local repetitions (40 test executions). The full Python suite passed: 1,172 tests run with two Linux Bubblewrap tests skipped on macOS. A deliberate 150 ms post-return scheduling delay also passed both deadline checks. Candidate-bound CI and six-platform soaks passed. The separate Windows subagent reviewer failure from the preceding run remains unexplained; ten local acceptance repetitions passed. Do not release until the current matrix and required evidence pass.

Release evidence follow-up

The signed candidate-bound R266 record passed the six-platform and required safety-log validation. A separate offline runtime probe observed the same earlier update_plan result bytes on both the second and third native-provider requests. This fails R258 as written; a signed failed record is retained and cannot be waived. The provider-continuation/context approach requires a design decision before changing the runtime. The previously observed intermittent Windows cancellation and reviewer failures remain unexplained even though this candidate’s matrix passed. No release manifest or public release has been created.

@coderabbitai

coderabbitai Bot commented Oct 31, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@ghfind-review ghfind-review Bot added review: high ghfind author score; see https://ghfind.com area: providers LLM providers, models, retries, and streaming clients bug Something isn't working release-blocker Required acceptance gate that must pass before the release labels Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: providers LLM providers, models, retries, and streaming clients bug Something isn't working release-blocker Required acceptance gate that must pass before the release review: high ghfind author score; see https://ghfind.com

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant