Skip to content

Add experimental global-budget Ouro GRPO recipe - #4

Closed
0z5a wants to merge 3 commits into
feat/ouro-shared-providerfrom
feat/ouro-budgeted-grpo
Closed

0z5a wants to merge 3 commits into
feat/ouro-shared-providerfrom
feat/ouro-budgeted-grpo

Conversation

@0z5a

@0z5a 0z5a commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Incremental review: Files changed.

Direct dependency and ancestor review: #3 → #1. Baseline: cleaned #3 head 94129cedf3a91edd92d16bbd3499258b93590cff. This fork PR shows only this layer.

Adds a synchronous global-budget Ouro GRPO recipe using VIME's existing Megatron optimizer, loss, gradient reduction and checkpoint path. Each update uses one deterministic K on every DP rank; prefill, decode and training execute the same budget. Group identities and checkpoint schedule/data contracts are preserved.

Historical fixed-budget rollout companion: 0z5a/vllm-rlt#1. The public-contract follow-up is #5, consolidated upstream in #459.

Variant Mean step (s) Ratio vs K=4 / latency reduction Job (s) / GPU-h Training reward Truncated Mean response tokens
fixed4 44.526 1.000× / 0.00% 633.63 / 0.3520 1/96 73/96 436.1
fixed2 22.786 1.954× / 48.83% 499.86 / 0.2777 0/96 61/96 378.0
fixed3 33.568 1.326× / 24.61% 556.56 / 0.3092 1/96 70/96 428.5
mixed 33.415 1.333× / 24.96% 558.63 / 0.3104 1/96 78/96 450.6

2×H20, full Ouro-1.4B, six updates per arm, one seed, 16 completions/update and a 512-token cap. Step ratios include generation/scoring/training/publication, not evaluation or checkpoint saving. They do not establish equal-quality acceleration. Fixed4 finishes at 1/4 held-out success at K=4; every final mixed evaluation is 0/4. The mixed pilot has not established the quality/compute tradeoff needed to justify adding a bucket scheduler.

14 budget/normalization tests, provider/HF numerical checks, real two-rank schedule checks, tiny mixed-depth save/resume and full-model K=4 save/resume pass. All four full-model arms finish with complete checkpoints and normal process exit. Completed-task model files have since been cleaned up.

Test scripts, validation reports and logs were removed. Mixed-depth bucketing, learned stopping and time-to-target remain unestablished.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@0z5a

0z5a commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

Superseded by vllm-project#459, now merged into 0z5a-branch. This historical Ouro layer is included in the consolidated upstream integration; closing the duplicate PR.

@0z5a 0z5a closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant