Repository navigation
Record MiniCPM-o throughput search, quality gates and peer baselines - #51
amy-why-3459 wants to merge 44 commits into
Conversation
…tics Refs #49 Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Refs #49. Campaign remains INCONCLUSIVE pending held-out service and quality results. Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Reviewed head 3c1b2ff with focus on scoring provenance and merge integration. One P2 scorer-resume defect is noted inline; no evidence was found that the retained campaign actually exercised it.
Integration blockers are distinct from GPU results:
- Draft, mergeable=false: reconcile with main before readiness.
- Test-merge CI full-check fails inventory registration ('inventory directories differ: minicpmo45-l20x-20261021'; Python3.10:372 passed,9 skipped).
- Installed-package fails the hardcoded len(records)==90 assertion. The inventory machinery is absent from this PR head, so these failures arise when integrating with main, not proof that the standalone campaign failed.
- Register/classify the campaign in main's inventory framework without rewriting frozen artifacts; update installed-content expectations and rerun both Python versions. Admission/declaration checks are also currently red and need truthful resolution.
The FAIL/INCONCLUSIVE outcomes and unused reserved holdout are appropriately disclosed. Archiving negative results does not require achieving the campaign target. Review was source/evidence and CI inspection only, without service/GPU/audio rescoring/model calls.
| for i,w in zip(ids,wavs): | ||
| audio,sr=sf.read(w) | ||
| assert len(audio)>0 and np.isfinite(audio).all() | ||
| doc['rows'].setdefault(i,{}).update({'wav_sha256':hashlib.sha256(Path(w).read_bytes()).hexdigest(),'duration_s':len(audio)/sr}) |
There was a problem hiding this comment.
[P2] Reject stale scoring output before replacing audio provenance
If --out exists, completed task flags are loaded, then each row's WAV hash/duration is overwritten with the new inputs before completed WER/SIM/UTMOS tasks are skipped. Reusing an output path with the same prompt IDs and different WAVs silently binds old scores to new audio hashes. Verify IDs, audio hashes and scorer provenance before resuming, or explicitly invalidate/recompute affected scores before updating provenance. Add a changed-WAV resume regression. Preserve historical frozen scripts/results via a versioned successor where needed.
There was a problem hiding this comment.
Addressed in f62aebd without changing the historical score_outputs.py bytes: the successor score_outputs_v2.py verifies prompt ID, WAV hash, duration and scorer file hash before reusing or skipping a completed item, and refuses (leaving outputs untouched) on mismatch. Covered by tests/test_minicpm_score_resume.py (WAV swapped before resume). Also merged main (4b72141, 5d43495, cdfcd88, 772e8c1) to integrate the inventory/installed-knowledge changes from the review; head is now 772e8c1.
There was a problem hiding this comment.
Scoped, non-voting recheck at head 772e8c17110b268608f8caee4977eec38bc4e606: the changed-WAV resume finding is addressed in the versioned score_outputs_v2.py successor.
It compares prompt ID sets, WAV SHA-256/duration and scorer-directory file hashes before output writes or completed-task skips. The regression controls cover changed-WAV rejection with the existing output bytes untouched, ID/duration/scorer mismatches, matching-score preservation and changed scorer weight bytes. The historical score_outputs.py remains byte-identical to the previously reviewed revision.
The Ready CI passed both Python versions and installed-package checks; both full suites report 3,550 passed, 13 skipped and 284 subtests. Logs identify tested merge f88216bfd36e1d8884117423607ed4972b88f6bf, combining this head with 663f3b327998526a13aa31e72ad729025c18983c (also the API-reported base). This was not a test against later main 9dded7ad.
This clears the specific resume-binding defect in the successor only. The current-head knowledge-author declaration remains noncompliant. I did not execute the scorer, recompute historical scores, run local tests or perform model/device work. Throughput, speech quality and whole-PR/campaign acceptance remain outside this clearance.
Dong1017
added
the
area:workbench
ChatGPT Codex Connector
|
Size note before this PR is marked ready (Refs #273, F5/R6) This PR would add 13.3 MB in 300 files. Most of them are under #273 R6 proposes keeping large raw artifacts outside Git as GitHub Release assets, with an accessible URL and SHA-256 recorded in the repository. Byte budgets are still under discussion in #273. Before marking this PR ready, please consider moving the raw result files to a Release asset and keeping the manifest, summaries and gate results here. No other change is requested. |
Keep historical validator/score_outputs.py frozen. Successor score_outputs_v2.py checks prompt IDs, WAV hashes, durations and scorer-file provenance before writing provenance or skipping a completed task, and a changed-WAV regression shows a reused output is left untouched. Review: #51 (comment) Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: amy-why-3459 <amy-why-3459@users.noreply.github.com>
Keep both peer scopes, deepinfer and minicpmo_single_gpu_engine, and append the Issue #49 matrix notes after main's sections. Generated skill abstracts follow main in this merge and are refreshed from this commit next. Review: #51 (review) Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: amy-why-3459 <amy-why-3459@users.noreply.github.com>
Classify minicpmo45-l20x-20261021 as a campaign. Its manifest-named records are descriptive rather than digest maps, so inventory does not forge or recheck embedded hashes. The installed catalog counts 128 knowledge records once the peer card is kept. The two MiniCPM cases cite their typed records so the merged knowledge contract can resolve them. Review: #51 (review) Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: amy-why-3459 <amy-why-3459@users.noreply.github.com>
Regenerate packaged skill abstracts so source revisions and content hashes match the merge, including the MiniCPM case records. Review: #51 (review) Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: amy-why-3459 <amy-why-3459@users.noreply.github.com>
The merged matrices and case records change the coverage input digest. Regenerate the view; no coverage row was added or promoted. Review: #51 (review) Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: amy-why-3459 <amy-why-3459@users.noreply.github.com>
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision 323db123b66a83a5df1c97a34dc1739eabb38dbe --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy. Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md","source_revision":"323db123b66a83a5df1c97a34dc1739eabb38dbe","status":"ready"}Provider/settings: Uncovered / unsupported paths (341)
|
|
Following up on the size note (Refs #273 R6) Thanks for the versioned The size question from 2026-11-01 is still open, and the PR has grown since: 300 files then, 340 now at To restate it briefly: merging adds about 13 MB to Git history permanently. A later deletion does not recover it, and every clone pays for it — #193 and #195 are the precedent. Three ways this can land, and any of them unblocks the PR:
What I cannot do is decide it for you: you own this campaign's evidence, and I would be guessing at which files are load-bearing for replay. #273 R6 has been waiting on this thread and on #113 since 2026-11-01, and it is the last open item in that review. A short answer — even "option 2, because X" — closes it. No change to the evidence, verdicts or this PR's scope is requested here. |
|
R6 is decided, and the part that concerns this PR is yours to choose. The record is #273 issuecomment-6632466101; #273 is now closed with every remedy decided. What it means here:
Measured today at this PR's head, so the numbers are current rather than the 2026-10-31 figures in F5:
No existing evidence file grows; these are all new files. |
… conflicts with #354 Keep both appended matrix sections: main's vLLM Ascend v0.27.1rc1 notes and this branch's Issue #49 conditioning-transport notes. The installed catalog counts 129 knowledge records because each side added one distinct record. Generated skill abstracts follow main in this merge and are refreshed from this commit next. Skill coverage is regenerated from the merged matrices and registry. Agent-assisted-by: Cursor / grok-4.7 (conflict resolution) Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: amy-why-3459 <amy-why-3459@users.noreply.github.com>
Regenerate packaged skill abstracts so source revisions and content hashes match the merge commit, including the vLLM Ascend skill and the MiniCPM case records. Agent-assisted-by: Cursor / grok-4.7 (conflict resolution) Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: amy-why-3459 <amy-why-3459@users.noreply.github.com>
|
已把 main（6fefa010）合并进 |
Refs #49.
Records the search for higher MiniCPM-o 4.5 throughput on vLLM-Omni, with pinned vLLM-Omni, SGLang-Omni and single-GPU peer baselines on L20X. Reports concurrency1/2/4/8/16/32, delivered audio frames/s, TTFA p50/p95, mean RTF and native speech-quality scoring. No candidate has achieved the peer-throughput and speech-quality acceptance target; this PR remains draft.
Campaign evidence and reproduction commands preserve successful measurements, startup failures, rejected numerical/quality results, sanitized request records and implementation patches. The matched64-prompt peer reaches C16/C32 1493.4/2544.1 frames/s; the separate320-request repeated baseline remains available. Explored changes include stage replication, selective CFM precision, speaker graphs, graph coverage and audio-window geometry. Larger windows explicitly report their first-audio latency cost. Profiled requests are excluded from throughput aggregates, including the invalidated high-rate CPU sampling attempt.
The latest experimental vLLM source packs ordinary-turn conditioning with the existing tensor serializer. It preserves legacy float32 values while avoiding scalar-list construction and traversal. Its synthetic CPU transport microbenchmark and213 passing regression tests establish component behavior only;25/50-frame service and quality screens are pending. The earlier CPU-reference speaker implementation remains separately scoped in vLLM-Omni #9188.
All previous strict and noninferiority quality FAIL verdicts remain unchanged. Round-two final quality uses pinned updated main and256 unused paired prompts, with three repetitions aggregated within each prompt before bootstrapping256 clusters. All nine one-sided95% upper degradation bounds must meet WER+0.20 percentage points, SIM×100 drop0.50 and UTMOS drop0.05. This repeat design was frozen before any reserved holdout generation; those prompts remain unused. No global-optimum or combined service/quality PASS claim.
The knowledge package adds the pinned peer scope and a speaker-conversion case, with the generated index refreshed from committed source. Validation includes267 scaffold tests, Ruff, index consistency and wheel lookup from the earlier package change; these are knowledge-delivery checks. New evidence validators pass8 clustered-quality controls and10 performance-runner preflight controls. Raw audio, generated transcripts and private host logs remain excluded.