feat(benchmarks): add end-to-end retrieval evaluation harness - #163
Open
AhmedIrfan7 wants to merge 1 commit into
Open
feat(benchmarks): add end-to-end retrieval evaluation harness#163AhmedIrfan7 wants to merge 1 commit into
AhmedIrfan7 wants to merge 1 commit into
Conversation
The dataset adapters (DiDeMo, HiREST) score one modality against an official evaluator through the benchmark-ready core. Nothing evaluated the complete public product path, so regressions in fusion, ranking, or evidence delivery were invisible to a reproducible measurement. Add vidxp.benchmarks.end_to_end, a transport-neutral harness that drives the same public search operation a real client calls and reports, in one place, the five measures the product path needs: - whether relevant moments are found (recall_at_k at a relevance IoU), - timestamp and temporal-range accuracy (mean_top1_iou, per-case best_iou), - ranking quality across modalities (MRR, IoU-graded nDCG, per-modality contribution), - whether delivered evidence supports the result (evidence_support_rate over evidence ranges), - latency, failures, and degraded or partial evidence (latency percentiles, failed and no-result counts, per-case evidence degradation). The evaluator is dependency-injected: callers pass a SearchFn mapping a case to a public FusedSearchResult, so every metric is unit-tested with a deterministic fake and no models. application_search_fn binds the real VidXPApplication.search into that shape, building the same SearchCommand a client sends. A case whose search fails is recorded as failed and excluded from quality metrics so one broken query never hides the rest; application boundaries only wrap known errors, so the harness catches any exception. Scores keep VidXP's ordering-only meaning; calibrated scoring is a separate concern. Wiring the harness to a CLI, HTTP, or MCP surface is intentionally a follow-up so the tested core lands first. Closes grayhatdevelopers#76
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add a reproducible end-to-end evaluation harness that measures VidXP's complete public retrieval path in one place.
Closes #76.
Why
The dataset adapters (DiDeMo, HiREST) score a single modality against an official evaluator through the benchmark-ready core. Nothing evaluated the complete public product path — fusion, ranking across modalities, and evidence delivery — so a regression there was invisible to any reproducible measurement. Per the issue, existing component benchmarks are reused but do not substitute for evaluating the whole path.
What it adds
vidxp.benchmarks.end_to_end— a transport-neutral harness that drives the same publicsearchoperation a real client calls and reports the five measures the issue asks for:recall_at_kat a relevance IoU (default 0.5)mean_top1_iou, per-casebest_ioumean_reciprocal_rank, IoU-gradedndcg_at_k,modality_contributionevidence_support_rateover delivered evidence rangeslatency_ms_*,failed_cases,no_result_cases, per-caseevidence_degradedDesign. The evaluator is dependency-injected: a caller supplies a
SearchFnmapping anEvaluationCaseto a publicFusedSearchResult.application_search_fnbinds the realVidXPApplication.searchinto that shape, building the sameSearchCommanda client sends — so the harness measures the public product path, not an internal shortcut. Because the core is injected, every metric is unit-tested with a deterministic fake and no models. A case whose search fails is recorded as failed and excluded from the quality metrics, so one broken query never hides the rest (application_boundaryonly wraps known errors, so the harness catches any exception to stay robust).Scores keep VidXP's ordering-only meaning; calibrated scoring is a separate concern (#90).
Scope
This PR lands the tested core, the public-path adapter, and docs. Wiring the harness to a CLI / HTTP / MCP surface is intentionally left as a follow-up so the tested core lands first —
application_search_fnalready shows the exact binding a surface adapter needs. Happy to follow up with the CLI command if you'd prefer it in-scope.Testing
ruff check src tests— clean.python -m unittest discover -s tests— 667 passed, 7 skipped (env-gated ffmpeg/Pillow/symlink), 0 failed.unittest.TestCasesubclasses so CI'sunittest discovercollects them.