Skip to content

feat(benchmarks): add end-to-end retrieval evaluation harness - #163

Open
AhmedIrfan7 wants to merge 1 commit into
grayhatdevelopers:mainfrom
AhmedIrfan7:feat/end-to-end-retrieval-evaluation
Open

feat(benchmarks): add end-to-end retrieval evaluation harness#163
AhmedIrfan7 wants to merge 1 commit into
grayhatdevelopers:mainfrom
AhmedIrfan7:feat/end-to-end-retrieval-evaluation

Conversation

@AhmedIrfan7

Copy link
Copy Markdown

What

Add a reproducible end-to-end evaluation harness that measures VidXP's complete public retrieval path in one place.

Closes #76.

Why

The dataset adapters (DiDeMo, HiREST) score a single modality against an official evaluator through the benchmark-ready core. Nothing evaluated the complete public product path — fusion, ranking across modalities, and evidence delivery — so a regression there was invisible to any reproducible measurement. Per the issue, existing component benchmarks are reused but do not substitute for evaluating the whole path.

What it adds

vidxp.benchmarks.end_to_end — a transport-neutral harness that drives the same public search operation a real client calls and reports the five measures the issue asks for:

Requested measure Metric
Relevant moments found recall_at_k at a relevance IoU (default 0.5)
Timestamp / temporal-range accuracy mean_top1_iou, per-case best_iou
Ranking quality across modalities mean_reciprocal_rank, IoU-graded ndcg_at_k, modality_contribution
Evidence supports the result evidence_support_rate over delivered evidence ranges
Latency, failures, degraded/partial latency_ms_*, failed_cases, no_result_cases, per-case evidence_degraded

Design. The evaluator is dependency-injected: a caller supplies a SearchFn mapping an EvaluationCase to a public FusedSearchResult. application_search_fn binds the real VidXPApplication.search into that shape, building the same SearchCommand a client sends — so the harness measures the public product path, not an internal shortcut. Because the core is injected, every metric is unit-tested with a deterministic fake and no models. A case whose search fails is recorded as failed and excluded from the quality metrics, so one broken query never hides the rest (application_boundary only wraps known errors, so the harness catches any exception to stay robust).

Scores keep VidXP's ordering-only meaning; calibrated scoring is a separate concern (#90).

Scope

This PR lands the tested core, the public-path adapter, and docs. Wiring the harness to a CLI / HTTP / MCP surface is intentionally left as a follow-up so the tested core lands first — application_search_fn already shows the exact binding a surface adapter needs. Happy to follow up with the CLI command if you'd prefer it in-scope.

Testing

  • ruff check src tests — clean.
  • python -m unittest discover -s tests — 667 passed, 7 skipped (env-gated ffmpeg/Pillow/symlink), 0 failed.
  • 12 new tests cover the metric functions (temporal IoU, IoU-graded nDCG, media-scoped matching), aggregate scoring, evidence support/degradation, failure isolation, no-result handling, parameter validation, and the public-command adapter. New tests are unittest.TestCase subclasses so CI's unittest discover collects them.

The dataset adapters (DiDeMo, HiREST) score one modality against an official
evaluator through the benchmark-ready core. Nothing evaluated the complete
public product path, so regressions in fusion, ranking, or evidence delivery
were invisible to a reproducible measurement.

Add vidxp.benchmarks.end_to_end, a transport-neutral harness that drives the
same public search operation a real client calls and reports, in one place,
the five measures the product path needs:

- whether relevant moments are found (recall_at_k at a relevance IoU),
- timestamp and temporal-range accuracy (mean_top1_iou, per-case best_iou),
- ranking quality across modalities (MRR, IoU-graded nDCG, per-modality
  contribution),
- whether delivered evidence supports the result (evidence_support_rate over
  evidence ranges),
- latency, failures, and degraded or partial evidence (latency percentiles,
  failed and no-result counts, per-case evidence degradation).

The evaluator is dependency-injected: callers pass a SearchFn mapping a case to
a public FusedSearchResult, so every metric is unit-tested with a deterministic
fake and no models. application_search_fn binds the real
VidXPApplication.search into that shape, building the same SearchCommand a
client sends. A case whose search fails is recorded as failed and excluded
from quality metrics so one broken query never hides the rest; application
boundaries only wrap known errors, so the harness catches any exception.

Scores keep VidXP's ordering-only meaning; calibrated scoring is a separate
concern. Wiring the harness to a CLI, HTTP, or MCP surface is intentionally a
follow-up so the tested core lands first.

Closes grayhatdevelopers#76
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add end-to-end retrieval evaluation

1 participant