Skip to content

feat: add per-artifact verification result metric - #3049

Merged
fseldow merged 2 commits into
notaryproject:mainfrom
charleswool:feat/verification-result-metric
Oct 22, 2026
Merged

fseldow merged 2 commits into
notaryproject:mainfrom
charleswool:feat/verification-result-metric

Conversation

@charleswool

Copy link
Copy Markdown
Contributor

Description

The v2 provider exports three metrics today: ratify_verification_request and ratify_mutation_request (latency histograms with no labels) and ratify_system_error_count (#2873). None of them says what the verdict was, so an operator cannot answer "what share of images is being denied?" or alert on a spike in verification errors without parsing logs.

Change

  • New counter ratify_verification_result_count, incremented once per verified artifact with:
    • result: a closed set, succeeded | failed | error. failed means verification completed and the artifact did not satisfy the policy (e.g. unsigned / untrusted); error means no verdict could be reached (no executor configured, registry failure, etc.). Exported as metrics.VerificationResult* constants.
    • cache_hit: whether the outcome was served from the verify cache.
    • workload_namespace: as on the other Ratify metrics.
  • Reported at the same point the outcome is logged (feat: make the provider request path observable #2873), so cache hits and singleflight-deduplicated callers are counted exactly like the log line, and a request that fails before any artifact is processed (bad body) is not counted.
  • Per-artifact metrics now get the real namespace. workload_namespace was always empty in v2 because nothing called SetContextWithNamespace. The handler now builds a metrics-only context from the [namespace] prefix of the Gatekeeper key. This also fills the label on the existing ratify_system_error_count{error="verify_artifact"}. The executor still receives the original request context.

Label cardinality stays bounded: 3 × 2 × number of namespaces. No image reference, digest or error text is used as a label.

Example:

ratify_verification_result_count_total{cache_hit="false",result="failed",workload_namespace="team-a"} 3
ratify_verification_result_count_total{cache_hit="true",result="succeeded",workload_namespace=""} 41
ratify_verification_result_count_total{cache_hit="false",result="error",workload_namespace=""} 1

Which issue(s) does this PR resolve?

None filed; follow-up to #2873 ("outcome labels on request histograms ... left as a follow-up").

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update (the metrics table on ratify.dev)

Testing and verification

  • TestVerify_ReportsResultMetric: cache hit succeeded, cache hit failed with a namespaced key, cache miss with error. Each asserts exactly one report with the expected result, cache_hit and namespace.
  • TestVerify_InvalidRequestReportsNoResult: a malformed body reports nothing.
  • TestVerificationOutcome, TestReportVerificationResult and TestReportVerificationResult_Uninitialized: attribute mapping, plus a no-op when metrics are disabled.
  • The reporter is a package var in internal/httpserver, the same pattern as initMetricsFunc, so tests stay hermetic.
  • go build ./..., go vet, go test for internal/httpserver, pkg/metrics and cmd/..., and golangci-lint v2.12.2 (./..., 0 issues) pass.

Checklist

  • Does the affected code have corresponding tests?
  • Are the changes documented? The ratify.dev instrumentation table and the Grafana dashboards under instrumentation/ will follow separately.
  • Does this introduce breaking changes that would require an announcement or bumping the major version? No.
  • Do all new files have appropriate license header? No new files.

Copilot AI balanced review requested due to automatic review settings October 22, 2026 06:21
@github-actions github-actions Bot added the v2 label Oct 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The additive instrumentation preserves verification behavior, has focused test coverage, and has no identified blocking issues.

No open issues 🎉

What changed in this PR

Adds per-artifact verification outcome metrics to the v2 provider, extending request-path observability without changing verification behavior.

Changes:

  • Adds ratify_verification_result_count with result, cache-hit, and workload-namespace labels.
  • Reports outcomes alongside existing logs and supplies namespace context to per-artifact metrics.
  • Adds tests for outcome mapping, cache hits, verification errors, malformed requests, and disabled metrics.
File Description
pkg/​metrics/​stats_reporter.go Defines and reports the verification outcome counter.
pkg/​metrics/​stats_reporter_test.go Tests counter attributes and uninitialized behavior.
internal/​httpserver/​handlers.go Reports artifact outcomes with namespace-aware metrics context.
internal/​httpserver/​handlers_test.go Tests reporting paths and outcome classification.

Review effort: Balanced

@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.21053% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.44%. Comparing base (088f7be) to head (445f91b).

Files with missing lines Patch % Lines
pkg/metrics/stats_reporter.go 70.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3049      +/-   ##
==========================================
+ Coverage   85.32%   85.44%   +0.12%     
==========================================
  Files          53       53              
  Lines        2868     2886      +18     
==========================================
+ Hits         2447     2466      +19     
+ Misses        326      324       -2     
- Partials       95       96       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread internal/httpserver/handlers.go Resolved
Comment thread internal/httpserver/handlers.go Resolved
Comment thread pkg/metrics/stats_reporter.go Resolved
charleswool and others added 2 commits October 22, 2026 23:17
Add ratify_verification_result_count, a counter incremented once per
verified artifact with a bounded result label (succeeded, failed, error)
and a cache_hit label. It is reported at the same point the outcome is
logged, so cache hits and singleflight-deduplicated callers are counted.

Per-artifact metrics now carry the workload namespace parsed from the
Gatekeeper key, which was previously always empty.

Signed-off-by: Charles Wu <yuewu2@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add cache-miss succeeded/failed cases to TestVerify_ReportsResultMetric so the non-cached result reporting branch is exercised, and remove comments flagged as unnecessary in review.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Charles Wu <yuewu2@microsoft.com>
@charleswool
charleswool force-pushed the feat/verification-result-metric branch from 768382a to 445f91b Compare October 22, 2026 12:18
@fseldow
fseldow merged commit 9d1f75a into notaryproject:main Oct 22, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants