Skip to content

[6/6] Attribute requests to a caller and a client tool - #393

Merged
andrew merged 6 commits into
git-pkgs:mainfrom
wickedOne:analytics-6-source-attribution
Oct 2, 2026
Merged

andrew merged 6 commits into
git-pkgs:mainfrom
wickedOne:analytics-6-source-attribution

Conversation

@wickedOne

@wickedOne wickedOne commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Client and request-source attribution

Part 6 of six splitting #381 ("analytics dashboard") into reviewable pieces, as the review there asked for -- this is the half it singled out as needing the most thought, now out from behind 3,000 lines of dashboard JSON. #381 can be closed once this lands.

Series: #388 route coverage · #389 ecosystem stats · #390 analytics page · #391 /stats breakdown · #392 Grafana dashboard · #393 source attribution

Needs #389, #390 and #392, and branches off #392, so until the chain merges the Files changed tab here shows those too. The last commit is this part.

Package managers do not say who invoked them. A request from pip or go carries a Host, an Accept and a User-Agent -- no Referer, no originating URL, nothing naming a repository, pipeline or job. So the proxy attributes requests by the two things always present: the caller's address, and the tool name taken from the leading User-Agent token.

Client names are exported as proxy_client_requests_total{client} and proxy_client_response_bytes_total{client} because they come from a closed set -- a User-Agent is attacker-controlled, so anything unrecognised collapses to other rather than minting a time series per request. Addresses are not exported as labels at all: the caller set is unbounded and outside the proxy's control. Per-address detail lives in the access log and, when enabled, on the page.

ui_request_sources, off by default

The review's point: the sources card publishes caller IP addresses, tool names and per-caller volumes on /ui/analytics, and the proxy has no authentication of its own, with gating deferred to a reverse proxy (#123). Until now /ui exposed what was cached, not who called.

So the card is behind a new top-level ui_request_sources, defaulting to false, and the README says plainly what it exposes and that /ui is unauthenticated. The tracker still runs when the flag is off, so the Prometheus client metrics are unaffected -- only the addresses are withheld. Tests cover both states end to end.

trust_forwarded_for moves to the top level

Also from the review: access_log.trust_forwarded_for reads narrower than it acts, since it drives the structured log and the analytics table as well as the access log. It is now top-level trust_forwarded_for / PROXY_TRUST_FORWARDED_FOR. Nothing has been released with the old key.

The header is honoured only when the flag is set, and the value must parse as an IP: a load balancer that appends leaves the leftmost entry caller-controlled, and that string becomes a map key held for the process lifetime and a line in the access log.

Log fields

remote keeps its meaning. #381 changed it from host:port to host-only; per the review it stays as the TCP peer and a new remote_ip carries the attributed address, matching what accesslog.Entry already does with RemoteAddr/RemoteIP. The access log gains remote_ip, user_agent, client, ecosystem and bytes.

The table

It tracks 200 callers, evicting the least recently seen once full. Evicting rather than refusing matters in the deployment this is for: containerised CI gives every job a fresh address, so a refusing table would freeze on whoever arrived first.

The overflow row reports two counts separately -- callers currently tracked below the display cut, and fold-ins from eviction -- because only the first is exact. A caller evicted, returning and evicted again counts twice, and counting distinct callers instead would mean keeping every key ever seen, which is the unbounded set the limit exists to avoid.

@wickedOne wickedOne changed the title Analytics 6 source attribution Attribute requests to a caller and a client tool Oct 1, 2026
@wickedOne wickedOne changed the title Attribute requests to a caller and a client tool [6/6] Attribute requests to a caller and a client tool Oct 1, 2026
wickedOne and others added 5 commits October 1, 2026 16:15
- Add GetEcosystemStats, aggregating packages, versions, artifacts,
  cache size, downloads and downloaded bytes per ecosystem
- Publish six proxy_ecosystem_* gauges from it on the existing
  one-minute cache-stats tick
- Set and selectively delete rather than Reset, so no scrape lands on
  a half-populated vector
- Report artifacts with no package row under "unattributed" rather
  than dropping them, so the figures still add up

Part 2 of 6 splitting git-pkgs#381 up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Ring of download size by ecosystem, cache figures, per-ecosystem
  table, vulnerability overview and a Runtime card
- Add metrics.Gather so the page can render counters that were never
  in the database
- Add proxy_response_bytes_total and response-writer byte counting
- Serve a retained snapshot behind a staleness banner when the
  aggregation fails, rather than rendering old figures as current
- Extract the security overview into a shared component

Part 3 of 6 splitting git-pkgs#381 up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add downloaded_bytes, downloads and an ecosystems array, served from
  the snapshot the gauges and the page already share
- Add stats_unavailable so a failed aggregation is distinguishable
  from an idle proxy
- Regenerate the OpenAPI spec

Part 4 of 6 splitting git-pkgs#381 up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- 33 panels, no hardcoded data source UID
- Three ecosystem variables, because the label means three different
  things across /metrics: the package record, the request path, and
  the handler's own name

Part 5 of 6 splitting git-pkgs#381 up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Track per-caller request and byte counts in a bounded table
- Export proxy_client_requests_total and
  proxy_client_response_bytes_total over a closed label set; addresses
  are never labels
- Gate the sources card behind ui_request_sources, off by default,
  since /ui carries no authentication of its own
- Move access_log.trust_forwarded_for to a top-level key
- Keep remote as the TCP peer and add remote_ip alongside it

Part 6 of 6 splitting git-pkgs#381 up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@wickedOne
wickedOne force-pushed the analytics-6-source-attribution branch from 6916614 to 3655000 Compare October 1, 2026 14:19
The flag exists to withhold caller addresses. The by-client table carries
none, and the same figures are already public at /metrics, so gating it
left two metrics with no tile on the page in the default config.
@andrew
andrew merged commit e7817dc into git-pkgs:main Oct 2, 2026
6 checks passed
@wickedOne
wickedOne deleted the analytics-6-source-attribution branch October 2, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants