[2/6] Per-ecosystem cache and download statistics

[wickedOne]

Per-ecosystem cache and download statistics

Part 2 of six splitting #381 ("analytics dashboard") into reviewable pieces, as the review there asked for. #381 stays open as the umbrella and reference thread until the last of these lands; it is not meant to be merged.

Series: #388 route coverage · #389 ecosystem stats · #390 analytics page · #391 /stats breakdown · #392 Grafana dashboard · #393 source attribution

Independent of the other five. #390, #391, #392 and #393 all build on it.

/metrics reported cache size and artifact count as single numbers with no breakdown, and the database had the figures to break them down all along: artifacts.hit_count x artifacts.size, summed per ecosystem, is the bytes the proxy has actually handed to clients from cache.

GetEcosystemStats() returns per-ecosystem packages, versions, artifacts, cache size, downloads and downloaded bytes. It is several grouped aggregations merged in Go rather than one join, so ecosystems with nothing cached still appear and the artifacts table stays optional for a database inherited from git-pkgs. Six gauges are published from it on the existing one-minute cache-stats tick, so the added background cost is a multiple of work that already happened rather than something new.

Two details worth a look:

Artifacts whose version or package row has gone missing are reported under unattributed rather than dropped, so the per-ecosystem figures still add up to proxy_cache_size_bytes and proxy_cached_artifacts_total. A non-zero unattributed is a signal about the database, not noise.

The gauges are Set and selectively deleted rather than Reset and repopulated. Reset followed by a repopulating loop leaves a window in which a scrape sees the families empty or half filled, which renders as a gap on any panel built from them. Rows are summed per label first, because normalizing collapses aliases and a database holding both gem and rubygems rows arrives as two rows belonging to one label.

These are gauges rather than counters because eviction clears an artifact's size and drops its past hits out of the total, so the figure can step downwards. The README says so, and says to chart them with max_over_time rather than increase.

New metrics

proxy_ecosystem_downloaded_bytes, proxy_ecosystem_artifact_downloads, proxy_ecosystem_cache_size_bytes, proxy_ecosystem_cached_artifacts, proxy_ecosystem_packages, proxy_ecosystem_versions, all labelled ecosystem.

Two of these were proxy_downloaded_bytes and proxy_artifact_downloads in #381. Renamed to share the prefix with the other four, per the review there -- nothing is published yet, so it costs nothing now and would be a break later.