feat(prometheus): stream active connections, termination status and bandwidth - #13796
Open
AlinsRan wants to merge 6 commits into
Open
feat(prometheus): stream active connections, termination status and bandwidth#13796AlinsRan wants to merge 6 commits into
AlinsRan wants to merge 6 commits into
Conversation
…andwidth A TCP or UDP proxy could not be monitored from the stream metrics we had. The only one was a per route connection counter, nginx reports a stream $status of 200 for every failure that happens after the upstream connection is established, and no byte counter of a live session is reachable from Lua, so a long-lived connection produced nothing until it ended. Add three metrics, keyed by listen_addr because a session can end before any stream route is matched and because the byte counters come from nginx, which only knows the listening address: - apisix_stream_active_connections, live concurrency per listening address - apisix_stream_status, one count per finished session. $stream_session_reason narrows 200 down to genuine closes and folds timeouts and resets onto the codes nginx itself uses; plugin rejections are carried on ctx because a stream plugin rejects by closing the session, which never sets $status - apisix_stream_bandwidth, bytes per direction and side, read once a second from the nginx zone so it grows while a connection is still open apisix_stream_metrics_zone, $stream_session_reason and $stream_listen_addr come from apisix-nginx-module 1.19.9, so .requirements moves to the first runtime carrying it. Everything degrades on an older runtime: the collector logs once and stays off, and the status metric falls back to $status.
dosubot
Bot
added
size:XL
ci/linux-install-openresty.sh guards the debug deb with a hardcoded version and sha256 per architecture, and bails out when .requirements moves without them. Every job that installs the runtime failed on that guard. Digests taken from the published 1.3.14 release assets.
The live-gauge case scraped the metrics endpoint while its session was open and found no series, although the later case asserting 0 on the same series passed -- so the collector had published, just not before this block scraped. Whether it has ticked by then depends on when the stream plugins finish loading, which is not what this case is about. Read the zone directly, which is where the count lives while the session is open; the published gauge stays covered by the case after it.
The zone was published by a per-second timer pinned to worker 0, because the bandwidth counter needs a total-to-delta conversion and the baseline sat in worker memory, which only one worker could own. Moving the baseline into the metric dict removes that constraint: every worker claims against the same value, so the conversion can run wherever the scrape lands. The range is claimed with an atomic incr so two concurrent scrapes cannot both count it -- the loser hands the excess back. Drops up to two seconds of staleness (one timer tick plus one counter sync), the gap where a reload lost the bytes between the old worker's last tick and the new worker's first, and the timer itself.
The zone backing the bandwidth and active connection metrics is declared in the stream block, and it is now read while the exposition is built. The scraping blocks were generated without a stream block, so the reload into them dropped the zone and the scrape had nothing to read: bandwidth never left its baseline and the gauge kept whatever the previous block left behind. Publishing used to happen on a timer in the stream subsystem, which ran in the blocks that did have one, so this only surfaced now.
The atomic claim clamped the baseline to the claiming reader's own zone sample, so a reader whose sample was older than a concurrent one pulled the baseline backwards and the overlapping bytes were counted twice on the next read. The rewind path had the same shape: it could not tell a recreated zone from a claim still in flight. Serialise the conversion instead, with an expiring key in the metric dict. A reader that loses the race publishes nothing and the delta it skipped is picked up by the next one; the gauge is unaffected because it carries no baseline. Also stop latching the zone probe. The zone comes and goes with the configuration, and caching a miss left a worker blind for good once it came back. The endpoint serves a cache the privileged agent refills on refresh_interval, so the tests pin that interval short: at the default 15s no block lives long enough to observe its own traffic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A TCP or UDP proxy cannot be monitored from the Stream metrics APISIX has today.
apisix_stream_connection_totalis a per-Route connection counter, and three things are missing behind it:$statusof 200 for every failure that happens after the upstream connection is established — every post-connect path ends inngx_stream_proxy_finalize(s, NGX_STREAM_OK)— so an idle timeout or a reset is indistinguishable from a normal close.$bytes_sentand friends only exist at log time, and Lua has no handle on a live session, so a long-lived connection reports nothing until it ends.This adds three metrics, all keyed by
listen_addrrather than by Route: a session can end before any Stream Route is matched, and the byte counters come from NGINX, which only knows the listening address.apisix_stream_active_connectionslisten_addrapisix_stream_statuscode,listen_addr,nodeapisix_stream_bandwidthlisten_addr,type,sidecodeonly ever takes values NGINX itself uses for Stream sessions — no synthetic codes.$stream_session_reasonnarrows 200 down to genuine closes and folds timeouts and resets onto 400/502. A plugin rejection is carried onctxinstead, because a Stream plugin rejects by closing the session (plugin.lua'srun_plugincallsngx_exit(1)), so the code it returned never reaches$status; it is mapped through the same whitelist so an operator-suppliedrejected_codecannot put an arbitrary value on the metric.apisix_stream_bandwidthandapisix_stream_active_connectionsare read once a second from an NGINX shared memory zone, so they keep moving while a connection is open. Zone size isnginx_config.stream.metrics_zone_size, default1m.Runtime dependency
apisix_stream_metrics_zone,$stream_session_reasonand$stream_listen_addrcome from apisix-nginx-module 1.19.9, so.requirementsmoves to1.3.14, the first APISIX-Runtime carrying it.Everything degrades on an older runtime rather than breaking: the collector logs once and stays off, and the status metric falls back to
$status.Which issue(s) this PR fixes
N/A
Checklist