Skip to content

agent 1.6.2: take the push off the poll loop, and two fixes that followed - #15

Merged
DorwardTech merged 1 commit into
mainfrom
sync/agent-1.6.2
Sep 25, 2026
Merged

DorwardTech merged 1 commit into
mainfrom
sync/agent-1.6.2

Conversation

@DorwardTech

@DorwardTech DorwardTech commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Summary

Sync from the monorepo (agent/), covering 1.6.0 → 1.6.2.

1.6.0 — the push was on the poll loop's critical path. pollLoop reset its timer only after deliver() returned, and deliver() blocks on an HTTPS round trip, so the real sampling period was interval + game-server round trips + central's latency:

case <-timer.C:
    if !poll() { return }              // collect() + deliver() both run here
    timer.Reset(a.nextPollInterval())  // ...only THEN does the clock start

A venue configured for a 1 s in-game rate measured 2.4 s in production, and no value of the setting could fix it — work time can only lengthen that period, never shorten it. Batches now go to a delivery goroutine that owns every push for the process's life and outlives every reconnect, which also keeps buffered telemetry draining while the agent cannot reach the box — exactly when the buffer matters most.

Separately, telemetryInterval() splits how often to look from how often to record. Fast polling while print-server work is in flight is a safety property and stays; it no longer drags the telemetry rate with it, which had made a nominal 15 s idle cadence average 7.9 s.

1.6.1 — a queued batch could be discarded without ever being sent. Moving delivery onto its own goroutine let a batch be enqueued while another was in flight. The buffer is bounded and drops oldest-first, so the batch being sent is the one capacity evicts, and the unconditional pop then removed the batch that had taken its place at the head. Nothing logged it, nothing retried it, and it could only happen while the buffer was full enough to be evicting — during an outage, when the queue is the only record there is. Buffer.PopSent now removes a batch only if it is still the one Peek handed out.

1.6.2 — the sample reporting a game had ENDED could be thrown away. The end of a game is the exact moment the recording interval widens, so the poll that discovers it was measured against the new, wider idle interval and rejected for arriving too soon. At 1 s in play / 30 s idle the state change could sit undelivered for most of a minute. A poll reporting a different server state is now always recorded; the exemption is one-shot, so the idle rate resumes immediately after.

Related issue

n/a — upstream sync.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Refactor / maintenance

Checklist

  • gofmt -l . prints nothing
  • go vet ./... passes
  • go test ./... passes
  • Tests added/updated for the change
  • .env.example and the README config table updated (if configuration changed) — no new configuration; IDLE_POLL_INTERVAL is pre-existing and already documented
  • No secrets or tokens committed
  • Change is focused (one logical change) — three releases in one sync, see below

Notes for reviewers

Not one logical change, deliberately. This is a mirror sync, so it carries 1.6.0 and the two fixes that review found in it rather than splitting them. The commit body separates the three.

Deployment note — order matters. IDLE_POLL_INTERVAL at or above 30 s crosses central's default Agent offline threshold, and config.go warns about it at startup. Raise that alert rule before raising the interval, or the site flaps offline in between. Leaving it unset keeps the 15 s default, which is safe.

New tests, each verified non-vacuous by reverting its fix:

  • TestDrainKeepsABatchEvictedWhileAnotherWasInFlight — reproduces the 1.6.1 loss exactly (central never received {"push_seq":2})
  • TestTheSampleThatEndsAGameIsNeverThrottled and TestAModeChangeBetweenGamesIsRecordedImmediately — the 1.6.2 transition
  • TestPopSentLeavesAHeadItDidNotSendAlone, TestPopSentStillDiscriminatesAfterALoad — the buffer contract, including that entry ids do not survive the spill file
  • plus the 1.6.0 cadence suite

Upstream also ran -race on the changed packages and the full monorepo suite.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Pi8roSs8BLnmFXgEgMVU7e


Generated by Claude Code

Summary by CodeRabbit

  • New Features
    • Telemetry continues sending in the background during server outages, without delaying polling.
    • Telemetry follows the active or idle cadence independently of polling. Slow polls and server-state changes are recorded promptly.
  • Bug Fixes
    • Prevented queued telemetry from being removed incorrectly when the queue changes during delivery.
  • Release
    • Updated the default agent version to 1.6.2.

…owed

Sync from the monorepo (agent/), covering 1.6.0 through 1.6.2.

1.6.0 — the push was on the poll loop's critical path. pollLoop reset its
timer only AFTER deliver() returned, and deliver() blocks on an HTTPS round
trip, so the real sampling period was `interval + game-server round trips +
central's latency`. A venue configured for a 1s in-game rate measured 2.4s in
production, and no value of the setting could fix it. Batches now go to a
delivery goroutine that owns every push for the process's life and outlives
every reconnect — which also means buffered telemetry keeps draining while the
agent cannot reach the box, exactly when the buffer matters most.

Separately, telemetryInterval() splits how often to LOOK from how often to
RECORD. Fast polling while print-server work is in flight is a safety property
and stays; it no longer drags the telemetry rate along with it, which had made
a nominal 15s idle cadence average 7.9s.

1.6.1 — a queued batch could be discarded without ever being sent. Moving
delivery onto its own goroutine let a batch be enqueued while another was in
flight; the buffer is bounded and drops oldest-first, so the batch being sent
is the one capacity evicts, and the unconditional pop then removed the batch
that had taken its place at the head. Nothing logged it and nothing retried
it, and it could only happen while the buffer was full enough to be evicting —
during an outage, when the queue is the only record there is. The drain now
removes a batch only if it is still the one it sent.

1.6.2 — the sample that reports a game has ENDED could be thrown away. The end
of a game is the exact moment the recording interval widens, so the poll that
discovers it was measured against the new, wider idle interval and rejected for
arriving too soon. At 1s in play and 30s idle the state change could sit
undelivered for most of a minute. A poll reporting a different server state is
now always recorded; the exemption is one-shot, so the idle rate resumes
immediately after.

Operators: IDLE_POLL_INTERVAL at or above 30s crosses central's default
"Agent offline" threshold, and config.go warns about it at startup. Raise that
alert rule before raising the interval, or the site flaps offline in between.

Gates: gofmt -l . clean, go vet ./... clean, go test ./... green, go build
./... clean, plus -race on the changed packages upstream.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pi8roSs8BLnmFXgEgMVU7e
@coderabbitai

coderabbitai Bot commented Sep 25, 2026

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 9bc3d18a-23b8-48ab-bac8-951abfb346fc

📥 Commits

Reviewing files that changed from the base of the PR and between d79df26 and 3763511.

📒 Files selected for processing (10)
  • .github/workflows/docker-image.yml
  • CHANGELOG.md
  • Dockerfile
  • docker-compose.yml
  • internal/app/app.go
  • internal/app/cadence_test.go
  • internal/app/deliver_test.go
  • internal/buffer/buffer.go
  • internal/buffer/buffer_test.go
  • internal/version/version.go
 ______________________________________________________________
< Good things come to those who commit early and review often. >
 --------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@DorwardTech
DorwardTech merged commit 3740e5d into main Sep 25, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant