Skip to content

Add the keygrabber sample, sink contract, retry queue and InfluxDB sink - #32

Merged
mikelangmayr merged 1 commit into
mainfrom
mike/keygrabber-sink
Sep 16, 2026
Merged

mikelangmayr merged 1 commit into
mainfrom
mike/keygrabber-sink

Conversation

@mikelangmayr

Copy link
Copy Markdown
Collaborator

First slice of the keygrabber: the storage boundary only. No daemon, no polling and no config yet, so nothing here runs on its own.

  • Add libby/keygrabber/ with Sample (keyword, group, peer, value, units, timestamp) and a Sink protocol of connect / is_connected / write / close. Nothing in Sample is Influx-shaped, so a second backend is a new sink rather than a change to the collector
  • write returns how many samples it stored, which can be fewer than it was handed, and raises SinkWriteError when a batch failed and is worth retrying
  • Add RetryingWriter, which wraps any sink and holds failed batches in a bounded queue with exponential backoff, dropping oldest when full so a database that stays down cannot grow memory without limit. It starts no thread and never sleeps: the caller drives flush_due() and the clock is injectable, so the backoff tests run in a millisecond
  • Fresh batches queue behind an existing backlog rather than overtaking it, and a retry that fails again stops the drain instead of hammering the backend
  • Add InfluxSink for InfluxDB 2.x: one measurement per keyword name, tags group / peer / units, single value field, whole batch as one request
  • Skip null values rather than raising. lasterror is nullable on every daemon and None most of the time, so a collection reading a whole peer would otherwise fail on every tick. Confirmed live against a daemon: a % read returns lasterror as {"ok": true, "value": null}
  • Write ints as floats, so one peer reporting 0 and another 0.5 for the same keyword cannot collide as int against float and have the write rejected. bool is checked first, being a subclass of int
  • Give absent units a none placeholder, since Influx drops an empty tag value and a keyword would otherwise split across two series depending on whether it declared units
  • Keep the Influx token out of InfluxConfig.__repr__, so a traceback or a logged config cannot leak it
  • Add an influxdb extra and install it in the tox test env, so the sink's tests run in CI instead of skipping. They skip cleanly when it is absent
  • Add libby.keygrabber.sink to the API reference; the Influx module is left out because it imports its client at module scope and the docs build does not install the extra
  • 28 sink tests, pylint 10.00/10, and the -W docs build verified locally. The write path is covered against an injected write API and the point mapping against the real client's line-protocol serializer; no live InfluxDB server was exercised

@prkrtg prkrtg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mikelangmayr
mikelangmayr merged commit 048a5b2 into main Sep 16, 2026
3 checks passed
@mikelangmayr
mikelangmayr deleted the mike/keygrabber-sink branch September 16, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants