Skip to content

feat(lib): count publisher sends and errors, skip failing publishers - #712

Open
arsenovicAtNominal wants to merge 2 commits into
mainfrom
issue-709-publisher-stats
Open

arsenovicAtNominal wants to merge 2 commits into
mainfrom
issue-709-publisher-stats

Conversation

@arsenovicAtNominal

@arsenovicAtNominal arsenovicAtNominal commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Second half of #709. Stacked on #713, which adds the notebook and the docs build that runs it, so review that first. Once #713 merges, change this PR's base to main.

  • Publish counting: Instrument.publish() counts what each publisher does with every Measurement and Command.
  • Failing publishers: a publisher that raises is logged, counted and skipped.
    • The first failure of each publisher is logged with its traceback; later failures are summarized at most once a minute.
    • The publishers after it still receive the data, and the instrument method returns normally.
    • Behaviour change: previously the error reached the caller, and later publishers missed that sample.
  • Instrument.publisher_stats: one PublisherStats snapshot per publisher: sent, errors, last_error, last_sent_ns, last_error_ns.
  • Instrument.channel_stats: the latest value, count and rate (state.rate(5.0)) of every channel, commands included. It doesn't need start().
  • Notebook card:
    • The Publishers section is a table of sent, errors and last error.
    • The Channels section comes from these counters.
    • A badge shows the error count.
  • ChannelState and MonitorState move unchanged from the feat(monitor): live terminal monitor as a publisher target #510 branch (issue-510-monitor-publisher, instro/lib/consumers/monitor/server.py) into instro/lib/stats.py. Once this merges, feat(monitor): live terminal monitor as a publisher target #510 can import them from there instead of keeping its own copy.

Demo notebook. #713 adds examples/notebooks/notebook_display.md. This PR adds to it a "A failing publisher" section, which uses a deliberately failing publisher, and calls to publisher_stats and channel_stats. It still needs no hardware or credentials.

Type of change

  • New feature (feat)

Verification

  • just check-python passes, and so does uv run pytest tests except tests/ethernetip, which I couldn't run because the locally built EtherNet/IP extension fails to load on my machine (unrelated to this change).
  • The strict docs build passes.
  • Notebook check: just test-notebooks passes.
    • All 13 code cells run; I checked that the MyST file converts to 13 code cells, not Markdown.
    • The recipe fails (exit 1) on a test notebook whose cell raises.
    • just notebook serves notebook_display.md as a notebook.
  • Overhead: Instrument.publish() with an 8-channel measurement and 3 publishers went from 1.47 µs (plain loop) to 3.12 µs per call. That's small next to instrument I/O, so the stats are always on.

Tests

  • Unit tests added or updated
    • tests/lib/test_publisher_stats.py:
      • a failing publisher is counted and logged once, and the next publisher still gets the data
      • channel stats cover commands without start()
      • repeated start()/stop() doesn't leak counters for dropped buffers
    • tests/awg/test_awg_config_software.py:
      • test_open_failure_in_a_publisher_drops_the_configured_channel asserted the old behaviour, where a publisher error aborts open(). It is now test_a_failing_publisher_does_not_fail_open.
      • Rollback on a driver failure stays covered by test_open_failure_after_a_channel_succeeds_drops_the_configured_channels.

Checklist

  • PR title follows Conventional Commits
  • Documentation updated: new "Publisher errors and stats" section in docs/library/publishers.md, the notebook-display guide, and the instro.lib.stats entries in docs/ref/library/instrument.md
  • Code follows the style/conventions of the surrounding code

Notes for reviewers

🤖 Generated with Claude Code

@greptile-apps

greptile-apps Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[High impact] The shared instrument publishing path changes across the library.

The PR appears safe to merge, with non-blocking fixes recommended for fast-channel rates and buffer cleanup during concurrent stats reads.

Findings

  1. P2 Fast channels show low rates ▶
  2. P2 Stopped buffers stay in memory ▶
Fix with agent prompt
### Issue 1
instro/lib/stats.py:33
`arrivals` keeps only 512 updates, even when they all fall within the requested window. A channel receiving 1,000 single-sample updates per second therefore reports at most 102.4/s from `rate(5.0)`, rather than 1,000/s. This affects both `channel_stats` and the notebook card.

Keep counts for the full time window, using time buckets if memory must stay bounded. Add a test that exceeds 512 updates within that window.

### Issue 2
instro/lib/instrument.py:200-204
`_publisher_counter()` can recreate a buffer's counter after `stop()` removes it. For example, `publisher_stats` can copy `publishers`, pause while another thread finishes `stop()`, then resume and create a counter for the old buffer.

That counter keeps the closed buffer alive. Later stops only remove the new buffer's counter, so repeated restarts can accumulate unused buffers. Coordinate counter creation with publisher removal so an older reader cannot restore a removed entry.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds publisher send/error counts, isolates publisher failures, and exposes channel counts and rates in Python and notebook cards.

  • Publisher failures no longer stop other publishers, and their results are counted.
  • Instrument callers can inspect channel values and rates without starting the daemon.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Instrument.publish(data)"] --> B["Update channel stats"]
    B --> C["Next publisher"]
    C --> D["publisher.publish(data)"]
    D -->|Returns| E["Count sent"]
    D -->|Raises| F["Count error and log"]
    E --> C
    F --> C
Loading

Reviews (1) · Last reviewed commit: "feat(lib): count publisher sends and err..." · Reviewed by Greptile

Comment thread instro/lib/stats.py Resolved
Comment thread instro/lib/instrument.py Resolved
@arsenovicAtNominal

Copy link
Copy Markdown
Collaborator Author

Heads-up from #711 review changes (20797ec). Rebasing onto issue-709-notebook-cards will conflict in three files:

  • instro/lib/publishers/nominal_core.py: the first argument is now dataset: str | Dataset. dataset_rid= stays as a keyword alias, and client= / base_url= are gone (fetch the Dataset with your client instead). Keep feat(lib): add notebook cards and Nominal connection options #711's signature and _credentials(), and re-apply only this PR's stats changes on top. NominalCorePublisher(dataset=stand_in) in the example notebook still works.
  • tests/lib/test_nominal_core_publisher.py: take feat(lib): add notebook cards and Nominal connection options #711's version. test_client_fetches_the_dataset and the client= / base_url= conflict cases were removed.
  • docs/library/notebook-display.md: Stefan asked to stop listing the card's contents and to drop the NominalCorePublisher detail sentence. On rebase, keep feat(lib): add notebook cards and Nominal connection options #711's one-sentence summary and fold the stats into it, for example "its publishers with their sent and error counts". Keep the super() extension docs and the instro.lib.set_display_options link. The new example-notebook paragraph is fine to keep.

@arsenovicAtNominal

Copy link
Copy Markdown
Collaborator Author

One more from #711 (69b3f8c): the "Channel count" row moved from the _display_fields() overrides in PSU, scope and AWG into Instrument._repr_html_, as getattr(self, "_num_channels", None) next to driver. This PR also edits _repr_html_, so expect a small conflict around the fields=[...] list. Keep the new ("Channel count", ...) row.

Base automatically changed from issue-709-notebook-cards to main November 12, 2026 12:11
arsenovicAtNominal and others added 2 commits November 12, 2026 09:02
Copy examples/notebooks/*.md into the docs, run them with myst-nb, and list
them under Notebooks on the Examples tab. Add the notebook display demo
(cards only; no hardware or Nominal credentials needed), the just notebook
and test-notebooks recipes, and the docs for adding a notebook.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Instrument.publish() counts what each publisher does with every Measurement
and Command. A publisher that raises is logged, counted and skipped, and the
others still receive the data. Add Instrument.publisher_stats and
channel_stats, a Publishers table and a channel rate and count on the
notebook card, and move ChannelState and MonitorState into instro.lib.stats.
The demo notebook gains a failing-publisher section and the stats calls.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@arsenovicAtNominal
arsenovicAtNominal changed the base branch from main to issue-709-docs-notebooks November 12, 2026 14:07
Base automatically changed from issue-709-docs-notebooks to main November 12, 2026 16:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant