Skip to content

Keep EIO completion notifications active - #299

Open
pauljccode wants to merge 1 commit into
rapid7:masterfrom
pauljccode:fix/eio-completion-notifications
Open

pauljccode wants to merge 1 commit into
rapid7:masterfrom
pauljccode:fix/eio-completion-notifications

Conversation

@pauljccode

@pauljccode pauljccode commented Oct 4, 2026 •

Copy link
Copy Markdown

An I/O completion can be stranded when a worker notifies the event loop between eio_done_poll stopping the async watcher and eio_async_cb restarting it. ev_async_start clears the pending notification, leaving a completed request queued without an event-loop callback.

Keep the watcher active while polling. Retain the required done-poll callback as a no-op because bundled libeio invokes it unconditionally.

Verification:

  1. Download check-eio-notifications.py and eio-notification-probe.c from the reproducer directory into the same directory. The runner requires Python 3, a C compiler, make and tar; it builds the bundled libeio 1.0.2 and libev 4.33.
  2. Run python3 check-eio-notifications.py /path/to/mettle baseline against master (4dd76ae), then repeat with candidate against this branch. The probe extracts the actual callbacks and forces the worker to finish in the stop/start interval.
  3. Confirm that both notifications are sent in each case, but only the corrected callbacks deliver both completions:
mode=stop-start notifications=2 completed=1 pending=1
mode=keep-active notifications=2 completed=2 pending=0

The race reproducer passed on Linux, Intel macOS and Apple Silicon macOS. Mettle Verify passed all eight Ruby/platform jobs, each with 65 examples and zero failures.

Acceptance of this commit passed 51 Linux and 46 Intel macOS examples. Both jobs build this PR's commit and verify installed files against the retained packages. The deterministic probe above isolates the notification race.

Keep the async watcher active while libeio drains its queue. A worker
can finish between stopping and restarting the watcher, and
 ev_async_start clears that pending notification.

Retain the required done-poll callback as a no-op; bundled libeio
invokes it unconditionally.
@pauljccode

Copy link
Copy Markdown
Author

I also ran a source-pinned Linux acceptance comparison with identical queue/watcher diagnostics and no forced scheduling delay. The baseline finished with two failures; this correction passed all 51 examples.

The baseline recorded notification clears across watcher restart and two stalls with completed work accumulating while callback counts stopped advancing. The corrected build recorded no clears. Source and installed-binary hashes were verified for both builds. The final notification transition in each stalled session was not directly sampled; the captured queue state supports the earlier controlled reproduction.

@pauljccode

Copy link
Copy Markdown
Author

Could someone restart and approve CI for this PR when they have a chance? The runs expired while awaiting approval. The separate fix in #298 has the same CI blocker. Happy to help with any additional testing. Thanks!

@dwelch-r7

Copy link
Copy Markdown
Contributor

Cosing and re-opening to trigger the CI again

@dwelch-r7 dwelch-r7 closed this Nov 12, 2026
@github-project-automation github-project-automation Bot moved this from Todo to Done in Metasploit Kanban Nov 12, 2026
@dwelch-r7

Copy link
Copy Markdown
Contributor

Cosing and re-opening to trigger the CI again

@dwelch-r7 dwelch-r7 reopened this Nov 12, 2026
@github-project-automation github-project-automation Bot moved this from Done to In Progress in Metasploit Kanban Nov 12, 2026
@dwelch-r7

Copy link
Copy Markdown
Contributor

@coderabbitai full review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants