Skip to content

shell: the e2e cut in shards played side by side (#862, step 3) - #867

Open
kitokey-sam[bot] wants to merge 6 commits into
mainfrom
issue-862-shards
Open

kitokey-sam[bot] wants to merge 6 commits into
mainfrom
issue-862-shards

Conversation

@kitokey-sam

@kitokey-sam kitokey-sam Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Refs #853 (its likely cause, the single-instance name, waited for). Closes #862 once its oracle holds, as step 3 of 3 (steps 1 and 2 merged in #865).


Module

modules/shell — exactly one.

  • This pull request touches a single module
  • ⚠️ Cross-module — the cross-module label is required, justification below

Deliverable

Out of cycle: #862, held, the coordinator's GO, an exception to the freeze. After #865, a full run took about 19.5 minutes, measured at 9eae465:

  • the tray's part, about 150 s;
  • the window's 47 scenarios, about 880 s summed (microphone_chosen 166 s, bar_opened 116 s, …);
  • main()'s own scenarios H to R, and the build.

The window's part is now the larger, and its scenarios are independent: each starts its own Kitokey.

What it changes

flowchart LR
  H[headless.sh, a full run] --> S[e2e_sharded.sh]
  S --> T[shard 0: the tray's part]
  S --> M[shard 1: main's own, H to R]
  S --> W1[shard 2: list 1]
  S --> W2[shard 3: list 2]
  S --> W3[shard 4: list 3]
  T & M & W1 & W2 & W3 --> J[one text, one logs judgement, one verdict]
Loading

Legend: each shard is a headless.sh session of its own, with its own GNOME, buses and PipeWire, played side by side. The three lists are balanced by e2e_times.txt.

  • scripts/headless.sh: a full run is handed to e2e_sharded.sh. KITOKEY_E2E_SHARDS (default 3) sets the number of lists; KITOKEY_E2E_SHARDS=1 runs it whole in one session, as before. A partial run (KITOKEY_E2E_ONLY) is never sharded. Its text and the session's log go where KITOKEY_E2E_OUT says.
  • scripts/e2e_sharded.sh (new):
    • It builds once, then plays the five shards side by side, and puts their text back in a whole run's order.
    • It judges every shard's logs together as a whole run's, with nothing « not owed », and gathers the screenshots into the run's evidence.
    • Each shard says « ok », « FAIL » or « NOT RUN ». The verdict is one: 77 if a shard could not run, and 1 if any failed.
  • scripts/e2e_shards.py (new):
  • scripts/e2e_times.txt (new): each window scenario's time from 9eae465's run. A scenario not listed counts as the median.
  • scripts/e2e.sh:
    • It writes to $KITOKEY_E2E_OUT.
    • KITOKEY_E2E_PART=tray plays the tray's part alone; =main plays main()'s own scenarios alone, not the tray's.
    • A shard ends as a partial run does: status 3, and its last line says whether its checks passed.
  • scripts/e2e_window.py: with KITOKEY_E2E_PART=main, the called scenarios are skipped. main()'s own part says its time, (main: N s), before the first call.
  • scripts/e2e_window.py, a session's own Kitokeys (4882305, 0155b8b):
    • instances() keeps those whose environ holds this session's XDG_RUNTIME_DIR. Before, a shard closing a window sent TERM to the other shards' Kitokeys, which share the binary (ines's reading).
    • Window.__enter__ waits, up to 20 s, until no Kitokey holds its single-instance name on the session's bus, com.napkinstack.Kitokey.SingleInstance…. Otherwise a start made while the one before is leaving hands over to it and leaves, which is shell e2e: a start ends right after « shell.started », no error — the single-instance race (#510)? #853's « shell.started, then nothing »; five sessions at once made it likelier.
    • e2e.sh's count of running Kitokeys after a second start is filtered by session the same way.
  • scripts/e2e_window.py, AD (4f93c92): it writes the two model files K leaves, when they are absent.
  • scripts/e2e_logs.py: a shard's logs are judged as a partial run's. The whole run's judgement is the one e2e_sharded.sh makes over all of them.

Oracle

  • The success criterion (shell e2e: measure where its 25–48 minutes go, then shard the window's scenarios #862): on the same head, the sharded run and the whole one (KITOKEY_E2E_SHARDS=1) give the same verdict and the same FAIL lines, with a known mutant and without it; the window's part takes less than 880/3 + 3 min; no shard reaches another's folder, bus or sound.
  • Seen failing: the whole run at this head with shell: e2e — D60's bar waited for whole, not read after a fixed pause (#850) #851's mutant (whole = lambda b: False): its two FAIL lines.
  • It passes now:
    • the sharded run at this head, without the mutant: « e2e: passed », its ok count that of the whole run;
    • the sharded run with the mutant: the same two FAIL lines, « e2e: FAILED »;
    • its wall time, said on its last lines ((the run, its shards side by side: N s)), against the whole run's;
    • the memory's peak while five sessions run, measured by the coordinator (free -m during the run). It decides whether N stays 3.

Exercised here, without a session: e2e_shards.py lists 3 gives 14, 16 and 17 scenarios, 47 in all, at about 294 s each. merge on sample shard texts puts the tray first, main()'s own next, then the scenarios in main()'s order whichever shard played them, and each shard's outside lines last. cargo test --locked --test e2e_only: 2 passed. bash -n on every script passes; py_compile passes; ruff (F,PLW2901) finds the same 7 in e2e_window.py as main, none in the new files.

Kind of oracle: the e2e's own verdict and lines, whole against sharded.


Review budget

  • Lines changed (excluding generated), at 4f93c92 against main: 323 (+295 −28). Of these, 50 are e2e_times.txt, data of one line a scenario.
  • Files touched: 7
  • Within budget

Contracts

  • No contract affected

Definition of Done

  • Oracle green: the coordinator's runs, whole against sharded, with and without the mutant
  • Lint, format, types: bash -n, py_compile, ruff as above
  • Unit tests: e2e_only's, 2 passed
  • E2E on critical journeys: the runs above
  • Affected documentation updated: the scripts' heads say it
  • Diff self-reviewed line by line

Test sheet

Verifier:

# Given · when · then Kind Result Evidence Commit Confirmed
S1 Given a full run at this head, when it runs sharded, then its verdict and FAIL lines are those of the same head run whole (KITOKEY_E2E_SHARDS=1), with nothing « not owed » in its logs' judgement automated
S2 Given #851's mutant, when the run is sharded, then the same two FAIL lines as the whole run's, and « e2e: FAILED » automated
S3 Given the sharded run, then its wall time, said on its last lines, is the window's part's under 880/3 + 3 min, and no shard's audio defaults change automated
S4 Given e2e_shards.py, when the lists are made, then each scenario main() calls is in exactly one list; and when the shards' texts are merged, each scenario's lines appear once, in main()'s order automated

Verifier's remarks:


Summary

DONE

  • A full shell e2e runs in five sessions side by side: the tray's part, main()'s own scenarios, and three balanced lists. The result is one text, one logs judgement and one verdict.

VERIFIED

  • Lists and merge on the measured times and on sample texts.
  • e2e_only 2 passed; syntax and lint as above.

ASSUMED

  • The scenarios are independent: each starts its own Kitokey, in its own folder. A scenario that needed another's leftovers would fail in a shard, which is what S1 would show.
  • Five GNOME sessions with WebKit fit in this machine's 23 GB, at about 1 to 1.5 GB each. Not measured; it is S3's run.

NOT VERIFIED

RISKS

  • A scenario that depends on what another scenario or main()'s body leaves behind fails only when sharded. Found and fixed so far:
  • A scenario that raises ends its shard's run there, as it ends a whole run. The scenarios after it in its list are said « NOT RUN — no shard said it », and its shard fails.
  • More memory and processes at once; KITOKEY_E2E_SHARDS lowers it, and 1 is the old way.
  • e2e_times.txt grows stale as scenarios change. The balance suffers, not the verdict. It is refreshed from any full run's (name: N s) lines.

🤖 Generated with Claude Code

…f its own (#862, step 3)

After #865, the window's 47 scenarios take ~880 s of a ~19-minute run.
headless.sh, for a full run, now hands it to e2e_sharded.sh: the tray's
part, main()'s own scenarios (H to R), and the others in three lists
balanced by their measured times (e2e_times.txt, longest first into the
lightest), each a headless.sh session of its own, side by side. Their
text is put back in a whole run's order, every shard's logs are judged
together as a whole run's, the screenshots gathered; one verdict, 77 if
a shard could not run. KITOKEY_E2E_SHARDS=1 runs it whole, as before.
e2e.sh writes where KITOKEY_E2E_OUT says and plays its tray part alone
(KITOKEY_E2E_PART=tray) or main()'s own alone (=main).

Agent-Session: 92ded11e-58ef-492d-ba56-c820b9011de8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kitokey-sam kitokey-sam Bot added the out-of-cycle Work outside the current cycle: an incident, a production defect; justified in the PR label Nov 15, 2026
@kitokey-sam

kitokey-sam Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

What a scenario called in a shard might lack, read at 00ccda7 (ines's point; BG and K's Whisper files are the known case, fixed by #866). Reading only, nothing run.

Every scenario's Window shares $WORK/data and $WORK/state (XDG_DATA_HOME and XDG_STATE_HOME, l.531) and $WORK's flag files. I compared what main()'s body writes there with what each of the 47 called scenarios reads.

  • The stand-in programs (SUMMARY, HEARING, RECORDER) are made when the module loads, so every shard has them.
  • The flag files main()'s body uses: slow-minutes, fail-minutes, fail-plainly, fail-hearing and slow-hearing are each touched and removed within it, so none is left behind. test-code is the scenario's own, and service_named makes it.
  • summary-args and hearing-args: each scenario that reads them either removes them first (context_gaps, own_models, models_advanced, minutes_checked, cancelled_at_once) or reads only what its own gesture has just written (confidential_kept, corrections_gated, press_on_the_page_left, confidential_every_way). A leftover could make a check pass by mistake in a whole run, never fail a shard.
  • K's model files ($WORK/data/kitokey/models, ggml-large-v3-turbo-q8_0.bin, ggml-small.bin):
    • The scenarios naming them make their own and set their own XDG_DATA_HOME (model_offered, models_listed, first_run, ollama_pulled).
    • list_refreshed looks for ggml-small-q8_0.bin as offered, not as present.
    • two_panes and pages_seen name the model only in a transcript header they write.
    • None reads K's two files besides BG.
  • $WORK/state: models_guided and transcription_guided check that no file there holds a key; in a shard there are fewer files, so the check is no weaker. recording_found_not_forgotten, trial_over and meeting_language read the logs of their own runs.

Not covered by this reading: a scenario that depends on what Kitokey itself left in $WORK/data or $WORK/state during an earlier scenario, such as a trash, settings or a recording found. That cannot be told without running. The line-by-line comparison of the whole and sharded runs is what shows it. Any line that differs names a scenario that must set up its own state, as BG does since cb855aa.

kitokey-sam Bot and others added 2 commits November 15, 2026 14:11
…other shard's (#862)

00ccda7's sharded run: the four window shards died together on WebDriver
errors (« session deleted because of page crash or hang »). instances()
read every process whose exe is the binary under test — the same for all
shards — and Window.__exit__ sent each a TERM: a shard closing a window
ended the others' Kitokeys (ines's reading). It now keeps those of this
session alone, by their XDG_RUNTIME_DIR, as headless.sh's own ended()
does; e2e.sh's count of running Kitokeys after a second start the same.

Agent-Session: 92ded11e-58ef-492d-ba56-c820b9011de8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… leaves them (#862)

4882305's sharded run: AD (two_panes) drew Settings with no model on this
computer — K, which writes two, played in another shard — so the guided
step stood in the models' place: « Settings, 420 px wide » failed on it,
and « Avancé » could not be clicked, ending its shard's run there. AD now
writes the same two files when they are not there; a whole run, where K
wrote them, is unchanged.

Agent-Session: 92ded11e-58ef-492d-ba56-c820b9011de8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kitokey-sam Bot and others added 3 commits November 15, 2026 15:01
…name on the session's bus (#853, #862)

4f93c92's sharded run: shard 4 died in first_run on « tauri-driver opened
no session in a minute; Kitokey's processes: [] », its log ending on
« shell.started » and nothing — #853's signature, five sessions side by
side making it likelier. A start made while the Kitokey before is still
leaving hands its arguments to it and leaves; /proc cannot say a process
that is exiting is Kitokey, but its single-instance name on the bus is
held to its last instant. Window.__enter__ now waits, up to 20 s, until
neither is there; a session that still opens none says whether the name
was held.

Agent-Session: 92ded11e-58ef-492d-ba56-c820b9011de8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: 92ded11e-58ef-492d-ba56-c820b9011de8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: 92ded11e-58ef-492d-ba56-c820b9011de8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

out-of-cycle Work outside the current cycle: an incident, a production defect; justified in the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

shell e2e: measure where its 25–48 minutes go, then shard the window's scenarios

0 participants