Skip to content

shell e2e: measure where its 25–48 minutes go, then shard the window's scenarios #862

Open

Description

@kitokey-sam

The shell's e2e takes 25 to 48 minutes per attest, and attests run one at a time under the machine lock. This issue measures where the time goes before choosing how to cut it, and proposes sharding the window's scenarios once the numbers say it pays.

Measured, from one run

The run is #847's attest at 3c16f92's parent: evidence-847, its shell logs' timestamps, in UTC.

Part From → to Duration Kitokey starts
GNOME's headless session up 09:12:40 — —
The tray's scenarios (e2e.sh A–S, 10 scenarios), 00-shell…jsonl 09:12:45 → 09:36:08 23.4 min 28
The window's scenarios (e2e_window.py, 47 called from main()), 01-shell…jsonl 09:37:01 → 09:52:50 15.8 min 150
After the window's last log line, to e2e.txt written 09:52:50 → 10:01:00 ~8 min —

The tray part is the larger. In it, consecutive Kitokey starts are 50 to 55 s apart: 09:13:36, 09:14:40, 09:15:32, 09:16:24 and so on. Each time, nothing is logged between shell.licence_read, at the start, and the scenario's next action. 28 starts × ~50 s ≈ 23 min, so nearly all of the part is spent between a start and its first gesture.

Assumption, not verified: e2e.sh's start() loops up to 100 times, each time calling busctl for the watcher's items, for the bus's names, and for each candidate menu. Its 0.1 s sleeps add 10 s at most, so the rest would be those calls, or the tray item registering late. A start in the window's part costs about 6 s (150 starts in 15.8 min).

Not in the evidence: time per scenario. e2e.txt and output.txt carry no timestamps, and the window's 150 starts cannot be tied to its 47 scenarios by counting: a static count of as browser and __enter__() gives 120. The last ~8 minutes cannot be attributed either.

Proposed, in this order

  1. Measure. No behaviour changes.
    • Each scenario's header gets its elapsed seconds, in both scripts.
    • start() says how long it took to find the tray item, and in how many tries.
    • One full run then gives the table this issue lacks.
  2. If start() costs what it seems to, fix it first. This is the largest lever: up to ~20 minutes on every run, with no added RAM. It also benefits the tray part, which cannot be sharded across sessions without a tray each.
  3. Then shard the window's scenarios. Run N sealed headless GNOME sessions in parallel, each playing a list given by KITOKEY_E2E_ONLY. The tray part runs in its own session. The shards' e2e.txt are merged in main()'s order, with one verdict.
    • The lists are balanced by step 1's times: greedy longest-first, as Bazel's shard_count, pytest-xdist's --dist loadgroup and Playwright's --shard do.
    • N = 3 to start. With 23 GB of RAM and about 1 to 1.5 GB per GNOME session with WebKit (an estimate to measure in step 1, from /proc/<pid>/status VmRSS of a session's processes), three sessions plus the build leave room. N grows only if the measured peak allows.
    • e2e_logs.py reads every shard's logs. A scenario's provoked lines are owed by whichever shard plays it, so the merge judges them once, over all shards.

Oracle (for step 3's pull request)

  • Same verdict, same FAILs. On the same head, the sharded run and the serial run give the same verdict and the same list of FAIL lines. That holds with one mutant known to fail a window scenario, for instance shell: e2e — D60's bar waited for whole, not read after a fixed pause (#850) #851's whole = lambda b: False, and with it removed.
  • Faster. Wall time below 15.8/N + 3 minutes for the window's part, measured on this machine; the tray part's time is reported separately.
  • No interference. Each shard's audio defaults and source-outputs are unchanged at its end. No shard reads another's folder, and each has its own WORK, XDG_* and bus.

Activity

  1. kitokey-sam commented on Nov 15, 2026

    @kitokey-sam
    ContributorAuthor

    The measure, at 7a4beb9 (#864), in the coordinator's full sealed run: « e2e: passed », 1823 ok, 0 FAIL, 2495 s in all. It ran beside another attest, so the times are a little high.

    Tray, 1502 s (25 min): A 59, B 3, C 52, D 105, E 53, F 181, G 73, L 108, M 275, S 593 s. S2's time is counted in S's.

    Window, 862 s (14.4 min) over its 47 scenarios, the longest first:

    Scenario s
    microphone_chosen 166.0
    bar_opened 115.7
    nothing_silent 57.2
    announced 36.3
    press_on_the_page_left 33.7
    models_advanced 31.2
    minutes_checked 30.5
    first_run 28.7
    sound_checked 28.2
    gestures_answer 24.1
    cancelled_at_once 23.5
    model_offered 22.5
    heard_checked 21.8
    confidential_kept 21.8
    context_written 19.8
    trial_over 17.6
    two_panes 17.1
    meeting_renamed 13.7
    context_before_start 12.9
    ollama_pulled 12.6
    window_looks 11.2
    models_listed 11.2
    service_named 10.3
    transcript_corrected 10.2
    pages_seen 9.7
    recording_found_not_forgotten 7.1
    bar_window 5.6
    app_language 5.0
    meeting_language 4.8
    context_gaps 4.3
    trial_left 4.2
    models_guided 4.2
    speech_offered 4.1
    transcription_guided 3.7
    minutes_copied 3.7
    microphone_rollback 3.7
    sidebar_folded 3.6
    corrections_gated 3.6
    meeting_told_once 2.8
    own_models 2.5
    microphone_latest_answer 2.4
    meeting_named 2.0
    actions_rows 1.9
    confidential_every_way 1.8
    word_after_a_gap 1.3
    recording_found 1.3
    list_refreshed 1.0

    What it means for step 3: after #865, the tray part should drop to about 3 minutes, and the window's 14.4 minutes become the larger. Sharded in three by longest-first, the window's floor is its longest scenario, microphone_chosen at 166 s, and its ideal is 862/3 ≈ 287 s. So the run would be about 5 min for the window plus about 3 min for the tray, against about 41 min measured, besides the build and GNOME's start.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions