Skip to content

shell e2e: BM acts on a held save before it exists — a loaded machine turns a late choice into an exception (#844) #858

Open

Description

@kitokey-lina

What

Scenario BM (microphone_rollback, #844, on main since f0448d7) failed in #825's attest, which does not touch the microphone. The machine was heavily loaded at that moment (load about 19). BM's own attest at 7b73dc9 was green.

From that attest's e2e output:

  • ok the bar's options: the microphone kept is named;
  • ok the headset alone chosen and refused: the bar names Digital Microphone…;
  • FAIL « A then B chosen, both saves held: the bar names B », got (False, 'Micro : USB Headset▾'). The headset (A) was chosen, Auto (B) was not, and the step's chosen is false;
  • FAIL « A's save refused while B stands… », got the refusal and Micro : Digital Microphone▾. With B never chosen, A is the last choice made, so going back to the microphone kept is the product's right answer: this FAIL follows from the one before;
  • ok A refused: nothing of it in the settings;
  • then JavascriptException: undefined is not an object (evaluating 'window.saves[2].go') (e2e_window.py l.4157). B's save was never made, so saves[2] does not exist. The exception ends the whole e2e run, and the scenarios after BM are not played.

Cause

It is the test's, not the product's. Two weaknesses together:

  1. Its waits are short for a loaded machine.

    • choose() waits 5 s for the list to show its three microphones, then 3 s for it to close after the click.
    • The step then waits 3 s for the saves to be counted.

    Under load, the second choice (B) did not complete within them.

  2. It goes on after a step that failed. It calls window.saves[N].go() or .refuse() without checking that the N-th save exists. A late or missing save then raises in the page, and the run stops on an exception instead of a FAIL line.

This is the same lesson as #836 and #851: wait for the state, bounded, before acting on it.

Fix (after the freeze, unless it reddens main again)

  • Before each saves[N].go() / .refuse(), wait, bounded (10 s), for window.saves.length > N.
    • If it never comes, the step's check fails with what was seen (the saves' count, the bar's name).
    • The scenario's later steps are not played: return after that check. No exception reaches main().
  • Give choose()'s two waits 10 s, as the other bar scenarios use.

Oracle

A mutant that delays the page's microphone_chosen call by 4 s, past choose()'s current waits:

  • today, BM ends on the JavascriptException;
  • with the fix, BM passes, since its waits cover 4 s, or fails on a check that names the missing save. Never an exception.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions