test(cli): stop the tree tests depending on ambient colour detection - #101
Open
justin13888 wants to merge 7 commits into
Open
test(cli): stop the tree tests depending on ambient colour detection#101justin13888 wants to merge 7 commits into
justin13888 wants to merge 7 commits into
Conversation
`dependable tree` styles its labels through `if_supports_color`, which asks the environment whether stdout can take ANSI. Both sets of tree tests assumed the answer would be no — the unit tests because a test harness captures stdout, the end-to-end ones because a pipe is not a TTY — and asserted on the plain text. That is a default, not a guarantee. `FORCE_COLOR` overrides it, and anything exporting it puts it in scope: a terminal multiplexer, a task runner, a CI image. Under `FORCE_COLOR=3` three tests fail on an unmodified checkout — `ascii_points_a_member_at_its_own_tree`, `tree_distinguishes_workspace_and_ external`, and `a_member_used_by_another_member_points_at_its_own_tree` — which makes `mise run test` fail for a contributor whose shell sets it while CI, which does not, stays green. A test that only holds in some terminals reports the terminal, not the code. The assertions are about the shape of the tree, never its colour, so both are made to say so: the unit tests strip any styling from `ascii` before matching, and the end-to-end `run` helper pins the child's colour environment rather than inferring it from the pipe. `dependable` itself is unchanged — honouring `FORCE_COLOR` through a pipe is what the variable is for.
Making the three failing tests colour-independent fixes today's failure but keeps nothing that way. `crates/dependable/src/output/table.rs` styles the manifest path and every status cell through the same `if_supports_color`, and no test file that spawns the binary pins its colour environment; the next assertion written across a style boundary reintroduces the same flake, and it reintroduces it silently, because the default run is the configuration where colour is off. `FORCE_COLOR` outranks the TTY check inside `supports-color`, so forcing it on is a genuinely different configuration rather than a repeat of the first run. A `test:color` task runs the suite under it, and CI runs that task after the ordinary one on the Linux leg. The suite is hermetic and takes seconds, so the second run costs little and is the only thing that holds the property. Also drops a comment in `ascii_marks_workspace_and_dedupe` that still claimed colour is disabled because a test harness is not a TTY. That is the assumption this branch exists to remove, and leaving it ten lines below the helper doc that contradicts it invites someone to conclude the stripping is redundant.
The colour-forced suite ran only in CI, so the guard caught a colour-dependent assertion one round-trip later than it needed to: `mise run ci` and the pre-push hook both passed, and the failure arrived from the remote. `[tasks.ci]` and the pre-push gate now include it, and the Quality block lists it beside the plain run. The exposure is wider than the three tests this branch repairs — `output/table.rs` styles the manifest path and every status cell through the same `if_supports_color`, and no test that spawns the binary pins its colour environment, so they pass today only because their assertions happen not to straddle a style boundary. This is what makes that prospective rather than permanent. The extra run costs seconds: the suite is hermetic and already built by then, so the second pass compiles nothing.
The step shelled out to `mise run test:color`. This file's own header states the convention -- explicit, stable workspace commands rather than delegation -- and every other step here follows it. Delegating also made the hook depend on `mise` being resolvable in the hook environment, which nothing guarantees. `Step` carries its own `env` block, so the forced-colour configuration the task supplied can be expressed here directly.
The colour leg was a second, byte-identical `cargo test --workspace` scheduled alongside the first in the same target directory. hk runs hook steps in parallel up to `HK_JOBS` (default: core count) and mise runs a task's `depends` list concurrently, and cargo releases its build-directory lock before any test binary starts -- so the two test phases overlapped rather than queued. That is not safe here. The CLI integration tests keep fixed-name scratch fixtures under `CARGO_TARGET_TMPDIR` and wipe each with `remove_dir_all` on entry, so two concurrent runs delete each other's fixtures. `github_summary` is the sharpest case: if the second run recreates the directory between the first run's two `check()` calls, `summary.md` comes back empty and the "appended, not clobbered" assertion fails on a count of 1 -- an unexplainable local failure of exactly the kind this branch exists to remove. Serialising costs almost nothing: `FORCE_COLOR` is in no fingerprint, so the second run's build is a no-op and only its test execution is added. A separate `CARGO_TARGET_DIR` would isolate the two as well, but it forces a full second build and roughly ten more gigabytes of target output for a run whose only difference is one environment variable. CI was unaffected either way -- its two legs are sequential job steps -- so the green checks were never evidence about this.
`pre-push`, `mise run ci` and the Linux CI job all run the colour leg; `hk check` did not. That hook is documented in this file as the manual/CI entry point, so a contributor could pass `hk check`, believe the gate satisfied, and still be blocked at push -- two gates that claim to be the same telling them different things. It carries the same `depends` on the plain run, so it queues behind it here as well.
`plain` took `(&DependencyGraph, &TreeOptions)`, so the only input it could ever receive was `ascii`'s own output -- and `ascii` emits nothing but CSI colour sequences. Three of the stripper's branches were therefore unreachable by any test: the non-CSI escape, a CSI final byte other than `m`, and a lone trailing `ESC`. The non-CSI branch also holds a latent wrong answer. It consumes the byte after `ESC` and drops it, then emits the remainder as text -- correct for a two-character escape, wrong for a string escape. If `label` ever wrapped a crate name in an OSC 8 hyperlink, the URL would be spliced onto the label and present as a tree-shape regression rather than as a limit of the stripper. Taking `&str` instead lets the branches be exercised with synthetic input, and states the contract the function actually implements rather than the narrower one its comment claimed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #100.
Three
treetests fail on an unmodified checkout wheneverFORCE_COLORis set, while CI — which does not set it — stays green:Cause
labelstyles throughtext.if_supports_color(Stream::Stdout, …).owo-colorsdelegates that tosupports-color, which honoursFORCE_COLORahead of the TTY check. Both test sets assumed the check would answer "no" — the unit tests because a harness captures stdout,fixture_tree.rsexplicitly ("Piped stdout is not a TTY, so labels are plain (uncolored) and assertable as text"), with the spawned child inheriting the parent's environment. Both are defaults, not guarantees, andFORCE_COLORis exported by terminal multiplexers, task runners and CI images.The consequence is that a contributor whose shell sets it sees
mise run testfail on a cleanmaster, cannot tell that failure from a real regression, and is blocked by the pre-push hook.The change
dependableitself is untouched: honouringFORCE_COLORthrough a pipe is what the variable is for, so the product is right and the tests were wrong to assume otherwise. Every failing assertion is about the shape of the tree and none is about its colour, so both sets are made to say that rather than relying on what the process inherited.crates/dependable/src/output/tree.rs— aplaintest helper runsasciiand strips CSI sequences; the fourasciicall sites in the unit tests go through it. Stripping rather than forcing colour off keeps the tests free of global process state, which matters because they run in parallel with each other.crates/dependable/tests/fixture_tree.rs—runclearsFORCE_COLORandCLICOLOR_FORCEand setsNO_COLOR=1on the child. The module doc no longer claims the pipe makes labels plain; it says the environment is pinned.fixture_tree.rsis the only test file that asserts on styled label text — the other eight that spawn the binary assert on JSON, SARIF, exit codes, or unstyled lines — so no shared helper is introduced for one caller.Judgement calls
asciiin-process, where the only lever is the globalowo_colors::set_override, and mutating global state from tests that run in parallel is worse than stripping. The end-to-end tests spawn a child, where per-child environment is precise and local. Reversed by: deciding one mechanism everywhere is worth the parallelism risk.dependablegains no--colorflag. One would make these tests trivial, but it is a user-facing feature nobody asked for and the tests do not need it. Reversed by: someone wanting to force or suppress colour from the command line, which is worth its own issue.plainhelper handles CSI sequences only. That is allowo-colorsemits here. Reversed by: a styling path that emits OSC sequences (a hyperlink, say), which would need the helper extended.Validation
Before this change the first of those four fails with the three tests above.
Review repairs
An independent review confirmed the diagnosis empirically — under
FORCE_COLOR=3,masterfails with exactly the three named tests and this branch passes — and traced theplainhelper's CSI state machine against the actualsupports-color2.1.0 and 3.0.2 sources: CSI parameter bytes are0x30..=0x3Fand intermediates0x20..=0x2F, all below the0x40..=0x7Efinal-byte range the helper terminates on, so no parameter byte can be mistaken for a terminator and no package name or box-drawing connector can be consumed. It also confirmedNO_COLOR=1plus the twoenv_removes is deterministic: removingFORCE_COLOR/CLICOLOR_FORCEempties the override step, andNO_COLORshort-circuits beforeCLICOLOR,TERM,COLORTERM,is_ciandIGNORE_IS_TERMINALare consulted.Two things came back, and both are fixed here.
A stale comment survived inside
ascii_marks_workspace_and_dedupe—// Color is disabled in the test harness (not a TTY), so labels are plain.— asserting exactly the belief this branch exists to remove, ten lines below the helper doc that calls it "a default, not a guarantee". Left in place it invites a contributor to conclude the stripping is redundant and revertplain(…)toascii(…), reintroducing the bug for that test. Removed.Nothing kept the property. The change fixed today's three failures but added no gate, and the exposure is wider than the fix:
output/table.rsstyles the manifest path and every status cell through the sameif_supports_color, and no test file that spawns the binary pins its colour environment — they pass only because their assertions happen not to straddle a style boundary. Amise run test:colortask now runs the suite withFORCE_COLOR=3, and CI runs it after the ordinary test step on the Linux leg. BecauseFORCE_COLORoutranks the TTY check, that is a genuinely second configuration rather than a repeat of the first. The suite is hermetic and runs in seconds.Validation
Follow-up: the guard now runs locally too
A second review found the colour-forced suite ran only in CI, so
mise run ciand the pre-push hook would both pass a colour-dependent assertion and the failure would arrive from the remote a round-trip later.[tasks.ci]and the pre-push gate now includetest:color, andCLAUDE.md's Quality block lists it beside the plain run. Measured cost: the second pass compiles nothing (0Compilinglines, ~1.6s), because the suite is hermetic and already built by then.The same review corrected a claim in this branch's first commit message. It cites
output/table.rsstyling the manifest path and every status cell as the motivation for the guard — but no test asserts ontable.rsstdout, andlist.rsstyles nothing, so the coloured leg's live coverage today is the tree unit tests alone. Its value fortable.rsis prospective: it catches the next assertion written across a style boundary, rather than protecting one that exists. Worth stating plainly, because the original wording reads as thoughtable.rsis covered now.Decisions taken
How to stop the colour run colliding with the plain run
Taken: serialize them — the colour task depends on the plain one, in both the mise gate and the git hook.
Rejected: giving the colour run its own
CARGO_TARGET_DIR— it works, and it is whatcargo llvm-covdoes for itself, but it forces a full second build and roughly ten more gigabytes of target output for a run whose only difference is one environment variable.FORCE_COLORis in no fingerprint, so a serialized second run rebuilds nothing and costs only its own test execution.Rejected: making the scratch fixtures collision-safe instead — that is the deeper fix and closes a wider class, including a
std::env::temp_dir()path that can already race the coverage step today, but it touches six test files and is not what put this pull request in the work order. Filed separately.Reverses: drop the
dependsfrom the colour task and addCARGO_TARGET_DIRto its env instead.Which gates the colour leg belongs to
Taken: hk
pre-push, hkcheck,mise run ci, and the Linux CI job.checkis added here because the file documents it as the manual/CI entry point, and a contributor who passeshk checkand is then blocked at push has been told two different things by two gates that claim to be the same.Rejected: adding it to the Windows and macOS CI legs — the mechanism is environment-variable driven rather than platform-driven, so one platform establishes it and two more triple the CI bill for no additional signal.
Rejected: pre-push only, matching the
coverageprecedent — coverage is explicitly informational with no threshold; this is a pass/fail correctness gate, so the precedent does not transfer.Reverses: remove the step from the
checkhook.Whether the hook invokes cargo directly or shells out to mise
Taken: invoke cargo directly with the hook step's own
envblock.Rejected:
mise run test:color—hk.pkl's own header states the convention that the file defines explicit stable workspace commands rather than delegating, and shelling out additionally makes the hook depend onmisebeing resolvable in the hook environment, which nothing guarantees.Reverses: restore
check = "mise run test:color"and drop the step's env block.Whether
fixture_tree.rsshould keep opting out of the colour configurationTaken: keep the pin. That file asserts tree SHAPE, so pinning its child's colour environment makes it deterministic under either leg, and the eight other binary-spawning test files still inherit the forced-colour configuration and provide the coverage the gate exists for.
Rejected: letting it inherit
FORCE_COLORso the gate covers the file that motivated the issue — the assertions there are about connectors and labels, and re-admitting ambient colour into them re-opens exactly the failure mode this change closes.Reverses: delete the three
env/env_removecalls in that file'srun()helper.Evidence that the serialization took effect
mise tasks deps ci, before and after:hk run check --all --format jsonl, step events in order —fmt,testandclippyall start within 4ms of each other, which is hk scheduling hook steps in parallel;test-colorstarts 1.4s later, only aftertestcompleted:The no-rebuild claim, from
mise run test:color's own output — the second leg's build is a no-op:Unresolved review notes
CARGO_TARGET_TMPDIR(eighteen throughworkdir, twelve throughcli_policy.rs'sconfig) wipe themselves on entry, and one fixed path understd::env::temp_dir()sits outside any target directory and can already race the coverage step. Not repaired here: it spans six test files and is wider than the scope that put this pull request in the work order. Filed as test(cli): scratch fixtures use fixed names and wipe on entry, so no two test processes can run at once #119.