Skip to content

test(cli): stop the tree tests depending on ambient colour detection - #101

Open
justin13888 wants to merge 7 commits into
masterfrom
test/tree-colour-independent
Open

test(cli): stop the tree tests depending on ambient colour detection#101
justin13888 wants to merge 7 commits into
masterfrom
test/tree-colour-independent

Conversation

@justin13888

@justin13888 justin13888 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Closes #100.

Three tree tests fail on an unmodified checkout whenever FORCE_COLOR is set, while CI — which does not set it — stays green:

$ FORCE_COLOR=3 cargo test --workspace
failures:
    output::tree::tests::ascii_points_a_member_at_its_own_tree
    tree_distinguishes_workspace_and_external
    a_member_used_by_another_member_points_at_its_own_tree

Cause

label styles through text.if_supports_color(Stream::Stdout, …). owo-colors delegates that to supports-color, which honours FORCE_COLOR ahead of the TTY check. Both test sets assumed the check would answer "no" — the unit tests because a harness captures stdout, fixture_tree.rs explicitly ("Piped stdout is not a TTY, so labels are plain (uncolored) and assertable as text"), with the spawned child inheriting the parent's environment. Both are defaults, not guarantees, and FORCE_COLOR is exported by terminal multiplexers, task runners and CI images.

The consequence is that a contributor whose shell sets it sees mise run test fail on a clean master, cannot tell that failure from a real regression, and is blocked by the pre-push hook.

The change

dependable itself is untouched: honouring FORCE_COLOR through a pipe is what the variable is for, so the product is right and the tests were wrong to assume otherwise. Every failing assertion is about the shape of the tree and none is about its colour, so both sets are made to say that rather than relying on what the process inherited.

  • crates/dependable/src/output/tree.rs — a plain test helper runs ascii and strips CSI sequences; the four ascii call sites in the unit tests go through it. Stripping rather than forcing colour off keeps the tests free of global process state, which matters because they run in parallel with each other.
  • crates/dependable/tests/fixture_tree.rsrun clears FORCE_COLOR and CLICOLOR_FORCE and sets NO_COLOR=1 on the child. The module doc no longer claims the pipe makes labels plain; it says the environment is pinned.

fixture_tree.rs is the only test file that asserts on styled label text — the other eight that spawn the binary assert on JSON, SARIF, exit codes, or unstyled lines — so no shared helper is introduced for one caller.

Judgement calls

  • Strip in the unit tests, pin the environment in the end-to-end ones. They are different problems: the unit tests call ascii in-process, where the only lever is the global owo_colors::set_override, and mutating global state from tests that run in parallel is worse than stripping. The end-to-end tests spawn a child, where per-child environment is precise and local. Reversed by: deciding one mechanism everywhere is worth the parallelism risk.
  • dependable gains no --color flag. One would make these tests trivial, but it is a user-facing feature nobody asked for and the tests do not need it. Reversed by: someone wanting to force or suppress colour from the command line, which is worth its own issue.
  • The plain helper handles CSI sequences only. That is all owo-colors emits here. Reversed by: a styling path that emits OSC sequences (a hyperlink, say), which would need the helper extended.

Validation

FORCE_COLOR=3 cargo test --workspace                                → 39 test binaries, 0 failures
cargo test --workspace (FORCE_COLOR unset)                          → 39 test binaries, 0 failures
cargo clippy --workspace --all-targets -- -D warnings               → clean
cargo fmt --all --check                                             → clean

Before this change the first of those four fails with the three tests above.


Review repairs

An independent review confirmed the diagnosis empirically — under FORCE_COLOR=3, master fails with exactly the three named tests and this branch passes — and traced the plain helper's CSI state machine against the actual supports-color 2.1.0 and 3.0.2 sources: CSI parameter bytes are 0x30..=0x3F and intermediates 0x20..=0x2F, all below the 0x40..=0x7E final-byte range the helper terminates on, so no parameter byte can be mistaken for a terminator and no package name or box-drawing connector can be consumed. It also confirmed NO_COLOR=1 plus the two env_removes is deterministic: removing FORCE_COLOR/CLICOLOR_FORCE empties the override step, and NO_COLOR short-circuits before CLICOLOR, TERM, COLORTERM, is_ci and IGNORE_IS_TERMINAL are consulted.

Two things came back, and both are fixed here.

  • A stale comment survived inside ascii_marks_workspace_and_dedupe// Color is disabled in the test harness (not a TTY), so labels are plain. — asserting exactly the belief this branch exists to remove, ten lines below the helper doc that calls it "a default, not a guarantee". Left in place it invites a contributor to conclude the stripping is redundant and revert plain(…) to ascii(…), reintroducing the bug for that test. Removed.

  • Nothing kept the property. The change fixed today's three failures but added no gate, and the exposure is wider than the fix: output/table.rs styles the manifest path and every status cell through the same if_supports_color, and no test file that spawns the binary pins its colour environment — they pass only because their assertions happen not to straddle a style boundary. A mise run test:color task now runs the suite with FORCE_COLOR=3, and CI runs it after the ordinary test step on the Linux leg. Because FORCE_COLOR outranks the TTY check, that is a genuinely second configuration rather than a repeat of the first. The suite is hermetic and runs in seconds.

Validation

mise run test:color   (FORCE_COLOR=3)   → 39 test binaries, 0 failures
cargo test --workspace (FORCE_COLOR unset) → 39 test binaries, 0 failures
cargo clippy --workspace --all-targets -- -D warnings → clean
cargo fmt --all --check → clean

Follow-up: the guard now runs locally too

A second review found the colour-forced suite ran only in CI, so mise run ci and the pre-push hook would both pass a colour-dependent assertion and the failure would arrive from the remote a round-trip later. [tasks.ci] and the pre-push gate now include test:color, and CLAUDE.md's Quality block lists it beside the plain run. Measured cost: the second pass compiles nothing (0 Compiling lines, ~1.6s), because the suite is hermetic and already built by then.

The same review corrected a claim in this branch's first commit message. It cites output/table.rs styling the manifest path and every status cell as the motivation for the guard — but no test asserts on table.rs stdout, and list.rs styles nothing, so the coloured leg's live coverage today is the tree unit tests alone. Its value for table.rs is prospective: it catches the next assertion written across a style boundary, rather than protecting one that exists. Worth stating plainly, because the original wording reads as though table.rs is covered now.


Decisions taken

  1. How to stop the colour run colliding with the plain run
    Taken: serialize them — the colour task depends on the plain one, in both the mise gate and the git hook.
    Rejected: giving the colour run its own CARGO_TARGET_DIR — it works, and it is what cargo llvm-cov does for itself, but it forces a full second build and roughly ten more gigabytes of target output for a run whose only difference is one environment variable. FORCE_COLOR is in no fingerprint, so a serialized second run rebuilds nothing and costs only its own test execution.
    Rejected: making the scratch fixtures collision-safe instead — that is the deeper fix and closes a wider class, including a std::env::temp_dir() path that can already race the coverage step today, but it touches six test files and is not what put this pull request in the work order. Filed separately.
    Reverses: drop the depends from the colour task and add CARGO_TARGET_DIR to its env instead.

  2. Which gates the colour leg belongs to
    Taken: hk pre-push, hk check, mise run ci, and the Linux CI job. check is added here because the file documents it as the manual/CI entry point, and a contributor who passes hk check and is then blocked at push has been told two different things by two gates that claim to be the same.
    Rejected: adding it to the Windows and macOS CI legs — the mechanism is environment-variable driven rather than platform-driven, so one platform establishes it and two more triple the CI bill for no additional signal.
    Rejected: pre-push only, matching the coverage precedent — coverage is explicitly informational with no threshold; this is a pass/fail correctness gate, so the precedent does not transfer.
    Reverses: remove the step from the check hook.

  3. Whether the hook invokes cargo directly or shells out to mise
    Taken: invoke cargo directly with the hook step's own env block.
    Rejected: mise run test:colorhk.pkl's own header states the convention that the file defines explicit stable workspace commands rather than delegating, and shelling out additionally makes the hook depend on mise being resolvable in the hook environment, which nothing guarantees.
    Reverses: restore check = "mise run test:color" and drop the step's env block.

  4. Whether fixture_tree.rs should keep opting out of the colour configuration
    Taken: keep the pin. That file asserts tree SHAPE, so pinning its child's colour environment makes it deterministic under either leg, and the eight other binary-spawning test files still inherit the forced-colour configuration and provide the coverage the gate exists for.
    Rejected: letting it inherit FORCE_COLOR so the gate covers the file that motivated the issue — the assertions there are about connectors and labels, and re-admitting ambient colour into them re-opens exactly the failure mode this change closes.
    Reverses: delete the three env/env_remove calls in that file's run() helper.

Evidence that the serialization took effect

mise tasks deps ci, before and after:

BEFORE (d58c3bb)          AFTER
ci                        ci
├── test:color            ├── test:color
├── test                  │   └── test
├── lint                  ├── test
└── fmt:check             ├── lint
                          └── fmt:check

hk run check --all --format jsonl, step events in order — fmt, test and clippy all start within 4ms of each other, which is hk scheduling hook steps in parallel; test-color starts 1.4s later, only after test completed:

seq 2  step_started    fmt         16:17:20.162781092
seq 3  step_started    test        16:17:20.166192340
seq 4  step_started    clippy      16:17:20.166256749
seq 5  step_completed  fmt         passed
seq 6  step_completed  clippy      passed
seq 7  step_completed  test        passed
seq 8  step_started    test-color  16:17:21.572976028
seq 9  step_completed  test-color  passed

The no-rebuild claim, from mise run test:color's own output — the second leg's build is a no-op:

[test:color] $ cargo test --workspace
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.18s

Unresolved review notes

  • Low — concurrent-unsafe scratch fixtures across the CLI test suite — thirty fixed scratch names under CARGO_TARGET_TMPDIR (eighteen through workdir, twelve through cli_policy.rs's config) wipe themselves on entry, and one fixed path under std::env::temp_dir() sits outside any target directory and can already race the coverage step. Not repaired here: it spans six test files and is wider than the scope that put this pull request in the work order. Filed as test(cli): scratch fixtures use fixed names and wipe on entry, so no two test processes can run at once #119.

`dependable tree` styles its labels through `if_supports_color`, which asks the
environment whether stdout can take ANSI. Both sets of tree tests assumed the
answer would be no — the unit tests because a test harness captures stdout, the
end-to-end ones because a pipe is not a TTY — and asserted on the plain text.

That is a default, not a guarantee. `FORCE_COLOR` overrides it, and anything
exporting it puts it in scope: a terminal multiplexer, a task runner, a CI
image. Under `FORCE_COLOR=3` three tests fail on an unmodified checkout —
`ascii_points_a_member_at_its_own_tree`, `tree_distinguishes_workspace_and_
external`, and `a_member_used_by_another_member_points_at_its_own_tree` —
which makes `mise run test` fail for a contributor whose shell sets it while
CI, which does not, stays green. A test that only holds in some terminals
reports the terminal, not the code.

The assertions are about the shape of the tree, never its colour, so both are
made to say so: the unit tests strip any styling from `ascii` before matching,
and the end-to-end `run` helper pins the child's colour environment rather than
inferring it from the pipe. `dependable` itself is unchanged — honouring
`FORCE_COLOR` through a pipe is what the variable is for.
Making the three failing tests colour-independent fixes today's failure but
keeps nothing that way. `crates/dependable/src/output/table.rs` styles the
manifest path and every status cell through the same `if_supports_color`, and
no test file that spawns the binary pins its colour environment; the next
assertion written across a style boundary reintroduces the same flake, and it
reintroduces it silently, because the default run is the configuration where
colour is off.

`FORCE_COLOR` outranks the TTY check inside `supports-color`, so forcing it on
is a genuinely different configuration rather than a repeat of the first run.
A `test:color` task runs the suite under it, and CI runs that task after the
ordinary one on the Linux leg. The suite is hermetic and takes seconds, so the
second run costs little and is the only thing that holds the property.

Also drops a comment in `ascii_marks_workspace_and_dedupe` that still claimed
colour is disabled because a test harness is not a TTY. That is the assumption
this branch exists to remove, and leaving it ten lines below the helper doc
that contradicts it invites someone to conclude the stripping is redundant.
The colour-forced suite ran only in CI, so the guard caught a colour-dependent
assertion one round-trip later than it needed to: `mise run ci` and the pre-push
hook both passed, and the failure arrived from the remote.

`[tasks.ci]` and the pre-push gate now include it, and the Quality block lists
it beside the plain run. The exposure is wider than the three tests this branch
repairs — `output/table.rs` styles the manifest path and every status cell
through the same `if_supports_color`, and no test that spawns the binary pins
its colour environment, so they pass today only because their assertions happen
not to straddle a style boundary. This is what makes that prospective rather
than permanent.

The extra run costs seconds: the suite is hermetic and already built by then,
so the second pass compiles nothing.
The step shelled out to `mise run test:color`. This file's own header states the
convention -- explicit, stable workspace commands rather than delegation -- and
every other step here follows it. Delegating also made the hook depend on `mise`
being resolvable in the hook environment, which nothing guarantees.

`Step` carries its own `env` block, so the forced-colour configuration the task
supplied can be expressed here directly.
The colour leg was a second, byte-identical `cargo test --workspace` scheduled
alongside the first in the same target directory. hk runs hook steps in parallel
up to `HK_JOBS` (default: core count) and mise runs a task's `depends` list
concurrently, and cargo releases its build-directory lock before any test binary
starts -- so the two test phases overlapped rather than queued.

That is not safe here. The CLI integration tests keep fixed-name scratch
fixtures under `CARGO_TARGET_TMPDIR` and wipe each with `remove_dir_all` on
entry, so two concurrent runs delete each other's fixtures. `github_summary` is
the sharpest case: if the second run recreates the directory between the first
run's two `check()` calls, `summary.md` comes back empty and the "appended, not
clobbered" assertion fails on a count of 1 -- an unexplainable local failure of
exactly the kind this branch exists to remove.

Serialising costs almost nothing: `FORCE_COLOR` is in no fingerprint, so the
second run's build is a no-op and only its test execution is added. A separate
`CARGO_TARGET_DIR` would isolate the two as well, but it forces a full second
build and roughly ten more gigabytes of target output for a run whose only
difference is one environment variable.

CI was unaffected either way -- its two legs are sequential job steps -- so the
green checks were never evidence about this.
`pre-push`, `mise run ci` and the Linux CI job all run the colour leg; `hk check`
did not. That hook is documented in this file as the manual/CI entry point, so a
contributor could pass `hk check`, believe the gate satisfied, and still be
blocked at push -- two gates that claim to be the same telling them different
things.

It carries the same `depends` on the plain run, so it queues behind it here as
well.
`plain` took `(&DependencyGraph, &TreeOptions)`, so the only input it could ever
receive was `ascii`'s own output -- and `ascii` emits nothing but CSI colour
sequences. Three of the stripper's branches were therefore unreachable by any
test: the non-CSI escape, a CSI final byte other than `m`, and a lone trailing
`ESC`.

The non-CSI branch also holds a latent wrong answer. It consumes the byte after
`ESC` and drops it, then emits the remainder as text -- correct for a
two-character escape, wrong for a string escape. If `label` ever wrapped a crate
name in an OSC 8 hyperlink, the URL would be spliced onto the label and present
as a tree-shape regression rather than as a limit of the stripper.

Taking `&str` instead lets the branches be exercised with synthetic input, and
states the contract the function actually implements rather than the narrower
one its comment claimed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(cli): three tree tests fail whenever FORCE_COLOR is set

1 participant