Upgrade check: ignore injected query memory faults in the post-upgrade log scan

[groeneai]

Changelog category (leave one):

• CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Stop the upgrade check from failing when work left behind by the stress phase hits that phase's own memory fault injection after the upgrade restart.

Description

Upgrade check (amd_release) reds on Error message in clickhouse-server.log when the only line the post-upgrade scan finds is an injected Code: 241 ... Query memory tracker: fault injected. Three runs in 90 days, all on unrelated PRs, 0 on master; in each job these lines were the whole scan output.

Root cause: stress workers 1 and 6 run with memory_tracker_fault_probability=0.05 (worker 1 in the shared database test_1). Work they leave behind keeps the settings of the query that created it and runs again on the upgraded server:

• a distributed DDL entry stores the initiator's changed settings, so a queued BACKUP ... ON CLUSTER of test_1 replays with fault injection (BackupsWorker: Failed to make internal backup);
• a pending Distributed batch is sent with its stored settings, and the receiving async-insert flush fails (AsynchronousInsertQueue: Failed insertion, empty query id, so the existing executeQuery entry misses it).

Nothing enables fault injection on the upgraded server itself, so this message can only come from carried-over stress work. The change adds it as one fixed-string scan entry, the same string the stress smoke check already tolerates (ci/jobs/scripts/stress/stress.py). It is MemoryTracker's message for both injection sites and does not match a real memory limit exceeded error.

Validation on the three jobs' real clickhouse-server.upgrade.log, with the scan cut from the script itself: base reports 1/1/2 lines, the fix 0, and removing the new entry restores the base output. A real backup failure, memory-limit error and DDL error appended to the log are still reported. 19 green upgrade logs give identical (empty) output.

The large DDL backlog in the #123464 job comes from 26.9 wedging its DDLWorker on KILL PART_MOVE_TO_SHARD, fixed on master by #122132.

The three failing runs

• 2026-10-02, #123464, BackupsWorker
• 2026-09-25, #117943, AsynchronousInsertQueue
• 2026-09-21, #120725, AsynchronousInsertQueue x2

Related: #122132
Related: #120990

---

Workflow [PR]
Sync PR [sync-upstream/pr/123581]

Version info

• Merged into: 26.10.1.1454-master (included in 26.10 and later)

[groeneai]

Internal second-model review: adjudication log (click to expand)

Pre-publication review: my own cold review plus an independent model (engine: codex; 0 findings). 1 finding total.

#	Sev	Finding	Verdict	Evidence / action
1	💡	The comment says the fault injection comes from "stress worker 1", but stress.py sets memory_tracker_fault_probability for every worker with i % 5 == 1 (workers 1 and 6 with the default --num-parallel)	AGREE, noted, not blocking	The entry is a fixed string and does not depend on which worker set the probability; only the wording is narrower. The PR description states workers 1 and 6.

Severity: ❌ blocker / ⚠️ major / 💡 nit. DISAGREE verdicts carry recorded evidence and are terminal per finding.

Session id: cron:clickhouse-review-slot-48:20261002-153100

[clickhouse-gh]

Workflow [PR], commit [cdc39b0]

Summary: ✅

---

AI Review

Summary

This PR teaches the upgrade-check log scrubber to ignore Query memory tracker: fault injected, matching the stress harness's existing treatment of the same injected failure. I checked the diff, the current upgrade_runner.sh logic, the prior PR discussion, and sampled the cited failing upgrade logs from both BackupsWorker and AsynchronousInsertQueue; they are consistent with carried-over stress-phase work replaying after restart, and I did not find a concrete compatibility signal that this new allow-list entry would now hide.

Final Verdict

• Status: ✅ Approve

[groeneai]

CI finish ledger - ea5ff28

Every failure below has an owner: a fixing PR (mine or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test	Reason	Owner / fixing PR
Stress test (arm_release) / Cannot start clickhouse-server, Check failed	Not from this PR, which touches only upgrade_runner.sh. The restart after the stress phase could not load t_ttl_replicated__fuzz_3, an AST fuzzer clone of the 05292_standalone_expression_scalar_subquery_with_join table whose TTL subquery reads merge() (Code: 393 THERE_IS_NO_QUERY while loading metadata)	#123423 (mine, merged 16:03Z) keeps that file's DDL out of the fuzzer. The run was dispatched at 15:56Z, before it merged, so I merged master @ cdc39b05dbf40fa32232cb3baf8f5ddacffad6b4
Mergeable Check, PR	rollups, red only because of the row above	#123423 (mine, merged)

All 149 checks completed, Finish Workflow is green and CH Inc sync passed. CIDB has no other FAIL or ERROR row on this head.

Session id: cron:our-pr-ci-monitor:20261002-190341