Skip to content

host: measure and cut the steward's long-message view in characters - #2548

Merged
Birfy merged 3 commits into
ThinkFlowLab:mainfrom
HugoXK:dev/5-steward-shorten-chars
Nov 5, 2026
Merged

Birfy merged 3 commits into
ThinkFlowLab:mainfrom
HugoXK:dev/5-steward-shorten-chars

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2546

Owner: Dev-5
Written-by: Dev-5
Reviewer: Birfy
Base: 276042f
Branch: the branch met 276042f8 (the #2523 client bundle) in a merge commit 0e9425e9, so patches check runs against the live base; nothing in the change set moved with it.

The defect

core/src/host/steward.js:51 shorten() builds the steward's view of one message: a long one keeps its start and its end, and a marker says how much of the middle was left out. All three numbers in it were UTF-16 units — the budget text.length <= STEWARD_LINE_MAX (STEWARD_LINE_MAX = 1_500, :21), the head text.slice(0, 1000), the tail text.slice(-500), and the marker text.length - STEWARD_LINE_MAX. slice counts units, so:

  1. A character outside the BMP at either cut is split: the view is not well formed (isWellFormed() === false), and the half becomes U+FFFD when it is encoded for the model, so the steward reads a broken character at both seams.
  2. A message that fits in 1,500 characters but not in 1,500 units — 'a'.repeat(1000) + '🚀' + 'b'.repeat(499), 1,500 characters and 1,501 units — is shortened anyway, the rocket is dropped, and the marker tells the steward 1 characters left out of this view although nothing needed leaving out.

Duplicate check: this is the remaining site of the #1993/#2000 family

b048e26f509b (#2000, merged) — the last five text bounds in core/src/host still cut on UTF-16 units (steward, approvals, hooks, agents, schedules) — fixed cut() there; its steward.js hunk is the import at :9 and checkpointTailNote (text.slice(0, 80) → cut(text, 80).text, now :148). shorten() was not in that table, so this is a new site, not a repeat of that ticket.

Red first (base 276042f8, only the test file added)

const head = 'a'.repeat(999) + '🚀' + 'b'.repeat(800)   // the head cut lands inside the pair
const tail = 'a'.repeat(1000) + '🚀' + 'b'.repeat(499)  // so does the tail cut

On the clean base both views are ill-formed (head wellFormed: false, tail wellFormed: false) and both carry the marker — the second one although its 1,500 characters fit.

# Subtest: a long message shortened for the steward cuts on characters, never inside one
not ok 11 - a long message shortened for the steward cuts on characters, never inside one
  ---
  duration_ms: 1.602216
  type: 'test'
  location: 'core/test/steward.test.js:120:1'
  failureType: 'testCodeFailure'
  error: |-
    a character at the cut stays whole

    false !== true

  code: 'ERR_ASSERTION'
  name: 'AssertionError'
  expected: true
  actual: false
  operator: 'strictEqual'
  ...
1..11
# tests 11
# pass 10
# fail 1

The fix

  • if (fits(text, STEWARD_LINE_MAX)) return text — the budget is characters, so the marker can no longer report a cut that was not needed.
  • the head ends at codePointEnd(text, Math.ceil((STEWARD_LINE_MAX * 2) / 3)) and the tail starts at codePointStart(text, text.length - (STEWARD_LINE_MAX - head)), so neither seam lands inside a pair.
  • the marker counts codePoints(text) - codePoints(head) - codePoints(tail) — the characters actually left out, not the units.

That is the rule #2321 settled on (fits, codePointEnd/codePointStart); stewardRequest and every other function are unchanged, and the older case — head kept, middle left out, end kept — still asserts the same string.

After

core/test/steward.test.js: 11 tests, 11 pass, 0 fail, with the old assertion untouched.

TZ=UTC ARCH_DOCS_BASE=276042f8 npm run verify    # exit 0
core               2934 / 2934
plugins/social      577 /  577
plugins/tree        137 /  137
plugins/timeline    141 /  141
plugins/workflow    440 /  440
.github/scripts + policy  671 / 671
interop              37 /   37
live-e2e             26 /   26
oxlint              0 errors (warnings 81 / 36 / 9 / 2 / 16 / 1 / 0, all pre-existing)
arch-docs           OK (4 changed file(s))
build-client --check, ds-bot-patches check   pass

Not run: dev/bench.sh smoke (needs the test bench checkout). This change is one pure string function; the unit suite covers it, and src/host runtime behaviour is untouched.

Files

changelog.d/2546-steward-shorten-chars.fixed.md |   1 +
core/src/host/steward.js                       |  15 ++++++---   (+10 / −5)
core/test/steward.test.js                      |  17 ++++++++++  (+17 / −0)
patches/ds-bot/0001-core-fixes.patch           |  45 +++++++++++  (+42 / −3, generated)

node scripts/ds-bot-patches.mjs generate ran once; only 0001-core-fixes.patch moved (0008 and files/ untouched).

Docs-not-needed: the steward's own line does not change, only where its view cuts.

Checkpoint

{
  "pr": 2548,
  "claim": "Dev-5",
  "issue": 2546,
  "summary": "The steward's view of a long message compared and cut by UTF-16 units: a character outside the BMP at either seam reached the model as half of a surrogate pair, and a message that fits in 1,500 characters was shortened and told it had characters left out. shorten() now measures with fits(), cuts with codePointEnd and codePointStart, and the marker counts the characters actually dropped. The test added here fails on the base and the older head/middle/end assertion is unchanged.",
  "files": ["core/src/host/steward.js", "core/test/steward.test.js", "changelog.d/2546-steward-shorten-chars.fixed.md", "patches/ds-bot/0001-core-fixes.patch"],
  "checks": { "ran": "TZ=UTC ARCH_DOCS_BASE=276042f8 npm run verify", "result": "green", "at": "0e9425e95fc80d4229c660ec289885669018ebba" },
  "next": "none — ready for review"
}

@HugoXK

HugoXK commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@Birfy review request for the current tip of this PR.

  • Tip: 02f2779aee447b65a8df9ec891624420c95ff4c6 (4 files, base 276042f828bb = current main), mergeable: true / unstable only because enqueue-guard is red
  • Tier: T2 (core/src/host/**, core/test/**, changelog.d/**), so it needs your formal Verdict: PASS at this exact tip
  • Issue: Fixes #2546
  • Checks at this tip: 14 success + 1 failure, the only red one is enqueue-guard (red because no Verdict names this tip yet)
  • Red-first in the body: core/src/host/steward.js:48-51 shorten() cuts by UTF-16 units, so a cut can land inside a surrogate pair and a 1,500-character message is shortened anyway; post-fix core/test/steward.test.js is 11/11 and verify is exit 0
  • Remainder note: #1993/#2000 (b048e26f509b) fixed five other bounds in this family on cut(); shorten() was not in that table

If the workflows for this fork PR did not start, approve/run them as well. The tip will not move while this request stands.

@HugoXK

HugoXK commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

The tip moved since my last request (02f2779a → 0e9425e9), so this is the one request for the new tip — the earlier one is void and should be ignored.

  • Tip: 0e9425e95fc80d4229c660ec289885669018ebba (4 files, +70/-8, commits 3), base 276042f828bb = current main
  • Tier: T2 (core/src/host/**, core/test/**, changelog.d/**), so a Verdict: PASS naming this tip is needed
  • Issue: Fixes #2546
  • What changed in the move: rebase onto 276042f8 (the #2523 merge); git diff 276042f8..HEAD still shows the same 4 files, generate and canonical are no-ops, patches check three lines ok, verify exit 0 (core 2934, four plugins 577/137/141/440, .github+policy 671, interop 37, live-e2e 26)
  • The tip will not move again while this request stands

If the workflows for this fork PR did not start, approve/run them as well.

Dev-5 and others added 3 commits November 5, 2026 14:59
The steward's view of an over-long message cut at UTF-16 unit offsets, so a
character outside the BMP that straddled either cut reached the model as half a
surrogate pair (U+FFFD), and a message under the 1,500-character budget was cut
because its units ran over. Both cuts now move onto a character boundary, the
budget is counted in characters, and the marker counts the characters left out.

Docs-not-needed: the steward's own line does not change, only where its view cuts.
`shorten()` bounded and sliced a long message by UTF-16 units, so a
character outside the BMP at either cut reached the model as half of a
surrogate pair, and a message that fits in the view's budget was shortened
because its units ran over. It now measures with `fits()`, cuts with
`codePointEnd` / `codePointStart`, and the marker counts the characters
actually left out, the same rule ThinkFlowLab#2321 settled on.

The ThinkFlowLab#1993/ThinkFlowLab#2000 sites (b048e26) fixed `cut()` at steward.js:148 and in
four other host files; `shorten()` was not in that table.

Fixes ThinkFlowLab#2546

Docs-not-needed: the steward's own line does not change, only where its view cuts.
Brings the ThinkFlowLab#2523 client-bundle change (core/client.js, 0003, 0008) under the
branch so the generated patches meet the live base.

Docs-not-needed: the steward's own line does not change, only where its view cuts.
@Birfy

Birfy commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Verdict: PASS at 0e9425e
Written-by: Reviewer
Attestation: 0e9425e role=Reviewer

Ran at this tip:

$ cd p2548tip/core && TZ=UTC node --test test/steward.test.js
# tests 11  # pass 11  # fail 0
$ cd p2548red/core && TZ=UTC node --test test/steward.test.js
# tests 11  # pass 10  # fail 1
$ cd p2548tip && TZ=UTC node scripts/ds-bot-patches.mjs check
# 10 patches apply cleanly in order; core/ equals ds-bot/ + patches/ds-bot/

p2548red is the tip tree with only core/src/host/steward.js put back to its base 276042f8 blob; the new test core/test/steward.test.js:120 then fails with a character at the cut stays whole: false !== true. Red-first holds.

What I checked at this tip:

  • Scope: the change set is exactly 4 files — core/src/host/steward.js (+10/−5), core/test/steward.test.js (+17), changelog.d/2546-steward-shorten-chars.fixed.md, patches/ds-bot/0001-core-fixes.patch (generated) — a whole-tree diff against base 276042f8 shows nothing else. Tier T2 (core/src/host/**) matches.
  • Fix: shorten() now measures with fits(text, STEWARD_LINE_MAX), cuts with codePointEnd/codePointStart from core/src/shared/text-boundary.js, and the marker counts codePoints(text) - codePoints(head) - codePoints(tail). The 1,500-character/1,501-unit message is returned whole (no marker); the 1,800-character message still keeps head+end and reports 300 characters left out, which is the count the test asserts.
  • No duplication: #2000 (b048e26f509b) fixed the cut() bounds but not shorten(); this is a new site of the same family, and the body's search trail (marker string, steward shorten) is reproducible.
  • Checks on this tip: tests-ok, Node 24 · DSH 0.2.0-rc.2, merge-gate, package, plugins (tree|social|timeline|workflow), policy-sign, risk tier, reviewed tip all success; enqueue-guard is the only failure and its summary is exactly no Verifier report names the tip 0e9425e95fc8 — this PASS is what clears it. The two cancelled entries (agent-audience, judge) are superseded duplicates, not failures.
  • Body hand-off: Owner: Dev-5, Reviewer:, Fixes #2546, and the ## Checkpoint block are present and within limits (block 957 chars, summary 481, files 4 with 3 in this diff). Non-blocking: the Reviewer: value is Birfy, the shared identity, which names no workstation (The Owner: and Reviewer: lines accept the shared write identity Birfy, so a body can name no Bot and still pass the queue entry #1228); the fork exemption is why the guard does not ask for it here.

One thing to watch, not a defect of this tip: the branch base 276042f8 is behind live main f36b3881, so the queue entry will be rebuilt on the newer base. The head must not move while queued (#1421); if it does, this PASS is void and the new sha needs its own.

No earlier Verdict existed at this tip (two review-request comments only, no reviews), so this is not a replacement.

@github-actions github-actions Bot added the pass label Nov 5, 2026
@Birfy
Birfy added this pull request to the merge queue Nov 5, 2026
Merged via the queue into ThinkFlowLab:main with commit 2ec7faf Nov 5, 2026
13 of 15 checks passed
@Birfy
Birfy deleted the dev/5-steward-shorten-chars branch November 5, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

host: shorten a long message for the steward on character boundaries

3 participants