fix(string): index WTF-8 positions without unchecked UTF-8

[proggeramlug]

Fixes #10725.

The UTF-16-position walker already reads bytes through wtf8_step, so it now accepts a byte slice directly. startsWith and endsWith pass the string payload without constructing an unchecked &str from WTF-8 that may contain a lone surrogate. The other walker call site passes its bytes to the new signature.

A runtime unit test covers positions before and after lone high and low surrogates. A gap fixture checks the same cases against Node.

Verification:

• CARGO_TARGET_DIR=/Users/amlug/projects/perry/agent-trees/fix-10645/target RUST_TEST_THREADS=1 cargo test -p perry-runtime --lib starts_and_ends_positions_cross_lone_surrogates -- --test-threads=1
• node --experimental-strip-types test-files/test_gap_10725_wtf8_starts_ends_position.ts
• rustfmt --check --edition 2021 on the three changed Rust files, and git diff --check

Summary by CodeRabbit

• Bug Fixes

  • Fixed position-based startsWith and endsWith behavior for strings containing lone surrogate characters.
  • Preserved correct UTF-16 position handling across surrogate boundaries.
  • Updated string searching to safely handle malformed or nonstandard UTF-8 content.

• Tests

  • Added regression coverage for matching and non-matching searches around lone surrogates.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a3dbc75c-2a44-46b8-9604-c8fb53b351d0

📥 Commits

Reviewing files that changed from the base of the PR and between c1569e2 and 6990fe3.

📒 Files selected for processing (5)

• changelog.d/10989-wtf8-position-search.md
• crates/perry-runtime/src/string/compare.rs
• crates/perry-runtime/src/string/mod.rs
• crates/perry-runtime/src/string/slice_ops.rs
• test-files/test_gap_10725_wtf8_starts_ends_position.ts

Included review availability: Your plan provides up to 8 included reviews per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The UTF-16 offset helper now traverses raw WTF-8 bytes. startsWith, endsWith, and indexOf pass byte slices instead of UTF-8 string views. Tests cover positions around lone surrogates.

Changes

WTF-8 position handling

Layer / File(s)	Summary
Raw payload position conversion crates/perry-runtime/src/string/mod.rs, crates/perry-runtime/src/string/compare.rs, crates/perry-runtime/src/string/slice_ops.rs, test-files/test_gap_10725_wtf8_starts_ends_position.ts, changelog.d/10989-wtf8-position-search.md	utf16_offset_to_byte_offset accepts raw byte slices and performs bounds-checked WTF-8 traversal. Position-based startsWith and endsWith use string payload bytes, and indexOf passes h.as_bytes(). Tests cover matching positions across lone high and low surrogates.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 6990f

String searches now preserve UTF-16 positions across lone surrogates, with regression coverage for the affected behavior. No concrete merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The change meets the coding requirements in issue #10725. The startsWith and endsWith position paths now pass WTF-8 payload bytes to utf16_offset_to_byte_offset instead of creating a string view. The …
Out of Scope Changes check	✅ Passed	The changed files support issue #10725. They update the shared UTF-16 offset helper, its callers, the affected startsWith and endsWith paths, the changelog, and regression coverage. No unrelated produ…
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 4 files. (1 skipped: 1 …
Title check	✅ Passed	The title clearly identifies the main fix: safe WTF-8 position handling for string operations.
Description check	✅ Passed	The description explains the change, references issue #10725, lists affected behavior, and provides focused verification commands. It omits the template headings and checklist, but it contains the req…

✨ Finishing Touches 💡 1

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[proggeramlug]

Landed on main in merge train 257 (#11039, v0.5.1640), main 990b3eeada.

Carried at head 6990fe3356. CI on the train head passed every job except the known public-baseline lint step: all 6 gap shards, cargo-test, e2e-scoped, gc-stress, check, warnings and security-audit green.

This train was split by blast radius after an earlier 35-PR assembly hit five gap regressions: it carries only PRs touching no lowering path. Trains rebase-merge, so commits get new SHAs and GitHub cannot mark this merged. Closed as landed.