fix(hir): preserve code points in typed string for-of

[proggeramlug]

Summary

Typed string for...of loops now yield Unicode code points: ä中😀Ö produces four values, including the complete two-code-unit emoji. Both function-body and module-initialization lowering reuse the runtime string iterator's WTF-8 conversion before traversing an Array<String>.

Changes

• Materialize typed strings into code-point arrays instead of indexing their UTF-16 code units. Preserve the iterator holder type when a nested loop is lowered through a forced lazy arm.
• Add full-content regression assertions for typed parameters, local/module variables, dynamic strings, adjacent astral characters, lone surrogates, empty/ASCII strings, assignment heads, break, continue, return, nested iterator fallback, and for await...of. Explicitly check that bracket indexing and charCodeAt still expose individual surrogate halves.
• Add a changelog fragment. No version bump or runtime helper changes.

This reuses existing conversion machinery and introduces O(n) temporary storage for typed string loops, including when a loop exits early. The ASCII control remains correct but is slower due to materialization (about 2.4–4.4× in this development-profile sweep). A streaming fast path would be a separate optimization. Unicode indexed-access prefix decoding is unchanged and remains tracked by #10055; no shared string/char_ops.rs edits overlap #10055 or #10061.

Related issue

Fixes #10062.

Test plan

• Fresh baseline from origin/main at 603b074ace01464bc66fc07cc8d532f26ccf5a0f; reproduced the exact reported diagnostic and all three historical Unicode checksum mismatches. The new regression fails on the baseline compiler and passes with this change.
• cargo build --profile perry-dev --locked -p perry -p perry-runtime-static -p perry-stdlib-static, with matching archives pinned through PERRY_RUNTIME_DIR, before and after.
• cargo test --profile perry-dev --locked -p perry-hir --lib -- --test-threads=1: 397 passed, 1 ignored.
• Native regression byte-compared with Node 26.5.1 at LLVM optimization levels 0, s, and z, both normally and under moving-GC stress. Each full stress run reported 476 copying minors / 19,675 moved objects and preserved the complete expected output.
• The original bounded diagnostic also passed with default native roots under moving-GC stress (15 copying minors / 28 moved objects).
• Related regressions pass: test_gap_9431_array_from_lone_surrogate.ts, test_gap_sso_concat_string_index.ts, and test_gap_array_proto_iterator_replaced_7542.ts.
• Exact issue Unicode/ASCII benchmark sources run serially at all five sizes; every post-fix checksum matches Node.
• Rust formatting, file-size gate, Node-version consistency, test registration, and git diff --check.

Local native validation used Windows x64, Ryzen 5 7640HS, LLVM 22.1.8, the pinned Rust nightly, and the perry-dev profile. The comprehensive/related fixtures use PERRY_RS4GC=0 because their iterator/async lowering introduces Windows funclet EH, which the default RS4GC pipeline rejects under existing #7354. The issue's diagnostic and benchmarks use the default configuration. Stress used seed 10062, rate 1, allocation floor 0, and evacuation verification.

Screenshots / output

Exact bounded diagnostic:

before: 1:228,1:20013,1:55357,1:56832,1:214,
after:  1:228,1:20013,2:55357,1:214,
Node:   1:228,1:20013,2:55357,1:214,

Median milliseconds per invocation from the unchanged issue benchmark driver. These are development-profile measurements on this Windows host, not release-performance claims or comparisons with the issue's macOS baseline. Baseline Unicode timings below produced incorrect checksums.

n	Unicode before	Unicode after	Unicode Node	ASCII before	ASCII after	ASCII Node
100	0.139	0.025	0.009	0.009	0.022	0.008
1,000	12.284	0.296	0.088	0.093	0.221	0.085
10,000	1,159.573	2.691	0.891	0.838	2.553	0.841
100,000	timeout (60 s)	27.670	8.696	8.513	25.906	8.535
1,000,000	skipped after timeout	425.772	82.804	83.778	369.734	83.637

Post-fix Unicode checksums: 383743935, 926322819, 754247844, 103819597, 618978563.
ASCII checksums before and after: 379242312, 993701591, 233623069, 542806320, 180357488.

Checklist

• No workspace version bump or changes to CLAUDE.md / CHANGELOG.md.
• Commit uses the repository's fix: convention.
• Read CONTRIBUTING.md and followed the Code of Conduct.

Summary by CodeRabbit

• Bug Fixes

  • Fixed string for...of loops so astral characters are yielded as complete code points instead of separate surrogate halves.
  • Preserved existing code-unit behavior for indexed access and charCodeAt.
  • Corrected behavior across regular and asynchronous loops, module-level iteration, control flow, assignment targets, and lone surrogates.

• Tests

  • Added comprehensive regression coverage for string iteration scenarios, including mixed characters, adjacent astral characters, empty strings, and nested loops.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: fc920194-3dd4-4c33-b571-b0e78e4f8e8b

📥 Commits

Reviewing files that changed from the base of the PR and between 603b074 and 066b226.

📒 Files selected for processing (4)

• changelog.d/10080-string-for-of-code-points.md
• crates/perry-hir/src/lower/stmt_loops.rs
• crates/perry-hir/src/lower_decl/body_stmt.rs
• test-files/test_gap_string_for_of_code_points.ts

Included review availability: Your plan provides up to 8 included reviews per hour; 5 remain after this review.

---

📝 Walkthrough

Walkthrough

String for...of lowering now converts non-lazy string iterables into code-point arrays. Regression tests cover astral characters, lone surrogates, control flow, indexing, module initialization, and for await...of.

Changes

String for-of lowering

Layer / File(s)	Summary
Materialize string code points crates/perry-hir/src/lower/stmt_loops.rs, crates/perry-hir/src/lower_decl/body_stmt.rs	String iteration uses Expr::ForOfToArray and an Array<String> holder for non-lazy loops.
Validate iteration semantics test-files/test_gap_string_for_of_code_points.ts, changelog.d/10080-string-for-of-code-points.md	Regression tests cover code points, surrogate handling, loop control, indexing, module initialization, and async iteration. The changelog records the fix.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 066b2

String iteration now preserves Unicode code points while retaining UTF-16 indexing behavior, with coverage for surrogate and control-flow cases. No actionable merge risk remains.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 58.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 3 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.
Linked Issues check	❓ Inconclusive	The implementation satisfies the core coding requirements in #10062. Both lowering paths use Expr::ForOfToArray with the existing WTF-8 conversion, and Array<String> holders preserve complete code…	The reviewed evidence needs benchmark results or reviewable benchmark coverage that compares Unicode checksums with Node at all completed sizes and retains the ASCII control.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the main change: preserving Unicode code points in typed string for-of loops.
Description check	✅ Passed	The description includes all required sections, explains the implementation and scope, references issue #10062, documents extensive validation, and completes the checklist. It also notes performance i…
Out of Scope Changes check	✅ Passed	The changes stay within #10062. The two HIR lowering updates implement typed string code-point iteration. The regression test verifies the linked behavior. The changelog documents the same fix. No unr…

Full details: Linked Issues check

Explanation

The implementation satisfies the core coding requirements in #10062. Both lowering paths use Expr::ForOfToArray with the existing WTF-8 conversion, and Array<String> holders preserve complete code points. The new test covers typed parameters and locals, dynamic strings, module initialization, astral and BMP text, adjacent astral characters, lone surrogates, control flow, nested lazy lowering, async iteration, and UTF-16 indexing with charCodeAt. The diff does not include Unicode benchmark checksums or benchmark-source coverage, so that #10062 requirement cannot be verified from the reviewed evidence.

Full details: Docstring Coverage

Explanation

Docstring coverage is 58.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 3 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/issue-10062-string-for-of

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[proggeramlug]

Landed with preserved authorship via #10082 (rebase merge 435d639); the merged tree was verified byte-identical to the audited Train160 tree. Closing this superseded original PR.