Skip to content

fix(vm): decode tables in iteration order so >32-entry sequences stay lists - #5

Open
smn wants to merge 1 commit into
sync-upstream-mainfrom
fix-decode-table-ordering-over-32-entries
Open

fix(vm): decode tables in iteration order so >32-entry sequences stay lists#5
smn wants to merge 1 commit into
sync-upstream-mainfrom
fix-decode-table-ordering-over-32-entries

Conversation

@smn

@smn smn commented Sep 3, 2026

Copy link
Copy Markdown

Stacked on #4 (sync-upstream-main). This PR's base is the sync branch so the diff is exactly the one fix commit. Merge #4 first, or retarget this base to main once #4 lands.

The bug

A Lua table holding a dense integer sequence longer than 32 entries decodes to a scrambled, integer-keyed map instead of an ordered list. Lua.Table.deep_cast/1 then mis-casts it to a map, silently dropping the list shape and its order.

{[decoded], _} = Lua.eval!(Lua.new(), "t={} for i=1,41 do t[i]=i*10 end return t")
Lua.Table.deep_cast(decoded)
# before: %{1 => 10, 2 => 20, ...}   (a map, keys scrambled)
# after:  [10, 20, 30, ..., 410]      (an ordered list)

Root cause

The 32 is not a Lua number — it's Erlang's own map representation switch. Erlang maps iterate in ascending key order while they're a flatmap (≤32 entries) but in unordered hash order once they grow into a hashmap (>32). Lua.VM.Value.decode/2 materialized every table through Lua.VM.Table.to_map/1 (an Erlang map), so for a sequence of 33+ it emitted {k, v} pairs that no longer started at {1, _}. deep_cast/1 detects a list by exactly that leading {1, _} pair, so it fell through to the map branch.

The VM itself never lost the ordering — ipairs/pairs iterate 1..n correctly for any size. Only the decode boundary threw it away by routing through a map.

entries Erlang map kind decoded first key deep_cast result
≤ 32 flatmap 1 list ✅
≥ 33 hashmap e.g. 18 map ❌

The fix

Add Lua.VM.Table.to_list/1, which materializes the whole table in pairs/1 iteration order (array keys first in index order, then hash keys in insertion order) by folding the existing, tested next_entry/2. decode/2 now uses it instead of to_map/1. to_map/1 stays for the callers that only need membership (display, string.gsub replacement lookups).

Decoded pairs now match pairs/1 order exactly, so sequences of any size round-trip as ordered, 1-indexed lists.

Tests

  • Lua.VM.Value — a 41-element sequence decodes in order (fails on main, passes here)
  • Lua.VM.Table.to_list/1 — equals a pairs walk; keeps a 41-element sequence 1-indexed; empty table → []
  • Lua.Table — end-to-end: a 41-element Lua list deep_casts to an ordered list, not a map

Full suite (2875), format, dialyzer (0 errors) and the lua53 suite all pass.

Context

Surfaced downstream in the Turn platform, where a Lua app manifest declaring 41 journeys installed nothing and still reported success — the 33+ entry section decoded to a map and was discarded. Worth upstreaming to tv-labs/lua.

🤖 Generated with Claude Code

`Lua.VM.Value.decode/2` materialized tables through an Erlang map
(`Lua.VM.Table.to_map/1`). Erlang maps switch from an ordered flatmap to
an unordered hashmap at 32 entries, so a dense integer sequence longer
than 32 decoded as a scrambled, integer-keyed list of pairs that no
longer started at key 1. `Lua.Table.deep_cast/1` detects a list by its
leading `{1, _}` pair, so it then mis-cast such a sequence to a map —
silently dropping the list shape (and its order) for any array over 32
elements.

The VM itself keeps the sequence ordered (`ipairs`/`pairs` iterate
1..n); only the decode boundary lost it. Add `Lua.VM.Table.to_list/1`,
which materializes the whole table in `pairs/1` iteration order (array
keys first in index order, then hash keys in insertion order) by folding
the existing, tested `next_entry/2`, and decode through it instead of
`to_map/1`. Sequences of any size now round-trip as ordered, 1-indexed
lists.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant