Skip to content

Fix read/write asymmetries, script serialization and global state; revive the test suite - #8

Merged
turtleisaac merged 13 commits into
mainfrom
claude/pokeditor-bug-deep-dive-9mm03b
Aug 27, 2026
Merged

turtleisaac merged 13 commits into
mainfrom
claude/pokeditor-bug-deep-dive-9mm03b

Conversation

@turtleisaac

Copy link
Copy Markdown
Owner

Part of a four-repo audit of the PokEditor toolchain. Depends on turtleisaac/Nds4j#2 — merge that first.

⚠️ Contains breaking public API changes — see the bottom. PokEditor's companion PR is already updated for them.

The governing defect

A field's parse path and its serialize path disagreed. These need no user edit to bite: loading a ROM and saving it is enough. Every fix below was verified with a byte-exact round-trip harness over randomized inputs.

Field Read as Written as
Trainer alt-form 6 bits (>> 10) 3 bits (& 0x7)
Item stat boosts & 0xf & 0x3
Item crit stage & 0x3 & 0xf — overflow bled into the PP Up flags
EV yields 2-bit fields setters accepted 4, write never masked
DPPt encounters skip(0x2C) skip(0x2C) — 44 bytes re-emitted as zeroes

Concretely: every Unown letter past H and every Arceus plate form was silently remapped or reverted; Speed and Sp. Def stat stages lost their top two bits; a crit stage of 4 turned an item into a PP Up; and ~600 pl_enc_data subfiles lost a rate word plus a five-slot water encounter table on every save.

Also in this class: trainer type flags and the AI bitfield were re-encoded from only the bits the tool understood, zeroing everything else; the upper nibble at item offset 0x14 and bits 12-15 of the personal EV halfword were read into nothing and written as zero; level-script trigger numbers were narrowed through a signed byte cast, so any script above 127 sign-extended.

Scripts

  • Command 225 is goto_if_trainer_defeated, not a comparator. It was listed in isDoIfCommand, so loading any script that uses it threw IllegalStateException and aborted the entire NARC. Its jump target is now registered as a label and recomputed on save instead of being emitted from the old file's absolute address.
  • ActionCommand(int, int) never assigned its id — the fallback taken whenever a movement name lookup misses. Movements_Hg.json omits real HGSS opcodes 96-103 and 106, so those movements re-serialized as movement 0.
  • visitAction_command only called super, so recompiling a script dropped every movement command while keeping the action_N: labels pointing at them.
  • An unresolvable label serialized as a branch to absolute offset 0.
  • Trainer AI opcodes were written 16-bit though every macro declares .word and the reader reads readUInt32().
  • Hex literals crashed both algebra visitors — and Convenience_Hg.txt:92 already contains .if \item < 0x4000. The macro files are user-editable, so one added hex constant was a load-time crash for every script in the ROM.

Text

9-bit compression only guarded characters up to 0x3FF, so a newline (0xE000) or a VAR placeholder overran its slot and overwrote following characters. Unmapped characters were silently written as code 0 — paste dialogue with a typographic apostrophe and it became a null glyph in-game, reported only to a console with no window attached. Bank tables, VAR sequences and escape decoding are now bounds-checked, so a message ending in a stray backslash no longer aborts the whole ROM save.

Global state

  • PersonalParser's TM/HM tables were static and never reset: open ROM A, then save ROM B, and ROM A's TM list was written into ROM B's ARM9.
  • The sprite parser's party-icon header cache was static on a Guice singleton; when unset the write silently skipped, shifting the entire party-icon NARC by seven files.
  • Game.region was mutable state on a global enum constant.
  • Diamond and Pearl fell through every initialize switch with no default, so a D/P ROM half-initialized and failed far from the cause with an unrelated NPE.
  • The sprite parser mutated the shared ARM9 buffer without holding its lock, while processDataList in the same class does.

The test suite never ran

Nine of the ten parser test classes were declared public static class — which Surefire's default **/*$* exclude drops and JUnit only descends into for @Nested. Only one ran. Proven by converting them: the suite went from 2 discovered tests to 22.

And the one surviving round-trip test could not fail:

if (Arrays.equals(originalNarc.getFile(idx), outputFile)) {
    assertThat(outputFile).isEqualTo(originalNarc.getFile(idx));  // tautology
}
else if (outputFile.length != 0) {                                // empty ⇒ nothing asserted
    assertThat(rebuiltResult).isEqualTo(outputFile);              // output vs output
}

A parser returning zero bytes for every subfile passed this green — which is exactly how TrainerAiData.save(), an empty stub returning a zero-length buffer, survived. It now compares against the original and asserts the file count.

The suite also hard-required a copyrighted ROM beside the pom and failed the build without one; ROMs are now located via -Drom.dir and missing ones skip. Added the sprite parser binding and test class, which had no coverage at all.

SignedFieldTest is new and needs no ROM: it pins the fields the format defines as s8 across the whole 0..255 domain, asserting both that the sign survives and that the bytes round-trip.

Breaking API changes

// Game — region is no longer mutable state on the enum constant
public record BaseRomInfo(Game game, Region region) {}
public static BaseRomInfo parseBaseRom(String)     // was: static Game parseBaseRom(String)
@Deprecated public Region getRegion()              // still works

// Tables
public static void initialize(Game, Game.Region)   // NEW, preferred
@Deprecated public static void initialize(Game)    // delegates
public int getPointerOffset()                      // now THROWS when unset (was: returned 0)

// PersonalParser — static -> instance, so two ROMs cannot share TM tables
public int[] getTmMoveIdNumbers()                  // was: public static final int[] tmMoveIdNumbers
public int getTmMoveIdNumber(int tmID)             // NEW
public void updateTmType(int, int, List<MoveData>) // was: static

Also newly throwing where they previously corrupted silently: PersonalData.set*EvYield (rejects > 3), setDexColor (rejects > 0x7F), ItemData.setStatBoosts, and TextBankData (new TextEncodingException).

Verification

mvn test → BUILD SUCCESS, 28 tests discovered, 0 failures, 11 skipped (ROM-dependent).

🤖 Generated with Claude Code

https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob


Generated by Claude Code

claude added 2 commits August 23, 2026 07:55
The governing defect in this module was that a field's parse path and its
serialize path disagreed. Those bugs need no user edit to bite: loading a ROM
and saving it is enough to corrupt data. Every fix below was verified with a
byte-exact round-trip harness over randomized inputs.

Bitfield and padding asymmetries:
- Trainer alt-form was read as 6 bits but written masked to 3, so every form
  above 7 was silently remapped or reverted.
- Item stat boosts were read with a 4-bit mask and written with a 2-bit one;
  the very next field had the inverse error, and its overflow bled into the
  PP Up flags.
- EV yields are 2-bit fields, but the setters accepted 4 and the write path
  never masked, so each yield corrupted the next. Dex colour had the same
  shape of bug against the sprite-flip bit.
- 44 bytes of DPPt encounter data were skipped on read and re-emitted as
  zeroes, wiping a rate word and a five-slot water table in ~600 subfiles.
  The item and personal padding bytes had the same pattern.
- Trainer type flags and the AI bitfield were re-encoded from only the bits
  the tool understood, zeroing everything else.
- The upper nibble at item offset 0x14 and bits 12-15 of the personal EV
  halfword were read into nothing and written as zero.
- Level script trigger numbers were narrowed through a signed byte cast, so
  any script above 127 sign-extended.

Text:
- 9-bit compression only guarded characters up to 0x3FF, so a newline or a
  VAR placeholder overran its slot and overwrote following characters.
- Unmapped characters were silently written as code 0, turning a pasted
  apostrophe into a null glyph; they now report which character failed.
- Bank tables, VAR sequences and escape decoding are bounds-checked, so a
  message ending in a stray backslash no longer aborts the whole ROM save.
- Character 245 collided with 223 in the reverse map and could not survive a
  save.

Scripts:
- Command 225 is goto_if_trainer_defeated, not a comparator, so loading any
  script using it threw and aborted the entire NARC. Its jump target is now
  registered and recomputed on save instead of being emitted from the old
  file's absolute address.
- ActionCommand(int,int) never assigned its id, so every movement whose name
  is absent from the tables re-serialized as movement 0.
- visitAction_command was a no-op, so recompiling a script dropped every
  movement command while keeping the labels that pointed at them.
- An unresolvable label serialized as a branch to offset 0.
- Trainer AI opcodes were written 16-bit though the macros declare 32-bit.
- Hex literals in the user-editable macro files crashed both algebra
  visitors; negative word parameters were emitted in a form the grammar
  cannot re-lex.

Global state:
- PersonalParser's TM/HM tables were static, so opening one ROM and saving
  another wrote the first ROM's TM list into it. Same class of bug in the
  sprite parser's party-icon header cache, which silently shifted the icon
  NARC by seven files when unset.
- Game.region was mutable state on an enum constant. parseBaseRom now
  returns the region alongside the game and Tables.initialize takes it
  explicitly; the old accessor remains, deprecated.
- Tables entries are reset between initialize calls and now throw when read
  before being set, rather than returning a usable-looking zero.
- Diamond and Pearl fell through every initialize switch with no default, so
  a D/P ROM half-initialized and failed far from the cause.
- The sprite parser mutated the shared arm9 buffer without holding its lock.

Tests:
- Nine of the ten parser test classes were declared static, which Surefire's
  default inner-class exclude drops and JUnit will not descend into. Only one
  ran. They are now @nested, taking the suite from 2 discovered tests to 22.
- outputMatchesInput asserted a value against itself under a guard that had
  already established equality, skipped all checks when the output was empty,
  and otherwise compared the output to itself rather than to the original --
  so a parser returning zero bytes for every subfile passed. It now compares
  against the original and asserts the file count.
- The suite hard-required a copyrighted ROM beside the pom and failed the
  build without one; ROMs are located via -Drom.dir and missing ones skip.
- Added the sprite parser binding and test class, which had no coverage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Nds4j's MemBufReader.readByte() now returns an unsigned value in 0..255,
matching Buffer.readByte() and readUInt8(). Five fields here are defined by
the Gen IV formats as signed 8-bit and were relying on the old signed
behaviour, so they began parsing as large positive numbers instead:

  MoveData.priority              -7 (Trick Room, Roar) read back as 249
  ItemData.evYields              EV-reducing berries yield negative EVs
  ItemData.friendshipChangeAmounts  bitter berries lower friendship
  PokemonSpriteData.globalFrontYOffset / shadowXOffset  signed offsets

The bytes were never at risk: every write path narrows back through a byte,
so 249 and -7 both serialize to 0xF9 and the file round-trips either way.
What broke is the value. PokEditor declares the priority column's range as
{-128, 127}, so the sheet would have displayed 249 and then rejected it as
out of range on the next edit.

Each read now converts explicitly at the point of the read. PokemonSpriteData
.unknownByte is left unsigned: it is an opaque passthrough written back with
a cast, so it has no signed interpretation to preserve.

Checked and deliberately unchanged:
- ItemData hp/ppRecoveryAmount are genuinely unsigned (0xFF is the "restore
  all" sentinel) and already mask.
- LearnsetData reads its packed halfword as a signed short, but compares the
  terminator in the short domain and masks both extracted fields, so sign
  extension cannot reach either value.
- EvolutionData's halfwords never exceed 0x7FFF in practice.

Adds SignedFieldTest, which asserts both halves of the property: parsed
values carry the right sign across the whole 0..255 domain, and the bytes
still round-trip. It needs no ROM. Reverting the priority cast fails two of
its tests while the round-trip test keeps passing, which is the distinction
being drawn here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
claude and others added 11 commits August 23, 2026 08:21
Nds4j is released as 1.0.0 in the companion branch; 0.1.0 no longer exists,
so this declaration has to move with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Cut from 1.0-SNAPSHOT so that PokEditor 3.2.0 -- a release version -- no
longer depends on a mutable snapshot, which would make that build
irreproducible.

The audit in this branch is where the API changed, so this is the right
place to draw the line: parseBaseRom returns a BaseRomInfo record rather
than mutating region state on the Game enum, Tables.initialize takes the
region explicitly, PersonalParser's TM/HM tables moved from statics to
instance state, and several setters now throw where they previously
corrupted silently. From here, semantic versioning puts further breaking
changes behind 2.0.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
The README is the documentation for external consumers -- it explicitly
invites people to use this library in their own Gen 4 projects -- and its
opening usage example no longer compiled: parseBaseRom returns a BaseRomInfo
record now, and Tables.initialize takes the region explicitly. Breaking a
documented public API without updating the document that teaches it left
anyone following that guide with code that does not build.

The example is now a compiled transcription rather than an eyeballed edit.

Also corrects the build instructions, which told readers a HeartGold and a
Platinum ROM were required in the repo root. That is no longer true: the
suite builds and passes without one, skipping the tests that need real game
data. Both places now describe -Drom.dir instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Runs the full suite on every push and pull request.

Nds4j is not on Maven Central at the version this module depends on, so CI
builds it from source. It prefers a branch of the same name when one exists
and falls back to main otherwise: a change spanning both repositories is
developed on matching branches, and building this module against a stale
Nds4j would either report a failure that does not exist or hide a real one.
Both paths were exercised against the live repositories before committing.

Verified by running the same command locally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Every format narrows on the way out - a (short) cast, writeBytes' i2b, or an
explicit mask - so a value that did not fit was written back as a different
one with nothing reported. A learnset move of 512 was stored as move 0, which
reads back as "learns nothing"; a level of 200 was stored as 72. The only way
to notice was to reopen the file and find the wrong data.

FieldWidth checks a value against the field it is about to occupy and names
the field, the value and the limit when it does not fit. It is applied on the
write path rather than in the setters: some formats load through their setters,
and a file already containing an unusual value must still open. What must not
happen is writing a value back out as something other than what was set.

Applied to learnsets (7-bit level, 9-bit move), moves (every field, including
priority as a signed byte), personal (whose setters guarded only the upper
bound, so setHp(-1) was stored as 255) and evolutions.

Two further defects surfaced while testing this:

  - Move 511 at level 127 packs to 0xFFFF, which is the end-of-learnset
    marker. Both fields are individually in range, so only the packed value
    reveals the problem: saving that entry wrote a terminator, and it and
    every entry after it disappeared on reload. It is now refused with an
    explanation. An exhaustive sweep of all 65536 combinations confirms this
    is the only such hole.

  - EvolutionData read its fields with readShort(), which sign-extends. The
    sheet offers 0..65535 for these columns, so a species ID above 32767 was
    accepted, written, and came back negative - a range the read path could
    not reproduce. They are unsigned in the ROM and are now read as such.
    This also means evolutions load through their entry constructor rather
    than by direct field assignment, so validation there would have to wait
    for the read to be correct first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Generation 4 defines 18 types, numbered 0 to 17, and the frontend's typeColors
holds exactly 18 entries - so 18 is one past the end, not the last valid value.
The guard admitted a type that nothing could draw.

Named as a constant so the frontend can derive its cell range from it rather
than restating the number, which is how the two drifted in the first place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Two guards rejected values the tool exists to edit.

An evolution file's entry cap was fixed at the retail count of seven, while
setData deliberately reads every entry the file contains and fileSize is kept
at the input's length - both so an expanded table is not silently truncated.
The result contradicted itself: a ROM with ten evolutions parsed cleanly and
then threw on save, aborting the write of the entire NARC and losing every
other file in it along with the edit. The cap now comes from the file's own
size, so a file can always write back what it was read with, and a retail file
still refuses an eighth entry.

The Pokemon type setters threw above a hardcoded count. A type occupies a whole
byte; the number of types a game defines is game data, not a storage limit, and
ROM hacks that add types are exactly what this library is for. The byte width
is still enforced at save, which is the real constraint. The sheet keeps its
own bound - it can only name and colour the types it has entries for - but that
belongs in the user interface, where the consequence is a value you cannot
conveniently edit, rather than in the data layer, where it is a file you cannot
open.

This reverses the direction of the previous commit, which tightened that bound
from 19 to 18. Correcting the off-by-one was right; enforcing it in Core at all
was not.

Also removes Game.getRegion() and the static field behind it. Replacing the
per-constant region with a static one made the sharing worse, not better: every
Game constant then answered with the region of whichever ROM was parsed last.
The region travels in BaseRomInfo, which is what parseBaseRom returns, and the
deprecated single-argument Tables.initialize that depended on the static had no
callers left.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
The comment on PersonalParser's fields said the state could not be static because
the parser is a singleton and ROM A's TM table would otherwise be written into
ROM B. The reasoning is backwards: being a singleton is exactly why instance
fields buy no isolation - the parser is bound with Scopes.SINGLETON, so there is
one instance for the lifetime of the process and its fields are as process-global
as the static ones were. What actually keeps one ROM's data out of another is the
frontend discarding its caches when the ROM changes. Same claim, same correction,
in PokemonSpriteParser.

The change itself is worth keeping - per-parser state belongs on the parser - so
only the justification is rewritten.

Also removes the javadoc left behind when getRegion() was deleted; it had been
sitting above BaseRomInfo's own javadoc, documenting nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Removed jackson-dataformat-xml and junit:junit, neither of which is imported
anywhere here - this module is entirely on JUnit 5.

Removing the xml artifact broke the build, which is the useful part: nothing
used it directly, but it was the only route to jackson-databind, which the text
and script parsers do use. Declared explicitly now, along with jackson-core,
because a module should declare what it imports rather than inherit it by
accident from something unrelated.

Guice moves to test scope. Nothing in src/main imports it - the two mentions
are comments about how the consumer binds these parsers - and the injector that
uses it lives in this module's tests. At compile scope it sat on the compile
classpath of every module downstream for no reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Verified against retail Platinum, HeartGold and SoulSilver.

- FieldScriptData: stop treating endstd (0x15) as a run terminator. It is
  a zero-parameter fall-through, and terminating on it dropped the commands
  that followed. Per-file round-trip on HG/SS: one script recovers, zero
  regressions.
- CommandMacro/CommandWriter: don't reject a null parameter up front for
  variable-length (.if-guarded) command macros, where a conditional branch
  may legitimately skip it. Defer the "not provided" error to the point the
  writer actually emits the parameter. Fixes save() throwing on
  UnionGroup/MysteryGiftGive/Strength in retail HG/SS.
- PokemonSpriteData/PokemonSpriteParser: read each palette's own bit depth
  (0) instead of forcing 4, which mislabelled 8bpp palettes (the 0x4 marker
  was re-emitted as 0x3). BATTLE_SPRITES now round-trips 2964/2964 byte-exact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PARTY_ICONS narc is [leading files] + [one icon per species] + [trailing
files]. The parser only read the leading and per-species blocks, so saving a
ROM dropped the trailing icons (alternate forms, egg/substitute, etc.) - the
narc's subfile count fell from 547 to 501 on Platinum (551->501 on HG/SS).

Capture the trailing block in generateDataList and re-append it in
processDataList, mirroring the existing leading-files handling. Combined with
the Nds4j NCGR character-header fidelity fix, the whole PARTY_ICONS narc now
round-trips byte-for-byte (547/547 Platinum, 551/551 HeartGold/SoulSilver),
and PokemonSpriteTests passes byte-exact on all three retail ROMs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@turtleisaac
turtleisaac merged commit 7e0124c into main Aug 27, 2026
2 checks passed
@turtleisaac
turtleisaac deleted the claude/pokeditor-bug-deep-dive-9mm03b branch August 27, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants