Fix read/write asymmetries, script serialization and global state; revive the test suite - #8
Merged
Merged
Conversation
The governing defect in this module was that a field's parse path and its serialize path disagreed. Those bugs need no user edit to bite: loading a ROM and saving it is enough to corrupt data. Every fix below was verified with a byte-exact round-trip harness over randomized inputs. Bitfield and padding asymmetries: - Trainer alt-form was read as 6 bits but written masked to 3, so every form above 7 was silently remapped or reverted. - Item stat boosts were read with a 4-bit mask and written with a 2-bit one; the very next field had the inverse error, and its overflow bled into the PP Up flags. - EV yields are 2-bit fields, but the setters accepted 4 and the write path never masked, so each yield corrupted the next. Dex colour had the same shape of bug against the sprite-flip bit. - 44 bytes of DPPt encounter data were skipped on read and re-emitted as zeroes, wiping a rate word and a five-slot water table in ~600 subfiles. The item and personal padding bytes had the same pattern. - Trainer type flags and the AI bitfield were re-encoded from only the bits the tool understood, zeroing everything else. - The upper nibble at item offset 0x14 and bits 12-15 of the personal EV halfword were read into nothing and written as zero. - Level script trigger numbers were narrowed through a signed byte cast, so any script above 127 sign-extended. Text: - 9-bit compression only guarded characters up to 0x3FF, so a newline or a VAR placeholder overran its slot and overwrote following characters. - Unmapped characters were silently written as code 0, turning a pasted apostrophe into a null glyph; they now report which character failed. - Bank tables, VAR sequences and escape decoding are bounds-checked, so a message ending in a stray backslash no longer aborts the whole ROM save. - Character 245 collided with 223 in the reverse map and could not survive a save. Scripts: - Command 225 is goto_if_trainer_defeated, not a comparator, so loading any script using it threw and aborted the entire NARC. Its jump target is now registered and recomputed on save instead of being emitted from the old file's absolute address. - ActionCommand(int,int) never assigned its id, so every movement whose name is absent from the tables re-serialized as movement 0. - visitAction_command was a no-op, so recompiling a script dropped every movement command while keeping the labels that pointed at them. - An unresolvable label serialized as a branch to offset 0. - Trainer AI opcodes were written 16-bit though the macros declare 32-bit. - Hex literals in the user-editable macro files crashed both algebra visitors; negative word parameters were emitted in a form the grammar cannot re-lex. Global state: - PersonalParser's TM/HM tables were static, so opening one ROM and saving another wrote the first ROM's TM list into it. Same class of bug in the sprite parser's party-icon header cache, which silently shifted the icon NARC by seven files when unset. - Game.region was mutable state on an enum constant. parseBaseRom now returns the region alongside the game and Tables.initialize takes it explicitly; the old accessor remains, deprecated. - Tables entries are reset between initialize calls and now throw when read before being set, rather than returning a usable-looking zero. - Diamond and Pearl fell through every initialize switch with no default, so a D/P ROM half-initialized and failed far from the cause. - The sprite parser mutated the shared arm9 buffer without holding its lock. Tests: - Nine of the ten parser test classes were declared static, which Surefire's default inner-class exclude drops and JUnit will not descend into. Only one ran. They are now @nested, taking the suite from 2 discovered tests to 22. - outputMatchesInput asserted a value against itself under a guard that had already established equality, skipped all checks when the output was empty, and otherwise compared the output to itself rather than to the original -- so a parser returning zero bytes for every subfile passed. It now compares against the original and asserts the file count. - The suite hard-required a copyrighted ROM beside the pom and failed the build without one; ROMs are located via -Drom.dir and missing ones skip. - Added the sprite parser binding and test class, which had no coverage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Nds4j's MemBufReader.readByte() now returns an unsigned value in 0..255,
matching Buffer.readByte() and readUInt8(). Five fields here are defined by
the Gen IV formats as signed 8-bit and were relying on the old signed
behaviour, so they began parsing as large positive numbers instead:
MoveData.priority -7 (Trick Room, Roar) read back as 249
ItemData.evYields EV-reducing berries yield negative EVs
ItemData.friendshipChangeAmounts bitter berries lower friendship
PokemonSpriteData.globalFrontYOffset / shadowXOffset signed offsets
The bytes were never at risk: every write path narrows back through a byte,
so 249 and -7 both serialize to 0xF9 and the file round-trips either way.
What broke is the value. PokEditor declares the priority column's range as
{-128, 127}, so the sheet would have displayed 249 and then rejected it as
out of range on the next edit.
Each read now converts explicitly at the point of the read. PokemonSpriteData
.unknownByte is left unsigned: it is an opaque passthrough written back with
a cast, so it has no signed interpretation to preserve.
Checked and deliberately unchanged:
- ItemData hp/ppRecoveryAmount are genuinely unsigned (0xFF is the "restore
all" sentinel) and already mask.
- LearnsetData reads its packed halfword as a signed short, but compares the
terminator in the short domain and masks both extracted fields, so sign
extension cannot reach either value.
- EvolutionData's halfwords never exceed 0x7FFF in practice.
Adds SignedFieldTest, which asserts both halves of the property: parsed
values carry the right sign across the whole 0..255 domain, and the bytes
still round-trip. It needs no ROM. Reverting the priority cast fails two of
its tests while the round-trip test keeps passing, which is the distinction
being drawn here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Nds4j is released as 1.0.0 in the companion branch; 0.1.0 no longer exists, so this declaration has to move with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Cut from 1.0-SNAPSHOT so that PokEditor 3.2.0 -- a release version -- no longer depends on a mutable snapshot, which would make that build irreproducible. The audit in this branch is where the API changed, so this is the right place to draw the line: parseBaseRom returns a BaseRomInfo record rather than mutating region state on the Game enum, Tables.initialize takes the region explicitly, PersonalParser's TM/HM tables moved from statics to instance state, and several setters now throw where they previously corrupted silently. From here, semantic versioning puts further breaking changes behind 2.0.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
The README is the documentation for external consumers -- it explicitly invites people to use this library in their own Gen 4 projects -- and its opening usage example no longer compiled: parseBaseRom returns a BaseRomInfo record now, and Tables.initialize takes the region explicitly. Breaking a documented public API without updating the document that teaches it left anyone following that guide with code that does not build. The example is now a compiled transcription rather than an eyeballed edit. Also corrects the build instructions, which told readers a HeartGold and a Platinum ROM were required in the repo root. That is no longer true: the suite builds and passes without one, skipping the tests that need real game data. Both places now describe -Drom.dir instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Runs the full suite on every push and pull request. Nds4j is not on Maven Central at the version this module depends on, so CI builds it from source. It prefers a branch of the same name when one exists and falls back to main otherwise: a change spanning both repositories is developed on matching branches, and building this module against a stale Nds4j would either report a failure that does not exist or hide a real one. Both paths were exercised against the live repositories before committing. Verified by running the same command locally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Every format narrows on the way out - a (short) cast, writeBytes' i2b, or an
explicit mask - so a value that did not fit was written back as a different
one with nothing reported. A learnset move of 512 was stored as move 0, which
reads back as "learns nothing"; a level of 200 was stored as 72. The only way
to notice was to reopen the file and find the wrong data.
FieldWidth checks a value against the field it is about to occupy and names
the field, the value and the limit when it does not fit. It is applied on the
write path rather than in the setters: some formats load through their setters,
and a file already containing an unusual value must still open. What must not
happen is writing a value back out as something other than what was set.
Applied to learnsets (7-bit level, 9-bit move), moves (every field, including
priority as a signed byte), personal (whose setters guarded only the upper
bound, so setHp(-1) was stored as 255) and evolutions.
Two further defects surfaced while testing this:
- Move 511 at level 127 packs to 0xFFFF, which is the end-of-learnset
marker. Both fields are individually in range, so only the packed value
reveals the problem: saving that entry wrote a terminator, and it and
every entry after it disappeared on reload. It is now refused with an
explanation. An exhaustive sweep of all 65536 combinations confirms this
is the only such hole.
- EvolutionData read its fields with readShort(), which sign-extends. The
sheet offers 0..65535 for these columns, so a species ID above 32767 was
accepted, written, and came back negative - a range the read path could
not reproduce. They are unsigned in the ROM and are now read as such.
This also means evolutions load through their entry constructor rather
than by direct field assignment, so validation there would have to wait
for the read to be correct first.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Generation 4 defines 18 types, numbered 0 to 17, and the frontend's typeColors holds exactly 18 entries - so 18 is one past the end, not the last valid value. The guard admitted a type that nothing could draw. Named as a constant so the frontend can derive its cell range from it rather than restating the number, which is how the two drifted in the first place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Two guards rejected values the tool exists to edit. An evolution file's entry cap was fixed at the retail count of seven, while setData deliberately reads every entry the file contains and fileSize is kept at the input's length - both so an expanded table is not silently truncated. The result contradicted itself: a ROM with ten evolutions parsed cleanly and then threw on save, aborting the write of the entire NARC and losing every other file in it along with the edit. The cap now comes from the file's own size, so a file can always write back what it was read with, and a retail file still refuses an eighth entry. The Pokemon type setters threw above a hardcoded count. A type occupies a whole byte; the number of types a game defines is game data, not a storage limit, and ROM hacks that add types are exactly what this library is for. The byte width is still enforced at save, which is the real constraint. The sheet keeps its own bound - it can only name and colour the types it has entries for - but that belongs in the user interface, where the consequence is a value you cannot conveniently edit, rather than in the data layer, where it is a file you cannot open. This reverses the direction of the previous commit, which tightened that bound from 19 to 18. Correcting the off-by-one was right; enforcing it in Core at all was not. Also removes Game.getRegion() and the static field behind it. Replacing the per-constant region with a static one made the sharing worse, not better: every Game constant then answered with the region of whichever ROM was parsed last. The region travels in BaseRomInfo, which is what parseBaseRom returns, and the deprecated single-argument Tables.initialize that depended on the static had no callers left. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
The comment on PersonalParser's fields said the state could not be static because the parser is a singleton and ROM A's TM table would otherwise be written into ROM B. The reasoning is backwards: being a singleton is exactly why instance fields buy no isolation - the parser is bound with Scopes.SINGLETON, so there is one instance for the lifetime of the process and its fields are as process-global as the static ones were. What actually keeps one ROM's data out of another is the frontend discarding its caches when the ROM changes. Same claim, same correction, in PokemonSpriteParser. The change itself is worth keeping - per-parser state belongs on the parser - so only the justification is rewritten. Also removes the javadoc left behind when getRegion() was deleted; it had been sitting above BaseRomInfo's own javadoc, documenting nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Removed jackson-dataformat-xml and junit:junit, neither of which is imported anywhere here - this module is entirely on JUnit 5. Removing the xml artifact broke the build, which is the useful part: nothing used it directly, but it was the only route to jackson-databind, which the text and script parsers do use. Declared explicitly now, along with jackson-core, because a module should declare what it imports rather than inherit it by accident from something unrelated. Guice moves to test scope. Nothing in src/main imports it - the two mentions are comments about how the consumer binds these parsers - and the injector that uses it lives in this module's tests. At compile scope it sat on the compile classpath of every module downstream for no reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Verified against retail Platinum, HeartGold and SoulSilver. - FieldScriptData: stop treating endstd (0x15) as a run terminator. It is a zero-parameter fall-through, and terminating on it dropped the commands that followed. Per-file round-trip on HG/SS: one script recovers, zero regressions. - CommandMacro/CommandWriter: don't reject a null parameter up front for variable-length (.if-guarded) command macros, where a conditional branch may legitimately skip it. Defer the "not provided" error to the point the writer actually emits the parameter. Fixes save() throwing on UnionGroup/MysteryGiftGive/Strength in retail HG/SS. - PokemonSpriteData/PokemonSpriteParser: read each palette's own bit depth (0) instead of forcing 4, which mislabelled 8bpp palettes (the 0x4 marker was re-emitted as 0x3). BATTLE_SPRITES now round-trips 2964/2964 byte-exact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PARTY_ICONS narc is [leading files] + [one icon per species] + [trailing files]. The parser only read the leading and per-species blocks, so saving a ROM dropped the trailing icons (alternate forms, egg/substitute, etc.) - the narc's subfile count fell from 547 to 501 on Platinum (551->501 on HG/SS). Capture the trailing block in generateDataList and re-append it in processDataList, mirroring the existing leading-files handling. Combined with the Nds4j NCGR character-header fidelity fix, the whole PARTY_ICONS narc now round-trips byte-for-byte (547/547 Platinum, 551/551 HeartGold/SoulSilver), and PokemonSpriteTests passes byte-exact on all three retail ROMs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of a four-repo audit of the PokEditor toolchain. Depends on turtleisaac/Nds4j#2 — merge that first.
The governing defect
A field's parse path and its serialize path disagreed. These need no user edit to bite: loading a ROM and saving it is enough. Every fix below was verified with a byte-exact round-trip harness over randomized inputs.
>> 10)& 0x7)& 0xf& 0x3& 0x3& 0xf— overflow bled into the PP Up flagsskip(0x2C)skip(0x2C)— 44 bytes re-emitted as zeroesConcretely: every Unown letter past H and every Arceus plate form was silently remapped or reverted; Speed and Sp. Def stat stages lost their top two bits; a crit stage of 4 turned an item into a PP Up; and ~600
pl_enc_datasubfiles lost a rate word plus a five-slot water encounter table on every save.Also in this class: trainer type flags and the AI bitfield were re-encoded from only the bits the tool understood, zeroing everything else; the upper nibble at item offset
0x14and bits 12-15 of the personal EV halfword were read into nothing and written as zero; level-script trigger numbers were narrowed through a signed byte cast, so any script above 127 sign-extended.Scripts
goto_if_trainer_defeated, not a comparator. It was listed inisDoIfCommand, so loading any script that uses it threwIllegalStateExceptionand aborted the entire NARC. Its jump target is now registered as a label and recomputed on save instead of being emitted from the old file's absolute address.ActionCommand(int, int)never assigned itsid— the fallback taken whenever a movement name lookup misses.Movements_Hg.jsonomits real HGSS opcodes 96-103 and 106, so those movements re-serialized as movement 0.visitAction_commandonly calledsuper, so recompiling a script dropped every movement command while keeping theaction_N:labels pointing at them..wordand the reader readsreadUInt32().Convenience_Hg.txt:92already contains.if \item < 0x4000. The macro files are user-editable, so one added hex constant was a load-time crash for every script in the ROM.Text
9-bit compression only guarded characters up to
0x3FF, so a newline (0xE000) or aVARplaceholder overran its slot and overwrote following characters. Unmapped characters were silently written as code 0 — paste dialogue with a typographic apostrophe and it became a null glyph in-game, reported only to a console with no window attached. Bank tables, VAR sequences and escape decoding are now bounds-checked, so a message ending in a stray backslash no longer aborts the whole ROM save.Global state
PersonalParser's TM/HM tables werestaticand never reset: open ROM A, then save ROM B, and ROM A's TM list was written into ROM B's ARM9.staticon a Guice singleton; when unset the write silently skipped, shifting the entire party-icon NARC by seven files.Game.regionwas mutable state on a global enum constant.initializeswitch with nodefault, so a D/P ROM half-initialized and failed far from the cause with an unrelated NPE.processDataListin the same class does.The test suite never ran
Nine of the ten parser test classes were declared
public static class— which Surefire's default**/*$*exclude drops and JUnit only descends into for@Nested. Only one ran. Proven by converting them: the suite went from 2 discovered tests to 22.And the one surviving round-trip test could not fail:
A parser returning zero bytes for every subfile passed this green — which is exactly how
TrainerAiData.save(), an empty stub returning a zero-length buffer, survived. It now compares against the original and asserts the file count.The suite also hard-required a copyrighted ROM beside the pom and failed the build without one; ROMs are now located via
-Drom.dirand missing ones skip. Added the sprite parser binding and test class, which had no coverage at all.SignedFieldTestis new and needs no ROM: it pins the fields the format defines ass8across the whole0..255domain, asserting both that the sign survives and that the bytes round-trip.Breaking API changes
Also newly throwing where they previously corrupted silently:
PersonalData.set*EvYield(rejects> 3),setDexColor(rejects> 0x7F),ItemData.setStatBoosts, andTextBankData(newTextEncodingException).Verification
mvn test→ BUILD SUCCESS, 28 tests discovered, 0 failures, 11 skipped (ROM-dependent).🤖 Generated with Claude Code
https://claude.ai/code/session_01RqYGZhtVb8KrPA9ftp5bob
Generated by Claude Code