Fix DECFLOAT values with 16-31 decimal places (SignSpecialPlaces mask $2f -> $3f)#5
Open
mariuz wants to merge 1 commit into
Open
Fix DECFLOAT values with 16-31 decimal places (SignSpecialPlaces mask $2f -> $3f)#5mariuz wants to merge 1 commit into
mariuz wants to merge 1 commit into
Conversation
TBCD.SignSpecialPlaces keeps the number of decimal places in bits 0..5 (0..63) and the sign in bit 7, so the correct mask for the places field is $3f. SQLDecFloatEncode and SQLDecFloatDecode both masked with $2f, which drops bit 4: any DECFLOAT value with 16..31 decimal places lost exactly 16 places in both directions, i.e. was read back (and sent) multiplied by 10^16. Values with 0..15 or 32..34 places were unaffected, which is why the bug could hide for so long. Reproduced against Firebird 6.0 with round trips at 8, 15, 16, 17, 22, 31 and 33 decimal places: before the fix the 16, 17, 22 and 31 cases came back as e.g. 1.2345678901234567 -> 12345678901234567 on both the getAsBCD path and the SQLParams AsBCD path; after the fix all round trips are exact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3.
TBCD.SignSpecialPlaceskeeps the decimal-places count in bits 0..5 and the sign in bit 7, so the places mask must be$3f. Both DECFLOAT conversion paths inTFB30ClientAPImasked with$2f, dropping bit 4 — any value with 16..31 decimal places lost exactly 16 places (i.e. arrived multiplied by 10^16) on both theGetAsBCDread path (SQLDecFloatDecode) and theAsBCDparameter path (SQLDecFloatEncode). Values with 0..15 or 32..34 places were unaffected, which is how the bug stayed hidden.The change is the two masks only (the third
$2fin the file is inside a commented-out debugwritelnand was left alone).Verification (Firebird 6.0, LI-T6.0.0.2076, Linux aarch64, fpc 3.2.2): round trips at 8, 15, 16, 17, 22, 31 and 33 decimal places, decode via
GetAsBCD+BCDToStrand encode viaSQLParams[].AsBCD+ server-sideCAST(? AS VARCHAR(60)). Before: the 16/17/22/31-place cases all came back shifted (1.2345678901234567→12345678901234567). After: all fourteen round trips are exact, and the previously-correct cases are unchanged. Issue #3 has the full before/after table.🤖 Generated with Claude Code