Skip to content

Send CJK-script references to the fine-tuned reference review - #212

Open
thesanogoeffect wants to merge 2 commits into
mainfrom
claude/project-thread-b9b6s7
Open

thesanogoeffect wants to merge 2 commits into
mainfrom
claude/project-thread-b9b6s7

Conversation

@thesanogoeffect

@thesanogoeffect thesanogoeffect commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

Before: with the v5.1 parser, Chinese references often skip the fine-tuned review. On unspaced full-width references (GB/T 7714 style), v5.1 puts the authors, title and journal all into the title and leaves the authors empty. That parse passes every source check, because the checks only test that each field occurs in the reference string, not that it is assigned right. v4.5 failed worse on these references and usually lost the year too, so missing_printed_year sent them to review and the review rescued them. On the sealed blind set, zh reference matching F1 dropped from 0.94 to 0.71 (11 papers), while review calls fell from 1,154 to 743.

After: any reference with Han, kana or Hangul gets a cjk_script flag and goes to review when REF_LLM_BASE_URL is set. For those references the review's parse is kept whenever no source check gets worse. Coverage can't tell CJK fields apart, so requiring a strict gain would reject a correct split.

How: cjk_script joins long_reference as an input-level signal that improves() ignores when it compares flag sets. Latin-script references keep the old rule (strictly fewer flags or higher coverage). Supported title, author and year evidence still can't disappear, and the per-list review cap is unchanged.

Checked on non-sealed references with the ONNX parser and the default routing settings. No LLM calls were made, so this measures routing only:

share sent to review v4.5 v5.1 (main) v5.1 + this PR
24 real mixed-script train rows 33% 21% 100%
48 synthetic GB/T 7714 refs 46% 8% 100%
full-width refs, NFKC 96% 12% 100%
full-width refs, OCR-artifact fix 96% 25% 100%
200 non-CJK controls 27% 15.5% 15.5%

On synthetic full-width references, v5.1 gets the title right on 0 of 24 (v4.5 on 4). On spaced Chinese references, real mixed-script rows and the non-CJK controls, v5.1 is better than v4.5. There are no real pure-Chinese references outside the sealed set.

Possible follow-ups, not in this PR: a general missing_authors flag, or normalising CJK punctuation and spacing before parsing (24 of 24 titles right on spaced input), plus real GB/T 7714 training rows.

The review gate's coverage check reads an unspaced CJK run as one word,
so a v5.1 parse that lumps authors, title and journal into one span
passes every source check and skips the review. v4.5's weaker parses
were flagged and rescued by it; v5.1's are not.

Flag references containing Han, kana or Hangul as cjk_script, treat it
as an input signal like long_reference, and keep the review's parse for
such references when no source check gets worse.
@codecov

codecov Bot commented Oct 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.57%. Comparing base (eeaf588) to head (d4413cf).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #212   +/-   ##
=======================================
  Coverage   90.56%   90.57%           
=======================================
  Files         309      309           
  Lines       57337    57346    +9     
  Branches    10383    10386    +3     
=======================================
+ Hits        51929    51940   +11     
+ Misses       3449     3448    -1     
+ Partials     1959     1958    -1     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@thesanogoeffect
thesanogoeffect marked this pull request as ready for review October 18, 2026 16:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant