Repository navigation
Take a printed non-English references heading before weaker list guesses - #213
Merged
Merged
Conversation
The reference locator read its printed-heading override with an English-only
pattern, so a German "Literatur" list lost to whatever an earlier, weaker step
offered. On EXCITE dev that cost 67 references on two papers: the front-role
model took a one-row body section ("Auswertung") for the header on one, and the
layout fallback took the last unclassified section (a two-row talk title) on
the other.
- The printed-heading override now also accepts any heading the multilingual
grammar matches as a whole.
- A front-role header whose section holds no dated row is not taken.
- The header-text fallback runs before the "last unclassified section" layout
fallback.
- The grammar knows the German "Quellen" compounds ("Zitierte und verwendete
Quellen", "Literatur und Quellen", "Quellen und Literatur"); bare "Quellen"
stays out because it also opens table notes.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #213 +/- ##
==========================================
- Coverage 90.56% 90.56% -0.01%
==========================================
Files 309 309
Lines 57337 57343 +6
Branches 10383 10383
==========================================
+ Hits 51929 51933 +4
Misses 3449 3449
- Partials 1959 1961 +2 🚀 New features to boost your workflow:
|
On EXCITE dev 32707 one list sits under two printed headings, "References" (the longer half) and a later "13 Literatur" (the shorter half). The multilingual override took the later German one and lost 12 references the English-only override had read. English headings now go first and the multilingual grammar only decides when none matches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: a paper with a German "Literatur" list could lose the whole list. The locator's printed-heading check only knew English headings, so an earlier, weaker step won. On EXCITE dev this happened twice. On one paper the front-role model took a one-row body section ("Auswertung") as the header; on another the layout fallback took the last unclassified section, a two-row talk title. In both cases the segmenters then ran on body sentences.
After: a printed references heading in any language the heading grammar lists wins over those guesses. An English heading still goes first when both are printed. The German "Quellen" compound headings are recognized too.
How:
RefLocator.collect_reference_rowstries English headings first, then any heading that matches the multilingual grammar as a whole.ref_patternsadds "Zitierte und verwendete Quellen", "Verwendete Literatur", "Literatur und Quellen" and "Quellen und Literatur". Bare "Quellen" is left out because it also opens German table notes.Two front-role tests used "Список литературы", which the printed-heading step now catches first. They now use a heading outside the grammar, so they still test the model path.
Evidence: I ran EXCITE dev (174 papers) on declan with the first commit (cb22d3d) against the 685765d baseline. The setup was identical and the scorer was held constant.
Footnote, mixed and English papers were unchanged. Only five papers changed which section they took as the list. 32707 regressed (21 → 9 clean) because its list sits under both "References" and "13 Literatur" and the German half was picked. The second commit (040a5de) restores English-first order for that case. I have not re-run dev for it: the order only differs where an English heading exists, so I expect 32707 back at the baseline and the other four unchanged.
Known leftover: on 40753 the "Literatur" section also takes in a publisher's series list in the back matter, so 15 of its 16 references now come out clean but about 34 false entries come along with them. Splitting a section at large page gaps would need care for books with one list per chapter, so it's not in this PR.
Tests: the new
tests/extract/test_ref_locator_printed_heading.pyfails on main and passes here. The extract, ocr and structure suites pass.