Skip to content

fix: count a label's masked entities under any label in the NER tables - #101

Merged
cristiangirlea merged 2 commits into
mainfrom
test/masked-column
Oct 26, 2026
Merged

cristiangirlea merged 2 commits into
mainfrom
test/masked-column

Conversation

@cristiangirlea

Copy link
Copy Markdown
Owner

Fixes #84.

In the NER measurement the per-label "Masked" column counted only predictions carrying the same label as the gold entity, so it always equalled that label's overlapping recall; the published tables showed it (PER 0.88/0.88 on the invented sample, 0.56/0.56 on WNUT-17). The column was meant to count entities hidden under a span of any label, as the all-labels row does. The all-labels matching now runs once per sentence with one-to-one matching kept, and its hidden entities are shared out by label, so the label rows add up to the all-labels row and a single span over two names masks one of them in every row. The printed definition and both documentation sections say precisely what the column counts.

Both tables were re-measured on this machine: only the Masked column moves on WNUT-17 (PER 0.56 to 0.63, ORG 0.40 to 0.45, LOC 0.59 to 0.69 at the default threshold). On the invented sample this machine's int8 inference produced two spans fewer than the GitHub runner's earlier run, which the section already describes as processor noise; the differences are listed and the redact tool's description still holds.

Files: scripts/ner_measure.py, tests/python/test_ner_measure.py (the two wrong-label cases now assert the masked count in the label's row, and a new case covers one span over two names), docs/testing.md.

Checks on this head, merged with main after the static-analysis and owner-cap merges: the full deterministic sequence passed, py-analysis (ruff and mypy --strict) clean, python-tests 407 (395 before, 11 from main, 1 from this branch). Review: independent review by Codex (gpt-6.1-sol): no functional defect; its one house-rule finding, a two-line comment, is fixed. The model smoke workflow runs on this PR because the measurement script is on its path list.

@cristiangirlea
cristiangirlea merged commit d3c6253 into main Oct 26, 2026
13 checks passed
@cristiangirlea
cristiangirlea deleted the test/masked-column branch October 26, 2026 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: the per-label Masked column in ner_measure.py equals overlapping recall instead of counting any-label masking

1 participant