Repository navigation
fix: count a label's masked entities under any label in the NER tables - #101
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #84.
In the NER measurement the per-label "Masked" column counted only predictions carrying the same label as the gold entity, so it always equalled that label's overlapping recall; the published tables showed it (PER 0.88/0.88 on the invented sample, 0.56/0.56 on WNUT-17). The column was meant to count entities hidden under a span of any label, as the all-labels row does. The all-labels matching now runs once per sentence with one-to-one matching kept, and its hidden entities are shared out by label, so the label rows add up to the all-labels row and a single span over two names masks one of them in every row. The printed definition and both documentation sections say precisely what the column counts.
Both tables were re-measured on this machine: only the Masked column moves on WNUT-17 (PER 0.56 to 0.63, ORG 0.40 to 0.45, LOC 0.59 to 0.69 at the default threshold). On the invented sample this machine's int8 inference produced two spans fewer than the GitHub runner's earlier run, which the section already describes as processor noise; the differences are listed and the redact tool's description still holds.
Files:
scripts/ner_measure.py,tests/python/test_ner_measure.py(the two wrong-label cases now assert the masked count in the label's row, and a new case covers one span over two names),docs/testing.md.Checks on this head, merged with main after the static-analysis and owner-cap merges: the full deterministic sequence passed, py-analysis (ruff and mypy --strict) clean, python-tests 407 (395 before, 11 from main, 1 from this branch). Review: independent review by Codex (gpt-6.1-sol): no functional defect; its one house-rule finding, a two-line comment, is fixed. The model smoke workflow runs on this PR because the measurement script is on its path list.