[cristiangirlea]

Fixes #84.

Draft until one more commit: the invented-sample NER table in docs/testing.md still shows the old per-label Masked cells. That table was published from the model workflow on GitHub's runner (AMD EPYC 7763), and int8 inference on another CPU moves its other columns too: the unchanged script reproduces the same shifts on a Ryzen 9 9950X. So it will be taken from this pull request's own model job summary, which runs model-ner because scripts/ner_measure.py is on the workflow's paths:, and added as a follow-up commit once its non-Masked columns are checked against the published ones.

What

In scripts/ner_measure.py the per-label Masked column counted only predictions carrying the entity's own label, so in a label's row it always equalled overlapping recall (PER 0.88 and 0.88 in the invented sample). Masked now means what the all-labels row already measured: a gold entity is masked when a predicted span of any label shares a character with it, one gold span per predicted span. The match runs once per sentence and each masked gold span is attributed to its own label, so the label rows' Masked counts add up exactly to the all-labels row. Exact and overlapping scoring are unchanged.

Measured again

WNUT-17, on the same CPU as the published table (Ryzen 9 9950X), measured again after the fix; every non-Masked column and the all-labels rows are identical:

Threshold	PER	ORG	LOC
0.85	0.56 → 0.63	0.40 → 0.45	0.59 → 0.69
0.5	0.62 → 0.85	0.52 → 0.59	0.70 → 0.85

The paragraph after the table now quotes the per-label figures. Its 0.59, 0.78 and "against 0.80" figures didn't move.

Files

scripts/ner_measure.py, tests/python/test_ner_measure.py, docs/testing.md, CHANGELOG.md (Unreleased, Fixed).

Tests

• A PER entity covered only by an ORG span is masked in the PER row and not overlapping.
• One ORG span over two PER names masks one of them, not both.
• One ORG span over a PER and an ORG name masks one gold span in total across the label rows.
• At every threshold, the label rows' Masked counts sum to the all-labels row, in the sample and in the WNUT reader's test.
• Two existing assertions that encoded the bug now expect the corrected counts.
• Red first: 4 failures before the fix, then 2 != 1 for the cross-label case before the second fix.

Checks

The full command block of AGENTS.md, run as one command on this head (89459e0, the two fix commits merged with main at 11ffa30) in a worktree, PHP 8.5:

• python-tests: Ran 409 tests, OK
• py-analysis: All checks passed! and Success: no issues found in 11 source files
• php-tests: 148; php-integration: 192; integration: PASS
• mcp-tests: 125, OK
• composer-install, php-analysis and php-psr18-tests (178): ok
• laravel-tests: 94; symfony-tests: 96; tls-check: 7 + 3

The WNUT-17 measurement itself: quality-fetcher (SHA-256 verified), then model-worker, then ner_measure.py with NER_DATASET=wnut17.

Review

Independent read-only review by codex (another model family) of the first fix commit. It found one blocking defect: each label row ran its own match, so one prediction could mask a gold span in two rows. That is fixed in the second commit, with the cross-label test above. It also confirmed that exact and overlapping are untouched and that the prose matches the table. Security: nothing open.

[cristiangirlea]

Closing as a duplicate of #101, which merged the same fix for #84 first, with the same single matching per sentence (each masked gold span credited to its own label) that this branch reached after review.