diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b2cbf9..6529417 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,10 @@ All notable changes to this project are documented in this file. The format foll - Locks moved past published advisories: `certifi` 2026.10.19 and `filelock` 4.2.1 in `requirements-model.lock`, `opentelemetry-api` 1.45.1 and `pyjwt` 2.15.0 in `requirements-mcp.lock`. +### Fixed + +- `scripts/ner_measure.py`'s Masked column counts, in a label's row too, an entity covered by a predicted span of any label (one entity per predicted span), as the all-labels and register rows already did; before, a label's row counted only that label's predictions, so it repeated overlapping recall. The WNUT-17 table in `docs/testing.md` is re-measured; its other columns are unchanged. + ## [0.3.0] - 2026-10-25 ### Added diff --git a/docs/testing.md b/docs/testing.md index e5d2143..2e851d7 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -70,7 +70,7 @@ PHP branch support was verified locally on 2026-09-17 on PHP 8.2.33, 8.3.33, 8.4 | Layer | Scope and counting | | --- | --- | -| Python worker | 406 tests: lifecycle, validation and HTTP, the per-token capacity cap (one token refused at its cap with its own `Retry-After` while another submits, a repeated key answered there, the total still enforced with the `Retry-After` of whichever limit applies, jobs restored from the table counted until they expire, the default unchanged, and every out-of-range value refused at start), the optional JSON log (every line parses and holds only its event's keys, an input, token, header, query string, idempotency key and a failing task's exception text never reach it, `off` writes nothing and a bad `BRIDGE_LOG` stops the start, the route templates, the 401 without an owner, consistent job timings, and a stderr that fails or blocks costing a request nothing), the drain (submissions refused with 503, queued jobs failed at once, running ones finished within `BRIDGE_DRAIN_SECONDS`, a second signal), `/readyz` and its reasons for any configured token, the 503 for a full connection budget and the 431 for an oversized request line or header block, per-client tokens (each source and every way it is refused at start, a job and its idempotency key scoped to the submitting token, through the job table and a table from before owners, the `owner` metrics series, and no token in an error, the metrics, the table, a child's environment or a start message), the job table in SQLite (a store reopened on its file fails unfinished jobs as `service_restarted`, serves retained results, deletes expired rows from the file and never holds an input; a submission the table refuses is a 429, a later write it refuses leaves the job served, and a key whose row outlived a failed delete runs again) and idempotency keys, process crashes, cancellation, timeouts, retention, the Retry-After a full or closing store gives and its header on every 429, allocation/exit races, the warm pool (that successive jobs share a child in which no live container or frame holds a reference to a finished job's input or result, after a success, a failure and real task payloads, with the detector checked against a planted needle; that a cancelled, timed-out or crashed child is replaced; that a child is recycled after its share of jobs; that reads, submissions and cancellations answer at once while a child is being started, that a child whose start ends after the store closed does not outlive it, and that a job assigned to or waiting for a live child does not fail when no replacement can start; that a job whose child died before taking it is tried once more on another, only once, that a job sent but not yet taken can still be cancelled or time out, and that a job whose child crashed after taking it is not; that a child, idle or running a job, ignores the SIGINT a terminal's Ctrl+C sends the whole process group; that the service token is absent from a child's environment and from its environment block when the service started it; that a message for an earlier job lands nowhere; that a child that never reports ready is replaced and that children dying at start are not restarted in a loop; and that a cancellation does not hold up reads), redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading, the uncased model's reading of the whole text and the merge driven by stubs, the fetcher's pins and the model directories an ONNX worker checks for before it starts, batched cross-encoder scoring, the NER measurement's sample, scoring and WNUT-17 reader, the load measurement's settings, percentiles, error counting and table, the ranking measurement's metrics, BM25, dataset reader and fit to the worker's limits, the public datasets' pins and licence gate, the workflows' runners, action pins and checkout credentials, both Claude Code hooks (the post-edit rules, and the pre-call push guard through its own cases), progress bounds, the service's reading of its BRIDGE_* settings (defaults, every variable, malformed and out-of-range values, a state directory that cannot hold the table), and the model-name table every result reports from | +| Python worker | 409 tests: lifecycle, validation and HTTP, the per-token capacity cap (one token refused at its cap with its own `Retry-After` while another submits, a repeated key answered there, the total still enforced with the `Retry-After` of whichever limit applies, jobs restored from the table counted until they expire, the default unchanged, and every out-of-range value refused at start), the optional JSON log (every line parses and holds only its event's keys, an input, token, header, query string, idempotency key and a failing task's exception text never reach it, `off` writes nothing and a bad `BRIDGE_LOG` stops the start, the route templates, the 401 without an owner, consistent job timings, and a stderr that fails or blocks costing a request nothing), the drain (submissions refused with 503, queued jobs failed at once, running ones finished within `BRIDGE_DRAIN_SECONDS`, a second signal), `/readyz` and its reasons for any configured token, the 503 for a full connection budget and the 431 for an oversized request line or header block, per-client tokens (each source and every way it is refused at start, a job and its idempotency key scoped to the submitting token, through the job table and a table from before owners, the `owner` metrics series, and no token in an error, the metrics, the table, a child's environment or a start message), the job table in SQLite (a store reopened on its file fails unfinished jobs as `service_restarted`, serves retained results, deletes expired rows from the file and never holds an input; a submission the table refuses is a 429, a later write it refuses leaves the job served, and a key whose row outlived a failed delete runs again) and idempotency keys, process crashes, cancellation, timeouts, retention, the Retry-After a full or closing store gives and its header on every 429, allocation/exit races, the warm pool (that successive jobs share a child in which no live container or frame holds a reference to a finished job's input or result, after a success, a failure and real task payloads, with the detector checked against a planted needle; that a cancelled, timed-out or crashed child is replaced; that a child is recycled after its share of jobs; that reads, submissions and cancellations answer at once while a child is being started, that a child whose start ends after the store closed does not outlive it, and that a job assigned to or waiting for a live child does not fail when no replacement can start; that a job whose child died before taking it is tried once more on another, only once, that a job sent but not yet taken can still be cancelled or time out, and that a job whose child crashed after taking it is not; that a child, idle or running a job, ignores the SIGINT a terminal's Ctrl+C sends the whole process group; that the service token is absent from a child's environment and from its environment block when the service started it; that a message for an earlier job lands nowhere; that a child that never reports ready is replaced and that children dying at start are not restarted in a loop; and that a cancellation does not hold up reads), redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading, the uncased model's reading of the whole text and the merge driven by stubs, the fetcher's pins and the model directories an ONNX worker checks for before it starts, batched cross-encoder scoring, the NER measurement's sample, scoring and WNUT-17 reader, the load measurement's settings, percentiles, error counting and table, the ranking measurement's metrics, BM25, dataset reader and fit to the worker's limits, the public datasets' pins and licence gate, the workflows' runners, action pins and checkout credentials, both Claude Code hooks (the post-edit rules, and the pre-call push guard through its own cases), progress bounds, the service's reading of its BRIDGE_* settings (defaults, every variable, malformed and out-of-range values, a state directory that cannot hold the table), and the model-name table every result reports from | | MCP server | 125 tests, offline with hash-pinned wheels: the protocol client against an in-process worker, every tool through the SDK's in-memory client with test tasks enabled, the entry point as a subprocess, result validators on synthetic data, the index builder (chunking, the format and each corruption it refuses, building against the worker, caching, its command line) `bridge_search` (the model gate, path confinement, reloads, rerank on and off), the reachability of every answer in the LLM evaluation, that a default search sees only part of its corpus and that its two-step question takes two steps, that the demo worker reaches none of the semantic evaluation's answers, that a tool waits out a full worker for its Retry-After within the call's budget or with its own backoff without one, before each of a search's jobs and within one budget, and says so when it stays full, the evaluation runner's command, isolation checks, scoring and per-backend launcher and index check without starting `claude`, a check that the server's model names match the worker's, that every path parameter says where it resolves and every position in a result says what it counts, and a check that every timing claim in a tool description is a measured median | | PHP contracts + HTTP | 192 checks per PHP version, a full fault-worker's Retry-After read through the client among them and a two-token `tokens-worker` where another token's read and cancel are 404 and the same idempotency key is two jobs (on which `tests/integration.py` then fills one token's `BRIDGE_OWNER_CAPACITY` of 8 while the other still submits up to the worker's capacity of 12); this **includes** the 148 contract checks (Retry-After from a loopback peer, present, absent, malformed and repeated, and a header block over 16 KiB refused), not 192 + 148; the contract checks include a fake transport's view of each request, the wait deadline across a transport failure, and that no file in `src/` outside `Laravel/` and `Symfony/` names a framework | | PSR-18 transport | The 148 contract checks plus 30 through `Psr18Transport` and in-repo fakes of the PSR-7, PSR-17 and PSR-18 interfaces (`tests/php/psr18_fakes.php`): a round trip with its method, URL, headers and body, a body at the 262144-byte cap and one over it read one byte past the cap and no further, five redirect statuses refused unread, a non-JSON or missing content type, malformed JSON, a 429's Retry-After from one header, a repeated one and none, a client exception and a body read failure as `transport_error` without their text; 178 checks by `php-psr18-tests` with `.cache/php-vendor` on the include path, once, in the MCP server job; `php-tests` skips them with one printed line; no real Symfony HttpClient or Guzzle runs | @@ -293,21 +293,21 @@ docker compose -f docker/compose.yaml --profile model down `NER_DATASET=wnut17 scripts/ner_measure.py` runs the WNUT-17 test split (Derczynski et al., emerging and rare entities in social media and forum posts) through the redact task, one sentence per job, at the same two thresholds as the invented sample. The split is CC BY 4.0 as its repository's README states, pinned to a commit and a SHA-256 in `scripts/public_data.py` and stored by `quality-fetcher`. Its person entities count as PER, corporation and group as ORG, and location as LOC; product and creative-work have no label among the task's three, so they are not gold spans and a span over one counts against precision. Rule spans are not counted. The table adds F1 to the invented sample's columns and replaces its registers with one row over all three labels. -Measured 2026-10-21 UTC, commit `e77310f`, runner: local, Windows 11 with Docker Desktop, CPU: AMD Ryzen 9 9950X 16-Core Processor. -Model `Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8`; WNUT-17 test split, `emerging.test.annotated` at SHA-256 `2aa79b764e56`, CC-BY-4.0: 1287 sentences with 810 entities mapped to labels, person to PER, corporation and group to ORG, location to LOC; product and creative-work have no label here, so a span over one counts against precision. Each sentence is its tokens joined by single spaces; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: gold spans that share a character with a predicted span of any label, one gold span per predicted span; a label's row counts only that label's predictions, so there it equals overlapping recall. +Measured 2026-10-26 UTC, commit `cf61afd`, runner: local, Windows 11 with Docker Desktop, CPU: AMD Ryzen 9 9950X 16-Core Processor. +Model `Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8`; WNUT-17 test split, `emerging.test.annotated` at SHA-256 `2aa79b764e56`, CC-BY-4.0: 1287 sentences with 810 entities mapped to labels, person to PER, corporation and group to ORG, location to LOC; product and creative-work have no label here, so a span over one counts against precision. Each sentence is its tokens joined by single spaces; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: gold spans that share a character with a predicted span of any label, one gold span per predicted span, in a label's row too. | Threshold | Label | Gold | Predicted | Precision, exact | Recall, exact | F1, exact | Precision, overlapping | Recall, overlapping | F1, overlapping | Masked | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.85 | all labels | 810 | 939 | 0.42 | 0.49 | 0.45 | 0.45 | 0.52 | 0.48 | 0.59 | -| 0.85 | PER | 429 | 310 | 0.75 | 0.54 | 0.63 | 0.78 | 0.56 | 0.65 | 0.56 | -| 0.85 | ORG | 231 | 487 | 0.17 | 0.36 | 0.23 | 0.19 | 0.40 | 0.26 | 0.40 | -| 0.85 | LOC | 150 | 142 | 0.57 | 0.54 | 0.55 | 0.62 | 0.59 | 0.60 | 0.59 | +| 0.85 | PER | 429 | 310 | 0.75 | 0.54 | 0.63 | 0.78 | 0.56 | 0.65 | 0.63 | +| 0.85 | ORG | 231 | 487 | 0.17 | 0.36 | 0.23 | 0.19 | 0.40 | 0.26 | 0.45 | +| 0.85 | LOC | 150 | 142 | 0.57 | 0.54 | 0.55 | 0.62 | 0.59 | 0.60 | 0.69 | | 0.5 | all labels | 810 | 1892 | 0.24 | 0.55 | 0.33 | 0.26 | 0.60 | 0.36 | 0.78 | -| 0.5 | PER | 429 | 395 | 0.63 | 0.58 | 0.61 | 0.67 | 0.62 | 0.65 | 0.62 | -| 0.5 | ORG | 231 | 1278 | 0.08 | 0.45 | 0.14 | 0.09 | 0.52 | 0.16 | 0.52 | -| 0.5 | LOC | 150 | 219 | 0.43 | 0.63 | 0.51 | 0.48 | 0.70 | 0.57 | 0.70 | +| 0.5 | PER | 429 | 395 | 0.63 | 0.58 | 0.61 | 0.67 | 0.62 | 0.65 | 0.85 | +| 0.5 | ORG | 231 | 1278 | 0.08 | 0.45 | 0.14 | 0.09 | 0.52 | 0.16 | 0.59 | +| 0.5 | LOC | 150 | 219 | 0.43 | 0.63 | 0.51 | 0.48 | 0.70 | 0.57 | 0.85 | -On real user-generated text the models find about half of the entities at the default threshold: exact F1 0.45 over all three labels, 0.63 for persons (precision 0.75, recall 0.54), 0.55 for places and 0.23 for organisations, where 487 predicted spans meet 231 gold ones. At 0.85, 0.59 of the entities share a character with a predicted span of some label (the Masked column, one entity per predicted span; an overlap, not a count of characters hidden), against 0.80 in the invented sample's informal messages; at 0.5 the share is 0.78, at the cost of 1278 ORG spans, nine in ten of them wrong. Part of the ORG excess may be WNUT-17's 269 products and creative works, which this mapping leaves without a label; how much was not counted. WNUT-17 was built from entities that are rare or new, which makes it harder than text naming well-known people and companies. +On real user-generated text the models find about half of the entities at the default threshold: exact F1 0.45 over all three labels, 0.63 for persons (precision 0.75, recall 0.54), 0.55 for places and 0.23 for organisations, where 487 predicted spans meet 231 gold ones. At 0.85, 0.59 of the entities share a character with a predicted span of some label (the Masked column, one entity per predicted span; an overlap, not a count of characters hidden): 0.63 of persons, 0.45 of organisations and 0.69 of places, against 0.80 in the invented sample's informal messages; at 0.5 the share is 0.78, at the cost of 1278 ORG spans, nine in ten of them wrong. Part of the ORG excess may be WNUT-17's 269 products and creative works, which this mapping leaves without a label; how much was not counted. WNUT-17 was built from entities that are rare or new, which makes it harder than text naming well-known people and companies. These figures carry what the models do with this public, English, informal text, at character-level matching on tokens joined by single spaces, so `>` reads as `& gt ;` and punctuation stands apart from words, as in the file. They are not WNUT-17's own entity F1, which counts six types; and they are one run: int8 inference is not bit-identical across CPUs, as the section above notes. The redact tool's description still quotes the invented sample's figures. To reproduce, run `model-quality` as above, or the NER half alone: diff --git a/scripts/ner_measure.py b/scripts/ner_measure.py index 6e85f02..3eb3454 100644 --- a/scripts/ner_measure.py +++ b/scripts/ner_measure.py @@ -95,23 +95,24 @@ def predictions(spans: list) -> list: return [(span["start"], span["end"], span["label"]) for span in spans if span["source"].startswith("model:")] -def matched(gold: list, predicted: list, same_label: bool) -> int: +def matched(gold: list, predicted: list, same_label: bool) -> list: """Gold spans overlapped by a predicted span, each predicted span matching one gold span at most.""" - unused, count = list(predicted), 0 + unused, found = list(predicted), [] for start, end, label in gold: match = next((p for p in unused if (p[2] == label or not same_label) and p[0] < end and start < p[1]), None) if match is not None: unused.remove(match) - count += 1 - return count + found.append((start, end, label)) + return found -def score(gold: list, predicted: list) -> dict: +def score(gold: list, predicted: list, masked: list | None = None) -> dict: """Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span, so one span over two names finds one of them. Masked: a shared character under any label, which is what a - redaction hides.""" + redaction hides; masked, when given, is that match already made over the whole sentence.""" return {"gold": len(gold), "predicted": len(predicted), "exact": len(set(gold) & set(predicted)), - "overlapping": matched(gold, predicted, True), "masked": matched(gold, predicted, False)} + "overlapping": len(matched(gold, predicted, True)), + "masked": len(matched(gold, predicted, False) if masked is None else masked)} def rate(part: int, whole: int) -> str: @@ -171,16 +172,19 @@ def add(total: dict, counts: dict) -> None: def tally(items: list, spans: dict) -> list: """Rows of (threshold, register or label, counts) from each threshold's spans per item: each register in the - sample's order, then each label, whose rows count only that label's gold and predicted spans.""" + sample's order, then each label, whose rows count only that label's gold and predicted spans, except that the + sentence's masked gold spans, matched once under any label, are each counted in their own label's row.""" registers = list(dict.fromkeys(item["register"] for item in items)) rows = [] for threshold, found_per_item in spans.items(): totals = {name: {} for name in registers + list(LABELS)} for item, found in zip(items, found_per_item): gold, predicted = gold_spans(item), predictions(found) - add(totals[item["register"]], score(gold, predicted)) + hidden = matched(gold, predicted, False) + add(totals[item["register"]], score(gold, predicted, hidden)) for label in LABELS: - add(totals[label], score([g for g in gold if g[2] == label], [p for p in predicted if p[2] == label])) + add(totals[label], score([g for g in gold if g[2] == label], [p for p in predicted if p[2] == label], + [g for g in hidden if g[2] == label])) rows += [(threshold, name, counts) for name, counts in totals.items()] return rows @@ -207,13 +211,13 @@ def main() -> None: if wnut: pinned = public_data.DATASETS["wnut17"] print(f"""Measured {now} UTC, commit `{os.environ.get("BENCH_COMMIT", "unknown")}`, runner: {os.environ.get("BENCH_RUNNER", "unspecified")}, CPU: {cpu_model()}. -Model {", ".join(f"`{model}`" for model in sorted(models))}; WNUT-17 test split, `{pinned["url"].rsplit("/", 1)[1]}` at SHA-256 `{pinned["sha256"][:12]}`, {pinned["licence"]}: {len(items)} sentences with {sum(len(item["gold"]) for item in items)} entities mapped to labels, person to PER, corporation and group to ORG, location to LOC; product and creative-work have no label here, so a span over one counts against precision. Each sentence is its tokens joined by single spaces; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: gold spans that share a character with a predicted span of any label, one gold span per predicted span; a label's row counts only that label's predictions, so there it equals overlapping recall. +Model {", ".join(f"`{model}`" for model in sorted(models))}; WNUT-17 test split, `{pinned["url"].rsplit("/", 1)[1]}` at SHA-256 `{pinned["sha256"][:12]}`, {pinned["licence"]}: {len(items)} sentences with {sum(len(item["gold"]) for item in items)} entities mapped to labels, person to PER, corporation and group to ORG, location to LOC; product and creative-work have no label here, so a span over one counts against precision. Each sentence is its tokens joined by single spaces; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: gold spans that share a character with a predicted span of any label, one gold span per predicted span, in a label's row too. """) print(f1_table(tally(items, spans))) return entities = sum(len(item["entities"]) for item in items) print(f"""Measured {now} UTC, commit `{os.environ.get("BENCH_COMMIT", "unknown")}`, runner: {os.environ.get("BENCH_RUNNER", "unspecified")}, CPU: {cpu_model()}. -Model {", ".join(f"`{model}`" for model in sorted(models))}; `{sample.relative_to(REPO).as_posix()}`, {len(items)} invented sentences with {entities} labelled entities; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: a shared character under any label, the share of entities a redaction hides. +Model {", ".join(f"`{model}`" for model in sorted(models))}; `{sample.relative_to(REPO).as_posix()}`, {len(items)} invented sentences with {entities} labelled entities; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: a shared character with a predicted span of any label, in a label's row too, one gold span per predicted span: the share of entities a redaction hides. | Threshold | Register or label | Gold | Predicted | Precision, exact | Recall, exact | Precision, overlapping | Recall, overlapping | Masked | | --- | --- | --- | --- | --- | --- | --- | --- | --- |""") diff --git a/tests/python/test_ner_measure.py b/tests/python/test_ner_measure.py index 886e30f..f670f01 100644 --- a/tests/python/test_ner_measure.py +++ b/tests/python/test_ner_measure.py @@ -101,6 +101,30 @@ def test_a_wrong_label_is_neither_exact_nor_overlapping_but_still_masks(self): self.assertEqual(measure.score([(0, 5, "PER")], [(0, 5, "ORG")]), {"gold": 1, "predicted": 1, "exact": 0, "overlapping": 0, "masked": 1}) + def test_a_label_s_row_counts_an_entity_masked_under_another_label(self): + items = [{"register": "informal", "text": "ada in rome", "entities": [["PER", "ada"], ["LOC", "rome"]]}] + found = [{"start": 0, "end": 3, "label": "ORG", "source": "model:ORG", "score": 0.9}] + counts = {name: value for _, name, value in measure.tally(items, {0.85: [found]})} + self.assertEqual(counts["PER"], {"gold": 1, "predicted": 0, "exact": 0, "overlapping": 0, "masked": 1}) + self.assertEqual(counts["ORG"], {"gold": 0, "predicted": 1, "exact": 0, "overlapping": 0, "masked": 0}) + self.assertEqual(counts["LOC"], {"gold": 1, "predicted": 0, "exact": 0, "overlapping": 0, "masked": 0}) + + def test_a_label_s_row_still_matches_one_gold_span_per_predicted_span(self): + # One ORG span over "ines and pedro" hides both, but is counted for one name, as in the register's row. + items = [{"register": "informal", "text": "ines and pedro", "entities": [["PER", "ines"], ["PER", "pedro"]]}] + found = [{"start": 0, "end": 14, "label": "ORG", "source": "model:ORG", "score": 0.9}] + counts = {name: value for _, name, value in measure.tally(items, {0.85: [found]})} + self.assertEqual(counts["PER"], {"gold": 2, "predicted": 0, "exact": 0, "overlapping": 0, "masked": 1}) + self.assertEqual(counts["informal"]["masked"], 1) + + def test_one_predicted_span_masks_one_gold_span_across_the_label_rows(self): + # One ORG span over "Ada Acme" hides both names but is counted for one, in the label rows as in the register's. + items = [{"register": "news", "text": "Ada Acme", "entities": [["PER", "Ada"], ["ORG", "Acme"]]}] + found = [{"start": 0, "end": 8, "label": "ORG", "source": "model:ORG", "score": 0.9}] + counts = {name: value for _, name, value in measure.tally(items, {0.85: [found]})} + self.assertEqual(counts["news"]["masked"], 1) + self.assertEqual(sum(counts[label]["masked"] for label in measure.LABELS), 1) + def test_only_the_model_s_spans_are_predictions(self): spans = [{"start": 0, "end": 3, "label": "PER", "source": "model:PER", "score": 0.9}, {"start": 4, "end": 7, "label": "LOC", "source": "model:LOC", "score": 0.6}, @@ -123,12 +147,15 @@ def test_rows_add_up_by_register_and_by_label_for_each_threshold(self): self.assertEqual(counts[(0.85, "informal")], {"gold": 2, "predicted": 0, "exact": 0, "overlapping": 0, "masked": 0}) self.assertEqual(counts[(0.5, "informal")], {"gold": 2, "predicted": 1, "exact": 1, "overlapping": 1, "masked": 1}) - # Acme predicted as a place is an ORG missed and a LOC that is wrong, in each label's own row; in the - # register's row it is still a name masked. + # Acme predicted as a place is an ORG missed and a LOC that is wrong, in each label's own row; it is still a + # name masked, in the register's row and in ORG's. self.assertEqual(counts[(0.85, "news")], {"gold": 2, "predicted": 2, "exact": 1, "overlapping": 1, "masked": 2}) - self.assertEqual(counts[(0.85, "ORG")], {"gold": 1, "predicted": 0, "exact": 0, "overlapping": 0, "masked": 0}) + self.assertEqual(counts[(0.85, "ORG")], {"gold": 1, "predicted": 0, "exact": 0, "overlapping": 0, "masked": 1}) self.assertEqual(counts[(0.85, "LOC")], {"gold": 1, "predicted": 1, "exact": 0, "overlapping": 0, "masked": 0}) self.assertEqual(counts[(0.85, "PER")], {"gold": 2, "predicted": 1, "exact": 1, "overlapping": 1, "masked": 1}) + for threshold in spans: + self.assertEqual(sum(counts[(threshold, label)]["masked"] for label in measure.LABELS), + sum(counts[(threshold, register)]["masked"] for register in ("informal", "news"))) def test_a_full_worker_is_waited_out(self): # Two thresholds over the sample and a development-set run within five minutes exceed the worker's 128 slots. @@ -196,10 +223,11 @@ def test_parsed_sentences_are_scored_like_the_sample(self): {"start": 12, "end": 16, "label": "ORG", "source": "model:ORG", "score": 0.9}] counts = {name: value for _, name, value in measure.tally(items, {0.85: [found]})} self.assertEqual(counts["all labels"], {"gold": 2, "predicted": 2, "exact": 1, "overlapping": 1, "masked": 2}) - # A label's row sees only that label's predictions, so the ORG span over Oslo masks nothing in LOC's row. - self.assertEqual(counts["LOC"], {"gold": 1, "predicted": 0, "exact": 0, "overlapping": 0, "masked": 0}) + # The ORG span over Oslo is no LOC prediction, but it hides Oslo, so LOC's row counts it as masked. + self.assertEqual(counts["LOC"], {"gold": 1, "predicted": 0, "exact": 0, "overlapping": 0, "masked": 1}) self.assertEqual(counts["ORG"], {"gold": 0, "predicted": 1, "exact": 0, "overlapping": 0, "masked": 0}) self.assertEqual(counts["PER"], {"gold": 1, "predicted": 1, "exact": 1, "overlapping": 1, "masked": 1}) + self.assertEqual(sum(counts[label]["masked"] for label in measure.LABELS), counts["all labels"]["masked"]) def test_f1_is_the_harmonic_mean_of_precision_and_recall(self): self.assertEqual(measure.f1(3, 4, 6), "0.60") # precision 0.5, recall 0.75