Skip to content

Make Multitext an actual Mapping, breaking keys(), len(), and duplicate langs - #42

Draft
imnasnainaec wants to merge 4 commits into
mainfrom
multitext-mapping
Draft

Make Multitext an actual Mapping, breaking keys(), len(), and duplicate langs#42
imnasnainaec wants to merge 4 commits into
mainfrom
multitext-mapping

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Multitext hand-wrote get, __contains__, keys, values and items on top
of __getitem__ / __iter__ / __len__ — the set collections.abc.Mapping
derives from those three. Inheriting it drops the five, and makes
isinstance(mt, Mapping) true.

What changes for a caller

before after
keys(), values(), items() lists views
len(mt) every form, lang-less included languages
a language on two forms two keys; values() had both texts one key, the first form; forms keeps both
a lang-less form mt[None] returned its text, None in mt was False not a key at all
del mt["en"], two en forms dropped the first, "en" in mt stayed true drops both
repr(mt) always dict-shaped pairs when the forms are not one per language

Views

Typing the class as a Mapping while returning lists is three suppressed Liskov
violations: a list is not a KeysView.

  • A caller typed against Mapping[str, Text] would be promised set operations it
    has not got. Those work now, and a caller wanting a list can say so.
  • Cost: one assertion in the suite, which mypy pointed at, and one line in the
    guides.

One key per language, and len() counting them

A choice, not something the ABC forces: multidict.MultiDict is a registered
Mapping whose keys() repeats and whose len() counts pairs — what this class
did.

  • Deduping is what gets __getitem__, len(), dict() and the three views to
    agree with each other.
  • Before, len(mt) counted a lang-less form that keys() had always excluded, so
    it could exceed len(mt.keys()); and a repeated language made len(mt) 2 where
    dict(mt) held 1 key.
  • werkzeug's MultiDict makes the same call for the same reason.

A lang-less form is not a key

The internal lookup matched it, so mt[None], mt.get(None) and
None in mt.keys() all reached it while __iter__, the views and len() left it
out. That is a key the mapping answered for but never reported — and a KeysView,
which is a Set, claiming an element it would not yield.

Deletion by language, assignment by form

del mt["en"] takes every en form, so the key is gone afterwards. Assignment
updates only the form it names.

  • Both MultiDicts collapse on assignment instead — safe for them, because a
    value is a str.
  • Here the mapping hands back a Text, but the Form carrying it also holds
    annotations and out-of-schema residue that no key reaches.
  • This is the wrong library for the most-used mutator to be the one operation that
    silently discards residue. forms stays the way to edit a duplicate
    deliberately.

Repr shape

A repeated or lang-less form rendered as a dict literal that cannot exist, whose
keys contradict keys(). The inherited views repr through the mapping, so that
text reached KeysView(...) too.

Multitext({'en': 'dog', 'fr': 'chien'})          # one form per language
Multitext([('en', 'colour'), ('en', 'color')])   # a repeated language
Multitext([(None, 'x')])                         # a lang-less form
  • Dict-shaped is kept for forms that are one per language — all a schema-valid
    document has.
  • The pair-list fallback keeps every form visible, with the shape itself as the
    signal that forms holds more than the mapping reaches.

What does not change

  • Fidelity, at all. The writer emits every form off forms, duplicates and
    lang-less ones included, and consults only __bool__; no writer path touches
    len() or the views. duplicate-form-lang reads forms too.
  • bool(mt) still answers "is there anything to serialize", which residue and a
    lang-less form each defeat on their own, so it stays independent of len().
  • MutableMapping is still not inherited. clear and popitem have no clear
    meaning for a form list that can hold forms no key reaches, so the two mutators
    the class already had are still the only two.

How rare these shapes are

Over the 34,904 multitexts reachable from an entry in the 24 loadable 0.13 corpus
files:

  • duplicate languages: 0 — they appear only in the hand-authored negative
    fixture
  • lang-less forms: 2, both in the LIFT spec's own example documents

So this is about the mapping being coherent for input the reader accepts, not
about a shape most files carry.

Coverage

  • tests/test_text.py, new — the mapping semantics, which need no reader.
    • The views, insertion order, first-form resolution, both mutators, deleting
      through the mapping while iterating it, truthiness, equality, both repr shapes.
  • tests/test_reader.py — the reader's share, on the two negative fixtures that
    already carry these shapes.
    • Replaces a hand-built model that had no business in a reader test.
  • tests/test_property_roundtrip.py — the Multitext strategy drew unique,
    never-None langs, so the round-trip properties never saw either shape.
    • Langs now come from the same pool plus None. Over 600 draws, roughly a third
      of multitexts carry a duplicate and a fifth a lang-less form, with most
      examples still schema-valid.
  • tests/test_writer.py — a multitext holding only residue and no forms.
    • Empty as a mapping, so the writer's decision to emit it cannot come from
      len(). No corpus fixture has that shape, and losing it would drop the
      attribute silently.

Guide and CHANGELOG updated to match.

python scripts/check.py green: 586 passed, 98% coverage. mkdocs build --strict
green.

🤖 Generated with Claude Code


This change is Reviewable

imnasnainaec and others added 2 commits August 26, 2026 15:31
The class hand-wrote get, __contains__, keys, values and items on top of
__getitem__ / __iter__ / __len__ — which is the set collections.abc.Mapping
derives from those three. Inheriting it drops them, and makes
isinstance(mt, Mapping) true for consumers that ask.

keys(), values() and items() therefore return views rather than lists. A view
is what a Mapping promises, and typing the class as one while returning lists
would have been three suppressed Liskov violations; set operations on keys()
work now, and a caller wanting a list can say so. __iter__ and __len__ read
forms directly, since the views are built on them.

__len__ counts languages rather than forms. It counted every form including a
lang=None one, which keys() has always excluded, so len(mt) could exceed
len(mt.keys()) on schema-invalid input; as a Mapping that would leave
len(mt) != len(list(mt)). __bool__ still answers "is there anything to
serialize", which residue and a lang-less form each defeat on their own, so it
stays independent of len().

The two mutators stay as they are. MutableMapping is not inherited: clear and
popitem have no clear meaning for a form list that can hold forms no key
reaches.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A language repeated across forms was yielded once per form, so the inherited
views walked it twice and resolved both to the first form's text: values()
reported that text twice and never the second form's, and len() counted 2
where dict() held 1 key. A repeated language is exactly the schema-invalid
input validation reports as duplicate-form-lang, so it is real FLEx and WeSay
output rather than a hypothetical.

__iter__ now yields each language once — the one __getitem__ answers with —
and __len__ counts those, so len(mt) == len(dict(mt)) whatever the forms hold.
Nothing is hidden: forms still holds every form, which is where
duplicate-form-lang reads from and where a lang-less form was already the
docstring's example of content no mapping can represent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
imnasnainaec added a commit that referenced this pull request Aug 26, 2026
Inheriting collections.abc.Mapping changes three things callers can see:
keys(), values() and items() return views rather than lists, len() counts
languages rather than forms, and a language spelled on two forms becomes one
key. Accepting that is a judgement about how much of a 0.x API is worth
breaking for a correct protocol, which nothing here shares a file with — the
rest of this branch deletes duplicated code without changing what anything
returns.

It is proposed on its own in #42, so it can be taken or refused without
holding up six changes that are only refactors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
imnasnainaec added a commit that referenced this pull request Sep 3, 2026
Inheriting collections.abc.Mapping changes three things callers can see:
keys(), values() and items() return views rather than lists, len() counts
languages rather than forms, and a language spelled on two forms becomes one
key. Accepting that is a judgement about how much of a 0.x API is worth
breaking for a correct protocol, which nothing here shares a file with — the
rest of this branch deletes duplicated code without changing what anything
returns.

It is proposed on its own in #42, so it can be taken or refused without
holding up six changes that are only refactors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
imnasnainaec and others added 2 commits September 4, 2026 14:49
A form with no lang was matched by _find, so mt[None], mt.get(None) and
None in mt all reached it while __iter__, the views and len() left it out —
a key the mapping answered for but never reported, and a KeysView claiming to
hold an element it would not yield. _find now skips lang-less forms, so the
key set the mapping answers for is the set it reports.

__delitem__ removes every form for the language rather than the first, so
del mt["en"] leaves "en" not in mt. Assignment still updates only the form it
names: a Form carries annotations and residue that no key reaches, and a
mutator should not discard content it cannot show the caller.

__iter__ walks a snapshot of forms. The inherited views iterate it live, so
deleting through the mapping while iterating it skipped the language after
each removal rather than raising the way a dict does.

__repr__ stays dict-shaped only while the forms are one per language, and
falls back to a list of pairs otherwise. A repeated or lang-less form rendered
as a dict literal that cannot exist and whose keys contradict keys(), and the
views repr through the mapping, so that text reached KeysView(...) too.

The docstring gains the first-form rule, the deletion rule, and the two
deliberate deviations from Mapping: bool() asking whether there is anything to
serialize, and dataclass equality over exact form lists. tests/test_text.py
collects the mapping semantics, which need no reader; the reader's own share of
them moves onto the two negative fixtures that already carry these shapes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Multitext strategy drew unique, never-None langs, so the round-trip
properties never saw a repeated language or a lang-less form — the two
schema-invalid shapes the reader accepts and the writer has to re-emit. Langs
are now drawn from the same pool plus None, which reaches both shapes while
leaving most examples schema-valid: over 600 draws, roughly a third carry a
duplicate and a fifth a lang-less form.

A multitext holding only residue and no forms is empty as a mapping, so the
writer's decision to emit it cannot come from len(). No corpus fixture has that
shape and losing it would drop the attribute silently, so a generated document
covers the touched-entry round-trip for it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@imnasnainaec imnasnainaec self-assigned this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant