Retire five duplicated implementations, three of them the stdlib's job

CHANGELOG.md

@@ -24,9 +24,11 @@ releases may contain breaking changes.
 
 - Project scaffolding: package skeleton, vendored LIFT 0.13 RELAX NG schema,
   test corpus with provenance, corpus-prep and large-file-generator tooling.
-- Full object model: all 35 LIFT 0.13 elements as typed dataclasses;
-  `sil_lift.load()` / `Lexicon.load()` full-document reader that keeps LIFT
-  residue per node in `Extras`; LIFT-version guard.
+- Full object model: all 35 LIFT 0.13 elements as typed dataclasses.
+  `Entry.all_senses()` walks every subsense depth-first in document order,
+  which `Entry.senses` (top level only) does not. `sil_lift.load()` /
+  `Lexicon.load()` full-document reader that keeps LIFT residue per node in
+  `Extras`; LIFT-version guard.
 - `Lexicon.save()` writer with byte-fidelity passthrough — unchanged
   documents and untouched entries are written byte-identically; touched entries
   re-serialize canonically with all out-of-schema content preserved. Fidelity
@@ -51,14 +53,12 @@ releases may contain breaking changes.
   against the most recent `save()`.
 - LIFT-folder handling: `RangesFile` (standalone `.lift-ranges` documents,
   same fidelity guarantees), automatic companion discovery/tracking on load
-  (`Lexicon.ranges_files`, matching companion filenames across case and
-  Unicode normalization differences), `save()` writes companions together,
+  (`Lexicon.ranges_files`), `save()` writes companions together,
   `all_ranges()` merged view, `media_refs()` / `missing_media()` helpers,
   build-from-scratch helpers `Lexicon.add_ranges_file()` /
   `RangesFile.add_range()` / `Range.add_element()` (`save()` writes and
   header-references a new companion beside the `.lift`); vendored
-  `schemas/lift-ranges-0.13.rng` — the first schema for standalone ranges
-  documents.
+  `schemas/lift-ranges-0.13.rng` for standalone ranges documents.
 - Zipped LIFT packages: `sil_lift.load()` reads a `.zip` (both the flat and
   folder-wrapped layouts, junk entries like `__MACOSX` ignored),
   `Lexicon.save_zip()` writes one (carrying media, `WritingSystems/`, and other

docs/en/guides/bulk-edit-glosses.md

@@ -12,18 +12,10 @@ import sil_lift
 path = "dictionary.lift"
 lex = sil_lift.load(path)
 
-
-def iter_senses(senses):
-    """Yield every sense, including subsenses (recursive)."""
-    for sense in senses:
-        yield sense
-        yield from iter_senses(sense.subsenses)
-
-
 edited_glosses = 0
 
 for entry in lex.entries:
-    for sense in iter_senses(entry.senses):
+    for sense in entry.all_senses():
         for gloss in sense.glosses:
             if gloss.lang != "en":
                 continue
@@ -47,7 +39,8 @@ print(f"edited {edited_glosses} gloss(es) across {len(changed)} entry(ies)")
 
 A few things worth noting:
 
-- `Sense.subsenses` is itself a `list[Sense]`, so `iter_senses` recurses into it — a bulk edit that only walked `entry.senses` would silently skip any gloss nested under a subsense.
+- `entry.all_senses()` yields every sense _and subsense_, depth-first in document order.
+    - `entry.senses` holds only the top level, so a bulk edit that walked it would silently skip any gloss nested under a subsense.
 - `gloss.text` is a `Text`, not a plain string: `str(gloss.text)` flattens it for matching, and the replacement is written back with `sil_lift.Text([new])` rather than mutating the string in place.
 - `lex.changed_entries()` reports which entries differ from the file as loaded. Since an entry's digest covers its whole subtree, an edit to a nested subsense reports the entry that contains it.
     - It compares serialized content, so assigning a field the value it already had isn't reported.

docs/en/guides/folder-media.md

@@ -12,8 +12,6 @@ lex.all_ranges()                            # merged {id: Range} view
 lex.all_ranges()["grammatical-info"].elements
 ```
 
-Companion discovery handles the real world: a `range/@href` that points at an existing file is used; FieldWorks' dangling absolute `file://C:/...` hrefs fall back to the href's basename next to the `.lift`; and the conventional `<name>.lift-ranges` sibling is picked up even when nothing references it.
-
 `lex.save()` writes the `.lift` and every tracked companion together. Edits to a `RangesFile` save back to _its_ file; untouched ranges keep their exact bytes. Standalone use:
 
 ```python
@@ -23,6 +21,16 @@ ranges.sort()
 ranges.save()
 ```
 
+### Companion discovery
+
+Several candidates are tried, and every distinct file among them is loaded.
+
+- A header `range/@href` that points at an existing file is used as given.
+- An href that resolves to nothing falls back to its basename next to the `.lift` — FieldWorks writes dangling absolute `file://C:/...` paths from the exporting machine, and that fallback is what makes them work locally.
+- The conventional `<name>.lift-ranges` sibling is picked up even when nothing references it.
+
+Names that differ only in case or Unicode normalization still match — `Dict.LIFT` finds `Dict.lift-ranges` — unless several files match one name, which loads none of them and is reported as [`ambiguous-ranges-file`](validate.md#problem-codes).
+
 Pass `resolve_ranges=False` to `load()` to skip companion discovery.
 
 ## Media

docs/en/guides/read-edit-write.md

@@ -24,14 +24,17 @@ entry.lexical_unit["en"] = "grove"      # plain strings are coerced
 
 `Text` is structured — an ordered list of `str` and `Span` fragments — because `<text>` can contain nested `<span>` markup. `str(text)` flattens to plain text; the fragments keep the markup for round-tripping.
 
-Glosses are _form-shaped_ in LIFT (each `<gloss>` carries its own language), so a sense has `glosses: list[Form]` plus a helper:
+Glosses are _form-shaped_ in LIFT (each `<gloss>` carries its own language), so a sense has `glosses: list[Form]` plus helpers:
 
 ```python
-sense = entry.senses[0]
+sense = entry.senses[0]                 # top level only
 sense.gloss("en")                       # Text | None
-entry.gloss_langs()                     # {"en", "id"}
+entry.all_senses()                      # every sense and subsense, document order
+entry.gloss_langs()                     # {"en", "id"}, subsenses included
 ```
 
+Reach for `all_senses()` whenever a question concerns the whole entry: counting senses, collecting languages, finding media. `entry.senses` gives the top level, which is what you want only when the nesting itself matters.
+
 ## Saving
 
 ```python

src/sil_lift/_cli.py

@@ -30,7 +30,7 @@
 from ._validate import iter_problems
 
 if TYPE_CHECKING:
-    from collections.abc import Iterator, Sequence
+    from collections.abc import Sequence
     from typing import BinaryIO, TextIO
 
     from ._model import Entry, Sense
@@ -95,16 +95,6 @@ def _cmd_validate(args: argparse.Namespace) -> int:
     return 1 if failed else 0
 
 
-def _iter_senses(entry: Entry) -> list[Sense]:
-    senses: list[Sense] = []
-    stack = list(entry.senses)
-    while stack:
-        sense = stack.pop()
-        senses.append(sense)
-        stack.extend(sense.subsenses)
-    return senses
-
-
 def _cmd_stats(args: argparse.Namespace) -> int:
     from ._zip import lift_source
 
@@ -121,7 +111,7 @@ def _cmd_stats(args: argparse.Namespace) -> int:
                 pronunciations.extend(variant.pronunciations)
             for pronunciation in pronunciations:
                 media += len(pronunciation.media)
-            for sense in _iter_senses(entry):
+            for sense in entry.all_senses():
                 senses += 1
                 examples += len(sense.examples)
                 media += len(sense.illustrations)
@@ -195,18 +185,14 @@ def _cmd_check_media(args: argparse.Namespace) -> int:
     return 1 if missing else 0
 
 
-def _iter_leaf_senses(senses: Sequence[Sense]) -> Iterator[Sense]:
-    """Depth-first leaf senses, document order.
+def _leaf_senses(entry: Entry) -> list[Sense]:
+    """The entry's senses that carry content, document order.
 
     A sense with subsenses is a LIFT grouping node (e.g. numbered "1a"/"1b"
     under a bare "1") whose own gloss/definition are conventionally empty —
     its subsenses carry the content and get the rows instead.
     """
-    for sense in senses:
-        if sense.subsenses:
-            yield from _iter_leaf_senses(sense.subsenses)
-        else:
-            yield sense
+    return [sense for sense in entry.all_senses() if not sense.subsenses]
 
 
 def _text_or_empty(text: Text | None) -> str:
@@ -265,7 +251,7 @@ def _cmd_export(args: argparse.Namespace) -> int:
             detected: set[str] = set()
             with open_reader(lift_path) as reader:
                 for entry in reader:
-                    for sense in _iter_leaf_senses(entry.senses):
+                    for sense in _leaf_senses(entry):
                         detected.update(g.lang for g in sense.glosses if g.lang is not None)
                         detected.update(sense.definition.keys())
             langs = sorted(detected)
@@ -294,7 +280,7 @@ def _cmd_export(args: argparse.Namespace) -> int:
                 for entry in reader:
                     forms = entry.lexical_unit.forms
                     lexeme = str(forms[0].text) if forms else ""
-                    for sense in _iter_leaf_senses(entry.senses):
+                    for sense in _leaf_senses(entry):
                         pos = sense.grammatical_info.value if sense.grammatical_info else ""
                         row = [entry.id or "", entry.guid or "", sense.id or "", lexeme, pos]
                         for lang in langs:

src/sil_lift/_model.py

@@ -224,17 +224,31 @@ class Entry(_Extensible):
     relations: list[Relation] = field(default_factory=list)
     etymologies: list[Etymology] = field(default_factory=list)
 
+    def all_senses(self) -> list[Sense]:
+        """Every sense and subsense, depth-first in document order.
+
+        LIFT nests senses arbitrarily deep, so anything asking a question of
+        "the entry's senses" — how many there are, what languages they gloss,
+        what media they reference — means this list rather than
+        :attr:`senses`, which holds only the top level.
+        """
+        return list(_walk_senses(self.senses))
+
     def gloss_langs(self) -> set[str]:
         """Every language that has a gloss in any sense or subsense."""
         langs: set[str] = set()
-        stack = list(self.senses)
-        while stack:
-            sense = stack.pop()
+        for sense in self.all_senses():
             langs.update(g.lang for g in sense.glosses if g.lang is not None)
-            stack.extend(sense.subsenses)
         return langs
 
 
+def _walk_senses(senses: list[Sense]) -> Iterator[Sense]:
+    """Depth-first pre-order over a sense list and every subsense under it."""
+    for sense in senses:
+        yield sense
+        yield from _walk_senses(sense.subsenses)
+
+
 @dataclass(slots=True)
 class MediaRef:
     """One media reference in the document, with its owner's identity."""
@@ -946,14 +960,11 @@ def media_refs(self) -> Iterator[MediaRef]:
             for pronunciation in pronunciations:
                 for media in pronunciation.media:
                     yield MediaRef(media.href, "media", entry.id, entry.guid)
-            stack = list(entry.senses)
-            while stack:
-                sense = stack.pop()
+            for sense in entry.all_senses():
                 for illustration in sense.illustrations:
                     yield MediaRef(
                         illustration.href, "illustration", entry.id, entry.guid, sense.id
                     )
-                stack.extend(sense.subsenses)
 
     def missing_media(self) -> list[MediaRef]:
         """Media references whose files don't exist in the LIFT folder layout.

src/sil_lift/_reader.py

@@ -87,6 +87,8 @@ def _attach_ranges_source(ranges_file: RangesFile, data: bytes, root: etree._Ele
     from ._scan import scan
     from ._writer import _RangeRecord, _RangesSourceInfo, range_digest
 
+    # As in _attach_source: byte regions only mean anything in an
+    # ASCII-compatible encoding, and only this check rules the others out.
     encoding = root.getroottree().docinfo.encoding
     if encoding is not None and encoding.lower() not in ("utf-8", "us-ascii", "ascii"):
         return
@@ -119,9 +121,13 @@ def _attach_source(lexicon: Lexicon, data: bytes, root: etree._Element) -> None:
     from ._scan import scan
     from ._writer import _EntryRecord, _SourceInfo, entry_digest, header_digest
 
+    # The regions scan reports are offsets into these bytes, and the writer
+    # splices them into a document it declares UTF-8. expat parses UTF-16
+    # perfectly well and would report offsets into UTF-16 bytes, tags and all,
+    # so nothing downstream would notice; this is the only thing that says no.
     encoding = root.getroottree().docinfo.encoding
     if encoding is not None and encoding.lower() not in ("utf-8", "us-ascii", "ascii"):
-        return  # byte scanning assumes an ASCII-compatible encoding
+        return
     result = scan(data)
     if result is None:
         return