fix(context): keep formatted and linked text in section titles - #175
Conversation
getHeadingText read only a heading's direct text and inlineCode children, so text inside emphasis, strong and links was dropped. A heading that was entirely a link or bold got an empty title and fell through to "Introduction". Read inline text recursively, and drop permalink anchors (a # link whose text is only ¶, §, # or a zero-width space) inside <h2>, which would otherwise now end up in the title.
🦋 Changeset detectedLatest commit: 496088a The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![human0-bot[bot]](https://avatars.githubusercontent.com/in/3310509?s=60&v=4){kind=small}
There was a problem hiding this comment.
Approved — the recursive heading extraction and h2 permalink handling address the reported title loss without changing section bodies or lower-level anchors.
Docs & rules — low risk, comment: the predicate and changeset contain details not reproducible from tracked files.
Question: .changeset/keep-heading-formatting.md includes corpus counts and before/after comparisons with no checked-in reproduction material.
This codebase is managed by Human0.
| // without this they end in the symbol. Only <h2>, which becomes the section title: | ||
| // anchors in other headings stay in the content as before. A rule, not remove(): | ||
| // the link rule would match <a> first. | ||
| const PERMALINK_TEXT = /^[¶§#🔗]?$/u; |
There was a problem hiding this comment.
Docs & rules · comment: The permalink predicate also accepts 🔗, which is not listed in the PR description. Please add the symbol to the description/comment or remove it from the regex so the documented scope matches the code.
This codebase is managed by Human0.
| "@neuledge/context": patch | ||
| --- | ||
|
|
||
| Keep bold, italic, link and inline-code text in section titles. Only a heading's top-level text was read, so words inside formatting or links were dropped from the title, which has the highest search weight. In the Python docs, 300 of 6,475 sections lost words (`Numeric Types — int, float, complex` became `Numeric Types — , , `), including 152 FAQ and guide sections whose heading is a link and which were titled "Introduction". Markdown was hit too: `## Using [superjson](...)` became "Using ". Heading permalinks such as Sphinx's "¶" stay out of the title. |
There was a problem hiding this comment.
Docs & rules · question: Can you add the source or a short reproduction procedure for the corpus counts and before/after comparisons in this changeset? They are not reproducible from the tracked scripts, fixtures, or artifacts.
This codebase is managed by Human0.
Releases @neuledge/context 1.2.10 -> 1.2.11 (patch). Consumes four changesets: #147 (keep installed docs when a rebuild fails), #175 (keep formatted and linked text in section titles), #180 (keep code language fences in HTML pre/code blocks) and #185 (clean up abandoned staging directories). @neuledge/registry 0.0.23 -> 0.0.24 is the automatic dependent bump for the private workspace package. Verified before merging: npm dist-tags.latest is 1.2.10 and 1.2.11 is not published yet. The code reviewer approved the current head (eb13c24), and CI on main is green at ef14e81.
1 participant
Problem
getHeadingTextbuilds a section's title from the heading's directtextandinlineCodechildren only. Text inside a link,**strong**or*emphasis*is one level deeper, so it is dropped. When the whole heading is a link or bold, the title comes out empty, and the empty string falls through to the "content before the first h2" branch, so the section is filed as "Introduction".sectionTitlehas the highest weight in search (bm25(chunks_fts, 5.0, 10.0, 1.0)), so the lost words are the ones that count most.HTML docs are hit hardest, because turndown turns Sphinx's cross-references into links inside headings. Python 3.14 (
python-3.14-docs-html, the registry's source), before and after:int,float,complexNumeric Types — , ,Numeric Types — int, float, complexstrText Sequence Type —Text Sequence Type — strsitemoduleConstants added by the moduleConstants added by the site moduleOrderedDictobjectsobjectsOrderedDict objectsIntroductionWhy are Python strings immutable?The FAQ and several guides wrap every heading in a back-link to the page's contents (
<a class="toc-backref">), so the whole title is link text.Counts from parsing every page:
The Markdown cases: tRPC's
## Using [superjson](...)and## Using [devalue](...)are both titled "Using ", and Drizzle's FAQ## **Should I usegenerateorpush?**is filed as "Introduction".Fix
getHeadingTextreads inline text recursively (getInlineText):textandinlineCodevalues, and the children of any node that has them. Nodes without text (html,image,break) are still skipped, as before.<a class="headerlink" href="#...">¶</a>in every heading, rustdoc uses§, VuePress#and Docusaurus a zero-width space. The old code only dropped them by accident, as link text. A turndown rule now drops an<a href="#...">inside an<h2>whose text is only one of those symbols. The<h2>is the heading that becomes the section title. The rule matches on text, not class names, so it covers generators outside the registry too. It is a rule rather thanturndown.remove(), because turndown's link rule matches<a>beforeremove()is consulted.The rule is limited to
<h2>on purpose. The permalinks in<h3>and below are part of section content, and they count towardisTableOfContents's link ratio. Dropping them everywhere let the intro section of 35 Python pages, Sphinx's navigation bar included, pass that filter for the first time. That is a separate question, so this PR leaves the content alone: every section body is identical to main, apart from one line inusing/cmdline.htmlwhere an## ...¶heading already sat inside a body and now reads without the anchor.Not changed
<h3>and below stay in content, as before. Dropping them would change which sections the table-of-contents filter keeps, as described above. Happy to follow up if you want that.using/cmdline.html, "1.2. Environment variables" gets no section of its own. Its<h2>ends up inside the last part of "1.1. Command line". I have not looked into why. It is separate from this change.Test
Two tests. Both fail on
fc503b7and pass with the fix:build.test.ts: a link, a fully bold heading with inline code, and emphasis giveUsing superjson,Should I use generate or push?andThe strict option. On main:Using,Introduction,The option.html.test.ts: Sphinx markup (atoc-backrefquestion, a<code>cross-reference,¶permalinks), a rustdoc§anchor and a Docusaurus zero-width anchor nested in a<span>giveWhy are Python strings immutable?,Constants added by the site module,TraitsandSetup. An<h3>permalink stays in the content. On main the first two areIntroductionandConstants added by the module.I broke each part of the fix in turn, and each time a test went red:
inlineCodenot read<h3>permalink disappears from content)<h2>parent only, instead ofclosest("h2")Setupkeeps its zero-width space)§not treated as a permalinkTraits§)Setupkeeps its zero-width space)Validation
tsc -p tsconfig.build.jsonandbiome ci --error-on-warningsare clean..changeset/keep-heading-formatting.md(patch).