Run the JavaScript parser on TypeScript 7 - #8690
Draft
knutwannheden wants to merge 13 commits into
Draft
Conversation
TypeScript 7.0 ships without a compiler API, so 6.0 is the last release carrying the classic surface this module parses against. Three 6.0 behaviour changes needed handling: - `types` defaults to `[]`, which silently emptied type attribution for ambient `@types/*` packages; `types: ["*"]` restores enumeration. - `strict` defaults to true, adding a `null` constituent to every nullable union; it is now set explicitly, since strictness belongs to the parsed project rather than to the compiler this module pins. - Diagnostics 1212 and 1540 are new and non-fatal; the AST is unchanged in both cases, so they join `excludedCodes`. Also drops the deprecated `baseUrl` from the test and fixtures tsconfigs.
`getChildren`, `getChildCount`, `getChildAt`, `getFirstToken` and `getLastToken` surface the punctuation and keyword tokens that node fields omit, and the parser depends on them at 110 sites across 57 functions for the offsets its whitespace placement needs. TypeScript 7 exposes none of the five: its AST is deserialised from the Go compiler, which never materialises token-level children. They are reconstructible from `forEachChild`, `createScanner` and `node.jsDoc`, all of which 7 does export -- a prototype of that reconstruction matches the native implementation on 409699 nodes across 357 files, exactly, for all five entry points. The call sites move behind named helpers so that reconstruction has one place to land. The helpers delegate to the native methods, so behaviour is unchanged; inverting the rewrite mechanically reproduces the previous source except at the one optional-chain site, where `?.` becomes an equivalent `&&` guard. Also drops three untyped `getBaseTypes` calls, which are public API.
TypeScript 7 ships no compiler API: `typescript`'s main entry is a version string, and the AST, checker and program live behind `unstable/*` subpath exports. This adds the two pieces the parser will need there, each covered by a conformance test against the TypeScript 6 tree it has to reproduce. `ts7/token-navigation.ts` rebuilds getChildren/getChildAt/getChildCount/ getFirstToken/getLastToken, which 7 does not expose, from forEachChild, createScanner and the source text. Across the parser's own 26 sources the reconstructed token stream is identical to TypeScript 6's native one -- kinds, order and offsets -- once two equivalences are accounted for: `EndOfFileToken` is renamed `EndOfFile`, and an elision in a binding pattern is a zero-width `BindingElement` rather than an `OmittedExpression`. `ts7/program.ts` replaces createProgram and the CompilerHost. The Go process owns parsing and resolution, so in-memory sources are served over filesystem callbacks and everything else defers to the real filesystem; node_modules and the bundled lib files resolve without a module-resolution hook. The compiler is on `module: node20`, since 7 is ESM-only and this package is CommonJS. `node16` reports TS1479 on every `unstable/*` import; `node20` permits require(esm), which Node supports from 22.12, and leaves the emitted output CommonJS. That floor is declared in `engines.node`; with the ts7 modules kept out of the build the emit is byte-identical to node16, so it constrains building and testing rather than consumers of the package. The ts7 modules stay out of the published package, since they run against `typescript7`, a devDependency, and only the conformance test loads them. TypeScript 6 stays the compiler the parser runs on. The visitor cannot span both, because kind numbers differ and it compares against named members of one enum.
knutwannheden
changed the title
The checker answers over IPC from the Go process, which changes three things for the parser. An identifier that names a type resolves to `any` through `getTypeAtLocation`, since the name references a type rather than being an expression with one; the type comes from `getTypeFromTypeNode` on the type node containing it. `typeAtLocation` routes on that, which takes the identifiers in a fixture covering the constructs type-mapping.ts branches on from 46/57 agreeing with TypeScript 6 to all of them. Symbol flags already agreed for every identifier, which is what the mapping actually branches on. Two of those now agree by being more specific: routing through the type node gives `Base<string>` where 6 gives `typeof Base`, and `Circle[]` where 6 gives `T[]`. Both name the same reference and the instantiated form is what the LST wants, so the test records the pairs rather than hiding them. A symbol's declarations arrive as handles into the project's tree, so `declarationsOf` and `valueDeclarationOf` resolve them. `getAmbientModules`, `getFullyQualifiedName` and `signatureToString` have no counterpart, and are substituted here.
The parser reads kinds, node types and type guards from `typescript` in seven files. TypeScript 7 spells some of them differently and serves them from `unstable/ast` and `unstable/sync` rather than one entry point, so the binding moves to `compiler.ts` and the seven files import from there. None of the 1048 references change: TypeScript's namespace carries its types as well as its values, so re-exporting it as the default leaves every `ts.SyntaxKind` and `ts.Node` resolving as before. Pointing the module at TypeScript 7 now reports 184 errors, which is what the switch costs. They gather into renamed guards, program construction, the symbol and signature accessors, and the four known field renames.
knutwannheden
marked this pull request as draft
August 28, 2026 04:50
`compiler.ts` points at TypeScript 7 and the dependency moves, so the parser builds against `unstable/ast` and `unstable/sync` rather than a single entry point. Token navigation runs on the reconstruction, which returns the scanned tokens as nodes so the parser's call sites are unchanged. Beyond renames, the switch touched: - `parse` and `parseOnly` open a session instead of building a program. The compiler owns resolution, so the host and its `resolveModuleNameLiterals` override are gone, and paths are absolute throughout since that is how the compiler reads them. - Options take the shape a tsconfig holds. The API's `CompilerOptions` names `JsxEmit` and `ModuleResolutionKind`, and neither enum is reachable from any exported path (microsoft/TypeScript#64067). - Diagnostics carry flattened text and offsets, so nothing needs flattening. - Symbols hand back handles, and types answer through accessors. - `package-exported-types` reads a package's declaration entry from its manifest and its `exports` map, since there is no resolver to ask (microsoft/TypeScript#64069). - The visit methods for kinds the compiler no longer produces are removed, all of them JSDoc or Bundle, none reachable. `(symbol as any).parent` returned a private numeric handle rather than a symbol, which the compiler rejected as an empty symbol handle at runtime; `getParent()` is the accessor.
Three defects kept the parser from round-tripping, taking it from 377 of 644 parser tests to 614. Token start skipped whitespace only, so a comment ahead of a token left its start equal to its full start and the prefix came back empty, dropping the comment. Trivia is whitespace and comments both, which is what skipTrivia walks. Children were rebuilt on every ask, and callers locate a node among its siblings by identity, so the lookups failed. They are built once per node now. A NodeArray reports `hasTrailingComma` unset whatever the source says, so a trailing comma is read back off the text. (cherry picked from commit 939b8f054e323e3dc386eaf6278100cbbd86af70)
A named member holds whichever of `?` and `!` follows its name in one `postfixToken`, where they used to be separate fields, so optional members and definite assignments were both being dropped. A scanned token stands for punctuation the compiler leaves out of its tree and has no identity in the compiler process, which rejects it with `getNodeId requires a RemoteNode`; the type mapping asks about real nodes only. 634 of 644 parser tests pass, from 614. (cherry picked from commit b255265e1d742db452795b6391868491e8816871)
The compiler reparses JSDoc into the tree: a `@param` or `@returns` type becomes a type annotation on the declaration it documents, positioned inside the comment it came from and flagged Reparsed. Read as syntax it prints as an annotation the source never had, so those nodes are left out of a node's children and out of the type information the LST carries. It also reports syntax errors from inside JSDoc, which the previous release let a parser turn off and this one does not, so a diagnostic whose position falls in a comment is not treated as a parse failure. 636 of 644 parser tests pass, from 634. (cherry picked from commit a5d78e0a4f278b45a51315915fa7553e1e6dd17c)
An elision in a binding pattern is a binding element with nothing bound rather than an omitted expression, so it has no name to map and the hole stands on its own. Sources held in memory were reachable only at the root the session opened at: a directory holding them reported neither its entries nor its own existence, so an import of a file below the root resolved to nothing and everything it named came back untyped. Both answers now account for the sources in hand. All 644 parser tests pass. (cherry picked from commit ea7f163a82fb731a46a0dab60008ceb1608f441e)
The suite is green on TypeScript 7: 2035 tests, the same as on 6. Type attribution reaches the checker through the type node enclosing a name, which is what an identifier in a type position resolves through; asking about the identifier itself answers `any`. Module names are derived against the root the session opens at, since the paths reaching the compiler are absolute. Package resolution is done here, as the compiler exposes no resolver: an `exports` map is read for the condition that names declarations, trying `import` before `require`, and the declaration files are looked for in the package built for the running platform, which is where they ship. A class exported as the default carries `default` as its symbol name rather than the name it was written with, which is only on the declaration, so qualified names are built from there. (cherry picked from commit 0b5797b7958fa567d1ec603ecc6dc54e7bb1c210)
The TypeScript 6 package the conformance test compares against declares the same `tsc` bin as TypeScript 7, and npm linked that one, so the scripts were compiling against 6 while the module runs on 7. They name the compiler by path instead. (cherry picked from commit 0d1ed738de1eb3c45b2b1096099d46e70c8b412e)
A parse was starting a compiler and loading its libraries each time, which costs 59ms against the 0.09ms the parse itself takes. The process now lives as long as the parser and a project is opened per batch, taking the suite from 205s to 126s. Reusing it means a path can be parsed twice with different text, and the compiler answers with what it read the first time, so each batch announces its sources as changed. (cherry picked from commit 70c562d3503f7041deb77672c50b79d0cbd37eeb)
knutwannheden
changed the title
Base automatically changed from
typescript-7-exposure-in-rewrite-javascript
to
main
August 28, 2026 11:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #8689, which moves this module to TypeScript 6 — review that first.
The parser runs on TypeScript 7. The full suite passes at 2035 tests, the same count as on 6, with no test disabled or removed.
TypeScript 7.0 ships no compiler API in the sense 6 did:
typescript's main entry islib/version.cjs, which exports a version string, and the AST, checker and program construction live behindunstable/*subpath exports that the 7.1 iteration plan does not schedule for stabilisation. Waiting for 7.1 would not change that — 7.1 stabilises the Content Mapper, Emit and Language Service APIs, none of which are what a parser reads. So this builds onunstable/*and pins the deviations with a conformance test that compares the two compilers directly.The seam
JavaScriptParserreached token-level navigation —getChildren,getChildCount,getChildAt,getFirstToken,getLastToken— at 110 sites across 57 functions. Those five surface the punctuation and keyword tokens that node fields omit, which is where whitespace placement gets its offsets, and TypeScript 7 exposes none of them: the AST is deserialised from the compiler process, which never materialises token-level children.The first commit moves those sites behind named helpers, still delegating to the native methods, so it is behaviour-preserving. The rewrite was mechanical and checked as such: inverting it programmatically reproduces the previous source byte-for-byte in
parser-utils.ts, and inparser.tseverywhere except one site, wherenode.elseStatement?.getChildAt(...)becomes!!node.elseStatement && childAt(node.elseStatement, ...). Both short-circuit before evaluating the argument.compiler.tsis then the single place the parser binds to a compiler release. None of the 1048ts.*references change: the namespace carries its types as well as its values, so re-exporting it leaves everyts.SyntaxKindandts.Noderesolving as before, and the release-specific renaming lives in one file.Reading the TypeScript 7 tree
ts7/token-navigation.tsrebuilds the five missing calls fromforEachChild(aNodemethod in 7, not a free function),createScanner, and the source text. Across the parser's own sources the reconstructed token stream is identical to TypeScript 6's native one — kinds, order and offsets — once two equivalences hold, both recorded in the test rather than skipped:EndOfFileTokenis renamedEndOfFile.const [a, , b] = …) is a zero-widthBindingElementwhere 6 emits anOmittedExpression. The equivalence is conditional on zero width.Four details had to be right for the offsets to line up, and the conformance test found each of them:
posis its full start, sogetTokenFullStart()is the source rather thangetTokenStart().|of| "a" | "b"disappears.Symbol; tagging it with anelementsfield silently capturedNamedImportsandArrayLiteralExpression, whose braces and brackets then vanished.skipTrivia. Searching for whitespace alone leftgetStart()equal togetFullStart()ahead of a comment, and every comment before a token was dropped.Children are built once per node, because callers locate a node among its siblings by identity.
Program construction and the checker
ts7/program.tsreplacescreateProgramand theCompilerHost, neither of which exists in 7. The compiler owns parsing and module resolution, so a program is described to it: in-memory sources are served over filesystem callbacks, and returningundefineddefers to the real filesystem.resolveModuleNameLiterals, whichparser.tsoverrode because its sources are not on disk, has no counterpart to port — the model removes the need.Starting a compiler and loading its libraries costs 59ms against the 0.09ms a parse takes, so the process lives as long as the parser and a project is opened per batch. Reusing it means a path can be parsed twice with different text and the compiler answers with what it read first, so each batch announces its sources as changed.
ts7/checker.tscovers what moving the checker into that process changes. An identifier that names a type resolves toanythroughgetTypeAtLocation— the name references a type rather than being an expression with one — and the type comes fromgetTypeFromTypeNodeon the type node containing it. Routing on that takes agreement with TypeScript 6 from 46/57 identifiers to all of them across a fixture covering the constructstype-mapping.tsbranches on. Symbol flags agreed for every identifier before and after, which matters more, since flags are what the mapping branches on.Two of those now agree by being more specific: routing through the type node gives
Base<string>where 6 givestypeof Base, andCircle[]where 6 givesT[]. Both name the same reference and the instantiated form is the one the LST wants.Deviations worth knowing
JSDoc never entered the tree before, because the parser sets
jsDocParsingMode: ParseNone. TypeScript 7 offers no equivalent and reparses JSDoc types into the AST as nodes of the declaration they annotate, positioned inside the comment and flaggedReparsed. Read as syntax they print as annotations the source never had, so they are excluded from a node's children and from the type information the LST carries. It also reports syntax errors from inside JSDoc, so a diagnostic whose position falls in a comment is not treated as a parse failure.(symbol as any).parentreturned a private numeric handle rather than a symbol, which the compiler rejected at runtime as an empty symbol handle.getParent()is the accessor, and this is the shape of failure the internal-API casts were always going to have: they type-check throughas anyand break only when run.A class exported as the default carries
defaultas its symbol name rather than the name it was written with, which is only on the declaration, so qualified names are built from there.Filed upstream
Six behaviours are gaps or bugs rather than deliberate changes, each verified against
7.0.2and7.1.0-dev.20260827.1:CompilerOptionsreferencesJsxEmitandModuleResolutionKind, neither exported from any path, and omitsesModuleInterop/allowSyntheticDefaultImportswhichtscaccepts. This is why options here take the shape a tsconfig holds.resolveModuleName, sopackage-exported-typesreadsexportsmaps itself.jsDocParsingMode, and reparsed JSDoc appears as syntax.@typescript/olddeclares atscbin that wins overtypescript@7, so the build names the compiler by path.DocumentIdentifierfails with aTypeErrorfrompath.js.Nodeis missinggetChildren(),getFirstToken(),getLastToken()and related getters microsoft/TypeScript#63892 (already open) — the missing child and token getters this PR reconstructs.microsoft/TypeScript#63861, which left
NodeArray.hasTrailingCommaunset, is fixed in 7.1-dev but not in 7.0.2, so trailing commas are read off the source here.Build and tests
Cold builds are faster; the suite is not yet:
The suite gap is one compiler process per parser instance, at 60.5ms against 1.5ms for a reused one, across roughly two thousand instances. A compiler shared between parsers would close it, and is left out of this PR because its lifetime is a design question rather than a detail.
Scope
module: node20is required: TypeScript 7 is ESM-only and this package is CommonJS, sonode16reports TS1479 on everyunstable/*import. That needs Node 22.12, whichenginesand the module's CLAUDE.md now state.The 44
visitJSDoc*methods and the ones forBundleandCommaListExpressionare removed. The dispatch map is built from the liveSyntaxKind, so kinds the compiler no longer produces were unreachable.