feat(desktop): find design-token sources in a picked folder

[wudilyy999]

Summary

First slice of #484: re-export the three brand token extractors from the core entry, and add scanTokenSources, which finds the files in a folder that carry design tokens and runs the extractors over them. Nothing calls the scan yet. PR 3 of the series wires it to the import action and loads it with await import(...) at the IPC handler, so it stays off the startup path (CLAUDE.md constraint 5). The two changes are separate commits so the re-export and the scan can be reviewed apart.

• Re-export (packages/core/src/index.ts): extractFromTailwindConfig, extractFromCssVars and importDtcgJson by name, no export *. They are side-effect-free text parsers, and the scan is not reachable from core's entry.
• Scan (apps/desktop/src/main/design-system-tokens.ts): returns the tokens per source file for tailwind.config.{js,cjs,mjs,ts}, CSS containing :root, [data-theme] or @theme, and tokens.json / *.tokens.json. It also reports the files it skipped (over the size cap, invalid JSON) and whether the file cap cut the walk short, as data for the preview and not as log lines.

What the scan does and does not do:

• Reads text only and executes nothing from the folder. The Tailwind config goes to the existing extractor, which parses it as text; a test scans a config that would write a file if it were run and checks that no file appears.
• Skips node_modules, .git, build output and the rest of the existing ignore list (IGNORED_DIRS, now exported from design-system.ts), and never follows symlinks.
• Is bounded by named limits (DEFAULT_TOKEN_SCAN_LIMITS): depth 6, 10,000 files looked at, 256,000 bytes per file.
• Reports a file as skipped only when it is over the size cap or is not valid JSON. A file that is not what its name says does not fail the scan: valid JSON that is not shaped like DTCG tokens, and a Tailwind config or CSS with unbalanced braces, produce no tokens and no error. Read errors (an unreadable file or directory) throw instead of being skipped, so a permissions problem is not hidden. Only a pathological file, such as JSON nested tens of thousands of levels deep, overflows the stack and fails the scan with that error.

The existing "Link design system repo" scan (scanDesignSystem) is unchanged, and the two write to different stores. The existing scan keeps an app-scoped list of candidate values in the app config. The import this series builds writes <workspace>/DESIGN.md, per design, and will be its own action with its own IPC channel (PR 3); it does not read or replace the snapshot.

One difference from the plan on #484: the walk is new and the existing one is not reused. The existing walk stops after 160 candidate files in directory order and keeps only a prompt summary, so on a larger repo it can stop before it reaches the token files, and it has no tests. This scan shares its ignore list (IGNORED_DIRS) and has its own limits, named constants that tests override. They are separate from the snapshot's caps, so the two scans do not share a budget.

Run on real folders: open-cowork (3 sources, 39 tokens) and this repo (6 sources, 190 tokens), about 12 ms each, neither truncated.

Type of change

• Bug fix
• New feature
• Refactor (no behavior change)
• Documentation
• Build / CI / tooling
• Breaking change

Linked issue

Refs #484 (PR 1 of 4)

Checklist

• I checked the linked issue / relevant context before starting
• pnpm lint && pnpm typecheck && pnpm test passes locally
• Added/updated tests for the change
• Added a changeset (pnpm changeset) if user-visible
• Updated docs if behavior changed

Nothing is user-visible until the scan is wired in, so the changeset comes with that PR. The docs page is PR 4.

Principles

• Compatibility: nothing calls the scan; the only change to existing code is export on IGNORED_DIRS.
• Upgradeability: parsing stays in the existing extractors; the scan only finds the files and groups the results.
• No bloat: no new dependencies; 139 lines of source outside tests.
• Elegance: one function, one structured result; limits are a named object that tests override.

Checks

• pnpm -r typecheck, pnpm lint, pnpm test: 10 tasks passed; core 560 tests, shared 326, desktop 181 files / 2,640 tests.
• design-system-tokens.test.ts (11 tests): a Tailwind config, :root CSS, @theme CSS and DTCG JSON in one folder, in a stable order; @theme and :root in one file; ignored directories; depth limit; file cap with the truncation flag; size cap; invalid JSON; valid JSON that is not DTCG-shaped; a malformed config and CSS; a config that is parsed and not run; symlinks to a directory and to a file are not followed.
• Each of the ignore list, the depth limit, the truncation flag, the size cap, the JSON error report and the two symlink checks fails its test when removed.

[github-actions]

Findings

• [Minor] A single structurally-invalid token file can abort the whole scan — readTokens only translates a SyntaxError (from JSON.parse) into a skippable invalid-json result, while every other throw is re-thrown (apps/desktop/src/main/design-system-tokens.ts:99-106). importDtcgJson(...) runs inside the same try, so if it throws for valid JSON that is not DTCG-shaped (e.g. a plain tokens.json from another tool), or if extractFromTailwindConfig/extractFromCssVars throw on an unusual tailwind.config.* / CSS file, scanTokenSources rejects and the whole folder import fails over one unrelated file. This is the exact case the scan is meant to tolerate: it already reports invalid-json and too-large as data rather than failing.
  Suggested fix / next action: either broaden the recoverable branch to cover extractor/shape failures (add a skipped reason such as invalid-tokens) so a bad file only excludes itself, or confirm and document that importDtcgJson and the extractors only throw for genuine I/O errors. Add a test for (a) valid JSON with a non-DTCG shape and (b) a malformed tailwind.config.js, since the current tests only cover '{ not json' (apps/desktop/src/main/design-system-tokens.test.ts).

Questions

• Can importDtcgJson throw anything other than SyntaxError for in-size files? The error instanceof SyntaxError narrowing implies a deliberate choice to re-throw everything else, but the intent isn't obvious from the diff.

Summary

• Review mode: initial
• The re-export and the scan are small, focused, and consistent with the project constraints I could check in the public repo: no new dependencies, no direct provider SDK import, all LLM/parsing work stays in the existing packages/core extractors, IGNORED_DIRS is only widened from const to export const (apps/desktop/src/main/design-system.ts:9), and nothing references scanTokenSources yet, so there is no startup-path cost. Symlink and size limits are enforced and tested.
• Highest-severity item is the Minor above. No Blocker or Major issues found.
• Issue validation for #484 could not be performed in this run (no issue fetch available); the PR uses Refs #484 and describes itself as PR 1 of 4, which is the correct link style for a partial slice, so no closure claim is being over-asserted.
• Residual risk: skipped can list any over-cap .css file (every .css is a candidate via sourceKind, and the size check runs before token detection), so a large token-less stylesheet may appear as a skipped token source in the PR 3 preview. Worth a look when the preview surfaces this list, but not a merge concern here.
• Deferring the changeset to PR 3 is reasonable as long as a changeset covering the new @open-codesign/core exports is added before the next version/release.

Testing

• Not run (automation). The added design-system-tokens.test.ts covers ordering, both CSS block forms, ignored dirs, depth/file/size caps, invalid JSON, text-only config parsing, and symlink non-following. Gap: no coverage for non-SyntaxError failures from the DTCG importer or the extractors (see finding above).

Open-CoDesign Bot

[github-actions]

Findings

No new findings. The single Minor from the previous review is addressed:

• The prior concern was that readTokens narrows recovery to SyntaxError, so a token file that makes importDtcgJson throw for another reason would reject the whole scan (apps/desktop/src/main/design-system-tokens.ts:99-108). The intent is now documented at apps/desktop/src/main/design-system-tokens.ts:100-101: importDtcgJson accepts any parsed value, so only malformed JSON is treated as skippable, and read errors propagate deliberately.
• New tests exercise exactly the cases the prior review asked for: valid JSON that is not DTCG-shaped ([1, 2, 3], null, a package.json-shaped object) and a malformed tailwind.config.js plus unbalanced CSS both return { sources: [], skipped: [], truncated: false } (apps/desktop/src/main/design-system-tokens.test.ts:126-147). These assertions would fail if importDtcgJson or the extractors threw for those inputs, so the tests validate the documented behavior rather than merely restating it.

Questions

None. The prior open question — whether importDtcgJson can throw anything other than SyntaxError for in-size files — is answered by the added comment and the new non-DTCG-shape tests.

Summary

• Review mode: follow-up after new commits
• Scope is unchanged from the prior review: re-export three packages/core brand extractors by name (no export *) and add an unused scanTokenSources walker. No new dependencies, no direct provider SDK imports, all parsing stays in the existing packages/core extractors, and IGNORED_DIRS is only widened from const to export const (apps/desktop/src/main/design-system.ts:9). Nothing references scanTokenSources, so there is no startup-path cost and the lazy-load constraint is respected.
• The diff since the previously reviewed commit is test-only plus a two-line comment; no behavioral change beyond the recoverable branch now being backed by tests.
• Residual risk (unchanged, non-blocking): readTokens still re-throws any non-SyntaxError from extractFromTailwindConfig/extractFromCssVars on the config/CSS paths (apps/desktop/src/main/design-system-tokens.ts:98, :111-112), so an unexpected extractor throw would abort the scan for the whole folder. There is no evidence the extractors throw on in-size text today; if the PR 3 preview ever surfaces such a case, add a dedicated skipped reason (e.g. invalid-tokens) instead of failing the import.
• Residual risk (unchanged, non-blocking): skipped can list any over-cap .css file, because every .css is a candidate and the size check runs before token detection, so a large token-less stylesheet can appear as a skipped token source. Worth re-checking when PR 3 renders the list.
• Linked issue: the PR uses Refs #484 and describes itself as PR 1 of 4, which is the correct link style for a partial slice; there is no closure claim to validate. The issue #484 body was not available in this run, so I could not re-check its acceptance criteria against this slice.
• Deferring the changeset to PR 3 is reasonable as long as the new @open-codesign/core export addition is covered before the next version bump.

Testing

• Not run (automation). The added design-system-tokens.test.ts covers ordering, both CSS block forms, ignored dirs, depth/file/size caps, invalid JSON, non-DTCG-shaped JSON, malformed config/CSS, text-only config parsing, and symlink non-following. No Playwright coverage is expected for an unused main-process helper.

Open-CoDesign Bot