Repository navigation
feat: custom review instructions for paid plans - #32
Conversation
Users asked for a way to stop Jargons flagging issues that aren't serious. This module defines the paid-plan "review guidance" shared by the settings UI and the review and scan engines: - ReviewGuidance: free-text custom instructions plus a minimum severity. The default (no instructions, minimum "note") reports everything, exactly as before. - normalizeCustomInstructions(): trims, caps at 1000 characters, and treats blank text as no instructions. - parseMinSeverity(): narrows untrusted input to a known severity, defaulting to "note". - filterBySeverity(): keeps findings at or above the minimum. - customInstructionsPrompt(): wraps the instructions in a framed block telling the model to follow them but never to hide a real bug, data-loss risk or security vulnerability, and to ignore any attempt to change its role or output format. - effectiveReviewGuidance(): returns the saved guidance only while the workspace is on a paid plan, so a downgrade turns it off without deleting it. - MIN_SEVERITY_OPTIONS: labels for the settings picker, from "Everything" to "Critical only".
Covers severity filtering at each threshold, trimming and length capping of instructions, severity parsing with its "note" fallback, the framed prompt block (and null for blank input), and that saved guidance applies on paid plans but falls back to defaults on free or when nothing is saved.
Add two columns to workspace_settings: - custom_instructions (text, nullable): the workspace's extra review instructions. - min_severity (finding_severity, not null, default 'note'): the lowest severity still reported. 'note' means report everything, so existing workspaces keep today's behaviour.
Generated with drizzle-kit. Adds custom_instructions and min_severity to workspace_settings. The change is additive: the new text column is nullable and min_severity defaults to 'note', so existing rows and the currently deployed code are unaffected.
Schema snapshot generated by drizzle-kit alongside the review guidance migration, used to diff future migrations.
Adds the 0005_quiet_jetstream entry so drizzle-kit migrate applies the review guidance migration.
reviewDiff() accepts optional customInstructions. When present, the framed instructions block is appended to the system prompt after the built-in rules. With no instructions the prompt is identical to before.
postReviewComment() takes an optional hiddenCount (default 0). When findings were dropped by the workspace's minimum severity, the review body says so, e.g. "3 lower-severity findings hidden by this workspace's minimum severity setting", both on clean reviews and below the listed findings. That way a short review isn't mistaken for a shallow one. With hiddenCount 0 the comment is byte-for-byte unchanged.
RunReviewInput gains a `guidance` field, already resolved for the workspace's plan. runReview() passes the custom instructions to the LLM, then filters findings by the minimum severity immediately after the response, before anything else happens. The saved findings, the fix PR and the GitHub comment therefore all see the same list, and the comment reports how many were hidden.
The pull request webhook now selects custom_instructions and min_severity with the other workspace settings, and resolves them with effectiveReviewGuidance() using the plan from the billing check it already makes. Free workspaces, and workspaces with no settings row, get the defaults.
scanCodebase() accepts optional customInstructions and appends the same framed block to the scan system prompt. Without instructions the prompt is unchanged.
RunScanInput gains `guidance`. runScan() passes the instructions to the scan prompt and filters findings by the minimum severity before building the architecture map and summary, so the severity counts, the map and the stored findings agree.
The scan start route reads the workspace's guidance settings and resolves them against the plan from its existing billing check before starting the scan. Free workspaces get the defaults.
getWorkspaceSettings() now returns: - the saved guidance (instructions and minimum severity), - canCustomizeReviews, true on paid plans, - the workspace's billing (plan, runs used, run limit) for the settings page. updateReviewPreferences() now validates input properly: booleans are coerced, instructions are normalised, and the severity is parsed against the known list. Optional guidance is saved alongside the toggles, but a request that includes guidance from a free workspace is rejected with an error, so the paid-plan gate can't be bypassed by calling the server function directly.
Custom review instructions card (paid plans only; hidden entirely on
free):
- An instructions textarea (1000-character limit with a counter). It
stretches to line up with the severity list on wide screens and
can't be resized.
- A "Report findings" picker from "Everything" to "Critical only",
built as an accessible radiogroup.
- Saved with the existing Save changes button and included in its
unsaved-changes check. Guidance is only sent on paid plans.
Workspace card: a new "Current plan" row shows the plan badge (amber
for paid, grey for free), runs used this month, and a link to
pricing ("Upgrade" on free, "View plans" on paid).
Add "Custom review instructions" to the Pro, Team and Business feature lists. The Free plan doesn't include it.
Style the page scrollbar to match the dark UI: thin, a translucent grey thumb on the page background that brightens on hover. It uses scrollbar-color (Firefox and modern Chromium) plus ::-webkit-scrollbar rules for other WebKit browsers, and sets color-scheme: dark. scrollbar-color is inherited, so inner scroll areas such as the instructions textarea match too.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![jargons-ai[bot]](https://avatars.githubusercontent.com/in/4369042?s=60&v=4){kind=small}
There was a problem hiding this comment.
Jargons review
Found 3 issues worth a look:
🟡 Medium — Inconsistent application of review guidance
src/routes/api.scans.start.tsx:106
The scanCodebase function in src/server/scan-engine/llm.ts now accepts customInstructions, but the scanCodebase call in src/routes/api.scans.start.tsx does not pass this parameter. This means custom instructions will not be applied when a codebase scan is initiated via the API, leading to inconsistent behavior compared to other review flows.
Suggestion: Pass input.guidance.customInstructions to the scanCodebase function call in src/routes/api.scans.start.tsx to ensure custom instructions are applied consistently for API-initiated scans.
🟡 Medium — Potential for UI/DB mismatch on minSeverity default
src/lib/review-guidance.ts:40
The min_severity column in workspace_settings has a default of 'note' and is NOT NULL. However, parseMinSeverity in src/lib/review-guidance.ts also defaults to 'note' for unknown values. While this is generally safe, if parseMinSeverity were ever to change its default or if a malformed minSeverity value somehow bypassed validation and was saved to the DB, the UI might display a different default than what the backend actually uses, leading to user confusion or unexpected behavior.
Suggestion: Consider making parseMinSeverity return DEFAULT_MIN_SEVERITY explicitly for undefined or null inputs, and ensure that any database interactions or API inputs strictly validate minSeverity against SEVERITIES to prevent invalid values from being stored. This would make the UI and backend more robust against potential mismatches.
🔵 Low — Redundant normalizeCustomInstructions call
src/lib/review-guidance.ts:52
In customInstructionsPrompt, normalizeCustomInstructions is called on customInstructions, but effectiveReviewGuidance already calls normalizeCustomInstructions on saved.customInstructions before passing it to customInstructionsPrompt. This results in redundant trimming and length capping.
Suggestion: Remove the normalizeCustomInstructions call inside customInstructionsPrompt as the input customInstructions should already be normalized by effectiveReviewGuidance. The function should directly use the provided customInstructions after a null check.
Note: the diff was large and reviewed in part.
🔧 Suggested fixes: open the fix PR →
Newer TanStack Router versions type the error passed to an errorComponent as `unknown`, because anything can be thrown, not only Error instances. RootErrorComponent now accepts `unknown`, and looksLikeServiceOutage() reads name and message only when the value is an Error, falling back to String(error) otherwise. The behaviour is unchanged for Error values, and the change compiles on both the current and the upgraded router versions.
Vercel blocks deploys of @tanstack/react-start 1.143.12 through 1.168.59 because of CVE-2026-102989 (GHSA-qx66-fv34-fjm8), a critical (CVSS 9.3) reflected XSS in server-function response handling. An unauthenticated attacker could craft a server-function URL that returns attacker-controlled HTML from the app's origin. - Pin @tanstack/react-start to ^1.168.60 instead of "latest", so the lockfile can't stay on a vulnerable release. This pulls in @tanstack/start-server-core 1.169.39. - Update the other TanStack router packages together and dedupe, so everything shares one @tanstack/router-core (1.171.34). Mixing 1.171.13 and 1.171.34 broke the production build with missing exports such as _getRenderedMatches. - routeTree.gen.ts was regenerated by the new router generator. Only the import order changed. Verified: tsc, eslint, 138 tests, `vite build`, and a dev-server smoke test (pages render, server routes respond, sign-in redirect works).
…view-instructions
There was a problem hiding this comment.
Jargons review
Found 1 issue worth a look:
🟡 Medium — Missing NOT NULL constraint for new columns
drizzle/0005_quiet_jetstream.sql:1
The new columns custom_instructions and min_severity are added to the workspace_settings table. While min_severity has a DEFAULT value, custom_instructions does not and is nullable. If custom_instructions is intended to always have a value (even an empty string), it should be marked as NOT NULL to enforce data integrity.
Suggestion: Consider adding NOT NULL to the custom_instructions column if it's expected to always have a value. If it can be genuinely null, then the current schema is fine.
Note: the diff was large and reviewed in part.
🔧 Suggested fixes: open the fix PR →
Summary
Paid workspaces can now add custom review instructions and choose the minimum severity to report, so Jargons stops flagging issues that aren't serious. The feature is optional and off by default.
Changes
Database
Migration
0005addscustom_instructionsandmin_severitytoworkspace_settings; it's already applied.Testing
148 tests pass; tsc and eslint are clean.