diff --git a/ROADMAP.md b/ROADMAP.md index 8365b2d..2bf1471 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,9 +4,26 @@ This is the living implementation record. A checked implementation item means co ## Release-only mode and merge gates -The latest staged-release brief governs release order. #28–#30 remain unchanged by this work. Aryan explicitly requested live staff-card status and an acceptance notice on October 11; that narrow feature is on the separate `feat/live-staff-case-status` draft branch, not deployed or blanket-approved for release. No diagnostics or new backend access accompanies it. +The latest staged-release instructions govern this work. Do not add diagnostics or unrelated features. Keep each release isolated, verified and separated by 24 clean hours unless Aryan explicitly says go. Omi Support Bot is the product name; technical identifiers remain unchanged. -### October 11 current production and review gate +### October 12 current production and next gates + +- Reports 1 and 2 were graded PASS. Phase 2 shipped the foundation and six approved contributor fixes at main `8321439`, deployment `5b25759b-d01e-4645-ade1-c869a482b323`, with `8e9e453b-d60b-4001-86c0-8303deaf123d` as that release's rollback target. The observed Phase 2 suite was 638 tests and offline 8/8. A later same-code deployment, `6ddd1673-2e45-4cac-9d0d-28c71a9107fd`, was the healthy active copy immediately before Phase 3. +- Phase 2 deployed October 11 at 12:08:24 UTC / 17:38:24 IST. Health returned HTTP 200, `status=ok`, `staffHandoff=configured`; settings and variables were unchanged. All four synthetic posts received one reply. The extra long guide exercised clipping at 1,900 characters and retained one complete Source line and both URLs, but its body was incomplete. Staff delivery was screenshot-confirmed; Aryan intentionally removed the test cards. The public filing formatter hid phone/order offline; a live filing draft was not inspected. Preserve these limitations rather than treating the smoke as blanket answer quality approval. +- #41 (`feat/live-staff-case-status`, reviewed head `e5cff9e`) passed 661 local tests and Node 20/24 plus Postgres CI, then merged with two-parent commit `b9eeb16` after Aryan explicitly said go. Initial deployment: `ee7dba58-1720-478d-a612-d893dbab85ab`. Current production after the same-image restart test: `4a8e93d5-771c-486e-91f6-335839f05be4`, created October 12 at 01:02:10 UTC / 06:32:10 IST. Phase 3 rollback: the verified pre-release `6ddd1673` deployment above. Health is HTTP 200, `status=ok`, `staffHandoff=configured`; no settings or variables changed. English and Spanish live cards progressed through Awaiting staff acceptance, Accepted, In progress, Waiting for customer, restart/Refresh, and Closed. Each customer received one correctly localized acceptance notice; Refresh re-rendered the same cards after restart. Both customer threads archived and locked after `/done`. Login-to-old-SIGTERM was 4.0 seconds and the old copy logged `drain completed=true`. The Accept button is removed after acceptance and the database permits one notice per acceptance cycle; the reviewer accepted that tested protection with no live retest required. `phase3-production-smoke-oct12.json` preserves full REST snapshots, notices and smoke-window logs for REPORT 3 review. Aryan intentionally deleted the Closed cards after screenshots; that is cleanup, not failed delivery. The existing Spanish closing-feedback card remains English, and damaged-item tracking intake awaits Phase 4. REPORT 3 was graded PASS. +- Phase 4 is implemented on `fix/howto-routing-and-long-answers`, from main `8321439`: technical topics select retrieval lanes without asserting a failure or forcing a handoff; real failures and protected person-required paths remain. Triage has no separate topic-only escalation rule; tests preserve real model/reviewer handoffs. Offline-capability questions and diagnostic-log guides are not incident reports by themselves. Navigation/confusion is not a fault; conditional failure clauses are hypothetical, while an independent actual failure and protected person-required routes remain. +- Phase 4 answers get an approximately 1,500-character draft budget, brief supported parts in requested order and complete-item overflow clipping. Wrapped bullets/steps remain atomic; Source links and data-preservation warnings stay whole. An oversized first item gets an honest truncation note, not partial guidance. +- Phase 4 multipart planning retains a neutral search query for each distinct part (bounded to eight). More than three parts can select four complementary official pages within eight chunks and 9,000 evidence characters. Normal query/citation defaults remain; multipart citations can use up to four retrieved URLs on one Source line. Production's stored corpus and the eval's joint multipart corpus share coverage selection. Ordinary evidence concatenation is not newly sliced, preserving the earlier Help Center regression fix. +- Physical damaged/replacement intake asks for photos/video, private order email and delivery date, not tracking. Account/code/receipt changes retain their prior intake; delivered-but-missing parcels retain tracking intake. Private staff order retention is unchanged. +- Local Phase 4 suite after merging current main: 707 tests. The routing correction has six own English/Spanish navigation/hypothetical how-tos and six incident reports, plus clause-boundary and protected-lane regressions; seven groups failed before the fix. The October 11 comparison at `945773a` stopped after eight candidate replies while main completed 162; retain both as interrupted evidence, not a passed gate. The eight completed replies had no recorded execution errors or timeouts; a 109.9 s total is the sum of normally completed stages, not a timed-out request. No sleep transition or Node crash report was found near the interruption, but no exit status was retained, so the exact termination cause is unknown. `phase4-interruption-diagnosis-oct12.json` preserves all eight row/stage latencies and this uncertainty. +- **Fresh October 12 comparison completed, quality approval still held.** Five candidate cases ran serially first, without errors, in 66.197/79.940/34.697/56.338/53.643 s (all under 90 s). Main `b9eeb16` ran 02:25:04–03:09:18 UTC; candidate runtime `251d20e` ran 03:09:18–03:53:25 UTC. Both completed all 162 valid rows (81×2/concurrency three), with no logged execution errors, a normal scoring exit 1 and no termination signal. The supervisor exited 0. Shared environment/harness/lockfile and clean pinned commits were guarded; temporary caffeinate assertions were verified active. The inter-arm gap was 81 ms. Independent live source/provider variation remains; this is not a frozen-corpus or isolated-host causal benchmark. +- Mutually exclusive structural pass-both/flaky/fail-both: main **71/6/4**, candidate **70/5/6**. Nearest-rank total wall latency includes every row: main p50/p95 **50.601/79.179 s**, candidate **49.957/80.067 s**. Numerical gates pass (70 ≥ 69; p50 delta −0.644 s; p95 delta +0.888 s). This is not the requested quality pass: structural unwanted-handoff flags are **12 main / 15 candidate**, missing-handoff flags **2 / 2**. Candidate's missing flags are `gs-035` in both runs; no implementation is tuned from protected outputs. Duplicate source labels: 1 main / 0 candidate. No end-ellipsis/ellipsis-before-source pattern was found, but that is only an advisory clipping screen, not semantic proof. Six second reviews per arm; 18/162 matched replies were byte-identical. Full private outputs, per-stage percentiles and per-case escalation deltas remain outside Git in `holdout-phase4-main-oct12.json`, `holdout-phase4-branch-oct12.json`, `holdout-phase4-oct12-comparison-summary.json` and provenance. Independent factual/how-to/protected-escalation/clipping grading is required before any merge or deploy. +- Phase 4b is separately implemented in draft #43 (`fix/localized-closing-card`, `3413f48`): stored case language controls closing-card title, description, optional honest-review text and buttons for the 14 acceptance languages, with English fallback and unchanged IDs/URLs. Seven new tests had red-before proof; its independent suite is 668 and Node 20/24/Postgres CI is green. Spanish wording explicitly says marked resolved rather than claiming customer-confirmed resolution. Review and release remain pending after #42. +- Scoped Phase 5 preparation is local only at `3688d99` on #28's branch after merging main with `42c6d42`; 750 tests and isolated SQL pass. Its actual read-only private-destination gate **failed**. The first check hit a REST-only SDK availability limitation; Aryan explicitly approved its tested metadata hydration repair and one rerun. The rerun returned **`untrusted_role_can_view`** for the current-production-configured staff channel. Each check used six GETs and one single-member read, no gateway login or external write. Stop #28: no push, merge, deploy, permission/allowlist change or silent recheck. Preserve both private preflight JSONs and reconcile the audience with an administrator before new direction. #29/#30 remain untouched. + +**Deployment order:** gated/reviewed #42 → reviewed localized closing-card fix (Phase 4b) → reviewed/private-preflight #28 → reviewed #29 → reviewed #30. One deployment and its own smoke report at a time, 24 clean hours unless Aryan explicitly says go. Private diagnostics remain parked. N12 credential rotation remains deferred by Aryan (Oct 9), accepted risk. + +### Historical October 11 foundation and Phase 2 record - #33 and #32 merged into #26's foundation; #26 merged into main as `2150383`. Its pre-release suite was 626/626, with Node 20/24 and database CI green. - Initial release `565be7b7-66ab-48a9-97e5-2cd8dedbc48d` succeeded on October 10 at 13:20:42 UTC. Direct `node index.js` startup replaced the unset override/effective npm startup; draining went from unset to 100 s. `/health` and its default timeout are unchanged; no service variables changed. Auto-deploy remains disabled; this was an explicit deploy. @@ -18,11 +35,10 @@ The latest staged-release brief governs release order. #28–#30 remain unchange - All four Phase 2 synthetic posts have one observed bot reply. The extra long-guide reply exercises the 1,900-character limit: the body ends in an ellipsis and one complete Source line retains both Help Center URLs. Both handoff cards were screenshot-confirmed, then intentionally removed by Aryan; do not recreate them. Phone hidden on private intake, order retained by explicit approval. The public filing formatter hides both in an offline check, but no live GitHub draft was inspected or issue filed. No missing-env, boot, model, review, staff-delivery or unhandled errors, and no second looks, in the checked smoke window. Full replies and timing lines are retained outside Git in `phase2-production-smoke-oct11.json`. - Phase 2 quality exceptions: the first guide omitted documented language/deletion steps; the clipped guide is incomplete despite preserving citations. The suggested test title containing “privacy” changed the crash report's route to privacy; a synthetic body-only comparison routes app, and Phase 2 did not change that classifier. Replacement intake still asks for tracking. These are not a blanket quality pass or evidence that the redaction fixes caused a new routing regression. #35 received the requested Changes requested review and remains unmerged. -**Current staged release order:** Phase 2 → #41 live staff status (review first) → `fix/howto-not-escalated` (same-day 81×2/concurrency-3 main comparison and review) → #28 durable filing/private-destination checks (live read-only preflight and review) → #29 → #30. Each deploy has its own smoke report and 24 clean hours unless Aryan explicitly says go. #28–#30 receive only their separately authorized phase corrections; this #41 work does not modify them. Broader knowledge/access work remains after these reliability gates, with diagnostics still parked. ### Historical release plan (superseded by the staged order above) -The gate/comparison entries below are retained as history, not a claim that #26 is still undeployed or that the old two-week #29/#30 decision precedes the latest staged instructions. +The earlier entries below are historical evidence, not a claim that the foundation is still undeployed or authority to skip current gates. 1. Complete and report Aryan's four #24 smoke tests, their exact timing lines, second-look lines and errors, and the private staff card. Screenshots received on October 9 confirm the four replies and replacement card. Totals: deletion 24.760 s, replacement 30.631 s, Spanish 31.251 s, developer 53.540 s; no second looks or model/staff-delivery failures in that window. Functional checks pass, but do not convert this into a blanket answer-quality signoff: the Undo wording overpromises recovery and the replacement intake asks for tracking details. 2. Release approved #25 with a merge commit, deploy main, verify HTTP 200/status `ok`/staff handoff `configured`, and record a real successful rollback target. Read but do not change overlap/draining variables. Set only Healthcheck Path `/health`, leave timeout default, verify the health-gated deployment and measure new Discord login versus old SIGTERM. If overlap exceeds ten seconds, tell Aryan before any further deployments. @@ -42,9 +58,9 @@ Customer-impacting follow-ups take priority over deferred plumbing: developer pa ## Current milestone -**Workflow foundation plus the approved Phase 2 fixes are live at `8321439`; Report 1 passed and Report 2 awaits grading.** Live staff-card status and acceptance notices are implemented on the separate draft branch, not production. No GCP permissions or customer-content access have been granted. +**Foundation, approved Phase 2 fixes and #41 are live at `b9eeb16`.** REPORT 3 was graded PASS; the reviewer accepted button removal plus the tested one-notice database invariant without a live retest. Phase 4's fresh comparison is complete and meets numerical limits, but independent quality/escalation grading remains pending. #43 is draft; #28's real audience preflight is stopped. No GCP permissions or customer-content access have been granted. -The earlier #24/#25 smoke prerequisites are complete. Subsequent releases still require their own review, smoke report and rollback evidence; never redeploy over an unfinished reply. +The older #24/#25 prerequisites are complete. Each subsequent release still needs its own review, smoke report, clean interval and rollback evidence; never redeploy over an unfinished reply. ## Already working before this milestone @@ -74,9 +90,9 @@ These capabilities remain subject to answer-quality checks. Existing functionali - [x] Add PR checks with no provider/Discord secrets, and a disposable Postgres integration job. - [x] Complete hosted CI for runtime head `3960f24`: rules on Node 20/24 and the disposable Postgres job pass. - [x] Run a focused three-run communication evaluation: eight cases meet the pass threshold, one is flaky, none fail the threshold. This is not the 81-case held-out gate or a production smoke test. -- [x] Deploy #26/#33/#32 and record private delivery, two in-flight drains, overlap and rollback evidence; Report 1 graded PASS, with the quality exceptions above. Approved Phase 2 fixes are subsequently deployed and reported separately. +- [x] Deploy the foundation and approved Phase 2 fixes; record real private delivery, two in-flight restart/drain checks, overlap and rollback evidence. Reports 1 and 2 passed; recorded answer-quality exceptions remain. -### Requested live staff-card follow-through — draft, not live +### Live staff-card follow-through — #41 deployed; REPORT 3 passed Validation on October 11: 647 unit/regression tests pass, offline customer evaluation 8/8, and the isolated SQL check passes with an old-schema upgrade, concurrent notice claims and card/close/reopen synchronization. The first hosted Postgres job caught a real microsecond-watermark bug that the initial millisecond-resolution fixture missed. It was reproduced with a sub-millisecond timestamp and corrected by stamping the exact server timestamp only for the rendered projection; a newer status remains pending. Hosted PR CI on the corrected head and the protected customer-wording gate are separate checks, not implied by local results. @@ -87,7 +103,7 @@ Phase 3 correction: Phase 2 main merged into #41 with a merge commit. An unconfi - [x] Preserve the first accepting staff member per acceptance cycle, reset acceptance/work state on customer-authorized reopening, and separate the suggested specialist from the actual acceptor. - [x] Send one localized, no-ping acceptance update in the bound customer thread (or verified original-message reply). No ETA, private intake, backend check or resolution claim. Cross-copy claims and a per-cycle nonce prevent repeated Accept clicks from repeating the notice. - [x] On an unconfirmed notice, keep the real acceptance and tell staff to follow up via Jump; do not blindly replay an uncertain send or claim the customer was notified. A crash between the durable claim and Discord delivery can leave this optional notice unconfirmed; staff follow-up is the fallback, not an exactly-once guarantee. -- [ ] Send #41 for review before merge; run any required wording gate, then smoke-test accept/double-accept/start/wait/refresh/restart/close in the separately approved rollout. The latest staged order places #41 before #28; #28's audience validation is a later gate, not claimed as present here. No competing audience checker or new backend access was added. The protected file has not been opened or rerun for this feature. +- [x] #41 reviewed, merged and deployed; REPORT 3 graded PASS. English/Spanish notices, work/wait, restart/Refresh and `/done` were observed live. The reviewer accepted button removal and tested notice uniqueness without a live duplicate-click retest. No new backend access was added. ## Milestone 2 Complete case handling and delivery recovery diff --git a/answerPipeline.js b/answerPipeline.js index 8d7a454..8231a91 100644 --- a/answerPipeline.js +++ b/answerPipeline.js @@ -3,7 +3,6 @@ const { stripFalseCertainty, stripPingNarration, formatDiscordReply, - clipForDiscord, DISCORD_REPLY_MAX, } = require('./utils'); const { stripHowtoBleed, stripShopBleed, stripUnsupportedClaims } = require('./honesty'); @@ -47,18 +46,71 @@ const TRAILING_SOURCE = new RegExp( 'iu' ); -// Clip the body and keep the source line whole. A link cut short opens nothing. +// Reply items are atomic: a wrapped bullet is one item, not a series of lines. +// Prose can stop at a complete sentence; an oversized first list item cannot. +// Keep this separate from the raw question/title clipper used elsewhere. +function clipCompleteReplyUnits(answer, max) { + const text = String(answer || '').trim(); + if (text.length <= max) return text; + const bodyBudget = Math.max(0, max - 2); // newline plus the truncation marker + const units = []; + let current; + let offset = 0; + const finish = () => { + if (current) units.push(current); + current = undefined; + }; + for (const line of text.split('\n')) { + const listItem = /^\s*(?:[-*+•]|\d+[.)])\s+\S/u.test(line); + if (!line.trim()) { + finish(); + } else if (listItem || !current) { + finish(); + current = { start: offset, end: offset + line.length, listItem }; + } else { + current.end = offset + line.length; + } + offset += line.length + 1; + } + finish(); + + let end = 0; + for (const unit of units) { + if (unit.end <= bodyBudget) { + end = unit.end; + continue; + } + if (!unit.listItem) { + const prefix = text.slice(unit.start, bodyBudget); + for (const sentenceEnd of prefix.matchAll(/[.!?。!?]["'”’\])]*(?=\s|$)/gu)) { + const candidate = unit.start + sentenceEnd.index + sentenceEnd[0].length; + // The prefix boundary itself is not a sentence boundary if the next + // original character continues a URL, word or punctuation group. + if (!text[candidate] || /\s/u.test(text[candidate])) end = candidate; + } + } + break; + } + const body = text.slice(0, end).trim(); + if (body) return `${body}\n…`; + const note = 'This reply was too long to display in full.'; + return note.length <= max ? note : ''; +} + +// Clip at a complete reply item and keep the source line whole. A link cut +// short opens nothing, including on multi-part answers with several sources. function clipKeepingSource(answer, max = DISCORD_REPLY_MAX) { const text = String(answer || '').trim(); if (text.length <= max) return text; const sourceAt = text.search(TRAILING_SOURCE); - if (sourceAt < 0) return clipForDiscord(text, max); + if (sourceAt < 0) return clipCompleteReplyUnits(text, max); const tail = text.slice(sourceAt); const gap = /^\n\s*\n/.test(tail) ? '\n\n' : '\n'; const sourceLine = tail.trim(); const bodyBudget = max - sourceLine.length - gap.length; - if (bodyBudget < 200) return clipForDiscord(text, max); - return `${clipForDiscord(text.slice(0, sourceAt), bodyBudget)}${gap}${sourceLine}`; + if (bodyBudget < 0) return EMPTY_ANSWER_FALLBACK; // never publish a severed source URL + const body = clipCompleteReplyUnits(text.slice(0, sourceAt), bodyBudget); + return body ? `${body}${gap}${sourceLine}` : sourceLine; } function presentReviewedAnswer(answer) { @@ -107,7 +159,7 @@ function addUnsyncedDataWarning(answer, question, { dataLossRisk = false, langua const sourceLine = sourceAt < 0 ? '' : current.slice(sourceAt).trim(); const body = sourceAt < 0 ? current : current.slice(0, sourceAt).trim(); const bodyBudget = Math.max(200, 1_750 - warning.length - sourceLine.length - 4); - return [clipForDiscord(body, bodyBudget), warning, sourceLine].filter(Boolean).join('\n\n'); + return [clipCompleteReplyUnits(body, bodyBudget), warning, sourceLine].filter(Boolean).join('\n\n'); } module.exports = { diff --git a/commandcode.js b/commandcode.js index 7176463..7c32a28 100644 --- a/commandcode.js +++ b/commandcode.js @@ -4,6 +4,7 @@ const { stripStaffLies } = require('./honesty'); const { EMPTY_ANSWER_FALLBACK, UNSYNCED_DATA_WARNING, hasUnsyncedDataRisk } = require('./answerPipeline'); const { isInstructionSentence, SOURCE_LABEL_NAME, SOURCE_LABEL_SEPARATOR } = require('./supportSteps'); const { redactSensitive } = require('./privacy'); +const { normalizeSubQuestions } = require('./retrieval'); const PLANNED_HANDOFF_GUIDANCE = 'A person from the Omi team will reply in this thread if staff delivery succeeds. The application adds that line after delivery; do not repeat or claim it already happened. Don\'t tell the customer to contact support, email, use a contact form or post in another channel. Give relevant official steps and ask only for missing details.'; @@ -143,33 +144,48 @@ function parseSearchPlan(raw, question) { .replace(/\s+/g, ' ') .trim() .slice(0, 500); - const queries = []; - const seen = new Set(); - for (const value of Array.isArray(data.search_queries) ? data.search_queries : []) { - const query = String(value || '').replace(/\s+/g, ' ').trim().slice(0, 180); - const key = query.toLowerCase(); - if (!query || seen.has(key)) continue; - seen.add(key); - queries.push(query); - if (queries.length >= 4) break; - } const shortList = (value, limit = 5) => { const out = []; for (const item of Array.isArray(value) ? value : []) { const text = String(item || '').replace(/\s+/g, ' ').trim().slice(0, 240); - if (!text || out.includes(text)) continue; + if (!text || out.some((prior) => prior.toLowerCase() === text.toLowerCase())) continue; out.push(text); if (out.length >= limit) break; } return out; }; + const mustAnswer = shortList(data.must_answer, 8); + let subQuestions = normalizeSubQuestions(data.sub_questions); + // Older planner outputs remain valid. On a genuinely multipart request, + // missing queries fall back to the neutral requirement, never a guessed fact. + if (!subQuestions.length && mustAnswer.length > 3) { + subQuestions = normalizeSubQuestions(mustAnswer.map((part, index) => ({ + question: part, query: Array.isArray(data.search_queries) ? data.search_queries[index] : part, + }))); + } + const multipart = subQuestions.length > 3; + const queries = []; + const seen = new Set(); + for (const value of [ + ...subQuestions.map((part) => part.query), + ...(Array.isArray(data.search_queries) ? data.search_queries : []), + ]) { + if (typeof value !== 'string') continue; + const query = value.replace(/\s+/g, ' ').trim().slice(0, 180); + const key = query.toLowerCase(); + if (!query || seen.has(key)) continue; + seen.add(key); + queries.push(query); + if (queries.length >= (multipart ? 8 : 4)) break; + } return { standaloneQuestion: standaloneQuestion || String(question || '').trim(), queries, device: String(data.device || '').trim().slice(0, 80), topic: String(data.topic || '').trim().slice(0, 80), customerGoal: String(data.customer_goal || '').replace(/\s+/g, ' ').trim().slice(0, 500), - mustAnswer: shortList(data.must_answer, 5), + mustAnswer: subQuestions.length ? subQuestions.map((part) => part.question) : mustAnswer, + subQuestions, customerFacts: shortList(data.customer_facts, 6), supportKind: String(data.support_kind || 'other').trim().slice(0, 60), wantsPerson: data.wants_person === true, @@ -366,11 +382,11 @@ function trailingSourceLabel(line) { : ''; } -function groundedSourceLine(answer, sources, sourceIds, language = 'en') { +function groundedSourceLine(answer, sources, sourceIds, language = 'en', { subQuestions = [] } = {}) { const urls = evidenceUrls(sources); const chosen = [ ...new Set((sourceIds || []).map((id) => urls.get(String(id))).filter(Boolean)), - ].slice(0, 2); + ].slice(0, normalizeSubQuestions(subQuestions).length > 3 ? 4 : 2); const text = String(answer || '').split('\n').map(unwrapEmphasis).join('\n'); const modelLabel = text.split('\n').map((line) => { const citation = sourceCitationSuffix(line); @@ -431,6 +447,10 @@ async function understandQuestion({ question, threadHistory = [], route, onUsage role: 'system', content: 'Include data_loss_risk as a boolean in the JSON. Set it true only when the customer says audio, a recording, or a transcript from a session they made is missing, empty, blank, lost, or not syncing. Do not set it for a general how-to about finding, exporting, or turning off recordings. When true, the case needs a person; do not suggest reinstalling, logging out, or clearing local storage.', }, + { + role: 'system', + content: 'Multipart search coverage: represent every distinct sub-question in sub_questions as {"question":"neutral requirement","query":"short English official-search wording"}, in the order asked. Produce one search query for each distinct sub-question, without adding an expected answer or diagnosis. Use at most eight distinct parts; group closely related requirements only when necessary. For more than three parts, one query per part takes priority over generic search variants; otherwise keep two to four short searches. Include these pairs alongside the existing JSON fields.', + }, { role: 'user', content: `Route: ${route?.lane || 'unknown'} / ${route?.area || 'unknown'}\nEarlier thread:\n${history || '(none)'}\n\nCustomer question:\n<<>>`, @@ -476,7 +496,7 @@ async function queryAgent({ temperature: 0.4, messages: [ { role: 'system', content: `${buildSystemPrompt(route)}${handoffPlanned || route?.escalate ? `\n\n${PLANNED_HANDOFF_GUIDANCE}` : ''}` }, - { role: 'system', content: 'Write the customer reply entirely in the language and script of the latest customer message. If it is romanized Hindi, use Latin letters rather than Devanagari. A person-needed request can still receive an official, sourced policy or how-to answer; keep escalation for the action or decision and do not promise an outcome. If understanding.dataLossRisk is true, warn in the customer language not to reinstall, log out, or clear local recordings before staff checks; do not append an English warning to a non-English reply.' }, + { role: 'system', content: 'Write the customer reply entirely in the language and script of the latest customer message. If it is romanized Hindi, use Latin letters rather than Devanagari. A person-needed request can still receive an official, sourced policy or how-to answer; keep escalation for the action or decision and do not promise an outcome. If understanding.dataLossRisk is true, warn in the customer language not to reinstall, log out, or clear local recordings before staff checks; do not append an English warning to a non-English reply. final_answer has a hard budget of 1,500 characters including whitespace and any inline links. For multi-part questions, answer the parts the sources support briefly in the order asked, rather than writing a guide. Use one short, complete bullet per part; combine closely related steps and omit extra background before dropping a requested part. State unsupported parts briefly without guessing. Keep necessary safety warnings and material uncertainty.' }, { role: 'user', content: buildUserPrompt({ @@ -602,6 +622,7 @@ async function reviewAnswer({ question, threadHistory = [], draft, removedBySafe }; } const send = post || axios.post.bind(axios); + const citationLimit = normalizeSubQuestions(understanding?.subQuestions).length > 3 ? 4 : 2; let data; for (let attempt = 0; attempt < 2; attempt += 1) { const compact = attempt === 1; @@ -622,6 +643,10 @@ async function reviewAnswer({ question, threadHistory = [], draft, removedBySafe role: 'system', content: `Clarification: the ban on steps from static fallback applies only to troubleshooting a tech or firmware symptom. For a how-to FAQ, the Omi team static product facts are usable below the Help Center and docs when not contradicted; do not invent a source URL. For refund, billing, subscription, account, and data requests, answer any supported policy or procedure from official pages while keeping escalation for the human action. General policy pages cannot verify a specific order status, location, tracking, or delivery date; remove or reject any such claim unless a verified order lookup tool fact supports it. If understanding.dataLossRisk is true, retain a brief warning in the customer language not to reinstall, log out, or clear local recordings while staff checks; do not add an English warning to a non-English reply. Use the customer language and script throughout, including a localized source label; romanized Hindi stays romanized. Do not append an English Source line to an existing localized citation.${handoffPlanned ? ` ${PLANNED_HANDOFF_GUIDANCE}` : ''}`, }, + ...(citationLimit === 4 ? [{ + role: 'system', + content: 'This request contains more than three distinct sub-questions. Its one Source line may contain at most four exact URLs from the retrieved evidence, instead of the ordinary two, when needed to support the requested parts. Keep each supported part brief and in the order asked. Keep the answer body within 1,500 characters without removing material safety guidance or uncertainty.', + }] : []), { role: 'user', content: `Earlier thread:\n${modelThread(compact ? threadHistory.slice(-3) : threadHistory) @@ -662,7 +687,7 @@ async function reviewAnswer({ question, threadHistory = [], draft, removedBySafe ? '' : safety.fallback ? safety.answer - : groundedSourceLine(stripStaffLies(safety.answer), sources, sourceIds, understanding?.replyLanguage); + : groundedSourceLine(stripStaffLies(safety.answer), sources, sourceIds, understanding?.replyLanguage, { subQuestions: understanding?.subQuestions }); const hasAnswer = Boolean(String(answer || '').trim()); return { final_answer: answer, diff --git a/docs.js b/docs.js index 66bd1f0..d2b2c47 100644 --- a/docs.js +++ b/docs.js @@ -11,6 +11,7 @@ const { isSingleItemDeletionQuestion, isDeveloperIntent, isDeveloperPageCompatibleQuestion, + normalizeSubQuestions, } = require('./retrieval'); let indexCache = { at: 0, text: '' }; let indexLookup = null; @@ -116,10 +117,10 @@ async function rememberPages(store, pages) { })); } -async function storedDocs(question, store, plannedQueries = []) { +async function storedDocs(question, store, plannedQueries = [], { subQuestions = [] } = {}) { if (!store?.searchDocPages) return ''; try { - const queries = supportQueries(question, plannedQueries); + const queries = supportQueries(question, plannedQueries, { subQuestions }); const batches = await Promise.all( queries.map((query) => Promise.all([ @@ -133,7 +134,12 @@ async function storedDocs(question, store, plannedQueries = []) { const helpSets = batches.map((batch) => batch[1].filter((row) => compatibleDevicePage(question, row))); const githubSets = batches.map((batch) => batch[2].filter((row) => compatibleDevicePage(question, row))); const developerQuestion = isDeveloperIntent(question); - const ranked = mergeRanked([...allSets, ...helpSets, ...githubSets], developerQuestion ? 24 : 12); + const multipart = normalizeSubQuestions(subQuestions).length > 3; + const resultSets = [...allSets, ...helpSets, ...githubSets]; + // All candidate rows were already fetched under the existing per-search + // limits. Multipart selection happens before the eight-chunk output cap. + const candidateLimit = multipart ? resultSets.reduce((count, rows) => count + rows.length, 0) : developerQuestion ? 24 : 12; + const ranked = mergeRanked(resultSets, candidateLimit); if (developerQuestion) { const developerPage = (row) => /^https:\/\/docs\.omi\.me\/(?:api-reference|docs?\/developer)\//i.test(String(row.url || '')); ranked.sort((a, b) => Number(developerPage(b)) - Number(developerPage(a)) || b.fusedScore - a.fusedScore); @@ -148,18 +154,18 @@ async function storedDocs(question, store, plannedQueries = []) { rows.push(row); if (rows.length >= 8) break; } - return formatEvidence(rows); + return formatEvidence(multipart ? rankLocalChunks(queries, ranked, 8, { customerQuestion: question, subQuestions }) : rows); } catch (err) { console.error('[Docs] stored lookup failed:', err.message); return ''; } } -async function relevantDocs(question, { fetchImpl, store, queries: plannedQueries = [] } = {}) { +async function relevantDocs(question, { fetchImpl, store, queries: plannedQueries = [], subQuestions = [] } = {}) { const fetchFn = fetchImpl || fetch; const saved = activeStore(store); - const queries = supportQueries(question, plannedQueries); - const stored = await storedDocs(question, saved, plannedQueries); + const queries = supportQueries(question, plannedQueries, { subQuestions }); + const stored = await storedDocs(question, saved, plannedQueries, { subQuestions }); if (stored) return stored; try { const indexText = await docsIndexText(fetchFn); @@ -167,6 +173,17 @@ async function relevantDocs(question, { fetchImpl, store, queries: plannedQuerie const allPages = pagesFromIndex(indexText); const picked = []; const seen = new Set(); + const parts = normalizeSubQuestions(subQuestions); + // Give each part a chance before a broad overview consumes six fetch slots. + if (parts.length > 3) { + for (const part of parts) { + const page = topPages(part.query, allPages, 1, question)[0]; + if (!page || seen.has(page.url)) continue; + seen.add(page.url); + picked.push(page); + if (picked.length >= 4) break; + } + } for (const query of queries) { for (const page of topPages(query, allPages, 3, question)) { if (seen.has(page.url)) continue; @@ -191,7 +208,7 @@ async function relevantDocs(question, { fetchImpl, store, queries: plannedQuerie const pages = fetched.filter(Boolean); const chunks = pages.flatMap((page) => chunkDocument(page)); await rememberPages(saved, pages); - return formatEvidence(rankLocalChunks(queries, chunks, 8, { customerQuestion: question })); + return formatEvidence(rankLocalChunks(queries, chunks, 8, { customerQuestion: question, subQuestions })); } catch (err) { console.error('[Docs] lookup failed:', err.message); return ''; diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 39afe99..c5d9316 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,6 +1,8 @@ # Omi Support Bot Architecture -Omi Support Bot solves documented problems and routes unresolved cases toward a person. Customer reassurance must describe actions and evidence the system can verify. The workflow foundation and approved Phase 2 fixes are deployed at main `8321439`; Report 1 passed and the Phase 2 smoke/report evidence is recorded in [the roadmap](../ROADMAP.md). Live staff-card status/acceptance notices described below are draft-branch work, not newly deployed behavior. The recording-diagnostics gateway remains a target design, not newly granted access. The existing official-source answer and review pipeline remains in place. +Omi Support Bot solves documented problems and routes unresolved cases toward a person. Reassurance describes verified actions and evidence. The foundation, approved Phase 2 fixes and #41 are live at main `b9eeb16`; reports 1–3 passed. Phase 4 is implemented on its review branch, not deployed. [The roadmap](../ROADMAP.md) separates implementation, validation and release. Recording diagnostics remain a target design, not granted access. + +The fresh Phase 4 comparison is complete and meets numerical latency/structural-score limits, not the independent escalation/answer-quality gate. More unwanted-handoff flags require review before shipping. Closing-card localization is a separate draft #43, not part of this runtime snapshot. #28's local audience checker rejected the real configured staff destination (`untrusted_role_can_view`); its optional engineering workflow is not released, and no permissions/allowlists were changed to make the check pass. ## Current capabilities and gaps @@ -68,6 +70,10 @@ For later log correlation, a dedicated gateway identity can read only an approve Use the current question plus relevant case context. Retrieve official evidence, answer the actual goal, and retain the review and source formatting checks. Preserve what the customer tried; do not repeat failed steps. Pure thanks receive no reply. Clarify only when the missing detail changes the answer. +Phase 4 separates technical topics from reported failures. APP/DESKTOP/FIRMWARE still select their retrieval lane, but a deterministic handoff requires an observed failure or an independently protected person-required signal. Capability questions and diagnostic-guide topics are not failures by themselves; a real fault inside a question still is. Model/reviewer rejection, human requests and data-loss signals remain independent handoff grounds, not suppressed by a how-to label. Navigation/confusion predicates are removed as failure signals, not the rest of the message. Conditional clauses do not assert a fault; independent main-clause failures still do, including when the customer omits a comma. + +Multipart planning retains bounded neutral `{question, query}` pairs in the requested order. With more than three distinct parts, retrieval selects up to four complementary pages, without increasing the eight-chunk/9,000-character output budget. Stored and in-memory evaluation corpora use the same coverage selector before formatting; the ordinary concatenation path remains untouched. Grounded multipart citations may include four exact retrieved URLs, while ordinary replies retain two. The answer model gets an approximately 1,500-character budget and concise supported coverage. Overflow presentation keeps complete bullets/steps or prose sentences plus the intact Source line; data-loss warnings use the same boundary and are never shortened away. + ### Missing recordings Acknowledge the impact and immediately protect potentially unsynced data. Provide a private verification step, obtain diagnostic consent, and inspect the permitted pipeline metadata. Explain observed facts without guessing root cause or promising recovery. If unresolved, deliver one staff case containing the diagnostic timeline, attempted steps and missing information. Logs can locate a failure stage; they cannot recreate absent audio. @@ -76,6 +82,8 @@ Acknowledge the impact and immediately protect potentially unsynced data. Provid Use the exact verified order privately when available. Refund, replacement, billing, account and privacy decisions remain with staff. Give safe documented steps where applicable and one next step. Once staff delivery succeeds, do not also tell the customer to email another channel. If delivery fails, use the existing honest fallback. +Phase 4 damaged-item/replacement intake requests photos or video, the order email privately and the delivery date. This requires a physical-item damage/replacement signal, not replacing an account field, receipt or redemption code. Ordinary missing-delivery intake keeps its tracking details; private order retention and public redaction boundaries are unchanged. + ### Human handoff and closure The current case states are `queued`, `delivered`, `accepted`, `resolved`, and `closed`, with an explicit reopening transition. An authenticated staff acceptance records its actor and time; delivery alone does not count. Delivery failure is an attempt outcome, not resolution. Staff get the current intake and a link to the conversation; a retained cumulative summary and a distinct `awaiting_customer` state are follow-up work. Reminders and target response times belong to a real staffed queue; do not publish an SLA before an owner commits to it. diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md index 82ac1db..1c1abe5 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/IMPLEMENTATION_PLAN.md @@ -1,6 +1,22 @@ # Omi Support Bot Implementation Checklist -Implement the [architecture](ARCHITECTURE.md) incrementally, retaining the working answer/review, official retrieval, public commands and private staff delivery. [The roadmap](../ROADMAP.md) is the authoritative progress record. The foundation and approved Phase 2 fixes are deployed at `8321439`; Report 1 passed and Report 2 awaits grading. Live staff-card status and acceptance notices are implemented only on a separate draft branch. Each runtime change needs regression checks and release verification. No cloud access grants accompany this foundation. +Implement the [architecture](ARCHITECTURE.md) incrementally, retaining the answer/review, official retrieval, public commands and private delivery. [The roadmap](../ROADMAP.md) is authoritative. The foundation, Phase 2 and #41 are live at `b9eeb16`; reports 1–3 passed. Phase 4 remains a review branch. No cloud access grants accompany this work. + +## Current staged release checklist + +- [x] Phase 2 release and REPORT 2 graded PASS; preserve the clipping, retrieval and synthetic-title limitations in the roadmap. +- [x] #41 review approved at `e5cff9e`, 661 tests and Node 20/24/Postgres CI green. +- [x] Release #41 after Aryan's explicit go; REPORT 3 graded PASS. Real English/Spanish accept, work/wait, restart/Refresh and closure passed. Button removal and tested database notice uniqueness were accepted without a live retest. +- [x] Phase 4 routing separates technical topics from failures, with eight own English/Spanish how-tos, eight failures and capability/diagnostic-guide edge checks; genuine reviewer/protected handoffs remain. +- [x] Phase 4 draft budget and complete-item clipping preserve full Source URLs and data-loss warnings. +- [x] Phase 4 multipart planning/retrieval/citation coverage uses up to four pages within existing evidence caps; production/eval joint coverage and unchanged ordinary evidence paths have offline fixtures. +- [x] Phase 4 replacement intake requests damage evidence, private order email and delivery date while preserving account and missing-delivery intake. +- [x] Current-main merge and structural navigation/conditional correction: 707 unit tests pass. Own six how-tos/six failures plus clause-boundary guards supplement the earlier red-before coverage. +- [x] Five serial candidate cases finish under 90 s, then both fresh same-day 81×2/concurrency-3 arms complete. Preserve private JSONs, all rows, percentiles, per-case escalation deltas and supervised exit/heartbeat provenance. Numerical limits pass; this is not independent answer-quality approval. +- [ ] Obtain independent #42 factual/how-to/protected-escalation/clipping grading. Unwanted handoff flags increased 12 → 15, missing flags remain 2 per arm; no code tuning from protected results and no merge/deploy while held. +- [x] Separate closing-card localization draft #43 uses stored case language with English fallback and unchanged feedback/link semantics; 668 tests and hosted CI pass. Review/release smoke remain pending after #42. +- [ ] Resolve #28's actual `untrusted_role_can_view` audience stop before another approved preflight/review/merge. Its scoped corrections are retained locally (750 tests and isolated SQL pass), not pushed/deployed. No server permission change; #29/#30 remain untouched. +- [ ] Review/deploy gated #42, then the separate localized closing-card PR, then #28, #29 and #30 in order, each with review, smoke and 24 clean hours. ## Release and security prerequisites @@ -18,8 +34,8 @@ Implement the [architecture](ARCHITECTURE.md) incrementally, retaining the worki - [x] Stop new work on shutdown; drain accepted work and queue unstarted requests. Coordinate overlapping copies through Postgres claims. Started requests with uncertain delivery remain an operator-reconciliation task. - [x] Make `/done`, ratings/reopening and Telegram delivery operate on the same case state. Failed customer delivery must not resolve a case. - [x] Record real staff acceptance through an authenticated private-card action; preserve the first accepting actor/time. -- [x] Draft: project accepted/work/waiting/resolved/closed state into the existing private cards; persist message references, serialize edits and recover missed updates after restart. Retire deleted cards without reposting. -- [x] Draft: send one localized customer-thread acceptance notice naming the actual staff member; retain uncertainty and direct staff to Jump on notification failure. Clear prior ownership on a new reopened acceptance cycle. +- [x] Live #41: project accepted/work/waiting/resolved/closed state into the existing private cards; persist message references, serialize edits and recover missed updates after restart. Retire deleted cards without reposting. +- [x] Live #41: send one localized customer-thread acceptance notice naming the actual staff member; retain uncertainty and direct staff to Jump on notification failure. Clear prior ownership on a new reopened acceptance cycle. - [x] Phase 3 correction: every unconfirmed acceptance notification explicitly tells staff to reply in the customer's thread, in an ephemeral response. Confirmed boot/30-second sweep wiring and stale-card repair after a failed edit; Phase 2 main merged without a rebase. Combined unit suite 661 pass, offline 8/8. - [ ] Review and run the full protected gate for the new canned acceptance wording, then smoke-test real accept/start/wait/close/reopen and restart recovery. No deployment or protected-file read accompanies this implementation. - [ ] Persist staff-approved issue drafts and action results so a restart does not invalidate the File workflow. Decide whether approved customer updates should reach linked GitHub issues; do not document an inactive path as working. @@ -35,7 +51,7 @@ Acceptance: restart during OTP, overlapping copies, provider slowdown, failed/un - [ ] For each new adapter, define a fixed allowlist, customer/case authorization where needed, consent, limits, kill switch and metadata-only audit; validate synthetically, then pilot in the private staff card before reviewed customer enablement. - [ ] Extend order facts only through exact-owner-verified private/ephemeral delivery, never a normal customer-thread message or model prompt. Keep existing `/order` self-service enabled; do not add order/account writes. - [ ] Obtain backend-team agreement and privacy review before linked-account metadata/sign-in implementation. Define available/stale/unknown states for device/version/sync/plan and any provable unsynced count; exclude content and unrelated customers. -- [ ] Follow the latest staged order: Phase 2 → reviewed #41 → routing/intake comparison → reviewed/private-preflight #28 → reviewed #29 → reviewed #30. The previous two-week go/no-go order is superseded, not a reason to bypass review. Keep deployments separate, with 24 clean hours unless Aryan explicitly says go; add only the authorized phase corrections. Other open queue/monitoring/retention work remains deferred. +- [ ] Latest order: gated #42 → localized closing card → reviewed/private-preflight #28 → reviewed #29 → reviewed #30. Each release stays separate with 24 clean hours unless Aryan says go. No unrelated feature additions. ## Phase 2 Verified metadata diagnostics diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index cec5f36..022597c 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -2,6 +2,8 @@ Operate the capabilities that are actually enabled. The [roadmap](../ROADMAP.md) distinguishes implemented, validated and deployed work. Never paste credentials, verification codes, customer messages, private identifiers or raw backend logs into public issues or operational chat. +October 12 release holds: #42's fresh comparison is complete and meets numerical limits, but factual/how-to/protected-escalation/clipping review is pending; do not approve it from pass-both or latency alone. #43 closing-card localization is a separate draft. #28's actual read-only preflight returned `untrusted_role_can_view`, so stop its rollout and obtain administrator/direction review of the real audience; never add unverified roles, change permissions or silently repeat a failed private-destination check. No deployment accompanied these checks. + ## Startup and deployment Production uses the existing `DATABASE_URL` for support state as well as knowledge. Startup applies additive, repeatable schemas before Discord login. If database initialization fails, the process must not start answering with a memory-only coordination or identity store. Development/evaluation without `DATABASE_URL` uses isolated in-memory stores. @@ -10,6 +12,8 @@ Keep one primary running service. Shared claims and worker slots protect overlap Before deploying, pass PR tests and relevant answer-quality checks, finish the existing smoke-test prerequisite, and record the current successful deployment as the rollback target. Verify the new deployment, health endpoint, Discord login, staff receipt and actual customer replies. Configuration presence is not a staff-delivery test. Do not change unrelated Railway variables or cloud permissions during rollout. +Remaining staged order is gated #42 → localized closing-card fix → #28 → #29 → #30, with one smoke report and 24 clean hours between releases unless Aryan explicitly says go. Aryan granted that exception for #41, which is live at main `b9eeb16`. Current deployment after the required same-image restart test is `4a8e93d5-771c-486e-91f6-335839f05be4`, created October 12 at 01:02:10 UTC / 06:32:10 IST; Phase 3's rollback target is `6ddd1673-2e45-4cac-9d0d-28c71a9107fd`. REPORT 3 was graded PASS; the reviewer accepted button removal plus tested database notice uniqueness without a live retest. The next deployment needs its own gates and 24 clean hours from this latest restart unless Aryan says go again. Verify the active deployment and interval before dispatch; never use a historical release's hard-coded rollback helper for a later release. No automatic rollout is booked. + ## Customer verification Order links are customer-owned, encrypted in Postgres, and expire after 30 days. Codes expire after ten minutes by default, have five attempts, and are limited to three sends per hour per Discord-user/email pair. Those limits survive restart. `/unlink` revokes the binding and pending challenge, even if Shopify/email delivery is unavailable; it does not reset the anti-abuse limit. diff --git a/handoff.js b/handoff.js index ae7d02f..6ff516e 100644 --- a/handoff.js +++ b/handoff.js @@ -413,11 +413,23 @@ function staffDetailsRequest(route, question) { if (route.area === 'privacy' || route.lane === 'privacy') { return 'If they did not provide the account email, ask for it privately.'; } + if (route.lane === 'account') { + return 'If they did not provide the purchase email, ask for it privately.'; + } + const physicalItem = '(?:omi|devices?|units?|hardware|casing|housing|necklaces?|glasses|chargers?|wearables?|items?)'; + const itemWords = '(?:(?:my|our|your|the|this|a|an|new|omi|damaged|defective|broken)\\s+){0,3}'; + const damagedItem = new RegExp(`\\b${physicalItem}\\b`, 'i').test(text) && + /\b(?:damag(?:e|ed)|defect(?:ive)?|broken|crack(?:ed)?|faulty)\b/i.test(text); + const replacingItem = new RegExp(`\\b(?:replac(?:e|ing)\\s+${itemWords}${physicalItem}|replacement\\s+(?:(?:for|of)\\s+)?${itemWords}${physicalItem}|${physicalItem}\\s+(?:needs?|requires?)\\s+(?:a\\s+)?replacement)\\b`, 'i').test(text); + if ((route.area === 'shop' || ['shop', 'money'].includes(route.lane)) && + (damagedItem || replacingItem)) { + return 'If they did not provide photos or a video of the damage, the order email and the delivery date, ask for them. Ask for the order email privately.'; + } if (/\b(?:bulk|wholesale|sales|business|enterprise|company|corporate|\d+\s+(?:units|devices|omis))\b/i.test(text) && (route.area === 'shop' || route.lane === 'shop')) { return 'If they did not provide the company and quantity, ask for both.'; } - if (route.lane === 'money' || route.lane === 'account' || + if (route.lane === 'money' || /\b(?:refund|charg(?:e|ed)|bill(?:ing)?|payment|purchase|subscription|invoice)\b/i.test(text)) { return 'If they did not provide the purchase email, ask for it privately.'; } diff --git a/index.js b/index.js index 510543b..73c39e9 100644 --- a/index.js +++ b/index.js @@ -770,7 +770,7 @@ async function answerMessage(message, { directHistory = [] } = {}) { ); const retrievalStart = performance.now(); const [docsText, officialCodeText, feedbackText, releaseText] = await Promise.all([ - relevantDocs(sourceQuestion, { queries: searchPlan.queries }), + relevantDocs(sourceQuestion, { queries: searchPlan.queries, subQuestions: searchPlan.subQuestions }), route.area === 'shop' ? '' : github.searchOfficialCode(sourceQuestion, { queries: searchPlan.queries }), router.isTechLane(route) || route.lane === 'unknown' ? relevantFeedback(sourceQuestion, { queries: searchPlan.queries }) diff --git a/retrieval.js b/retrieval.js index 949fad2..3981068 100644 --- a/retrieval.js +++ b/retrieval.js @@ -190,7 +190,7 @@ function queryTerms(text, limit = 14) { return terms; } -function uniqueQueries(question, planned = []) { +function uniqueQueries(question, planned = [], limit = 5) { const seen = new Set(); const out = []; for (const value of [question, ...(planned || [])]) { @@ -199,7 +199,7 @@ function uniqueQueries(question, planned = []) { if (!clean || seen.has(key)) continue; seen.add(key); out.push(clean); - if (out.length >= 5) break; + if (out.length >= limit) break; } return out; } @@ -222,7 +222,27 @@ function isDeveloperPageCompatibleQuestion(question) { /\b(?:firmware|flash\w*|dev\s*kit|devkit|developer|compil\w*|source|repos?itor\w*|repos?|github|npm|flutter)\b/i.test(String(question || '')); } -function supportQueries(question, planned = []) { +function normalizeSubQuestions(value) { + const seen = new Set(); + const parts = []; + for (const item of Array.isArray(value) ? value : []) { + const rawQuestion = typeof item === 'string' ? item : item?.question; + if (typeof rawQuestion !== 'string') continue; + const question = rawQuestion + .replace(/\s+/g, ' ').trim().slice(0, 240); + const key = question.toLowerCase(); + if (!question || seen.has(key)) continue; + const rawQuery = typeof item === 'string' ? item : item?.query ?? item?.search_query; + const query = (typeof rawQuery === 'string' ? rawQuery : question) + .replace(/\s+/g, ' ').trim().slice(0, 180) || question.slice(0, 180); + seen.add(key); + parts.push({ question, query }); + if (parts.length >= 8) break; + } + return parts; +} + +function supportQueries(question, planned = [], { subQuestions = [] } = {}) { const text = [question, ...(planned || [])].filter(Boolean).join(' '); const variants = []; if (isDeveloperIntent(question) && /\bapps?\b/i.test(question) && /\bomi\b/i.test(question)) { @@ -249,7 +269,68 @@ function supportQueries(question, planned = []) { 'delete conversation transcript cloud phone local synced recording copies Offline Sync Manage Storage' ); } - return uniqueQueries(question, [...variants, ...(planned || [])]); + const parts = normalizeSubQuestions(subQuestions); + // Preserve every interpreted part before optional synonyms or generic searches. + // Ordinary requests keep the existing five-query budget. Multipart requests + // have at most eight part queries plus the customer's original wording. + return parts.length > 3 + ? uniqueQueries(question, [...parts.map((part) => part.query), ...(planned || []), ...variants], 9) + : uniqueQueries(question, [...variants, ...(planned || [])]); +} + +function queryMatchScore(query, row) { + const terms = queryTerms(query, 20); + const title = String(row.title || '').toLowerCase(); + const body = String(row.body || row.blurb || '').toLowerCase(); + return terms.reduce((score, term) => score + (title.includes(term) ? 4 : body.includes(term) ? 1 : 0), 0); +} + +function selectSubQuestionEvidence(rows, subQuestions, limit = 8) { + const parts = normalizeSubQuestions(subQuestions); + if (parts.length <= 3) return rows.slice(0, limit); + const byPage = new Map(); + for (const [index, part] of parts.entries()) { + const matches = rows.map((row) => ({ row, score: queryMatchScore(part.query, row) })) + .filter((hit) => hit.score > 0) + .sort((a, b) => b.score - a.score || + sourceAuthority(b.row.source || sourceKind(b.row.url)) - sourceAuthority(a.row.source || sourceKind(a.row.url))); + const row = matches[0]?.row; + if (!row) continue; + const key = String(row.url || row.title || ''); + const page = byPage.get(key) || { key, firstPart: index, rows: [] }; + page.rows.push(row); + byPage.set(key, page); + } + // Prefer pages answering several parts, then preserve the requested order. + // One useful chunk per page goes first so the unchanged character cap cannot + // spend its budget on repeats before complementary guidance arrives. + const selectedPages = [...byPage.values()] + .sort((a, b) => b.rows.length - a.rows.length || a.firstPart - b.firstPart) + .slice(0, 4); + const allowedPages = new Set(selectedPages.map((page) => page.key)); + for (const row of rows) { + const key = String(row.url || row.title || ''); + if (allowedPages.size >= 4) break; + if (!allowedPages.has(key)) { + allowedPages.add(key); + selectedPages.push({ key, rows: [row] }); + } + } + const selected = []; + const seen = new Set(); + for (const row of [ + ...selectedPages.map((page) => page.rows[0]), + ...selectedPages.flatMap((page) => page.rows.slice(1)), + ...rows, + ]) { + const page = String(row.url || row.title || ''); + const key = `${page}#${row.chunk_index ?? row.chunkIndex ?? 0}`; + if (!allowedPages.has(page) || seen.has(key)) continue; + seen.add(key); + selected.push(row); + if (selected.length >= limit) break; + } + return selected; } function diverseTop(rows, limit, maxPerPage = 2) { @@ -288,7 +369,7 @@ function mergeRanked(resultSets, limit = 8) { return diverseTop(ranked, limit); } -function rankLocalChunks(queries, rows, limit = 8, { customerQuestion = '' } = {}) { +function rankLocalChunks(queries, rows, limit = 8, { customerQuestion = '', subQuestions = [] } = {}) { const wanted = (queries || []).map((query) => new Set(queryTerms(query, 20))); const consumerQuestion = !isDeveloperIntent(customerQuestion || queries?.[0] || ''); const singleItemDeletion = consumerQuestion && isSingleItemDeletionQuestion(customerQuestion || queries?.[0] || ''); @@ -320,7 +401,9 @@ function rankLocalChunks(queries, rows, limit = 8, { customerQuestion = '' } = { }) .filter((row) => row.rank > 0) .sort((a, b) => b.rank - a.rank || b.authority - a.authority); - return diverseTop(ranked, limit); + return normalizeSubQuestions(subQuestions).length > 3 + ? selectSubQuestionEvidence(ranked, subQuestions, limit) + : diverseTop(ranked, limit); } function formatEvidence(rows, { maxChars = 9_000, maxPerPage = 2 } = {}) { @@ -343,9 +426,10 @@ function formatEvidence(rows, { maxChars = 9_000, maxPerPage = 2 } = {}) { const body = cleanDocument(row.body, 1_700); if (!body) continue; const block = `[${id} | ${sourceLabel(kind)} | ${trust}]\n${row.title || 'Untitled'}${section}\n${url}\n${body}`; - if (used + block.length > maxChars && blocks.length) break; + const separatorChars = blocks.length ? 2 : 0; + if (used + separatorChars + block.length > maxChars && blocks.length) break; blocks.push(block); - used += block.length; + used += separatorChars + block.length; perPage.set(url, count + 1); } return blocks.join('\n\n'); @@ -361,6 +445,39 @@ function combineEvidence(...values) { .join('\n\n'); } +function evidenceChunkRows(evidence) { + const text = String(evidence || ''); + const markers = [...text.matchAll(/^\[S\d+\s*\|[^\n]*\]\s*$/gm)]; + const chunks = []; + for (const [index, marker] of markers.entries()) { + const block = text.slice(marker.index, markers[index + 1]?.index ?? text.length).trim().split('\n'); + const url = String(block[2] || '').trim(); + const source = sourceKind(url); + // Authority comes from the known URL, never from a prose trust label. + if (!['docs', 'help', 'website', 'github'].includes(source)) continue; + const body = block.slice(3).join('\n').trim(); + if (!body) continue; + chunks.push({ url, title: String(block[1] || '').trim(), body, source, chunkIndex: `fetched-${index}` }); + } + return chunks; +} + +function jointMultipartEvidence(question, docsEvidence, officialChunks, { queries = [], subQuestions = [] } = {}) { + const parts = normalizeSubQuestions(subQuestions); + const seen = new Set(); + const chunks = []; + for (const row of [...evidenceChunkRows(docsEvidence), ...(officialChunks || [])]) { + const key = `${row.url || ''}\n${cleanDocument(row.body)}`; + if (seen.has(key)) continue; + seen.add(key); + chunks.push(row); + } + return formatEvidence(rankLocalChunks( + supportQueries(question, queries, { subQuestions: parts }), chunks, 8, + { customerQuestion: question, subQuestions: parts } + )); +} + module.exports = { MAX_DOCUMENT, CHUNK_SIZE, @@ -376,8 +493,11 @@ module.exports = { isDeveloperIntent, isDeveloperPageCompatibleQuestion, supportQueries, + normalizeSubQuestions, + selectSubQuestionEvidence, mergeRanked, rankLocalChunks, formatEvidence, combineEvidence, + jointMultipartEvidence, }; diff --git a/router.js b/router.js index c57563d..276012f 100644 --- a/router.js +++ b/router.js @@ -50,12 +50,7 @@ const WEAK_SHOP = [/\border\s*(#|number|id|num)\b/i, /\border\s*no\.?\s*#?\d{3,} const SHOP = [...STRONG_SHOP, ...WEAK_SHOP]; const ORDER_STATUS_OPENING = 'Let’s work through your delivery question.'; -const FIRMWARE = [ - /\bfirmware\b/i, - /\brma\b/i, - /\bwarehouse\b/i, - /\bserial( number)?\b/i, - /\breplacement (device|unit|omi)\b/i, +const FIRMWARE_FAILURE = [ /powers? off (by itself|after)/i, /dies? seconds after/i, /turns? off .+ (100\s*%|battery)/i, @@ -65,7 +60,16 @@ const FIRMWARE = [ /shuts? (itself )?off after/i, /\bnot charging\b/i, /\bwon'?t charge\b/i, +]; + +const FIRMWARE = [ + /\bfirmware\b/i, + /\brma\b/i, + /\bwarehouse\b/i, + /\bserial( number)?\b/i, + /\breplacement (device|unit|omi)\b/i, /\bcharger\b/i, + ...FIRMWARE_FAILURE, ]; const DESKTOP = [ @@ -90,22 +94,26 @@ const INTEGRATION_FAILURE = [ /\b(?:google|gmail|calendar|integration|extension).{0,70}\b(?:blocked|error|fail(?:ed|s|ing)?|cannot connect|won'?t connect|not connect|only (?:shows|gets|syncs))\b/i, ]; -const APP = [ - /\b(android|iphone|ios)\b/i, - /\bapple watch\b/i, +const APP_FAILURE = [ ...INTEGRATION_FAILURE, /\b(the )?app (crash|crashed|force.?clos)/i, /\bcrash(ed|es|ing)?\b/i, + /\bdidn'?t sync\b/i, + /\b(recordings?|clips?) (are )?(missing|gone)\b/i, + /\bstill nothing recorded\b/i, +]; + +const APP = [ + /\b(android|iphone|ios)\b/i, + /\bapple watch\b/i, /\blisten socket\b/i, /\b1011\b/, /\btranscription\b/i, /\bwss:\/\/api\.omi/i, - /\bdidn'?t sync\b/i, - /\b(recordings?|clips?) (are )?(missing|gone)\b/i, + /\bapp stayed open\b/i, /\b(disconnected|offline).{0,40}\bapp\b/i, /\bapp.{0,40}(disconnected|offline)\b/i, - /\bapp stayed open\b/i, - /\bstill nothing recorded\b/i, + ...APP_FAILURE, ]; const CAPTURE_FAILURE = [ @@ -278,13 +286,14 @@ function classifyRoute(text) { return { area: 'unknown', lane: 'faq', escalate: wantHuman, wantHuman }; } if (any(s, FIRMWARE)) { - return { area: 'firmware', lane: 'firmware', escalate: true, wantHuman }; + const deviceRequest = /\brma\b|\breplacement\s+(?:device|unit|omi)\b/i.test(s); + return { area: 'firmware', lane: 'firmware', escalate: wantHuman || deviceRequest || looksLikeReportedFailure(s), wantHuman }; } if (any(s, DESKTOP)) { - return { area: 'desktop', lane: 'tech', escalate: true, wantHuman }; + return { area: 'desktop', lane: 'tech', escalate: wantHuman || looksLikeReportedFailure(s), wantHuman }; } if (any(s, APP)) { - return { area: 'app', lane: 'tech', escalate: true, wantHuman }; + return { area: 'app', lane: 'tech', escalate: wantHuman || looksLikeReportedFailure(s), wantHuman }; } if (any(s, WEAK_SHOP)) { return { area: 'shop', lane: 'shop', escalate: true, wantHuman }; @@ -381,9 +390,50 @@ function looksLikeHardFailure(text) { return /\b(app (was|stayed) open|kept the app (open|running)|blue light)\b/i.test(s) && /\b(nothing|still)\b/i.test(s); } +function conditionalFailureContext(clause) { + // A past event in a temporal clause is an observation, not a hypothetical. + if (/^(?:when(?:ever)?|while|cuando|mientras)\b/i.test(clause) && + /\b(?:crashed|failed|froze|stopped|fall[oó]|se\s+detuvo|se\s+bloque[oó])\b/i.test(clause)) return clause; + // Customers omit commas. An action followed by a separate finite fault + // predicate starts the main clause; keep it. A coordinated condition + // ("...and the app fails") still belongs to the hypothetical instead. + const action = /\b(?:open(?:s|ed|ing)?|launch(?:es|ed|ing)?|start(?:s|ed|ing)?|select(?:s|ed|ing)?|choos(?:e|es|ing)|record(?:s|ed|ing)?|abro|abrimos|inici[oó]|inicio|intento|selecciono|grabo)\b/i.exec(clause); + if (!action) return 'condition'; + const remainder = clause.slice(action.index + action[0].length); + const consequence = /\b(?:(?:it|they|i|we|(?:the|my)\s+(?:app|application|device|transcript|recording|firmware))\s+(?:crash(?:es|ed|ing)?|fail(?:s|ed|ing)?|freez(?:es|ing)|froze|stop(?:s|ped)?|can't|cannot|doesn't|won't|(?:is|stays?)\s+(?:offline|broken)|(?:get|see|receive)\s+(?:an?\s+)?error)|se\s+(?:bloquea|congela)|aparece\s+(?:un\s+)?error)\b/i.exec(remainder); + if (!consequence || /\b(?:and|or|y|o)\s*$/i.test(remainder.slice(0, consequence.index))) return 'condition'; + return 'condition ' + remainder.slice(consequence.index); +} + +function looksLikeReportedFailure(text) { + const s = String(text || '').replace(/[\u2018\u2019]/g, "'") + // In a diagnostic noun phrase, crash/error names the type of artifact, + // not an assertion that the customer's app crashed or raised an error. + // Reported failures elsewhere in the same question remain untouched. + .replace(/\b(?:crash|error|exception|failure)s?\s+(logs?|reports?|guides?|documentation)\b/gi, 'diagnostic $1') + // Not knowing a navigation/procedure is not a broken capability. Remove + // just that grammatical signal; missing recordings or a separate incident + // in the remainder still reaches the ordinary failure checks below. + .replace(/\b(?:can't|cannot|can not|couldn't|could not)\s+(?:find|locate|see\s+where|figure\s+out|work\s+out)\b/gi, 'navigation') + .replace(/\b(?:i'm|i am)\s+lost\b|\bnot\s+sure\s+how\b|\b(?:don't|do not|didn't|did not)\s+know\s+(?:how|where)\b/gi, 'navigation') + .replace(/\b(?:no\s+(?:puedo|podemos|consigo)\s+(?:encontrar|localizar|ver\s+d[oó]nde|averiguar|entender)|estoy\s+perdid[oa]|no\s+s[eé]\s+(?:c[oó]mo|d[oó]nde))\b/gi, 'navigation') + // A condition describes a circumstance to explain, not an observed fault. + // Stop at clause/sentence boundaries so a main or contrasting clause can + // still report a genuine crash, missing capture, or failed operation. + .replace(/\b(?:if|when(?:ever)?|while|in\s+case|si|cuando|mientras|en\s+caso\s+de\s+que)\b(?:(?![,;:\n]|[.!?](?:\s|$)|\b(?:but|however|yet|pero|sin\s+embargo|entonces|then)\b).)*/gi, conditionalFailureContext); + if (looksLikeHardFailure(s) || looksLikeCaptureFailure(s) || + any(s, APP_FAILURE) || any(s, FIRMWARE_FAILURE)) return true; + // Topic names select the product lane, not a support incident. Only an + // observed failure/negated capability adds the deterministic handoff signal. + const englishFailure = /\b(?:not\s+working|broken|errors?|fail(?:ed|s|ing)?|stopped|stuck|missing|lost|gone|cannot|can't|couldn't|could\s+not|won't|doesn't|didn't|unavailable|freez(?:e|es|ing)|froze|never\s+(?:get|gets|receive|receives)|no\s+(?:audio|transcript|recording|response))\b/i; + const spanishFailure = /\b(?:no\s+(?:funciona|funcionan|abre|abren|graba|graban|sincroniza|sincronizan|conecta|conectan|termina|terminan)|dej[oó]\s+de\s+(?:funcionar|grabar|transcribir)|fall(?:a|an|o|os|ando)|atascad[oa]s?|rot[oa]s?|perdid[oa]s?|faltan?)\b/i; + const connectionFailure = /\b(?:is|was|stays?|stayed|keeps?|kept|show(?:s|ing)?|says?|reports?|became|went)\s+(?:as\s+)?(?:disconnected|offline)\b/i; + return englishFailure.test(s) || spanishFailure.test(s) || connectionFailure.test(s); +} + function looksLikeProductQuestion(text) { const s = String(text || ''); - if (looksLikeHardFailure(s)) return false; + if (looksLikeReportedFailure(s)) return false; if ( any(s, STRONG_MONEY) || any(s, WEAK_MONEY) || @@ -587,15 +637,18 @@ function cannedReply(route, question) { const batteryReply = batteryReadingReply(question); if (batteryReply) return batteryReply; if (lane === 'firmware') { + if (route?.escalate === false) return 'You need guidance on using or setting up your Omi device.'; return 'Your Omi device isn’t working as expected. A device review is needed to identify the cause and a safe next step.'; } if (lane === 'tech' && route?.area === 'desktop') { + if (route?.escalate === false) return 'You need guidance on using or setting up the computer app.'; return 'The computer app isn’t working as expected. A closer review is needed to find a safe next step without repeating checks you’ve already tried.'; } if (lane === 'tech' && route?.area === 'unknown') { return 'You’re running into a technical problem. A closer review is needed to identify the cause and a safe next step.'; } if (lane === 'tech') { + if (route?.escalate === false) return 'You need guidance on using or setting up the phone app.'; return 'The phone app isn’t working as expected. A closer review is needed to find a safe next step without repeating checks you’ve already tried.'; } if (/\bblue\b/i.test(String(question || '')) && /\blight\b/i.test(String(question || ''))) { @@ -676,10 +729,14 @@ function describe(route) { return 'Account limits and plans. Answer phone vs computer if they asked. Do not invent prices.'; } if (lane === 'firmware' || area === 'firmware') { + if (route?.escalate === false) return 'Device usage, setup or firmware guidance. Answer the requested procedure from official sources without assuming a device fault.'; return 'The device itself is failing. They may already have paired.'; } - if (area === 'desktop') return 'Computer app install or bug.'; + if (area === 'desktop') return route?.escalate === false + ? 'Computer app usage or setup. Answer the requested procedure from official sources without assuming a fault.' + : 'Computer app install or bug.'; if (area === 'app') { + if (route?.escalate === false) return 'Phone app or transcription usage and setup. Answer the requested procedure from official sources without assuming a fault.'; return 'The phone app is reporting a problem. Do not name a cause or where recordings are.'; } if (lane === 'shop') return 'Order, shipping, or customs.'; diff --git a/scripts/customer-eval.js b/scripts/customer-eval.js index cd4e96a..68af91f 100644 --- a/scripts/customer-eval.js +++ b/scripts/customer-eval.js @@ -190,7 +190,7 @@ async function replyFor(scene, route) { const sourceQuestion = searchPlan.standaloneQuestion || scene.ask; const usePublicSource = process.env.EVAL_PUBLIC_SOURCE === '1'; const [pages, code, feedback, publicSource] = await Promise.all([ - usePublicSource ? '' : relevantDocs(sourceQuestion, { queries: searchPlan.queries }), + usePublicSource ? '' : relevantDocs(sourceQuestion, { queries: searchPlan.queries, subQuestions: searchPlan.subQuestions }), github.searchOfficialCode(sourceQuestion, { queries: searchPlan.queries }), router.isTechLane(route) || route.lane === 'unknown' ? relevantFeedback(sourceQuestion, { queries: searchPlan.queries }) @@ -201,7 +201,8 @@ async function replyFor(scene, route) { await loadOfficialSource(fetch, { saveDocPage: async (page) => sourcePages.push(page) }); const chunks = sourcePages.flatMap((page) => chunkDocument(page)); return formatEvidence( - rankLocalChunks(supportQueries(sourceQuestion, searchPlan.queries), chunks, 8) + rankLocalChunks(supportQueries(sourceQuestion, searchPlan.queries, { subQuestions: searchPlan.subQuestions }), chunks, 8, + { customerQuestion: sourceQuestion, subQuestions: searchPlan.subQuestions }) ); })() : '', diff --git a/scripts/live-eval.js b/scripts/live-eval.js index f6b0139..67a8fc9 100644 --- a/scripts/live-eval.js +++ b/scripts/live-eval.js @@ -181,6 +181,18 @@ function makeMessage(content, channel, history, authorId) { return message; } +function evaluationDocsEvidence(retrieval, question, options, officialDocs, officialChunks) { + if (retrieval.normalizeSubQuestions?.(options.subQuestions).length > 3) { + return retrieval.jointMultipartEvidence(question, officialDocs, officialChunks, options); + } + const official = retrieval.formatEvidence( + retrieval.rankLocalChunks(retrieval.supportQueries(question, options.queries), officialChunks, 7, { + customerQuestion: options.customerQuestion || question, + }) + ); + return retrieval.combineEvidence(officialDocs, official); +} + async function run(root, selected, { runs = 3, concurrency = 1, onProgress = () => {} } = {}) { // These overrides make it impossible for the harness to contact actual // customers or staff. They are set before index.js captures env values. @@ -217,17 +229,12 @@ async function run(root, selected, { runs = 3, concurrency = 1, onProgress = () const officialChunks = [...helpPages, ...websitePages].flatMap((page) => retrieval.chunkDocument(page)); const realRelevantDocs = docs.relevantDocs; docs.relevantDocs = async (question, options = {}) => { - const official = retrieval.formatEvidence( - retrieval.rankLocalChunks(retrieval.supportQueries(question, options.queries), officialChunks, 7, { - customerQuestion: options.customerQuestion || question, - }) - ); const officialDocs = await realRelevantDocs(question, options); // The live harness has no Postgres index, so it loads Help Center and docs // separately. Put the directly matched docs chunks before the broad local // Help Center matches so the answer/reviewer context limits retain both. // Source authority is still carried by each block and enforced by policy. - return retrieval.combineEvidence(officialDocs, official); + return evaluationDocsEvidence(retrieval, question, options, officialDocs, officialChunks); }; const utils = inRoot('utils.js'); utils.typingDelay = async () => {}; @@ -354,4 +361,4 @@ async function main() { if (require.main === module) main().catch((err) => { console.error(err.message); process.exitCode = 1; }); -module.exports = { cases, selectCases, parseCaseLine, loadCasesFile, judge, summarize, captureModelUsage, makeChannel, makeMessage, run }; +module.exports = { cases, selectCases, parseCaseLine, loadCasesFile, judge, summarize, captureModelUsage, makeChannel, makeMessage, evaluationDocsEvidence, run }; diff --git a/test/answerBudget.test.js b/test/answerBudget.test.js new file mode 100644 index 0000000..542fee4 --- /dev/null +++ b/test/answerBudget.test.js @@ -0,0 +1,21 @@ +const assert = require('node:assert/strict'); +const test = require('node:test'); +const { queryAgent } = require('../commandcode'); + +test('answer instructions budget the reply and prioritize brief supported multipart answers', async t => { + const previousKey = process.env.CMD_API_KEY; + process.env.CMD_API_KEY = 'test-only-key'; + t.after(() => { + if (previousKey === undefined) delete process.env.CMD_API_KEY; + else process.env.CMD_API_KEY = previousKey; + }); + await queryAgent({ + question: 'Explain recording, language, sharing and deleting a conversation.', + post: async (_url, body) => { + assert.match(body.messages[0].content, /hard budget of 1,500 characters/i); + assert.match(body.messages[0].content, /parts.*briefly.*order asked/i); + assert.match(body.messages[0].content, /sources support/i); + return { data: { choices: [{ message: { content: '{"final_answer":"Use the app recording button.","confidence":0.9,"escalate":false}' } }] } }; + }, + }); +}); diff --git a/test/answerPipeline.test.js b/test/answerPipeline.test.js index e1cc34f..37ce723 100644 --- a/test/answerPipeline.test.js +++ b/test/answerPipeline.test.js @@ -8,6 +8,7 @@ const { addUnsyncedDataWarning, hasUnsyncedDataRisk, stripUnverifiedOrderClaims, + UNSYNCED_DATA_WARNING, } = require('../answerPipeline'); test('reviewed answers preserve legitimate conjunctions and link lines', () => { @@ -184,6 +185,63 @@ test('an answer that fits is not clipped, with or without a source line', () => assert.ok(sourced.endsWith('Source: https://help.omi.me/en/articles/10401566')); assert.equal(sourced.includes('…'), false); const long = presentReviewedAnswer(`${step} `.repeat(30)); - assert.equal(long.length, 1900); + assert.ok(long.length <= 1900); assert.ok(long.endsWith('…')); + assert.ok(long.slice(0, -2).endsWith(step)); +}); + +test('overflow keeps whole numbered steps and their wrapped continuation lines', () => { + const source = 'Source: https://help.omi.me/en/articles/recording-reference'; + const first = '1. Start the recording with the microphone button.'; + const continuation = ' Keep the app recording while the meeting is in progress.'; + const oversized = `2. ${'Review the transcript and make sure the meeting has finished. '.repeat(40)}`; + const shown = presentReviewedAnswer(`${first}\n${continuation}\n${oversized}\n3. Open the summary.\n\n${source}`); + assert.equal(shown, `${first}\n${continuation.trim()}\n…\n${source}`); + assert.ok(shown.length <= 1900); +}); + +test('overflow never retains part of a bullet even when its first line fits', () => { + const source = 'Fuente: https://help.omi.me/es/articles/recording-reference'; + const complete = '- Abre la conversación y revisa el resumen.'; + const shown = presentReviewedAnswer(`${complete}\n- Comprueba los detalles de la grabación.\n ${'Espera a que termine el procesamiento. '.repeat(70)}\n- Guarda los cambios.\n\n${source}`); + assert.equal(shown, `${complete}\n…\n${source}`); + assert.doesNotMatch(shown, /Comprueba|procesamiento|Guarda/); +}); + +test('overflow clips prose at a whole sentence and does not split a word', () => { + const sentence = 'Open the conversation and read its summary.'; + const shown = presentReviewedAnswer(`${sentence} ${'supercalifragilisticexpialidocious '.repeat(80)}`); + assert.equal(shown, `${sentence}\n…`); +}); + +test('a first item too long to display is not partially shown and keeps complete sources', () => { + const source = '来源:https://help.omi.me/en/articles/recording-reference'; + const shown = presentReviewedAnswer(`- ${'A single long checklist item with no safe item boundary. '.repeat(50)}\n\n${source}`); + assert.ok(shown.length <= 1900); + assert.ok(shown.endsWith(source)); + assert.doesNotMatch(shown, /checklist item/); +}); + +test('four long source URLs remain complete while numbered steps fit the shared reply limit', () => { + const urls = [1, 2, 3, 4].map((part) => `https://help.omi.me/en/articles/${part}-${'official-guide-'.repeat(12)}`); + const source = `Fuente: ${urls.join(' ')}`; + const steps = [ + `1) Open the app and check the recording controls. ${'Read each control carefully. '.repeat(10)}`, + `2) Review the completed conversation. ${'Read the transcript before continuing. '.repeat(10)}`, + `3) Choose the appropriate language. ${'Check the saved choice before the meeting. '.repeat(30)}`, + ]; + const shown = presentReviewedAnswer(`${steps.join('\n')}\n\n${source}`); + assert.ok(shown.length <= 1900); + assert.equal(shown, `${steps.slice(0, 2).map((step) => step.trim()).join('\n')}\n…\n${source}`); + assert.equal((shown.match(/^Fuente:/gmu) || []).length, 1); + for (const url of urls) assert.ok(shown.includes(url)); +}); + +test('making room for the data-loss warning also keeps whole steps and complete sources', () => { + const source = 'Source: https://help.omi.me/en/articles/recording-reference'; + const first = '1. Keep the existing recordings in place.'; + const oversized = `2. ${'Review the stored recording before continuing. '.repeat(50)}`; + const shown = addUnsyncedDataWarning(`${first}\n${oversized}\n\n${source}`, 'The recording from my meeting is missing.'); + assert.equal(shown, `${first}\n…\n\n${UNSYNCED_DATA_WARNING}\n\n${source}`); + assert.ok(shown.length <= 1750); }); diff --git a/test/commandcode.test.js b/test/commandcode.test.js index e0639d2..76ba938 100644 --- a/test/commandcode.test.js +++ b/test/commandcode.test.js @@ -674,6 +674,87 @@ test('parseSearchPlan clips and deduplicates unsafe output shape', () => { assert.deepEqual(parsed.customerFacts, ['has Omi']); }); +test('multipart planning preserves one neutral query for each distinct requested part', async (t) => { + const previousKey = process.env.CMD_API_KEY; + process.env.CMD_API_KEY = 'test-only-key'; + t.after(() => { + if (previousKey === undefined) delete process.env.CMD_API_KEY; + else process.env.CMD_API_KEY = previousKey; + }); + const parts = [ + { question: 'Start a phone recording', query: 'Omi phone microphone recording' }, + { question: 'Choose a transcription language', query: 'Omi transcription language settings' }, + { question: 'Find completed conversations', query: 'Omi completed conversations list' }, + { question: 'Delete an individual conversation', query: 'Omi app individual conversation deletion' }, + { question: 'Find action items', query: 'Omi conversation action items tasks' }, + { question: 'Set up a speech profile', query: '' }, + ]; + const plan = await planSearch({ + question: 'Please explain recording, language, conversations, deletion, tasks and speech profile.', + post: async (_url, body) => { + assert.match(body.messages[0].content, /one search query for each distinct sub-question/i); + return { data: { choices: [{ message: { content: JSON.stringify({ + standalone_question: 'How do I use the recording and conversation features?', + sub_questions: [...parts, parts[1]], search_queries: ['Omi getting started'], + }) } }] } }; + }, + }); + assert.equal(plan.subQuestions.length, 6); + assert.deepEqual(plan.mustAnswer, parts.map((part) => part.question)); + for (const part of plan.subQuestions) { + assert.ok(plan.queries.includes(part.query), part.question); + assert.ok(part.query.trim(), 'a missing query falls back to the neutral question, not an answer'); + } +}); + +test('multipart planner fields stay bounded, deduplicated and safe for malformed shapes', () => { + const parts = Array.from({ length: 10 }, (_, index) => ({ question: `Feature ${index + 1}`, query: `Search feature ${index + 1}` })); + const plan = parseSearchPlan(JSON.stringify({ sub_questions: [ + { question: {}, query: 'invalid shape' }, { question: ' ', query: 'blank part' }, + { question: 'Feature 1', query: {} }, { question: 'FEATURE 1', query: 'duplicate' }, ...parts, + ] }), 'Fallback'); + assert.equal(plan.subQuestions.length, 8); + assert.equal(plan.queries.length, 8); + assert.deepEqual(plan.subQuestions[0], { question: 'Feature 1', query: 'Feature 1' }); + assert.ok(plan.queries.every((query) => !query.includes('[object Object]'))); + assert.deepEqual(plan.mustAnswer, plan.subQuestions.map((part) => part.question)); + const legacy = parseSearchPlan(JSON.stringify({ must_answer: ['First', 'Second', 'Third', 'Fourth'], search_queries: 'bad shape' }), 'Fallback'); + assert.deepEqual(legacy.queries, ['First', 'Second', 'Third', 'Fourth']); +}); + +test('paired planner queries cannot be crowded out by generic searches on a shorter request', () => { + const parts = [ + { question: 'Change the language', query: 'Omi transcription language' }, + { question: 'Find saved summaries', query: 'Omi conversation summaries' }, + { question: 'Remove one conversation', query: 'Omi individual conversation deletion' }, + ]; + const plan = parseSearchPlan(JSON.stringify({ sub_questions: parts, search_queries: ['overview', 'guide', 'basics', 'introduction'] }), 'Fallback'); + assert.equal(plan.queries.length, 4, 'ordinary planner query cap stays unchanged'); + for (const part of parts) assert.ok(plan.queries.includes(part.query)); +}); + +test('multipart review may cite four retrieved pages while ordinary review still cites two', async () => { + const evidence = Array.from({ length: 5 }, (_, index) => + `[S${index + 1} | Official Help Center]\nPart ${index + 1}\nhttps://help.omi.me/en/articles/fixture-${index + 1}\nOfficial instructions.`).join('\n\n'); + const subQuestions = Array.from({ length: 4 }, (_, index) => ({ question: `Requested part ${index + 1}`, query: `Topic ${index + 1}` })); + const sourceIds = ['S1', 'S2', 'S3', 'S4', 'S5']; + const multipart = groundedSourceLine('Here are the four requested steps.', evidence, sourceIds, 'en', { subQuestions }); + const ordinary = groundedSourceLine('Here is the requested step.', evidence, sourceIds); + assert.equal((multipart.match(/https:\/\/help\.omi\.me\//g) || []).length, 4); + assert.equal((ordinary.match(/https:\/\/help\.omi\.me\//g) || []).length, 2); + await reviewAnswer({ + question: 'Explain four app features.', draft: 'Here are the four requested steps.', sources: evidence, + understanding: { subQuestions }, lane: 'faq', + post: async (_url, body) => { + assert.match(body.messages[0].content, /at most four exact URLs/i); + return { data: { choices: [{ message: { content: JSON.stringify({ + final_answer: 'Here are the four requested steps.', grounded: true, relevant: true, + sources_used: sourceIds, confidence: 0.9, + }) } }] } }; + }, + }).then((reviewed) => assert.equal((reviewed.final_answer.match(/https:\/\/help\.omi\.me\//g) || []).length, 4)); +}); + test('source lines can only use URLs from retrieved evidence blocks', () => { const answer = groundedSourceLine( 'Hold the button.\n\nSource: https://docs.omi.me', diff --git a/test/docs-multipart.test.js b/test/docs-multipart.test.js new file mode 100644 index 0000000..ce8736d --- /dev/null +++ b/test/docs-multipart.test.js @@ -0,0 +1,36 @@ +const assert = require('node:assert/strict'); +const test = require('node:test'); +const { relevantDocs } = require('../docs'); + +test('multipart cold-index retrieval fetches complementary pages before broad overview hits consume the slots', async () => { + const parts = [ + { question: 'Start microphone capture', query: 'microphone capture start' }, + { question: 'Select a transcription language', query: 'transcription language selection' }, + { question: 'Remove one conversation', query: 'individual conversation deletion' }, + { question: 'Find extracted tasks', query: 'extracted action tasks' }, + ]; + const pages = parts.map((part, index) => ({ + title: part.query, url: `https://docs.omi.me/fixture/part-${index}.md`, + body: `${part.query}: follow the documented control for this feature.`, + })); + const generic = Array.from({ length: 3 }, (_, index) => ({ + title: 'App overview', url: `https://docs.omi.me/fixture/overview-${index}.md`, + body: 'App overview: introductory information about the app.', + })); + const fetched = []; + const evidence = await relevantDocs('Omi app overview', { + queries: ['App overview'], subQuestions: parts, + fetchImpl: async (url) => { + if (String(url).endsWith('/llms.txt')) return { + ok: true, text: async () => [...generic, ...pages].map((page) => `- [${page.title}](${page.url}): ${page.title}`).join('\n'), + }; + fetched.push(String(url)); + const page = [...generic, ...pages].find((item) => item.url === url); + return { ok: true, text: async () => page?.body || '' }; + }, + }); + assert.ok(fetched.length <= 6, 'existing page-fetch limit is unchanged'); + for (const page of pages) assert.ok(evidence.includes(page.url), page.title); + assert.equal(new Set([...evidence.matchAll(/^https:\/\/\S+/gm)].map((match) => match[0])).size, 4); + assert.ok(evidence.length <= 9_000); +}); diff --git a/test/docs.test.js b/test/docs.test.js index 0f428c7..0ec8047 100644 --- a/test/docs.test.js +++ b/test/docs.test.js @@ -3,6 +3,43 @@ const test = require('node:test'); const { relevantDocs, storedDocs, topPages, canonicalizeGithubDocsEvidence } = require('../docs'); const { pages: developerPages, store: developerStore } = require('./fixtures/developer-docs'); +test('multipart stored retrieval preserves complementary Help Center pages within the existing evidence budget', async () => { + const subQuestions = [ + { question: 'Start microphone capture', query: 'microphone capture start' }, + { question: 'Select a transcription language', query: 'transcription language selection' }, + { question: 'Remove one conversation', query: 'individual conversation deletion' }, + { question: 'Find extracted tasks', query: 'extracted action tasks' }, + ]; + const pages = subQuestions.map((part, index) => ({ + url: `https://help.omi.me/en/articles/multipart-${index + 1}`, + title: part.query, source: 'help', chunk_index: 0, rank: 0.4, + body: `${part.query}: use the documented app control. ${'More documentation. '.repeat(65)}`, + })); + const generic = Array.from({ length: 12 }, (_, index) => ({ + url: `https://docs.omi.me/general-app-${index}`, title: 'General app guide', + source: 'docs', chunk_index: 0, rank: 0.9, + body: 'An introduction to the app and other unrelated features.', + })); + const searches = []; + const evidence = await relevantDocs('Please explain capture, language, conversation management and tasks.', { + subQuestions, queries: ['general app guide'], + store: { searchDocPages: async (query, _limit, sources = []) => { + searches.push(query); + if (sources.includes('github')) return []; + const page = pages.find((item) => query === item.title); + if (page) return [page]; + return sources.includes('help') ? [pages[0]] : generic; + } }, + fetchImpl: async () => { throw new Error('the fixture store supplies all evidence'); }, + }); + for (const [index, part] of subQuestions.entries()) { + assert.ok(searches.includes(part.query), `part ${index + 1} has its own search`); + assert.ok(evidence.includes(pages[index].url), `part ${index + 1} retains its page`); + } + assert.ok(evidence.length <= 9_000); + assert.ok((evidence.match(/^\[S\d+ \|/gm) || []).length <= 8); +}); + test('GitHub docs citations use the exact page published in the official docs index', async () => { const indexText = '- [Omi Setup](https://docs.omi.me/onboarding/omi.md): Get started'; const githubUrl = 'https://github.com/BasedHardware/omi/blob/abc123/docs/onboarding/omi.mdx'; diff --git a/test/handoff.test.js b/test/handoff.test.js index c54821b..a46a6e5 100644 --- a/test/handoff.test.js +++ b/test/handoff.test.js @@ -209,6 +209,46 @@ test('staff cards request details appropriate to the ticket and flag possible da assert.match(loss.fields.find((field) => field.name === 'Why').value, /^Possible data loss: they should not reinstall, log out or clear storage\./); }); +test('damaged-item and replacement intake asks for evidence and order email, not tracking', () => { + const base = { message: { url: 'https://discord.com/channels/1/2/3', author: { id: '99' }, channel: { id: '2' } }, staffOnly: true, area: 'shop' }; + for (const [lane, question] of [ + ['shop', 'The housing arrived cracked. My order number is 38192. What should I do?'], + ['money', 'I purchased a unit and would like a replacement for the defective device.'], + ['shop', 'The device was damaged in the box; I need the team to help.'], + ['shop', 'Can you replace my Omi?'], + ['shop', 'This unit needs a replacement.'], + ]) { + const card = formatStaffTicket({ ...base, lane, question }).discord.embeds[0]; + const intake = card.fields.find((field) => field.name === 'Staff').value; + assert.match(intake, /photos? or (?:a )?video.*damage/i, question); + assert.match(intake, /order email.*privately/i, question); + assert.match(intake, /delivery date/i, question); + assert.doesNotMatch(intake, /tracking/i, question); + if (question.includes('38192')) assert.match(card.description, /38192/); + } + const crash = formatStaffTicket({ ...base, area: 'app', lane: 'tech', question: 'The app crashes when I try to replace a task.' }).discord.embeds[0]; + assert.doesNotMatch(crash.fields.find((field) => field.name === 'Staff').value, /photos|order email/i); +}); + +test('replacement intake stays specific to physical items and preserves account and delivery intake', () => { + const base = { message: { url: 'https://discord.com/channels/1/2/3', author: { id: '99' }, channel: { id: '2' } }, staffOnly: true, area: 'shop' }; + for (const question of [ + 'I would like to replace my redemption code.', + 'My account details need replacing after a plan change.', + 'I need to replace the device linked to my subscription account.', + ]) { + const card = formatStaffTicket({ ...base, lane: 'account', question }).discord.embeds[0]; + const intake = card.fields.find((field) => field.name === 'Staff').value; + assert.match(intake, /purchase email.*privately/i, question); + assert.doesNotMatch(intake, /photo|video|damage|tracking/i, question); + } + const receipt = formatStaffTicket({ ...base, lane: 'money', question: 'Could you replace my purchase receipt for the Omi device?' }).discord.embeds[0]; + assert.doesNotMatch(receipt.fields.find((field) => field.name === 'Staff').value, /photo|video|damage/i); + const delivery = formatStaffTicket({ ...base, lane: 'shop', question: 'The parcel is marked delivered but never arrived.' }).discord.embeds[0]; + assert.match(delivery.fields.find((field) => field.name === 'Staff').value, /order number and the delivery date tracking shows/i); + assert.doesNotMatch(delivery.fields.find((field) => field.name === 'Staff').value, /photo|video|damage/i); +}); + test('staff ticket infers shop/account labels from fair-use text', () => { const ticket = formatStaffTicket({ message: { diff --git a/test/index.test.js b/test/index.test.js index 63a097c..9b40f65 100644 --- a/test/index.test.js +++ b/test/index.test.js @@ -40,6 +40,7 @@ const modelCalls = []; let modelReply = {}; let modelDown = false; let searchPlanQueries = []; +let plannerSubQuestions = []; const plannerCalls = []; let plannerSupportKind = 'other'; let plannerMessageKind = 'question'; @@ -84,6 +85,7 @@ commandcode.understandQuestion = async ({ question, onUsage }) => { wantsPerson: plannerWantsPerson, dataLossRisk: plannerDataLossRisk, queries: searchPlanQueries, + subQuestions: plannerSubQuestions, }; }; commandcode.reviewAnswer = async (args) => { @@ -373,6 +375,7 @@ test.beforeEach(() => { modelReply = {}; modelDown = false; searchPlanQueries = []; + plannerSubQuestions = []; plannerCalls.length = 0; plannerSupportKind = 'other'; plannerMessageKind = 'question'; @@ -649,6 +652,33 @@ test('large docs evidence still passes later Help Center blocks to the answer st } }); +test('multi-part planning reaches production retrieval without dropping individual sub-questions', async t => { + const previousKey = process.env.CMD_API_KEY; + t.after(() => { + docsObserver = null; + if (previousKey === undefined) delete process.env.CMD_API_KEY; else process.env.CMD_API_KEY = previousKey; + }); + process.env.CMD_API_KEY = 'test-only-key'; + plannerSubQuestions = [ + { question: 'Start a phone recording', query: 'Omi phone microphone recording instructions' }, + { question: 'Choose a transcription language', query: 'Omi app transcription language settings' }, + { question: 'Read a conversation summary', query: 'Omi conversation summary view' }, + { question: 'Remove one saved conversation', query: 'Omi individual conversation deletion' }, + ]; + searchPlanQueries = plannerSubQuestions.map(part => part.query); + const lookups = []; + docsObserver = async (question, options) => { + lookups.push({ question, options }); + return '[S1 | Official Help Center | authoritative]\nApp controls\nhttps://help.omi.me/en/articles/fixture-controls\nThe app provides recording, language and conversation controls.'; + }; + const result = await ask('Please explain phone recording, transcription language selection, reading summaries and removing one saved conversation.'); + assert.equal(result.modelCalled, true); + assert.equal(lookups.length, 1); + assert.deepEqual(lookups[0].options.subQuestions, plannerSubQuestions); + assert.deepEqual(lookups[0].options.queries, searchPlanQueries); + assert.deepEqual(modelCalls.at(-1).understanding.subQuestions, plannerSubQuestions); +}); + test('a how-to question in #vector-test gets the model answer and no Handoff', async () => { const r = await ask('How do I pair my Omi with a new phone?'); assert.equal(r.modelCalled, true); diff --git a/test/live-eval.test.js b/test/live-eval.test.js index c2800de..82efe88 100644 --- a/test/live-eval.test.js +++ b/test/live-eval.test.js @@ -1,6 +1,6 @@ const assert = require('node:assert/strict'); const test = require('node:test'); -const { parseCaseLine, loadCasesFile, judge, summarize, captureModelUsage } = require('../scripts/live-eval'); +const { parseCaseLine, loadCasesFile, judge, summarize, captureModelUsage, evaluationDocsEvidence } = require('../scripts/live-eval'); test('live-eval keeps per-call model usage next to stage timings', () => { const state = { timings: { planner: 10 }, modelCalls: [] }; @@ -54,3 +54,24 @@ test('three-run scoring accepts two passes and lists that case as flaky', () => assert.deepEqual(summary.fail, []); assert.deepEqual(summary.latency.planner, { p50: 2, p95: 3 }); }); + +test('multi-part live evidence uses one joint corpus while ordinary lookup keeps later Help Center blocks', () => { + const parts = ['recording', 'language', 'summaries', 'deletion'].map(question => ({ question, query: question })); + const calls = []; + const chunks = [{ url: 'https://help.omi.me/en/articles/fixture-guide', body: 'Fixture guidance' }]; + const docs = '[Documentation blocks]\n' + 'D'.repeat(9000); + const help = '[Help Center blocks]\n' + 'H'.repeat(8000); + const retrieval = { + normalizeSubQuestions: subQuestions => subQuestions || [], + jointMultipartEvidence: (...args) => { calls.push(args); return 'joint coverage'; }, + supportQueries: () => ['neutral query'], rankLocalChunks: () => chunks, + formatEvidence: () => help, combineEvidence: (...blocks) => blocks.join('\n\n'), + }; + const options = { queries: parts.map(part => part.query), subQuestions: parts }; + assert.equal(evaluationDocsEvidence(retrieval, 'A multi-part request', options, docs, chunks), 'joint coverage'); + assert.deepEqual(calls[0], ['A multi-part request', docs, chunks, options]); + const ordinary = evaluationDocsEvidence(retrieval, 'One ordinary request', {}, docs, chunks); + assert.ok(ordinary.length > 17000); + assert.ok(ordinary.includes('[Help Center blocks]')); + assert.equal(calls.length, 1, 'ordinary evidence must not be re-budgeted or merged through multipart policy'); +}); diff --git a/test/retrieval.test.js b/test/retrieval.test.js index 2752643..4176692 100644 --- a/test/retrieval.test.js +++ b/test/retrieval.test.js @@ -13,6 +13,7 @@ const { uniqueQueries, isDeveloperIntent, isDeveloperPageCompatibleQuestion, + jointMultipartEvidence, } = require('../retrieval'); const { pages: developerPages } = require('./fixtures/developer-docs'); @@ -137,6 +138,78 @@ test('query expansion keeps exact product terms and deduplicates planned searche ); }); +test('multipart query expansion never spends its subquestion slots on generic variants', () => { + const subQuestions = Array.from({ length: 6 }, (_, index) => ({ question: `Requested feature ${index + 1}`, query: `Omi feature-${index + 1}` })); + const queries = supportQueries('How do I remove one memory in the app and use other features?', ['generic overview'], { subQuestions }); + for (const part of subQuestions) assert.ok(queries.includes(part.query), part.query); + assert.equal(queries[0], 'How do I remove one memory in the app and use other features?'); +}); + +test('multipart local evidence selects one useful chunk from each of four complementary pages before extras', () => { + const subQuestions = [ + { question: 'Start microphone capture', query: 'microphone capture start' }, + { question: 'Select a transcription language', query: 'transcription language selection' }, + { question: 'Remove one conversation', query: 'individual conversation deletion' }, + { question: 'Find extracted tasks', query: 'extracted action tasks' }, + ]; + const pages = subQuestions.flatMap((part, index) => Array.from({ length: 2 }, (_, chunkIndex) => ({ + url: `https://help.omi.me/en/articles/fixture-${index}`, title: part.query, + body: `${part.query}. ${'Additional official documentation. '.repeat(48)}`, + source: 'help', chunkIndex, + }))); + const generic = Array.from({ length: 8 }, (_, index) => ({ + url: `https://docs.omi.me/generic-${index}`, title: 'App overview', + body: 'App overview phone app menus and controls.', source: 'docs', chunkIndex: 0, + })); + const queries = supportQueries('Omi app overview', ['app overview'], { subQuestions }); + const rows = rankLocalChunks(queries, [...generic, ...pages], 8, { + customerQuestion: 'Omi app overview', subQuestions, + }); + assert.equal(new Set(rows.slice(0, 4).map((row) => row.url)).size, 4); + const evidence = formatEvidence(rows); + for (let index = 0; index < 4; index += 1) assert.ok(evidence.includes(`https://help.omi.me/en/articles/fixture-${index}`)); + assert.ok(evidence.length <= 9_000); +}); + +test('multipart joint corpus retains Help Center coverage rather than filling the budget with fetched docs first', () => { + const subQuestions = [ + { question: 'Start microphone capture', query: 'microphone capture start' }, + { question: 'Select a transcription language', query: 'transcription language selection' }, + { question: 'Remove one conversation', query: 'individual conversation deletion' }, + { question: 'Find extracted tasks', query: 'extracted action tasks' }, + ]; + const fetchedRows = Array.from({ length: 7 }, (_, index) => ({ + url: `https://docs.omi.me/fixture-overview-${index}.md`, title: 'App overview', + source: 'docs', body: `App overview for general setup. ${'Background details. '.repeat(90)}`, chunkIndex: 0, + })); + const docsEvidence = formatEvidence(fetchedRows); + const help = subQuestions.map((part, index) => ({ + url: `https://help.omi.me/en/articles/joint-fixture-${index}`, + title: part.query, source: 'help', body: `${part.query}: follow this documented control. ${'Official information. '.repeat(50)}`, + chunkIndex: 0, + })); + const result = jointMultipartEvidence('Omi app overview', docsEvidence, help, { queries: ['Omi app overview'], subQuestions }); + for (const page of help) assert.ok(result.includes(page.url)); + assert.equal(new Set([...result.matchAll(/^https:\/\/\S+/gm)].map((match) => match[0])).size, 4); + assert.ok(result.length <= 9_000); + assert.ok((result.match(/^\[S\d+ \|/gm) || []).length <= 8); + + const mixedParts = subQuestions.map((part, index) => ({ ...part, query: index ? part.query : 'App overview' })); + const mixed = jointMultipartEvidence('Omi app overview', docsEvidence, help, { subQuestions: mixedParts }); + assert.match(mixed, /Official documentation \| authoritative/); + assert.match(mixed, /App overview\nhttps:\/\/docs\.omi\.me\/fixture-overview-/); + assert.match(mixed, /Official Help Center \| authoritative/); + assert.ok(mixed.length <= 9_000); +}); + +test('the evidence character budget includes separators between chunks', () => { + const row = { url: 'https://help.omi.me/en/articles/budget-fixture', title: 'Official page', body: 'Grounded information. '.repeat(50), source: 'help' }; + const one = formatEvidence([row]); + const maxChars = one.length * 2; + const result = formatEvidence([row, { ...row, chunkIndex: 1 }], { maxChars }); + assert.ok(result.length <= maxChars); +}); + test('voice input without an answer adds the intended chat-delivery vocabulary', () => { const queries = supportQueries( 'The transcription appears after I ask, but nothing answers', diff --git a/test/router-reporting.test.js b/test/router-reporting.test.js new file mode 100644 index 0000000..21144c9 --- /dev/null +++ b/test/router-reporting.test.js @@ -0,0 +1,69 @@ +const assert = require('node:assert/strict'); +const test = require('node:test'); +const router = require('../router'); + +const howtos = [ + 'I cannot locate the language menu in the iPhone app; please point me to it.', + "I'm lost in the Android settings and don't know where to enable transcription.", + 'For the desktop app, what happens if the connection fails while my laptop is offline?', + 'No puedo encontrar dónde elegir el idioma en mi iPhone; necesito una guía.', + 'Estoy perdida en los ajustes de Android, no sé cómo seleccionar la transcripción.', + '¿Qué debo hacer si la aplicación de iPhone falla cuando el teléfono está sin conexión?', +]; +const failures = [ + "I can't locate any voice clips I captured this morning; the recordings are missing in the Android app.", + "I'm lost trying to recover a meeting: my iPhone app froze and the transcript never saved.", + "The desktop app stays offline after reconnecting to the router; it doesn't sync.", + 'Estoy perdido: la aplicación de iPhone se bloquea y muestra error 503 al abrir una conversación.', + 'Cuando abro la transcripción en Android, la aplicación deja de responder y falla cada vez.', + 'Si intento iniciar una grabación en mi iPhone, aparece un error y no graba nada.', +]; + +for (const [i, question] of howtos.entries()) { + test(`navigation or hypothetical how-to ${i + 1} does not escalate`, () => { + assert.equal(router.classify(question).escalate, false); + }); +} +for (const [i, question] of failures.entries()) { + test(`actual report alongside navigation or a condition ${i + 1} still escalates`, () => { + assert.equal(router.classify(question).escalate, true); + }); +} + +test('navigation and uncertainty grammar does not negate an independent incident', () => { + for (const navigation of [ + "can't find", 'cannot locate', "couldn't see where", "can't figure out", 'cannot work out', + "I'm lost", 'I am lost', 'not sure how', "don't know how", "don't know where", + ]) { + assert.equal(router.classify(`In the iPhone app, ${navigation} the transcription settings.`).escalate, false, navigation); + assert.equal(router.classify(`In the iPhone app, ${navigation} the transcription settings. The app crashed with error code 503.`).escalate, true, navigation); + } +}); + +test('only conditional clauses are excluded, not the actual failure before or after them', () => { + for (const marker of ['if', 'when', 'while', 'in case', 'whenever']) { + assert.equal(router.classify(`Please explain Android recording ${marker} my phone is offline.`).escalate, false, marker); + assert.equal(router.classify(`My Android app crashed ${marker} I opened a transcript.`).escalate, true, marker); + assert.equal(router.classify(`${marker} my phone is offline, Android transcription stops with error 503.`).escalate, true, marker); + } + assert.equal(router.classify('Explain iPhone recording when the phone is offline, but my saved recordings are gone.').escalate, true); +}); + +test('navigation and hypothetical wording do not bypass protected person-required lanes', () => { + for (const question of [ + "I can't figure out this Android setting; I want a refund.", + 'Explain desktop recording when it is offline; please delete my account.', + 'Estoy perdida en Android y necesito una persona del equipo. I need a human to help.', + "I cannot locate the billing page; my subscription was charged twice.", + ]) assert.equal(router.classify(question).escalate, true, question); +}); + +test('independent loss and unpunctuated main-clause faults remain reports', () => { + for (const question of [ + 'I cannot locate Monday’s iPhone recordings; they are gone.', + 'When I launch the Android app it crashes before the home screen loads.', + 'Cuando abro la aplicación en Android se bloquea con error 503.', + 'When my desktop app froze this morning I had to stop the meeting.', + ]) assert.equal(router.classify(question).escalate, true, question); + assert.equal(router.classify('Explain Android recording if the phone is offline and the app fails.').escalate, false); +}); diff --git a/test/router.test.js b/test/router.test.js index a2f5eb9..1424d14 100644 --- a/test/router.test.js +++ b/test/router.test.js @@ -3,6 +3,93 @@ const test = require('node:test'); const router = require('../router'); const orderFlow = require('../orderFlow'); +test('technical setup questions retain their topic lane without a handoff', () => { + const questions = [ + ['I am setting up transcription on my iPhone and would like the setup steps.', 'app', 'tech'], + ['I want to choose the recording language in the Android app.', 'app', 'tech'], + ['Please explain sharing a transcript from the desktop app.', 'desktop', 'tech'], + ['Tell me the recommended way to check the installed firmware version.', 'firmware', 'firmware'], + ['Quiero configurar la transcripción en mi iPhone; explica los pasos.', 'app', 'tech'], + ['Necesito instrucciones para elegir el idioma en Android.', 'app', 'tech'], + ['Explícame las opciones de grabación en Omi Desktop.', 'desktop', 'tech'], + ['Busco una guía para actualizar el firmware de mi Omi.', 'firmware', 'firmware'], + ]; + for (const [question, area, lane] of questions) { + const route = router.classify(question); + assert.equal(route.area, area, question); + assert.equal(route.lane, lane, question); + assert.equal(route.escalate, false, question); + assert.equal(router.skipModel(route), false, question); + } +}); + +test('reported technical failures still escalate in their topic lane', () => { + const failures = [ + ['My iPhone app freezes with an error whenever I open a recording.', 'app'], + ['Transcription stopped working during our call.', 'app'], + ["Android doesn't save my new transcript.", 'app'], + ["My macOS app can't open the saved conversation.", 'desktop'], + ['The desktop app crashes when I start a recording.', 'desktop'], + ['The firmware update failed before it finished.', 'firmware'], + ['El firmware se queda atascado y no termina la actualización.', 'firmware'], + ['La aplicación en mi iPhone dejó de funcionar al abrir una conversación.', 'app'], + ]; + for (const [question, area] of failures) { + const route = router.classify(question); + assert.equal(route.area, area, question); + assert.equal(route.escalate, true, question); + } +}); + +test('question-shaped failure reports do not lose their technical escalation', () => { + for (const question of [ + 'How do I use transcription again? It stopped on my iPhone.', + 'Where is the transcript? Android says it failed to sync.', + ]) { + const route = router.classify(question); + assert.equal(route.area, 'app', question); + assert.equal(route.escalate, true, question); + } +}); + +test('capability and diagnostic-guide questions do not report an incident', () => { + for (const question of [ + 'Can I record offline in the Android app?', + 'How do I record offline in the Android app?', + 'How do I read error logs from the Android app when developing an integration?', + 'Where can I find documentation about crash logs in the desktop app?', + ]) assert.equal(router.classify(question).escalate, false, question); +}); + +test('asking for a guide alongside an observed error or disconnect still escalates', () => { + for (const question of [ + 'Where can I find crash logs? The Android app crashes when I open a recording.', + 'Can I record offline? My Android app keeps showing disconnected.', + 'How do I inspect error logs? The desktop app gives an error when I open it.', + ]) assert.equal(router.classify(question).escalate, true, question); +}); + +test('quiet technical lanes do not invent a problem in routing guidance or fallback', () => { + for (const route of [ + { area: 'app', lane: 'tech', escalate: false }, + { area: 'desktop', lane: 'tech', escalate: false }, + { area: 'firmware', lane: 'firmware', escalate: false }, + ]) { + assert.doesNotMatch(router.describe(route), /failing|reporting a problem|bug/i); + assert.doesNotMatch(router.cannedReply(route, 'Please explain the setup'), /isn.t working|problem|cause/i); + } +}); + +test('topic-only routing does not weaken explicit staff, replacement or protected requests', () => { + for (const question of [ + 'I want to configure transcription on my iPhone. I need a person to help.', + 'Please explain the firmware update; I need a replacement device.', + 'The desktop app is fine, but I want a refund.', + 'I want to delete my data in the Android app.', + 'I recorded a call on my iPhone and the recordings are missing.', + ]) assert.equal(router.classify(question).escalate, true, question); +}); + test('tax, duties, and customs stay shop/money and never skip as tech', () => { const tax = router.classify('import tax on order #20716'); assert.equal(tax.area, 'shop'); diff --git a/test/triage.test.js b/test/triage.test.js index 6bc799a..2688ff7 100644 --- a/test/triage.test.js +++ b/test/triage.test.js @@ -3,6 +3,35 @@ const test = require('node:test'); const triage = require('../triage'); const router = require('../router'); +test('technical topic labels and filing metadata alone do not request a person', () => { + for (const question of [ + 'Please explain using transcription on an iPhone.', + 'Quiero instrucciones para grabar con Android.', + 'I would like setup instructions for the desktop app.', + 'Busco información sobre el firmware de Omi.', + ]) { + const merged = triage.merge(router.classify(question), { + area: 'app', lane: 'tech', labels: ['app', 'tech'], + topic: 'Recording setup', file_issue: true, escalate: false, + }, question); + assert.equal(merged.escalate, false, question); + } +}); + +test('technical how-to triage preserves a genuine model handoff and protected routes', () => { + const question = 'Please explain recording with the Android app'; + const merged = triage.merge(router.classify(question), { + escalate: true, reason: 'Official evidence did not answer the requested procedure.', + }, question); + assert.equal(merged.escalate, true); + for (const protectedQuestion of [ + 'I need a person to help set up my iPhone.', + 'I want a refund for the device.', + 'Delete my data from Android.', + 'I recorded on my iPhone and now the clips are missing.', + ]) assert.equal(triage.merge(router.classify(protectedQuestion), { escalate: false }, protectedQuestion).escalate, true); +}); + test('sanitizeTopic drops greetings and keeps the problem', () => { assert.equal(triage.sanitizeTopic('Hi, I just got my omi'), ''); assert.match(triage.sanitizeTopic('fair use warning after one day'), /fair use/i);