Skip to content

Match a version question to the newest release for the product it names - #35

Open
singhharsh1708 wants to merge 1 commit into
BasedHardware:mainfrom
singhharsh1708:fix/release-note-newest-and-product
Open

singhharsh1708 wants to merge 1 commit into
BasedHardware:mainfrom
singhharsh1708:fix/release-note-newest-and-product

Conversation

@singhharsh1708

Copy link
Copy Markdown
Contributor

Customer problem

Asked "What is the latest version of the desktop app?", the bot is handed release v0.12.452 as the matching release note while v0.12.464 is out. bestNote counts every shared word, repeats included, so the pick is decided by words like "the", "what" and "app". Asked "Is there a firmware update for my Omi?" or "What's new in the phone app?", it is handed a desktop release, because the last fifteen releases are all desktop builds. The prompt tells the model to use that note for "which version to get".

Change

bestNote keeps to notes for the product the question names (desktop, firmware or phone), ignores the words every version question shares, counts a shared word once, and breaks ties by the newest release. A plain "latest version" question gets the newest release for that product. A question about one change still needs a note that mentions it. A product with no release in the window gets no note.

Evidence

Run against the live BasedHardware/omi release feed on 2026-10-10. Before: the desktop questions got v0.12.452, the firmware question v0.12.464 and the phone app questions v0.12.462, all desktop builds. After: the desktop questions get v0.12.464, and the firmware and phone app questions get nothing.

Verification

  • Added or updated a focused behavior test: three in test/releases.test.js, on a feed copied from the real one. All three fail on main
  • npm test: 629/629
  • EVAL_NO_MODEL=1 node scripts/customer-eval.js: 8/8
  • Live-model evaluation not needed: the prompt and the model path are unchanged
  • No secrets, customer PII, .env, or railway.toml
  • No new claim that a person was pinged, an issue was filed, or an email was sent without confirmed success

bestNote counted every shared word, repeats included, so "What is the
latest version of the desktop app?" was matched on words like "the" and
"app" and came back with a release twelve builds old. A firmware or
phone app question was handed a desktop release, because the last
fifteen releases are all desktop builds.

It now keeps to notes for the product the question names, ignores the
words every version question shares, counts a shared word once and
breaks ties by the newest release.

@aryanorastar aryanorastar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this fixes a real problem (firmware questions got desktop notes). Three changes before merge:
(1) when the question names no product and no specific change ("what is the latest version?"), return an empty string instead of the newest note; main returns an empty string there today;
(2) skip prereleases: keep prerelease from the API and skip notes whose name says candidate, beta or dev, so customers are not pointed at unreleased builds;
(3) productOf should recognise firmware tags such as Omi_CV1_v3.0.15, Omi_DK2_v2.0.x and omi_firmware_v…. Real tags end in -macos / -desktop-* (desktop), -mobile-cm / -android* / -ios-cm (phone).
Please add tests for each.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants