Return a test run's results instead of scraping them out of its logs

docs/gotchas/tooling.md

@@ -33,6 +33,26 @@ When a section grows to 10+ items, graduate it to its own doc.
 
 - **The watch service and the lock file do not share a version vocabulary.** A watch's `baselineVersion`, and the `currentVersion` it reports back, are the *asset-delivery content hash* for the place — that's what the service's Roblox driver reads, and it compares them by string equality. `deploy.nevermore.lock.json` holds the *Open Cloud place version* (an integer). They are never equal, so handing the lock's number over as a baseline reads as drift on the very first poll and dispatches a rebuild of the build that just shipped. The CLI therefore sends no baseline at all — the service's first poll adopts what it sees, which is what a baseline was for — and treats every version the service reports as an opaque "something moved" token, asking Open Cloud what the place is actually at before deciding to rebuild. Anything comparing a service version against a lock version is wrong even when the types line up.
 
+- **Open Cloud truncates long engine logs, so anything that must be read back exactly belongs in the script's return value.** A script printing 20,001 lines came back with 7,627 of them — the last contiguous block, with the head dropped — and the size of the window varies run to run (6,715 and 3,236 lines on other runs), so there is no line count to stay under. A Luau execution task's return value is not subject to this: it arrives on the task as `output.results`, a flat array of the returned values (`return t, "s", 42` → three entries), and Roblox serializes them itself, so a returned Lua table is real nested JSON. Do **not** `HttpService:JSONEncode` the result — that lands as a double-encoded string.
+
+- **An oversize return value annihilates the task rather than truncating the value.** A ~2.1MB return value arrives complete and intact; a ~4.2MB one fails the whole task — state `FAILED`, no `output`, and no error message saying why. So code reading a return value has to treat "the task reported no output" as *unknown* rather than empty, and fall back to the logs; `getTaskReturnValues` in `open-cloud-client.ts` draws exactly that line (`undefined` = nothing came back, `[]` = the script returned nothing).
+
+- **`Jest.runCLI` resolves with a wrapper, not the results.** jest-lua's `runCLI(...)` resolves with `{ globalConfig, results }`; every count (`numFailedTests`, `numTotalTests`, ...) lives on the inner `results`, which is the `AggregatedResult`. Reading them off the outer table gets `nil` for all of them. This is not a loud failure: `nil` counts default to zero, and zero failures over zero tests is indistinguishable from a clean run — a `numFailedTests > 0` check written against the wrapper sat in `NevermoreTestRunnerUtils` for its whole life and never fired once, so test failures were only ever caught by scraping jest's printed summary. `_resultsFromJest` now accepts either shape and **fails closed** when it can recognize neither, because "shape I cannot read" must never be spelled the same way as "nothing failed". Note `numTotalTests == 0` is a readable result (a package with no specs) — only a *missing* count means the shape is unknown.
+
+- **`AggregatedResult.success` is inverted in jest-lua. Never read it.** `TestScheduler.lua:434` assigns `aggregatedResults.success = anyTestFailures or aggregatedResults.snapshot.failure or anyReporterErrors`, where upstream jest negates that whole expression. The field is therefore true exactly when the run **failed**, and false on a clean run — so `result.success ~= false` fails every passing suite, which is how a batch of four came back `0 passed, 4 failed` with three of them green. Do not read it in the other sense either: that depends on the missing `not` staying missing. Read the underlying signals instead — `numFailedTests`, `numFailedTestSuites`, `numRuntimeErrorTestSuites`, `wasInterrupted`, and `snapshot.failure`. (`anyReporterErrors` has no other trace on the result and is simply unavailable.)
+
+- **`numFailedTestSuites` already counts the suites that failed to run, so never add `numRuntimeErrorTestSuites` to it.** `helpers.lua` `addResult` increments `numRuntimeErrorTestSuites` for any suite with a `testExecError`, and then increments `numFailedTestSuites` through the `numFailingTests > 0 or testExecError` branch of a *separate* statement. Summing the two reports one broken suite as two — and `2 of 1 suite(s) failed` for a single-spec package. Report `numFailedTestSuites`, and keep `numRuntimeErrorTestSuites` as its own verdict term: a suite that is both skipped and broken lands in `numPendingTestSuites`, so it is the only count that sees it.
+
+- **Validate jest's counts with the invariant, not with a list of field names.** `total == passed + failed + skipped` holds by construction — `helpers.lua:91` derives `numTotalTests` as `numPassingTests + numFailingTests + numPendingTests + numTodoTests` and every aggregate accumulates the same per-suite fields. Checking the sum catches a rename or a move of *any* count, including ones the checking code does not know the name of; validating names one at a time only ever covers the names known when it was written. This matters because a missing count reads as `0`, and zero failures over zero tests is spelled exactly like a clean run. `NevermoreTestResults.fromJest` asserts the sum and fails closed on a mismatch.
+
+- **A failure reason must never be built by formatting counts that may be zero.** The bug above surfaced as the reason `0 test(s) and 0 test suite(s) failed` — a string that asserts nothing failed while failing the package, and which reads as neither a pass nor a failure. `_resultsFromJest` now assembles the reason from the causes that actually hold and falls back to naming its own confusion, so a reason that says nothing failed is unconstructible. Both CLI readers additionally warn when a run reports failure over counts where nothing failed: clean counts across every package with every package failed is the signature of a wrong verdict, not of broken tests.
+
+- **A module-scope `require("Jest")` makes everything in the file untestable.** All three bugs above shipped green out of one file, because `NevermoreTestRunnerUtils` requires Jest at module scope and so cannot load in any harness that does not have Jest and the Nevermore loader. The verdict logic now lives in `NevermoreTestResults`, which requires nothing at all, takes plain tables, returns plain tables and does no logging — so it runs under Lune (`cd src/nevermore-test-runner && npm test`). If a package's logic keeps producing bugs a test would catch, look at what its file drags in before writing another integration check. Note a dependency-free Luau module does not need the loader line, and 312 of the repo's ~1250 package modules already omit it.
+
+- **Documenting a *local* Luau function with `--[=[ … ]=]` breaks `lint:moonwave`.** moonwave-extractor treats any `--[=[` block as a doc comment and rejects one it cannot attach to a class: `error: Function requires @within tag`. It also aborts on the first diagnostic, so one bad block hides every other. Use plain `--[[ … ]]` for local helpers and keep `--[=[ … ]=]` for public members (or add `@within` explicitly, as `@prop` blocks do).
+
+- **A structured channel that is plumbed but not flowing looks exactly like one that works.** The first version of the results-return path shipped inert: the runner returned its table, the batch runner captured it, the parser preferred it — and because the counts were all zero and nothing said where they came from, the verdict silently fell back to log scraping and every check passed. Both readers now state provenance unconditionally (`countsSource`, plus an `info` line naming how many packages returned counts), warn when a run fell back to scraping, and warn when the two channels disagree about the same run's totals. When adding a channel that has a fallback, make using the fallback louder than using the channel.
+
 - **`--script-text` loses everything after the first line when invoked through `npx` on Windows**: the `npx.cmd` shim truncates a multi-line argument, so `nevermore test --cloud --script-text '<line 1>\n<line 2>'` silently runs only line 1 (and prints `(no output)` when line 1 produced none). Either write the script as a single line with `;` separators, or bypass the shim: `node tools/nevermore-cli/dist/nevermore.js test --cloud --script-text '...'`, which passes newlines through intact.
 
 ## Claude Code hooks

docs/testing/testing.md

@@ -394,20 +394,29 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 ```
 
 Replace `mypackage` with the key used in your Rojo project tree.
 
+Returning `results` is what tells the CLI how the run went. Test output is log text and the
+engine truncates a long run's logs, so counts scraped back out of it are exactly what goes
+missing on the runs where they matter most; returning the table sends them out as a value
+instead. A script that returns nothing still works — the CLI falls back to reading the logs.
+
 ### NevermoreTestRunnerUtils
 
 The `@quenty/nevermore-test-runner` package provides `NevermoreTestRunnerUtils`, which handles the test execution lifecycle:
 
 - If a `jest.config` is found under the given root, it runs Jest tests
 - If no `jest.config` is found, boot success is the test (smoke test)
 - Detects Open Cloud vs local execution context and exits appropriately
+- Returns a [TestRunResults](/api/NevermoreTestRunnerUtils) table — counts, a capped failure
+  list, and the run's verdict — or `nil` when no test run was attempted, which is how a real
+  game server tells itself apart from a test place and falls through to its normal boot
 
 ## Running tests
 

package.json

@@ -23,7 +23,7 @@
     "build:ts": "pnpm -r --filter \"./tools/**\" --filter \"!./tools/nevermore-vscode\" run build",
     "format": "stylua --config-path=stylua.toml src games plugins",
     "format:ts": "prettier --ignore-path .gitignore --write \"tools/**/*.{ts,tsx,js,jsx}\"",
-    "lint:luau": "luau-lsp analyze --sourcemap=sourcemap.json --base-luaurc=.luaurc --defs=globalTypes.d.lua --flag:LuauSolverV2=false --ignore=**/node_modules/** --ignore=**/*.story.lua --ignore=**/*.client.lua --ignore=**/*.server.lua src",
+    "lint:luau": "luau-lsp analyze --sourcemap=sourcemap.json --base-luaurc=.luaurc --defs=globalTypes.d.lua --flag:LuauSolverV2=false --ignore=**/node_modules/** --ignore=**/*.story.lua --ignore=**/*.test.luau --ignore=**/*.client.lua --ignore=**/*.server.lua src",
     "lint:moonwave": "npx lerna exec --parallel -- moonwave-extractor extract src",
     "lint:prettier": "prettier --ignore-path .gitignore --check \"tools/**/*.{ts,tsx,js,jsx}\"",
     "lint:selene": "npx lerna exec --parallel -- selene --no-summary --num-threads=1 --config=../../selene.toml src",
@@ -55,4 +55,4 @@
       "released"
     ]
   }
-}
+}

src/access/test/scripts/Server/ServerMain.server.lua

@@ -8,8 +8,9 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()

src/animations/test/scripts/Server/ServerMain.server.lua

@@ -10,6 +10,7 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/binder/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/blend/test/scripts/Server/ServerMain.server.lua

@@ -11,6 +11,7 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/brine/test/scripts/Server/ServerMain.server.lua

@@ -8,8 +8,9 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()

src/brio/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/camera/test/scripts/Server/ServerMain.server.lua

@@ -11,6 +11,7 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/characterutils/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/chatproviderservice/test/scripts/Server/ServerMain.server.lua

@@ -10,8 +10,9 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()

src/clienttranslator/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/clipcharacters/test/scripts/Server/ServerMain.server.lua

@@ -10,8 +10,9 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()

src/cmdrservice/test/scripts/Server/ServerMain.server.lua

@@ -11,8 +11,9 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()

src/conditions/test/scripts/Server/ServerMain.server.lua

@@ -10,8 +10,9 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local AdorneeConditionUtils = require("AdorneeConditionUtils")

src/coreguienabler/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/datastore/test/scripts/Server/ServerMain.server.lua

@@ -12,8 +12,9 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local Maid = require("Maid")

src/deathreport/test/scripts/Server/ServerMain.server.lua

@@ -11,8 +11,9 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()

src/ellipticcurvecryptography/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/elo/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/experiencecalculator/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/fakeskybox/test/scripts/Server/ServerMain.server.lua

@@ -8,8 +8,9 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()

src/friendutils/test/scripts/Server/ServerMain.server.lua

@@ -7,6 +7,7 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/fzy/test/scripts/Server/ServerMain.server.lua

@@ -8,6 +8,7 @@ local loader = root:FindFirstChild("LoaderUtils", true).Parent
 local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end

src/gameconfig/test/scripts/Server/ServerMain.server.lua

@@ -11,8 +11,9 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()

src/gameproductservice/test/scripts/Server/ServerMain.server.lua

@@ -11,8 +11,9 @@ local require = require(loader).bootstrapGame(root)
 
 local NevermoreTestRunnerUtils = require("NevermoreTestRunnerUtils")
 
-if NevermoreTestRunnerUtils.runTestsIfNeededAsync(root) then
-	return
+local results = NevermoreTestRunnerUtils.runTestsIfNeededAsync(root)
+if results then
+	return results
 end
 
 local serviceBag = require("ServiceBag").new()