[Bug]: CI skips 85 tests because unittest discovery cannot collect module-level pytest functions

[Hasnain2430]

Primary affected area

Other or unsure — continuous integration and test collection.

Affected capability

Not capability-specific.

What happened?

CI discovers tests with python -m unittest discover -s tests, which collects only unittest.TestCase subclasses. Eight test files define module-level pytest functions instead, so unittest discovery collects nothing from them and reports success. 85 tests never run in CI, with no warning that they were skipped.

The counts reconcile exactly:

Runner	Tests
python -m unittest discover -s tests -q (what CI runs)	655
python -m pytest	740
Difference	85

Per file:

File	pytest collects	unittest collects
tests/test_upload_service.py	48	0
tests/test_codex_plugin.py	10	0
tests/test_agent_ablation.py	7	0
tests/test_artifact_download.py	7	0
tests/test_tusd_hooks.py	6	0
tests/test_native_ingestion.py	4	0
tests/test_upload_page.py	2	0
tests/test_tusd_live.py	1	0
Total	85	0

The uncovered set is not peripheral. It is most of the media-ingestion path: the upload service, the tus hooks, the upload page, artifact download, and the end-to-end native ingestion tests that assert an imported video becomes searchable. Two of those test_native_ingestion.py tests currently fail on a local Windows checkout with AssertionError: Autonomous indexing did not finish without status polling, which is the kind of regression this gap is designed to hide. I have not yet established whether that failure is environmental, and I am not reporting it as a defect here.

This also means the contributor validation instructions and CI exercise different test sets. docs/CONTRIBUTING.md directs contributors to run the suite with pytest, so a contributor can see a failure that CI will never reproduce, and CI can pass a change that the documented command would have caught. (Separately, the documented command needed a correction, which I sent as #155.)

How can we reproduce it?

uv run --no-sync python -m unittest discover -s tests -p "test_upload_service.py" -q
# Ran 0 tests in 0.000s
# NO TESTS RAN

uv run --no-sync python -m pytest -q --collect-only tests/test_upload_service.py
# 48 tests collected

The same holds for each of the eight files above. To see the aggregate:

uv run --no-sync python -m unittest discover -s tests -q      # Ran 655 tests
uv run --no-sync python -m pytest -q --collect-only           # 740 tests collected

Environment

• VidXP version: 0.4.0, source checkout at 24b916a
• Operating system: Windows 11
• Installation: source, uv sync --frozen with the contributor extras
• Python: 3.12.14 in the project venv
• CI reference: .github/workflows/ci.yml, the python -m unittest discover -s tests -q step

The collection gap itself is platform-independent: it follows from the discovery mechanism, not from the operating system.

Logs or other details

Two directions, whichever you prefer:

1. Run the suite in CI with python -m pytest instead. It collects both styles, so all 740 tests run and the 85 stop being invisible. Lowest-effort fix, but it changes the runner CI standardizes on.
2. Keep unittest as the runner and add a collection guard — a test that asserts the discovered count matches the number of test functions under tests/, so a file written in the unsupported style fails loudly instead of silently.

I am happy to send either as a pull request, but the choice of runner is a maintainer decision so I would rather ask first than guess. Option 2 is the smaller change and does not disturb the existing CI contract.

Found while fixing two Windows path assertions (#154), one of which lives in a file CI never runs.

[Hasnain2430]

Following up to close out the one open question in the report: the two test_native_ingestion.py failures I mentioned were environmental, not a defect. They pass once the scene model artifacts are prepared.

uv run --no-sync vidxp prepare --modalities scene --yes
uv run --no-sync python -m pytest -q tests/test_native_ingestion.py
4 passed in 104.31s

So there is no ingestion bug here, and I am withdrawing that part of the report. The collection gap itself is unaffected: those 4 tests still never run in CI, along with the other 81.

Two smaller observations from confirming this, in case either is useful:

• The failure message a contributor sees when model artifacts are missing is AssertionError: Autonomous indexing did not finish without status polling, which does not point at the cause. docs/CONTRIBUTING.md tells contributors that an unprepared fresh checkout is expected and then directs them to run the suite, so this is the first thing some contributors will hit. A precondition check that skips or fails with "scene model artifacts are not prepared" would save the diagnosis. Happy to send that if you want it.
• vidxp prepare --modalities scene prompts for confirmation and exits 0 without downloading anything when stdin is not a terminal. In a scripted or CI setting that reads as success, and the missing models only surface later as the misleading assertion above. --yes is the correct flag and works; the exit code on a declined-by-EOF prompt may still be worth a look.

I can open either of those as its own issue rather than expanding this one, if you would prefer to keep this thread on the collection gap.

[Hasnain2430]

Correction to my previous comment: the second observation was wrong, and I withdraw it.

I claimed vidxp prepare exits 0 when it declines at the confirmation prompt with no terminal on stdin. It does not. typer.confirm(..., abort=True) raises typer.Abort, which Typer turns into exit code 1:

uv run --no-sync vidxp prepare --modalities action > /dev/null 2>&1; echo $?
1

The exit 0 I reported came from my own shell pipeline. I had piped the command into tail, so the status I read was tail's, not the command's. The behavior is correct: the prompt refuses to proceed without confirmation, --yes is the documented way through, and the exit status reflects the refusal. Apologies for the noise.

My first observation stands, and it is the smaller of the two: when scene model artifacts are missing, test_native_ingestion.py fails with AssertionError: Autonomous indexing did not finish without status polling, which does not name the cause. That is the only thing from that comment worth acting on, and it is minor next to the collection gap in the report above.