diff --git a/CHANGELOG.md b/CHANGELOG.md index 56a5623..10c94ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## 0.4.0a2 — pytest outcome correction + +- Record final pytest outcomes after expected-failure processing; mixed suites containing xfail or xpass remain inconclusive. +- Preserve ordinary assertion failures, runtime errors, and existing unittest behavior. +- Align the package, CLI, and report version. + ## 0.4.0a1 — public alpha - Compare selected unittest or pytest cases on a Git baseline and a committed or working-tree fix. diff --git a/README.md b/README.md index 060886f..2838e9e 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ It runs the same selected Python tests against a Git baseline and your current code, including uncommitted edits and new, non-ignored files. It records individual test outcomes, source hashes, snapshots, and raw logs. -Version 0.4.0a1 is a public alpha for Python projects with a known pre-fix Git +Version 0.4.0a2 is a public alpha for Python projects with a known pre-fix Git commit. It has no runtime package dependencies, account, or model API. It has been tested locally on Windows with Python 3.12; see [VERIFICATION.md](VERIFICATION.md) for the exact results and current limits. @@ -19,12 +19,12 @@ for the exact results and current limits. Requires Python 3.11+ and Git. Install the alpha from its tagged source: ```sh -python -m pip install "git+https://github.com/agammann/bugbix.git@v0.4.0-alpha.1" +python -m pip install "git+https://github.com/agammann/bugbix.git@v0.4.0-alpha.2" bugbix --version ``` -Alternatively, download the wheel from the [alpha release](https://github.com/agammann/bugbix/releases/tag/v0.4.0-alpha.1) -and install that file with `python -m pip install /path/to/bugbix-0.4.0a1-py3-none-any.whl`. +Alternatively, download the wheel from the [alpha release](https://github.com/agammann/bugbix/releases/tag/v0.4.0-alpha.2) +and install that file with `python -m pip install /path/to/bugbix-0.4.0a2-py3-none-any.whl`. Bugbix is not published to PyPI. ## Quick start @@ -91,9 +91,10 @@ Exit `0` means `regression_observed`; exit `1` means another completed verdict; exit `2` means configuration, snapshot, or orchestration failure. CLI output is JSON. For `inconclusive`, the CLI and report name the blocking run, test, and exception type when available. Runner failures include a short cause; the full trace remains -in the evidence folder. An assertion failure in one test does not override an error -or skip in another selected test. Select the relevant test and rerun, or repair -the test error. The report lists Git-visible changed files and links both stdout +in the evidence folder. An assertion failure in one test does not override an error, +skip, expected failure (`xfail`), or unexpected pass (`xpass`) in another selected +test. Remove an obsolete expected-failure marker when checking a fixed test. Select the relevant test and rerun, or +repair the test error. The report lists Git-visible changed files and links both stdout and stderr. A failure-to-pass transition cannot be reported as a regression when the only Git-visible changes are test files. Test-file recognition uses common Python names and `test/` or `tests/` folders; review the changed-file list. diff --git a/VERIFICATION.md b/VERIFICATION.md index f6bc697..b1efc4b 100644 --- a/VERIFICATION.md +++ b/VERIFICATION.md @@ -5,7 +5,29 @@ known pre-fix Git commit from a fix. This record separates local observations from broader product claims. The project does not yet have independent user adoption evidence. -## Local product checks +## 0.4.0a2 checks — 2026-10-02 + +Windows 11, CPython 3.12.14, pytest 9.1.1: + +- All 22 product tests passed without skips, including mixed ordinary/expected + failures and strict/non-strict unexpected passes. +- Before the fix, a mixed pytest suite incorrectly returned `regression_observed` + and recorded an xfail-to-xpass case as an ordinary failure-to-pass transition. + The corrected runner reads pytest's final report and returns `inconclusive`. +- A wheel was installed in a separate virtual environment and run outside the + source checkout. Its six Python module files matched the source. The package, + CLI, and report version agreed on `0.4.0a2`. +- The installed CLI passed five disposable example checks: a real assertion + transition, a weak test, an import error, a skipped test, and a timeout. +- The installed CLI checked Bugbix's new mixed-xfail regression test against + baseline `3176427c3744009772c5be2bc1a276b8464c3fa7`: two assertion failures + before the fix, two passes after it, yielding `regression_observed`. + +These are local controlled checks. The earlier third-party repository trials +below were not rerun for this correction. Arbitrary pytest plugins and detached +background processes remain outside the verified scope. + +## Earlier alpha checks — 2026-09-27 Test host: Windows 11 build 26200, CPython 3.12.14, 2026-09-27. The source suite exercises positive and negative verdicts, pytest and unittest, skipped @@ -15,7 +37,7 @@ in a separate virtual environment and invoked outside the source tree. | Check | Observed result | | --- | --- | -| [Source suite on the alpha source](evidence/alpha-source-tests.txt) | 20 tests passed | +| [Source suite on the first alpha source](evidence/alpha-source-tests.txt) | 20 tests passed | | Installed wheel smoke check | `bugbix --version` returned `Bugbix 0.4.0a1` | | Package consistency | Wheel Python source bytes matched the files in this repository | @@ -55,7 +77,7 @@ python -m pip wheel --no-deps --wheel-dir dist . ``` The GitHub Actions workflow runs the suite on Windows and Ubuntu with Python -3.12. A passing workflow is separate evidence from the local results above; +3.11 and 3.12. A passing workflow is separate evidence from the local results above; check the latest run before assuming either platform passed remotely. ## Boundaries diff --git a/bugbix/__init__.py b/bugbix/__init__.py index d1ce10b..1691e44 100644 --- a/bugbix/__init__.py +++ b/bugbix/__init__.py @@ -1,3 +1,3 @@ """Bugbix: regression evidence for the change in front of you.""" -__version__ = "0.1.0" +__version__ = "0.4.0a2" diff --git a/bugbix/cli.py b/bugbix/cli.py index 5b729d3..d9d53d6 100644 --- a/bugbix/cli.py +++ b/bugbix/cli.py @@ -3,12 +3,13 @@ import subprocess import sys +from . import __version__ from .core import check def main(argv=None): parser = argparse.ArgumentParser(description="Does your regression test catch the old bug?") - parser.add_argument("--version", action="version", version="Bugbix 0.4.0a1") + parser.add_argument("--version", action="version", version=f"Bugbix {__version__}") commands = parser.add_subparsers(dest="command", required=True) run = commands.add_parser("check", help="Compare selected tests on old and current code") run.add_argument("--repo", default=".") diff --git a/bugbix/core.py b/bugbix/core.py index c22c37f..414e4f7 100644 --- a/bugbix/core.py +++ b/bugbix/core.py @@ -13,6 +13,8 @@ import sys import time +from . import __version__ + MAX_FILE = 10 * 1024 * 1024 MAX_TOTAL = 100 * 1024 * 1024 @@ -310,7 +312,7 @@ def check(repo, base, head, tests, output, python=sys.executable, timeout=30.0, blockers = [{"run": "all", "issue": "no_non_test_changes"}] else: blockers = blockers_for(runs) if verdict == "inconclusive" else [] - report = {"schema_version": 2, "product": "Bugbix", "version": "0.4.0a1", + report = {"schema_version": 2, "product": "Bugbix", "version": __version__, "verdict": verdict, "reason": reason, "regression_tests": transitions, "blockers": blockers, "changed_files": changed_files, "repo": str(repo), "base": base_oid, "head": head_oid, "tests": tests, diff --git a/bugbix/pytest_runner.py b/bugbix/pytest_runner.py index a3d8c22..4b9c4d1 100644 --- a/bugbix/pytest_runner.py +++ b/bugbix/pytest_runner.py @@ -28,12 +28,9 @@ def pytest_collectreport(self, report): if report.failed: self.collection_errors.append(report.nodeid) - def pytest_runtest_makereport(self, item, call): - # This hook sees the original exception, allowing an assertion failure - # to be distinguished from a setup/runtime error. - import pytest - - report = pytest.TestReport.from_item_and_call(item, call) + def record_report(self, report, call): + # Inspect the final report after pytest has applied xfail and other hooks, + # while retaining the original exception for assertion/error distinction. if report.failed: if hasattr(report, "wasxfail"): status = "unexpected_success" @@ -46,8 +43,8 @@ def pytest_runtest_makereport(self, item, call): elif report.skipped: self.record(report.nodeid, "expected_failure" if hasattr(report, "wasxfail") else "skip") elif call.when == "call": - self.record(report.nodeid, "pass") - return report + self.record(report.nodeid, + "unexpected_success" if hasattr(report, "wasxfail") else "pass") def main(): @@ -58,7 +55,13 @@ def main(): try: import pytest - recorder = Recorder() + class Plugin(Recorder): + @pytest.hookimpl(hookwrapper=True, tryfirst=True) + def pytest_runtest_makereport(self, item, call): + outcome = yield + self.record_report(outcome.get_result(), call) + + recorder = Plugin() exit_code = int(pytest.main(["-q", "-p", "no:cacheprovider", *paths], plugins=[recorder])) payload = {"events": [{"id": nodeid, "status": status, **({"error_type": recorder.error_types[nodeid]} diff --git a/pyproject.toml b/pyproject.toml index a04f642..1ff42ff 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "bugbix" -version = "0.4.0a1" +version = "0.4.0a2" description = "Check whether a regression test catches the bug before your fix." requires-python = ">=3.11" readme = "README.md" diff --git a/tests/test_product.py b/tests/test_product.py index 3bd83fc..3c46819 100644 --- a/tests/test_product.py +++ b/tests/test_product.py @@ -177,6 +177,35 @@ def test_pytest_function_catches_dirty_tree_fix(self): self.root / "focused", repeat=1, runner="pytest") self.assertEqual(focused["verdict"], "regression_observed") + def test_pytest_xfail_outcomes_block_mixed_regression(self): + try: + import pytest # noqa: F401 + except ImportError: + self.skipTest("pytest is not installed in this test environment") + self.change_test("import pytest\nfrom pricing import total\n" + "def test_discount():\n assert total(100, 20, 10) == 88\n" + "@pytest.mark.xfail(reason='pending', strict=False)\n" + "def test_marked():\n assert total(100, 20, 10) == 88\n") + report = self.run_check(runner="pytest") + self.assertEqual(report["verdict"], "inconclusive") + outcomes = {(item["run"], item.get("status")) for item in report["blockers"]} + self.assertIn(("base-1", "expected_failure"), outcomes) + self.assertIn(("head-1", "unexpected_success"), outcomes) + + def test_pytest_xpass_does_not_count_as_an_ordinary_pass(self): + try: + import pytest # noqa: F401 + except ImportError: + self.skipTest("pytest is not installed in this test environment") + for strict in (False, True): + with self.subTest(strict=strict): + self.change_test("import pytest\n" + f"@pytest.mark.xfail(reason='pending', strict={strict})\n" + "def test_marked():\n assert True\n") + report = check(self.repo, "HEAD", "working", ["test_pricing.py"], + self.root / f"xpass-{strict}", repeat=1, runner="pytest") + self.assertEqual(report["verdict"], "inconclusive") + def test_pytest_runtime_error_is_inconclusive(self): try: import pytest # noqa: F401