Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 41 additions & 5 deletions .github/workflows/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,61 @@ permissions:

on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: bugbix-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false

jobs:
test:
name: ${{ matrix.os }} / Python ${{ matrix.python-version }}
runs-on: ${{ matrix.os }}
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
os: [windows-latest, ubuntu-latest]
python-version: ['3.11', '3.12']
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: ${{ matrix.python-version }}
- name: Install test dependency
run: python -m pip install pytest
run: python -m pip install pytest==9.1.1 build==1.6.1
- name: Run product tests
run: python -m unittest discover -s tests -v
- name: Build wheel
run: python -m pip wheel --no-deps --wheel-dir dist .
- name: Build and verify wheel and source archive
run: python scripts/package.py
- name: Install actual artifacts and verify consumer/recovery
run: python scripts/consumer.py --out ../bugbix-clean-consumer
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
with:
name: bugbix-packages
path: artifacts/
if-no-files-found: error
overwrite: true
publish:
needs: test
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.sha }}
persist-credentials: false
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: bugbix-packages
path: release-artifacts
- uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
script: |
const publish = require('./scripts/release.cjs');
await publish({ github, context, core });
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@ __pycache__/
build/
dist/
.venv/
artifacts/
reports/
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Changelog

## 1.0.0

2026-10-06

- Establish the v1 CLI, verdict/exit and report schema 2 stability contract for selected Python regression checks.
- Retain the verified unittest/pytest engine, final xfail/xpass handling and original-checkout preservation behavior.
- Deliver a prebuilt wheel and installable source archive with MIT licensing and SHA256 checksums.
- Verify both actual artifacts in fresh environments, including positive/negative verdicts, recovery and checkout preservation.
- Gate main-branch release publication on the existing Windows/Linux and Python 3.11/3.12 checks and complete artifact verification.

## 0.4.0a2 — pytest outcome correction

- Record final pytest outcomes after expected-failure processing; mixed suites containing xfail or xpass remain inconclusive.
Expand Down
17 changes: 15 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing

Bugbix is an experimental Python command-line tool. Focused bug reports and
Bugbix is a Python command-line tool for bounded regression evidence. Focused bug reports and
small changes that improve its stated use case are welcome.

Before opening an issue, include the Bugbix version, Python and Git versions,
Expand All @@ -11,7 +11,7 @@ raw test output, which may include private paths or other sensitive data.

For code changes:

1. Run `python -m pip install pytest` in a disposable environment.
1. Run `python -m pip install pytest==9.1.1 build==1.6.1` in a disposable environment.
2. Run `python -m unittest discover -s tests -v`.
3. Add a test for a changed verdict or runner behavior when it protects against
a concrete failure.
Expand All @@ -21,3 +21,16 @@ For code changes:
The current scope is selected Python unittest and pytest tests against a known
pre-fix Git commit. Discuss broader runner or language support before building
it.

## Build and verify the delivered artifacts

```sh
python scripts/package.py
python scripts/consumer.py --out ../bugbix-consumer-check
```

Choose a new consumer directory outside the source checkout. Packaging verifies runtime source bytes, CLI entry point, version, MIT license and archive contents, then writes the wheel/source archive and checksums under ignored `artifacts/`. The consumer installs each actual artifact in its own fresh virtual environment, exercises the installed CLI outside source, verifies positive/negative verdicts and exit codes, preserves the dirty fixture checkout, and tests recovery into a new output directory. Reports remain in the consumer directory.

The workflow retains Windows/Linux and Python 3.11/3.12 checks. Pull requests and manual dispatches have read-only permissions. Main pushes publish only after all four lanes pass, using the same-run verified wheel/source artifact. The publisher checks the exact five-file package/checksum set, current main/tag commit, matching draft, and complete uploaded digests/sizes before publishing. Published versions remain unchanged; same-commit draft retries can resume.

Follow the [v1 contract](docs/stability.md). Add concrete regression coverage for changes to engine behavior; documentation/version packaging changes need actual installation checks rather than duplicate engine tests.
6 changes: 6 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
include LICENSE README.md CHANGELOG.md CONTRIBUTING.md VERIFICATION.md DIRECTION.md NAMING.md
include evidence/alpha-source-tests.txt
recursive-include bugbix *.py
recursive-include tests *.py
recursive-include docs *.md
recursive-include scripts *.py *.cjs
30 changes: 19 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,23 +9,29 @@ It runs the same selected Python tests against a Git baseline and your current
code, including uncommitted edits and new, non-ignored files. It records individual
test outcomes, source hashes, snapshots, and raw logs.

Version 0.4.0a2 is a public alpha for Python projects with a known pre-fix Git
commit. It has no runtime package dependencies, account, or model API. It has
Version 1.0.0 supports selected Python tests against a known pre-fix Git
commit, with a documented [v1 CLI and report contract](docs/stability.md). It has no runtime package dependencies, account, or model API. It has
been tested locally on Windows with Python 3.12; see [VERIFICATION.md](VERIFICATION.md)
for the exact results and current limits.

## Install

Requires Python 3.11+ and Git. Install the alpha from its tagged source:
Requires Python 3.11+ and Git. Download the wheel and `SHA256SUMS` from the [v1.0.0 release](https://github.com/agammann/bugbix/releases/tag/v1.0.0). Check the wheel's SHA256 before installing it. On PowerShell, use `Get-FileHash <wheel> -Algorithm SHA256`; on Linux, use `sha256sum -c SHA256SUMS` after downloading both artifacts listed in the manifest.

```sh
python -m pip install "git+https://github.com/agammann/bugbix.git@v0.4.0-alpha.2"
python -m pip install /path/to/bugbix-1.0.0-py3-none-any.whl
bugbix --version
```

Alternatively, download the wheel from the [alpha release](https://github.com/agammann/bugbix/releases/tag/v0.4.0-alpha.2)
and install that file with `python -m pip install /path/to/bugbix-0.4.0a2-py3-none-any.whl`.
Bugbix is not published to PyPI.
Expect `Bugbix 1.0.0`. The wheel is prebuilt and has no runtime package dependencies. Bugbix is not published to PyPI.

For a source installation, download `bugbix-1.0.0.tar.gz` from the same release, verify its checksum, and run:

```sh
python -m pip install /path/to/bugbix-1.0.0.tar.gz
```

Pip builds the source package using its declared setuptools backend; obtaining build dependencies may need network access. The resulting runtime remains dependency-free. Keep wheels/source archives from one version together and check the installed CLI version after upgrading. Source and distributions are [MIT licensed](LICENSE), copyright 2026 agammann.

## Quick start

Expand Down Expand Up @@ -88,7 +94,7 @@ repository root and its `src/` directory first.
| `unstable` | Repeated runs produced different outcomes. |

Exit `0` means `regression_observed`; exit `1` means another completed verdict;
exit `2` means configuration, snapshot, or orchestration failure. CLI output is JSON.
exit `2` means configuration, snapshot, or orchestration failure. Parsed check output is JSON; command syntax errors print argparse diagnostics to stderr with exit 2.
For `inconclusive`, the CLI and report name the blocking run, test, and exception
type when available. Runner failures include a short cause; the full trace remains
in the evidence folder. An assertion failure in one test does not override an error,
Expand All @@ -101,7 +107,7 @@ Python names and `test/` or `tests/` folders; review the changed-file list.

Each evidence folder contains `report.md`, `report.json`, and per-run snapshots,
stdout, stderr, and structured test results. Default: two runs per version, a
30-second timeout per run. Configure `--repeat` (1–10) and `--timeout` (up to 3600).
30-second timeout per run. Configure `--repeat` (1–10) and `--timeout` (up to 3600).

## How it works

Expand All @@ -117,7 +123,7 @@ must already work on the baseline. A missing helper produces `inconclusive`.

## Scope and limits

- This alpha supports selected Python test files or individual cases with unittest or pytest.
- Version 1 supports selected Python test files or individual cases with unittest or pytest.
Other languages and runners are not supported. Package-relative test imports
and project-specific pytest plugins may require additional setup.
- `--base` must be a real pre-fix commit. If `HEAD` already contains some or all
Expand Down Expand Up @@ -147,10 +153,12 @@ must already work on the baseline. A missing helper produces `inconclusive`.
## Run the product tests

```sh
python -m pip install pytest
python -m pip install pytest==9.1.1 build==1.6.1
python -m unittest discover -s tests -v
```

See [VERIFICATION.md](VERIFICATION.md) for observed results, [DIRECTION.md](DIRECTION.md)
for the product scope and alternatives, [CHANGELOG.md](CHANGELOG.md) for release
changes, and [CONTRIBUTING.md](CONTRIBUTING.md) for bug reports and contributions.

For interrupted or inconclusive checks, upgrades and removal, see [Recovery](docs/recovery.md). The [v1 stability contract](docs/stability.md) records the supported CLI, exit codes and report schema 2.
34 changes: 30 additions & 4 deletions VERIFICATION.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,35 @@
# Verification

The alpha's supported claim is narrow: selected Python tests can distinguish a
The supported claim is narrow: selected Python tests can distinguish a
known pre-fix Git commit from a fix. This record separates local observations
from broader product claims. The project does not yet have independent user
adoption evidence.

## 1.0.0 release checks — 2026-10-06

Windows, CPython 3.12.14, pytest 9.1.1, build 1.6.1 and setuptools 84.0.0:

- All 22 product tests passed without skips. The regression-check engine is
unchanged from the current 0.4.0a2 source; the runtime version is now 1.0.0.
- The wheel and source archive were built and checked for matching version,
six runtime modules, MIT license, entry point and zero runtime dependencies.
The source archive also includes the CLI/report contract and recovery guide.
- Each actual distribution was installed in its own fresh virtual environment
outside the source checkout. The installed CLI reported 1.0.0, imported from
that environment and matched all six runtime module bytes.
- Each installed distribution passed nine controlled consumer cases: a real
assertion transition, a weak test, a broken fix, an import error, refusal to
overwrite existing evidence, recovery into a new output folder, a skipped
test, a timeout and a command syntax error. Help also exited successfully.
The checks preserved the original fixture checkout, branch, index and files;
refusal to reuse an output directory preserved the existing evidence bytes.

These are current local checks. The workflow requires the source suite, package
checks and both clean distribution consumers on Windows and Ubuntu with Python
3.11 and 3.12 before main can publish the five release assets. Check the actual
workflow and release before assuming remote success or availability. The dated
alpha and third-party records below are historical and were not rerun for v1.

## 0.4.0a2 checks — 2026-10-02

Windows 11, CPython 3.12.14, pytest 9.1.1:
Expand Down Expand Up @@ -71,9 +96,10 @@ error as a successful regression check.
## Reproduce the product suite

```sh
python -m pip install pytest
python -m pip install pytest==9.1.1 build==1.6.1
python -m unittest discover -s tests -v
python -m pip wheel --no-deps --wheel-dir dist .
python scripts/package.py
python scripts/consumer.py --out ../bugbix-clean-consumer
```

The GitHub Actions workflow runs the suite on Windows and Ubuntu with Python
Expand All @@ -86,5 +112,5 @@ Only Windows and Python 3.12 have been verified locally. The third-party runs
were focused tests, not complete upstream test suites. The two successful
projects demonstrate technical utility, not demand. Generated files, external
services, untracked prior baselines, and other languages remain outside this
alpha's verified scope. Test code runs with the caller's permissions; the
verified scope. Test code runs with the caller's permissions; the
snapshot directories are not a security sandbox.
2 changes: 1 addition & 1 deletion bugbix/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Bugbix: regression evidence for the change in front of you."""

__version__ = "0.4.0a2"
__version__ = "1.0.0"
17 changes: 17 additions & 0 deletions docs/recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Recover a check and remove Bugbix

A completed `inconclusive` result has evidence. Read its blockers and the linked stdout/stderr before changing the test environment. A missing dependency, shared fixture, skipped/xfail case or runner error must be resolved in the selected interpreter or test selection. Keep the original report, correct the cause, and run again with a new `--out` directory. An existing output folder is refused with exit 2 and remains untouched.

For an interruption or orchestration failure, a partial evidence directory may lack a complete report. Retain it while investigating; it is not a completed verdict. Stop the interrupted test processes if necessary, check the original repository state, and rerun into a new directory. Bugbix's normal timeout handling stops the main test process tree. Intentionally detached processes remain outside the supported scope.

`--base` must identify the real pre-fix commit. If HEAD already includes the fix, choose an older saved commit. Bugbix cannot restore unsaved pre-fix working-tree state. Snapshot operations preserve the original checkout; tests themselves run with your permissions and can access paths/services outside their copied source. Use trusted code and review evidence before sharing it.

Reports contain source copies and raw output. Retention and sharing are your responsibility. You may archive a complete evidence directory as a unit; its Markdown links refer to the adjacent per-run directories. Removing an evidence directory does not change the tested checkout. Verify its absolute path before removing anything.

To remove the installed package, use its environment's interpreter:

```sh
python -m pip uninstall bugbix
```

An isolated virtual environment can be removed after preserving any evidence saved inside it. Bugbix has no account, daemon, scheduled task, telemetry or hosted resource to remove. Existing Git repositories and saved evidence remain independently usable.
17 changes: 17 additions & 0 deletions docs/stability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Bugbix v1 stability contract

Version 1.0.0 supports selected Python unittest or pytest cases against a known pre-fix Git commit and a current working tree or committed fix. Python 3.11+ and Git are required. The installed runtime has no package dependencies, account, model API or network service. Pytest and the tested project's dependencies belong in the interpreter selected by `--python`; Bugbix does not install them.

Within v1, `bugbix check` and the documented options (`--repo`, `--base`, `--head`, repeated `--test`, `--runner`, `--out`, `--python`, `--timeout`, `--repeat`) remain supported. Defaults remain working-tree head, unittest, two repetitions and 30 seconds per run. `--version` prints `Bugbix <version>` and exits 0. Help exits 0. Command syntax errors use argparse diagnostics on stderr and exit 2.

A parsed check prints JSON to stdout. Exit 0 means `regression_observed`; exit 1 means another completed verdict (`not_demonstrated`, `fix_fails`, `inconclusive`, `unstable`). Configuration, snapshot or orchestration errors exit 2 and include a JSON error. A failure-to-pass transition needs matching assertion outcomes, passing current tests and a non-test source change. Errors, skips, expected failures, unexpected passes, missing results, timeouts and input mutation prevent a qualifying regression. Repeated disagreement produces `unstable`.

Written `report.json` retains `schema_version: 2`. Verdict names and the documented top-level data remain supported within v1; readers should allow additional fields. Reports include source snapshots/hashes, selected tests, revisions, runs, blockers and limits. Timing, temporary paths, human-readable reason strings, raw traces and Markdown layout are observational output rather than machine parsing contracts. Old evidence is retained as produced; the CLI does not rewrite or migrate existing report folders.

Snapshot operations leave the original checkout, branch and index intact. Evidence is written to a new directory outside that repository; existing output folders are refused. Only selected current test files are transplanted onto the baseline, so shared fixtures/helpers must already work there. The documented file/size limits and rejection of submodules, symlinks and Git LFS pointers remain part of the supported scope. Tests execute with the caller's permissions and inherited dependencies; snapshots are not an execution sandbox or a hermetic environment.

Windows and Linux on Python 3.11/3.12 are the CI lanes. Local release checks use Windows Python 3.12.14 and pytest 9.1.1. Other interpreter versions/platforms, languages, arbitrary pytest plugins, detached background processes and full-suite/bug-relevance guarantees require separate evidence.

## Upgrade from 0.4.0a2

Install the v1 wheel or source archive in your environment and check `bugbix --version`. The check options, verdicts, exit codes and report schema 2 are unchanged. Keep older evidence folders for comparison; use a new output directory for every v1 run. No repository migration, service registration or account change is needed. Follow [recovery and removal](recovery.md) for interrupted/inconclusive runs and uninstalling the CLI.
7 changes: 4 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
[build-system]
requires = ["setuptools>=68"]
requires = ["setuptools>=77.0.3"]
build-backend = "setuptools.build_meta"

[project]
name = "bugbix"
version = "0.4.0a2"
version = "1.0.0"
description = "Check whether a regression test catches the bug before your fix."
requires-python = ">=3.11"
readme = "README.md"
authors = [{name = "agammann"}]
license = {text = "MIT"}
license = "MIT"
license-files = ["LICENSE"]
dependencies = []

[project.urls]
Expand Down
Loading
Loading