[WIP] Gaia charts#362
Open
mrosseel wants to merge 309 commits into
Open
Conversation
The pifinder systemd service has a restricted PATH that doesn't include sudo. Use /run/current-system/sw/bin/sudo for all calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These commands are called outside of sudo (which has secure_path), so they need full paths in the restricted pifinder service PATH. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bump nixpkgs input to nixos-25.11 - Update Python 3.12 → 3.13 across all references - Switch simplejpeg wheel to cp313 - Add explicit pyproject/build-system to 13 packages (required by 25.11) - Add dontCheckRuntimeDeps for packages with env-level deps - Rename hardware.pulseaudio → services.pulseaudio Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- u-boot SD image: use custom ubootSD with CONFIG_CMD_SYSBOOT, hardcoded load address, correct partition (0:2) - RPi.GPIO: patch cpuinfo.c for aarch64 where /proc/cpuinfo lacks Hardware line and u-boot replaces firmware DTB - I2C1: add overlay to fdtoverlay chain (nixos-hardware was bypassed) - sudo: substituteInPlace pifinder-src to use /run/wrappers/bin/sudo - polkit: add reboot/shutdown D-Bus rules for pifinder user - /boot: mount firmware at /boot/firmware so extlinux.conf works Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- u-boot: disable PCI/USB/NET probing (CONFIG_PREBOOT="", CONFIG_PCI=n, CONFIG_USB=n, etc.) — eliminates ~3-4s of unnecessary hardware scanning - boot-splash: move earlier with DefaultDependencies=false, poll for SPI device instead of relying on systemd restart - Disable NetworkManager-wait-online (7.1s savings, NM still works async) - pwm-permissions: no longer blocks pifinder startup (runs in parallel) - pifinder-src: pre-compile .pyc bytecode at build time Net result: splash ~7s (was 21s), menu ~45s (was 65s). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add bashInteractive to systemPackages (mkForce strips default bash) - Add download-buffer-size 512MB for large PiFinder repo tarball - Fix migration_progress OLED binary args to match expected format (<percent> <stage_num> <stage_total> <stage_name>) - Fix nixos-rebuild progress: count copying/building lines instead of parsing non-existent (N/M) format Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Unified Python dev shell: extract python-packages.nix as single source of truth for both Pi production and x86 dev environments, add flake devShell, replace old shell.nix + pip venv with `use flake` - Pygame keyboard capture: monkey-patch luma's _abort to intercept pygame KEYDOWN events, fixing arrow keys on Wayland where PyHotKey/pynput cannot access keyboard - Cedar detect: build from source instead of shipping prebuilt binaries, add Cargo.lock for reproducible builds - Software update UI: channel-based browsing (stable/beta/unstable), scrolling text for long labels, 7x square secret to unlock unstable channel, always show channels even when empty - Fix pre-commit hooks: replace nox/virtualenv with direct devShell tool invocation, fix mypy config for nix environment - Fix numpy.math removal in tetra3 (Python 3.13 compat) - Various Python 3.13 compatibility fixes across codebase Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix test mode images: drain debug commands before blocking capture() so mode switch is processed even when camera hardware is unresponsive - Fix PAM errors: create dedicated pifinder PAM service (auth-only, no session/account management that requires setuid) - Fix service PATH: add /run/wrappers/bin and /run/current-system/sw/bin to pifinder systemd service path (camera switch, sudo, systemctl) - Fix AP networking: enable firewall ports 53/67 UDP (DNS/DHCP) and 80 TCP (web UI), add dnsmasq for NM shared mode, AP autoconnect - Fix tetra3 rev pin: update to cded265 (np.math.factorial fix) - Remove bootstrap workflow (no longer used) - Remove absolute paths from sys_utils.py (service PATH handles resolution) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fetch catalog_images.tar.zst from files.miker.be during SD image build, unpack at build time, and copy writable into the root filesystem. Not included in the regular system closure (no Cachix bloat). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split pifinder-src into stable astro-data (~193MB) derivation + code (~57MB) for smaller OTA updates - Add bootstrapMode option: minimal NixOS without pifinder-src, runs nixos-rebuild on first boot for migration from Raspbian - Custom tarball builder (replaces make-system-tarball.nix) to avoid nix sandbox symlink conflict during closure copy - Bootstrap tarball: 648MB with boot firmware, nix closure, extlinux - Conditionally disable pifinder/cedar/watchdog services in bootstrap Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration tarball is now a complete v2.5.0 system — no two-stage first-boot rebuild needed. Users get a working PiFinder immediately after migration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
images/ was fully stripped but console.py and splash.py load images/welcome.png on startup. Keep that file, strip the rest. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After migration from Raspbian, the tarball includes /nix-path-registration with Nix store path data. This oneshot service loads it into the Nix DB on first boot so nix-store and nixos-rebuild work correctly. Uses ConditionPathExists so it's a no-op on normal boots. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace nixos-rebuild with direct nix store path deployment: - CI builds push to Cachix and stamp pifinder-build.json per branch - Upgrade service uses nix copy + switch-to-configuration (no OOM) - Software UI resolves store paths from per-branch build metadata - Delete versions.json (replaced by pifinder-build.json) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nix copy --from uses only the specified source store. If Cachix doesn't have every transitive dependency (e.g. shadow from nixos cache), the copy fails. nix build with --max-jobs 0 uses all configured substituters (Cachix + cache.nixos.org) as fallbacks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename nox.yml to lint.yml, use nix develop for ruff/mypy/pytest - Move ruff builtins config into pyproject.toml, target py313 - Add notebook lint exclusions (E402, F841) - Fix ambiguous variable name in software.py (l -> lbl) - Remove unused import in sys_utils.py - Delete noxfile.py, requirements.txt, requirements_dev.txt - Add DEPENDENCIES.md (auto-generated from nix) and generation script - Update .pre-commit-config.yaml with ruff lint+format hooks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
native-wait job polls build-native for 15 min on ubuntu-latest. If the self-hosted aarch64 runner doesn't pick up the job in time, build-emulated kicks in with QEMU instead of queuing forever. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add openFirewall = true to samba service - Remove manual port 22 (openssh.openFirewall defaults to true) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avahi fails to start after switch-to-configuration because the old PID file persists at /run/avahi-daemon/pid. Add ExecStartPre to remove it, preventing mDNS (pifinder.local) from breaking on upgrades. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DeterminateSystems/nix-installer-action defaults to determinate: true which requires FlakeHub auth. This fails on forks (brickbots/PiFinder) because OIDC tokens don't work cross-repo. Use upstream Nix instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrates brickbots/main (through brickbots#523) into the nixos branch. Conflict resolutions: - gps_ubx_parser.py / test_gps_ubx_parser.py: both branches carry the UBX NAV-SVINFO/NAV-SAT decoding fix (upstream brickbots#524). Kept nixos's later refinement that counts satellites at quality >= 2 (signal acquired) instead of >= 4 (code locked), which stopped the seen count flapping to zero during marginal re-acquisition. - ui/software.py: both branches independently built manifest-driven update logic. Kept nixos's _fetch_update_manifest (channels dict via _entry_from_manifest) as the single source and adapted upstream's _migration_version_info_from_manifest to consume it; threaded migration_url/migration_sha256_url through _entry_from_manifest so the migration-tarball selection still works. Both features retained. - test_software.py: unioned both suites; dropped upstream's obsolete TestFetchUpdateManifest (tested the removed raw-dict signature) and reshaped the migration fixtures to mirror real manifest entries. - nixos_migration.sh / nixos_migration_init.sh: took upstream's version. nixos deliberately dropped its Raspbian-side copies (brickbots#521); the canonical pre-migration implementation lives on main.
The update manifest marked a build available: true whenever store_path merely looked like a valid Nix store path, never checking the binary cache. A build whose attic push failed (or was GC'd) was still advertised, so a device selecting it hit 'no substituter can build it' and stranded the upgrade. set_available now probes each cache's narinfo and only advertises a build when one actually serves the closure; otherwise available: false with reason 'not in cache' / 'cache unreachable' (the UI already blocks these). Defaults to the dev + release caches, so the gate is active without any workflow change. --skip-cache-check retained for offline/tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PiFinder's imx462 module has a 74.25 MHz oscillator; on Raspbian the
imx290 overlay parameter clock-frequency=74250000 sets this, but
fdtoverlay cannot apply overlay parameters, so the NixOS DTB kept the
37.125 MHz default — wrong INCKSEL/PLL programming, sensor enumerates
but never streams ("Camera frontend has timed out"). Apply a compiled
override overlay (cam1_clk + sensor node) after the camera overlay.
No kernel change: nixos-hardware's raspberry-pi-4 module already pins
the RPi vendor kernel (stable_20250916), whose imx290 driver carries
imx462 support. ADR 0008 rewritten accordingly.
The nixos branch replaced nox + requirements*.txt with uv, but an
upstream merge kept .github/workflows/nox.yml while dropping
python/noxfile.py, so the nox check failed on every nixos-line PR
("Failed to load Noxfile"). Rewrite the workflow to run the same
checks through uv (ruff pinned to 0.4.8 like upstream's noxfile, mypy,
pytest smoke/unit/ui), keeping the "nox" check name.
Fix what crept in while CI was dead:
- ObservationsDatabase resolved utils.observations_db as an import-time
default argument, so the UI-harness sandbox patch never applied — on a
runner without ~/PiFinder_data every UI test died with "unable to
open database file" (and locally the suite wrote to the real data
dir). Resolve the path at call time.
- UIMigrationConfirm/UIMigrationProgress documented in _COVERAGE_SKIP:
smoke-driving them risks starting a real migration download.
- UBXParser.reader also accepts the aiofiles reader from_file passes;
software.py declares the gettext-installed _ for mypy.
Update CLAUDE.md dev commands (uv, no tetra3 submodule).
Re-applied the deepchart feature (Gaia star charts) as a clean delta on top of current nixos, replacing the cat_images image backend with the new object_images package (POSS images + generated Gaia charts via chart_provider / poss_provider / gaia_chart / star_catalog). Adds the limiting-magnitude entry UI (ui/lm_entry.py) and the Obj Chart / Set LM menus. Merge decisions (theirs=deepchart, ours=nixos): - object_details.py: object_images.get_display_image replaces cat_images.get_display_image; nixos Contrast Reserve, telemetry and the DM_CONTRAST page preserved; deepchart chart-generator and progressive-image handling integrated. - menu_structure.py: keep both UITelemetryList and UILMEntry imports; union the nixos Image menu with deepchart Obj Chart / Set LM menus. - log / solver / main / camera_debug / sqm conflicts resolved to nixos (deepchart changes there were dead code, disabled test hacks, or already upstream). - cat_images.py removed (object_images fully replaces it); obsolete test_cat_images.py removed and test_equipment.py decoupled from it. Carried nixos NSEW-label / object-size-box overlays onto the object_images backend: ported the cardinal/size/vertex geometry helpers and an add_orientation_overlays() into object_images/image_utils.py, called from poss_provider so the image_nsew / image_bbox settings work again. Replaced the obsolete star_catalog tests (which targeted the removed _read_binary_index API) with real coverage of the v3 CompressedIndex reader: header/tile-count parsing, cumulative per-run tile offsets, second-run offset_base, missing-tile lookups (before/after/gap/past-run-length), single-run, and version validation. Made CompressedIndex.close() idempotent to fix a double-close ValueError the new tests exposed. The healpy-dependent _parse_records / _apply_proper_motion paths still need on-device coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gaia deep charts load from utils.data_dir/gaia_stars at runtime, but nothing provisioned it. Add a fixed-output derivation that fetches the ~454 MB catalog tarball and symlink it into PiFinder_data from the activation script, mirroring the pifinder-src / astro_data pattern so both fresh flashes and in-place upgrades deliver it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
star_catalog.py needs healpy for query_disc tile selection, but it was neither declared in pyproject.toml nor imported unconditionally. The uv2nix env shipped without it, and a try/except import guard (_HEALPY_AVAILABLE) degraded silently: every render logged an error and returned zero tiles, so the deep chart showed no stars and the SQUARE view drew blank instead of failing at startup. - Declare healpy in pyproject.toml and relock (prebuilt aarch64 wheel; pulls astropy). - Drop the _HEALPY_AVAILABLE guard: import healpy plainly and remove the six silent-degradation branches. A missing mandatory dep now fails loudly at import. - Fix pre-existing DTZ lint in the proper-motion epoch calc: use timez.utc_now() instead of naive datetime.now(), and snapshot it once (the old code called now() twice, risking a year/day-of-year straddle across a year boundary).
…hair _draw_crosshair_simple rebuilt the screen buffer and rebound self.screen / self.draw. The text layouters (designator, type/const, mag/size, aka, description, contrast interpretation) captured the module's original ImageDraw in __init__, so after the swap they drew onto a discarded image and vanished -- blacking out the object description screen once a chart had been viewed (the LOCATE->POSS->DESC cycle always passes through the chart). Paste the inverted pixels back into the existing screen buffer instead, preserving object identity.
The nixos branch replaced nox + requirements*.txt with uv, but an
upstream merge kept .github/workflows/nox.yml while dropping
python/noxfile.py, so the nox check failed on every nixos-line PR
("Failed to load Noxfile"). Rewrite the workflow to run the same
checks through uv (ruff pinned to 0.4.8 like upstream's noxfile, mypy,
pytest smoke/unit/ui), keeping the "nox" check name.
Fix what crept in while CI was dead:
- ObservationsDatabase resolved utils.observations_db as an import-time
default argument, so the UI-harness sandbox patch never applied — on a
runner without ~/PiFinder_data every UI test died with "unable to
open database file". Resolve the path at call time.
- UIMigrationConfirm/UIMigrationProgress documented in _COVERAGE_SKIP:
smoke-driving them risks starting a real migration download.
- UBXParser.reader also accepts the aiofiles reader from_file passes;
software.py declares the gettext-installed _ for mypy.
Update CLAUDE.md dev commands (uv, no tetra3 submodule).
…doers rule matches The PiFinder-Type (screen_direction) menu and other restart paths call sys_utils.restart_pifinder(), which ran "sudo systemctl restart pifinder". The NixOS sudoers rule allows exactly "systemctl restart pifinder.service" — sudo matches arguments verbatim, so the restart was refused and silently never happened: the UI showed "Restarting..." but the old process kept running. Net effect: switching PiFinder Type right→left left the integrator dead-reckoning with the old IMU geometry (reversed left/right motion) until a manual reboot. Use the full unit name (matches every other systemctl call in the codebase), pass -n so a denied sudo fails instead of hanging on a password prompt, and log a restart failure instead of swallowing it.
New shell.nix + flake devShell for aarch64-linux; .envrc runs it via direnv; nixos-pr-build and release workflows build and push the dev shell closure to Attic; dev_guide documents the on-device dev environment.
DisplayPygame_128/_320 installed _patch_pygame_keyboard, which replaced luma's _abort with a version calling pygame.event.get() every render frame. But set_keyboard_queue was never called, so _keyboard_queue stayed None and those KEYDOWNs were drained from the SDL queue and discarded -- racing the wired main.py event poll (ADR 0004) and losing ~29 of every 30 keypresses in the pygame emulator. Remove the dead mechanism (_patch_pygame_keyboard, _build_key_map, _PYGAME_KEY_MAP, set_keyboard_queue) so luma's non-draining _abort runs and main.py's poll is the sole SDL event consumer.
git and rsync were added to environment.systemPackages unconditionally for on-device development (clone/pull/rsync a checkout). They are not used by the NixOS image updater (manifest/store-path, ADR 0003), so keep them out of the production image by gating on pifinder.devMode -- available on dev/netboot builds, absent from production.
The BCM2711 hardware I2C block corrupts transfers when a slave stretches the clock — the BNO055 does so routinely. The 10 kHz &i2c1 workaround only lowered the odds (an errno-121 NACK still killed the IMU process on pifinder-mr2) while making every IMU transaction ~10x slower than 100 kHz. - hardware.nix: replace the 10 kHz &i2c1 overlay with a bit-banged i2c-gpio bus on GPIO2/3 (implements clock stretching per spec, ~60-100 kHz effective); disable &i2c1; drop the i2cFrequency option. - PiFinder/i2c_bus.py (new): get_i2c() hands out the i2c-gpio adapter when the device tree provides one (sysfs discovery), else the default hardware bus — no board-model sniffing, boot config is the single source of truth. - imu_pi.py: survive transient bus errors in the monitor loop (skip the sample, retry); re-create Imu() after ~2 s of consecutive errors to restore IMUPLUS mode after a sensor power glitch. - hardware_detect.py, battery_bq25895.py: route through get_i2c() so the rev-4 charger probe and battery telemetry follow the same bus. - deps: adafruit-extended-bus (ExtendedI2C on an arbitrary /dev/i2c-N). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UNCZgdNkUKk6Wzv42b11zq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a gaia achart view to all objects, down to mag 17 but with an auto limiting magnitude taking into account SQM and equipment. This auto limit is overridable.
TODO: