Feat/working branch - #159
Open
simonvanlierde wants to merge 1297 commits into
Open
Conversation
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
simonvanlierde
added a commit
that referenced
this pull request
Jul 12, 2026
- guard the getUser slot by sequence instead of promise identity (CodeQL missing-await) - read api.generated.ts directly instead of existsSync-then-read (CodeQL TOCTOU) - drop redundant location.hash guard and document server-side OAuth enforcement (CodeQL bypass FP) - anchor the Scalar request-blocking regex in docs e2e (CodeQL regex anchor) - catch Exception instead of BaseException in shutdown steps (code-quality) - add stamp-freshness HEALTHCHECK to the backups image (Trivy DS-0026) - use docstring bodies for protocol stubs instead of bare ellipses (code-quality) - remove unused Union import from alembic migration (code-quality)
simonvanlierde
added a commit
that referenced
this pull request
Jul 12, 2026
- break auth-services<->core import cycle: inline redis bool coercion in blocklist_store - break auth models<->user_database cycle: move get_user_db/get_auth_async_session to user_manager - rpi-cam ws auth: hash camera rate-limit bucket via rate_limit_bucket_key like all other callers - rpi-cam models: replace non-exhaustive match with dict lookup in to_http_error - connection_manager: sanitize camera_id in logs; pairing: drop unused _STATUS_WAITING - support_services/test_upload_security: replace del-parameter idiom (noqa B027 / pass) - test_email_providers: match provider URLs with startswith instead of substring - codeql config: enable AlertSuppression packs; exclude py/unused-import (ruff owns it) and informational py/cyclic-import - suppress verified false positives inline (token fingerprint sha256, secret-name logging, masked email logging)
simonvanlierde
force-pushed
the
feat/working-branch
branch
from
July 14, 2026 14:31
b0c5b17 to
11f74b6
Compare
simonvanlierde
added a commit
that referenced
this pull request
Jul 14, 2026
- guard the getUser slot by sequence instead of promise identity (CodeQL missing-await) - read api.generated.ts directly instead of existsSync-then-read (CodeQL TOCTOU) - drop redundant location.hash guard and document server-side OAuth enforcement (CodeQL bypass FP) - anchor the Scalar request-blocking regex in docs e2e (CodeQL regex anchor) - catch Exception instead of BaseException in shutdown steps (code-quality) - add stamp-freshness HEALTHCHECK to the backups image (Trivy DS-0026) - use docstring bodies for protocol stubs instead of bare ellipses (code-quality) - remove unused Union import from alembic migration (code-quality)
simonvanlierde
added a commit
that referenced
this pull request
Jul 14, 2026
- break auth-services<->core import cycle: inline redis bool coercion in blocklist_store - break auth models<->user_database cycle: move get_user_db/get_auth_async_session to user_manager - rpi-cam ws auth: hash camera rate-limit bucket via rate_limit_bucket_key like all other callers - rpi-cam models: replace non-exhaustive match with dict lookup in to_http_error - connection_manager: sanitize camera_id in logs; pairing: drop unused _STATUS_WAITING - support_services/test_upload_security: replace del-parameter idiom (noqa B027 / pass) - test_email_providers: match provider URLs with startswith instead of substring - codeql config: enable AlertSuppression packs; exclude py/unused-import (ruff owns it) and informational py/cyclic-import - suppress verified false positives inline (token fingerprint sha256, secret-name logging, masked email logging)
Picks up the accumulated backend API-surface changes from the review-fixes tasks: admin user-delete anonymize/delete content param, email-validation GET->POST, pairing register/poll/claim audience split (device-only vs public), pairing fingerprint pattern validation, and dropped category_name sort option on materials/product-types.
The Components section always showed "Add components after saving the product", including in view mode, where the product is saved and the Add component button is right there and works. Beyond contradicting the UI, the hint's accessible name contains the button's, so anything resolving "Add component" — a screen reader, or a test locator — matched the tooltip first. - pass editMode to SectionConfig.tooltip and return the hint only while editing, which is the only state where it is true - drop the E2E workaround that filtered the two apart - depend on the ref object rather than its .current in useReturnFocus, a render-time ref access that could never have worked as a dependency
`prod-up` started the API and `prod-migrate` ran afterwards, so the new code served against the old schema for an unbounded window. Fine for the cutover, which is a full outage behind a closed tunnel, but not for a routine release. - gate api on the migrator completing successfully, declared required: false so a profile-less `prod-up` still starts exactly as before (migrator sits behind the `migrations` profile); verified both shapes start correctly - document in the cutover runbook when to use each ordering
Verification listed checks but never said what constitutes a pass, who decides, or what happens on failure — and nothing alerts, so the decision is manual. - state the go/no-go criteria explicitly, and that anything unresolved means rolling back rather than fixing forward - smoke every public origin, not only the API, and check no container is restarting or unhealthy - record the absent schema rollback as an accepted risk: all 40 migrations define downgrade() and the newest is round-trip tested, but only one step on an empty schema, so a re-added column comes back empty - name an owner for the accepted risks - correct the start-then-migrate gap, which the migrations profile now closes
- split the single containers gate into per-leg path filters so a PR only runs the smoke legs whose image build inputs or service wiring actually changed - build the docker-smoke matrix from a JSON leg list emitted by the changes job, skipping the job outright when nothing container-relevant moved - add merge_group to the triggers and treat it like a push to main, since the event carries no PR context for path filtering - expose the Actions runtime to BuildKit and route CI builds through bake so the cache_from/cache_to gha entries in compose.yaml take effect - key concurrency by event and cancel only pull_request runs
The gallery spec mocks /products and the image URLs outright, so a broken upload, storage write, or thumbnail pipeline could not fail the suite. - drive the real file chooser, then Save, which is what actually uploads: picking a file only stages it client-side - record the /uploads/ URLs already on the page first and assert a *new* one appears. Seeded products put thumbnails in the DOM, so asserting that "an uploaded image exists" passes without having uploaded anything - fetch the stored bytes back, separating a client-side render from a real round-trip; verified the assertion fails when the upload POST is aborted
…symmetry) - audit product deletions during account erasure against the acting admin's id, not the erased user's - skip the re-save path for animated originals that carry EXIF instead of flattening them to one frame - reword the image-resize limiter NOTE now that AppServices.image_resize_limiter no longer exists - restore the already-claimed pairing record with its actual remaining TTL instead of a fresh one - add a regression test pinning that FileSystemStorage.delete propagates non-missing OSErrors
- gate the smoke harness itself (justfile, docker-smoke action) on the orchestration filter, and backend/pyproject.toml + .python-version on the backend filter, so harness and image-input edits still smoke something - add merge_group to the security workflow and treat it like a push to main, so an enabled merge queue cannot stall on a required security check - drop the now-dead run-containers output and guard the smoke matrix against an empty smoke-legs value - restore cancel-in-progress for push runs while keeping schedule and merge-queue runs uncancellable
The suite covered these steps as fragments, each from a fresh login as the seeded superuser — which never proved they compose, and never proved a normal contributor can do them at all, since a superuser passes every ownership check by construction. - add a journey spec running as `alice`, a seeded verified non-superuser: create, add a dimension, upload an image, add a child component, then read the result back as an anonymous visitor - assert the guest cannot see Edit Product, so the block cannot quietly pass if the context ever inherits a session - let loginAndReachProducts take credentials, defaulting to the admin - wait for the picked file to reach the form before saving; setFiles returns early, so both upload tests could save a product with no images and never issue the upload Registration is not chained in: a password signup is unverified, product creation requires verification, and the E2E stack has no mail transport, no stored token, and no admin route to set is_verified. The spec header records this; auth.spec.ts still covers signup to the verify-email prompt.
- CPVCard.tsx no longer draws a decorative rotated glyph — the app's "no ornament" rule wins over the taste-call left in TODOS.md - useCaptureEntity's create/upload failure toasts and the product-page ErrorState now show a specific title instead of a generic "Something went wrong", matching every other error call site
- give the seeded Dell XPS 13 a nested component tree, so dev and CI have a product whose parts can carry photographs - resolve seeded image parents by querying component ids rather than walking the created tree, which lazy-loads mid-await - keep the seed photographs above 800px: narrower originals generate only the 200px derivative, leaving every consumer without a srcset
- give the committed fixture a photograph per part, so builds without API access show the grid rather than the list - assert in e2e that every plate image decodes: a fixture pointing at a file that never shipped renders an empty frame nothing else notices - correct the homepage main-link count, which predated the blueprint's own record link
- build against the seeded compose.e2e.yaml backend and assert the hero shows a live record, since a failed fetch silently degrades to the fixture - assert every plate carries a two-candidate srcset and a sizes hint, which the one-width fixture cannot produce - run the lane from test-e2e-full-stack, before the app's, on the same stack
Every domain term the app puts in front of a contributor, defined in ordinary words: records and components, the three circularity notes, reference data and CPV, media and pairing, account states. Written for the repair-café and citizen-scientist audiences PRODUCT.md records, where none of it is assumed knowledge. - new page under Guides, cross-linked from the data collection guide - component advice points at the guide's rule of thumb rather than restating it
- the six digits from the authenticator app were labelled "Setup code", "Current code" and "Authentication code" on three screens; all three now say Authentication code - the measurement placeholder was "> 0", a validation predicate rather than a hint; it is now "e.g. 12", and LocalizedFloatInput's default matches its only call site - the physical properties rows gained an edit-mode line naming the units and what an empty field means - two generic failures said "An error occurred. Please try again later."; they now name what failed and what to do
The gallery specs mock the API with data URLs and the list specs never look at a thumbnail, so the suite passed against broken images: a failed load quietly swaps in ProductCard's placeholder. Fetch the requested URL instead of reading the rendered element, since the rig's cross-port 127.0.0.1 origins are blocked by the API's same-site resource policy.
The /uploads CORP relaxation rode on settings.debug, which also turns on SQL echo, DEBUG logging and raw storage error detail — so the E2E rig could not have its images load without diverging from production in three unrelated ways. - derive settings.uploads_allow_cross_origin from the environment (dev and testing), so staging and prod cannot opt in whatever their env files say - cover both deployed environments, not just prod, in the strict-policy test - assert decoded images in the www and app E2E lanes now that cross-port loopback loads are allowed, instead of fetching the URL beside the page
Cloudflare already cached /uploads by file extension and by the origin's Cache-Control, so this changes no behaviour — it states the intent where it can be reviewed and pins it against a zone-setting change made outside this repo. - cache /uploads on prod's api host for a year, matching the origin's immutable content-addressed URLs - assert the media rule and the staging bypass stay disjoint by host, since the cache phase is the one place two rules could set the same thing
- make phone editing actions flow safely around detail fields - clarify property hierarchy and compact edit-only visuals - improve index filters, creation labels, and card facts - add responsive browser and component regression coverage
…ions in PRODUCT.md
- Add an ordered `UserRole` (`contributor`, `lab`) alongside the existing
account booleans, with an explicit rank map rather than StrEnum ordering,
which agrees with the tier order only by coincidence.
- Store it as VARCHAR with a validated CHECK constraint, added NOT VALID then
validated so the migration takes no rewrite; every existing account is
backfilled to `contributor`, fail-closed.
- Restrict non-image research-file upload to lab accounts on the product and
component routes, rejecting before the multipart body is parsed.
- Tier the upload quota by role, resolved as a SQL CASE inside the existing
conditional UPDATE so the reservation stays one atomic statement.
- Add `PUT /admin/users/{id}/role`, audit-logged and superuser-only; role is
absent from `UserUpdate` entirely, so no self-service PATCH can reach it.
- Add a read-only report listing accounts over the quota their role grants,
for the deploy that introduces the tiers.
Acceptance was stamped only at registration, so every account predating the terms columns held no publication licence and could never grant one — the dataset release excluded their records permanently. - Add `POST /users/me/accept-terms`, which takes no body: the version is stamped server-side, since the column is evidence of what the person was shown. An acceptance already past the current version is never moved back. - Move `MINIMUM_RELEASE_TERMS_VERSION` into the module that owns terms versioning and import it in the release script, so the prompt and the release can never disagree about who is covered. - Expose `terms_acceptance_required` on the user read schema, derived from that threshold, so clients never compare versions themselves. - Rewrite the release script's source-level guard to pin the intent — the threshold is imported, never redefined, never derived from the current version — rather than the literal it asserted before.
Covers both preceding backend changes: the role and quota fields on the user read schema, and the terms acceptance route and flag. Generated by `just openapi`, `just datamodel`, and `just codegen`; not hand-edited.
- Map `role` and the upload quota onto the frontend user, defaulting a missing role to `contributor` so an incomplete payload never reads as `lab`. - Add a files service covering list, upload, and remove, validating the extension against the backend's allowlist by final suffix only, so dotted research filenames pass and double extensions do not. - Add a "Research files" block to the Media section, rendered only for a lab account in edit mode on a saved record it owns. Files attach to an existing record and upload immediately, so none of this joins the draft save cycle. - Add expo-document-picker, accepting any type and validating the filename afterwards, since pickers report MIME types inconsistently per platform. Hiding the block is presentation; the control is the route dependency, which refuses a non-lab upload whatever the client renders.
- Mount a dialog globally that asks accounts still owing acceptance, offering the public terms page, "Not now", and Accept. It states plainly that declining costs nothing: a grant extracted by withholding access is the one whose validity gets questioned. - Keep the dismissal in a shared store rather than hook state, so the account screen's row reopens the mounted dialog instead of toggling its own copy. It is not persisted, so the next login asks again. - Add the account-screen row beside the About section rather than inside it, which would make a section of static links depend on auth context.
- Describe the three independent account privileges in the security baseline, since conflating any two of them is a privilege escalation. - Replace the cutover runbook's single-quota guidance with the role tiers, and add the step that promotes lab accounts after the fail-closed backfill. - Audit the seven migrations added after that runbook was written: none aborts on data or drops any, so only a lock-contention pre-flight was needed. Note that one CONCURRENTLY index drop escapes the single-transaction abort rule, and correct the revision literals, which had gone stale by 27 migrations. - Document research-file upload as a lab capability and the tiered quotas in the collection guide, install guide, and dataset release notes.
- add `wcag22a`/`wcag22aa` to the axe tag set in www, docs, and app, which enforces `target-size` (2.5.8), the only 2.2-only rule axe-core 4.12 ships - narrow the documented hand-check gap to 2.4.11 and 2.4.13, which have no axe rule at all - add the public accessibility statement at /accessibility, linked from the footer, claiming partial conformance and naming what is untested - record that the compliant artifact is a per-domain register entry, which is why the page omits a response window and an escalation body for now
The prompt is modal and its dismissal lived in memory, so every full page load re-opened it. That is nagging rather than asking, and it broke 59 authenticated E2E tests: seeded accounts are created programmatically and so record no terms acceptance, and later clicks hit the modal scrim while `toBeVisible` assertions kept passing, since visibility is not occlusion. - Hold the dismissal in sessionStorage on web, so a reload does not re-ask while a later sign-in still does. Native keeps in-memory state, which has the same lifetime there. - Suppress the prompt through sessionStorage in the shared E2E login helper, the same way the guest welcome card is already handled, rather than clicking it away on every navigation. - Cover the prompt itself end to end: acceptance persists across a reload, and a dismissal clears the way without recording anything. The acceptance test is one-shot per database, since no endpoint un-accepts, and says so when it fails.
- justify the two conditional test.skip calls with biome-ignore comments - parallelize independent per-plate assertions in landing.spec.ts - suppress the live-lane loop's await, which must stay sequential since scrollIntoViewIfNeeded races if plates are checked concurrently
- import the account screen at the top level instead of require()ing it inside the first test, so a cold jest cache no longer charges the module transform against the 15s testTimeout - retry the products FAB geometry assertion with toPass(), since scrolling to the terminal count row can pull in another page of products between the two boundingBox() calls
- extract EditModeOverlay from ProductImageGalleryContent and ReadOnlySpecRow from LocalizedFloatInput so neither is redefined on every parent render - pass a module-scope renderTabBar to Tabs instead of an inline arrow - lift regex literals to module constants in the ProductTags and formTokens tests - batch the boundingBox() reads in product-detail.spec.ts instead of awaiting one per loop iteration
| stripped = value.strip() | ||
| if label in {"prod", "staging"} and stripped.startswith(KNOWN_SECRET_PLACEHOLDER_PREFIXES): | ||
| if is_optional: | ||
| sys.stdout.write(f"{label}: optional secret {name} is an unfilled placeholder (provider not configured?)\n") |
| from PIL import Image as PILImage | ||
| from PIL.ExifTags import IFD | ||
|
|
||
| import scripts.build_dataset_release as release |
| import httpx | ||
| import pytest | ||
|
|
||
| import scripts.zenodo_deposit as zenodo |
| try: | ||
| if target.read_text() == content: | ||
| return None | ||
| except FileNotFoundError: |
| // dimensions from onLoad, add if it bothers anyone. | ||
| const clampTranslationToBounds = useCallback( | ||
| (velocityX: number, velocityY: number) => { | ||
| 'worklet'; |
| // declares `matrix` on its internal extracted-props type, so the native branch | ||
| // needs the cast. | ||
| export function matrixProp(m: Matrix): { transform: Matrix } { | ||
| 'worklet'; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat/working-branch → main: security hardening, MFA, brand refresh, and repo restructure
TL;DR
A ~2.5-month working branch (728 commits) intended as a single squash-merge. It hardens the auth/account-security stack, adds TOTP MFA with recovery codes, removes dead data-model surface (organizations, newsletter), rebuilds the public homepage around live stats, rebrands toward R9lab / ReLab, and restructures the two frontend subrepos.
Why squash
The branch grew organically over ~2.5 months, so its intermediate history is noisy. Squashing collapses it into one clean, coherent commit on
main. The changes were checked againstmainacross multiple angles — correctness, security, test coverage, performance, and over-engineering — before landing.Highlights by area
🔐 Auth & account security (largest area)
The auth module was substantially reworked — service layer split into focused units (
lifecycle,login_flow,mfa_service,session_flow,token_store,email/*,oauth/*) and hardened end to end.🗄️ Data model & migrations
17 new migrations. Notable removals and reshapes:
email_canonical),has_usable_password, MFA/recovery tables.📁 File storage hardening
Paginated media lists, real zip-size enforcement, malware scanning extended to device thumbnails, quota owner keying fixed, sensitive-key rejection in local storage, JSON-recursion guard, dotted filenames accepted, uploaded filenames validated against MIME type.
🌐 Public site (
www, formerlyfrontend-web)Homepage rebuilt: new
Hero,SiteFooter, privacy page, and aStatsPanelfed by new public stats API (/stats/totals,/stats/categories,/stats/series) — monthly activity chart with per-category part counts. New brand/token CSS, theme handling, 404 page, and a Vitest suite for the new components.📷 RPi camera plugin
WebSocket relay bounded against unresponsive devices, camera-flapping-to-offline fixed, livestream/recording lifecycle corrected, device key kept on the LAN, response ownership verified, device-assertion lifetime capped, circuit breaker made Redis-only and atomic.
📱 App (React Native, formerly
frontend-app)MFA challenge screen + pending-login routing, OAuth callback via URL fragment, new-product drafts can set type/material before first save, "already live" stream dialog, single-flight auth/MFA submits, gallery no longer silently deletes unrenderable images, static-background refactor (parallax scaffolding removed), plus a large sweep of review-driven bug fixes.
🎨 Brand → R9lab / ReLab
R9lab logo/wordmark/flask marks wired into www, docs, and app UIs; email templates rebranded with hosted wordmark; README wordmark. Asset generators live in
assets/logo-src/.Copy pass across the app UI, docs, and www: unified sign-in/sign-out terminology, sentence-case titles and buttons, plainer error and empty-state messages, and less boilerplate — with tests and e2e specs updated to match.
🏗️ Infra / CI / deploy
needs, OpenAPI/app-codegen freshness gates, JUnit → Codecov Test Analytics, per-PR RN a11y lint, Playwright browser caching.X-Forwarded-For.🧭 Repo restructure
frontend-web→wwwfrontend-app→appRenames (with history preserved) touch devcontainers, CI, docs, and configs — the bulk of the file count.
Verification
Each subrepo's loop (
just fix→just check→just test) was run as changes landed. Beyond the test suites, the diff was reviewed across correctness, security, test-coverage, performance, and simplification angles. New migrations include a downgrade/upgrade round-trip test, and new logic (stats, auth flows, storage, MFA) ships with regression tests.