Skip to content

Fix/db storage #111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Amazing-Stardom wants to merge 22 commits into
base: master
Choose a base branch
from
Draft

Fix/db storage #111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f74d50c
Prune Database Storage and Limit Job Retention
Amazing-Stardom Sep 5, 2026
0762452
Offload Diff Artifacts to Blob Storage
Amazing-Stardom Sep 5, 2026
ba0d94a
Consolidate Diff Storage Architecture and Documentation
Amazing-Stardom Sep 6, 2026
bb54a1a
Robust Blob Storage and Migration Logic
Amazing-Stardom Sep 6, 2026
2e04fd0
Implement Diff Archival to Blob Storage
Amazing-Stardom Sep 8, 2026
3108e3e
Implement Async Diff Archival with Bulk Metadata Updates
Amazing-Stardom Sep 10, 2026
5d80b5f
Refactor Diff Archival to Preloaded Changes Archival
Amazing-Stardom Sep 11, 2026
2db1988
Optimize Preloaded Changes Archival and Logging
Amazing-Stardom Sep 12, 2026
3a8f1f2
Fix JSON Deserialization for Confidence Field
Amazing-Stardom Sep 12, 2026
3a60014
Restrict JSON Fallback Decoding and Update Cron Schedule Helpers
Amazing-Stardom Sep 12, 2026
a26db26
Add Code Context to Comments and Refactor Feedback UI
Amazing-Stardom Sep 13, 2026
ff14745
Standardize Review Worker Logging to Zerolog
Amazing-Stardom Sep 13, 2026
82ed4a8
Migrate Preloaded Changes Archival to River Jobs
Amazing-Stardom Sep 18, 2026
3dc5346
Refactor Job Archival and Update UI Cron Controls
Amazing-Stardom Sep 20, 2026
eb2c142
Update Project Dependencies
Amazing-Stardom Sep 20, 2026
efb22bb
Migrate Diff Storage Offloading to River Job Queue
Amazing-Stardom Sep 20, 2026
095a5ba
Document Vulnerability Verification and Suppression Policy
Amazing-Stardom Sep 20, 2026
8662b06
Update Dependencies, Fix Archival Errors, Unify UI
Amazing-Stardom Sep 20, 2026
30604ad
Implement dashboard date range perist and other fixes
LinceMathew Sep 20, 2026
61c71e6
Update Job Queue Context for Startup Queries
Amazing-Stardom Sep 20, 2026
d2c7033
Add Error Logging and Simplify Blobstore Reads
Amazing-Stardom Sep 20, 2026
bb4b68e
Refactor Job Processing and Enhance Error Reporting
Amazing-Stardom Sep 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,9 +180,9 @@ RUN echo "👤 Creating non-root user..." && \
useradd -u 1001 -r -g livereview -d /app -s /sbin/nologin livereview && \
echo "User 'livereview' created successfully"

# Create directories
# Create directories (including lrdata/blobs for local blob storage)
RUN echo "📁 Creating application directories..." && \
mkdir -p /app/db/migrations /app/data /app/logs && \
Comment thread
Amazing-Stardom marked this conversation as resolved.
Comment thread
Amazing-Stardom marked this conversation as resolved.

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

A new directory /app/lrdata/blobs has been created to handle local blob storage.

mkdir -p /app/db/migrations /app/data /app/logs /app/lrdata/blobs && \
chown -R livereview:livereview /app && \
echo "Directories created and permissions set"

Expand Down
87 changes: 45 additions & 42 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ docker-local-rebuild:
docker-local-stop:
docker compose down

# Python executable mapping (works out-of-the-box on Ubuntu where python3 is present but python is not)
Comment thread
Amazing-Stardom marked this conversation as resolved.
PYTHON ?= $(shell command -v python3 2>/dev/null || echo python)

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

Using a dynamic Python executable improves portability by handling both python3 and python.

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

A Python variable has been introduced to ensure cross-platform compatibility.


# Go parameters
GOENV=env -u GOROOT
GOCMD=$(GOENV) go
Expand Down Expand Up @@ -151,48 +154,48 @@ email-preview:

# Version management targets
version:
@python scripts/lrops.py version
@$(PYTHON) scripts/lrops.py version
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-bump:
@python scripts/lrops.py bump $(ARGS)
@$(PYTHON) scripts/lrops.py bump $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-patch:
@python scripts/lrops.py bump --type patch $(ARGS)
@$(PYTHON) scripts/lrops.py bump --type patch $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-minor:
@python scripts/lrops.py bump --type minor $(ARGS)
@$(PYTHON) scripts/lrops.py bump --type minor $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-major:
@python scripts/lrops.py bump --type major $(ARGS)
@$(PYTHON) scripts/lrops.py bump --type major $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Version management targets that allow dirty working directory
version-bump-dirty:
@python scripts/lrops.py bump --allow-dirty
@$(PYTHON) scripts/lrops.py bump --allow-dirty
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-patch-dirty:
@python scripts/lrops.py bump --type patch --allow-dirty
@$(PYTHON) scripts/lrops.py bump --type patch --allow-dirty

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The $(PYTHON) variable should be used for a consistent Python executable.


version-minor-dirty:
@python scripts/lrops.py bump --type minor --allow-dirty
@$(PYTHON) scripts/lrops.py bump --type minor --allow-dirty
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-major-dirty:
@python scripts/lrops.py bump --type major --allow-dirty
@$(PYTHON) scripts/lrops.py bump --type major --allow-dirty
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Dry-run version targets
version-bump-dry:
@python scripts/lrops.py bump --dry-run --allow-dirty
@$(PYTHON) scripts/lrops.py bump --dry-run --allow-dirty
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-patch-dry:
@python scripts/lrops.py bump --type patch --dry-run --allow-dirty
@$(PYTHON) scripts/lrops.py bump --type patch --dry-run --allow-dirty
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-minor-dry:
@python scripts/lrops.py bump --type minor --dry-run --allow-dirty
@$(PYTHON) scripts/lrops.py bump --type minor --dry-run --allow-dirty
Comment thread
Amazing-Stardom marked this conversation as resolved.

version-major-dry:
@python scripts/lrops.py bump --type major --dry-run --allow-dirty
@$(PYTHON) scripts/lrops.py bump --type major --dry-run --allow-dirty
Comment thread
Amazing-Stardom marked this conversation as resolved.

build-versioned:
@python scripts/lrops.py build
@$(PYTHON) scripts/lrops.py build
Comment thread
Amazing-Stardom marked this conversation as resolved.

# ============================================================================
# Frozen DOCKER DEPENDENCY versions (docker/docker-deps.env)
Expand Down Expand Up @@ -222,15 +225,15 @@ build-versioned:
# shows up in the report when it falls behind, it's just never auto-applied
# by update-docker-deps/update-docker-deps-yes or the pre-build check.
# Override for one run with:
# python3 scripts/check_docker_deps.py --include-pinned [--yes]
# $(PYTHON) scripts/check_docker_deps.py --include-pinned [--yes]
Comment thread
Amazing-Stardom marked this conversation as resolved.
check-docker-deps:
@python3 scripts/check_docker_deps.py --check
@$(PYTHON) scripts/check_docker_deps.py --check
Comment thread
Amazing-Stardom marked this conversation as resolved.

update-docker-deps:
@python3 scripts/check_docker_deps.py
@$(PYTHON) scripts/check_docker_deps.py
Comment thread
Amazing-Stardom marked this conversation as resolved.

update-docker-deps-yes:
@python3 scripts/check_docker_deps.py --yes
@$(PYTHON) scripts/check_docker_deps.py --yes
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Smoke-test that every pinned Docker dependency binary is actually present
# and invokable INSIDE a built image (dbmate, river, riverui, vl-convert,
Expand All @@ -257,7 +260,7 @@ verify-docker-deps:
# 8. Interactive confirmation prompt before build execution
# Files: scripts/lrops.py (lines 634-826), Dockerfile (multi-stage), ui/package.json
docker-build:
@python scripts/lrops.py build --docker $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# DOCKER-BUILD-PUSH: Same as docker-build but automatically pushes to registry
# Implementation: scripts/lrops.py:cmd_build() with push=True flag
Expand All @@ -270,21 +273,21 @@ docker-build:
# Registry: Configurable via --registry, defaults to GitLab Container Registry
# Tags: <registry>/<image>:<version> and optionally <registry>/<image>:latest
docker-build-push:
@python scripts/lrops.py build --docker --push $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --push $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Interactive Docker build with tag selection
docker-interactive:
@python scripts/lrops.py docker
@$(PYTHON) scripts/lrops.py docker
Comment thread
Amazing-Stardom marked this conversation as resolved.

docker-interactive-push:
@python scripts/lrops.py docker --push $(ARGS)
@$(PYTHON) scripts/lrops.py docker --push $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Dry-run Docker targets
docker-build-dry:
@python scripts/lrops.py build --docker --dry-run $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --dry-run $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

docker-interactive-dry:
@python scripts/lrops.py docker --dry-run
@$(PYTHON) scripts/lrops.py docker --dry-run

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The $(PYTHON) variable should be used for a consistent Python executable.


# Legacy build-push for backward compatibility (now uses versioning)
build-push: docker-build-push
Expand Down Expand Up @@ -501,7 +504,7 @@ security-gh-secret-scanning:

# Regenerate machine-readable and markdown triage artifacts from the latest OSV report.
security-triage: security-osv
@python3 scripts/extract_osv_report.py \
@$(PYTHON) scripts/extract_osv_report.py \
Comment thread
Amazing-Stardom marked this conversation as resolved.
--input security_issues/osv-scanner-latest.json \
--csv security_issues/osv-triage-latest.csv \
--md security_issues/osv-triage-latest.md
Expand Down Expand Up @@ -636,7 +639,7 @@ sync-docs-sources:
# Exits 1 if anything is behind - usable in CI, or just run
# `make sync-docs-sources` to actually pull the update in.
check-docs-sources:
@python3 scripts/docindex/check_docs_sources.py
@$(PYTHON) scripts/docindex/check_docs_sources.py
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Generate a token-compact schema dump of the prod DB (public schema) for LLM context.
.PHONY: compressed-schema
Expand All @@ -646,7 +649,7 @@ compressed-schema:
exit 1; \
fi
@mkdir -p db
@set -a && . ./.env.prod && set +a && python3 scripts/llm-schema.py db/schema-compressed.txt
@set -a && . ./.env.prod && set +a && $(PYTHON) scripts/llm-schema.py db/schema-compressed.txt
Comment thread
Amazing-Stardom marked this conversation as resolved.
@echo "✅ Wrote db/schema-compressed.txt"

# Export a full snapshot of the prod DB (schema + data) using .env.prod's
Expand Down Expand Up @@ -758,10 +761,10 @@ ghcr-login: docker-context-setup

# Multi-architecture Docker build targets
docker-multiarch:
@python scripts/lrops.py build --docker --multiarch $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --multiarch $(ARGS)

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The $(PYTHON) variable should be used for a consistent Python executable.


docker-multiarch-push:
@python scripts/lrops.py build --docker --multiarch --push $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --multiarch --push $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.
@echo "ℹ️ Optional GitHub release publish: make release-gh"
@echo " Optional explicit override: make release-gh VERSION=$$(git describe --tags --abbrev=0 2>/dev/null || true)"

Expand All @@ -771,41 +774,41 @@ release-gh:
@python3 $(RELEASE_GH_SCRIPT) --repo $(GH_REPO) $(if $(VERSION),--version $(VERSION),)

docker-multiarch-dry:
@python scripts/lrops.py build --docker --multiarch --dry-run $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --multiarch --dry-run $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Vendor multi-arch dry run (Phase 9 validation)
vendor-docker-multiarch-dry:
@python scripts/lrops.py build --docker --multiarch --dry-run --vendor-prompts $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --multiarch --dry-run --vendor-prompts $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Vendor single-arch builds
vendor-docker-build-dry:
@python scripts/lrops.py build --docker --dry-run --vendor-prompts $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --dry-run --vendor-prompts $(ARGS)

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The $(PYTHON) variable should be used for a consistent Python executable.


vendor-docker-build:
@python scripts/lrops.py build --docker --vendor-prompts $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --vendor-prompts $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

vendor-docker-build-push:
@python scripts/lrops.py build --docker --push --vendor-prompts $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --push --vendor-prompts $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Vendor multi-arch push (with optional latest tagging via ARGS="--latest")
vendor-docker-multiarch-push:
@python scripts/lrops.py build --docker --multiarch --push --vendor-prompts $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --multiarch --push --vendor-prompts $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Cross-compilation Docker build targets (faster ARM builds)
docker-multiarch-cross:
@echo "🚀 Building multi-arch images using cross-compilation for faster ARM builds"
@python scripts/lrops.py build --docker --multiarch $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --multiarch $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

docker-multiarch-cross-push:
@echo "🚀 Building and pushing multi-arch images using cross-compilation"
@python scripts/lrops.py build --docker --multiarch --push $(ARGS)
@$(PYTHON) scripts/lrops.py build --docker --multiarch --push $(ARGS)
Comment thread
Amazing-Stardom marked this conversation as resolved.

# Interactive multi-architecture Docker build
docker-interactive-multiarch:
@python scripts/lrops.py docker --multiarch
@$(PYTHON) scripts/lrops.py docker --multiarch

docker-interactive-multiarch-push:
@python scripts/lrops.py docker --multiarch --push
@$(PYTHON) scripts/lrops.py docker --multiarch --push

cplrops:
@cp lrops.sh ../gh/LiveReview/
Expand Down Expand Up @@ -1121,7 +1124,7 @@ docs/openapi.yaml internal/api/docs/spec.go: $(API_SPEC_INPUTS) typed-install
@chmod 755 docs internal/api/docs
@PATH="$(TYPED_BIN_DIR):$$PATH" typed -config config/typed.yaml > /tmp/lr_typed_build.log 2>&1 || (echo "❌ Typed generation failed. Logs:" && cat /tmp/lr_typed_build.log && exit 1)
@$(GOCMD) run internal/api/docs/spec.go > /tmp/lr_spec_build.log 2>&1 || (echo "❌ OpenAPI spec generation failed. Logs:" && cat /tmp/lr_spec_build.log && exit 1)
@python3 scripts/openapi/fix-openapi-spec.py docs/openapi.yaml
@$(PYTHON) scripts/openapi/fix-openapi-spec.py docs/openapi.yaml


generate-openapi: docs/openapi.yaml
Expand Down Expand Up @@ -1503,7 +1506,7 @@ razorpay-webhook-ensure:
fi
@MODE_VALUE="$(MODE)"; \
if [ -z "$$MODE_VALUE" ]; then MODE_VALUE="$${RAZORPAY_MODE:-live}"; fi; \
python3 scripts/razorpay_webhook_ensure.py --base-url "$(BASE_URL)" --mode "$$MODE_VALUE" $(ARGS)
$(PYTHON) scripts/razorpay_webhook_ensure.py --base-url "$(BASE_URL)" --mode "$$MODE_VALUE" $(ARGS)

razorpay-webhook-ensure-dry:
@if [ -z "$(BASE_URL)" ]; then \
Expand All @@ -1512,7 +1515,7 @@ razorpay-webhook-ensure-dry:
fi
@MODE_VALUE="$(MODE)"; \
if [ -z "$$MODE_VALUE" ]; then MODE_VALUE="$${RAZORPAY_MODE:-live}"; fi; \
python3 scripts/razorpay_webhook_ensure.py --base-url "$(BASE_URL)" --mode "$$MODE_VALUE" --dry-run $(ARGS)
$(PYTHON) scripts/razorpay_webhook_ensure.py --base-url "$(BASE_URL)" --mode "$$MODE_VALUE" --dry-run $(ARGS)

razorpay-verify-plans:
@bash ./scripts/verify-razorpay-plans.sh $(DEPLOY_ACTUAL_ENV_FILE)
Expand Down
3 changes: 3 additions & 0 deletions config/osv-scanner.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@
id = "GO-2026-5932"
reason = "False positive — golang.org/x/crypto/openpgp sub-package is never imported by this project; only golang.org/x/crypto/bcrypt is used."

[[ignoredVulns]]
Comment thread
Amazing-Stardom marked this conversation as resolved.

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: info

The vulnerability identified as GO-2026-6452 is being ignored.

id = "GO-2026-6452"
reason = "TODO: Once OSV DB fixes the missing 'fixed' semver range, this should be removed. We are on patched v2.11.0, and the vulnerable parsing path is never called because we only write spreadsheets, never read them."

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

TODO: This ignored vulnerability should be removed once the OSV database is fixed.

Suggestions:

  1. Create a follow-up task to re-evaluate this ignored vulnerability once the OSV database is updated.

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

The TODO comment indicates a temporary ignore. Please track this for removal.

Suggestions:

  1. Create a follow-up task to monitor the OSV database for updates regarding 'GO-2026-6452'.
  2. Remove this ignore entry once the vulnerability is properly fixed or the OSV database reflects the correct status.

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

This is a placeholder comment. It should be tracked and removed once the OSV database has been fixed.

Suggestions:

  1. Create a tracking issue for this TODO item to ensure it's revisited and removed once the OSV DB is updated or the dependency is fully resolved.

4 changes: 3 additions & 1 deletion docker-entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ echo "🚀 Starting LiveReview application..."

# Ensure blob storage directory exists (runs as root)
mkdir -p /app/lrdata/blobs
chown -R livereview:livereview /app/lrdata/blobs
# Local dev only: chown may fail under Docker Desktop / rootless Docker / seccomp.
# Production (lrops.sh) runs standard Docker where this always succeeds — no || true needed there.
chown -R livereview:livereview /app/lrdata/blobs 2>/dev/null || true
Comment thread
Amazing-Stardom marked this conversation as resolved.

@LiveReview-Bot LiveReview-Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: warning

The failure of chown is being ignored, which may mask real permission issues.

Suggestions:

  1. Consider adding more specific logging for chown failures (e.g., chown ... 2>>/dev/stderr || true) to capture errors without stopping startup, especially if this is not strictly for local dev.


# Function to wait for PostgreSQL to be ready
wait_for_postgres() {
Expand Down
Loading
Loading