Skip to content

Name the psycopg2 dialect explicitly for the MetaTracker engine - #23

Merged
jamesray merged 1 commit into
mainfrom
jamesray/explicit-psycopg2-dialect
Nov 5, 2026
Merged

jamesray merged 1 commit into
mainfrom
jamesray/explicit-psycopg2-dialect

Conversation

@jamesray

@jamesray jamesray commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Why

SQLAlchemy 2.1 (released 24 September 2026) changed the default driver for plain postgresql:// URLs from psycopg2 to psycopg (v3). The swxsoc[tracker] extra pins only sqlalchemy<3, and this image installs psycopg2-binary, so every processing image built since 7 October fails the MetaTracker step:

ERROR: No module named 'psycopg'   (file_processor.py, _track_file_metatracker)

The file is still calibrated and uploaded, so the run reports success, but its metadata never reaches RDS. dev-padre has logged this on every file since its 22 October image; prod gdctb and today's prod padre images carry it too. Build logs confirm the cause: images built 9 September got SQLAlchemy 2.0.52, every image since 7 October got 2.1.x.

What

Spell out the DBAPI in the engine URL: postgresql+psycopg2://. The failed-file reprocessing path calls psycopg2.connect directly with a libpq URL and is unaffected.

Verification

  • Reproduced in dev on 5 November: after a full dev apply, a craft test file pushed through dev-impax and a REACH L1c re-trigger in dev-swxsoc_pipeline both processed and uploaded correctly but hit this error at the tracking step.
  • Unit tests run in CI; the local machine lacks the scientific dependency stack.
  • After merge, the image rebuild and a repeat of the dev push should show science_file_id populated and no psycopg error.

The same URL pattern exists in sdc_aws_concating_lambda; a matching PR is open there.

🤖 Generated with Claude Code

SQLAlchemy 2.1 (24 September 2026) made psycopg (v3) the default
driver for plain "postgresql://" URLs. The image installs
psycopg2-binary, so every processing image built since 7 October
fails the MetaTracker step with "No module named 'psycopg'": the file
is still calibrated and uploaded, but its metadata never reaches RDS.
dev-padre has logged this since its 22 October image.

Spell out "postgresql+psycopg2://" where the SQLAlchemy engine is
built. The failed-file reprocessing query already uses psycopg2
directly with a libpq URL and is unaffected.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jamesray
jamesray marked this pull request as ready for review November 5, 2026 22:42
Copilot AI balanced review requested due to automatic review settings November 5, 2026 22:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change explicitly selecting psycopg2 for MetaTracker database connections.

No open issues 🎉

What changed in this PR

This PR makes MetaTracker explicitly use the installed psycopg2 driver, preventing metadata tracking failures caused by SQLAlchemy selecting psycopg v3.

Changes:

  • Changes the engine URL scheme to postgresql+psycopg2://.
  • Adds a comment explaining why the driver must be specified.
File Description
lambda_function/​src/​file_processor/​file_processor.py Explicitly selects psycopg2 for the MetaTracker database connection and documents the reason.

Review effort: Balanced

@jamesray
jamesray merged commit fe53508 into main Nov 5, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants