Skip to content

Share one database connection per container - #19

Closed
jamesray wants to merge 4 commits into
mainfrom
jamesray/skip-archives-and-pool-db
Closed

jamesray wants to merge 4 commits into
mainfrom
jamesray/skip-archives-and-pool-db

Conversation

@jamesray

@jamesray jamesray commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

On 2026-09-08 a REACH backfill fired this Lambda 12,894 times in under an hour. Every invocation created a new SQLAlchemy engine, and the idle pooled connections left behind by each warm container pinned both db.t4g.micro MetaTracker databases at their ~70 connection limit: 5,283 remaining connection slots are reserved errors in prod, 5,146 in dev, and 2,465 foreign-key violations where the status row landed but the file row had not. Grafana lost its database connection as collateral.

The archives themselves are handled upstream now (swxsoc/sdc_aws_reach_sync#3 delivers them to a bucket nothing listens to), so this PR no longer special-cases them. It fixes the part that is this Lambda's own: connection churn.

What changes

  • One engine per container. database_engine(secret_arn, url) caches a sqlalchemy engine per secret ARN with pool_size=1, max_overflow=0, pool_pre_ping=True, and pool_recycle=300. The cache key is the secret ARN, never the URL, because the URL carries the password: each entry remembers the URL it was built from, and a rotated credential disposes the old pool and builds exactly one replacement.
  • Tables created once per engine. ensure_tracker_tables runs create_tables once per cached engine rather than on every event, and again after a rotation rebuilds the engine.
  • A warm container now holds at most one tracker connection across rotations.

Tests cover engine reuse per secret, engine replacement on rotation with the old pool disposed, and once-per-engine table creation. ruff check, ruff format, and black --check are clean for the whole tree (the test module needed two whitespace fixes to get black green, which is the same gate the CodeBuild pre_build runs). The four E501 lines flake8 reports all predate this branch.

Not in this PR

Reserved concurrency on the processing and artifacts Lambdas belongs in the architecture Terraform; the per-container cap here is only a floor until that lands.

🤖 Generated with Claude Code

Compressed archives are never science inputs, but 12,894 REACH tarballs
were run through the instrument package on 2026-09-08, each failing on
the file type after opening a fresh database pool. Skip them before any
download or MetaTracker write, and cache one single-connection engine
per Lambda container so a burst of invocations no longer exhausts the
tracker database's connection slots.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 8, 2026 17:52
Two whitespace-only differences in the existing tests made
black --check fail for the whole lambda_function tree, which is the same
gate the CodeBuild pre_build step runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The engine/table caches are currently keyed by password-bearing connection strings, which can accumulate pools and rerun table creation on secret rotation, undermining the “one connection per container” goal.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens the Lambda file-processing path against archive bursts by skipping known compressed archive inputs before any S3 download/instrument dispatch/tracking work occurs, and by reusing a single SQLAlchemy engine per warm container to reduce database connection pressure on MetaTracker.

Changes:

  • Add archive-suffix skip logic in handle_event and at the start of _process_file, with SDC_AWS_SKIP_SUFFIXES override support.
  • Cache and reuse a SQLAlchemy engine per container (single-connection pool) and run create_tables() once per container.
  • Add tests covering suffix matching/override behavior, skip short-circuiting, skip counts in handle_event, and engine reuse; document the behavior in README.md.
File summaries
File Description
README.md Documents the archive-skip behavior and single-engine-per-container approach.
lambda_function/tests/test_processor.py Adds tests for skip suffix logic, skip short-circuiting, skip-count response, and engine reuse.
lambda_function/src/file_processor/file_processor.py Implements archive skipping and container-scoped database engine reuse / one-time table creation.
Review details

Suppressed comments (1)

lambda_function/src/file_processor/file_processor.py:541

  • _TABLES_CREATED_FOR is keyed by connection_string, which includes the database password. If the secret rotates, this set will grow unbounded and create_tables() will run again even though it’s the same database. Keying table creation off a stable identifier that excludes the password keeps the “once per container” behavior intact across secret rotations.
            # Reuse this container's engine instead of opening a new pool
            engine = database_engine(connection_string)

            # Create tables once per container if they do not exist
            if connection_string not in _TABLES_CREATED_FOR:
                create_tables(engine)
                _TABLES_CREATED_FOR.add(connection_string)
  • Files reviewed: 4/4 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +80 to +104
"""
Return a cached SQLAlchemy engine for this connection string.

Parameters
----------
connection_string : str
PostgreSQL connection URL.

Returns
-------
Any
A shared ``sqlalchemy.engine.Engine`` with a single pooled connection,
pre-ping enabled, and connections recycled after five minutes.
"""
engine = _DATABASE_ENGINES.get(connection_string)
if engine is None:
engine = sqlalchemy_create_engine(
connection_string,
pool_size=1,
max_overflow=0,
pool_pre_ping=True,
pool_recycle=300,
)
_DATABASE_ENGINES[connection_string] = engine
return engine
jamesray and others added 2 commits September 8, 2026 11:21
Caching by the full connection URL would keep a stale pool alive for
every credential rotation a warm container lives through. Cache by the
secret ARN, remember the URL each engine was built from, dispose and
rebuild once when it changes, and track table creation per engine
instead of per URL.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The REACH sync no longer delivers archives anywhere the pipeline
listens, so the processor does not need to special-case them. The
per-container engine with a single pooled connection stays; that is
what stops a burst of invocations from exhausting the tracker database.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jamesray jamesray changed the title Skip archives and share one database connection per container Share one database connection per container Sep 9, 2026
@jamesray jamesray closed this Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants