Repository navigation
Conversation
Compressed archives are never science inputs, but 12,894 REACH tarballs were run through the instrument package on 2026-09-08, each failing on the file type after opening a fresh database pool. Skip them before any download or MetaTracker write, and cache one single-connection engine per Lambda container so a burst of invocations no longer exhausts the tracker database's connection slots. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two whitespace-only differences in the existing tests made black --check fail for the whole lambda_function tree, which is the same gate the CodeBuild pre_build step runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The engine/table caches are currently keyed by password-bearing connection strings, which can accumulate pools and rerun table creation on secret rotation, undermining the “one connection per container” goal.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR hardens the Lambda file-processing path against archive bursts by skipping known compressed archive inputs before any S3 download/instrument dispatch/tracking work occurs, and by reusing a single SQLAlchemy engine per warm container to reduce database connection pressure on MetaTracker.
Changes:
- Add archive-suffix skip logic in
handle_eventand at the start of_process_file, withSDC_AWS_SKIP_SUFFIXESoverride support. - Cache and reuse a SQLAlchemy engine per container (single-connection pool) and run
create_tables()once per container. - Add tests covering suffix matching/override behavior, skip short-circuiting, skip counts in
handle_event, and engine reuse; document the behavior inREADME.md.
File summaries
| File | Description |
|---|---|
| README.md | Documents the archive-skip behavior and single-engine-per-container approach. |
| lambda_function/tests/test_processor.py | Adds tests for skip suffix logic, skip short-circuiting, skip-count response, and engine reuse. |
| lambda_function/src/file_processor/file_processor.py | Implements archive skipping and container-scoped database engine reuse / one-time table creation. |
Review details
Suppressed comments (1)
lambda_function/src/file_processor/file_processor.py:541
_TABLES_CREATED_FORis keyed byconnection_string, which includes the database password. If the secret rotates, this set will grow unbounded andcreate_tables()will run again even though it’s the same database. Keying table creation off a stable identifier that excludes the password keeps the “once per container” behavior intact across secret rotations.
# Reuse this container's engine instead of opening a new pool
engine = database_engine(connection_string)
# Create tables once per container if they do not exist
if connection_string not in _TABLES_CREATED_FOR:
create_tables(engine)
_TABLES_CREATED_FOR.add(connection_string)
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| """ | ||
| Return a cached SQLAlchemy engine for this connection string. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| connection_string : str | ||
| PostgreSQL connection URL. | ||
|
|
||
| Returns | ||
| ------- | ||
| Any | ||
| A shared ``sqlalchemy.engine.Engine`` with a single pooled connection, | ||
| pre-ping enabled, and connections recycled after five minutes. | ||
| """ | ||
| engine = _DATABASE_ENGINES.get(connection_string) | ||
| if engine is None: | ||
| engine = sqlalchemy_create_engine( | ||
| connection_string, | ||
| pool_size=1, | ||
| max_overflow=0, | ||
| pool_pre_ping=True, | ||
| pool_recycle=300, | ||
| ) | ||
| _DATABASE_ENGINES[connection_string] = engine | ||
| return engine |
Caching by the full connection URL would keep a stale pool alive for every credential rotation a warm container lives through. Cache by the secret ARN, remember the URL each engine was built from, dispose and rebuild once when it changes, and track table creation per engine instead of per URL. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The REACH sync no longer delivers archives anywhere the pipeline listens, so the processor does not need to special-case them. The per-container engine with a single pooled connection stays; that is what stops a burst of invocations from exhausting the tracker database. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Why
On 2026-09-08 a REACH backfill fired this Lambda 12,894 times in under an hour. Every invocation created a new SQLAlchemy engine, and the idle pooled connections left behind by each warm container pinned both
db.t4g.microMetaTracker databases at their ~70 connection limit: 5,283remaining connection slots are reservederrors in prod, 5,146 in dev, and 2,465 foreign-key violations where the status row landed but the file row had not. Grafana lost its database connection as collateral.The archives themselves are handled upstream now (swxsoc/sdc_aws_reach_sync#3 delivers them to a bucket nothing listens to), so this PR no longer special-cases them. It fixes the part that is this Lambda's own: connection churn.
What changes
database_engine(secret_arn, url)caches asqlalchemyengine per secret ARN withpool_size=1,max_overflow=0,pool_pre_ping=True, andpool_recycle=300. The cache key is the secret ARN, never the URL, because the URL carries the password: each entry remembers the URL it was built from, and a rotated credential disposes the old pool and builds exactly one replacement.ensure_tracker_tablesrunscreate_tablesonce per cached engine rather than on every event, and again after a rotation rebuilds the engine.Tests cover engine reuse per secret, engine replacement on rotation with the old pool disposed, and once-per-engine table creation.
ruff check,ruff format, andblack --checkare clean for the whole tree (the test module needed two whitespace fixes to get black green, which is the same gate the CodeBuild pre_build runs). The fourE501lines flake8 reports all predate this branch.Not in this PR
Reserved concurrency on the processing and artifacts Lambdas belongs in the architecture Terraform; the per-container cap here is only a floor until that lands.
🤖 Generated with Claude Code