Skip to content

fix: add configurable S3 source error handling - #339

Open
neha-5456 wants to merge 1 commit into
codeforstartups:developmentfrom
neha-5456:fix/323-s3-source-error-handling
Open

neha-5456 wants to merge 1 commit into
codeforstartups:developmentfrom
neha-5456:fix/323-s3-source-error-handling

Conversation

@neha-5456

Copy link
Copy Markdown
Contributor

Summary

Adds configurable error handling to S3Source so document parsing failures can either be raised or skipped without stopping the entire ingestion pipeline.

Fixes #323

Changes

  • Added on_error option to S3Source:
    • "raise" — default behavior; parsing errors are propagated.
    • "skip" — logs the parsing failure and continues processing remaining S3 objects.
  • Added validation for unsupported on_error values.
  • Added warning logging when an object is skipped.
  • Ensured S3 access/download errors such as get_object failures are still propagated and are not silently skipped.
  • Updated Markdown and CSV decoding so parsing failures follow the configured error-handling policy.
  • Added tests covering:
    • Default raise behavior.
    • Skip behavior and continued ingestion.
    • Invalid on_error values.
    • Warning logging.
    • S3 get_object errors.
    • Invalid Markdown front matter.

Testing

uv run --no-sync pytest tests/test_s3_source.py -q

Result: 11 passed

uv run --no-sync ruff check src/dynavec/ingest.py tests/test_s3_source.py

Result: All checks passed!

uv run --no-sync mypy src/dynavec/ingest.py

Result: Success: no issues found in 1 source file

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(ingest): Add configurable error policy (raise vs skip) to S3Source

2 participants