feat(connectors): add Meilisearch source connector#3498
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3498 +/- ##
============================================
- Coverage 73.99% 65.17% -8.82%
Complexity 937 937
============================================
Files 1301 1301
Lines 147393 133177 -14216
Branches 122945 108805 -14140
============================================
- Hits 109063 86799 -22264
- Misses 34858 42847 +7989
- Partials 3472 3531 +59
🚀 New features to boost your workflow:
|
|
overall correct direction - cursor-by-primary-key is the right shape for Meilisearch, test coverage is present but coverage gap — only happy-path covered. No test for: (a) state restored across restart → cursor starts from last_primary_key, not from zero; (b) second poll produces zero duplicates. refer The elasticsearch_source.rs has state_persists_across_connector_restart as a pattern. For a cursor-based source, duplicate-freedom is the core invariant; it should be integration-tested. |
|
/author |
github-actions
Bot
added
S-waiting-on-author
|
/ready |
github-actions
Bot
added
S-waiting-on-review
|
/author |
github-actions
Bot
added
S-waiting-on-author
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either Thank you for your contribution! |
… into feat/meilisearch-source-connector
|
/ready |
github-actions
Bot
added
S-waiting-on-review
| valid_documents_and_last_primary_key(documents, primary_key, self.id); | ||
| let messages = self.documents_to_messages(documents)?; | ||
|
|
||
| let mut state = self.state.lock().await; |
There was a problem hiding this comment.
298-301,657-677 — hits non-empty, all PK invalid → last_primary_key stays None → same page re-polled forever; valid docs behind stuck. Unit test :974-985 locks this. Fix: fail poll (or hard-error) when hits.len()>0 && valid.is_empty().
| )) | ||
| }) | ||
| } | ||
|
|
There was a problem hiding this comment.
lib.rs:238-258 — open checks sortableAttributes only; README + fixture require filterable too. Cursor filter pk > N fails at poll with PermanentHttpError after first batch. Fix: mirror sortable check for filterableAttributes in open.
| # under the License. | ||
|
|
||
| [package] | ||
| name = "iggy_connector_meilisearch_source" |
There was a problem hiding this comment.
version 0.4.0; peers 0.4.1-edge.1. Fix: validate and align.
| let (documents, last_document_primary_key) = | ||
| valid_documents_and_last_primary_key(documents, primary_key, self.id); | ||
| let messages = self.documents_to_messages(documents)?; | ||
|
|
There was a problem hiding this comment.
298-305 + runtime/source.rs:464-517 — in-memory cursor advance before Iggy send; send fail ⇒ batch lost in live process. README:72-76 documents. Fix: #3635-class ack/rollback
| Error::Connection(format!("Failed to create Meilisearch client: {error}")) | ||
| }) | ||
| } | ||
|
|
There was a problem hiding this comment.
187-216 — health outer loop nests retry_sdk_open_operation(max_open_retries); budgets multiply. Fix: single open retry budget.
| (None, None) => None, | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
restored non-integer last_primary_key → every poll Err; no open validate. Fix: validate/wipe cursor at open.
| } else { | ||
| document | ||
| }; | ||
|
|
There was a problem hiding this comment.
ProducedMessage.id always None; integer PK known. ES same (:418). Fix: map PK→id optional.
|
/author |
github-actions
Bot
added
S-waiting-on-author
3 participants
Summary
meilisearch-sdkMotivation
This splits the source portion out of #3404 so the Meilisearch source can be reviewed and merged independently from the sink connector.
Closes #3496.
Validation
cargo sort --no-format --workspacecargo test -p iggy_connector_meilisearch_sourcecargo clippy -p iggy_connector_meilisearch_source --all-targets -- -D warningsenv CARGO_BIN_EXE_iggy-server=... CARGO_BIN_EXE_iggy-connectors=... cargo test -p integration --test mod -- connectors::meilisearch::meilisearch_sourcegit diff --check