Skip to content

Commit 285bc69

Browse files
andinuxclaude
andcommitted
feat(network): drain-all receive with chunk cap; rename to receive_changes
Rework the chunked-download client API for ergonomics and caller control. - cloudsync_network_sync() now drains an entire chunked /check stream in a single call, fetching already-available chunks back-to-back with no delay. wait_ms/max_retries are spent only while the server payload is not yet ready (HTTP 202), not while paging through chunks already available. - Add cloudsync_network_receive_changes([max_chunks]) as the canonical receive function: drains all available chunks by default; max_chunks caps pages per call for progress/traffic control, resuming across calls via the in-memory page cursor. cloudsync_network_check_changes() is retained as a deprecated, fully-functional alias (removed in a future major). - Add a shared network_drain_changes() helper backing both sync and receive_changes. - Surface new JSON fields: receive.chunks/bytes/complete and send.chunks/bytes. receive.rows and receive.tables are now cumulative across the whole drain. Docs (API.md, CHANGELOG), integration tests (single-sync drain, capped receive, and deliberate alias coverage), the sync benchmark, the example apps, and the .claude command docs are migrated to the new name. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 95ede89 commit 285bc69

11 files changed

Lines changed: 478 additions & 110 deletions

File tree

‎.claude/commands/stress-test-sync-sqlitecloud.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -114,13 +114,13 @@ Create a bash script at `/tmp/stress_test_concurrent.sh` that:
114114
- Each iteration does:
115115
a. **UPDATE** — run `UPDATE <table> SET value = value + 1;` repeated `NUM_UPDATES` times (skip if 0)
116116
b. **DELETE** — run `DELETE FROM <table> WHERE rowid IN (SELECT rowid FROM <table> ORDER BY RANDOM() LIMIT 10);` repeated `NUM_DELETES` times (skip if 0)
117-
c. **Sync using the 3-step send/check/check pattern:**
117+
c. **Sync using the 3-step send/receive/receive pattern:**
118118
1. `SELECT cloudsync_network_send_changes();` — send local changes to the server
119-
2. `SELECT cloudsync_network_check_changes();` — ask the server to prepare a payload of remote changes
119+
2. `SELECT cloudsync_network_receive_changes();` — ask the server to prepare a payload of remote changes
120120
3. Sleep 1 second (outside sqlite3, between two separate sqlite3 invocations)
121-
4. `SELECT cloudsync_network_check_changes();` — download the prepared payload, if any
121+
4. `SELECT cloudsync_network_receive_changes();` — download the prepared payload, if any
122122
- Each sqlite3 session must: `.load` the extension, call `cloudsync_network_init()`/`cloudsync_network_init_custom()`, `cloudsync_network_set_apikey()`/`cloudsync_network_set_token()` (depending on RLS mode), do the work, call `cloudsync_terminate()`
123-
- **Timing**: Log the wall-clock execution time (in milliseconds) for each `cloudsync_network_send_changes()`, `cloudsync_network_check_changes()` call. Define a `now_ms()` helper function at the top of the script and use it before and after each sqlite3 invocation that calls a network function, computing the delta. On **macOS**, `date` does not support `%3N` (nanoseconds) — use `python3 -c 'import time; print(int(time.time()*1000))'` instead. On **Linux**, `date +%s%3N` works fine. The script should detect the platform and define `now_ms()` accordingly. Log lines like: `[DB<N>][iter <I>] send_changes: 123ms`, `[DB<N>][iter <I>] check_changes_1: 45ms`, `[DB<N>][iter <I>] check_changes_2: 67ms`
123+
- **Timing**: Log the wall-clock execution time (in milliseconds) for each `cloudsync_network_send_changes()`, `cloudsync_network_receive_changes()` call. Define a `now_ms()` helper function at the top of the script and use it before and after each sqlite3 invocation that calls a network function, computing the delta. On **macOS**, `date` does not support `%3N` (nanoseconds) — use `python3 -c 'import time; print(int(time.time()*1000))'` instead. On **Linux**, `date +%s%3N` works fine. The script should detect the platform and define `now_ms()` accordingly. Log lines like: `[DB<N>][iter <I>] send_changes: 123ms`, `[DB<N>][iter <I>] receive_changes_1: 45ms`, `[DB<N>][iter <I>] receive_changes_2: 67ms`
124124
- Include labeled output lines like `[DB<N>][iter <I>] updated count=<C>, deleted count=<D>` for grep-ability
125125

126126
3. **Launches all workers in parallel** using `&` and collects PIDs
@@ -138,7 +138,7 @@ Create a bash script at `/tmp/stress_test_concurrent.sh` that:
138138
- Use `echo -e` to pipe generated SQL (with `\n` separators) into sqlite3
139139
- During database initialization (Step 1), insert `ROWS` initial rows per database in a single transaction so each DB starts with data to update/delete. Row IDs should be unique across databases: `db<N>_r<J>`
140140
- User IDs for rows must match the token's userId for RLS to work
141-
- The sync pattern requires **separate sqlite3 invocations** for send_changes and each check_changes call (with a 1-second sleep between the two check_changes calls), so that timing can be measured per-call from bash
141+
- The sync pattern requires **separate sqlite3 invocations** for send_changes and each receive_changes call (with a 1-second sleep between the two receive_changes calls), so that timing can be measured per-call from bash
142142
- **stderr capture**: All sqlite3 invocations must redirect both stdout and stderr to the log file. Use `>> "$LOG" 2>&1` (in this order — stdout redirect first, then stderr to stdout). For timed calls that capture output in a variable, redirect stderr to the log file separately: `RESULT=$(echo -e "$SQL" | $SQLITE3 "$DB" 2>> "$LOG")` and then echo `$RESULT` to the log as well. This ensures "Runtime error" messages from sqlite3 are never lost.
143143
- Use `/bin/bash` (not `/bin/sh`) for arrays and process management
144144

@@ -191,7 +191,7 @@ Report the test results including:
191191
| Rows per iteration | ROWS |
192192
| Iterations per database | ITERATIONS |
193193
| Total CRUD operations | N × ITERATIONS × (UPDATE_ALL + DELETE_FEW) |
194-
| Total sync operations | N × ITERATIONS × 3 (1 send_changes + 2 check_changes) |
194+
| Total sync operations | N × ITERATIONS × 3 (1 send_changes + 2 receive_changes) |
195195
| Duration | start to finish time |
196196
| Total errors | count |
197197
| Error types | categorized list |

‎.claude/commands/test-sync-roundtrip-sqlitecloud-rls.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -267,16 +267,16 @@ For each of the four SQLite databases, execute the sync operations:
267267
SELECT cloudsync_network_send_changes();
268268

269269
-- Check for changes from server (repeat with 2-3 second delays)
270-
SELECT cloudsync_network_check_changes();
271-
-- Repeat check_changes 3-5 times with delays until it returns more than 0 received rows or stabilizes
270+
SELECT cloudsync_network_receive_changes();
271+
-- Repeat receive_changes 3-5 times with delays until it returns more than 0 received rows or stabilizes
272272
```
273273

274274
**Recommended sync order:**
275275
1. Sync Database 1A (send + check)
276276
2. Sync Database 2A (send + check)
277277
3. Sync Database 1B (send + check)
278278
4. Sync Database 2B (send + check)
279-
5. Re-sync all databases (check_changes) to ensure full propagation
279+
5. Re-sync all databases (receive_changes) to ensure full propagation
280280

281281
### Step 10: Verify RLS Enforcement
282282

@@ -333,7 +333,7 @@ SELECT COUNT(*) FROM <table_name> WHERE id = 'malicious_1';
333333
**Also verify the malicious row does NOT appear in User 2's databases after syncing:**
334334
```sql
335335
-- In Database 2A or 2B (User 2)
336-
SELECT cloudsync_network_check_changes();
336+
SELECT cloudsync_network_receive_changes();
337337
SELECT * FROM <table_name> WHERE id = 'malicious_1';
338338
-- Expected: 0 rows (the malicious row should not sync to legitimate User 2 databases)
339339
```
@@ -405,7 +405,7 @@ The test FAILS if:
405405
- Always use the Homebrew sqlite3 binary, NOT `/usr/bin/sqlite3`
406406
- The cloudsync extension must be built first with `make`
407407
- SQLiteCloud tables need cleanup before re-running tests
408-
- `cloudsync_network_check_changes()` may need multiple calls with delays
408+
- `cloudsync_network_receive_changes()` may need multiple calls with delays
409409
- Run `SELECT cloudsync_terminate();` on SQLite connections before closing to properly cleanup memory
410410
- Ensure both test users exist in Supabase auth before running the test
411411
- The RLS policies must use `auth_userid()` to work with SQLiteCloud token authentication

‎.claude/commands/test-sync-roundtrip-supabase-rls.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -275,16 +275,16 @@ For each of the four SQLite databases, execute the sync operations:
275275
SELECT cloudsync_network_send_changes();
276276

277277
-- Check for changes from server (repeat with 2-3 second delays)
278-
SELECT cloudsync_network_check_changes();
279-
-- Repeat check_changes 3-5 times with delays until it returns more than 0 received rows or stabilizes
278+
SELECT cloudsync_network_receive_changes();
279+
-- Repeat receive_changes 3-5 times with delays until it returns more than 0 received rows or stabilizes
280280
```
281281

282282
**Recommended sync order:**
283283
1. Sync Database 1A (send + check)
284284
2. Sync Database 2A (send + check)
285285
3. Sync Database 1B (send + check)
286286
4. Sync Database 2B (send + check)
287-
5. Re-sync all databases (check_changes) to ensure full propagation
287+
5. Re-sync all databases (receive_changes) to ensure full propagation
288288

289289
### Step 10: Verify RLS Enforcement
290290

@@ -341,7 +341,7 @@ SELECT COUNT(*) FROM <table_name> WHERE id = 'malicious_1';
341341
**Also verify the malicious row does NOT appear in User 2's databases after syncing:**
342342
```sql
343343
-- In Database 2A or 2B (User 2)
344-
SELECT cloudsync_network_check_changes();
344+
SELECT cloudsync_network_receive_changes();
345345
SELECT * FROM <table_name> WHERE id = 'malicious_1';
346346
-- Expected: 0 rows (the malicious row should not sync to legitimate User 2 databases)
347347
```
@@ -414,7 +414,7 @@ The test FAILS if:
414414
- Always use the Homebrew sqlite3 binary, NOT `/usr/bin/sqlite3`
415415
- The cloudsync extension must be built first with `make`
416416
- PostgreSQL tables need cleanup before re-running tests
417-
- `cloudsync_network_check_changes()` may need multiple calls with delays
417+
- `cloudsync_network_receive_changes()` may need multiple calls with delays
418418
- Run `SELECT cloudsync_terminate();` on SQLite connections before closing to properly cleanup memory
419419
- Ensure both test users exist in Supabase auth before running the test
420420
- The RLS policies must use `auth.uid()` to work with Supabase JWT authentication

‎.claude/commands/test-sync-roundtrip-supabase.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -136,8 +136,8 @@ In the SQLite session:
136136
SELECT cloudsync_network_send_changes();
137137

138138
-- Check for changes from server (repeat with 2-3 second delays)
139-
SELECT cloudsync_network_check_changes();
140-
-- Repeat check_changes 3-5 times with delays until it returns more than 0 received rows or stabilizes
139+
SELECT cloudsync_network_receive_changes();
140+
-- Repeat receive_changes 3-5 times with delays until it returns more than 0 received rows or stabilizes
141141

142142
-- Verify final data
143143
SELECT * FROM <table_name>;
@@ -164,7 +164,7 @@ Report the test results including:
164164
- Always use the Homebrew sqlite3 binary, NOT `/usr/bin/sqlite3`
165165
- The cloudsync extension must be built first with `make`
166166
- PostgreSQL tables need cleanup before re-running tests
167-
- `cloudsync_network_check_changes()` may need multiple calls with delays
167+
- `cloudsync_network_receive_changes()` may need multiple calls with delays
168168
- run `SELECT cloudsync_terminate();` on SQLite connections before closing the properly cleanup the memory
169169

170170
## Permissions

0 commit comments

Comments
 (0)