Repository navigation
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ColumnSpec maps flat and nested catalog columns to tensors. resolve() validates against catalog meta (nested sub-column dtypes come from NestedDtype.column_dtypes) without computing data, and reports every problem at once. Nested columns are padded to the batch maximum with a mask and lengths, fully vectorized via nest.len() and nest.to_flat(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Partitions are sharded across DDP ranks and DataLoader workers from a shared per-epoch permutation, so every row is read exactly once per epoch for any worker count and world size. Each shard streams its partitions through lsdb's CatalogStream; batches are cut within a chunk and never span chunks. Supports fixed epochs and open-ended streaming. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Context-manager session modeled on Hyrax's train_stream: iterate (batch, metrics), stop(), close() returns the model. The optimization step is a user callable (SupervisedStep covers the common case). Periodic and final atomic checkpoints, rank-0 only; DDP runs under model.join() so uneven batch counts across ranks do not hang. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #7 +/- ##
===========================================
- Coverage 100.00% 95.42% -4.58%
===========================================
Files 2 14 +12
Lines 7 634 +627
===========================================
+ Hits 7 605 +598
- Misses 0 29 +29 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
Training path from the initial design (§5–6). Stacked on the spec/collate PR; merge that first.
Solution Description
CatalogIterableDataset: streams collated batches from an LSDB catalog through lsdb'sCatalogStream. Partitions are sharded across DDP ranks and DataLoader workers from a shared per-epoch permutation, so every row appears exactly once per epoch for anynum_workersand world size. Batches never span a chunk;batch_size=Noneyields whole chunks. Supportsepochs=N(reshuffled per epoch) andepochs=None(open-ended).TrainSession: context-manager loop modeled on Hyrax'strain_stream— iterate(batch, metrics),stop(),close()returns the model. Skips empty and below-min_batch_sizebatches; periodic and final atomic checkpoints (rank 0 only);resume_from; callbacks as the logging hook; DDP runs undermodel.join().SupervisedStepcovers loss + optimizer; anystep(model, batch) -> metricscallable works, which is the seam for a future Hyrax backend.docs/examples/train_from_catalog.py.Tests: no duplicated or dropped rows across workers and simulated ranks, epoch reshuffling and reproducibility, batches staying within chunks, empty partitions, distributed client path, session behaviour, and a real 2-process gloo DDP test with uneven batch counts.
🤖 Generated with Claude Code