Skip to content

[Claude-led] Feature/spec collate - #6

Draft
dougbrn wants to merge 2 commits into
devfrom
feature/spec-collate
Draft

dougbrn wants to merge 2 commits into
devfrom
feature/spec-collate

Conversation

@dougbrn

@dougbrn dougbrn commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Change Description

First implementation PR for the initial design (docs/design/initial-design.md, §4).

Solution Description

  • Adds lsdb>=0.11 and torch>=2.4 as core dependencies.
  • ColumnSpec / Flat / Nested: maps flat and nested catalog columns to tensors. resolve() validates against catalog meta (nested sub-column types via NestedDtype.column_dtypes) without computing any data, and reports every problem in one error.
  • Nested columns are padded to the batch maximum with a <key>_mask and <key>_lengths, fully vectorized with nest.len() / nest.to_flat(). Rows with empty or null nested values have length 0. Options: max_length with head/tail truncation, pad_to, pad_value, null fill, transform=.
  • resolve_device / move_batch helpers (default "cpu", "auto" opt-in).

Tests compare against a per-row reference implementation and include a fixture catalog that raises if any partition is computed.

Note: CI only triggers on main, so this PR gets no CI until dev is added to the workflow triggers (design doc §12).

🤖 Generated with Claude Code

dougbrn and others added 2 commits October 15, 2026 11:58
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ColumnSpec maps flat and nested catalog columns to tensors. resolve()
validates against catalog meta (nested sub-column dtypes come from
NestedDtype.column_dtypes) without computing data, and reports every
problem at once. Nested columns are padded to the batch maximum with a
mask and lengths, fully vectorized via nest.len() and nest.to_flat().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougbrn
dougbrn changed the base branch from main to dev October 15, 2026 19:45
@codecov

codecov Bot commented Oct 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.50173% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.59%. Comparing base (c9ed114) to head (e266875).

Files with missing lines Patch % Lines
src/lsdb_ml/spec.py 95.79% 9 Missing ⚠️
src/lsdb_ml/device.py 86.36% 3 Missing ⚠️
src/lsdb_ml/collate.py 98.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##               dev       #6      +/-   ##
===========================================
- Coverage   100.00%   95.59%   -4.41%     
===========================================
  Files            2        5       +3     
  Lines            7      295     +288     
===========================================
+ Hits             7      282     +275     
- Misses           0       13      +13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant