feat: split raw data storage into per-dataset SQLite files - #8219

Draft

Mikhail Astafev (astafan8) wants to merge 23 commits into

microsoft:mainfrom

astafan8:feature/split-raw-data-sqlite

Commits on Jun 12, 2026

feat: split raw data storage into per-dataset SQLite files

Mikhail Astafev
and
Copilot
committed
efca7d8

Commits on Jun 30, 2026

address PR review feedback from Jens and Copilot

Mikhail Astafev
and
Copilot
committed
f9a3ea4
Add update_raw_data_paths helper for relocated raw data files

Mikhail Astafev
and
Copilot
committed
32b1c96
Add update_raw_data_paths documentation to intro.rst and Database.ipynb

Mikhail Astafev
and
Copilot
committed
d897d95

Commits on Jul 2, 2026

Merge branch 'main' into feature/split-raw-data-sqlite
Mikhail Astafev (astafan8)
authored
dce30f5
Fix ruff format and unused import lint issues

Mikhail Astafev
and
Copilot
committed
ce732d6
Fix pyright type errors in test_raw_data_storage

Mikhail Astafev
and
Copilot
committed
a195b32

Commits on Jul 3, 2026

Merge branch 'main' into feature/split-raw-data-sqlite
Mikhail Astafev (astafan8)
authored
5577d3a

Commits on Jul 16, 2026

Add purge_orphaned_datasets and cleanup_datasets management functions

Mikhail Astafev
and
Copilot
committed
07266e6
Add towncrier news fragment for split raw data storage

Mikhail Astafev
and
Copilot
committed
0f21ee9
Address PR review comments: move queries to queries.py, add prints, simplify

Mikhail Astafev
and
Copilot
committed
c76baad
Use log.debug for per-dataset reports, Path.unlink() for file removal

Mikhail Astafev
and
Copilot
committed
84e119d
Merge branch 'main' into feature/split-raw-data-sqlite
Mikhail Astafev (astafan8)
authored
8f4ff32

Commits on Jul 17, 2026

Fix extract/export and unsubscribe to use _data_conn for split storage

Mikhail Astafev
and
Copilot
committed
91dd316
Merge branch 'main' into feature/split-raw-data-sqlite
Mikhail Astafev (astafan8)
authored
ddbee78

Commits on Jul 18, 2026

Apply suggestions from code review

Mikhail Astafev (astafan8)
and
Copilot
authored
2aeb373

Commits on Jul 21, 2026

Update docs/dataset/dataset_design.rst
Mikhail Astafev (astafan8)
authored
7dd5e23
Reuse existing qcodes helpers instead of duplicating logic

Mikhail Astafev
and
Copilot
committed
ae3c879

Commits on Jul 23, 2026

Avoid empty results table in main DB for split raw data storage

Mikhail Astafev
and
Copilot
committed
a2edfac

Commits on Jul 27, 2026

Extract subscriber create/queue helpers for clarity

Mikhail Astafev
and
Copilot
committed
cc04231
Merge pull request #405 from astafan8/feature/split-raw-data-no-empty-table
Mikhail Astafev (astafan8)
authored
d4af63c
Address review: make sqlite-file connect helper private, use dataclass for query result

Mikhail Astafev
and
Copilot
committed
198f919
Update news fragment to mention no empty results table in main DB

Mikhail Astafev
and
Copilot
committed
94884fc