Repository navigation
Add tests for reading dfsu files with a submesh custom block - #65
Merged
Merged
Conversation
From version 2027, the MIKE_FM custom block of a dfsu 2001 file may have 6 values, the 6th being the submesh type. Add test files with 5 and 6 value custom blocks, a locally refined file, and a legacy spectral file, and tests for reading them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ryan-kipawa
force-pushed
the
feature/submesh-backwards-compatibility
branch
from
September 29, 2026 11:29
302f48c to
9d29532
Compare
Only a data type 2001 file with a 6 value custom block and no frequency and direction static items is read as a submesh file. All other files, including other data types with 6 values, are interpreted as before. Add tests for edge cases: legacy spectral static item checks, frequency or direction only legacy files, 3D sigma submesh files, other data types with 6 values, and appending to submesh files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
__HasSpectralStaticItems reads static items 10 and 11, which moved the file pointer of the dfs file. It now saves the file pointer and moves it back before returning, using the new DfsFile.FindStaticItem for static items and FindItem for dynamic items. __Init reads the first static item with ReadStaticItemNext again, as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
__HasSpectralStaticItems now opens its own read-only DfsFile to check static items 10 and 11, so the file pointer of the dfsu file is never moved. This removes DfsFile.FindStaticItem and the file pointer restore. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
__IsSubMesh decides whether the "MIKE_FM" custom block is a submesh custom block, including the data type and custom block count checks that were at the call site. In build mode it returns False, as DfsuBuilder does not create files with a 6 value custom block. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Instead of opening a separate dfs file, __IsSubMesh reads static items 10 and 11 from the dfsu file, and __Init reads the first static item by number. In append mode, __Init already moves the file pointer to the end of the file after reading the static items. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Downstream libraries need to know whether a 2001 file has a submesh custom block, and telling one apart from a legacy spectral custom block requires the static item check in __IsSubMesh. Expose its result rather than have each library repeat it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Legacy spectral 2001 files have a 6 value custom block and 11 static items, where item 10 is the wave frequency in hertz and item 11 is the wave direction in radians. __IsSubMesh now requires exactly these items, with matching element counts, and reads any other 6 value custom block in a 2001 file as a submesh custom block. Frequency-only, direction-only, degree unit and extra static item variants are now read as submesh files. Tests follow the new rule. The 2004 test no longer asserts the number of sigma layers, which the dfsu specification reads from another custom block position, and the append test creates testdata/testtmp itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the tests that built modified copies of the test data files, and the CopyDfsuModified helper. Extend the append test to the legacy spectral file, and check the appended data and IsSubMesh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 2D submesh type 0 test now checks its expected values directly, as the locally refined test does, so Dfsu2D_CustomBlock5.dfsu is no longer needed. Correct the test data comment: the 3D sigma-z pair has the same mesh, but its dynamic data only matches to within float32 rounding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ryan-kipawa
marked this pull request as ready for review
September 29, 2026 14:37
…kwards-compatibility Master reformatted the code with ruff and narrowed the test imports. Reapply the submesh detection in DfsuFile.py on master's formatted version, and import os, numpy and assert_array_equal explicitly in test_dfsu_file.py, as tests/test_util.py no longer re-exports numpy.testing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DfsuFile decides the layout of a data type 2001 custom block by asking whether the file is the legacy Spectral0D exception, as the format owner asked, instead of whether it is a submesh file: 1. 5 values or fewer: pre-2027 layout, no submesh type. 2. Exactly 6 values, 11 static items, item 10 a frequency in hertz and item 11 a direction in radians or degrees (wave or generic EUM item), with lengths equal to spec values 5 and 6: legacy Spectral0D. 3. Otherwise, 6 values or more: MIKE 2027 layout, spec value 5 is the number of sigma layers, spec value 6 the submesh type, values from 7 on ignored. The guard that treated anything other than exactly 6 values as "not a submesh" is gone, so a 7 value file reads with the right number of sigma layers. Other data types read as on master. SubMeshType is None for files without a submesh type, otherwise 0 or 1. IsSubMesh is removed. In build mode, __Init takes the submesh type the builder wrote and uses it instead of reading the static items. Tests rewrite test data at test time with DfsBuilder: the 3D sigma-z file with a 7 value custom block, and the legacy Spectral0D file with its direction item in degrees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SetSubMeshType(None | 0 | 1), default None. None writes the custom block as before (5 values for non-spectral types). 0 or 1 writes 6 values with spec value 6 as the submesh type. Other values raise ValueError, and spectral builder types raise, as SetNumberOfSigmaLayers does for 2D. The mesh geometry is not checked. CreateDfsu passes the builder's submesh type to DfsuFile.DfsuFileBuild, so the returned DfsuFile reports the same SubMeshType as the file opened again from disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DfsuBuilder appends the submesh type where it builds the 2001 custom blocks, instead of replacing the last custom block. The spectral dfsu file types are defined once, and SetSubMeshType declares that it takes numpy integers. The build path in DfsuFile states that DfsuBuilder writes 6 values only with a submesh type, and raises if that no longer holds, so PR 76 must revisit it. The DfsuFile docstring describes the legacy Spectral0D rule and the 4 value layout, and the CHANGELOG describes what users can do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stray docstring after the return in OpenAppend made pyrefly infer that it may return None. Move it into DfsuFileBuild, where it belongs. Pass dfsFile to __IsLegacySpectral0D, and narrow Frequencies and Directions in the legacy Spectral0D test. The warning count drops from 338 to 328, under CI's ceiling of 330. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A legacy data type 2001 point spectrum with only frequencies or only directions, such as MIKE IO's pt_freq_spectra.dfsu, read as Dfsu1D after the legacy check started to require both axes. The check now accepts one static item per present axis (spec value > 0), at least one axis, and exactly 9 + number of present axes static items. The item count is checked by number, not by the read position, so a 6-value file with both spec values 0 is never the exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop references to PRs, issues and people, and keep only what the code needs to explain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…kwards-compatibility # Conflicts: # CHANGELOG.md
np.size rejected Frequencies and Directions, which may be None, under the Python 3.11 numpy stubs CI uses, adding 3 pyrefly warnings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #54
Summary
From MIKE 2027 (FM file specification, section 4.3), a data type 2001 dfsu file can have 6 values in its
MIKE_FMcustom block. Spec value 5 is the number of sigma layers, and spec value 6 is the submesh type: 1 for a local refinement submesh, 0 otherwise. Master reads every 6-value 2001 file as a legacy Spectral0D (rose-plot) file, so it can't open 2027 files that are 3D or locally refined. Following the format owner's review,DfsuFilenow decides whether a 2001 file is the legacy Spectral0D exception. Any other file with 6 or more values reads with the standard 2027 layout.How
DfsuFilereads a data type 2001 custom block:A file is the legacy Spectral0D exception only when all of these are true:
So both two-axis files (11 static items) and single-axis files, with frequencies only or directions only (10 static items), stay Spectral0D. An example of a single-axis file is MIKE IO's
pt_freq_spectra.dfsu.Data types 2000, 2002, 2003 and 2004 read as on master.
Public API:
DfsuBuilderpasses its submesh type to theDfsuFileit returns fromCreateFile, because the static items have not been written yet when it is created. When PR 76 adds Spectral0D writing, it must look at this hand-off again: with a 6-value block and no submesh type, building now raises.Evidence
Opening the
testdata/submesh/files withDfsuFile.Open:Dfsu3DSigma.After: it reads as
Dfsu3DSigmaZwith 1 sigma layer andSubMeshType == 0.Dfsu2Din the standard layout, not as Spectral0D.None, 0 and 1 check:SubMeshTypeon both the returned object and the file opened again from disk.uv run pytest: 211 passed, 1 skipped, 8 xfailed.main, 3.5.0, run): the same results as with master's mikecore (run) on Ubuntu and Windows, Python 3.12 and 3.14.test_eum_conversionfails on both OSes, andtest_select_area_rotated_UTM_2fails on Ubuntu only.Merge Danger
Door: two-way
The file writing is opt-in.
DfsuBuilderwrites the same 5-value block as today unlessSetSubMeshTypeis called. Once users have written files with a submesh type, mikecore releases before this one can't open them, except for 2D files with submesh type 0. TheSetSubMeshTypedocstring says so.Blast Radius: 2001-readers
IsSubMeshfrom the first version of this PR is removed. It was never released.🤖 Generated with Claude Code