Skip to content

Add tests for reading dfsu files with a submesh custom block - #65

Merged
ryan-kipawa merged 23 commits into
masterfrom
feature/submesh-backwards-compatibility
Nov 11, 2026
Merged

ryan-kipawa merged 23 commits into
masterfrom
feature/submesh-backwards-compatibility

Conversation

@ryan-kipawa

@ryan-kipawa ryan-kipawa commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #54

Summary

From MIKE 2027 (FM file specification, section 4.3), a data type 2001 dfsu file can have 6 values in its MIKE_FM custom block. Spec value 5 is the number of sigma layers, and spec value 6 is the submesh type: 1 for a local refinement submesh, 0 otherwise. Master reads every 6-value 2001 file as a legacy Spectral0D (rose-plot) file, so it can't open 2027 files that are 3D or locally refined. Following the format owner's review, DfsuFile now decides whether a 2001 file is the legacy Spectral0D exception. Any other file with 6 or more values reads with the standard 2027 layout.

How DfsuFile reads a data type 2001 custom block:

 read MIKE_FM custom block
-if 6 values
-  values 5, 6 = frequencies, directions               # every 6-value 2001 file
+if 5 values or fewer                                 # before MIKE 2027
+  value 5 (if present) = sigma layers; SubMeshType = None
+elif exactly 6 values and is legacy Spectral0D file
+  values 5, 6 = frequencies, directions; SubMeshType = None
+else                                                 # 6+ values: MIKE 2027 layout
+  value 5 = sigma layers; value 6 = SubMeshType; values 7+ ignored

A file is the legacy Spectral0D exception only when all of these are true:

  • it has exactly 6 values;
  • an axis is present when its spec value (5 for frequencies, 6 for directions) is greater than 0, and at least one is present;
  • after the 9 standard static items it has exactly one static item per present axis, and no more;
  • the frequency item (if present) comes first, is a frequency in hertz (wave or generic EUM item type), and is as long as spec value 5;
  • the direction item (if present) follows it, is a direction in radians or degrees (wave or generic EUM item type), and is as long as spec value 6.

So both two-axis files (11 static items) and single-axis files, with frequencies only or directions only (10 static items), stay Spectral0D. An example of a single-axis file is MIKE IO's pt_freq_spectra.dfsu.

Data types 2000, 2002, 2003 and 2004 read as on master.

Public API:

 DfsuFile
+  SubMeshType            # None | 0 | 1. None = file has no submesh type (compare with `is None`)
 DfsuBuilder
+  SetSubMeshType(value)  # None (default: writes 5 values, as today) | 0 | 1 (writes 6 values)
+                         # ValueError for other values; raises for spectral builders

DfsuBuilder passes its submesh type to the DfsuFile it returns from CreateFile, because the static items have not been written yet when it is created. When PR 76 adds Spectral0D writing, it must look at this hand-off again: with a 6-value block and no submesh type, building now raises.

Evidence

Opening the testdata/submesh/ files with DfsuFile.Open:

  • Before (master):
    Dfsu2D_CustomBlock6_Submesh0                  Dfsu2D          layers=0 sigma=0
    Dfsu2D_CustomBlock6_Submesh1_LocallyRefined   ERROR: File is not a valid dfsu file. It can not be opened
    Dfsu3DSigmaZ_CustomBlock5                     Dfsu3DSigmaZ    layers=3 sigma=1
    Dfsu3DSigmaZ_CustomBlock6_Submesh0            ERROR: File is not a valid dfsu file. It can not be opened
    DfsuSpectral0D_CustomBlock6_Legacy            DfsuSpectral0D  layers=0 sigma=0
    
    After:
    Dfsu2D_CustomBlock6_Submesh0                  Dfsu2D          layers=0 sigma=0 SubMeshType=0
    Dfsu2D_CustomBlock6_Submesh1_LocallyRefined   Dfsu2D          layers=0 sigma=0 SubMeshType=1
    Dfsu3DSigmaZ_CustomBlock5                     Dfsu3DSigmaZ    layers=3 sigma=1 SubMeshType=None
    Dfsu3DSigmaZ_CustomBlock6_Submesh0            Dfsu3DSigmaZ    layers=3 sigma=1 SubMeshType=0
    DfsuSpectral0D_CustomBlock6_Legacy            DfsuSpectral0D  layers=0 sigma=0 SubMeshType=None
    
  • Before (first version of this PR): the 3D sigma-z file, rewritten with a 7-value custom block, reads as Dfsu3DSigma.
    After: it reads as Dfsu3DSigmaZ with 1 sigma layer and SubMeshType == 0.
  • Tests make variants of these files at test time; no new binary test files are added. The variants cover:
    • a legacy Spectral0D file with directions in degrees, which reads as Spectral0D;
    • a legacy Spectral0D file with generic EUM item types, which reads as Spectral0D;
    • a legacy Spectral0D file with 7 values, which reads as Dfsu2D in the standard layout, not as Spectral0D.
    • single-axis legacy files, frequency-only and direction-only, which read as Spectral0D. A file whose single static item is not a frequency or direction reads in the standard layout.
  • Builder round trips for 2D and 3D sigma-z with None, 0 and 1 check:
    • the number of custom block values on disk;
    • SubMeshType on both the returned object and the file opened again from disk.
  • Checks:
    • uv run pytest: 211 passed, 1 skipped, 8 xfailed.
    • pre-commit passes.
    • pyrefly: 0 errors, 328 warnings (ceiling 330).
  • Downstream MIKE IO (main, 3.5.0, run): the same results as with master's mikecore (run) on Ubuntu and Windows, Python 3.12 and 3.14.
    • Ubuntu: 1157 passed. Windows: 1057 passed, 101 skipped.
    • Every run fails the same tests as with master's mikecore. test_eum_conversion fails on both OSes, and test_select_area_rotated_UTM_2 fails on Ubuntu only.
    • All 30 dfsu files in MIKE IO's test data read the same as on master.

Merge Danger

Door: two-way

The file writing is opt-in. DfsuBuilder writes the same 5-value block as today unless SetSubMeshType is called. Once users have written files with a submesh type, mikecore releases before this one can't open them, except for 2D files with submesh type 0. The SetSubMeshType docstring says so.

Blast Radius: 2001-readers

  • Only data type 2001 files with 6 or more custom block values read differently.
  • A 6-value 2001 file that master read as Spectral0D, but that fails the exception check, now reads with the 2027 layout. An example is a file whose spectral static items have the wrong unit or length. All 30 dfsu files in MIKE IO's test data read the same as on master.
  • IsSubMesh from the first version of this PR is removed. It was never released.
  • PR 76 must be rebased onto this and must revisit the build-time hand-off.

🤖 Generated with Claude Code

ryan-kipawa and others added 2 commits September 29, 2026 13:29
From version 2027, the MIKE_FM custom block of a dfsu 2001 file may have
6 values, the 6th being the submesh type. Add test files with 5 and 6
value custom blocks, a locally refined file, and a legacy spectral file,
and tests for reading them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ryan-kipawa
ryan-kipawa force-pushed the feature/submesh-backwards-compatibility branch from 302f48c to 9d29532 Compare September 29, 2026 11:29
ryan-kipawa and others added 10 commits September 29, 2026 13:40
Only a data type 2001 file with a 6 value custom block and no frequency
and direction static items is read as a submesh file. All other files,
including other data types with 6 values, are interpreted as before.

Add tests for edge cases: legacy spectral static item checks, frequency
or direction only legacy files, 3D sigma submesh files, other data types
with 6 values, and appending to submesh files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
__HasSpectralStaticItems reads static items 10 and 11, which moved the
file pointer of the dfs file. It now saves the file pointer and moves it
back before returning, using the new DfsFile.FindStaticItem for static
items and FindItem for dynamic items. __Init reads the first static item
with ReadStaticItemNext again, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
__HasSpectralStaticItems now opens its own read-only DfsFile to check
static items 10 and 11, so the file pointer of the dfsu file is never
moved. This removes DfsFile.FindStaticItem and the file pointer restore.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
__IsSubMesh decides whether the "MIKE_FM" custom block is a submesh
custom block, including the data type and custom block count checks
that were at the call site. In build mode it returns False, as
DfsuBuilder does not create files with a 6 value custom block.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Instead of opening a separate dfs file, __IsSubMesh reads static items
10 and 11 from the dfsu file, and __Init reads the first static item by
number. In append mode, __Init already moves the file pointer to the
end of the file after reading the static items.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Downstream libraries need to know whether a 2001 file has a submesh
custom block, and telling one apart from a legacy spectral custom block
requires the static item check in __IsSubMesh. Expose its result rather
than have each library repeat it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Legacy spectral 2001 files have a 6 value custom block and 11 static
items, where item 10 is the wave frequency in hertz and item 11 is the
wave direction in radians. __IsSubMesh now requires exactly these items,
with matching element counts, and reads any other 6 value custom block
in a 2001 file as a submesh custom block. Frequency-only, direction-only,
degree unit and extra static item variants are now read as submesh files.

Tests follow the new rule. The 2004 test no longer asserts the number of
sigma layers, which the dfsu specification reads from another custom
block position, and the append test creates testdata/testtmp itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the tests that built modified copies of the test data files, and
the CopyDfsuModified helper. Extend the append test to the legacy
spectral file, and check the appended data and IsSubMesh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 2D submesh type 0 test now checks its expected values directly, as
the locally refined test does, so Dfsu2D_CustomBlock5.dfsu is no longer
needed. Correct the test data comment: the 3D sigma-z pair has the same
mesh, but its dynamic data only matches to within float32 rounding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ryan-kipawa
ryan-kipawa marked this pull request as ready for review September 29, 2026 14:37
ryan-kipawa and others added 11 commits November 10, 2026 10:18
…kwards-compatibility

Master reformatted the code with ruff and narrowed the test imports.
Reapply the submesh detection in DfsuFile.py on master's formatted
version, and import os, numpy and assert_array_equal explicitly in
test_dfsu_file.py, as tests/test_util.py no longer re-exports
numpy.testing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DfsuFile decides the layout of a data type 2001 custom block by asking
whether the file is the legacy Spectral0D exception, as the format owner
asked, instead of whether it is a submesh file:

1. 5 values or fewer: pre-2027 layout, no submesh type.
2. Exactly 6 values, 11 static items, item 10 a frequency in hertz and
   item 11 a direction in radians or degrees (wave or generic EUM item),
   with lengths equal to spec values 5 and 6: legacy Spectral0D.
3. Otherwise, 6 values or more: MIKE 2027 layout, spec value 5 is the
   number of sigma layers, spec value 6 the submesh type, values from 7
   on ignored.

The guard that treated anything other than exactly 6 values as "not a
submesh" is gone, so a 7 value file reads with the right number of sigma
layers. Other data types read as on master.

SubMeshType is None for files without a submesh type, otherwise 0 or 1.
IsSubMesh is removed. In build mode, __Init takes the submesh type the
builder wrote and uses it instead of reading the static items.

Tests rewrite test data at test time with DfsBuilder: the 3D sigma-z file
with a 7 value custom block, and the legacy Spectral0D file with its
direction item in degrees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SetSubMeshType(None | 0 | 1), default None. None writes the custom block
as before (5 values for non-spectral types). 0 or 1 writes 6 values with
spec value 6 as the submesh type. Other values raise ValueError, and
spectral builder types raise, as SetNumberOfSigmaLayers does for 2D. The
mesh geometry is not checked.

CreateDfsu passes the builder's submesh type to DfsuFile.DfsuFileBuild,
so the returned DfsuFile reports the same SubMeshType as the file opened
again from disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DfsuBuilder appends the submesh type where it builds the 2001 custom
blocks, instead of replacing the last custom block. The spectral dfsu
file types are defined once, and SetSubMeshType declares that it takes
numpy integers. The build path in DfsuFile states that DfsuBuilder
writes 6 values only with a submesh type, and raises if that no longer
holds, so PR 76 must revisit it. The DfsuFile docstring describes the
legacy Spectral0D rule and the 4 value layout, and the CHANGELOG
describes what users can do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stray docstring after the return in OpenAppend made pyrefly infer that
it may return None. Move it into DfsuFileBuild, where it belongs. Pass
dfsFile to __IsLegacySpectral0D, and narrow Frequencies and Directions in
the legacy Spectral0D test. The warning count drops from 338 to 328, under
CI's ceiling of 330.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A legacy data type 2001 point spectrum with only frequencies or only
directions, such as MIKE IO's pt_freq_spectra.dfsu, read as Dfsu1D after
the legacy check started to require both axes. The check now accepts one
static item per present axis (spec value > 0), at least one axis, and
exactly 9 + number of present axes static items. The item count is
checked by number, not by the read position, so a 6-value file with both
spec values 0 is never the exception.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop references to PRs, issues and people, and keep only what the code
needs to explain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…kwards-compatibility

# Conflicts:
#	CHANGELOG.md
np.size rejected Frequencies and Directions, which may be None, under the
Python 3.11 numpy stubs CI uses, adding 3 pyrefly warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ryan-kipawa
ryan-kipawa merged commit d549b72 into master Nov 11, 2026
18 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New submesh feature (2027) makes all dfsu fail!

1 participant