Fix incompatability issue for moorings products

[mphemming]

This PR has been closed. Refer to #189 instead.

V2 branch has been merged into this one - see comments indicating which code can be ignored as changes are only due to this.

Description

netCDF4.stringtochar raises an AttributeError when converting fixed-width byte strings when using netcdf4==1.7.4 (pinned in dataflow-orchestration). This broke mooring product generation and required a monkeypatch.

I investigated whether we could downgrade netcdf4 in dataflow-orchestration but constraining versions is not ideal there.

Instead we can use a helper function for numpy byte-to-character conversion, covering source_file and instrument_id in the products. This removes the need for a monkeypatch and doesn't effect ncwriter used by ingestion and moorings NRT pipelines.

I tested update_mooring_products_flow.py locally while importing this branch aodntools and removing the monkeypatch and is working.

[mphemming]

I had to merge in the v2 branch, resulting in some of the changes below.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[mphemming]

Ignore changes here. Due to merge of V2 branch

[mphemming]

Ignore changes here. Due to merge of V2 branch

[mphemming]

Changes above due to merge of V2 branch only

[mphemming]

Ignore changes here. Due to merge of V2 branch

[mphemming]

Ignore changes here. Due to merge of V2 branch

[mphemming]

Ignore changes here. Due to merge of V2 branch

[mphemming]

Ignore changes here. Due to merge of V2 branch

[mhidas]

V2 branch has been merged into this one - see comments indicating which code can be ignored as changes are only due to this.

As with all recent PR's in this repository (e.g. #180, #180), please

• Start your new dev branch from the v2 branch, not master; and
• Create your PR to merge the changes back into v2, not master.

Basically, think of v2 as the new main branch. For now we're keeping master because the legacy pipelines still use it. Once that's all been decommissioned, we can do some branch renaming to make this more obvious.

[mhidas]

netCDF4.stringtochar raises an AttributeError when converting fixed-width byte strings when using netcdf4==1.7.4

See Unidata/netcdf4-python#1464
If it's the same issue, it looks like it's been fixed, so that should come through in the next version (who knows when?).
In the meantime, it looks like specifying stringtochar(..., encoding='ascii') could be the simplest solution for us.

[mphemming]

, encoding='ascii'

Thanks Marty, I tried this and tested locally. Seems to be working.

I created a new branch based on the v2 branch: #189
It got confusing in this branch after restoring back to master. So to be safe it will be better to review this new branch instead. I will close this PR.