Repository navigation
IsValidDataset: Confidence errors where its siblings only warn #560
Description
Description:
IsValidDataset checks four things about a dataset's source information and treats one of them differently. Missing source information, a missing author and a missing title are each a Warning. Confidence left at Undefined is an Error.
That reads as an accident of ordering rather than a decision. Confidence is the field a reader can least judge without knowing the data, and it is the only one of the four that blocks.
It is also why there is a backlog. Create.UnitTestDataSet says in its own summary that it produces a dataset "set up to be valid with .ci checks of unit tests", and its confidence parameter defaults to Confidence.Undefined, which is the one value that makes the check error. Fixtures generated the obvious way fail the check the method says they satisfy.
This is latent while dataset compliance is advisory. It stops being latent the moment that check gates a merge anywhere, because the first pull request touching a generated fixture then meets a blocking error on a field the tooling itself left unset.
Steps to reproduce:
Generate a dataset with Create.UnitTestDataSet without passing confidence, then run dataset compliance over the result. It reports an Error, "Dataset confidence level has not been correctly set".
Clear the author or the title on the same dataset instead. The same check reports a Warning and passes.
Expected behaviour:
One decision on the level, and either answer works. Either Confidence drops to Warning and joins its three siblings, or the other three rise to Error and the rule says plainly that a dataset must declare its provenance. What should not stay is one field of four blocking with no stated reason.
Separately, and whichever way that goes: UnitTestDataSet should not default to the value that fails its own stated contract. Default to a real level, require the caller to pass one, or stop claiming the output is valid against the .ci checks.
Fixing the default does not fix the fixtures already generated, which is the smaller half but is somebody's edit either way.
Test file(s):
None needed. Any dataset produced by Create.UnitTestDataSet without an explicit confidence reproduces it.
Metadata
Metadata
Assignees
Type
Fields
Projects
- Status
Activity
IsakNaslundBh commentedon Oct 21, 2026
Thanks @sakanni
Think you are right in that the confidence should not be defaulted, and method changed to force a non-undefined input for it to run.
With that said - the confidence level on the unit-test datasets are not as critical for the production code. They are not part of the shipped production code, and never seen by the end-consumer. The reason for enforcing that enum to be set to something other than "undefined" for datasets in general is for users of the tool to get a clear-tired understanding of the state of the datasets they are using in production scripts. Here it is critical, especially if a low confidence dataset is being used.
I cant remeber how the filtering of datasets on the bot check is working, but not sure the unit-test datasets are even checked. To some extent I do not think the unit-tests datasets need the same level checks (for the reason stated above).
Need to have another think about this though, but
TL/DR:
Don't think this check is critical for unit-test datasets. However, it is critical for distributed datasets.
sakanni commentedon Oct 25, 2026
The
Confidencerule has never failed a pull request. Checking every dataset-compliance run available across the repositories that require it, there is no failure on record, and there are runs that concluded success while reportingConfidence: Undefinedagainst files they had just read. So the rule has carried Error severity since 2021 without that severity ever having an effect, which makes this a question of setting a standard rather than changing one.On the decision itself, @IsakNaslundBh's answer is that
Confidencestays an error for production datasets and becomes a warning for unit-test datasets, with the field still expected on both. We have implemented that in CI rather than here, in BHoM/CI_Toolkit#40: the rule is right about the dataset either way, and what changes is a judgement about what a file is for, which is a convention about where CI puts things rather than somethingIsValidDatasetshould have to know. So there is no change proposed toTest_Toolkitfor this.The
Create.UnitTestDataSetdefault staysUndefinedand we are not proposing to change it either. Worth recording only because it explains the shape of what you will see: a fixture generated the obvious way carries that value, so the warnings will be common rather than rare, and they are a default rather than a step anyone skipped.That has merged, so I am closing this on that basis. Reopen it if you would rather the severity went the other way, or if the default is worth revisiting after all.