Skip to content

ci-dataset-compliance: neither a crash nor a skip reports what it did #32

Closed

Description

@sakanni

ci-dataset-compliance can finish without saying what it examined, in two different ways.

A file the engine cannot open takes the process down with it. IsValidDataset opens the file before it parses anything, and that open is unguarded, so the exception escapes and the runner dies before it writes any output. The check goes red with no annotation, no coverage line and no name for the file that caused it, and the findings it had already gathered for every other file in the run are lost. ComplianceRunner had the same defect and #28 fixed it. The dataset runner was not changed.

A skip reports nothing either, and that one is not a bug. The check reads only the dataset files a pull request changed, so skipping a pull request that changed none is correct. But the job's only output in that case is a log notice. Nothing reaches the job summary, so a green that had nothing to check and a green that checked everything look the same there, and a correct skip cannot be told from a pathspec that missed a file.

Impact

No observed instance of the crash. It needs a file that exists but cannot be read, which on a runner means a lock or a permissions problem rather than a bad dataset. Skips are the normal case: most pull requests change no datasets at all.

What's needed

  • A per-file catch in DatasetComplianceRunner, as fix(ci-compliance): survive a file the engine throws on #28 did for ComplianceRunner: count the file as failed, annotate it by name, carry on, and fail the run on the failure.
  • A job summary line when a compliance job skips, naming the pathspec it searched.

Evidence

Reproduced by holding a dataset file open with no sharing and running the dataset runner over it and one other file. The process exits on an unhandled IOException with empty output: no coverage line, and the second file's genuine finding never reported. The same two files with nothing locked produce both.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions