Skip to content

Make AWS full tests use verified S3 inputs - #105

Open
edmundmiller wants to merge 1 commit into
devfrom
fix/aws-full-tests-20261112
Open

edmundmiller wants to merge 1 commit into
devfrom
fix/aws-full-tests-20261112

Conversation

@edmundmiller

Copy link
Copy Markdown

ENA staging failures have been breaking release tests. The latest AWS run hit a connection reset on Nextflow 26.04.7, which does not retry that error (nextflow#7798). An earlier run staged a full-sized but corrupted FASTQ copy.

AWS full test
  → Nextflow 26.10.5 / strict parser
  → verified private S3 samplesheet
  → final pipeline result determines GitHub status

The existing launch action cannot select the remote Nextflow version, so this migrates to the pinned action-seqera-launch revision and retains a launch-result artifact. The generic test_full profile and public ENA samplesheet are unchanged; only the nf-core AWS launch uses the mirror.

The mirror is live in eu-west-1: 12 original FASTQs, 60.67 GB, 482,526,530 matching read pairs. Complete ENA MD5, gzip, and paired-FASTQ validation passed before server-side copying; destination full-object checksums match. Contents are restricted to the nf-core AWS account. Existing cached originals were not changed or deleted.

PR checklist

  • Describe the change and reason.
  • Changed-workflow hooks and actionlint pass.
  • nextflow lint . on 26.10.5 passes: no errors, existing deprecation warnings.
  • nf-core pipelines lint — not run locally.
  • Full pipeline test — not rerun; the data-validation job is not a pipeline smoke test.

Related data source: test-datasets#1972.

Pin Nextflow 26.10.5 so connection-reset staging failures can retry.
Use the checksum-validated private mirror for AWS tests while leaving
public ENA samplesheets unchanged for other users.

Migrate to the supported Seqera launch action to select the remote
runtime explicitly and wait for the final workflow result. Preserve a
launch-result artifact on success or failure.
@github-actions

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 665a138

+| ✅ 243 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   2 tests had warnings |!
Details

❗ Test warnings:

  • nextflow_config - Config manifest.version should end in dev: 1.0.0
  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).

❔ Tests ignored:

  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • files_unchanged - File ignored due to lint config: assets/nf-core-deepmutscan_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-deepmutscan_logo_dark.png

✅ Tests passed:

Run details

  • nf-core/tools version 4.2.2
  • Run at 2026-11-12 21:51:02

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant