nf-test: A Nextflow testing framework

Introduction

nf-test is a testing framework built specifically for Nextflow — it has no use outside that context.

nf-core (the community’s collection of standardised Nextflow pipelines and modules) originally tested pipelines, only with success status. Some went further and followed the modules CI at the time. with pytest. Every assertion needed its expected value written into the test by hand, including MD5 checksums copied in from a real run. Any incidental change to the output meant editing checksums by hand across many tests. nf-test replaces this with snapshot testing: run the test once, capture everything nf-test can capture about the output, and compare against that snapshot on every future run. A failing test then means “the output changed” rather than “my hardcoded assertion mismatches” — you decide whether the change was expected, and either fix the code or accept the new snapshot.

Installing and scaffolding tests

Install nf-test the same way you install Nextflow itself, with a single curl command:

curl -fsSL https://get.nf-test.com | bash

Run nf-test init inside a Nextflow project to set it up. It creates nf-test.config and a tests/ directory.

nf-test init

nf-test generate scaffolds a test skeleton for a process, workflow, pipeline, or function, given the Nextflow file to test:

nf-test generate process modules/local/my_process.nf

The generated skeleton isn’t a working test — it’s a starting point with the structure filled in and placeholders for the parts only you know: which inputs to give the process and what to assert about its output.

Four levels of testing

nf-test tests four kinds of Nextflow code, matching how a pipeline is built up from smaller pieces:

• Function — a single Groovy (the language underlying Nextflow) or Nextflow function, tested in isolation.
• Process — a single module.
• Workflow — a subworkflow chaining several modules together.
• Pipeline — the whole thing, end to end.

Every test follows the same three-part shape, regardless of level:

• an optional setup step, which prepares whatever the test needs — often by running an upstream process to produce real input data;
• a when step, which declares the specific inputs (and, for a pipeline test, parameters) to run with;
• a then step, which asserts the run succeeded (or failed, on purpose) and optionally snapshots the output.

Extending nf-test

Beyond nf-test’s own assertions, a growing set of plugins add content-aware comparisons for specific file formats: nft-bam for BAM alignments, nft-vcf for variant calls, nft-tiff for imaging data, and nft-utils — maintained by the nf-core community — which adds sanitizeOutput() to strip non-reproducible content (timestamps, absolute paths, channel index numbers) out of a snapshot automatically, instead of hand-picking an accessor per output. You can also write your own comparison functions in Groovy, alongside a pipeline’s other library code.

nf-core’s own tooling talks to nf-test directly, too: nf-core modules test/nf-core subworkflows test generate and run a component’s nf-test snapshot, and nf-core pipelines lint checks the content of a pipeline’s nf-test files (for example, that versions.yml is included in every snapshot) — nf-test isn’t a separate, disconnected test runner from the nf-core toolchain.

A worked example

nf-core/modules tests every module this way, with one test folder per module. Test inputs come from a small shared test-data repository, referenced via params.modules_testdata_base_path, rather than files committed alongside the test itself. Take BWAMEM2_INDEX, which builds a BWA-MEM2 alignment index from a reference FASTA:

modules/nf-core/bwamem2/index/tests/main.nf.test

nextflow_process {

    name "Test Process BWAMEM2_INDEX"
    script "../main.nf"
    process "BWAMEM2_INDEX"

    test("fasta") {
        when {
            process {
                """
                input[0] = [
                    [id: 'test'],
                    file(params.modules_testdata_base_path + 'genomics/sarscov2/genome/genome.fasta', checkIfExists: true)
                ]
                """
            }
        }
        then {
            assertAll(
                { assert process.success },
                { assert snapshot(process.out).match() }
            )
        }
    }
}

There’s no setup step: the process only needs a reference file, given directly in when as input[0] — nf-test addresses a process’s inputs positionally, by index, in the order the process declares them. then asserts success and snapshots the entire process.out — every output channel, including versions. Running it once produces a snapshot like this (trimmed to the interesting fields):

modules/nf-core/bwamem2/index/tests/main.nf.test.snap

{
  "index": [
    [
      { "id": "test" },
      [
        "genome.fasta.0123:md5,b02870de80106104abcb03cd9463e7d8",
        "genome.fasta.amb:md5,3a68b8b2287e07dd3f5f95f4344ba76e",
        "genome.fasta.ann:md5,c32e11f6c859f166c7525a9c1d583567",
        "genome.fasta.bwt.2bit.64:md5,d097a1b82dee375d41a1ea69895a9216",
        "genome.fasta.pac:md5,983e3d2cd6f36e2546e6d25a0da78d66"
      ]
    ]
  ],
  "versions_bwamem2": [ [ "BWAMEM2_INDEX", "bwamem2", "2.2.1" ] ]
}

That snapshot is committed to git alongside the test. Every future run diffs against it.

BWAMEM2_MEM (the actual alignment step) needs an index to align against, so its test builds one first with a setup block, rather than committing a separate hand-built fixture index to the repo:

modules/nf-core/bwamem2/mem/tests/main.nf.test

nextflow_process {

    name "Test Process BWAMEM2_MEM"
    script "../main.nf"
    process "BWAMEM2_MEM"

    setup {
        run("BWAMEM2_INDEX") {
            script "../../index/main.nf"
            process {
                """
                input[0] = Channel.of([
                    [:],
                    [file(params.modules_testdata_base_path + 'genomics/sarscov2/genome/genome.fasta', checkIfExists: true)]
                ])
                """
            }
        }
    }

    test("sarscov2 - fastq, index, fasta, false") {
        when {
            process {
                """
                input[0] = Channel.of([
                    [ id:'test', single_end:true ],
                    [file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/fastq/test_1.fastq.gz', checkIfExists: true)]
                ])
                input[1] = BWAMEM2_INDEX.out.index
                input[2] = Channel.of([[:], [file(params.modules_testdata_base_path + 'genomics/sarscov2/genome/genome.fasta', checkIfExists: true)]])
                input[3] = false
                """
            }
        }
        then {
            assertAll(
                { assert process.success },
                { assert snapshot(
                    bam(process.out.bam[0][1]).getHeaderMD5(),
                    bam(process.out.bam[0][1]).getReadsMD5(),
                    process.out.findAll { key, val -> key.startsWith("versions") }
                ).match() }
            )
        }
    }
}

BWAMEM2_INDEX.out.index in the when block refers straight back to the setup run — real index output, not a stand-in. The snapshot here is narrower than BWAMEM2_INDEX’s: a BAM file’s own MD5 isn’t reproducible on its own (its header embeds a timestamp), so the test uses nft-bam’s getHeaderMD5() and getReadsMD5() to hash the header and the aligned reads separately, skipping the part that legitimately varies run to run.

That narrower snapshot has a real cost: naming bam[0][1] twice by hand means any other output channel — say a log file, or one a future version of the module adds — isn’t captured at all, and the snapshot no longer shows which file each hash came from. sanitizeOutput() (from nft-utils) is the fix nf-core has converged on, and it’s worth understanding on its own: it snapshots the entire process.out, like the INDEX test did, but sanitizes only the specific channels you name instead of leaving the whole snapshot exposed to non-reproducible noise. SNAPALIGNER_ALIGN’s test — another aligner module, same setup-then-align shape — shows the pattern:

modules/nf-core/snapaligner/align/tests/main.nf.test

then {
    assert process.success
    assertAll(
        { assert snapshot(sanitizeOutput(process.out, readsMD5Keys: ["bam"], unstableKeys: ["bai"])).match() }
    )
}

readsMD5Keys: ["bam"] hashes just the aligned reads in the bam channel, the same header problem as above. unstableKeys: ["bai"] keeps the BAM index’s filename in the snapshot but drops its checksum, since a BAM index genuinely isn’t reproducible byte-for-byte. Every other channel is left untouched and snapshotted as-is — nothing needs to be enumerated by hand, so a newly added output channel shows up in the next diff instead of silently going untested. This is the version of the pattern to reach for by default; the direct nft-bam accessors in the MEM test above are for the narrower case where you don’t want the rest of process.out in the snapshot at all.

Testing failure paths, too

A test isn’t limited to the success path. process.failed asserts a process failed on purpose, and its captured standard output and error can be snapshotted the same way as any other output — so a specific, expected error message is covered by regression testing, not just the happy path.

NotePipeline-scale example: nf-core/sarek’s annotation tests

nf-core/sarek tests its whole pipeline this way, not just individual modules. Rather than writing each nextflow_pipeline test by hand, Sarek’s tests describe each case as a plain data map — parameters, and whether it’s expected to fail — and a shared helper (tests/lib/UTILS.groovy) turns each one into a full test:

tests/annotation_vep.nf.test

def test_scenario = [
    [
        name: "Fails with profile test --dbnsfp and no dbnsfp_tbi",
        params: [ /* ... */ dbnsfp: '...', tools: 'vep' ],
        failure: true,
        snapshot: 'stdout',
        snapshot_include: 'dbnsfp inconsistency',
    ],
    [
        name: "-profile test --tools vep --vep_loftee",
        params: [ /* ... */ tools: 'vep', vep_loftee: true ],
        vcf_header_check: '|LoF|',
    ],
]

test_scenario.each { scenario -> test(scenario.name, UTILS.getTest(scenario)) }

The first scenario is exactly the failure-path pattern above in practice: it expects the pipeline to fail, and snapshots only the lines of standard output containing dbnsfp inconsistency — the specific warning this test exists to catch. The second doesn’t test failure at all; it asserts a produced VCF’s header contains |LoF|, confirming a particular annotation plugin actually ran. Once UTILS.getTest() exists, adding a new pipeline test case is just adding another map to the list — it stops looking like nf-test code and starts looking like plain configuration.

Snapshot everything you can

The instinct when writing a test is to assert one specific, meaningful property — “the output has 3 lines” — rather than snapshot the whole thing. nf-test’s snapshot mechanism argues for the opposite: snapshot everything you reasonably can, and let the diff on a future run tell you whether a change is a regression or expected. A specific, hand-picked assertion needs rewriting by hand every time that value legitimately changes; a snapshot just needs nf-test test --update-snapshot, a diff review, and a commit once you’re satisfied it’s correct.

NotePushing the idea further: snapshotting a release’s “mega test”

Every nf-core pipeline release runs a full-scale test on AWS against real-sized data — nf-core calls this the mega test. Its output already lands in an S3 bucket; nothing about producing it involves nf-test at all. That doesn’t stop nf-test from checking it: an nf-test case that runs no pipeline logic itself can point at the release’s S3 path instead, list every file the mega test produced, pull out tool versions from specific files, and snapshot that listing. The test doesn’t generate anything — it just turns a result someone would otherwise eyeball once into a snapshot that’s diffed on every future release. It’s the same “snapshot everything, let the diff decide” idea from above, stretched to output the test itself never ran.

Key takeaways

• The four test scopes — function, process, workflow, pipeline — mirror how a Nextflow pipeline is actually built: small pieces composed into bigger ones.
• Every test follows the same three stages: an optional setup, a when that declares the inputs, and a then that asserts and snapshots.
• The plugin system adds content-aware comparisons — BAM, VCF, TIFF, and more — instead of leaving every module to hand-roll its own.
• setup chains real upstream output into a test instead of committing hand-built fixtures.

Further resources

• nf-test.com — documentation, with worked examples for every assertion type.
• plugins.nf-test.com — the plugin catalogue.
• #nf-test and #nft-plugins on the nf-core Slack.
• nf-core/modules — hundreds of real module tests to read for more examples.

Community pipelines with pipeline-level nf-test suites worth reading, beyond Sarek above:

• nf-core/demultiplex
• nf-core/seqinspector
• nf-core/rnavar

Tip

Writing assertions and snapshot content by hand is exactly the kind of task current AI coding assistants are good at — worth trying alongside the documentation’s own examples when starting a new test.