Managing software environments with

03-Mar-2025

The problem

Full reproducibility requires the possibility to recreate the system that was originally used to generate the results.

Conda is a package, dependency, and environment manager

  • package: any type of program (e.g. multiqc, snakemake etc.)

flowchart LR
    multiqc(multiqc)

Conda is a package, dependency, and environment manager

  • package: any type of program (e.g. multiqc, snakemake etc.)
  • dependency: other software required by a package

flowchart LR
    multiqc(multiqc) -.-> numpy(numpy)
    multiqc -.-> matplotlib(matplotlib)
    multiqc -.-> python(python)

Conda is a package, dependency, and environment manager

  • package: any type of program (e.g. multiqc, snakemake etc.)
  • dependency: other software required by a package
    • dependencies in turn have their own dependencies

flowchart LR
    multiqc(multiqc) -.-> numpy(numpy)
    multiqc -.-> matplotlib(matplotlib)
    multiqc -.-> python(python)
    matplotlib -.-> python
    matplotlib -.-> numpy
    matplotlib -.-> fonttools(fonttools)
    numpy -.-> python
    numpy -.-> libcxx(libcxx)

Conda is a package, dependency, and environment manager

  • package: any type of program (e.g. multiqc, snakemake etc.)
  • dependency: other software required by a package
    • dependencies in turn have their own dependencies
  • environment: a distinct collection of packages

flowchart LR
    subgraph environment
    style environment fill:#00000000, stroke-width:1px
    direction LR
    multiqc(multiqc) -.-> numpy(numpy)
    multiqc -.-> matplotlib(matplotlib)
    multiqc -.-> python(python)
    matplotlib -.-> python
    matplotlib -.-> numpy
    matplotlib -.-> fonttools(fonttools)
    numpy-.->python
    numpy -.-> libcxx(libcxx)
    end

Conda channels

Channels are remote directories containing packages

flowchart TD
    ch1[(channel1)] --- p1[package1]
    ch1[(channel1)] --- p2[package2]
    ch1[(channel1)] --- p3[package3]

    ch2[(channel2)] --- p4[package4]
    ch2[(channel2)] --- p5[package5]
    ch2[(channel2)] --- p6[package6]

Conda channels

Two common examples are:

  • bioconda (a channel specializing in bioinformatics software)
  • conda-forge (a community-led channel made up of thousands of contributors)

flowchart TD
    ch1[(bioconda)] --- p1[bowtie2]
    ch1[(bioconda)] --- p2[fastqc]
    ch1[(bioconda)] --- p3[snakemake]

    ch2[(conda-forge)] --- p4[numpy]
    ch2[(conda-forge)] --- p5[jupyter]
    ch2[(conda-forge)] --- p6[wget]

Conda channels

Two common examples are:

  • bioconda (a channel specializing in bioinformatics software)
  • conda-forge (a community-led channel made up of thousands of contributors)

flowchart TD
    ch1[(bioconda)] --- p1[bowtie2]
    ch1[(bioconda)] --- p2[fastqc]
    ch1[(bioconda)] --- p3[snakemake]

    ch2[(conda-forge)] --- p4[numpy]
    ch2[(conda-forge)] --- p5[jupyter]
    ch2[(conda-forge)] --- p6[wget]

    p5 -.-> l1([conda install -c conda-forge -c bioconda snakemake jupyter])
    p3 -.-> l1

Defining and sharing environments

Define a Conda environment in an environment.yml file:

environment.yml
channels:
  - conda-forge
  - bioconda
dependencies:
  - fastqc=0.11
  - sra-tools=2.8
  - snakemake=4.3.0
  - multiqc=1.3
  - bowtie2=2.3
  - samtools=1.6
  - htseq=0.9
  - graphviz=2.38.0

Conda command-line

  • Create a new environment:
conda env create -f environment.yml
  • Update an existing environment:
conda env update -f environment.yml
  • Update an existing environment:
conda env update -f environment.yml
  • Export environment (including all dependencies) to a file:
conda env export > environment.yml
  • Export historical environment (only packages explicitly installed):
conda env export --from-history > environment.yml

Conda, Anaconda, Miniconda, Miniforge…

  • Conda: The package manager itself, written in python
  • Anaconda:
    • The company behind Conda itself
    • An installer for Conda containing over 7,500 open-source packages
    • A cloud service where conda packages are hosted (anaconda.com)
  • Miniconda: A minimal installer for Conda, pre-configured to use the default channel
  • Miniforge: A minimal installer for Conda, pre-configured to use the conda-forge channel (use this!)

Questions?