Introduction to Apptainer

Lesson plan based around materials from CodeRefinery

Apptainer is the open source version of Singularity.

What are containers?

• Containers isolate software, dependencies, configurations, and system libraries from the host system
• The naming came from the idea of shipping containers (which are portable & standardized)
• Virtual machines are a similar concept, but these virtualise hardware, contain complete operating systems (including the kernel), and are managed by software known as a hypervisor
• Containers on the other hand share the host OS kernel, so they don’t contain complete operating systems, just user space system libraries
• This makes them lightweight, portable, and fast to start up

Docker vs Apptainer containers

• Apptainer (i.e. Singularity) is intended to run reproducibly across many system types, including HPC systems
• Docker is rarely allowed on clusters, requires root access
• Their images are somewhat different - Docker images are made up of layers (Base image -> launched as container -> edited -> used as new base layer -> launched -> edited -> etc.). Apptainer images squash layers into one file (.sif format)
• These files are easily shared and can be run on any system with Apptainer installed

Apptainer vs Singularity history & the .sif format

• Singularity was the original name of the open source project from 2015, but this turned partly commercial in 2018
• The open source part forked and joined the Linux foundation in 2021, becoming Apptainer
• The singularity image format (.sif) remains in Apptainer as a legacy of that
• Similarly when you install Apptainer, there are symlinks, so you can use singularity pull rather than apptainer pull
• For practical purposes, they are the same

Structure of an apptainer command

• apptainer [subcommand] [image] [additional commands]

• Example of pull and shell

apptainer pull docker://alpine
apptainer shell alpine_latest.sif
cat /etc/alpine-release
exit
cat /etc/alpine-release
cat /etc/debian_version

• Interact with the container from the host system:

apptainer exec alpine_latest.sif cat /etc/alpine-release
cat /etc/alpine-release

Building .sif from a definition file (.def)

• Building a custom container requires a .def file, specifying the registry and image for the base image, and various options for the container

example.def

Bootstrap: docker
From: debian:12.5-slim

%environment
        export PATH=$PATH:/root/.pixi/bin

%runscript
                cat /etc/debian_version

%post
        export PATH=$PATH:/root/.pixi/bin
        apt-get update && \
        apt-get install -y curl && \
        curl -fsSL https://pixi.sh/install.sh | bash && \
        apt-get clean && \
        pixi global install -c bioconda -c conda-forge  minigraph

• Then build the container: apptainer build minigraph.sif example.def

• apptainer run executes the runscript inside the container:

apptainer run minigraph.sif

Portability to HPC systems, & note on reproducibilty (.sif files vs rebuilding from a .def)

• .sif files are portable and will be highly reproducible

• You can also share a .def file, however consider that some tags on docker hub refer to rolling releases, e.g. latest. If you want future builds of the container to be identical, try to find a static tag, e.g. a github commit tag

• In addition, using commands from package managers like apt-get update in the .def will make the container less reproducible, as the package versions will change over time

• By default Apptainer/Singularity loads certain directories such as $HOME (see below), and this means local packages (e.g. Python, R, etc.) can be picked up and loaded instead of the ones in the container. To prevent this, do something like this.

Other useful things to know:

Mount binding

• Some folders are automatically bound from the host system (e.g., $HOME, $CWD, /tmp)
• Therefore don’t install software to those locations - they’ll install on your host system too
• Use an unmounted folder like /opt or /usr/local instead
• If you need to mount a directory to the container, e.g. a data directory, this is possible, e.g.:

apptainer exec --bind /scratch example.sif ls /scratch

Conversion from docker

• It’s possible to convert from docker images to singularity images - not covered today

Sandbox containers

• Singularity containers are basically uneditable - no so fun if you need to keep rebuilding from scatch during development
• There is a “sandbox container” feature which is editable, and it works like a file system within your file system
• When you are ready for a production container, you can convert the sandbox container to a regular container, though it would be preferable to rebuild from a definition file, so that there is a record of what was done

Tip

It’s advisable to use many small containers (minimal container for a single process/tool) rather than large inclusive containers.

Seqera containers resource

Converting from docker: if anyone uses Docker regularly - maybe they can take this one as an example?