Skip to content

Containers

Introduction

Containers are a independent runtime environment with it's own set of applications, dependencies, and libraries needed to run a specific application or operating system.

A popular container tool is Docker. However, Docker is not very secure or designed for use in an HPC environment. The WI-HPC cluster offers Apptainer (formally Singularity). If you are familiar with Docker and have it installed on your laptop, please see Working from a Dockerfile to convert a dockerfile to an Apptainer container.

Please see the wi-hpc-apptainer github repository for definition files and other examples.

Why Should I Use Apptainer?

  • Portable and publishable software enviornments
  • BYOE (Bring your own environment) approach
  • Able to install software without IT
  • Reproducibility of scientific results
  • Security and ability to verify

Libraries and Registries

Best Practices

  1. Do NOT use apptainer commands on the head nodes. When using the build, pull, shell, or exec commands, always request resouces.

  2. Publish your images on the Sylabs Cloud Library for ease of use and to save space in your lab share.

  3. Refer to the Apptainer documentation for best practices and examples. If you are still having issues please email the IT Help Desk at helpdesk@wistar.org

  4. Publish your images and/or definition files for ease of pulling.

Using Apptainer

When using Apptainer please ensure that you are either requesting resources or schedule a job.

Pulling Images

Some software is provided as an Apptainer image with the .sif or .simg file extension. More commonly however, a Docker image will be provided and this must be converted to an Apptainer image. For instance, if the installation directions are saying:

docker pull ubuntu:22.04

Then download and convert the Docker image to an Apptainer image with:

apptainer pull docker://ubuntu:22.04

This will produce the file ubuntu_22.04.sif in the current working directory, where 22.04 is a specific version of the software or a "tag".

Below is another example for the case where the image is on the Singularity Cloud Library:

apptainer pull library://sylabs/ubuntu

Unlike pull, build will convert the image to the latest Apptainer image format after downloading it.

Interacting with Images

Shell Command

Use the "shell" command to run a shell within the container

apptainer shell ./myimage.sif
Apptainer> python3 my_script.py

The apptainer shell command is very useful when you are trying to find certain files within the container.

Exec Command

To run a specific command that is defined within the container, use apptainer exec

apptainer exec myimage.sif python3 my_script.py

Run Command

To run the default command within Apptainer image use, for example:

apptainer run myimage.sif

This default run command references the %runscript section in the definition file. For example if the definition file had:

%runscript
    python3 my_script.py

It would run the same as the exec example from above.

Please see Building Images. Some containers might not have a default run command, see the documentation for the apptainer run command for more information.

Building Images

Apptainer images can be made from scratch using a definition file which is a text file that specifies the base image, the software to be installed and other information.

Create a definition file (e.g. myrecipe.def) and then run:

apptainer build myimage.sif myrecipe.def

For example definition files, please see the WI-HPC GitHub Apptainer folder.

Publishing Images

Publishing images to the Sylabs Library or Docker Hub (requires account) can be the best way to save your container, images, and definition files. As well as contributing to the scientific community so that anyone is able to re-create your research with the same exact environment as you did.

Once published, you can use the apptainer pull command to pull down your pre-build container/image. Interact with the container using the apptainer shell/run/exec/etc commands.

It is expected that the .sif file is deleted from your home folder or lab share once live on Sylabs Library or Docker Hub.

Working from a Dockerfile

Some software is provided as a Dockerfile instead of an actual container. In this case, if you have Docker installed on your local machine (e.g., laptop) then you can create the Docker image yourself and then transfer it to the WI-HPC cluster where the Apptainer image can be built.

On your local machine, after making the Docker image, get the image id by running this command:

docker images

Next, save that images as a tar file (say it was id 9c27e219663c):

docker save 9c27e219663c -o myimage.tar

Copy the myimage.tar to the WI-HPC Cluster by using scp and then create the Apptainer image. The commands might look as follows:

scp myimage.tar <username>@wi-hpc:/home/<username>
ssh <username>@wi-hpc
apptainer build myimage.sif docker-archive://myimage.tar

You may then follow the directions below for running myimage.sif which is a Apptainer container.