Skip to content

Using pixi in place of conda

Reference links:

pixi is a tool that the CQLS is recommending to use in place of conda for managing conda environments. In general, the solver of pixi is faster, and it is much easier to configure and use in an automatic fashion.

Tip

If you haven't yet, get your configuration set up to use pixi without any fuss.

See here regarding the 'Set PIXI_CACHE_DIR to continue.' message.

Quick start

The main difference you will encounter is regarding where the environments are stored when you are using pixi. With conda, environments are all (generally) stored in the same location, and they are activated using the conda activate <env> syntax. With pixi, the environments are stored individually, generally near where the data are processed.

Note

You can install tools globally as well with pixi. Tools installed with pixi global install need no environment activation to use.

To install a tool locally for a certain task, you can follow this pattern:

  1. Check out a node with salloc
  2. Navigate to the directory you would like to install the pixi env in.
  3. pixi init . (1)
  4. pixi add <package-0> <package-1> <package-2> (2)

Tip

You can search for packages with pixi search, or use the streamlit metadata app.

  1. pixi shell (3)
  2. Run the commands, or submit the commands to the queuing system using hqsub (4)
  1. This makes the current directory a pixi env. You can also specify a directory name to start the pixi env in a different dir, then cd to that dir.
  2. Replace <package-N> with the conda packages from conda-forge or bioconda
  3. Activates the pixi env for your shell
  4. Submit the commands while in the pixi shell and your job on the compute node will also have access to the software.

You'll have to activate the environment using pixi shell (or by using the shell-hook, see below) each time you want to run the software.

Adding packages

When you pixi add packages to your environment, the lockfile will automatically update and the software will be installed. To allow adding of multiple pacakges without install, you can add the --no-install flag to your --add command. In this way, you can pixi add --no-install pacakge1 and then later pixi add --no-install package2 before you finally run pixi install. This can be useful if you want to get all of the packages added to the environment, but you aren't quite sure the proper versions or dependencies you need before you start.

Specifying version numbers

You can specify versions or builds when adding packages to the environment. As an example, if you wanted to add a specific version of R to your environment, you could pixi add r-base=4.1.3 or pixi add 'r-base 4.1.3'. If you know there is a specific build you need (this can be relevant when using GPUs), you can do pixi add 'package version build', where version or build can be delared with wildcards and/or less-than or greater-than.

There are other options you can specify including channels, which can be important if a package happens to be in both bioconda and conda-forge.

Tip

Specifying a package with matchspec may be easier to do manually, by e.g. editing the pixi.toml file and adding package = { version = "VersionNumber", channel = "ChannelName", build = "BuildName" }

Sharing environments

One limitation of using pixi compared to conda, is that sharing environments is a little less straightforward. You cannot use someone else's pixi environment by running pixi shell, but it is relatively simple to copy the pixi.toml and pixi.lock files into your own directory, and run pixi install. One of the benefits of the lockfile is that replicating environments is much easier compared to conda, where even when all version numbers are specified, the solver still needs to be run.

Generating a shell hook for activation

One way to share an environment between users is to generate a shell hook using pixi shell-hook > activate.sh. Then, anyone can run source ./activate.sh to activate the environment and run commands from the environment.

Note: There is no way to manually deactivate the environment once activated. Therefore, exiting your shell is the only way to return to a clean shell.

Tip

When using the queuing system and running a shell script with your commands, you can add a line with the source activate.sh above your analysis commands. In this way, you don't have to worry about de-activation.

Using direnv to automatically activate your pixi envs

We have installed direnv to help facilitate pixi env activation.

The pixi devs have instructions for using direnv.

First, you need to enable direnv:

bash
echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
tcsh
echo 'eval `direnv hook tcsh`' >> ~/.cshrc

Then, in the pixi env directory:

.envrc
echo 'watch_file pixi.lock' > .envrc
echo 'eval "$(pixi shell-hook)"' >> .envrc
direnv allow

Then, when you enter the directory, your shell will automatically update with the pixi env.

Warning

If you use direnv in this way, you must submit jobs from the pixi env directory, or your shell will deactivate the env when leaving the directory.

Using direnv to manage your pixi env activation works best if you use the base of your project directory as a pixi env. For reproducible research, you can also manage the directory using git to make it easy to share your pixi.toml and pixi.lock files.

Future directions

The CQLS will document other methods of using pixi for reproducible environments, especially for reproducible research and GPGPU computing in the near future.