Give Pixi A Chance

(Apologies for no Thursday Drop, again. I got stuck in Kubernetes limbo, which — oddly enough — is a redundant term.)

Ev’rybody’s talking about uv, venv, poetry, and renv. Nixs and condas, and Cargos, and spacks. And, aye yai yai (with apologies to Mr. Lennon, RIP).

These days, things move way too fast, and that pace of change is a huge detriment to software development, reproducible data analyses, data engineering pipelines, backward compatibility, and, more often than not, just getting stuff done. In other words, “dependency hades” — along with configuration management — is quite a real phenomenon.

The non-exhaustive list of both ecosystem specific, and “generic” package/environment managers in the highly lyrically suspect opening sentence all attempt to help with this situation, and so does Pixi, the subject of this week’s WPE.

What Is Pixi?

Pixi is a Rust-based, cross-platform, multi-language package manager and workflow tool that’s built on the, er…solid? foundation of the Conda ecosystem. It supports multiple languages, including Python, C++, and R, using Conda packages (it also supports many other programming languages). This means you can manage packages for multiple ecosystems with the same tool, on a per-project basis. It works across Linux, Windows, and macOS, so your team can all take advantage of the same moving parts. And, it can work at both the project and system level.

Getting Started

They ask you to do the curl|bash dance (the script was safe as of the time of the Drop) to get set up, but there are Homebrew, native installer, and direct Cargo installation options as well. To see if it got added properly, you can do:

$ pixi info
      Pixi version: 0.14.0
          Platform: osx-arm64
  Virtual packages: __unix=0=0
                  : __osx=14.4=0
                  : __archspec=1=arm64
         Cache dir: /Users/hrbrmstr/Caches/rattler/cache
      Auth storage: /Users/hrbrmstr/.rattler/credentials.json

That “rattler” path refers to the Rust library and set of tools for the conda ecosystem.

I recommend running pixi --help (or tap here) just to see what’s available, and also recommend following their instructions for installing shell completions.

Just to get familiar with how it works, go ahead and do:

$ pixi init sup
✔ Initialized project in /Users/hrbrmstr/projects/p/sup

That creates the sup directory and adds a pixi.lock file, which is used to record the exact versions of dependencies that have been resolved and installed in a project. It also adds a pixi.toml:

[project]
name = "sup"
version = "0.1.0"
description = "Add a short description here"
authors = ["hrbrmstr <bob@rud.is>"]
channels = ["conda-forge"]
platforms = ["osx-arm64"]

[tasks]

[dependencies]

We use pixi add to add tools/dependencies to the project. These can be versioned with the familiar =##.## postfix (e.g., python=3.9).

$ pixi add R
✔ Added R

That fetched R for this project and updated our dependencies:

[dependencies]
R = ">=4.3,<5"

(You will want to run through the online docs for the full toml config spec for pixi).

It also created a .pixi directory:

$ tree .pixi -L 3
.pixi
└── envs
    └── default
        ├── bin
        ├── cmake
        ├── conda-meta
        ├── etc
        ├── fonts
        ├── include
        ├── lib
        ├── libexec
        ├── man
        ├── prefix
        ├── sbin
        ├── share
        ├── ssl
        └── var

There’s quite a bit of bits in there:

$ du -sh sup
738M    sup

so, make sure you’ve got plenty of storage space.

We can make sure it did what it said:

$ pixi run R --version --quiet
R version 4.3.2 (2023-10-31) -- "Eye Holes"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: aarch64-apple-darwin20.0.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
https://www.gnu.org/licenses/.

Pixi activated the local environment, then ran R in that context. You can do pixi shell to run things in the content of the environment without the need to prefix the commands with pixi run.

T[a]sk, t[a]sk

Most projects have steps that need to be run. Pixi calls these “tasks“, and they can be standalone items or depend on other tasks. Let’s add a task that installs some R packages.

[tasks]
# install necessary R packages
setup_r = "Rscript -e 'install.packages(c(\"jsonlite\", \"httr\"))'"

If you run pixi run setup_r, you’ll see it perform said operation. NOTE: For now, you should consider using one of R’s environment managers on top of pixi if you need to pin items to a specific version.

Before adding more tasks, let’s add some more pixi-level dependencies:

$ pixi add nodejs=20 r-recommended curl
✔ Added nodejs=20
✔ Added r-recommended
✔ Added curl

We’ll use this setup to make a reproducible Observable Framework dashboard. The code is on Codeberg, and there you’ll see we’ve added:

.
├── docs
│   ├── data
│   │   └── mtcars.json.R
│   └── index.md
├── observablehq.config.ts
└── package.json

into the mix. The R file will be run by Observable’s Framework on dev/build and use our pixi’d R to generate the mtcars.json file. We’ll also add some helper tasks to the config:

[project]
name = "sup"
version = "0.1.0"
description = "A reproducible Obserable Framework project"
authors = ["hrbrmstr <bob@rud.is>"]
channels = ["conda-forge"]
platforms = ["osx-arm64"]

[tasks]
# install necessary R packages
setup_r = "Rscript -e 'install.packages(c(\"jsonlite\", \"httr\"))'"

# setup node project
setup_node = "npm install"

# run in dev mode
dev = "npm run dev"

# run in dev mode
build = "npm run build"

[dependencies]
R = ">=4.3,<5"
r-recommended = ">=4.3,<5"
nodejs = "20.*"
curl = ">=8.5.0,<8.6"

If you execute pixi run dev, it will activate the local environment and fire up the server and browser. Similarly, pixi run build will build the “dashboard”.

Your Turn

If you want to build on this foundation, consider adding Python into the mix and using it plus curl to add more data loaders to the Framework project. Also, see what happens when you make the dev or build tasks dependent on other tasks. While you’re at it, make the index.md far less lame.

Otherwise, try converting one of your existing projects to a pixi-managed one.

The config also supports designating a README, so consider making a good one for either path you take.

Remember, this example project is on Codeberg.

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev ☮️