Run, systemd-run!; sysbench; sysfs

Today, we have a Drop in three acts involving running, measuring, and squinting.

We start with an alternate way to fire off long running jobs, sail through a visit to a trusty benchmarking tool, and slide into an area of your [linux] file system you likely rarely explore.

Run, systemd-run!

I grok that systemd is the cilantro of linux system and service managers, so if it’s not your cuppa, please make your way to another section.

Personally, I appreciate the expressiveness (which does add a bit of, at times, painful verbosity) of it, and other aspects, such as how it helps to sequence and manage dependencies of scheduled processes.

Similarly, there also strong feels regarding screen vs. tmux (et al.), especially in one use case where you choose to be in a mux’d session to fire off a potentially long running command.

Well, let’s add some 🌿 to that mux’d 🌯 with a nod to systemd-run.

This utility takes a command and runs it under systemd instead of your shell. You get a transient unit (a temporary service or scope) that systemd then tracks like any other. It has two running modes. In “service” mode (the default), it runs detached in a clean environment, parented by systemd. systemd-run returns once it starts, so add --wait to block until it finishes. In “scope” mode, the task runs synchronously, inherits your environment, and returns only when the command exits.

These are some of the more common flags you’ll find yourself using (if you adopt the utility):

• -u NAME — name the unit yourself instead of a generated name.
• -p KEY=VALUE — set a unit property (e.g. CPUQuota=50%).
• --uid= --gid= — run as a different user/group.
• --working-directory= / -d — set the working dir (-d uses the caller’s).
• -E NAME=VALUE — set an environment variable.
• -t / --pty — attach a pseudo-TTY for interactive programs.
• -P / --pipe — inherit stdin/stdout/stderr for shell pipelines.
• --user — talk to your user manager instead of system (highly recommended to always do this).

An example is warranted:

# background the benchmarker (prbly not the bestest example)
$ systemd-run --user --unit=sys-benchmark sysbench cpu --threads=4 --time=600 run
Running as unit: sys-benchmark.service; invocation ID: 83173d3b081344c7bcbaba3e60977c6d
# check on it
$ journalctl --user --unit sys-benchmark.service
Nov 14 09:48:28 medina systemd[1823]: Started sys-benchmark.service - /usr/bin/sysbench cp>
Nov 14 09:48:28 medina sysbench[3819130]: sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)
Nov 14 09:48:28 medina sysbench[3819130]: Running the test with following options:
Nov 14 09:48:28 medina sysbench[3819130]: Number of threads: 4
Nov 14 09:48:28 medina sysbench[3819130]: Initializing random number generator from curren>
Nov 14 09:48:28 medina sysbench[3819130]: Prime numbers limit: 10000
Nov 14 09:48:28 medina sysbench[3819130]: Initializing worker threads...
Nov 14 09:48:28 medina sysbench[3819130]: Threads started!
# check on it a different way
$ systemctl --user status sys-benchmark.service
● sys-benchmark.service - /usr/bin/sysbench cpu --threads=4 --time=600 run
     Loaded: loaded (/run/user/1000/systemd/transient/sys-benchmark.service; transient)
  Transient: yes
     Active: active (running) since Sat 2026-11-14 09:48:28 EST; 23s ago
   Main PID: 3819130 (sysbench)
      Tasks: 5 (limit: 73238)
     Memory: 2.3M (peak: 2.9M)
        CPU: 1min 32.965s
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/sys-benchmark.service
             └─3819130 /usr/bin/sysbench cpu --threads=4 --time=600 run
# stop it
$ systemctl --user stop sys-benchmark.service

I did let that finish, and this is the result:

Nov 14 09:58:28 medina sysbench[3819130]: CPU speed:
Nov 14 09:58:28 medina sysbench[3819130]:     events per second:  7852.89
Nov 14 09:58:28 medina sysbench[3819130]: General statistics:
Nov 14 09:58:28 medina sysbench[3819130]:     total time:                          600.0005s
Nov 14 09:58:28 medina sysbench[3819130]:     total number of events:              4711745
Nov 14 09:58:28 medina sysbench[3819130]: Latency (ms):
Nov 14 09:58:28 medina sysbench[3819130]:          min:                                    0.46
Nov 14 09:58:28 medina sysbench[3819130]:          avg:                                    0.51
Nov 14 09:58:28 medina sysbench[3819130]:          max:                                    1.73
Nov 14 09:58:28 medina sysbench[3819130]:          95th percentile:                        0.57
Nov 14 09:58:28 medina sysbench[3819130]:          sum:                              2399010.61
Nov 14 09:58:28 medina sysbench[3819130]: Threads fairness:
Nov 14 09:58:28 medina sysbench[3819130]:     events (avg/stddev):           1177936.2500/170.69
Nov 14 09:58:28 medina sysbench[3819130]:     execution time (avg/stddev):   599.7527/0.01
Nov 14 09:58:28 medina systemd[1823]: sys-benchmark.service: Consumed 39min 59.693s CPU time, 2.9M memory peak, 0B memory swap peak.

That accounting and logging are big reasons I’m a fan of using this for disconnected, long[ish]-running tasks.

Here are some more use cases:

# Build a kernel/project; depends on repo size
systemd-run --user --unit=build make -j"$(nproc)"
# Disk I/O benchmark; creates a test file
systemd-run --user --unit=fio-test fio --name=test --rw=readwrite --bs=1M --size=1G --runtime=300 --time_based --filename=fio.test
# Download a large file
systemd-run --user --unit=download curl -L -o large.iso https://example.com/large.iso
# Mirror/sync a tree
systemd-run --user --unit=sync-tree rsync -avP ./src/ ./backup/
# Compress a big file
systemd-run --user --unit=compress zstd -T0 -3 bigfile.bin
# Watch my own user journal indefinitely
systemd-run --user --unit=journal-follow journalctl --user -f
# Start a web app/server
systemd-run --user --unit=webapp python3 app.py

For those still seeking a “so what?”, I’ll argue that the real value-add of systemd-run is “ownership,” and — perhaps — even “persistence”. A tmux session keeps a terminal alive so you can reattach later, but the process still lives in your login session’s cgroup, and with KillUserProcesses=yes as the default, that session’s processes die on logout. systemd-run with --scope --user moves the process under user@.service, outside the session scope, so it survives logout — and with loginctl enable-linger, it starts at boot and survives logout with no login at all.

Beyond persistence, systemd-run makes the process a bona fide managed unit. You get resource control through -p flags like IOWeight=10 or CPU and memory limits. You get scheduling through --on-calendar= or --on-active=30, which create timer units without cron. You get remote and container targets through -H and -M. And you get observability: the unit shows in systemctl list-units and its output lands in journalctl -u.

So use tmux/screen for interactive shells you’ll come back to, and lean into systemd-run for when you want systemd to own the process — with those cgroups, logging, scheduling, and lifecycle control attached.

sysbench

Since I used sysbench in the first section, we should talk about it a bit more (but I’ll try to be brief).

This utility is a modular, cross-platform, multi-threaded benchmark tool primarily for (legacy?) database systems. Its stated purpose is to evaluate OS parameters that matter for a system running a database under intensive load, and the design goal is to get a quick impression of system performance without setting up complex database benchmarks or even installing a database at all.

sysbench spins up a specified number of threads, and they all execute requests in parallel. The actual workload depends on the test mode. You limit the run by total number of requests, total time, or both.

It ships with two database drivers (mysql, pgsql) and five compiled-in tests (fileio, cpu, memory, threads, mutex). Each test mode has its own options, and the suite was designed so adding new test modes is easy.

Reasons to reach for it:

• No database required for baseline work. The fileio, cpu, memory, threads, and mutex tests measure the OS and hardware directly. You can characterize a machine before a database ever touches it.
• Multi-threaded by default. --threads=N scales load across cores, so you see contention and throughput under parallelism, not just single-threaded performance.
• Bounded and repeatable. --events=N caps total requests, --time=N caps wall-clock seconds, --rand-seed=N makes RNG deterministic. Results are comparable across runs.
• Intermediate reporting. --report-interval=N emits periodic stats; --report-checkpoints dumps full stats at specified time offsets. You can watch behavior change over the course of a run, not just the final number.
• Latency detail. --percentile=95 computes tail latency; --histogram=on prints a latency histogram. Average throughput alone hides the outliers that matter for databases.
• Database testing when you want it. Point it at MySQL or PostgreSQL with host, port, user, password, and SSL options. Prepared statement mode is configurable. The same harness covers both synthetic OS tests and real database load.
• Config files. --config-file=FILENAME keeps long option sets out of shell history and makes runs reproducible.
• Extensible. New test modes plug in as modules, so the suite grows with your needs.

The typical workflow is three commands: prepare sets up test data (files or tables), run executes the benchmark, cleanup removes it. A file I/O run looks like sysbench fileio --file-test-mode=rndrw prepare, then sysbench fileio --file-test-mode=rndrw run, then cleanup. A CPU run is just sysbench cpu run.

sysfs

sysfs is a pseudo-filesystem that provides an interface to kernel data structures. It is commonly mounted at /sys and has been available since Linux 2.6.0. Many files are read-only, but some are writable, allowing kernel variables to be changed at runtime. Symbolic links are heavily used to connect entries across the filesystem tree, avoiding redundancy.

Some key components under /sys:

• /sys/block — one symbolic link per block device, pointing to the corresponding directory under /sys/devices.
• /sys/bus — one subdirectory per bus type (pci, usb, platform). Each contains devices/ (symlinks to /sys/devices) and drivers/ (one subdirectory per loaded driver).
• /sys/class — one subdirectory per device class (terminals, network, block, graphics, sound). Each contains symlinks to entries in /sys/devices.
• /sys/class/net — symlinks for every real or virtual networking device visible in the accessing process’s network namespace, each pointing to /sys/devices.
• /sys/dev — block/ and char/ subdirectories. Inside are symlinks named major:minor pointing to the sysfs directory for that device. Useful for looking up a device’s sysfs entry from the IDs returned by stat(2).
• /sys/devices — the kernel device tree. A filesystem representation of the hierarchy of device structures.
• /sys/firmware — interfaces for viewing and manipulating firmware-specific objects.
• /sys/fs — subdirectories for filesystems that chose to create them. /sys/fs/cgroup is conventionally a mount point for cgroup filesystems.
• /sys/kernel — files and subdirectories with information about the running kernel. Includes /sys/kernel/mm (memory management) and /sys/kernel/debug/tracing (ftrace mount point).
• /sys/module — one subdirectory per loaded module. Each may contain parameters/ (one file per module parameter, some writable), plus coresize, initstate, refcnt, version, holders/, sections/, and others.
• /sys/power — is the kernel’s global power-management interface. It exposes high-level suspend/sleep/hibernate state and tuning knobs for the whole system.

Most of us will never need to poke here, but for the curious, here are some things you can do:

# Find your network interface's sysfs entry
$ readlink /sys/class/net/enp1s0/
../../devices/pci0000:00/0000:00:01.2/0000:01:00.0/net/enp1s0
# Look up a device by major:minor ID (e.g. /dev/null = 1:3)
$ stat -c "%t %T" /dev/null
1 3
$ readlink /sys/dev/char/1:3
../../devices/virtual/mem/null
# List all PCI devices the kernel found
$ ls /sys/bus/usb/devices/
…
# Check a loaded module's parameters
$ ls /sys/module/<name>/parameters/
$ cat /sys/module/<name>/parameters/<param>
# List huge page sizes the kernel supports
ls /sys/kernel/mm/hugepages/
# See which drivers are loaded for a bus
ls /sys/bus/usb/drivers/

I kind of like having succinct access to network devide config info with it:

$ cat /sys/class/net/enp1s0/{speed,mtu,address}
1000
1500
84:47:09:35:81:2e

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:

• Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev
• Bluesky via <https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy>

☮️