This project isolates a given command and args using a container backend. Supported backends are bubblewrap, lxc and chroot. Latter is mainly thought for updating the underlying Linux system root.

A unique, fresh ZFS clone is used as dataset for each isolation, and discarded after use. A privileged daemon cleans up used clones and provides fresh clones.

For the lxc and chroot engines the clone’s root is mounted read-write, so changes are made on disk (ZFS copy-on-write), preventing high memory usage under load. The bwrap engine intentionally mounts the root read-only and layers writable tmpfs over /tmp and /home; guest writes elsewhere do not persist and are backed by memory.

The project consists of the main script isolate, the common bindings common.sh and seccomp_wrapper.c, which complements bubblewrap. Latter works around bubblewraps --new-session option, which prevents feeding input from the isolated environment to the host terminal, but also prevents job control of the spawned shell.

isolate can work without ZFS. In this case, modifications to the backing Linux system root are persistent. For such setup, see “Setup without ZFS”

Setup

System preparations

Copy isolate and common.sh to /usr/local/bin We expect a group isolate to exist. Users that shall be able to use isolate shall be members of this group.

On NixOS, skip this manual step and use the flake (see “NixOS”).

NixOS

This repository is a flake providing an overlay, a package and a NixOS module. The module installs isolate system-wide, creates the isolate group, and generates one systemd path+service pair per template for automatic clone regeneration. You remain responsible for creating the template ZFS dataset; it is only named in the config.

Add the flake as an input and wire the overlay + module (mirrors any other local-flake input):

inputs.isolate = {
  url = "path:./isolate";           # or a git URL
  inputs.nixpkgs.follows = "nixpkgs";
};

# in your machine's module list:
({ pkgs, ... }: {
  nixpkgs.overlays = [ isolate.overlays.default ];
  services.isolate = {
    enable = true;
    templates.debisl = {
      dataset  = "rpool/isolate/debisl";  # you create this dataset yourself
      count    = 10;                        # fresh clones to keep available
      stateDir = "/isolate";                # parent of the dataset's mountpoint
    };
  };
})
isolate.nixosModules.default

Add the users that may use ZFS features to the group, e.g. users.users.alice.extraGroups = [ "isolate" ];

nixos-rebuild switch succeeds even when the dataset does not exist yet. The regenerate service fails cleanly while the dataset is missing:

# before creating the dataset, this fails (by design):
systemctl start isolate-regenerate-debisl.service   # -> failed, "cannot open 'rpool/isolate/debisl'"

Create the template dataset as described below (Creating the backing Linux system root), install the seccomp wrapper with isolate -w /isolate/debisl (uses the prebuilt static binary shipped with the package, no compiler needed), then populate clones by (re)starting the unit:

systemctl restart isolate-regenerate-debisl.service

From then on each isolated run consumes a clone and, on exit, touches the signal file, which triggers the path unit to replenish clones automatically.

Creating the backing Linux system root

In the following, rpool/isolate/debisl is the dataset and /isolate/debisl its mountpoint. Both may be chosen arbitrarily.

Do the following steps as root user:

1. Create a system Linux system root dataset root, e.g. zfs create rpool/isolate/debisl
2. Create a template Linux system root using e.g. debootstrap /isolate/debisl
3. [only bwrap] Install the seccomp wrapper into the template: isolate -w /isolate/debisl This copies the prebuilt static binary shipped next to isolate (e.g. by the nix package). When none is shipped (e.g. a git checkout), it compiles seccomp_wrapper.c instead; you then need gcc and the libseccomp static libs:

   gentoo: install libseccomp, USE: static-libs
   debian: install libseccomp-dev
       

4. Chroot into the environment and install what you need isolate -e chroot -t /isolate/debisl bash
5. After you are done, create a snapshot

   zfs snapshot "rpool/isolate/debisl@$(date "+%Y%m%dT%H%M%SZ")"
       

6. Generate clones for usage by isolate: isolate -r rpool/isolate/debisl 5

Your ZFS structure will look somewhat like this:

NAME                                             USED  AVAIL     REFER  MOUNTPOINT
rpool/isolate                                   18.8G   488G     9.96G  /isolate
rpool/isolate/deb_gnuradio                      1.86G   488G     1.72G  /isolate/deb_gnuradio
rpool/isolate/deb_gnuradio_1677838587626715157   184K   488G     1.72G  /isolate/deb_gnuradio_1677838587626715157
rpool/isolate/deb_gnuradio_1677838587665325261   184K   488G     1.72G  /isolate/deb_gnuradio_1677838587665325261
rpool/isolate/deb_gnuradio_1677838587703660046   184K   488G     1.72G  /isolate/deb_gnuradio_1677838587703660046
rpool/isolate/debisl                            6.96G   488G     6.39G  /isolate/debisl
rpool/isolate/debisl_1677860441807895636           8K   488G     6.39G  /isolate/debisl_1677860441807895636
rpool/isolate/debisl_1677860480418781581           8K   488G     6.39G  /isolate/debisl_1677860480418781581
rpool/isolate/debisl_1677860488091186587           8K   488G     6.39G  /isolate/debisl_1677860488091186587

Updating the backing Linux system root

Do the following steps as root user:

1. Chroot into the environment and install what you need isolate -e chroot -t /isolate/debisl bash
2. After you are done, create a snapshot

   zfs snapshot "rpool/isolate/debisl@$(date "+%Y%m%dT%H%M%SZ")"
       

3. Generate clones for usage by isolate: isolate -f -r rpool/isolate/debisl 5

Enabling automatic clone regeneration

The path and system units we provide expect the isolation Linux system roots to be at rpool/isolate/xxx. If your paths are different, please modify the service and path files. Also, the number of clones is hardcoded in the unit file. You might want to adjust this too. A systemd path unit watches a signal file (touched by isolate on each exit) and triggers clone regeneration when an isolated environment exits and frees the used clone.

Copy isolate_regenerate@.path and isolate_regenerate@.service to /etc/systemd/system.

To enable regeneration for rpool/isolate/debisl, enable isolate_regenerate@debisl.path through systemctl enable --now isolate_regenerate@debisl.path

X11, Wayland, Pulseaudio and GPU passthrough

Display, audio and GPU passthrough are only supported with the bwrap engine.

X11 (-1) should work without any configuration.

Wayland (-3) binds the compositor socket ($XDG_RUNTIME_DIR/$WAYLAND_DISPLAY) into the guest and points WAYLAND_DISPLAY at it. Unlike X11, no xhost-style grant is needed: access is just filesystem permission on the socket, which the invoking user already holds (bwrap maps the in-container uid 0 to that user). For pure X11 clients on a Wayland host, XWayland means -1 keeps working.

GPU (-4) binds /dev/dri into the guest for hardware acceleration; combine with -3 for accelerated Wayland (-3 -4). Two guest-side prerequisites apply: the template rootfs must contain the GPU userspace drivers (e.g. Mesa), because bwrap exposes only the template filesystem, not the host; and the invoking user needs access to the render node (usually the render=/=video group, or a world-readable renderD*).

For pulseaudio (-2) to work, you will have to allow anonymous access onto the pulse socket. Add the following to /etc/pulse/default.pa:

load-module module-native-protocol-unix auth-anonymous=1 socket=/tmp/pulse-socket

Setup without ZFS

isolate can function without ZFS by using template mode. Create a backing Linux system root in a directory of your choice. You can spawn isolate as usual, but rather using the -t instead of the -z option. Modifications to the underlying system root are of course permanent in template mode.

Executing isolated programs

See isolate -h:

spawn:                        ./isolate [-e engine] [-t templatedir] [-1] [-2] [-n] [-p pwd] [-d dir_host:target_ctr]... [-v env_name:env_value]... [command] [args] ...
spawn:                        ./isolate [-e engine] [-z zfs_template] [-1] [-2] [-n] [-p pwd] [-d dir_host:target_ctr]... [-v env_name:env_value]... [command] [args] ...
refresh available zfs clones: ./isolate [-r template] [-f] [-s] [num_should_avail]
  [!] refresh ALWAYS uses the latest snapshot on the dataset
install seccomp wrapper:      ./isolate -w template

OPTIONS
         -e: engine: one of {bwrap, chroot, lxc}, defaults to bwrap
         -d: dirs to mount into the sandbox, mounts dir_host to target_ctr in the container
         -u: uid [integer] to use as base uid. uid and the following 65535 uids are mapped to [0:65535] in the sandbox.
         -g: gid [integer] to use as base gid. gid and the following 65535 gids are mapped to [0:65535] in the sandbox.
             NOTE: when using -u or -g, you should align the ownership of the template to the range specified.
         -i: ignore SIGINT, keep running. Current the workaround until signal passing into the guest is implemented.
         -v: var:value to pass into the sandbox
         -p: pwd: switch to this directory on spawn. Defaults to /
         -1: bind X11 socket into guest
         -2: bind Pulseaudio socket into guest
         -3: bind Wayland socket into guest (uses WAYLAND_DISPLAY/XDG_RUNTIME_DIR)
         -4: expose GPU (/dev/dri) into guest for hardware acceleration
         -n: share host network
         -x: trace
         -f: regenerate ALL templates of given zfs template
         -s: skip generating new clones
         -q: quiet: only print warnings and prompts
         -w: compile/install seccomp_wrapper into template's /usr/bin (arg: template dir or zfs dataset)

  add all users that should be able to use zfs features to the `isolate' group

ENVIRONMENT VARIABLES
   PRE_SPAWN_HOOK: command that is run before [command args] are run in the isolated environment
                   $ROOTFS references the root of the isolated environment to be started
   POST_SPAWN_HOOK: command that is run after [command args] has completed in the isolated environment
                   $ROOTFS references the root of the isolated environment to be started
   DISABLE_SECCOMP_WRAPPER [=!'']: disable the bwrap seccomp wrapper that prevents IOCTL to host
   LXC_NET_BR [=!'']: bridge to be used by LXC, defaults to br_vm
   LXC_MAP_TUN [!='']: map /dev/net/tun into the container
   LXC_CPU_QUOTA: percentage (1-100), this amount of CPU processing time will be available to the container. Default: 20
   LXC_MEM_MB_QUOTA: memory available to guest, in MB
   DEBUG_LXC_SPAWN [=!'']: if nonzero len, write lxc start log to /tmp/isolate_lxc_log

SUPPORTED BY ENGINE
  | engine | env | {U/G}ID | net | cmd+args | dirs | pwd | X11 | Pulseaudio | Wayland | GPU |
  |--------+-----|---------------+----------+------|-----+-----+------------+---------+-----|
  | chroot |     |         |  X  | X        |      |     |     |            |         |     |
  | bwrap  |  X  |    *    |  X  | X        | X    | X   |  X  |     X      |    X    |  X  |
  | lxc    |  X  |    X    |  X  | X        | X    | X   |     |            |         |     |

  * bwrap always creates a user namespace: in-container uid 0 maps to the
    unprivileged invoking user, so the sandbox never runs as real host root.
    -u/-g shift the base uid/gid of that mapping.