Infinite BusyBox with systemd

Lightweight virtual containers with PID 1.

In this article, I demonstrate a method to build one Linux system within another using the latest utilities within the systemd suite of management tools. The guest OS container design focuses upon BusyBox and Dropbear for the userspace system utilities, but I also work through methods for running more general application software so the containers are actually useful.

This tutorial was developed on Oracle Linux 7, and it likely will run unchanged on its common brethren (Red Hat, CentOS, Scientific Linux), and from here forward, I refer to this platform simply as V7. Slight changes may be necessary on other systemd platforms (such as SUSE, Debian or Ubuntu). Oracle's V7 runs only on the x86_64 platform, so that's this article's primary focus.

Required Utilities

Red Hat saw fit to remove the long-included BusyBox binary from its V7 distribution, but this easily is remedied by downloading the latest binary directly from the project's Web site. Since the /home filesystem gets a large amount of space by default when installing V7, let's put it there for now. Run the commands below as root until indicated otherwise:


cd /home
wget http://busybox.net/downloads/binaries/latest/busybox-x86_64

You also can get a binary copy of the Dropbear SSH server and client from this location:


wget http://landley.net/aboriginal/downloads/
↪binaries/extras/dropbearmulti-x86_64

For this article, I used the following versions:

  • BusyBox v1.21.1.

  • Dropbear SSH multi-purpose v2014.63.

These are static binaries that do not link against shared objects—nothing else is required to run them, and they are ideal for building a new UNIX-ish environment quickly.

Build a chroot

The chroot system call and the associated shell utility allow an arbitrary subdirectory somewhere on the system to be declared as the root for all child processes. The commands below populate the "chroot jail", then lock you in. Note that the call to chroot needs your change to the SHELL environment variable below, as you don't have bash inside the jail (and it's likely the default value of $SHELL):


export SHELL=/bin/sh
mkdir /home/nifty
mkdir /home/nifty/bin
cd /home/nifty/bin
cp /home/busybox-x86_64 /home/dropbearmulti-x86_64 .
chmod 755 busybox-x86_64 dropbearmulti-x86_64
./busybox-x86_64 --list | awk '{print "ln -s 
 ↪busybox-x86_64 " $0}' | sh
chroot /home/nifty
export PATH=/bin
ls -l
###(try some commands)
exit

Take some time to explore your shell environment after you launch your chroot above before you exit. Notice that you have a /bin directory, which is populated by soft links that resolve to the BusyBox binary. BusyBox changes its behavior depending upon how it is called—it bundles a whole system of utility programs into one convenient package.

Try a few additional UNIX commands that you may know. Some that work are vi, uname, uptime and (of course) the shell that you are working inside. Commands that don't work include ps, top and netstat. They fail because they require the /proc directory (which is dynamically provided by the Linux kernel)—it has not been mounted within the jail.

Note that few native utilities will run in the chroot without moving many dependent libraries (objects). You might try copying bash or gawk into the jail, but you won't be able to run them (yet). In this regard, BusyBox is ideal, as it depends upon nothing.

Build a Minimal UNIX System and Launch It

The systemd suite includes the eponymous program that runs as PID 1 on Linux. Among many other utilities, it also includes the nspawn program that is used to launch containers. Containers that are created by nspawn fix most of the problems with chroot jails. They provide /proc, /dev, /run and otherwise equip the child environment with a more capable runtime.

Next, you are going to configure a getty to run on the console of the container that you can use to log in. Being sure that you have exited your chroot from the previous step, run the following commands as root:


mkdir /home/nifty/etc
mkdir /home/nifty/root
echo 'NAME="nifty busybox container"' > 
 ↪/home/nifty/etc/os-release
cd /home/nifty
ln -s bin sbin
ln -s bin usr/bin
echo 'root::0:0:root:/root:/bin/sh' > 
 ↪/home/nifty/etc/passwd
echo 'console::respawn:/bin/getty 38400 /dev/console' >
 ↪/home/nifty/etc/inittab
tar cf - /usr/share/zoneinfo | (cd /home/nifty; tar xvpf -)
systemd-nspawn -bD /home/nifty

After you have executed the nspawn above, you will be presented with a "nifty login" prompt. Log in as root (there is no password—yet), and try a few more commands. You immediately will notice that ps and top work, and there is now a /proc.

You also will notice that the processes that appear in the child container also appear on the host system, but different PIDs will be assigned between the parent and child.

Note that you'll also receive the message: "The kernel auditing subsystem is known to be incompatible with containers. Please make sure to turn off auditing with 'audit=0' on the kernel command line before using systemd-nspawn. Sleeping for 5s..." The audit settings don't seem to impact the BusyBox container login, but you can adjust your kernel command line in your grub configuration (at least to silence the warning and stop the delay).

Running Dropbear SSH in Your Container

It's best if you configure a non-root user of your system and forbid network root logins. The reasoning will become clear when I address container security.

Run all of these commands as root within the container:


cd /bin
ln -s dropbearmulti-x86_64 dropbear
ln -s dropbearmulti-x86_64 ssh
ln -s dropbearmulti-x86_64 scp
ln -s dropbearmulti-x86_64 dropbearkey
ln -s dropbearmulti-x86_64 dropbearconvert

Above, you have established the names that you need to call Dropbear, both the main client and server, and the sundry key generation and management utilities.

You then generate the host keys that will be used by this container, placing them in a new directory /home/nifty/etc/dropbear (as viewed by the host):


mkdir /etc/dropbear
dropbearkey -t rsa -f /etc/dropbear/dropbear_rsa_host_key
dropbearkey -t dss -f /etc/dropbear/dropbear_dss_host_key
dropbearkey -t ecdsa -f /etc/dropbear/dropbear_ecdsa_host_key

Various directories are then created that you will need shortly:


mkdir -p /var/log/lastlog
mkdir /home
mkdir /var/run
mkdir /tmp
mkdir /var/tmp
chmod 01777 /tmp /var/tmp

You then create the inittab, which will launch syslogd and Dropbear once at startup (in addition to the existing getty that is respawned whenever it dies):


echo ::sysinit:/bin/syslogd >> /etc/inittab
echo '::sysinit:/bin/dropbear -w -p 2200' >> /etc/inittab

Next, you add a shadow file and create a password for root:


echo root:::::::: > /etc/shadow
chmod 600 /etc/shadow
echo root:x:0: > /etc/group
passwd -a x root

Note that the BusyBox passwd call used here generated an MD5 hash—there is a $1$ prefix in the second field of /etc/shadow for root. Additional hashing algorithms are available from this version of the passwd utility (the options -a s will generate a $5$ SHA256 hash, and -a sha512 will generate a $6$ hash). However, Dropbear seems to be able to work only with $1$ hashes for now.

Finally, add a new user to the system, and then halt the container:


adduser -h /home/luser -D luser
passwd -a x luser

halt

You should see container shutdown messages that are similar to a system halt.

When you next start your container, it will listen on socket 2200 for connections. If you want remote hosts to be able to connect to your container from anywhere on the network, run this command as root on the host to open a firewall port:


iptables -I INPUT -p tcp --dport 2200 --syn -j ACCEPT

The port will be open only until you reboot. If you'd like the open port to persist across reboots, use the firewall-config command from within the X Window System (set the port on the second tab in the GUI).

In any case, run the container with the previous nspawn syntax, then try to connect from another shell within the parent host OS with the following:


ssh -l luser -p 2200 localhost

You should be able to log in to the luser account under a BusyBox shell.

Executing Programs with Runtime Dependencies

If you copy various system programs from /bin or /usr/bin into your container, you immediately will notice that they don't work. They are missing shared objects that they need to run.

If you had previously copied the gawk binary in from the host:


cp /bin/gawk /home/nifty/bin/

you would find that attempts to execute it fail with "gawk: not found" errors (on the host, there usually will be explicit complaints about missing shared objects, which are not seen in the container).

You easily can make most of the 64-bit libraries available with an argument to nspawn that establishes a bind mount:


systemd-nspawn -bD /home/nifty --bind-ro=/usr/lib64

Then, from within the container, run:


cd /
ln -s usr/lib64 lib64

You then will find that many 64-bit binaries that you copy in from the host will run (running /bin/gawk -V returns "GNU Awk 4.0.2"—an entire Oracle 12c instance is confirmed to run this way). The read-only library bind mount also has the benefit of receiving security patches immediately when they appear on the host.

There is a significant security problem with this, however. The root user in the container has the power to mount -o remount,rw /usr/lib64 and, thus, gain write access to your host library directories. In general, you cannot give root to a container user that you don't know and trust—among other problems, these mounts can be abused.

You also might be tempted to mount the /usr/lib directory in the same manner. The difficulty you will find is that the systemd binary will be found under that directory tree, and nspawn will try to execute it in preference to BusyBox init. Enabling 32-bit runtime support likely will involve more directory and mounting gymnastics than was required for /usr/lib64.

And now, I'm going off on a tangent.

______________________

Charles Fisher has an electrical engineering degree from the University of Iowa and works as a systems and database administrator for a Fortune 500 mining and manufacturing corporation.