Infinite BusyBox with systemd

Infinite BusyBox

It might interesting to launch tens, hundreds, or even thousands of containers at once. You could launch the clones by making copies of the /home/nifty directory, then adjusting the systemd service file. To simplify, you will place your new containers in /home/nifty1, /home/nifty2, /home/nifty3 ... using integer suffixes on the directories to differentiate them.

Please make sure that you have disabled kernel auditing to remove the five-second delay when launching containers. At the very least, press e at the grub menu at boot time, and add the audit=0 to your kernel command line for a one-time boot.

I'm going to return to the subject of systemd "instantiated services" that I touched upon with the telnetd service file that replaced inetd. This technique will allow you to use one service file to launch all of your containers. Such a service has an @ character in the filename that is used to refer to a particular, differentiated instance of a service, and it allows the use of the %i placeholder within the service file for variable expansion. Run the following on the host as root to place your service file for instantiated containers:

echo '[Unit]
Description=nifty container # %i

ExecStart=/usr/bin/systemd-nspawn -bD /home/nifty%i
 ↪--bind-ro=/home/cbin:/cbin --bind-ro=/usr/share/zoneinfo
KillMode=process' > /etc/systemd/system/nifty@.service

The %i above first adjusts the description, then adjusts the launch directory for the nspawn. The content that will replace the %i is specified on the systemctl command line.

To test this, make a directory called /home/niftyslick. The service file doesn't limit you to numeric suffixes. You will adjust the SSH port after the copy. Run this as root on the host:

cd /home
mkdir niftyslick
(cd nifty; tar cf - .) | (cd niftyslick; tar xpf -)
sed "s/2200/2100/" < nifty/etc/inittab > niftyslick/etc/inittab

systemctl start nifty@slick.service

Bearing this pattern in mind, let's create a script to produce these containers in massive quantities. Let's make a thousand of them:

cd /home
for x in $(seq 1 999)
  mkdir "nifty${x}"
  (cd nifty; tar cf - .) | (cd "nifty${x}"; tar xpf -)
  sed "s/2200/$((x+2200))/" < nifty/etc/inittab > 
  systemctl start nifty@${x}.service

As you can see below, this test launches all containers:

$ ssh -l luser -p 3199 localhost
The authenticity of host '[localhost]:3199 ([::1]:3199)' 
 ↪can't be established.
ECDSA key fingerprint is 07:26:15:75:7d:15:56:d2:ab:9e:
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:3199' (ECDSA) 
 ↪to the list of known hosts.
luser@localhost's password: 
~ $ sh --help
BusyBox v1.21.1 (2013-07-08 11:34:59 CDT) multi-call binary.

Usage: sh [-/+OPTIONS] [-/+o OPT]... [-c 'SCRIPT' 
 ↪[ARG0 [ARGS]] / FILE [ARGS]]

Unix shell interpreter

~ $ cat /proc/self/cgroup

The output of systemctl will list each of your containers:

# systemctl
machine-nifty1.scope    loaded active running   Container nifty1
machine-nifty10.scope   loaded active running   Container nifty10
machine-nifty100.scope  loaded active running   Container nifty100
machine-nifty101.scope  loaded active running   Container nifty101
machine-nifty102.scope  loaded active running   Container nifty102

More detail is available with systemctl status:

machine-nifty10.scope - Container nifty10
   Loaded: loaded (/run/systemd/system/machine-nifty10.scope; 
  Drop-In: /run/systemd/system/machine-nifty10.scope.d
            ↪90-Description.conf, 90-Slice.conf,
   Active: active (running) since Tue 2014-11-18 23:01:21 CST; 
            ↪11min ago
   CGroup: /machine.slice/machine-nifty10.scope
            ↪2871 init      
            ↪2880 /bin/syslogd
            ↪2882 /bin/dropbear -w -p 2210

Nov 18 23:01:21 localhost.localdomain systemd[1]: 
 ↪Starting Container nifty10.
Nov 18 23:01:21 localhost.localdomain systemd[1]: 
 ↪Started Container nifty10.

The raw number of containers that you can launch with this approach is more directly impacted by kernel limits than general disk and memory resources. Launching the containers above used no swap on a small system with 2GB of RAM.

After you have investigated a few of the containers and their listening ports, the easiest and cleanest way to get all of your containers shut down is likely a reboot.

Container Security

A number of concerns are raised with these features:

1) Since BusyBox and Dropbear were not installed with the RPM host package tools, updates to them will have to be loaded manually. It will be important to check from time to time if new versions are available and if any security flaws have been discovered. If it is necessary to load new versions, the binaries should be copied to all containers that are potentially used, which should then be restarted (especially if a security issue is involved).

2) Control of the root user in the container cannot be passed to an individual that you do not trust. For a particular example, if the lib64/cbin/zoneinfo bind mounts above are used, the container root user can issue the command:

mount -o remount,rw /usr/lib64

at which point the container root will have full write privileges on your 64-bit libraries, container bin or zoneinfo. The systemd-nspawn man page goes even further, with the warning:

Note that even though these security precautions are taken systemd-nspawn is not suitable for secure container setups. Many of the security features may be circumvented and are hence primarily useful to avoid accidental changes to the host system from the container. The intended use of this program is debugging and testing as well as building of packages, distributions and software involved with boot and systems management.

The crux is that untrusted users cannot have the container root, any more than you would give them full system root. The container root will have the CAP_SYS_ADMIN privilege, which allows full control of the system. If you want to isolate non-root users further, the container environment does limit non-root users' visibility into host activities, as they cannot see the full process table.

3) Note that the BusyBox su and passwd utilities above do not work when installed in the manner outlined here. They lack the appropriate filesystem permissions. To fix this, chmod u+s busybox-x86_64 could be executed, but this is also distasteful from a security perspective. Removing the links and copying the BusyBox binary to su and passwd before applying the setuid privilege might be better, but only slightly. It would be best if su was unavailable and another mechanism was found for password changes.

4) The -w argument to the Dropbear SSH server above prevents root logins from the network. It is somewhat distasteful, from a security perspective, to relax this limitation. The net effect is that root is locked out of active use in the container when -w is forced, and su/passwd do not have setuid. If it is at all possible to live with such an arrangement for your container, try to do so, as the security is much improved.

systemd Controversy

There is a high degree of hostility toward systemd from users of Linux. This hostility is divided into two main complaints:

  • The classic inittab from UNIX System V should not be changed because it is well understood.

  • Increasing features are bundled into systemd that bring dangerous complexity to a critical system process.

Toward the first point, nostalgia for legacy systems is not always misguided, but it cannot be allowed to hinder progress unreasonably. A classic System V init is not able to nspawn and has far less control over processes running on a system. The features delivered by systemd surely justify the inconvenience of change in many situations.

Toward the second point, much thought was placed into the adoption of the architecture of systemd by skilled designers from diverse organizations. Those most critical of the new environment should acknowledge the technical success of systemd as it is adopted by the majority of the Linux community.

In any case, the next decade will see popular Linux server distributions equipped with systemd, and competent administrators will not have the option of ignoring it. It is unfortunate that the introduction of systemd did not include more guidance for the user community, but the new features are compelling and should not be overlooked.


Charles Fisher has an electrical engineering degree from the University of Iowa and works as a systems and database administrator for a Fortune 500 mining and manufacturing corporation.