I recently found myself needing a machine to compile binaries on for a CentOS server. I first considered actually spinning up a CentOS system on a VPS; however, that seemed a little overboard just for compiling, I then realized that this would be the perfect use for a container. I could have an identical system to the one where the binaries will be deployed on, and at little cost since it can simply be blown away when I’m done. In order to set up my compile machine I used LXC.

LXC, or “Linux Containers”, are a set of tools for creating full-featured containers. Compared to other tools such as systemd-nspawn, LXC is much more complex, and it has been used to build projects such as Docker. Docker has since moved away from LXC, however LXC is still one of the huge players in the Linux container game. The Linux container project also brings LXD, a daemon that can be used to manage containers. LXD makes a larger use of system images, as opposed to templates, in order to allow quick deployment of containers. Together these projects allow easy deployment and management of containers, as well as as advanced features and customizability.

In the following post I will go through the process of setting up several LXC containers, using both the traditional template-based method, as well as experiment with LXD to download image-based containers.

LXC

Requirements

On Arch Linux, which is what I’m using, the packages lxc and arch-install-scripts are required.

In order to see if the currently running kernel is properly configured to use lxc, run lxc-checkconfig, it should detect if anything is missing from the current configuration.

[root]# lxc-checkconfig                                                               [email protected]
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: missing
Network namespace: enabled
Multiple /dev/pts instances: missing

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: enabled
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
FUSE (for use with lxcfs): enabled

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/sbin/lxc-checkconfig

One of the features that it is normal to see missing is user namespaces. In order to use this feature on Arch Linux, a custom compiled kernel is necessary. The default kernel has user namespaces turned off due to security concerns.

Templates

The traditional way of setting up an LXC container is using templates. Templates are prebuilt shell scripts that will build a system image.

View the available templates in /usr/share/lxc/templates:

total 235K
drwxr-xr-x 2 root root   21 Aug 30 06:10 .
drwxr-xr-x 6 root root    8 Aug 30 06:10 ..
-rwxr-xr-x 1 root root  13K Aug 30 06:10 lxc-alpine
-rwxr-xr-x 1 root root  14K Aug 30 06:10 lxc-altlinux
-rwxr-xr-x 1 root root  11K Aug 30 06:10 lxc-archlinux
-rwxr-xr-x 1 root root  12K Aug 30 06:10 lxc-busybox
-rwxr-xr-x 1 root root  29K Aug 30 06:10 lxc-centos
-rwxr-xr-x 1 root root  11K Aug 30 06:10 lxc-cirros
-rwxr-xr-x 1 root root  20K Aug 30 06:10 lxc-debian
-rwxr-xr-x 1 root root  18K Aug 30 06:10 lxc-download
-rwxr-xr-x 1 root root  49K Aug 30 06:10 lxc-fedora
-rwxr-xr-x 1 root root  28K Aug 30 06:10 lxc-gentoo
-rwxr-xr-x 1 root root  14K Aug 30 06:10 lxc-openmandriva
-rwxr-xr-x 1 root root  16K Aug 30 06:10 lxc-opensuse
-rwxr-xr-x 1 root root  42K Aug 30 06:10 lxc-oracle
-rwxr-xr-x 1 root root  12K Aug 30 06:10 lxc-plamo
-rwxr-xr-x 1 root root  19K Aug 30 06:10 lxc-slackware
-rwxr-xr-x 1 root root  27K Aug 30 06:10 lxc-sparclinux
-rwxr-xr-x 1 root root 6.7K Aug 30 06:10 lxc-sshd
-rwxr-xr-x 1 root root  26K Aug 30 06:10 lxc-ubuntu
-rwxr-xr-x 1 root root  12K Aug 30 06:10 lxc-ubuntu-cloud

To get help on container creation use lxc-create --help. lxc-create is used to create a container from the templates.

There are many options:

[root]# lxc-create --help
Options :
  -n, --name=NAME               NAME of the container
  -f, --config=CONFIG           Initial configuration file
  -t, --template=TEMPLATE       Template to use to setup container
  -B, --bdev=BDEV               Backing store type to use
      --dir=DIR                 Place rootfs directory under DIR

  BDEV options for LVM (with -B/--bdev lvm):
      --lvname=LVNAME           Use LVM lv name LVNAME
                                (Default: container name)
      --vgname=VG               Use LVM vg called VG
                                (Default: lxc)
      --thinpool=TP             Use LVM thin pool called TP
                                (Default: lxc)

  BDEV options for Ceph RBD (with -B/--bdev rbd) :
      --rbdname=RBDNAME         Use Ceph RBD name RBDNAME
                                (Default: container name)
      --rbdpool=POOL            Use Ceph RBD pool name POOL
                                (Default: lxc)

  BDEV option for ZFS (with -B/--bdev zfs) :
      --zfsroot=PATH            Create zfs under given zfsroot
                                (Default: tank/lxc)

  BDEV options for LVM or Loop (with -B/--bdev lvm/loop) :
      --fstype=TYPE             Create fstype TYPE
                                (Default: ext3)
      --fssize=SIZE[U]          Create filesystem of
                                size SIZE * unit U (bBkKmMgGtT)
                                (Default: 1G, default unit: M)

Common options :
  -o, --logfile=FILE               Output log to FILE instead of stderr
  -l, --logpriority=LEVEL          Set log priority to LEVEL
  -q, --quiet                      Don't produce any output
  -P, --lxcpath=PATH               Use specified container path
  -?, --help                       Give this help list
      --usage                      Give a short usage message
      --version                    Print the version number

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

See the lxc-create man page for further information.

Configuration

Settings can be specified in the lxc config. I’m using ZFS as a backing store, btrfs and lvm are also supported as well as “none” which is the default.

I added my backend, lxc path, and zfs root.

[root]# nano /etc/lxc/lxc.conf
lxc.rootfs.backend = zfs
lxc.lxcpath = /var/lib/lxc
lxc.bdev.zfs.root = vault/lxc

I also set some container defaults, i’m using a systemd-network bridge setup the same way I detailed in a previous post on systemd-nspawn:

[root]# nano /etc/lxc/default.conf
lxc.network.type = veth
lxc.network.link = br0

I created a new ZFS dataset for my lxc containers

[root]# zfs create vault/var/lib/lxc -o mountpoint=/var/lib/lxc

Container Creation

I created a centos container using zfs as a backing store, it requires the AUR package yum:

[root]# lxc-create --name=centos-builder --bdev=zfs --template=centos
Host CPE ID from /etc/os-release:
This is not a CentOS or Redhat host and release is missing, defaulting to 6 use -R|--release to specify release
Checking cache download in /var/cache/lxc/centos/x86_64/6/rootfs ...
Cache found. Updating...
Loaded plugins: fastestmirror
Setting up Update Process
Loading mirror speeds from cached hostfile
 * base: mirror.it.ubc.ca
 * extras: mirror.it.ubc.ca
 * updates: mirror.it.ubc.ca
base                                                                        | 3.7 kB     00:00
extras                                                                      | 3.4 kB     00:00
updates                                                                     | 3.4 kB     00:00
updates/primary_db                                                          | 3.1 MB     00:00
No Packages marked for Update
Loaded plugins: fastestmirror
Cleaning repos: base extras updates
0 package files removed
Update finished
Copy /var/cache/lxc/centos/x86_64/6/rootfs to /usr/lib/lxc/rootfs ...
Copying rootfs to /usr/lib/lxc/rootfs ...
Storing root password in '/var/lib/lxc/centos-builder/tmp_root_pass'
Expiring password for user root.
passwd: Success

Container rootfs and config have been created.
Edit the config file to check/enable networking setup.

The temporary root password is stored in:

        '/var/lib/lxc/centos-builder/tmp_root_pass'

To reset the root password, you can do:

        lxc-start -n centos-builder
        lxc-attach -n centos-builder -- passwd
        lxc-stop -n centos-builder

A new data set has been created with the name of the container.

[root]# zfs list                                                                      [email protected]
NAME                               USED  AVAIL  REFER  MOUNTPOINT
vault                              337G  93.3G    96K  none
vault                              309G  93.3G    96K  none
vault/ROOT                        18.8G  93.3G    96K  none
vault/ROOT/default                18.8G  93.3G  10.2G  /
vault/home                         253G  93.3G   112G  /home
...
vault/var/lib/lxc                  224M   283G   100K  /var/lib/lxc
vault/var/lib/lxc/centos-builder   224M   283G   224M  /var/lib/lxc/centos-builder/rootfs

The config file made with the container can be found at /var/lib/lxc/centos-builder/config, it can be edited if the networking needs changing.

[root]# cat /var/lib/lxc/centos-builder/config
# Template used to create this container: /usr/share/lxc/templates/lxc-centos
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)

lxc.network.type = veth
lxc.network.link = br0
lxc.network.hwaddr = fe:a1:a9:24:d8:54
lxc.rootfs = /var/lib/lxc/centos-builder/rootfs
lxc.rootfs.backend = zfs

# Include common configuration
lxc.include = /usr/share/lxc/config/centos.common.conf

lxc.arch = x86_64
lxc.utsname = centos-builder

Start the Container

Start the container, and login to the console. The password will be in the container directory /var/lib/lxc/centos-builder/tmp_root_pass.

[root]# lxc-start -n centos-builder

Now attach to the console with lxc-console you should find yourself at the login of a brand-new machine.

[root]# lxc-console -n centos-builder
CentOS release 6.8 (Final)
Kernel 4.8.4-1-ARCH on an x86_64

centos-builder login:

Container Networking

Networking can now be setup in the container using a bridge, check the interface name and set up the network as if on any other computer.

Check the interface:

[[email protected] /]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
4: [email protected]: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d6:49:f8:16:59:d9 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Here the interface shows up as as [email protected]. Use the prefix eth0 as the interface.

I setup a static IP using systemd-networkd.

If a static IP is being used, the file /usr/lib/systemd/network/80-container-host0.network that would have setup DHCP needs to be masked.

[[email protected] ~]# ln -sf /dev/null /etc/systemd/network/80-container-host0.network

Start and enable systemd-networkd.

[[email protected] ~]# systemctl enable systemd-networkd
[[email protected] ~]# systemctl start systemd-networkd

Create the configuration file.

[[email protected] /]# nano /etc/systemd/network/vethernet.network
[Match]
Name=eth0

[Network]
DNS=192.168.0.1
Address=192.168.0.10/24
Gateway=192.168.0.1

Restart the service and you should have an IP.

[[email protected] /]# systemctl restart systemd-networkd
[[email protected] /]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
8: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f2:5a:ef:12:40:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.10/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f05a:efff:fe12:40a4/64 scope link
       valid_lft forever preferred_lft forever

Pre-Built Images

Alternatively to using templates, images that have been pre-built can also be downloaded using the lxc-download script.

Running lxc-download runs you through an interactive prompt that will let you download an image from the listed options.

You can also specify the release and architecture beforehand.

[root]# lxc-create -t download --help
LXC container image downloader

Special arguments:
[ -h | --help ]: Print this help message and exit.
[ -l | --list ]: List all available images and exit.

Required arguments:
[ -d | --dist <distribution> ]: The name of the distribution
[ -r | --release <release> ]: Release name/version
[ -a | --arch <architecture> ]: Architecture of the container

Optional arguments:
[ --variant <variant> ]: Variant of the image (default: "default")
[ --server <server> ]: Image server (default: "images.linuxcontainers.org")
[ --keyid <keyid> ]: GPG keyid (default: 0x...)
[ --keyserver <keyserver> ]: GPG keyserver to use
[ --no-validate ]: Disable GPG validation (not recommended)
[ --flush-cache ]: Flush the local copy (if present)
[ --force-cache ]: Force the use of the local copy even if expired

LXC internal arguments (do not pass manually!):
[ --name <name> ]: The container name
[ --path <path> ]: The path to the container
[ --rootfs <rootfs> ]: The path to the container's rootfs
[ --mapped-uid <map> ]: A uid map (user namespaces)
[ --mapped-gid <map> ]: A gid map (user namespaces)

Let’s set up Ubuntu 17.04 container non-interactively. When specifying LXC arguments, -- must be passed before the template specific arguments.

[root]#  lxc-create -t download -n buntzy -B zfs -- \
                   --dist ubuntu --release zesty --arch amd64
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu container (release=zesty, arch=amd64, variant=default)

To enable sshd, run: apt-get install openssh-server

For security reason, container images ship without user accounts
and without a root password.

Use lxc-attach or chroot directly into the rootfs to set a root password
or create user accounts.

Since this container doesn’t come with a root password, we need to attach directly to the container. There we can specify the root password and from then on connect normally using lxc-console.

[root]# lxc-start -n buntzy
[root]# lxc-attach -n buntzy
[email protected]:/#

LXD

LXD builds on the idea of using image-based containers and functions similar to the way lxc-download works. In order to use it on Arch, install the lxd (AUR) package and start the service lxd.

lxd requires the ability to run unprivileged containers, and since Arch does not have this enabled by default, you can either build your own kernel, or run containers with the security.privileged=true command. I take the latter approach in the following examples.

Setup

To configure lxd run lxd init, an interactive prompt should follow walking you through the configuration.

[root]# lxd init
Name of the storage backend to use (dir or zfs) [default=zfs]:
Create a new ZFS pool (yes/no) [default=yes]? no
Name of the existing ZFS pool or dataset: vault/ARCHON/lxd
Would you like LXD to be available over the network (yes/no) [default=no]?
Would you like stale cached images to be updated automatically (yes/no) [default=yes]?
Would you like to create a new network bridge (yes/no) [default=yes]? no
LXD has been successfully configured.

Networking

Available networks can be viewed with the network command, listing them I see the bridge I made previously.

[root]# lxc network list
+------+----------+---------+---------+
| NAME |   TYPE   | MANAGED | USED BY |
+------+----------+---------+---------+
| br0  | bridge   | NO      | 0       |
+------+----------+---------+---------+
| eno1 | physical | NO      | 0       |
+------+----------+---------+---------+

The default configuration can be edited with lxc profile edit default.

[root]# lxc profile edit default
name: default
config: {}
description: Default LXD profile
devices: {}
usedby:

I added my network bridge.

name: default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
usedby:

Remote Images

Using LXD you can talk to remote image servers and download containers.

To list the available images run lxc image list images. A long list of available images should appear in the following form.

| eda3a4fda048 | yes | Fedora 24 amd64 (20161028_01:27) | x86_64  | 227.93MB | Oct 28, 2016 at 12:00am (UTC) |

To create a container from one of these images, specify the distro release and architecture in the form distro/release/arch.

[root]# lxc launch images:fedora/24/amd64 hat -c security.privileged=true
Creating hat
Retrieving image: 100%
Starting hat

To view the current containers run lxc list

lxc list
+------+---------+----------------------+------+------------+-----------+
| NAME |  STATE  |         IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+------+---------+----------------------+------+------------+-----------+
| hat  | RUNNING | 192.168.0.223 (eth0) |      | PERSISTENT | 0         |
+------+---------+----------------------+------+------------+-----------+

Running Commands

Individual commands can be run inside the containers by using the lxc exec command. This can be used to get a shell by running bash, or an alternate shell, as well as used to run one off commands.

[root]# lxc exec hat -- dnf install cowsay
[root]# lxc exec hat -- sh -c 'cowsay "Containers are awesome!!!"'
 ___________________________
< Containers are awesome!!! >
 ---------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

This is just a small look into what LXC can do, and if you’re interested in experimenting more with LXC and LXD I would recommend reading the great series of posts by Stéphane Graber, the LXC and LXD project leader.