Recently, I’ve been experimenting with running Kubernetes in system containers, and those tests led me to wonder whether I could use system containers as a means of working around the issues I’ve experienced installing kubeadm, the simple-to-use tool for bootstrapping kubernetes clusters, on an atomic host.

On a regular CentOS or Fedora host, using kubeadm is a matter of installing rpms for the kubelet, kubectl, kubeadm itself, and for a set of Kubernetes networking tools, kubernetes-cni. On an atomic host, rpm-ostree package layering allows for installing rpms, but if existing kube rpms are already part the atomic host image, as they are for Fedora Atomic Host, you won’t be able to install the prescribed upstream kube versions. And even on a host without built-in kubernetes, like CentOS Atomic Continuous, rpm-ostree won’t abide rpm content stored in /opt.

I managed to make a kubadm system container that uses the same tmpfiles.d trick I used to link kubectl and etcdctl from the container into the the host’s /usr/local/bin to make the kubeadm tool available from the host.

For now, this works best on CentOS Atomic Host, with either the downstream or continuous branches.

With Fedora-based Atomic Hosts, there are two issues. First, Fedora 25 currently ships with an older version of runc. Soon after the updated version gets enough karma, this kubeadm system container will work with Fedora 25… with SELinux in permissive mode. I’m trying to track down why this is necessary with Fedora but not with CentOS.

To try it for yourself, start out by installing this kubeadm system container, starting the kubelet service, and kicking off kubeadm init:

# atomic install --system --name kubelet docker.io/jasonbrooks/kubeadm
# systemctl start kubelet
# kubeadm init --skip-preflight-checks --pod-network-cidr=10.254.0.0/16

Once the kubeadm init completes, you’ll need to follow the directions on screen to configure kubectl:

$ sudo cp /etc/kubernetes/admin.conf $HOME/
$ sudo chown $(id -u):$(id -g) $HOME/admin.conf
$ export KUBECONFIG=$HOME/admin.conf

Next we need to configure a networking plugin, along with RBAC rules. I’ve modified the the kube-flannel.yml file below to work with SELinux:

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
$ kubectl apply -f https://raw.githubusercontent.com/jasonbrooks/flannel/support-selinux-kube/Documentation/kube-flannel.yml

Assuming you want your master to do double-duty as a node for an all in one setup, run the following command:

$ kubectl taint nodes --all node-role.kubernetes.io/master-

Once all the pods in the kube-system namespace are up and running, which you can check with kubectl get pods -n kube-system, you can test out your cluster. I like to use this guestbook-go app:

$ kubectl apply -f https://gist.githubusercontent.com/jasonbrooks/c2cab426c315ec26266ddd2c78aa4b60/raw/f9199348a04d5d65b60ce974992f5fd589b3e1de/guestbookgo.yaml

$ kubectl get svc guestbook
NAME        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
guestbook   10.106.170.7   <nodes>       3000:32534/TCP   6m

Once all the pods for the guestbook app are are up and running, which you can check with kubectl get pods, you can test out the app by visiting the IP address of your host at the NodePort listed above (in this case 32534).

Running atomic containers list will show that most of the containers running are run via docker, with the exception of the kubelet container, run via runc:

$ atomic containers list
   CONTAINER ID IMAGE                COMMAND              CREATED          STATE     BACKEND    RUNTIME   
   81ab52fe1a6a docker.io/redis@sha2 docker-entrypoint.sh 2017-05-10 14:21 running   docker     docker    
   cf2561dbd708 docker.io/kubernetes /bin/sh -c /run.sh   2017-05-10 14:21 running   docker     docker    
   9cbbea14fc20 gcr.io/google_contai ./guestbook          2017-05-10 14:21 running   docker     docker    
   b8b91e19260e gcr.io/google_contai /pause               2017-05-10 14:21 running   docker     docker    
   e68cf8f192ab gcr.io/google_contai /pause               2017-05-10 14:21 running   docker     docker    
   f619b1092cfc gcr.io/google_contai /pause               2017-05-10 14:21 running   docker     docker    
   5d9b7966a292 gcr.io/google_contai /sidecar --v=2 --log 2017-05-10 12:27 running   docker     docker    
   f74a7d3da1bd gcr.io/google_contai /dnsmasq-nanny -v=2  2017-05-10 12:27 running   docker     docker    
   b6afd95588a9 gcr.io/google_contai /kube-dns --domain=c 2017-05-10 12:27 running   docker     docker    
   96a68d5a53d3 gcr.io/google_contai /pause               2017-05-10 12:27 running   docker     docker    
   75f981a02222 quay.io/coreos/flann /opt/bin/flanneld -- 2017-05-10 12:27 running   docker     docker    
   2c7d8e7af5c2 gcr.io/google_contai /usr/local/bin/kube- 2017-05-10 12:27 running   docker     docker    
   180eeb2e9a26 quay.io/coreos/flann /bin/sh -c 'set -e - 2017-05-10 12:26 running   docker     docker    
   1c35ebad2642 gcr.io/google_contai /pause               2017-05-10 12:26 running   docker     docker    
   f09b2e9f60ad gcr.io/google_contai /pause               2017-05-10 12:24 running   docker     docker    
   b980f99adec8 gcr.io/google_contai etcd --listen-client 2017-05-10 12:24 running   docker     docker    
   8d58bd57c632 gcr.io/google_contai kube-apiserver --kub 2017-05-10 12:24 running   docker     docker    
   2ffdb4612a64 gcr.io/google_contai kube-scheduler --add 2017-05-10 12:24 running   docker     docker    
   2cabf3d7a924 gcr.io/google_contai kube-controller-mana 2017-05-10 12:23 running   docker     docker    
   a5a8d203f7e7 gcr.io/google_contai /pause               2017-05-10 12:23 running   docker     docker    
   d25bc2c3808a gcr.io/google_contai /pause               2017-05-10 12:23 running   docker     docker    
   d3191cdaef80 gcr.io/google_contai /pause               2017-05-10 12:23 running   docker     docker    
   ef0d198666bc gcr.io/google_contai /pause               2017-05-10 12:23 running   docker     docker    
   kubelet      docker.io/jasonbrook /usr/bin/launch.sh   2017-05-10 12:22 running   ostree     runc

kubeadm is still considered beta for now, and there aren’t official Fedora or CentOS packages , so the system container I made pulls that packages directly from the upstream. Once kubeadm is considered GA, I expect to see it available in Fedora-packaged form.