Starting with Fedora 27 Atomic Host, the RPMs for Kubernetes, Flannel and Etcd are no longer included in the host’s image, but are installable instead either as system containers or via package layering.

System containers can serve as drop-in replacements for components that had been included in the Fedora Atomic image. Once installed, these components will be manageable using the same systemctl commands that apply to regular RPM-installed components. System containers are very flexible; you can easily run system container images based on CentOS, or on older (or newer) versions of Fedora on a Fedora 27 Atomic host.

Package layering makes it possible to install regular RPM packages from configured repositories. These additional layered packages are persistent across upgrades, rebases, and deploys. You must typically reboot after layering on packages, and not all packages may be installed in this way. For instance, RPMs that install content to /opt aren’t currently installable via package layering. Unlike with system containers, the packages you layer onto your host must be compatible with the version of Fedora the host is running.

If you’re running a Kubernetes cluster on Fedora Atomic Host that depends on the baked-in versions of these components, such as a cluster installed via the Ansible scripts in the kubernetes/contrib repo, you’ll need to choose one of these methods to migrate your cluster when upgrading to Fedora Atomic 27.

Migrating Kubernetes and related components using System Containers

To replace Kubernetes, Flannel, and Etcd with system containers, you would run the following commands. You could run these commands on a Fedora 26 Atomic Host, and then upgrade to 27. Upon rebooting, your components and any cluster based on them should be up and running.

System containers for master nodes

# atomic install --system --system-package=no --name kube-apiserver registry.fedoraproject.org/f27/kubernetes-apiserver

# atomic install --system --system-package=no --name kube-controller-manager registry.fedoraproject.org/f27/kubernetes-controller-manager

# atomic install --system --system-package=no --name kube-scheduler registry.fedoraproject.org/f27/kubernetes-scheduler

Note: the kube-apiserver system container provides the kubectl client.

System containers for worker nodes

# atomic install --system --system-package=no --name kubelet registry.fedoraproject.org/f27/kubernetes-kubelet

# atomic install --system --system-package=no --name kube-proxy registry.fedoraproject.org/f27/kubernetes-proxy

System container for etcd

# atomic install --system --system-package=no --storage=ostree --name etcd registry.fedoraproject.org/f27/etcd

When installed with the name etcd, the etcd system container expects to find stores etcd data in /var/lib/etcd/etcd.etcd. The etcd RPM is configured by default to store data in /var/lib/etcd/default.etcd, and the ansible scripts in kubernetes/contrib use /var/lib/etcd. On a system running etcd as configured by the kubernetes/contrib ansible scripts, you’d move your data as follows:

# systemctl stop etcd

# cp -r /var/lib/etcd/member /var/lib/etcd/etcd.etcd/

Note: the etcd container provides the etcdctl client. There will be more about the etcd container on this blog on Friday.

System container for flannel

# atomic install --system --system-package=no --name flanneld registry.fedoraproject.org/f27/flannel

Updating system containers

System container updates are independent of host updates. You can update a system container by pulling an updated version of the image, and then running the atomic containers update command.

# atomic pull registry.fedoraproject.org/f27/etcd
# atomic containers update etcd

You can then roll back to the previous system container version by running atomic containers rollback.

 # atomic containers rollback etcd

Migrating Kubernetes and related components using RPM Package Layering

During the upgrade to 27, you can opt to layer on particular packages by appending --install PACKAGE to the rpm-ostree rebase commands. Upon rebooting into 27, your components and any cluster based on them should be up and running.

To layer packages on master and etcd nodes, run the following command:

# rpm-ostree rebase fedora-atomic-27:fedora/27/x86_64/atomic-host --install kubernetes-master --install flannel --install etcd -r

To layer packages on worker nodes, run the following command:

# rpm-ostree rebase fedora-atomic-27:fedora/27/x86_64/atomic-host --install kubernetes-node --install flannel -r

Updating package layers

During regular rpm-ostree upgrades (with rpm-ostree upgrade or atomic host upgrade), your host will fetch updated package versions from your configured repositories.