Flannel and docker don't start

Question

I'm trying to set up a kubernetes cluster on 2 nodes , centos 7.1 using this guide. However when I attempt to start the services on the minion like so:

for SERVICES in kube-proxy kubelet docker flanneld; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES 
done

I get the following error:

-- Logs begin at Wed 2015-12-23 13:00:41 UTC, end at Wed 2015-12-23 16:03:54 UTC. --
Dec 23 16:03:47 sc-test2 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Dec 23 16:03:47 sc-test2 systemd[1]: Failed to start Docker Storage Setup.
-- Subject: Unit docker-storage-setup.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker-storage-setup.service has failed.
-- 
-- The result is failed.
Dec 23 16:03:47 sc-test2 systemd[1]: Unit docker-storage-setup.service entered failed state.
Dec 23 16:03:48 sc-test2 flanneld[36477]: E1223 16:03:48.187350 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:49 sc-test2 flanneld[36477]: E1223 16:03:49.189860 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:50 sc-test2 flanneld[36477]: E1223 16:03:50.192894 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:51 sc-test2 flanneld[36477]: E1223 16:03:51.194940 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:52 sc-test2 flanneld[36477]: E1223 16:03:52.197222 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:53 sc-test2 flanneld[36477]: E1223 16:03:53.199248 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:54 sc-test2 flanneld[36477]: E1223 16:03:54.201160 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)

I'm sure I set the key on the master with : etcdctl mk /coreos.com/network/config '{"Network":"172.17.0.0/16"}'

By far installation seems to be the hardest bit on using kubernetes :(

Solution

Today's christmas but I spent the whole day trying to get this to work :) This is what I did:

#1 FLANNEL

As mentioned I'd set the flannel etcd key on the master with:

etcdctl mk /coreos.com/network/config '{"Network":"172.17.0.0/16"}'

but I got this error when trying to start flannel on the minion:

Failed to retrieve network config: 100: Key not found (/atomic.io)

So I edited the /etc/sysconfig/flanneld file on the minion from:

# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://master:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/coreos.com/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

to:

# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://master:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/atomic.io/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

i.e. changed the FLANNEL_ETCD key.

After this systemctl start flanneld worked.

#2 DOCKER

I didn't find a way to make the version installed as a dependency by kubernetes work so I uninstalled it and following the docker docs for Centos installed docker-engine and manually created a docker.service file for systemctl.

cd /usr/lib/systemd/system

and the contents of the docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
Requires=flanneld.service
After=flanneld.service

[Service]
EnvironmentFile=/etc/sysconfig/flanneld
ExecStart=/usr/bin/docker daemon -H fd:// --bip=${FLANNEL_SUBNET}
Restart=on-failure
RestartSec=5


[Install]
WantedBy=multi-user.target

then start and enable the daemon with systemctl as well as query the status.

systemctl restart docker
systemctl enable docker
systemctl status docker

Comments

[Chris Westin]

I've been working on the same thing. The instructions from severalnines.com seem to depend on more storage; I was trying this out in centos guests in virtual box, and the docker startup failed because docker-storage-setup failed because it was unable to create a logical volume to hold docker images. Since I was working with virtual box, I just created larger (12GB) centos guests, and that worked. Then I was stuck on the flannel problem. It looks like their setup for the flanneld configuration was wrong. Your change here fixed it for me.

[Chris Westin]

Nope, spoke too soon. 12GB VMs wasn't large enough. Any ideas how big I should make it?

[Jonathan]

I don't think storage is an issue, I was able to get kubernetes working by uninstalling the existing docker package (installed via yum install docker) and installing docker-engine as per the docker docs. Then swapping out the service file values with the above.

[Jonathan]

I did find that adding the argument --storage-opt="dm.blocksize=64k" to the docker daemon command fixed it. From github.com/docker/docker/issues/6325