flannel是CoreOS提供用于解决Dokcer集群跨主机通讯的覆盖网络工具。它的主要思路是：预先留出一个网段，每个主机使用其中一部分，然后每个容器被分配不同的ip；让所有的容器认为大家在同一个直连的网络，底层通过UDP/VxLAN等进行报文的封装和转发。

flannel项目地址：

node节点部署

创建TLS 密钥和证书

etcd 集群启用了双向TLS 认证，所以需要为flanneld 指定与etcd 集群通信的CA 和密钥。

创建flanneld 证书签名请求：

cd /usr/local/kubernetes/crtscat > flanneld-csr.json <

生成flanneld 证书和私钥：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

创建配置目录：

mkdir -p /etc/flanneld/sslmv flanneld*.pem /etc/flanneld/ssl

向etcd 写入集群Pod 网段信息

该步骤只需在第一次部署Flannel 网络时执行，后续在其他节点上部署Flanneld 时无需再写入该信息

etcdctl \  --endpoints=https://192.168.16.235:2379,https://192.168.16.236:2379,https://192.168.16.237:2379,https://192.168.16.238:2379,https://192.168.16.239:2379 \  --ca-file=/etc/kubernetes/ssl/ca.pem \  --cert-file=/etc/flanneld/ssl/flanneld.pem \  --key-file=/etc/flanneld/ssl/flanneld-key.pem \  set /kubernetes/network/config '{"Network":"'172.18.0.0/16'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'

写入的 Pod 网段(${CLUSTER_CIDR}，172.18.0.0/16) 必须与kube-controller-manager的 --cluster-cidr选项值一致

安装和配置flanneld

前往 页面下载最新版的flanneld 二进制文件

下载程序

wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gztar -xf flannel-v0.10.0-linux-amd64.tar.gz -C /usr/local/srcmkdir /usr/local/flanneld/bin -pmv /usr/local/src/{flanneld,mk-docker-opts.sh} /usr/local/flanneld/bin/

创建flanneld的systemd unit 文件

cat > /usr/lib/systemd/system/flanneld.service << EOF[Unit]Description=Flanneld overlay address etcd agentAfter=network.targetAfter=network-online.targetWants=network-online.targetAfter=etcd.serviceBefore=docker.service[Service]Type=notifyExecStart=/usr/local/flanneld/bin/flanneld \\  -etcd-cafile=/etc/kubernetes/ssl/ca.pem \\  -etcd-certfile=/etc/flanneld/ssl/flanneld.pem \\  -etcd-keyfile=/etc/flanneld/ssl/flanneld-key.pem \\  -etcd-endpoints=https://192.168.16.235:2379,https://192.168.16.236:2379,https://192.168.16.237:2379,https://192.168.16.238:2379,https://192.168.16.239:2379 \\  -etcd-prefix=/kubernetes/networkExecStartPost=/usr/local/flanneld/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/dockerRestart=on-failure[Install]WantedBy=multi-user.targetRequiredBy=docker.serviceEOF

• mk-docker-opts.sh脚本将分配给flanneld 的Pod 子网网段信息写入到/run/flannel/docker 文件中，后续docker 启动时使用这个文件中的参数值为 docker0网桥
• flanneld 使用系统缺省路由所在的接口和其他节点通信，对于有多个网络接口的机器(内网和公网)，可以用 --iface 选项值指定通信接口(上面的 systemd unit 文件没指定这个选项)

所有node节点安装

分发证书：

ansible k8s-node -a 'mkdir -p /etc/flanneld/ssl'ansible k8s-node -m copy -a 'src=/etc/flanneld/ssl/ dest=/etc/flanneld/ssl'

安装flanneld:

ansible k8s-node -a 'mkdir -p /usr/local/flanneld/bin'ansible k8s-node -m copy -a 'src=/usr/local/flanneld/bin/  dest=/usr/local/flanneld/bin mode=0755'

分发systemd unit 文件：

ansible k8s-node -m copy -a 'src=/usr/lib/systemd/system/flanneld.service  dest=/usr/lib/systemd/system/flanneld.service'

修改docker与flanneld网络配置

需要修改的

vim /usr/lib/systemd/system/docker.serviceEnvironmentFile=-/run/flannel/dockerExecStart=/usr/bin/dockerd --log-level=error $DOCKER_NETWORK_OPTIONS # 修改成这样

修改后完整的

[Unit]Description=Docker Application Container EngineDocumentation=https://docs.docker.comAfter=network-online.target firewalld.serviceWants=network-online.target[Service]Type=notify# the default is not to use systemd for cgroups because the delegate issues still# exists and systemd currently does not support the cgroup feature set required# for containers run by dockerEnvironmentFile=-/run/flannel/dockerExecStart=/usr/bin/dockerd --log-level=error $DOCKER_NETWORK_OPTIONSExecReload=/bin/kill -s HUP $MAINPID# Having non-zero Limit*s causes performance problems due to accounting overhead# in the kernel. We recommend using cgroups to do container-local accounting.LimitNOFILE=infinityLimitNPROC=infinityLimitCORE=infinity# Uncomment TasksMax if your systemd version supports it.# Only systemd 226 and above support this version.#TasksMax=infinityTimeoutStartSec=0# set delegate yes so that systemd does not reset the cgroups of docker containersDelegate=yes# kill only the docker process, not all processes in the cgroupKillMode=process# restart the docker process if it exits prematurelyRestart=on-failureStartLimitBurst=3StartLimitInterval=60s[Install]WantedBy=multi-user.target

分发到所有node节点

ansible k8s-node -m copy -a 'src=/root/docker.service dest=/usr/lib/systemd/system/docker.service'

重启docker

ansible k8s-node -m systemd -a 'daemon-reload=yes enabled=yes name=docker state=restarted'

启动并检查flanneld

启动flanneld:

ansible k8s-node -m systemd -a 'name=flanneld daemon_reload=yes enabled=yes state=started'

检查分配给各flanneld 的Pod 网段信息

# 查看集群 Pod 网段(/16)etcdctl \  --endpoints=https://192.168.16.235:2379,https://192.168.16.236:2379,https://192.168.16.237:2379,https://192.168.16.238:2379,https://192.168.16.239:2379 \  --ca-file=/etc/kubernetes/ssl/ca.pem \  --cert-file=/etc/flanneld/ssl/flanneld.pem \  --key-file=/etc/flanneld/ssl/flanneld-key.pem \  get /kubernetes/network/config{"Network":"172.18.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}# 查看已分配的 Pod 子网段列表(/24)etcdctl \  --endpoints=https://192.168.16.235:2379,https://192.168.16.236:2379,https://192.168.16.237:2379,https://192.168.16.238:2379,https://192.168.16.239:2379 \  --ca-file=/etc/kubernetes/ssl/ca.pem \  --cert-file=/etc/flanneld/ssl/flanneld.pem \  --key-file=/etc/flanneld/ssl/flanneld-key.pem \  ls /kubernetes/network/subnets/kubernetes/network/subnets/172.18.52.0-24/kubernetes/network/subnets/172.18.64.0-24/kubernetes/network/subnets/172.18.74.0-24/kubernetes/network/subnets/172.18.16.0-24/kubernetes/network/subnets/172.18.29.0-24/kubernetes/network/subnets/172.18.6.0-24/kubernetes/network/subnets/172.18.5.0-24# 查看某一 Pod 网段对应的 flanneld 进程监听的 IP 和网络参数etcdctl \  --endpoints=https://192.168.16.235:2379,https://192.168.16.236:2379,https://192.168.16.237:2379,https://192.168.16.238:2379,https://192.168.16.239:2379 \  --ca-file=/etc/kubernetes/ssl/ca.pem \  --cert-file=/etc/flanneld/ssl/flanneld.pem \  --key-file=/etc/flanneld/ssl/flanneld-key.pem \  get /kubernetes/network/subnets/172.18.5.0-24{"PublicIP":"192.168.16.243","BackendType":"vxlan","BackendData":{"VtepMAC":"aa:ea:ca:03:a6:20"}}

各个节点部署完Flanneld 后，查看已分配的Pod 子网段列表：

# ip route   // node节点default via 192.168.16.1 dev ens192 proto static metric 100 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.6.0/24 via 172.18.6.0 dev flannel.1 onlink 172.18.16.0/24 via 172.18.16.0 dev flannel.1 onlink 172.18.29.0/24 via 172.18.29.0 dev flannel.1 onlink 172.18.52.0/24 via 172.18.52.0 dev flannel.1 onlink 172.18.64.0/24 via 172.18.64.0 dev flannel.1 onlink 172.18.74.0/24 via 172.18.74.0 dev flannel.1 onlink 192.168.16.0/24 dev ens192 proto kernel scope link src 192.168.16.243 metric 100

再重启docker查看dockerr0与flannel.1网段是否一致

ansible k8s-node -m systemd -a 'daemon-reload=yes name=docker state=restarted'~]# ifconfigdocker0: flags=4099
    
       mtu 1500        inet 172.18.6.1  netmask 255.255.255.0  broadcast 172.18.6.255        ether 02:42:f0:d9:93:d5  txqueuelen 0  (Ethernet)        RX packets 0  bytes 0 (0.0 B)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 0  bytes 0 (0.0 B)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0ens192: flags=4163
     
        mtu 1500        inet 192.168.16.244  netmask 255.255.255.0  broadcast 192.168.16.255        inet6 fe80::822:4c72:993b:a056  prefixlen 64  scopeid 0x20
              inet6 fe80::6450:5694:d6c3:b30e  prefixlen 64  scopeid 0x20
              inet6 fe80::d92e:5d8e:953a:c790  prefixlen 64  scopeid 0x20
              ether 00:50:56:98:66:11  txqueuelen 1000  (Ethernet)        RX packets 39810  bytes 21589285 (20.5 MiB)        RX errors 0  dropped 114  overruns 0  frame 0        TX packets 14350  bytes 1409987 (1.3 MiB)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0flannel.1: flags=4163
      
         mtu 1450        inet 172.18.6.0  netmask 255.255.255.255  broadcast 0.0.0.0        inet6 fe80::108f:96ff:fe7c:c88a  prefixlen 64  scopeid 0x20
               ether 12:8f:96:7c:c8:8a  txqueuelen 0  (Ethernet)        RX packets 0  bytes 0 (0.0 B)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 0  bytes 0 (0.0 B)        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0lo: flags=73
       
          mtu 65536        inet 127.0.0.1  netmask 255.0.0.0        inet6 ::1  prefixlen 128  scopeid 0x10
        
                 loop  txqueuelen 1000  (Local Loopback)        RX packets 22  bytes 2080 (2.0 KiB)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 22  bytes 2080 (2.0 KiB)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0