安装文档
Docker 需要安装 1.12.x ~ 1.13.1 之间的版本新版本不支持 Kubenetes Cluster 构建,文档
不使用 kube-apiserver 的 HTTP 端口,启用 Kubernetes 集群相关组件的 TLS 通信和双向认证,下面将使用工具生成各组件TLS的证书和私钥。
{ "signing": { "default": { "expiry": "87600h" }, "profiles": { "frognew": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } |
创建 CA 证书签名请求配置
{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "cloudnative" } ] } |
CN
即 Common Name,kube-apiserver从证书中提取该字段作为请求的用户名O
即 Organization,kube-apiserver 从证书中提取该字段作为请求用户所属的组下面使用cfss生成CA证书和私钥
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca |
创建 kube-apiserver 证书签名请求配置
{ "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.166.1.12", "192.166.1.2", "192.166.1.13", "10.96.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "cloudnative" } ] } |
注意:上面配置 hosts 字段中制定授权使用该证书的 IP 和域名列表,因为现在要生成的证书需要被 Kubernetes Master 集群各个节点使用,所以这里指定了各个节点的 IP 和 hostname。 同时还要指定集群内部 kube-apiserver 的多个域名和IP地址 10.96.0.1 (后边 kube-apiserver-service-cluster-ip-range=10.96.0.0/12 参数的指定网段的第一个 IP ),这里如果有 VIP 的,也是需要填写的。 |
下面生成 kube-apiserver 的证书和私钥:
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew apiserver-csr.json | cfssljson -bare apiserver $ ls apiserver* apiserver.csr apiserver-csr.json apiserver-key.pem apiserver.pem |
创建 admin 证书的签名请求配置
{ "CN": "kubernetes-admin", "hosts": [ "192.166.1.12", "192.166.1.2", "192.166.1.13" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "cloudnative" } ] } |
kube-apiserver 将提取 CN
作为客户端的用户名,这里是 kubernetes-admin
,将提取 O
作为用户所属的组,这里是 system:master
。 kube-apiserver 预定义了一些 RBAC 使用的ClusterRoleBindings,例如 cluster-admin 将组 system:masters与 ClusterRole cluster-admin 绑定,而 cluster-admin 拥有访问 kube-apiserver 的所有权限,因此 kubernetes-admin
这个用户将作为集群的超级管理员。
下面生成 kubernetes-admin
的证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew admin-csr.json | cfssljson -bare admin $ ls admin* admin.csr admin-csr.json admin-key.pem admin.pem |
下面创建 kube-controller-manager 客户端访问 ApiServer 所需的证书和私钥。 controller-manager 证书的签名请求配置
{ "CN": "system:kube-controller-manager", "hosts": [ "192.168.1.12", "192.168.1.2", "192.168.1.13" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-controller-manager", "OU": "cloudnative" } ] } |
kube-apiserver 将提取 CN
作为客户端的用户名,这里是 system:kube-controller-manager
。 kube-apiserver 预定义的 RBAC 使用的 ClusterRoleBindings system:kube-controller-manager
将用户 system:kube-controller-manager
与 ClusterRole system:kube-controller-manager
绑定。
下面生成证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew controller-manager-csr.json | cfssljson -bare controller-manager $ ls controller-manager* controller-manager.csr controller-manager-csr.json controller-manager-key.pem controller-manager.pem |
下面创建kube-scheduler客户端访问ApiServer所需的证书和私钥。 scheduler证书的签名请求配置
{ "CN": "system:kube-scheduler", "hosts": [ "192.166.1.12", "192.166.1.2", "192.166.1.13" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-scheduler", "OU": "cloudnative" } ] } |
kube-scheduler 将提取 CN
作为客户端的用户名,这里是 system:kube-scheduler
。 kube-apiserver 预定义的 RBAC 使用的 ClusterRoleBindings system:kube-scheduler
将用户system:kube-scheduler
与 ClusterRole system:kube-scheduler
绑定。
下面生成证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew scheduler-csr.json | cfssljson -bare scheduler $ ls scheduler* scheduler.csr scheduler-csr.json scheduler-key.pem scheduler.pem |
部署的 Master 节点集群由 Node1, Node2, Node3三个节点组成,每个节点上部署 kube-apiserver,kube-controller-manager,kube-scheduler 三个核心组件。 kube-apiserver 的3个实例同时提供服务,在其前端部署一个高可用的负载均衡器作为 kube-apiserve r的地址。 kube-controller-manager和 kube-scheduler 也是各自3个实例,在同一时刻只能有1个实例工作,这个实例通过选举产生。
将前面生成的 ca.pem, apiserver-key.pem, apiserver.pem, admin.pem, admin-key.pem, controller-manager.pem, controller-manager-key.pem, scheduler-key.pem, scheduler.pem 拷贝到各个节点的 /etc/kubernetes/pki 目录下 |
创建目录,并拷贝证书到对应 Node
$ mkdir -p /etc/kubernetes/pki $ scp ./{ca-key.pem,ca.pem,apiserver-key.pem,apiserver.pem,admin.pem,admin-key.pem,controller-manager.pem,controller-manager-key.pem,scheduler-key.pem,scheduler.pem} node02:/etc/kubernetes/pki |
下载 Kubernetes 二进制包,下载地址:
将 Kubernetes 二进制包解压后 kubernetes/server/bin 中的 kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet 拷贝到各节点的 /usr/local/bin 目录中: |
解压文件并拷贝到可执行目录
$ tar zxf kubernetes-server-linux-amd64.tar.gz && cp kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/ |
创建 kube-apiserver 的 systemd unit 文件,注意替换 INTERNAL_IP
变量的值
$ mkdir -p /var/log/kubernetes $ export INTERNAL_IP=192.166.1.12 $ cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF [Unit] Description=kube-apiserver After=network.target After=etcd.service [Service] EnvironmentFile=-/etc/kubernetes/apiserver ExecStart=/usr/local/bin/kube-apiserver \ --logtostderr=true \ --v=0 \ --advertise-address=${INTERNAL_IP} \ --bind-address=${INTERNAL_IP} \ --secure-port=6443 \ --insecure-port=0 \ --allow-privileged=true \ --etcd-servers=https://192.166.1.12:2379,https://192.166.1.2:2379,https://192.166.1.13:2379 \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --storage-backend=etcd3 \ --service-node-port-range=79-39999 \ --service-cluster-ip-range=10.96.0.0/12 \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --service-account-key-file=/etc/kubernetes/pki/ca-key.pem \ --experimental-bootstrap-token-auth=true \ --apiserver-count=3 \ --enable-swagger-ui=true \ --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds \ --authorization-mode=RBAC \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kubernetes/audit.log Restart=on-failure Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF |
生成好的配置如下:
[Unit] Description=kube-apiserver After=network.target After=etcd.service [Service] EnvironmentFile=-/etc/kubernetes/apiserver ExecStart=/usr/local/bin/kube-apiserver --logtostderr=true --v=0 --advertise-address=192.166.1.12 --bind-address=192.166.1.12 --secure-port=6443 --insecure-port=0 --allow-privileged=true --etcd-servers=https://192.166.1.12:2379,https://192.166.1.2:2379,https://192.166.1.13:2379 --etcd-cafile=/etc/etcd/ssl/ca.pem --etcd-certfile=/etc/etcd/ssl/etcd.pem --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem --storage-backend=etcd3 --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.pem --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem --client-ca-file=/etc/kubernetes/pki/ca.pem --service-account-key-file=/etc/kubernetes/pki/ca-key.pem --experimental-bootstrap-token-auth=true --apiserver-count=3 --enable-swagger-ui=true --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds --authorization-mode=RBAC --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/kubernetes/audit.log Restart=on-failure Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target |
--insecure-port:
禁用了不安全的 http 端口--secure-port:
指定 https 安全端口,kube-scheduler、kube-controller-manager、kubelet、kube-proxy、kubectl 等组件都将使用安全端口与 ApiServer 通信(实际上会由我们在前端部署的负载均衡器代理)。--authorization-mode=RBAC:
表示在安全端口启用 RBAC 授权模式,在授权过程会拒绝会授权的请求。kube-scheduler、kube-controller-manager、kubelet、kube-proxy、kubectl等组件都使用各自证书或 kubeconfig 指定相关的 User、Group 来通过 RBAC 授权。--admission-control:
为准入机制,这里配置了NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds
--service-cluster-ip-range:
指定 Service Cluster IP 的地址段,注意该地址段是 Kubernetes Service 使用的,是虚拟 IP,从外部不能路由可达。启动 kube-apiserve
$ systemctl daemon-reload $ systemctl enable kube-apiserver $ systemctl start kube-apiserver $ systemctl status kube-apiserver |
已经部署了 kube-apiserver,并且前面已经 kubernetes-admin 的证书和私钥,我们将使用这个用户访问 ApiServer。
$ kubectl --server=https://192.166.1.12:6443 \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --client-certificate=/etc/kubernetes/pki/admin.pem \ --client-key=/etc/kubernetes/pki/admin-key.pem \ get componentstatuses NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused etcd-1 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} |
上面使用 kubectl 命令打印出了 Kubernetes 核心组件的状态,因为还没有部署 kube-scheduler 和 controller-manager,所以这两个组件当前是不健康的。
前面面使用 kubectl 时需要指定 ApiServer 的地址以及客户端的证书,用起来比较繁琐。 接下来我们创建 kubernetes-admin 的 kubeconfig 文件 admin.conf。
$ cd /etc/kubernetes $ export KUBE_APISERVER="https://192.166.1.12:6443" |
配置 cluster
$ kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=admin.conf |
配置 credentials
$ kubectl config set-credentials kubernetes-admin \ --client-certificate=/etc/kubernetes/pki/admin.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/pki/admin-key.pem \ --kubeconfig=admin.conf |
配置 context
$ kubectl config set-context kubernetes-admin@kubernetes \ --cluster=kubernetes \ --user=kubernetes-admin \ --kubeconfig=admin.conf |
配置 default context
$ kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=admin.conf |
为了使用 kubectl 访问 apiserver 使用admin.conf,将其拷贝到 $HOME/.kube 中并重命名成 config:
$ cp /etc/kubernetes/admin.conf ~/.kube/config |
尝试直接使用 kubectl 命令访问 apiserver,这样就不用指定 CA 方式验证了
$ kubectl get cs NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused etcd-2 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} |
注意:已经把 kubernetes-admin 用户的相关信息和证书保存到了admin.conf 中,所以可以删除 /etc/kubernetes/pki 中的 admin.pem 和 admin-key.pem
|
高可用是需要对其 api 提供的 6443 端口进行负载策略,使用 Haproxy 结合 Corosync + Pacemaker 方式构建。
已经创建了 controller-manager.pem 和 controller-manage-key.pem,下面生成 controller-manager 的 kubeconfig 文件
$ cd /etc/kubernetes $ export KUBE_APISERVER="https://192.166.1.222:6443" |
配置 cluster
$ kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=controller-manager.conf |
配置 credentials
$ kubectl config set-credentials system:kube-controller-manager \ --client-certificate=/etc/kubernetes/pki/controller-manager.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/pki/controller-manager-key.pem \ --kubeconfig=controller-manager.conf |
配置 context
$ kubectl config set-context system:kube-controller-manager@kubernetes \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=controller-manager.conf |
配置 default context
$ kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=controller-manager.conf |
controller-manager.conf 文件生成后将这个文件分发到各个 Master 节点的 /etc/kubernetes 目录下。
$ scp controller-manager.conf node02:/etc/kubernetes ... |
创建 kube-controller-manager 的 systemd unit 文件
$ export KUBE_APISERVER="https://192.166.1.222:6443" $ cat > /usr/lib/systemd/system/kube-controller-manager.service <<EOF [Unit] Description=kube-controller-manager After=network.target After=kube-apiserver.service [Service] EnvironmentFile=-/etc/kubernetes/controller-manager ExecStart=/usr/local/bin/kube-controller-manager \ --logtostderr=true \ --v=0 \ --master=${KUBE_APISERVER} \ --kubeconfig=/etc/kubernetes/controller-manager.conf \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem \ --root-ca-file=/etc/kubernetes/pki/ca.pem \ --insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers \ --use-service-account-credentials=true \ --service-cluster-ip-range=10.96.0.0/12 \ --cluster-cidr=10.244.0.0/16 \ --allocate-node-cidrs=true \ --leader-elect=true \ --controllers=*,bootstrapsigner,tokencleaner Restart=on-failure Type=simple LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF |
配置文件展示
[Unit] Description=kube-controller-manager After=network.target After=kube-apiserver.service [Service] EnvironmentFile=-/etc/kubernetes/controller-manager ExecStart=/usr/local/bin/kube-controller-manager --logtostderr=true --v=0 --master=https://192.166.1.222:6443 --kubeconfig=/etc/kubernetes/controller-manager.conf --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem --root-ca-file=/etc/kubernetes/pki/ca.pem --insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers --use-service-account-credentials=true --service-cluster-ip-range=10.96.0.0/12 --cluster-cidr=10.244.0.0/16 --allocate-node-cidrs=true --leader-elect=true --controllers=*,bootstrapsigner,tokencleaner Restart=on-failure Type=simple LimitNOFILE=65536 [Install] |
--service-cluster-ip-range
和 kube-api-server
中指定的参数值一致在各节点上启动:
$ systemctl daemon-reload $ systemctl enable kube-controller-manager $ systemctl start kube-controller-manager $ systemctl status kube-controller-manager |
在 node1,node2,node3 上完成部署和启动后,查看一下状态:
$ kubectl get cs NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager Healthy ok etcd-1 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} |
三个 Master 节点上的 kube-controller-manager 部署完成,通过选举出一个 leader 工作。 分别在三个节点查看状态
$ systemctl status -l kube-controller-manager |
node01 机器服务状态
如果看到此状况,请把 apiserver-csr.json 定义的证书授权 IP 内容里把 VIP 填写进去,重新适应 cfssl 生成证书后放入各个 master 节点的 /etc/kubernetes/pki 上,并重启 kube-apiserver.service 服务方可正常。如图
上图中是没有把配置文件放入 /etc/kubernetes 中,下图为正确启动 log
其他机器状态
只需要保证一台服务开启即可。
下面生成kube-scheduler的kubeconfig文件
$ cd /etc/kubernetes $ export KUBE_APISERVER="https://192.166.1.222:6443" |
配置 cluster
$ kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=scheduler.conf |
配置 credentials
$ kubectl config set-credentials system:kube-scheduler \ --client-certificate=/etc/kubernetes/pki/scheduler.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/pki/scheduler-key.pem \ --kubeconfig=scheduler.conf |
配置 context
$ kubectl config set-context system:kube-scheduler@kubernetes \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=scheduler.conf |
配置 default context
$ kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=scheduler.conf |
scheduler.conf 文件生成后将这个文件分发到各个 Master 节点的 /etc/kubernetes 目录下。
$ scp scheduler.conf node02:/etc/kubernetes/ ... |
创建 kube-scheduler 的 systemd unit 文件
$ export KUBE_APISERVER="https://192.166.1.222:6443" $ cat > /usr/lib/systemd/system/kube-scheduler.service <<EOF [Unit] Description=kube-scheduler After=network.target After=kube-apiserver.service [Service] EnvironmentFile=-/etc/kubernetes/scheduler ExecStart=/usr/local/bin/kube-scheduler \ --logtostderr=true \ --v=0 \ --master=${KUBE_APISERVER} \ --kubeconfig=/etc/kubernetes/scheduler.conf \ --leader-elect=true Restart=on-failure Type=simple LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF |
配置文件展示
[Unit] Description=kube-scheduler After=network.target After=kube-apiserver.service [Service] EnvironmentFile=-/etc/kubernetes/scheduler ExecStart=/usr/local/bin/kube-scheduler --logtostderr=true --v=0 --master=https://192.166.1.222:6443 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true Restart=on-failure Type=simple LimitNOFILE=65536 [Install] WantedBy=multi-user.target |
在各节点上启动
$ systemctl daemon-reload $ systemctl enable kube-scheduler $ systemctl start kube-scheduler $ systemctl status kube-scheduler |
三个 Master 节点上的 kube-schedule r部署完成,通过选举出一个 leader 工作。
查看 Kubernetes Master 集群各个核心组件的状态全部正常。
$ kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-2 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} |
Kubernetes的一个Node节点上需要运行如下组件:
下面我们以在node1上为例部署这些组件:
$ wget https://github.com/containernetworking/cni/releases/download/v0.5.2/cni-amd64-v0.5.2.tgz $ mkdir -p /opt/cni/bin $ tar -zxvf cni-amd64-v0.5.2.tgz -C /opt/cni/bin $ ls /opt/cni/bin/ bridge cnitool dhcp flannel host-local ipvlan loopback macvlan noop ptp tuning |
首先确认将 kubelet 的二进制文件拷贝到 /usr/local/bin 下。
$ cp kubernetes/server/bin/kubelet /usr/local/bin |
创建 kubelet 的工作目录:
$ mkdir -p /var/lib/kubelet |
安装依赖包
$ yum install ebtables socat util-linux conntrack-tools |
接下来创建访问ApiServer的证书和私钥
{ "CN": "system:node:node2", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:nodes", "OU": "cloudnative" } ] } |
system:node:<node-name>
生成证书和私钥:
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew kubelet-csr.json | cfssljson -bare kubelet $ ls kubelet* kubelet.csr kubelet-csr.json kubelet-key.pem kubelet.pem |
把生成的秘钥文件拷贝到 /etc/kubernetes/pki/ 工作目录下
$ scp ./{kubelet.csr,kubelet-key.pem,kubelet.pem} node02:/etc/kubernetes/pki/ ... |
在 node2 节点生成 kubeconfig 文件
$ cd /etc/kubernetes $ export KUBE_APISERVER="https://192.166.1.222:6443" |
配置 cluster
$ kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kubelet.conf |
配置 credentials
$ kubectl config set-credentials system:node:node2 \ --client-certificate=/etc/kubernetes/pki/kubelet.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/pki/kubelet-key.pem \ --kubeconfig=kubelet.conf |
配置 context
$ kubectl config set-context system:node:node2@kubernetes \ --cluster=kubernetes \ --user=system:node:node2 \ --kubeconfig=kubelet.conf |
配置 default context
$ kubectl config use-context system:node:node2@kubernetes --kubeconfig=kubelet.conf |
创建 kubelet 的 systemd unit service 文件,注意替换 NodeIP
变量:
$ export KUBE_APISERVER="https://192.166.1.222:6443" $ export NodeIP="192.166.1.2" $ cat > /usr/lib/systemd/system/kubelet.service <<EOF [Unit] Description=kubelet After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet EnvironmentFile=-/etc/kubernetes/kubelet ExecStart=/usr/local/bin/kubelet \ --logtostderr=true \ --v=0 \ --address=${NodeIP} \ --api-servers==${KUBE_APISERVER} \ --cluster-dns=10.96.0.10 \ --cluster-domain=cluster.local \ --kubeconfig=/etc/kubernetes/kubelet.conf \ --require-kubeconfig=true \ --pod-manifest-path=/etc/kubernetes/manifests \ --allow-privileged=true \ --authorization-mode=Webhook \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --network-plugin=cni \ --cni-conf-dir=/etc/cni/net.d \ --cni-bin-dir=/opt/cni/bin \ --cgroup-driver=systemd Restart=on-failure [Install] WantedBy=multi-user.target EOF |
--pod-manifest-path=/etc/kubernetes/manifests
指定了静态Pod定义的目录。可以提前创建好这个目录 。
$ mkdir -p /etc/kubernetes/manifests |
配置文件参考
[Unit] Description=kubelet After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet EnvironmentFile=-/etc/kubernetes/kubelet ExecStart=/usr/local/bin/kubelet --logtostderr=true --v=0 --address=192.166.1.2 --api-servers==https://192.166.1.222:6443 --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.pem --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cgroup-driver=systemd Restart=on-failure [Install] WantedBy=multi-user.target |
启动 kubelet
$ systemctl daemon-reload $ systemctl enable kubelet $ systemctl start kubelet $ systemctl status kubelet |
检查 Node 是否被添加
$ kubectl get node NAME STATUS AGE VERSION node02 NotReady 2m v1.6.2 |
因为还未添加 Proxy 组件和 DNS 组件所以是 Notready 状态。
将 kuber-proxy 的二进制文件拷贝到 /usr/local/bin 下。 创建 kubelet 的工作目录
$ mkdir -p /var/lib/kube-proxy |
接下来创建访问ApiServer的证书和私钥
{ "CN": "system:kube-proxy", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-proxy", "OU": "cloudnative" } ] } |
生成证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew kube-proxy-csr.json | cfssljson -bare kube-proxy $ ls kube-proxy* kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem |
把生成的秘钥文件拷贝到 /etc/kubernetes/pki/ 工作目录下
$ scp ./{kube-proxy.csr,kube-proxy-key.pem,kube-proxy.pem} node02:/etc/kubernetes/pki/ ... |
在 node2 节点生成 kubeconfig 文件
$ cd /etc/kubernetes $ export KUBE_APISERVER="https://192.166.1.222:6443" |
配置 cluster
$ kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.conf |
配置 credentials
$ kubectl config set-credentials system:kube-proxy \ --client-certificate=/etc/kubernetes/pki/kube-proxy.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/pki/kube-proxy-key.pem \ --kubeconfig=kube-proxy.conf |
配置 context
$ kubectl config set-context system:kube-proxy@kubernetes \ --cluster=kubernetes \ --user=system:kube-proxy \ --kubeconfig=kube-proxy.conf |
配置 default context
$ kubectl config use-context system:kube-proxy@kubernetes --kubeconfig=kube-proxy.conf |
创建 kube-proxy 的 systemd unit service 文件,注意替换 NodeIP
变量:
$ export KUBE_APISERVER="https://192.166.1.222:6443" $ export NodeIP="192.166.1.22" $ cat > /usr/lib/systemd/system/kube-proxy.service <<EOF [Unit] Description=kube-proxy After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy EnvironmentFile=-/etc/kubernetes/kube-proxy ExecStart=/usr/local/bin/kube-proxy \ --logtostderr=true \ --v=0 \ --bind-address=${NodeIP} \ --kubeconfig=/etc/kubernetes/kube-proxy.conf \ --cluster-cidr=10.244.0.0/16 Restart=on-failure [Install] WantedBy=multi-user.target EOF |
启动 kubelet-proxy
$ systemctl daemon-reload $ systemctl enable kube-proxy $ systemctl start kube-proxy $ systemctl status -l kube-proxy |
flannel 以 DaemonSet 的形式运行在 Kubernetes 集群中。 由于我们的 etcd 集群启用了 TLS 认证,为了从 flannel 容器中能访问 etcd,我们先把 etcd 的 TLS 证书信息保存到 Kubernetes 的Secret 中。
$ kubectl create secret generic etcd-tls-secret --from-file=/etc/etcd/ssl/etcd.pem --from-file=/etc/etcd/ssl/etcd-key.pem --from-file=/etc/etcd/ssl/ca.pem -n kube-system $ kubectl describe secret etcd-tls-secret -n kube-system Name: etcd-tls-secret Namespace: kube-system Labels: <none> Annotations: <none> Type: Opaque Data ==== ca.pem: 1322 bytes etcd-key.pem: 1675 bytes etcd.pem: 1476 bytes |
下载启动 YAML
$ mkdir -p ~/k8s/flannel $ cd ~/k8s/flannel $ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml $ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml |
修改配置
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: ...... spec: template: metadata: ...... spec: ...... containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.7.1-amd64 command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "-etcd-endpoints=https://192.166.1.12:2379,https://192.166.1.2:2379,https://192.166.1.13:2379", "-etcd-cafile=/etc/etcd/ssl/ca.pem", "--etcd-certfile=/etc/etcd/ssl/etcd.pem", "-etcd-keyfile=/etcd/etcd/ssl/etcd-key.pem", "--iface=etn0" ] securityContext: privileged: true ...... volumeMounts: ...... - name: etcd-tls-secret readOnly: true mountPath: /etc/etcd/ssl/ ...... volumes: ...... - name: etcd-tls-secret secret: secretName: etcd-tls-secret |
flanneld 的启动参数中加入以下参数:
-etcd-endpoints:
配置etcd集群的访问地址-etcd-cafile:
配置etcd的CA证书,/etc/etcd/ssl/ca.pem从etcd-tls-secret这个Secret挂载--etcd-certfile:
配置etcd的公钥证书,/etc/etcd/ssl/etcd.pem从etcd-tls-secret这个Secret挂载--etcd-keyfile:
配置etcd的私钥,/etc/etcd/ssl/etcd-key.pem从etcd-tls-secret这个Secret挂载--iface:
当 Node 节点有多个网卡时用于指明具体的网卡名称,最需要主要的是此选项,有很多操作系统的网卡名称并不是 eth0,请注意修改!完整的配置如下:
--- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flannel data: cni-conf.json: | { "name": "cbr0", "type": "flannel", "delegate": { "isDefaultGateway": true } } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } --- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannel spec: template: metadata: labels: tier: node app: flannel spec: hostNetwork: true nodeSelector: beta.kubernetes.io/arch: amd64 tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule serviceAccountName: flannel containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.7.1-amd64 command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "-etcd-endpoints=https://192.166.1.12:2379,https://192.166.1.2:2379,https://192.166.1.13:2379", "-etcd-cafile=/etc/etcd/ssl/ca.pem", "--etcd-certfile=/etc/etcd/ssl/etcd.pem", "-etcd-keyfile=/etcd/etcd/ssl/etcd-key.pem", "--iface=etn0" ] securityContext: privileged: true env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run - name: flannel-cfg mountPath: /etc/kube-flannel/ - name: etcd-tls-secret readOnly: true mountPath: /etc/etcd/ssl/ - name: install-cni image: quay.io/coreos/flannel:v0.7.1-amd64 command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ] volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg - name: etcd-tls-secret secret: secretName: etcd-tls-secret |
下面部署flannel
$ kubectl create -f kube-flannel-rbac.yml $ kubectl create -f kube-flannel.yml |
查看状态
$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-flannel-ds-1b1vv 2/2 Running 0 14s $ ifconfig flannel.1 flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 10.244.0.0 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::b81f:4eff:fe06:71ce prefixlen 64 scopeid 0x20<link> ether ba:1f:4e:06:71:ce txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0 |
Kubernetes 支持 kube-dns 以 Cluster Add-On 的形式运行。Kubernetes 会在集群中调度一个 DNS 的 Pod 和 Service。
$ mkdir -p ~/k8s/kube-dns $ cd ~/k8s/kube-dns $ wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kubedns-cm.yaml $ wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kubedns-sa.yaml $ wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kubedns-svc.yaml.base $ wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kubedns-controller.yaml.base $ wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/transforms2sed.sed |
查看 transforms2sed.sed:
$ cat transforms2sed.sed s/__PILLAR__DNS__SERVER__/$DNS_SERVER_IP/g s/__PILLAR__DNS__DOMAIN__/$DNS_DOMAIN/g s/__MACHINE_GENERATED_WARNING__/Warning: This is a file generated from the base underscore template file: __SOURCE_FILENAME__/g |
将 $DNS_SERVER_IP
替换成 10.96.0.10
,将 DNS_DOMAIN
替换成 cluster.local.
。 注意 $DNS_SERVER_IP
要和 kubelet 设置的 --cluster-dns
参数一致。
s/__PILLAR__DNS__SERVER__/10.96.0.10/g s/__PILLAR__DNS__DOMAIN__/cluster.local/g s/__MACHINE_GENERATED_WARNING__/Warning: This is a file generated from the base underscore template file: __SOURCE_FILENAME__/g |
执行
$ cd ~/k8s/kube-dns $ sed -f transforms2sed.sed kubedns-svc.yaml.base > kubedns-svc.yaml $ sed -f transforms2sed.sed kubedns-controller.yaml.base > kubedns-controller.yaml |
上面的变量 DNS_SERVER
要和 kubelet 设置的 --cluster-dns
参数一致。
$ kubectl create -f kubedns-cm.yaml $ kubectl create -f kubedns-sa.yaml $ kubectl create -f kubedns-svc.yaml $ kubectl create -f kubedns-controller.yaml |
查看 kube-dns 的 Pod,确认所有 Pod 都处于 Running 状态:
$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-dns-1759312207-wd7m2 3/3 Running 0 1m kube-system kube-flannel-ds-1b1vv 2/2 Running 0 6m |
查看添加的节点状态
$ kubectl get nodes NAME STATUS AGE VERSION node02 Ready 54m v1.6.2 |
测试一下DNS功能是否好用
$ kubectl run bosybox --image=registry.aliyuncs.com/slzcc/busybox -i --tty sh $ nslookup kubernetes.default Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local |
kube-dns 是 Kubernetes 实现服务发现的重要组件之一,默认情况下只会创建一个 DNS Pod,在生产环境中我们可能需要对 kube-dns 进行扩容。
kubectl --namespace=kube-system scale deployment kube-dns --replicas=<NUM_YOU_WANT>
重复 1.7.2 的步骤,添加新的 Node节点。 在添加新节点时注意以下事项:
system:node:<node-name>
。每个 Node 节点是不同的。$ kubectl get nodes NAME STATUS AGE VERSION node02 Ready 54m v1.6.2 ... |