Page tree

如需转载请标注内容地址为: https://wiki.shileizcc.com/confluence/display/etcd/Etcd+use+Cfssl+in+Clusters

Skip to end of metadata
Go to start of metadata

创建证书

环境准备

首先保证安装 Cfssl  Cfsll install

准备机器

主机名IP备注
node01192.166.1.12cfssl,cfssljson
node02192.166.1.2
node03192.166.1.13

CA证书和私钥

创建:

ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "frognew": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

ca-config.json 中可以定义多个 profile,分别设置不同的 expiry 和 usages 等参数。如上面的 ca-config.json 中定义了名称为 frognew 的 profile,这个 profile 的 expiry 87600h 为 10 年,useages 中:

  • signing表示此CA证书可以用于签名其他证书,ca.pem中的CA=TRUE
  • server auth表示TLS Server Authentication
  • client auth表示TLS Client Authentication

创建CA证书签名请求配置:

ca-csr.json
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "cloudnative"
    }
  ]
}

下面使用 cfss 生成 CA 证书和私钥:

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2017/05/11 15:50:00 [INFO] generating a new CA key and certificate from CSR
2017/05/11 15:50:00 [INFO] generate received request
2017/05/11 15:50:00 [INFO] received CSR
2017/05/11 15:50:00 [INFO] generating key: rsa-2048
2017/05/11 15:50:00 [INFO] encoded CSR
2017/05/11 15:50:00 [INFO] signed certificate with serial number 15784603879474863111844291591890643469090743811
$ ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

etcd证书和私钥

创建etcd证书签名请求配置:

etcd-csr.json
{
    "CN": "slzcc",
    "hosts": [
      "127.0.0.1",
      "192.166.1.12",
      "192.166.1.2",
      "192.166.1.13",
      "node1",
      "node2",
      "node3"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "slzcc",
            "OU": "cloudnative"
        }
    ]
}

注意上面配置 hosts 字段中制定授权使用该证书的 IP 和域名列表,因为现在要生成的证书需要被 etcd 集群各个节点使用,所以这里指定了各个节点的 IP 和 hostname。

下面生成etcd的证书和私钥:

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew etcd-csr.json | cfssljson -bare etcd
2017/05/11 15:59:10 [INFO] generate received request
2017/05/11 15:59:10 [INFO] received CSR
2017/05/11 15:59:10 [INFO] generating key: rsa-2048
2017/05/11 15:59:11 [INFO] encoded CSR
2017/05/11 15:59:11 [INFO] signed certificate with serial number 92375075491683600471705917127105851217494569006
$ ls etcd*
etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem

对生成的证书可以使用 cfssl 或 openssl 查看:

cfssl-certinfo
$ cfssl-certinfo -cert etcd.pem
{
  "subject": {
    "common_name": "slzcc",
    "country": "CN",
    "organization": "slzcc",
    "organizational_unit": "cloudnative",
    "locality": "BeiJing",
    "province": "BeiJing",
    "names": [
      "CN",
      "BeiJing",
      "BeiJing",
      "slzcc",
      "hotstone",
      "slzcc"
    ]
  },
  "issuer": {
    "common_name": "slzcc",
    "country": "CN",
    "organization": "frognew",
    "organizational_unit": "cloudnative",
    "locality": "BeiJing",
    "province": "BeiJing",
    "names": [
      "CN",
      "BeiJing",
      "BeiJing",
      "frognew",
      "cloudnative",
      "slzcc"
    ]
  },
  "serial_number": "576009387263410194508123162490525803912519673637",
  "sans": [
    "node1",
    "node2",
    "node3",
    "127.0.0.1",
    "192.166.1.12",
    "192.166.1.2",
    "192.166.1.13"
  ],
  "not_before": "2017-05-11T08:53:00Z",
  "not_after": "2027-05-09T08:53:00Z",
  "sigalg": "SHA256WithRSA",
  "authority_key_id": "3:A6:29:22:CF:BE:63:46:98:7:5F:D7:86:45:67:15:F5:6D:C5:79",
  "subject_key_id": "CE:C7:FA:5F:13:0:9C:86:B1:71:B8:71:5C:60:C6:A2:F5:57:56:55",
  "pem": "-----BEGIN CERTIFICATE-----\nMIIEFDCCAvygAwIBAgIUZOUne98yiiCgIOiRot/sdeTm5yUwDQYJKoZIhvcNAQEL\nBQAwaTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0Jl\naUppbmcxEDAOBgNVBAoTB2Zyb2duZXcxFDASBgNVBAsTC2Nsb3VkbmF0aXZlMQ4w\nDAYDVQQDEwVzbHpjYzAeFw0xNzA1MTEwODUzMDBaFw0yNzA1MDkwODUzMDBaMGQx\nCzAJBgNVBAYTAkNOMRAwDgYDVQQIEwdCZWlKaW5nMRAwDgYDVQQHEwdCZWlKaW5n\nMQ4wDAYDVQQKEwVzbHpjYzERMA8GA1UECxMIaG90c3RvbmUxDjAMBgNVBAMTBXNs\nemNjMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArk0gekjVl9MDknu1\neXna6ZUbW9hAImv27KnH8rYbqmX1jClKmQVa0Z/7rbwdTSpIFvLBiSp/IQAwTFoz\n+aACBJDuaXFFov9I+Fo0+gAHYhX8k+aKi6QkRLdO598217124HVGYN4B56dUcQFi\nCCUQOmk8xUBdDmTsglb6EzjZn87VnKiEUu85XZIgflsFqcsW3CWvZr5g9yq+1M2c\nRovdjGCw+oLPCDyO+FCSUPO3/j5liOSXLUU2G0q2FZz9F1MXb3fdf6PEKoqGOXNh\nV6No6e3hxpKCwmLL7sLjvle82OVeBHV8T2eRFsaknO5vlI9caSe2HcZqJ3cE2zDa\nqClj7wIDAQABo4G4MIG1MA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEF\nBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUzsf6XxMAnIax\ncbhxXGDGovVXVlUwHwYDVR0jBBgwFoAUA6YpIs++Y0aYB1/XhkVnFfVtxXkwNgYD\nVR0RBC8wLYIFbm9kZTGCBW5vZGUyggVub2RlM4cEfwAAAYcEwKYBDIcEwKYBAocE\nwKYBDTANBgkqhkiG9w0BAQsFAAOCAQEAXmRvSDrucZfm6uEHagaqFg3px21i3j5K\nVorookErWbxpRrUqp1RBsEi69F34VHPDIvsqhmdx3zBOCVY2i83zwj1Xs6b/KgUV\noOWZE5pg4Kesie2iICTkAO2QpFn3wtEU+/DLRPnwHwQuzplv3Zne3/+gx0JfuU+a\nMsHG4WReqRmeHsL5d7SbNc5rbyoCiSmZ9t0b8x0ptUtwfE077i/t1LLFyQyzGMCq\nhS1XDtuU/XsZyLd2lKRPSn1RF8YtxGdLZhXVdQ27TN8jrM979FKIlVrK5MRKglcd\nBxkSbx321hQ6D3TSwDbf5p8gO9GPWK+yAtD1+GNWFzOK66KcX+DNtw==\n-----END CERTIFICATE-----\n"
}
openssl
$ openssl x509  -noout -text -in  etcd.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            64:e5:27:7b:df:32:8a:20:a0:20:e8:91:a2:df:ec:75:e4:e6:e7:25
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=BeiJing, L=BeiJing, O=frognew, OU=cloudnative, CN=slzcc
        Validity
            Not Before: May 11 08:53:00 2017 GMT
            Not After : May  9 08:53:00 2027 GMT
        Subject: C=CN, ST=BeiJing, L=BeiJing, O=slzcc, OU=hotstone, CN=slzcc
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ae:4d:20:7a:48:d5:97:d3:03:92:7b:b5:79:79:
                    da:e9:95:1b:5b:d8:40:22:6b:f6:ec:a9:c7:f2:b6:
                    1b:aa:65:f5:8c:29:4a:99:05:5a:d1:9f:fb:ad:bc:
                    1d:4d:2a:48:16:f2:c1:89:2a:7f:21:00:30:4c:5a:
                    33:f9:a0:02:04:90:ee:69:71:45:a2:ff:48:f8:5a:
                    34:fa:00:07:62:15:fc:93:e6:8a:8b:a4:24:44:b7:
                    4e:e7:df:36:d7:bd:76:e0:75:46:60:de:01:e7:a7:
                    54:71:01:62:08:25:10:3a:69:3c:c5:40:5d:0e:64:
                    ec:82:56:fa:13:38:d9:9f:ce:d5:9c:a8:84:52:ef:
                    39:5d:92:20:7e:5b:05:a9:cb:16:dc:25:af:66:be:
                    60:f7:2a:be:d4:cd:9c:46:8b:dd:8c:60:b0:fa:82:
                    cf:08:3c:8e:f8:50:92:50:f3:b7:fe:3e:65:88:e4:
                    97:2d:45:36:1b:4a:b6:15:9c:fd:17:53:17:6f:77:
                    dd:7f:a3:c4:2a:8a:86:39:73:61:57:a3:68:e9:ed:
                    e1:c6:92:82:c2:62:cb:ee:c2:e3:be:57:bc:d8:e5:
                    5e:04:75:7c:4f:67:91:16:c6:a4:9c:ee:6f:94:8f:
                    5c:69:27:b6:1d:c6:6a:27:77:04:db:30:da:a8:29:
                    63:ef
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                CE:C7:FA:5F:13:00:9C:86:B1:71:B8:71:5C:60:C6:A2:F5:57:56:55
            X509v3 Authority Key Identifier: 
                keyid:03:A6:29:22:CF:BE:63:46:98:07:5F:D7:86:45:67:15:F5:6D:C5:79

            X509v3 Subject Alternative Name: 
                DNS:node1, DNS:node2, DNS:node3, IP Address:127.0.0.1, IP Address:192.166.1.12, IP Address:192.166.1.2, IP Address:192.166.1.13
    Signature Algorithm: sha256WithRSAEncryption
         5e:64:6f:48:3a:ee:71:97:e6:ea:e1:07:6a:06:aa:16:0d:e9:
         c7:6d:62:de:3e:4a:56:8a:e8:a2:41:2b:59:bc:69:46:b5:2a:
         a7:54:41:b0:48:ba:f4:5d:f8:54:73:c3:22:fb:2a:86:67:71:
         df:30:4e:09:56:36:8b:cd:f3:c2:3d:57:b3:a6:ff:2a:05:15:
         a0:e5:99:13:9a:60:e0:a7:ac:89:ed:a2:20:24:e4:00:ed:90:
         a4:59:f7:c2:d1:14:fb:f0:cb:44:f9:f0:1f:04:2e:ce:99:6f:
         dd:99:de:df:ff:a0:c7:42:5f:b9:4f:9a:32:c1:c6:e1:64:5e:
         a9:19:9e:1e:c2:f9:77:b4:9b:35:ce:6b:6f:2a:02:89:29:99:
         f6:dd:1b:f3:1d:29:b5:4b:70:7c:4d:3b:ee:2f:ed:d4:b2:c5:
         c9:0c:b3:18:c0:aa:85:2d:57:0e:db:94:fd:7b:19:c8:b7:76:
         94:a4:4f:4a:7d:51:17:c6:2d:c4:67:4b:66:15:d5:75:0d:bb:
         4c:df:23:ac:cf:7b:f4:52:88:95:5a:ca:e4:c4:4a:82:57:1d:
         07:19:12:6f:1d:f6:d6:14:3a:0f:74:d2:c0:36:df:e6:9f:20:
         3b:d1:8f:58:af:b2:02:d0:f5:f8:63:56:17:33:8a:eb:a2:9c:
         5f:e0:cd:b7

安装etcd

下载etcd二进制文件包,并解压缩:

$ wget https://github.com/coreos/etcd/releases/download/v3.1.6/etcd-v3.1.6-linux-amd64.tar.gz
$ tar zxf etcd-v3.1.6-linux-amd64.tar.gz ; mv etcd-v3.1.6-linux-amd64/{etcd,etcdctl} /usr/bin/

在各节点创建 etcd 的数据目录

$ mkdir -p /var/lib/etcd

在每个节点上创建 etcd 的 systemd unit 文件 /usr/lib/systemd/system/etcd.service,注意替换 ETCD_NAME 和 INTERNAL_IP 变量的值:

$ export ETCD_NAME=node1
$ export INTERNAL_IP=192.166.1.12
$ cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
  --name ${ETCD_NAME} \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://${INTERNAL_IP}:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster node1=https://192.166.1.12:2380,node2=https://192.166.1.2:2380,node3=https://192.166.1.13:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF


/usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd   --name node1   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://192.166.1.12:2380   --listen-peer-urls https://192.166.1.12:2380   --listen-client-urls https://192.166.1.12:2379,https://127.0.0.1:2379   --advertise-client-urls https://192.166.1.12:2379   --initial-cluster-token etcd-cluster-1   --initial-cluster node1=https://192.166.1.12:2380,node2=https://192.166.1.2:2380,node3=https://192.166.1.13:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • 上面在启动参数中指定了 etcd 的工作目录和数据目录是/var/lib/etcd
  • --cert-file 和 --key-file 分别指定 etcd 的公钥证书和私钥
  • --peer-cert-file 和 --peer-key-file 分别指定了 etcd 的 Peers 通信的公钥证书和私钥。
  • --trusted-ca-file 指定了客户端的 CA 证书
  • --peer-trusted-ca-file 指定了 Peers 的 CA 证书
  • --initial-cluster-state new 表示这是新初始化集群,--name 指定的参数值必须在 --initial-cluster 

把生成的秘钥拷贝到其他 Node 机器的 /etc/etcd/ssl/ 目录下:

$ mkdir -p /etc/etcd/ssl/
$ scp ./{etcd-key.pem,ca.pem,etcd.pem} node02:/etc/etcd/ssl/

启动etcd

在各节点上启动 etcd:

$ systemctl daemon-reload
$ systemctl enable etcd
$ systemctl start etcd
$ systemctl status etcd

检查集群是否健康,在任一节点执行:

$ etcdctl \
  --ca-file=/etc/etcd/ssl/ca.pem \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --endpoints=https://node1:2379,https://node2:2379,https://node3:2379 \
  cluster-health
2017-05-11 17:08:21.190702 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-05-11 17:08:21.191848 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member bceb23be05645bc4 is healthy: got healthy result from https://192.166.1.12:2379
member d36644899dad062c is healthy: got healthy result from https://192.166.1.2:2379
member d36644899dad062c is healthy: got healthy result from https://192.166.1.13:2379
cluster is healthy

或者:

$ etcdctl --ca-file=certs/etcd-ca.pem --key-file=certs/etcd-key.pem --cert-file=certs/etcd.pem --endpoints=https://127.0.0.1:2379 ls
  • No labels