Page tree

如需转载请标注内容地址为: https://wiki.shileizcc.com/confluence/pages/viewpage.action?pageId=27000909

Skip to end of metadata
Go to start of metadata

未经允许不得转载

以 MongoDB 为例,使用 StatefulSet 完成 MongoDB 集群的创建,为每个 MongoDB 实例在共享存储中(这里采用 Ceph)申请一片存储空间,以实现一个无单点故障、高可用、可动态扩展的  MongoDB 集群。

前提条件

在创建 StatefulSet 之前,需要确保在 Kubernetes 集群中管理员已经创建好了共享存储,并能够与 StorageClass 对接,以实现动态存储供应的模式。

创建 StatefulSet

为了完成 MongoDB 集群的搭建,需要创建如下三个资源对象。

  • 一个 StorageClass,用于 StatefulSet 自动为各个应用 Pod 申请 PVC。
  • 一个 Headless Service,用于维护 MongoDB 集群的状态。
  • 一个 StatefulSet。

首先,保证拥有一个 StorageClass 对象。在 Ceph 中已经创建。

创建一个 StorageClass 对象。

ceph-storage-class.yaml
apiVersion: storage.k8s.io/v1 
kind: StorageClass 
metadata: 
  name: rbd 
provisioner: ceph.com/rbd 
parameters: 
  monitors: 10.140.0.8:6789,10.140.0.9:6789,10.140.0.10:6789
  adminId: admin 
  adminSecretName: ceph-admin-secret 
  adminSecretNamespace: kube-system 
  pool: kube 
  userId: admin 
  userSecretName: ceph-admin-secret 
  imageFormat: "2" 
  imageFeatures: layering

创建服务:

$ kubectl apply -f ceph-storage-class.yaml

mongo-sidecar 作为 MongDB 集群的管理者,将使用此 Headless Service 来维护各个 MongoDB 示例之间的集群关系,以及集群规模变化时的自动更新。

mongo-headless-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo

配置 mongo statefulset 配置文件:(项目 Github

statefulset-mongo.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo
        environment: test
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo:3.4
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--bind_ip"
            - 0.0.0.0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "rbd"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

其中的主要配置说明如下:

(1)在该 StatefulSet 的定义中包括两个容器:mongo 和 mongo-sidecar。mongo 是主服务程序,mongo-sidecar 是将多个 mongo 示例进行集群设置的工具。mongo-sidecar 中的环境变量如下。

  • MONGO_SIDECAR_POD_LABELS:设置为 mongo 容器的标签,用于 sidecar 查询它所需要管理的 MongoDB 集群示例。
  • KUBERNETES_MONGO_SERVICE_NAME:它的值为 "mongo",表示 sidecar 将使用 "mongo" 这个服务名来完成 MongoDB 集群的设置。

(2)replicas:表示 3 这个 MongoDB 集群由 3 个 mongo 实例组成。

(3)volumeClaimTemplates:是 StatefulSet 最重要的存储设置。在 annotations 段设置 volume.beta.kubernetes.io/storage-class="rbd" 表示使用名为 “rbd” 的 StirageClass 自动为每个 mongo Pod 实例分配后端存储。resources.requests.storage=10Gi 表示为每个 mongo 实例分配 10Gi 的磁盘空间。

创建 StatefulSet:

$ kubectl apply -f statefulset-mongo.yaml

最终可以看到 StatefulSet 依次创建并启动 3 个 mongo Pod 实例,它们的名字依次为 mongo-0、mongo-1、mongo-2:

$ kubectl get pods -l role=mongo
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          24m
mongo-1   2/2     Running   0          23m
mongo-2   2/2     Running   0          21m

StatefulSet 会用 volumeClaimTemplates 中的定义为每个 Pod 副本创建一个 PVC 实例,每个 PVC 的名称由 StatefulSet 定义中 volumeClaimTemplates 的名称和 Pod 副本的名字组合而成,查看系统中的 pvc,可以验证这一点:

$ kubectl get pvc
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongo-persistent-storage-mongo-0   Bound    pvc-b6ba6b9d-8c0c-11e9-a28c-42010aae0009   10Gi       RWO            rbd            25m
mongo-persistent-storage-mongo-1   Bound    pvc-e9f78d68-8c0c-11e9-a28c-42010aae0009   10Gi       RWO            rbd            23m
mongo-persistent-storage-mongo-2   Bound    pvc-14d6b6b9-8c0d-11e9-a28c-42010aae0009   10Gi       RWO            rbd            22m

下面是 mongo-0 这个 Pod 中的 volumes 设置,可以看到系统自动为其挂载了对应的 PVC:

$ kubectl get pod mongo-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 192.168.56.2/32
  creationTimestamp: "2019-06-11T05:49:44Z"
  generateName: mongo-
  labels:
    controller-revision-hash: mongo-7d764b574f
    environment: test
    role: mongo
    statefulset.kubernetes.io/pod-name: mongo-0
  name: mongo-0
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: mongo
    uid: b6b66046-8c0c-11e9-a28c-42010aae0009
  resourceVersion: "15315"
  selfLink: /api/v1/namespaces/default/pods/mongo-0
  uid: b6bc1311-8c0c-11e9-a28c-42010aae0009
spec:
  containers:
  - command:
    - mongod
    - --replSet
    - rs0
    - --smallfiles
    - --noprealloc
    image: mongo
    imagePullPolicy: Always
    name: mongo
    ports:
    - containerPort: 27017
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /data/db
      name: mongo-persistent-storage
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-pt4lj
      readOnly: true
  - env:
    - name: MONGO_SIDECAR_POD_LABELS
      value: role=mongo,environment=test
    - name: KUBERNETES_MONGO_SERVICE_NAME
      value: mongo
    image: cvallance/mongo-k8s-sidecar
    imagePullPolicy: Always
    name: mongo-sidecar
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-pt4lj
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: mongo-0
  nodeName: instance-2
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: mongo
  terminationGracePeriodSeconds: 10
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: mongo-persistent-storage
    persistentVolumeClaim:
      claimName: mongo-persistent-storage-mongo-0
  - name: default-token-pt4lj
    secret:
      defaultMode: 420
      secretName: default-token-pt4lj
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-06-11T05:49:54Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-06-11T05:51:10Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-06-11T05:51:10Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-06-11T05:49:54Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://07d93dba7a690a7223c65d7ba070d4aacb69564329f4bd1149076c70dc169bf1
    image: mongo:latest
    imageID: docker-pullable://mongo@sha256:93bd5412f16f3b9f7e12eb94813087f195dad950807a8ca74aa2db080c203990
    lastState: {}
    name: mongo
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2019-06-11T05:50:45Z"
  - containerID: docker://defad92f01673c47dbfef942c172e0d3ef1ab9aecd68bbad7bd79a617d27576e
    image: cvallance/mongo-k8s-sidecar:latest
    imageID: docker-pullable://cvallance/mongo-k8s-sidecar@sha256:cd62d32db488fbf78dfbaef020edd7fc09ee4d3fe5d50cc0579e747e8232c77f
    lastState: {}
    name: mongo-sidecar
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2019-06-11T05:51:10Z"
  hostIP: 10.174.0.8
  phase: Running
  podIP: 192.168.56.2
  qosClass: BestEffort
  startTime: "2019-06-11T05:49:54Z"

至此,一个由 3 个实例组成的 MongoDB 集群就创建完成了。其中每个实例都拥有稳定的名称和独立的存储空间。

如果存在问题,请查看 log 等方式进行解决:

$ kubectl logs -f -c mongo mongo-0

查看 MongoDB 集群状态

登入任意一个 mongo Pod,在 mongo 命令行界面用 rs.status() 命令查看 MongoDB 集群的状态,可以看到 mongo 集群已通过 sidecar 完成了创建。集群中包含了 3 个节点,每个节点的名称都是 StatefulSet 设置的 DNS 域名格式的网络标识名称:

  • mongo-0.mongo.default.svc.cluster.local
  • mongo-1.mongo.default.svc.cluster.local
  • mongo-2.mongo.default.svc.cluster.local

同时,可以看到 3 个 mongo 实例各自角色(PRIMARY 或 SECONDARY)也都进行了正确的设置。

$ kubectl exec -it mongo-0 -- mongo
Defaulting container name to mongo.
Use 'kubectl describe pod/mongo-0 -n default' to see all of the containers in this pod.
MongoDB shell version v3.4.20
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.20
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
Server has startup warnings:
2019-06-11T06:56:30.959+0000 I STORAGE  [initandlisten]
2019-06-11T06:56:30.959+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-06-11T06:56:30.959+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten]
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten]
rs0:PRIMARY>
rs0:PRIMARY> rs.status()
{
	"set" : "rs0",
	"date" : ISODate("2019-06-11T06:59:31.111Z"),
	"myState" : 1,
	"term" : NumberLong(1),
	"syncingTo" : "",
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1560236370, 1),
			"t" : NumberLong(1)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1560236370, 1),
			"t" : NumberLong(1)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1560236370, 1),
			"t" : NumberLong(1)
		}
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "192.168.56.4:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 181,
			"optime" : {
				"ts" : Timestamp(1560236370, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2019-06-11T06:59:30Z"),
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1560236209, 2),
			"electionDate" : ISODate("2019-06-11T06:56:49Z"),
			"configVersion" : 5,
			"self" : true,
			"lastHeartbeatMessage" : ""
		},
		{
			"_id" : 1,
			"name" : "192.168.89.5:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 90,
			"optime" : {
				"ts" : Timestamp(1560236360, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1560236360, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2019-06-11T06:59:20Z"),
			"optimeDurableDate" : ISODate("2019-06-11T06:59:20Z"),
			"lastHeartbeat" : ISODate("2019-06-11T06:59:29.392Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-11T06:59:30.387Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "192.168.56.4:27017",
			"syncSourceHost" : "192.168.56.4:27017",
			"syncSourceId" : 0,
			"infoMessage" : "",
			"configVersion" : 5
		},
		{
			"_id" : 2,
			"name" : "192.168.89.6:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 39,
			"optime" : {
				"ts" : Timestamp(1560236360, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1560236360, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2019-06-11T06:59:20Z"),
			"optimeDurableDate" : ISODate("2019-06-11T06:59:20Z"),
			"lastHeartbeat" : ISODate("2019-06-11T06:59:29.394Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-11T06:59:31.016Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "192.168.56.4:27017",
			"syncSourceHost" : "192.168.56.4:27017",
			"syncSourceId" : 0,
			"infoMessage" : "",
			"configVersion" : 5
		}
	],
	"ok" : 1
}

对于需要访问这个 mongo 集群的 Kubernetes 集群内部客户端来说,可以通过 Headless Service “mongo” 获取到后端的所有 Endpoints 列表,并组合为数据库链接串,例如 “mongodb://mongo-0.mongo,mongo-1.mongo,mongo-2.mongo:27017/dbname_?”。

可通过启动 busybox 应用来测试 DNS:

busybox.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: example-busybox
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example-busybox
  template:
    metadata:
      labels:
        app: example-busybox
        version: v1.0.0
    spec:
      containers:
      - image: slzcc/ansible:demo4
        command:
          - sleep
          - "3600000"
        name: busybox
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: example-busybox
  namespace: default
spec:
  ports:
  - port: 80
    name: http
    protocol: TCP
    targetPort: 80
  selector:
   app: example-busybox
$ nslookup mongo-0.mongo.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	mongo-0.mongo.default.svc.cluster.local
Address: 192.168.56.4

$ nslookup mongo-1.mongo.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	mongo-1.mongo.default.svc.cluster.local
Address: 192.168.89.5

$ nslookup mongo-2.mongo.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	mongo-2.mongo.default.svc.cluster.local
Address: 192.168.89.6

StatefulSet 的场景应用场景

下面对 MongoDB 集群场景的两种场景进行操作,说明 StatefulSet 对用状态应用的自动化管理功能。

MongoDB 集群的扩容

假设在系统运行过程中,3 个 mongo 实例不足以满足业务的需求,这时就需要用对 mongo 集群进行扩容。仅需要通过对 StatefulSet 进行 scale 操作,就能实现在 mongo 集群中自动添加新的 mongo 节点。

扩展实例命令:

$ kubectl scale --replicase=4 statefulset mongo
statefulset.apps/mongo scaled

查看实例数量:

$ kubectl get po -l role=mongo
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          10m
mongo-1   2/2     Running   0          9m13s
mongo-2   2/2     Running   0          8m5s
mongo-3   2/2     Running   0          32s

进入某个实例查看 mongo 集群的状态,可以看到第 4 个节点已经加入:

$ kubectl exec -it mongo-0 -- mongo
Defaulting container name to mongo.
Use 'kubectl describe pod/mongo-0 -n default' to see all of the containers in this pod.
MongoDB shell version v3.4.20
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.20
Server has startup warnings:
2019-06-11T06:56:30.959+0000 I STORAGE  [initandlisten]
2019-06-11T06:56:30.959+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-06-11T06:56:30.959+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten]
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2019-06-11T06:56:31.142+0000 I CONTROL  [initandlisten]
rs0:PRIMARY> rs.status()
{
	"set" : "rs0",
	"date" : ISODate("2019-06-11T07:06:26.678Z"),
	"myState" : 1,
	"term" : NumberLong(1),
	"syncingTo" : "",
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1560236780, 1),
			"t" : NumberLong(1)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1560236780, 1),
			"t" : NumberLong(1)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1560236780, 1),
			"t" : NumberLong(1)
		}
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "192.168.56.4:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 596,
			"optime" : {
				"ts" : Timestamp(1560236780, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2019-06-11T07:06:20Z"),
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1560236209, 2),
			"electionDate" : ISODate("2019-06-11T06:56:49Z"),
			"configVersion" : 6,
			"self" : true,
			"lastHeartbeatMessage" : ""
		},
		{
			"_id" : 1,
			"name" : "192.168.89.5:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 505,
			"optime" : {
				"ts" : Timestamp(1560236780, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1560236780, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2019-06-11T07:06:20Z"),
			"optimeDurableDate" : ISODate("2019-06-11T07:06:20Z"),
			"lastHeartbeat" : ISODate("2019-06-11T07:06:24.972Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-11T07:06:22.976Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "192.168.56.4:27017",
			"syncSourceHost" : "192.168.56.4:27017",
			"syncSourceId" : 0,
			"infoMessage" : "",
			"configVersion" : 6
		},
		{
			"_id" : 2,
			"name" : "192.168.89.6:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 455,
			"optime" : {
				"ts" : Timestamp(1560236780, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1560236780, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2019-06-11T07:06:20Z"),
			"optimeDurableDate" : ISODate("2019-06-11T07:06:20Z"),
			"lastHeartbeat" : ISODate("2019-06-11T07:06:24.972Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-11T07:06:22.974Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "192.168.56.4:27017",
			"syncSourceHost" : "192.168.56.4:27017",
			"syncSourceId" : 0,
			"infoMessage" : "",
			"configVersion" : 6
		},
		{
			"_id" : 3,
			"name" : "192.168.56.5:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 23,
			"optime" : {
				"ts" : Timestamp(1560236780, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1560236780, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2019-06-11T07:06:20Z"),
			"optimeDurableDate" : ISODate("2019-06-11T07:06:20Z"),
			"lastHeartbeat" : ISODate("2019-06-11T07:06:24.969Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-11T07:06:23.533Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "192.168.56.4:27017",
			"syncSourceHost" : "192.168.56.4:27017",
			"syncSourceId" : 0,
			"infoMessage" : "",
			"configVersion" : 6
		}
	],
	"ok" : 1
}

同时,系统也为 mongo-3 分配了一个新的 PVC 用于保存数据。

自动故障恢复

假设在系统运行过程中,某个 mongo 实例或其所在主机发生故障,则 StatefulSet 将会自动重建该 mongo 实例,并保证其身份(ID)和使用的数据(PVC)不变。

以 mongo-0 实例发生故障为例,StatefulSet 将会自动重建 mongo-0 实例,并未其挂载之前分配的 PVC “mongo-persistent-storage-mongo-0”。服务 “mongo-0” 在重新启动后,原数据库中的数据不会丢失,可继续使用。

$ kubectl get po -l role=mongo
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          11m
mongo-1   2/2     Running   0          10m
mongo-2   2/2     Running   0          9m11s
mongo-3   2/2     Running   0          98s
 
$ kubectl delete pod mongo-0
pod "mongo-0" deleted
 
$ kubectl get po -l role=mongo
NAME      READY   STATUS              RESTARTS   AGE
mongo-0   0/2     ContainerCreating   0          6s
mongo-1   2/2     Running             0          10m
mongo-2   2/2     Running             0          9m31s
mongo-3   2/2     Running             0          118s

查看此时的 mongo-0 的配置:

$ kubectl get pod mongo-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 192.168.56.6/32
  creationTimestamp: "2019-06-11T07:07:21Z"
  generateName: mongo-
  labels:
    controller-revision-hash: mongo-6cc9fc5786
    environment: test
    role: mongo
    statefulset.kubernetes.io/pod-name: mongo-0
  name: mongo-0
  namespace: default
...
  volumes:
  - name: mongo-persistent-storage
    persistentVolumeClaim:
      claimName: mongo-persistent-storage-mongo-0
  - name: default-token-pt4lj
    secret:
      defaultMode: 420
      secretName: default-token-pt4lj
...

进入某个实例查看 mongo 集群的状态,mongo-0 在发生故障前在集群中的角色为 PRIMARY,在其脱离集群后,mongo 集群会自动选出一个 SECONDARY 节点提升为 PRIMARY 节点(本例中为 mongo-2)。重启后的 mongo-0 则会称为一个新的 SECONDARY 节点。

$ kubectl exec -it mongo-1 -- mongo
...
rs0:PRIMARY> rs.status()
{
	"set" : "rs0",
	"date" : ISODate("2019-06-11T07:10:47.505Z"),
	"myState" : 1,
	"term" : NumberLong(2),
	"syncingTo" : "",
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1560237045, 1),
			"t" : NumberLong(2)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1560237045, 1),
			"t" : NumberLong(2)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1560237045, 1),
			"t" : NumberLong(2)
		}
	},
	"members" : [
		{
			"_id" : 1,
			"name" : "192.168.89.5:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 789,
			"optime" : {
				"ts" : Timestamp(1560237045, 1),
				"t" : NumberLong(2)
			},
			"optimeDate" : ISODate("2019-06-11T07:10:45Z"),
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1560236848, 1),
			"electionDate" : ISODate("2019-06-11T07:07:28Z"),
			"configVersion" : 9,
			"self" : true,
			"lastHeartbeatMessage" : ""
		},
		{
			"_id" : 2,
			"name" : "192.168.89.6:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 716,
			"optime" : {
				"ts" : Timestamp(1560237045, 1),
				"t" : NumberLong(2)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1560237045, 1),
				"t" : NumberLong(2)
			},
			"optimeDate" : ISODate("2019-06-11T07:10:45Z"),
			"optimeDurableDate" : ISODate("2019-06-11T07:10:45Z"),
			"lastHeartbeat" : ISODate("2019-06-11T07:10:47.462Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-11T07:10:45.468Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"configVersion" : 9
		},
		{
			"_id" : 3,
			"name" : "192.168.56.5:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 284,
			"optime" : {
				"ts" : Timestamp(1560237045, 1),
				"t" : NumberLong(2)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1560237045, 1),
				"t" : NumberLong(2)
			},
			"optimeDate" : ISODate("2019-06-11T07:10:45Z"),
			"optimeDurableDate" : ISODate("2019-06-11T07:10:45Z"),
			"lastHeartbeat" : ISODate("2019-06-11T07:10:47.464Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-11T07:10:45.471Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"configVersion" : 9
		}
	],
	"ok" : 1
}
从上面的例子中可以看到,Kubernetes 使用 StatefulSet 来搭建有状态的应用集群(MongoDB、MySQL 等),同部署无状态的应用一样简便。Kubernetes 能够保证 StatefulSet 中各应用实例在创建和运行的过程中,都具有固定的身份标识和独立的后端存储,还支持在运行时对集群规模进行扩容、保障集群的高可用等非常重要的功能。
  • No labels