这里k8s官方给我们提供了两种比较主流的连接k8s API cluster的语言, 一种是GO, 另外一种就是我们DevOps比较主流的Python.

今天这里我将给大家介绍如何使用python client扩展包去编写脚本并远程连接k8s API cluster.

这样我们平时除了可以远程使用ssh+kubectl去与kubernetes进行命令行数据交互外, 同样可以在远程不依赖命令行直接连接k8s cluster API cluster, 与k8s进行python API交互, 方便我们后期利用python去对k8s进行自动化集成与二次开发.

这里我们不多说, 直接进入我们的runbook配置环节:

一.部署python client环境连接我们的k8s API cluster.

1.安装python3.6.5源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel -y

2.编译安装python3.6.5以及pip package manager

# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz --no-check-certificate

# tar xf Python-3.6.5.tar.xz

# cd Python-3.6.5

# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

3.安装virtualenv

# pip3.6 install --upgrade pip

# pip3.6 install virtualenv

4.配置加载virtualenv环境

# virtualenv -p /usr/local/bin/python3 .py3env

# source .py3env/bin/activate

5.安装kubernetes python client扩展包

# pip install kubernetes

二.在k8s master获取API cluster URL与token

我们需要在创建python脚本前, 在k8s主机上创建一个admin权限的service account, 并获取其token 用来作为我们的脚本凭证.

1.抓取Cluster URL地址

# APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")

# echo $APISERVER

2.创建k8s admin-user

# mkdir -p /kube/role

# cd /kube/role

在kube-system下创建admin-user

# vi CreateServiceAccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system

给admin-user赋予admin权限

# vi CreateServiceAccount.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

# kubectl create -f CreateServiceAccount.yaml
# kubectl create -f RoleBinding.yaml

3.获取admin-user token

# Token=$(kubectl describe secret $(kubectl get secret -n kube-system | grep ^admin-user | awk '{print $1}') -n kube-system | grep -E '^token'| awk '{print $2}')

# echo $Token

最后将token与APISERVER地址返回内容复制到python client主机上, 供脚本使用.

三. 在python client主机上编写脚本

1. 创建目录结构

# mkdir -p /kube/auth

# cd /kube/auth

# touch token.txt

2. 创建token.txt文件并将k8s主机上获取的Token字符串复制到该文件

这里我们获取的token会引入到我们的脚本下, 作为bearer authorization的api key与远程k8s API建立认证连接.

3. 编写python client脚本

# vi k8s_auth.py

#!/usr/bin/env python


from kubernetes import client, config


def main():
    # Define the barer token we are going to use to authenticate.
    # See here to create the token:
    # https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/

    with open('token.txt', 'r') as file:
        Token = file.read().strip('\n')

    APISERVER = 'https://10.110.16.14:6443'

    # Create a configuration object
    configuration = client.Configuration()

    # Specify the endpoint of your Kube cluster
    configuration.host = APISERVER

    # Security part.
    # In this simple example we are not going to verify the SSL certificate of
    # the remote cluster (for simplicity reason)
    configuration.verify_ssl = False

    # Nevertheless if you want to do it you can with these 2 parameters
    # configuration.verify_ssl=True
    # ssl_ca_cert is the filepath to the file that contains the certificate.
    # configuration.ssl_ca_cert="certificate"
    configuration.api_key = {"authorization": "Bearer " + Token}

    # configuration.api_key["authorization"] = "bearer " + Token
    # configuration.api_key_prefix['authorization'] = 'Bearer'
    # configuration.ssl_ca_cert = 'ca.crt'
    # Create a ApiClient with our config
    client.Configuration.set_default(configuration)

    # Do calls
    v1 = client.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = v1.list_pod_for_all_namespaces(watch=False)
    for i in ret.items:
        print("%s\t%s\t%s" %
              (i.status.pod_ip, i.metadata.namespace, i.metadata.name))


if __name__ == '__main__':
    main()

这个脚本通过抓取同目录的k8s token字符串, 以及我们之前获取到的api server地址,利用python kubernetes扩展模块连接我们的远程API, 最终实现打印我们k8s所有namespace下的所有pods.

实际效果与kubectl get pods --all-namespaces类似.

4. 修改权限并执行

# chmod 755 k8s_auth.py

# ./k8s_auth.py

Listing pods with their IPs:
...
10.244.1.3	default	db
10.244.1.2	default	kubernetes-downwardapi-volume-example
10.244.1.245	default	nginx1-7-deployment-667547b6d8-c5nsr
10.244.1.239	default	nginx1-7-deployment-667547b6d8-mjxq4
10.244.1.247	default	nginx1-8-deployment-d46768cf9-888v8
10.244.1.242	default	nginx1-8-deployment-d46768cf9-fglq2
10.110.16.14	kube-system	etcd-kube-master
10.110.16.14	kube-system	kube-apiserver-kube-master
10.110.16.14	kube-system	kube-controller-manager-kube-master
10.244.1.241	kube-system	kube-dns-6f4fd4bdf-qsxrj
10.110.16.14	kube-system	kube-flannel-ds-5sdlg
10.110.16.15	kube-system	kube-flannel-ds-qctv4
10.110.16.14	kube-system	kube-proxy-5nscq
10.110.16.15	kube-system	kube-proxy-6g7jc
10.110.16.14	kube-system	kube-scheduler-kube-master
10.244.1.246	kube-system	kubernetes-dashboard-58f5cb49c-6dxgn
10.244.1.240	kube-system	tiller-deploy-cbb85d8dc-f6rsv
10.110.16.15	kube-system	traefik-ingress-lb-765c44656f-fkzxb
10.244.1.243	spinnaker	kubelive-create-bucket-wxlv4
10.244.1.244	spinnaker	kubelive-delete-jobs-zhn75

这样我们就成功的编写完成python client脚本连接k8s API cluster, 将所有namespace下的pod的基本信息打印出来.

Finished...

traefik 是一款开源的反向代理与负载均衡工具。它最大的优点是能够与常见的微服务系统直接整合，可以实现自动化动态配置。目前支持 Docker, Swarm, Mesos/Marathon, Mesos, Kubernetes, Consul, Etcd, Zookeeper, BoltDB, Rest API 等等后端模型。

由于微服务架构以及 Docker 技术和 kubernetes 编排工具最近几年才开始逐渐流行，所以一开始的反向代理服务器比如 nginx、apache 并未提供其支持，所以才会出现 Ingress Controller 来做 kubernetes 和前端负载均衡器如 nginx 之间做衔接；即 Ingress Controller 的存在就是为了能跟 kubernetes 交互，又能写 nginx 配置，还能 reload 它，这是一种折中方案；而 traefik 天生就是提供了对 kubernetes 的支持，也就是说 traefik 本身就能跟 kubernetes API 交互，感知后端变化，因此可以得知:  traefik 完全已经取代了 Ingress Controller 作为与k8s交互的代理，在此给大家这里整体架构如下:

为什么选择 traefik？

Golang 编写，单文件部署，与系统无关，同时也提供小尺寸 Docker 镜像。
支持 Docker/Etcd 后端，天然连接我们的微服务集群。
内置 Web UI，管理相对方便。
自动配置 ACME(Let’s Encrypt) 证书功能。
性能尚可，我们也没有到压榨 LB 性能的阶段，易用性更重要。

除了这些以外，traefik 还有以下特点：
Restful API 支持。
支持后端健康状态检查，根据状态自动配置。
支持动态加载配置文件和 graceful 重启。
支持 WebSocket 和 HTTP/2。

目前网上很多资料虽然能够成功在k8s下搭建ingress+traefik实现cluster内网的代理功能, 但如何将这个反向代理IP暴露在我们外网, 也就是不依赖于AWS等公有云, 在我们on-prem(本地k8s网络环境)通过定义hostnetwork去实现将我们的node ip直接作为我们在cluster下创建的反向代理ip,  将是我们本次需要给大家介绍的内容.

我们接着上一篇Kubernates1.9+Docker17离线安装部署, 给大家展开如何实现k8s通过使用Ingress+Traefik实现集群反向代理功能.

1.获取Traefik支持K8s仓库脚本

# cd /kube/

# git clone https://github.com/containous/traefik.git

# cd traefik/examples/k8s/

# ll

total 36
-rw-r--r-- 1 root root  140 Aug 26 04:23 cheese-default-ingress.yaml
-rw-r--r-- 1 root root 1805 Aug 26 04:23 cheese-deployments.yaml
-rw-r--r-- 1 root root  519 Aug 26 04:23 cheese-ingress.yaml
-rw-r--r-- 1 root root  509 Aug 26 04:23 cheese-services.yaml
-rw-r--r-- 1 root root  504 Aug 26 04:23 cheeses-ingress.yaml
-rw-r--r-- 1 root root 1144 Aug 26 06:40 traefik-deployment.yaml
-rw-r--r-- 1 root root 1206 Aug 26 04:23 traefik-ds.yaml
-rw-r--r-- 1 root root  694 Aug 26 04:23 traefik-rbac.yaml
-rw-r--r-- 1 root root  466 Aug 26 04:40 ui.yaml

这个目录下就是示例 Traefik 启动所需要的 yaml 文件，Traefik 提供了适配各个类型服务编排的部署方式，kubernetes 启动方式支持 Deployment 和 DaemonSet，二选一都可以, 这里笔者选择Deployment

2. Traefik脚本配置

还记得我之前提到的hostnetwork吗? 

这里我们需要在部署之前将我们的traefik-deployment.yaml下添加一行 hostNetwork: true

这样子就能保证我们的Traefik反向代理的会将自己Endpoints IP配置成我们的NodeIP, 供我们与K8S同一网段的设备访问.

我们可以这么配置:

# vi traefik-deployment.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      hostNetwork: true
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
        - name: admin
          containerPort: 8080
        args:
        - --api
        - --kubernetes
        - --logLevel=INFO
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

# vi traefik-rbac.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

这样子我们就完成了K8S脚本的初始修改, 接下来我们就可以直接开始我们的Traefik安装

3. Traefik安装

# kubectl apply -f traefik-rbac.yaml

clusterrole "traefik-ingress-controller" created
clusterrolebinding "traefik-ingress-controller" created

# kubectl apply -f traefik-deployment.yaml

serviceaccount "traefik-ingress-controller" created
deployment "traefik-ingress-controller" created
service "traefik-ingress-service" created

# kubectl apply -f ui.yaml

service "traefik-web-ui" created
ingress "traefik-web-ui" created

# kubectl get pods --all-namespaces -o wide

NAME                                          READY     STATUS    RESTARTS   AGE       IP             NODE
…
traefik-ingress-controller-795ffb7d78-ppw94   1/1       Running   0          5m        10.110.16.15   kube-node1

可以看到我们的traefik-ingress pods成功的打开了我们的hostnetwork, 这里显示的是我们的node ip而不是pod内网 ip, 也就是我们可以直接通过访问node ip 10.110.16.15来连接traefik反向代理, 利用traefik来分配我们的K8S的容器连接.

# kubectl get service --all-namespaces

NAMESPACE      NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
default        frontend                    ClusterIP   10.110.87.1      <none>        80/TCP                        4h
default        kubernetes                  ClusterIP   10.96.0.1        <none>        443/TCP                       1d
default        my-nginx                    ClusterIP   10.102.200.83    <none>        80/TCP                        4h
kube-system    kube-dns                    ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP                 1d
kube-system    kubernetes-dashboard        NodePort    10.108.187.244   <none>        443:32666/TCP                 1d
kube-system    tiller-deploy               ClusterIP   10.111.251.3     <none>        44134/TCP                     1d
kube-system    traefik-ingress-service     NodePort    10.100.70.138    <none>        80:31087/TCP,8080:31017/TCP   13m
kube-system    traefik-web-ui              ClusterIP   10.96.54.167     <none>        80/TCP                        13m
newegg-nginx   newegg-nginx-newegg-nginx   NodePort    10.97.200.124    <none>        80:32239/TCP                  8h

我们可以通过两种方式去访问我们的traefik后台管理员界面.

1).通过Node Port方式

我们可以在与K8S node同网段的机器去访问其Node暴露的端口31017

2).直接通过ui.yaml配置traefik+ingress反向代理, 这样我们就可以通过host域名直接访问后台

# vi ui.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  rules:
  - host: traefik-ui.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

在本地MacOS系统添加traefik-ui.k8s解析到10.110.16.15

# echo "10.110.16.15 traefik-ui.k8s" >> /etc/hosts

我们可以同样利用traefik反向代理, 通过http域名的方式访问admin后台.

4.部署两个服务nginx1-7和nginx1-8，配置Traefik去负载这两个服务：

配置一个deployment类型的1.7版本nginx实例, 并利用service开启其内部80端口监听.

# vi nginx1-7.yaml

apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-7
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx1-7-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-7
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

配置一个deployment类型的1.8版本nginx实例, 并利用service开启其内部80端口监听.

# vi nginx1-8.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-nginx
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-8
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx1-8-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-8
    spec:
      containers:
      - name: nginx
        image: nginx:1.8
        ports:
        - containerPort: 80

通过利用Ingress绑定1.7与1.8nginx实例下的serviceName, 给其定义不同的域名, 实现traefik反向代理.

# vi traefik.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: default
spec:
  rules:
  - host: traefik.nginx.io
    http:
      paths:
      - path: /
        backend:
          serviceName: my-nginx
          servicePort: 80
  - host: traefik.frontend.io
    http:
      paths:
      - path: /
        backend:
          serviceName: frontend
          servicePort: 80

我们这里通过配置MacOS本地Host DNS, 将所有的DNS记录指向我们暴露在外网的traefik IP, 从而让traefik作为代理接管所有访问K8S内部实例的连接, 实际的原理和我们之前用过的apache, nginx基本一致.

# echo "10.110.16.15 traefik.nginx.io traefik.frontend.io" >> /etc/hosts

这样子我们就可以直接在MacOS浏览器下访问traefik.nginx.io与traefik.frontend.io从而实现traefik反向代理.

5.配置HTTPS访问traefik管理员后台

1).创建并配置https证书并保存到k8s secret下.

# mkdir -p /opt/k8s/ssl

# cd /opt/k8s/ssl

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=traefik-ui.k8s"

# kubectl create secret generic traefik-cert --from-file=tls.crt --from-file=tls.key -n kube-system

2).配置traefik反向代理并保存到k8s configmap下.

# mkdir -p /opt/k8s/conf/

# cd /opt/k8s/ssl/

# vi traefik-ui.toml

defaultEntryPoints = ["http","https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    regex = "^http://traefik-ui.k8s/(.*)"
    replacement = "https://traefik-ui.k8s/$1"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/opt/k8s/ssl/tls.crt"
      keyFile = "/opt/k8s/ssl/tls.key"

# kubectl create configmap traefik-ui-conf --from-file=traefik-ui.toml -n kube-system

这样子我们就成功配置traefik反向代理, 仅使http://traefik-ui.k8s会重定向到https://traefik-ui.k8s, 实现https加密访问.

3).创建traefik deployment实例并支持https反向代理.

# vi traefik-deployment-ssl.yaml 

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: traefik-ingress-lb
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      restartPolicy: Always
      serviceAccountName: traefik-ingress-controller
      volumes:
      - name: ssl
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-ui-conf
      containers:
      - image: traefik
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: "/opt/k8s/ssl"
          name: "ssl"
        - mountPath: "/opt/k8s/conf"
          name: "config"
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
        - name: admin
          containerPort: 8080
          hostPort: 8080
        args:
        - --configFile=/opt/k8s/conf/traefik-ui.toml
        - --web
        - --web.address=:8080
        - --kubernetes
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: https
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

这里我们在原来的traefik deployment实例基础上将之前配置的secret与configmap导入该deployment下, 并mount ssl与conf目录用来让traefik获取我们在k8s node目录下保存的相应SSL证书与traefik反向代理配置.

最后我们通过service类型打开我们的deployment 443端口.

4).创建traefik管理员界面的ingress, 从而实现https域名访问.

# vi ui-ssl.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  tls:
    - secretName: traefik-cert
  rules:
  - host: traefik-ui.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

最后我们访问https://traefik-ui.k8s 验证结果.

这样子我们就成功的开启了traefik后台admin界面的https加密连接...

更多traefik+ingress相关内容: https://docs.traefik.io/user-guide/kubernetes/

这里简单的介绍一下我们这个自动化流水线所使用到的工具:

Jenkins Pipeline: 目前国内外DevOps, CI/CD比较主流的一种将我们软件开发周期所涉及到的环节通过pipeline流水线完美的串联在一起的自动化部署框架. 它提出了一种pipeline as a code的概念, 意在将我们的所有软件开发周期中涉及到的所有步骤(版本控制 - 代码检查 - 编译 - 单元测试 - 打包 - 测试环境部署 - 集成测试 - 功能测试 - 生产环境部署)通过代码的方式推送给我们的Jenkins进行pipeline自动化部署, 实现将我们的自动化流水线的配置代码化, 并同样实现版本控制. 

目前Jenkins官方有两种pipeline的写法, 一种叫做Declarative Pipeline, 另外一种叫做Scripted Pipeline. 前者较为方便阅读, 适合初学者入门, 但相对对集成模块的兼容性以及pipeline逻辑编写的功能性较后者相对弱一些, 后者较为专业, 可以实现逻辑编写, 并支持很多主流集成模块, 推荐具有groovy脚本编写经验的人员使用.

具体内容详见: http://www.showerlee.com/archives/1972

Kubernetes: 目前这几年流行的一种容器化管理系统, 用于自动部署、扩展和管理容器化（containerized）应用程序的开源系统。它旨在提供“跨主机集群的自动部署、扩展以及运行应用程序容器的平台”。它支持一系列容器工具, 目前主流会使用Docker作为他的主流配置容器.使用它的原因也在于它将是目前DevOps领域的一个主流的Docker容器管理系统, 目前国内外很多公司都即将或者已经把自己的传统的虚拟机架构转向为Docker微服务架构, 这里我们非常有必要去学习如何基于k8s下去做自动化部署.

具体内容详见: http://www.showerlee.com/archives/2200

Helm: 这个应该算作k8s的一个功能性的工具, 它作为k8s的包管理工具, 非常方便的帮助我们整合k8s下零散的部署脚本为一个具体的项目包, 最终简化了Kubernetes部署应用的版本控制、打包、发布、删除、更新等操作.

具体内容详见: http://www.showerlee.com/archives/2455

这里本次自动化流水线实现的部署内容就是通过编写pipeline脚本, 将一个从Docker官方拿到的centos docker镜像容器, 进行二次安装配置, 打包为一个nignx静态网站容器, 发布到我们k8s下, 期间我们会对这个容器进行一些常规的功能测试, 从而模拟我们在k8s下实现自动化流水线交付的完整架构.

本次内容涉及到的代码可以参考我的github仓库:

https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes

Okay, Let's roll out...

安装环境

Local Desktop: MacOS

Virtual Machine: Virtual Box

Virtual System: CentOS 7.4

Jenkins: Jenkins 2.138

Kubernetes: Kubernetes 1.9

Docker: 17.03.2-ce

Helm: helm-v2.7.0

kube-master 10.110.16.14

kube-node-1 10.110.16.15

一. 系统环境配置

1.关闭SELINUX和firewall

# vi /etc/sysconfig/selinux

...
SELINUX=disabled 
...

# setenforce 0

# systemctl stop firewalld  && systemctl disable firewalld

2.安装k8s环境.

http://www.showerlee.com/archives/2200

3.安装helm环境.

http://www.showerlee.com/archives/2455

4.安装Jenkins环境.

http://www.showerlee.com/archives/1880

二. Jenkins Pipeline配置

1.将jenkins用户添加到默认docker用户组下, 从而保证jenkins可以直接访问/var/run/docker.sock

# usermod -a -G docker jenkins

2.更改全局安全配置, 保证pipeline可以直接调用helm

进入Jenkins --> Manage Jenkins --> Configure Global Security
在Authorization下选择Project-based Matrix Authorization Strategy

配置Anonymous User具有Read Jenkins Jobs的权限

如图:

3.配置jenkins用户加载kubectl环境变量.

# mkdir -p /home/jenkins/.kube

# cp -i /etc/kubernetes/admin.conf /home/jenkins/.kube/config

# chown jenkins:jenkins /home/jenkins/.kube/config

4.配置github与dockerhub的Jenkins账户凭证, 后面pipeline对应模块需要调用.

5.创建pipeline任务

#!groovy

def kubectlTest() {
    // Test that kubectl can correctly communication with the Kubernetes API
    echo "running kubectl test"
    sh "kubectl get nodes"

}

def helmLint(String chart_dir) {
    // lint helm chart
    sh "/usr/local/bin/helm lint ${chart_dir}"

}

def helmDeploy(Map args) {
    //configure helm client and confirm tiller process is installed

    if (args.dry_run) {
        println "Running dry-run deployment"

        sh "/usr/local/bin/helm upgrade --dry-run --debug --install ${args.name} ${args.chart_dir} --set ImageTag=${args.tag},Replicas=${args.replicas},Cpu=${args.cpu},Memory=${args.memory},DomainName=${args.name} --namespace=${args.name}"
    } else {
        println "Running deployment"
        sh "/usr/local/bin/helm upgrade --install ${args.name} ${args.chart_dir} --set ImageTag=${args.tag},Replicas=${args.replicas},Cpu=${args.cpu},Memory=${args.memory},DomainName=${args.name} --namespace=${args.name}"

        echo "Application ${args.name} successfully deployed. Use helm status ${args.name} to check"
    }
}



timeout(time: 2000, unit: 'SECONDS') {
    node {
        println "----------------------------------------------------------------------------"
        stage 'Check out pipeline from GitHub Repo'
        //git url: 'https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git'
        git branch: 'master',
            credentialsId: 'showerlee-github',
            url: 'https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git'

        // Setup the Docker Registry (Docker Hub) + Credentials 
        registry_url = "https://index.docker.io/v1/" // Docker Hub
        docker_creds_id = "showerlee-dockerhub" // name of the Jenkins Credentials ID

        def pwd = pwd()
        def chart_dir = "${pwd}/charts/newegg-nginx"

        // Add build tag version
        Properties props = new Properties()
        File propsFile = new File("${pwd}/promote.properties")
        props.load(propsFile.newDataInputStream())
        def build_tag_raw = props.getProperty('BUILD_TAG')
        float build_tag = Float.parseFloat(build_tag_raw)+0.1;
        println("Set current build_tag="+build_tag+" temporarily")

        //Set build_tag to index.html
        sh """
        echo "<h1>Welcome Newegg Nginx Test Version: ${build_tag}</h1>" > index.html
        """

        def inputFile = readFile('config.json')
        def config = new groovy.json.JsonSlurperClassic().parseText(inputFile)
        println "pipeline config ==> ${config}"
        println "----------------------------------------------------------------------------"
        
        stage 'Register DockerHub'
        echo "[INFO] Register Dockerhub"
        docker.withRegistry("${registry_url}", "${docker_creds_id}") {
        
            // Set up the container to build 
            maintainer_name = "showerlee"
            container_name = "nginx-test"
            println "----------------------------------------------------------------------------"

            stage "Build Nginx Container"
            echo "[INFO] Building Nginx with docker.build(${maintainer_name}/${container_name}:${build_tag})"
            container = docker.build("${maintainer_name}/${container_name}:${build_tag}", '.')
            println "----------------------------------------------------------------------------"
            try {
                
                // Start Testing
                stage "Spin up Nginx Container"
                echo "[INFO] Spin up Nginx Container"
                
                // Run the container with the env file, mounted volumes and the ports:
                docker.image("${maintainer_name}/${container_name}:${build_tag}").withRun("--name=${container_name}  -p 80:80 ")  { c ->
                       
                    // wait for the django server to be ready for testing
                    // the 'waitUntil' block needs to return true to stop waiting
                    // in the future this will be handy to specify waiting for a max interval: 
                    // https://issues.jenkins-ci.org/browse/JENKINS-29037
                    //
                    waitUntil {
                        sh """
                        set +x
                        ss -antup | grep :::80[^0-9] | grep LISTEN | wc -l | tr -d '\n' > /tmp/wait_results
                        set -x
                        """
                        wait_results = readFile '/tmp/wait_results'

                        echo "[INFO] Wait Results(${wait_results})"
                        if ("${wait_results}" == "1")
                        {
                            echo "[INFO] Nginx is listening on port 80"
                            sh "rm -f /tmp/wait_results"
                            return true
                        }
                        else
                        {
                            echo "[INFO] Nginx is not listening on port 80 yet"
                            return false
                        }
                    } // end of waitUntil
                    
                    // At this point Nginx is running
                    echo "[INFO] Docker Container is running"
                    input 'You can check the running container on docker build server now! Click Proceed to next stage...'    
                    // this pipeline is using 3 tests 
                    // by setting it to more than 3 you can test the error handling and see the pipeline Stage View error message
                    MAX_TESTS = 3
                    for (test_num = 1; test_num <= MAX_TESTS; test_num++) {     
                        println "----------------------------------------------------------------------------"   
                        echo "Running Test(${test_num})"
                    
                        expected_results = 0
                        if (test_num == 1 ) 
                        {
                            // Test we can download the home page from the running docker container
                            echo "[INFO] Check validation of home page"
                            sh """
                            set +x
                            docker exec -t ${container_name} curl -s http://localhost | grep Welcome | wc -l | tr -d '\n' > /tmp/test_results
                            set -x
                            """
                            expected_results = 1
                        }
                        else if (test_num == 2)
                        {
                            // Test if port 80 is exposed
                            echo "[INFO] Check if port 80 is exposed"
                            sh """
                            set +x
                            docker inspect --format '{{ (.NetworkSettings.Ports) }}' ${container_name}
                            docker inspect --format '{{ (.NetworkSettings.Ports) }}' ${container_name} | grep map | grep '80/tcp:' | wc -l | tr -d '\n' > /tmp/test_results
                            set -x
                            """
                            expected_results = 1
                        }
                        else if (test_num == 3)
                        {
                            // Test there's nothing established on the port since nginx is not running:
                            echo "[INFO] Check if nothing established from nginx container"
                            sh """
                            set +x
                            docker exec -t ${container_name} ss -apn | grep 80 | grep ESTABLISHED | wc -l | tr -d '\n' > /tmp/test_results
                            set -x
                            """
                            expected_results = 0
                        }
                        else
                        {
                            err_msg = "Missing Test(${test_num})"
                            echo "[ERROR] ${err_msg}"
                            currentBuild.result = 'FAILURE'
                            error "Failed to finish container testing with Message(${err_msg})"
                        }
                        
                        // Now validate the results match the expected results
                        stage "Test(${test_num}) - Validate Results"
                        test_results = readFile '/tmp/test_results'
                        echo "[INFO] Test(${test_num}) Results($test_results) == Expected(${expected_results})"
                        sh """
                        set +x
                        if [ \"${test_results}\" != \"${expected_results}\" ]; 
                        then 
                            echo \" --------------------- Test(${test_num}) Failed--------------------\"
                            echo \" - Test(${test_num}) Failed\"
                            exit 1
                        else 
                            echo \" - Test(${test_num}) Passed\"
                            exit 0
                        fi
                        set -x
                        """

                        echo "[INFO] Finished Running Test(${test_num})"
                    
                        // cleanup after the test run
                        sh "rm -f /tmp/test_results"
                        currentBuild.result = 'SUCCESS'
                    }
                }
                
            } catch (Exception err) {
                err_msg = "Test had Exception(${err})"
                currentBuild.result = 'FAILURE'
                error "FAILED - Stopping build for Error(${err_msg})"
            }
            println "----------------------------------------------------------------------------"
            stage "Push to DockerHub"
            input 'Do you approve to push?'
            container.push()
            currentBuild.result = 'SUCCESS'
            println "----------------------------------------------------------------------------"
            stage "Push properties to git repo"
            echo "Push current build_tag="+build_tag+" to git repo"
            withCredentials([usernamePassword(credentialsId: 'showerlee-github', passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
                sh """
                set +x
                git --version
                echo 'BUILD_TAG=${build_tag}' > ${pwd}/promote.properties
                git config --global user.email "showerlee@vip.qq.com"
                git config --global user.name "showerlee"
                git add ${pwd}/promote.properties ${pwd}/index.html
                git commit -m"Update docker tag to ${build_tag}"
                git push --set-upstream https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git master
                set -x
                """
            }
            println "----------------------------------------------------------------------------"
            
        }
        
        stage ('helm test') { 
            echo "[INFO] Start helm test"   
            // run helm chart linter
            echo "[INFO] Run helm chart linter"
            helmLint(chart_dir)

            // dry-run helm chart installation
            echo "[INFO] Dry-run helm chart installation"
            helmDeploy(
                dry_run       : true,
                name          : config.app.name,
                chart_dir     : chart_dir,
                tag           : build_tag,
                replicas      : config.app.replicas,
                cpu           : config.app.cpu,
                memory        : config.app.memory
            )
            println "----------------------------------------------------------------------------"
        }
        
        stage ('helm deploy') {
            input 'Do you approve to deploy?'
            echo "[INFO] Start helm deployment"
            // Deploy using Helm chart
            helmDeploy(
                dry_run       : false,
                name          : config.app.name,
                chart_dir     : chart_dir,
                tag           : build_tag,
                replicas      : config.app.replicas,
                cpu           : config.app.cpu,
                memory        : config.app.memory
            )
            echo "[INFO] Deployment Finished..."
        }
        
        ///////////////////////////////////////
        //
        // Coming Soon Feature Enhancements
        //
        // 1. Add Docker Compose testing as a new Pipeline item that is initiated after this one for "Integration" testing
        // 2. Make sure to set the Pipeline's "Throttle builds" to 1 because the docker containers will collide on resources like ports and names
        // 3. Should be able to parallelize the docker.withRegistry() methods to ensure the container is running on the slave
        // 4. After the tests finish (and before they start), clean up container images to prevent stale docker image builds from affecting the current test run
    }
}

6.执行pipeline任务构建

Started by user admin
Obtained Jenkinsfile from git https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git
[Pipeline] timeout
Timeout set to expire in 33 min
[Pipeline] {
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/kube-helm-pipeline@2
[Pipeline] {
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Check out pipeline from GitHub Repo)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Check out pipeline from GitHub Repo
Proceeding
[Pipeline] git
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git # timeout=10
Fetching upstream changes from https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials github credential
 > git fetch --tags --progress https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision ce4fc2593333f3ed89cd5654966919b0aa97628d (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ce4fc2593333f3ed89cd5654966919b0aa97628d
 > git branch -a -v --no-abbrev # timeout=10
 > git branch -D master # timeout=10
 > git checkout -b master ce4fc2593333f3ed89cd5654966919b0aa97628d
Commit message: "Update docker tag to 1.5"
 > git rev-list --no-walk ce4fc2593333f3ed89cd5654966919b0aa97628d # timeout=10
[Pipeline] pwd
[Pipeline] echo
Set current build_tag=1.6 temporarily
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ echo '<h1>Welcome Newegg Nginx Test Version: 1.6</h1>'
[Pipeline] readFile
[Pipeline] echo
pipeline config ==> [app:[memory:128Mi, replicas:3, name:newegg-nginx, cpu:10m], pipeline:[library:[branch:master], enabled:true]]
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Register DockerHub)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Register DockerHub
Proceeding
[Pipeline] echo
[INFO] Register Dockerhub
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withDockerRegistry
$ docker login -u showerlee -p ******** https://index.docker.io/v1/
Login Succeeded
[Pipeline] {
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Build Nginx Container)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Build Nginx Container
Proceeding
[Pipeline] echo
[INFO] Building Nginx with docker.build(showerlee/nginx-test:1.6)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker build -t showerlee/nginx-test:1.6 .
Sending build context to Docker daemon 2.164 MB

Step 1/6 : FROM centos:centos7
 ---> 5182e96772bf
Step 2/6 : MAINTAINER showerlee
 ---> Using cache
 ---> cc77ae5c175a
Step 3/6 : RUN yum -y update         && yum clean all         && yum install -y epel-release         && yum install -y nginx iproute
 ---> Using cache
 ---> 2c718c146ba5
Step 4/6 : EXPOSE 80
 ---> Using cache
 ---> 829b29a719e6
Step 5/6 : COPY index.html /usr/share/nginx/html/
 ---> Using cache
 ---> 22e805b84cee
Step 6/6 : CMD nginx -g daemon off;
 ---> Using cache
 ---> de02105d9033
Successfully built de02105d9033
[Pipeline] dockerFingerprintFrom
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Spin up Nginx Container)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Spin up Nginx Container
Proceeding
[Pipeline] echo
[INFO] Spin up Nginx Container
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker run -d --name=nginx-test -p 80:80 showerlee/nginx-test:1.6
[Pipeline] dockerFingerprintRun
[Pipeline] waitUntil
[Pipeline] {
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
[Pipeline] readFile
[Pipeline] echo
[INFO] Wait Results(1)
[Pipeline] echo
[INFO] Nginx is listening on port 80
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ rm -f /tmp/wait_results
[Pipeline] }
[Pipeline] // waitUntil
[Pipeline] echo
[INFO] Docker Container is running
[Pipeline] input
You can check the running container on docker build server now! Click Proceed to next stage...
Proceed or Abort
Approved by admin
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] echo
Running Test(1)
[Pipeline] echo
[INFO] Check validation of home page
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
[Pipeline] stage (Test(1) - Validate Results)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Test(1) - Validate Results
Proceeding
[Pipeline] readFile
[Pipeline] echo
[INFO] Test(1) Results(1) == Expected(1)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
 - Test(1) Passed
[Pipeline] echo
[INFO] Finished Running Test(1)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ rm -f /tmp/test_results
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] echo
Running Test(2)
[Pipeline] echo
[INFO] Check if port 80 is exposed
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
map[80/tcp:[{0.0.0.0 80}]]
[Pipeline] stage (Test(2) - Validate Results)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Test(2) - Validate Results
Proceeding
[Pipeline] readFile
[Pipeline] echo
[INFO] Test(2) Results(1) == Expected(1)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
 - Test(2) Passed
[Pipeline] echo
[INFO] Finished Running Test(2)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ rm -f /tmp/test_results
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] echo
Running Test(3)
[Pipeline] echo
[INFO] Check if nothing established from nginx container
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
[Pipeline] stage (Test(3) - Validate Results)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Test(3) - Validate Results
Proceeding
[Pipeline] readFile
[Pipeline] echo
[INFO] Test(3) Results(0) == Expected(0)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
 - Test(3) Passed
[Pipeline] echo
[INFO] Finished Running Test(3)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ rm -f /tmp/test_results
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker stop 05a19a7c8519a04cc03029d7273a8e2ae3363e74a03b6d4167dfbec0bf6ee978
05a19a7c8519a04cc03029d7273a8e2ae3363e74a03b6d4167dfbec0bf6ee978
+ docker rm -f 05a19a7c8519a04cc03029d7273a8e2ae3363e74a03b6d4167dfbec0bf6ee978
05a19a7c8519a04cc03029d7273a8e2ae3363e74a03b6d4167dfbec0bf6ee978
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Push to DockerHub)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Push to DockerHub
Proceeding
[Pipeline] input
Do you approve to push?
Proceed or Abort
Approved by admin
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker tag showerlee/nginx-test:1.6 index.docker.io/showerlee/nginx-test:1.6
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker push index.docker.io/showerlee/nginx-test:1.6
The push refers to a repository [docker.io/showerlee/nginx-test]
aedd55edeb74: Preparing
85e9d4859663: Preparing
1d31b5806ba4: Preparing
85e9d4859663: Layer already exists
1d31b5806ba4: Layer already exists
aedd55edeb74: Retrying in 5 seconds
aedd55edeb74: Retrying in 4 seconds
aedd55edeb74: Retrying in 3 seconds
aedd55edeb74: Retrying in 2 seconds
aedd55edeb74: Retrying in 1 second
aedd55edeb74: Pushed
1.6: digest: sha256:8ee214129c880a0d759837daf30c31772797fa6709f11edd9e30db6891710de4 size: 948
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Push properties to git repo)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Push properties to git repo
Proceeding
[Pipeline] echo
Push current build_tag=1.6 to git repo
[Pipeline] withCredentials
[Pipeline] {
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
git version 1.8.3.1
[master cdbb62c] Update docker tag to 1.6
 2 files changed, 2 insertions(+), 2 deletions(-)
To https://****:****@github.com/****/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git
   ce4fc25..cdbb62c  master -> master
Branch master set up to track remote branch master from https://****:****@github.com/****/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git.
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] stage
[Pipeline] { (helm test)
[Pipeline] echo
[INFO] Start helm test
[Pipeline] echo
[INFO] Run helm chart linter
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ /usr/local/bin/helm lint /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx
==> Linting /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, no failures
[Pipeline] echo
[INFO] Dry-run helm chart installation
[Pipeline] echo
Running dry-run deployment
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ /usr/local/bin/helm upgrade --dry-run --debug --install newegg-nginx /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx --set ImageTag=1.6,Replicas=3,Cpu=10m,Memory=128Mi,DomainName=newegg-nginx --namespace=newegg-nginx
[debug] Created tunnel using local port: '39460'

[debug] SERVER: "localhost:39460"

Release "newegg-nginx" does not exist. Installing it now.
[debug] CHART PATH: /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx

NAME:   newegg-nginx
REVISION: 1
RELEASED: Sun Aug 26 01:38:02 2018
CHART: newegg-nginx-1.0.0
USER-SUPPLIED VALUES:
Cpu: 10m
DomainName: newegg-nginx
ImageTag: "1.6"
Memory: 128Mi
Replicas: 3

COMPUTED VALUES:
ContainerPort: 80
Cpu: 10m
DomainName: newegg-nginx
Image: showerlee/nginx-test
ImagePullPolicy: Always
ImageTag: "1.6"
Imagetag: latest
Memory: 128Mi
Replicas: 3
ServicePort: 80
ServiceType: NodePort

HOOKS:
MANIFEST:

---
# Source: newegg-nginx/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: newegg-nginx-newegg-nginx
  labels:    
    app: newegg-nginx-newegg-nginx
    version: 1.0.0
    release: newegg-nginx
spec:
  type: "NodePort"
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: newegg-nginx-newegg-nginx
---
# Source: newegg-nginx/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: newegg-nginx-newegg-nginx
  labels:    
    app: newegg-nginx-newegg-nginx
    version: 1.0.0
    release: newegg-nginx
spec:
  replicas: 3
  template:
    metadata:
      labels:        
        app: newegg-nginx-newegg-nginx
        version: 1.0.0
        release: newegg-nginx
    spec:
      containers:
        - name: newegg-nginx
          image: "showerlee/nginx-test:1.6"
          imagePullPolicy: "Always"
          ports:
            - containerPort: 80
              protocol: TCP
          resources:
            requests:
              cpu: "10m"
              memory: "128Mi"
---
# Source: newegg-nginx/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: newegg-nginx-newegg-nginx
  labels:    
    app: newegg-nginx-newegg-nginx
    version: 1.0.0
    release: newegg-nginx
spec:
  rules:
  - host: newegg-nginx.buyabs.corp
    http:
      paths:
      - path: /
        backend:
          serviceName: newegg-nginx-newegg-nginx
          servicePort: 80
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (helm deploy)
[Pipeline] input
Do you approve to deploy?
Proceed or Abort
Approved by admin
[Pipeline] echo
[INFO] Start helm deployment
[Pipeline] echo
Running deployment
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ /usr/local/bin/helm upgrade --install newegg-nginx /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx --set ImageTag=1.6,Replicas=3,Cpu=10m,Memory=128Mi,DomainName=newegg-nginx --namespace=newegg-nginx
Release "newegg-nginx" does not exist. Installing it now.
NAME:   newegg-nginx
LAST DEPLOYED: Sun Aug 26 01:38:24 2018
NAMESPACE: newegg-nginx
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Ingress
NAME                       HOSTS                     ADDRESS  PORTS  AGE
newegg-nginx-newegg-nginx  newegg-nginx.buyabs.corp  80       1m

==> v1/Pod(related)
NAME                                        READY  STATUS             RESTARTS  AGE
newegg-nginx-newegg-nginx-56c478c888-6n6rt  0/1    ContainerCreating  0         1m
newegg-nginx-newegg-nginx-56c478c888-hsbcp  0/1    ContainerCreating  0         1m
newegg-nginx-newegg-nginx-56c478c888-vdfgb  0/1    ContainerCreating  0         1m

==> v1/Service
NAME                       TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)       AGE
newegg-nginx-newegg-nginx  NodePort  10.97.200.124  <none>       80:32239/TCP  1m

==> v1beta1/Deployment
NAME                       DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
newegg-nginx-newegg-nginx  3        3        3           0          1m


[Pipeline] echo
Application newegg-nginx successfully deployed. Use helm status newegg-nginx to check
[Pipeline] echo
[INFO] Deployment Finished...
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline
Finished: SUCCESS

7.验证结果

这样我们就成功的利用Jenkins集成k8s+helm实现将我们的静态网站部署到我们的kubernetes集群中.

大功告成...

最近研究了下kubernetes用的比较火的Helm, Helm作为一个包管理工具, 它把Kubernetes资源(比如deployments、services或 ingress等) 打包到一个chart中，方便我们将其chart保存到chart仓库用来存储和分享, Helm支持发布应用配置的版本管理, 使发布可配置, 它最终简化了Kubernetes部署应用的版本控制、打包、发布、删除、更新等操作。

其实Helm和我们的ansible playbook有一些类似的地方就是, 它支持变量预定义, 使我们每一个kube脚本将一些重复的配置使用变量代替, 方便我们对一个project release的管理和批量部署, 升级, 回滚等操作.

Let's roll out...

安装环境

Local Desktop: MacOS

Virtual Machine: Virtual Box

Virtual System: CentOS 7.4

Kubernetes: Kubernetes1.9

Docker: 17.03.2-ce

Helm: helm-v2.7.0

kube-master 10.110.16.10

kube-node-1 10.110.16.11

一. 系统环境配置

1.关闭SELINUX和firewall

# vi /etc/sysconfig/selinux

...
SELINUX=disabled 
...

# setenforce 0

# systemctl stop firewalld  && systemctl disable firewalld

2.安装k8s环境.

http://www.showerlee.com/archives/2200

二. Helm配置

1.Helm安装

# wget https://storage.googleapis.com/kubernetes-helm/helm-v2.7.0-linux-amd64.tar.gz

# tar -zxvf helm-v2.7.0-linux-amd64.tar.gz

# mv linux-amd64/helm /usr/local/bin/

2.添加tiller到k8s service account

# kubectl create serviceaccount --namespace kube-system tiller

# kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

3.使用阿里云tiller镜像以及tiller账户初始化helm, 将tiller部署到k8s deployment下.

# vi ~/.helm/repository/repositories.yaml

Tip: username, password为你的阿里云账号密码

apiVersion: v1
generated: 2018-04-13T23:48:19.490774427-04:00
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/stable-index.yaml
  certFile: ""
  keyFile: ""
  name: stable
  password: "password"
  url: https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
  username: "username"
- caFile: ""
  cache: /root/.helm/repository/cache/local-index.yaml
  certFile: ""
  keyFile: ""
  name: local
  password: ""
  url: http://127.0.0.1:8879/charts
  username: ""

#  helm init --service-account tiller --upgrade --tiller-image=registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.7.0

Tip: 这里helm可以理解为一个操作tiller服务的客户端, tiller作为部署到k8s下的一个deployment, 负责去将我们的chart脚本解析给k8s去做进一步的部署工作.

4.检查tiller是否部署到k8s

# kubectl get pods --namespace kube-system

NAME                                  READY     STATUS    RESTARTS   AGE
etcd-kube-master                      1/1       Running   0          26d
kube-apiserver-kube-master            1/1       Running   0          26d
kube-controller-manager-kube-master   1/1       Running   1          26d
kube-dns-6f4fd4bdf-54smn              3/3       Running   0          26d
kube-flannel-ds-gwl2z                 1/1       Running   0          26d
kube-flannel-ds-m754s                 1/1       Running   0          26d
kube-proxy-697qx                      1/1       Running   0          26d
kube-proxy-cvfd9                      1/1       Running   0          26d
kube-scheduler-kube-master            1/1       Running   1          26d
tiller-deploy-cf797bfbf-rnk4k         1/1       Running   0          1h

5.创建一个chart范例

# helm create helm-chart

# tree ./helm-chart

./helm-chart
├── charts
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── ingress.yaml
│   ├── NOTES.txt
│   └── service.yaml
└── values.yaml

Tip: 可以看到helm默认创建了一个chart表结构, 这里的templates下面放的大部分为k8s的部署脚本, values.yaml和chart.yaml为主要的参数文件存放一些变量供k8s yaml文件调用, 有需要的小伙伴可以将自己的k8s脚本与默认进行替换.

6.检查chart语法

# helm lint ./helm-chart

7.使用默认chart部署到k8s

# helm install --name example1 ./helm-chart --set service.type=NodePort

Tip: 这里 --name命名我们这个chart release的名称, --set service.type=NodePort为将我们的任意node的ip映射到我们部署的pod, 以供访问.

# helm install --name example1 ./helm-chart --set service.type=NodePort
NAME:   example1
LAST DEPLOYED: Sat Apr 14 01:08:16 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                 TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)       AGE
example1-helm-chart  NodePort  10.105.111.66  <none>       80:25146/TCP  0s

==> v1beta1/Deployment
NAME                 DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
example1-helm-chart  1        1        1           0          0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
example1-helm-chart-7975cbf9b7-86vx5  0/1    ContainerCreating  0         0s


NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services example1-helm-chart)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

我们可以使用上面的NOTES去访问我们的部署网站

# curl 10.110.16.10:25146

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

8.查看当前的部署列表

# helm ls

NAME    	REVISION	UPDATED                 	STATUS  	CHART           	NAMESPACE
example1	1       	Sat Apr 14 01:08:16 2018	DEPLOYED	helm-chart-0.1.0	default

# kubectl get deployment

NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
example1-helm-chart   1         1         1            1           4m

9.打包chart

# helm package ./helm-chart --debug

10.使用包去做release部署

# helm install --name example2 helm-chart-0.1.0.tgz --set service.type=NodePort

11.升级当前release

# helm upgrade example2 ./helm-chart

12.回滚当前release

# helm rollback example2 1

13.删除该release

# helm delete example2

# helm del --purge example2

14.查看release历史删除记录

Tip: 如果删除时未使用--purge参数可查看删除记录

# helm ls --deleted -d

NAME    	REVISION	UPDATED                 	STATUS 	CHART           	NAMESPACE
example2	2       	Sat Apr 14 00:14:54 2018	DELETED	helm-chart-0.1.0	default

这里作者就不继续介绍helm chart的一些语法结构了, 有需要的小伙伴可以直接访问Helm官方去查看相关文档

https://docs.helm.sh

Finished...

安装环境

System: CentOS 7.4

Kubernetes: Kubernetes1.9

Docker: 17.03.2-ce

kube-master 10.110.16.10

kube-node-1 10.110.16.11

一. NFS配置:

1. NFS依赖包安装

在Master与Node分别安装NFS组件

# yum install nfs-utils -y

Tip: 这里需保证nfs-utils安装到所有master和node中, 否则容器挂载NFS时会报错.

2. 为Master下mysql data和wordpress源码配置NFS共享目录

# systemctl enable nfs-server && systemctl start nfs-server

# mkdir -p /kube/mysql-db

# mkdir -p /kube/wordpress

# chown nfsnobody:nfsnobody /kube/mysql-db

# chown nfsnobody:nfsnobody /kube/wordpress

# chmod 755 /kube/mysql-db

# chmod 755 /kube/wordpress

# echo -e "/kube/mysql-db    kube-*(rw,sync,no_subtree_check,no_root_squash)" > /etc/exports

# echo -e "/kube/wordpress    kube-*(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports

Tip: 这里kube-*限制只有kube相关的server才能连接Master下NFS共享目录, no_root_squash参数保证wordpress-mysql pod在初始化mysql配置的时候向在其下挂载的/var/lib/mysql目录有写入权限

3.应用配置

# exportfs -a

二. Persistent volume配置

1.为mysql data与wordpress源码存储创建Persistent volume
# kubectl create -f mysql-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    app: mysql
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /kube/mysql-db
    server: kube-master

# kubectl create -f wordpress-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: wp-pv
  labels:
    app: wordpress
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /kube/wordpress
    server: kube-master

2.创建存放mysql data的PVC

# kubectl create -f mysql-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-pv-claim
  labels:
    app: mysql
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

3.创建存放wordpress源码的PVC
# kubectl create -f wordpress-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

查看绑定

# kubectl get pvc

NAME             STATUS    VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-pv-claim   Bound     mysql-pv   5Gi        RWO                           3m
wp-pv-claim      Bound     wp-pv      5Gi        RWO                           6s

三. Secret配置

1.创建mysql root password

# kubectl create secret generic mysql-pass --from-literal='password=countonme'

四. Deployment配置

1.部署mysql deployment with PVC
# kubectl create -f mysql-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

2.部署wordpress deployment with PVC

# kubectl create -f wordpress-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
    spec:
      containers:
      - image: wordpress:4.8-apache
        name: wordpress
        env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql
        - name: WORDPRESS_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          claimName: wp-pv-claim

3.Service配置

Tip: 这里我们开启了node IP的80端口的外部访问权限, 可以方便我们直接利用主机去访问虚拟机任意Node地址从而登录我们的Wordpress网站.

# kubectl create -f wp-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
      nodePort: 80
  selector:
    app: wordpress
    tier: frontend
  type: NodePort

Tip: 这里service定义的name: wordpress-mysql保证我们wordpress-deployment.yaml定义的如下环境变量可以作为有效的域名成功去访问我们的mysql容器, 保证网站服务器与数据库服务器的通讯.

env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql

五. 验证结果

1.访问wordpress主页

这里我们可以直接在浏览器访问任意node的IP地址从而进入wordpress主页

添加相关信息并初始化安装WordPress

完成安装

后台

主页

2.查看NFS主机对在容器里的mysql data与wordpress root dir的目录挂载.

有兴趣的小伙伴可以直接访问如下代码仓库去下载本文相关代码:

https://git.showerlee.com/showerlee/kube-deploy/tree/master/wordpress-mysql

Finished...

后记:

如果我们使用helm包管理去部署wordpress, 将大大简化我们的工作量.

这里我的代码仓库提供了wordpress chart部署脚本, 以下是详细的部署步骤:

Prerequisite:

Kubernetes cluster setup

http://www.showerlee.com/archives/2200

Helm setup

http://www.showerlee.com/archives/2455

Helm deployment:

# git clone git@git.showerlee.com:showerlee/kube-deploy.git

# cd kube-deploy

# kubectl create secret generic mysql-pass --from-literal='password=countonme'

# helm install --name wordpress-mysql ./wordpress-helm-chart --set service.type=NodePort

NAME:   wordpress-mysql
LAST DEPLOYED: Sat Apr 14 03:09:46 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/PersistentVolume
NAME      CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                   STORAGECLASS  REASON  AGE
mysql-pv  5Gi       RWO           Recycle         Bound   default/mysql-pv-claim  1s
wp-pv     5Gi       RWO           Recycle         Bound   default/wp-pv-claim     1s

==> v1/PersistentVolumeClaim
NAME            STATUS  VOLUME    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
mysql-pv-claim  Bound   mysql-pv  5Gi       RWO           1s
wp-pv-claim     Bound   wp-pv     5Gi       RWO           1s

==> v1/Service
NAME             TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)    AGE
wordpress-mysql  ClusterIP  None           <none>       3306/TCP   1s
wordpress        NodePort   10.110.14.233  <none>       80:80/TCP  1s

==> v1/Deployment
NAME             AGE
wordpress-mysql  1s
wordpress        1s


NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services wordpress-mysql-wordpress-helm-chart)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

Secrets是一个包含敏感数据的对象，例如我们常用的密码，令牌或密钥等,  我们编写yaml如果直接明文这些信息则会将我们的敏感信息暴漏在我们的脚本中; 所以将其放置在Secret对象中可以更好地控制它的使用方式，并降低意外暴露的风险。

Pod可以引用我们事先创建好的Secrets键值对到环境变量, 通过获取环境变量的键值对动态更新我们Pod的环境配置, 从而实现动态配置更新.

1. 创建一个secret

# kubectl create secret generic secret-demo --from-literal='password=countonme'

2. 查看创建好的secret

# kubectl get secret secret-demo

NAME          TYPE      DATA      AGE
secret-demo   Opaque    1         13s

3.创建一个Pod并引用这个secret

# vi secret-env-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    env:
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: secret-demo
          key: password

# kubectl create -f secret-env-pod.yaml

4.查看secret

# kubectl describe secret

5.查看变量是否引入Pod

# kubectl exec -ti httpd-pod env

PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=httpd-pod
TERM=xterm
PASSWORD=countonme
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
HTTPD_PREFIX=/usr/local/apache2
NGHTTP2_VERSION=1.18.1-1
OPENSSL_VERSION=1.0.2l-1~bpo8+1
HTTPD_VERSION=2.4.29
HTTPD_SHA256=777753a5a25568a2a27428b2214980564bc1c38c1abf9ccc7630b639991f7f00
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=
HOME=/root

可以看到Pod的环境变量里已经引入一组键值对PASSWORD=countonme

6.向Pod挂载目录写入secret文件.

# vi secret-vol-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod-secret-vol
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    volumeMounts:
    - name: secret
      mountPath: "/mnt"
      readOnly: true
  volumes:
  - name: secret
    secret:
      secretName: secret-demo

# kubectl create -f secret-vol-pod.yaml

# kubectl exec -it httpd-pod-secret-vol cat /mnt/password

countonme

可以看到该Pod下面有一个文件名为password, 内容为countonme的文本文件. 

Config Map

1.创建config map

# vi cfgmap-demo.yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  name: cfgmap-demo

# kubectl create -f cfgmap-demo.yaml

2.查看config map

# kubectl get configmap cfgmap-demo -o yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  creationTimestamp: 2018-02-24T07:11:01Z
  name: cfgmap-demo
  namespace: default
  resourceVersion: "1064654"
  selfLink: /api/v1/namespaces/default/configmaps/cfgmap-demo
  uid: de9248d1-1931-11e8-9e24-00163e0e24bf

3. 修改config map

# vi cfgmap-demo.yaml

添加一行键值对

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
  http_port: "80"
kind: ConfigMap
metadata:
  name: cfgmap-demo

更新config map

# kubectl replace -f cfgmap-demo.yaml

查看更新后的config map

# kubectl get configmap cfgmap-demo -o yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_port: "80"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  creationTimestamp: 2018-02-24T07:11:01Z
  name: cfgmap-demo
  namespace: default
  resourceVersion: "1065520"
  selfLink: /api/v1/namespaces/default/configmaps/cfgmap-demo
  uid: de9248d1-1931-11e8-9e24-00163e0e24bf

4.创建一个Pod并引用这个config map

# vi cfgmap-env-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: cfgmap-httpd-pod
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    envFrom:
    - configMapRef:
        name: cfgmap-demo

# kubectl create -f cfgmap-env-pod.yaml

5.查看config map的键值对是否引入Pod

# kubectl exec -ti cfgmap-httpd-pod env

PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=cfgmap-httpd-pod
TERM=xterm
db_port=3306
http_port=80
http_url=http://www.example.com
database=db.example.com
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
HTTPD_PREFIX=/usr/local/apache2
NGHTTP2_VERSION=1.18.1-1
OPENSSL_VERSION=1.0.2l-1~bpo8+1
HTTPD_VERSION=2.4.29
HTTPD_SHA256=777753a5a25568a2a27428b2214980564bc1c38c1abf9ccc7630b639991f7f00
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=
HOME=/root

可以看到我们Config map下的所有键值对已经成功引入Pod环境变量.

相关代码:

https://git.showerlee.com/showerlee/kube-deploy

Finished...

持久化卷下PV和PVC概念:

Persistent Volume（PV）是由管理员设置的存储，它是群集的一部分。就像节点是集群中的资源一样，PV 也是集群中的资源。 PV 是 Volume 之类的卷插件，但具有独立于使用 PV 的 Pod 的生命周期。此 API 对象包含存储实现的细节，即 NFS、iSCSI 或特定于云供应商的存储系统

PersistentVolumeClaim（PVC）是用户存储的请求。它与 Pod 相似。Pod 消耗节点资源，PVC 消耗 PV 资源。Pod 可以请求特定级别的资源（CPU 和内存）。PVC声明可以请求特定的大小和访问模式（例如，可以以读/写一次或 只读多次模式挂载）

它和普通Volume的区别是什么呢？

普通Volume和使用它的Pod之间是一种静态绑定关系，在定义Pod的文件里，同时定义了它使用的Volume。Volume是Pod的附属品，我们无法单独创建一个Volume，因为它不是一个独立的K8S资源对象。

如何简单理解持久化卷?

我们需要首先创建一个独立的持久化卷(PV)资源对象, 然后创建一个与PV绑定的PVC存储请求, 这个请求会事先定义accessModes, resources等资源配置, 最终我们会在Pod中挂载定义好的PVC以供我们数据存储使用

Let's start...

一. NFS安装配置

我们这里利用NFS去实现k8s持久化卷的配置

1,安装NFS server

# yum install nfs-utils -y

2.启动NFS服务

# systemctl enable nfs-server

# systemctl start nfs-server

3.配置NFS共享目录

# mkdir /srv/pv-demo

# chown nfsnobody:nfsnobody /srv/pv-demo

# chmod 755 /srv/pv-demo

# echo -e "/srv/pv-demo    kube-master(rw,sync)" > /etc/export

4.生效共享目录

# exportfs -a

因为资源有限, 我们最终在Master上创建一个NFS共享目录/srv/pv-demo, 以供我们后面的持久化卷使用, 有富裕的小伙伴可以创建一台与kube-master同一网段的独立server去充当NFS服务器, 

二. Persistent Volume配置

1.创建Persistent Volume

# vi pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-demo
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /srv/pv-demo
    server: kube-master

Tip: 这里的定义卷的大小为5G, 使用的accessmodes为ReadWriteOnce, PVC policy为Recycle, NFS共享目录为我们之前在master创建好的/srv/pv-demo, server为我们定义好的本地host kube-master

# kubectl create -f pv.yaml

2. 查看PV

# kubectl get pv pv-demo

NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM              STORAGECLASS   REASON    AGE
pv-demo   5Gi        RWO            Recycle          Bound     default/pvc-demo                            1h

3.创建Persistent Volume Claim

Tip: 这里PVC可以理解为在PV请求的资源, 也就是说所有我们的数据都会保存在PVC里, 任何PVC的删除操作都会清除我们存储在这里的数据.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

# kubectl create -f pvc.yaml

4.查看PVC

# kubectl get pvc pvc-demo

NAME       STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-demo   Bound     pv-demo   5Gi        RWO                           1h

5.创建一个Pod并使用该PVC

# vi pvpod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod
spec:
  containers:
  - image: httpd
    name: httpd-pod
    imagePullPolicy: Always
    volumeMounts:
    - mountPath: "/usr/local/apache2/htdocs/"
      name: httpd-volume
  volumes:
    - name: httpd-volume
      persistentVolumeClaim:
        claimName: pvc-demo

Tip: 这里需要保证claimName的值与我们之前创建的PVC name一致.

# kubectl create -f pvpod.yaml

Tip: 这里我们将PVC挂载到Pod的Apache根目录"/usr/local/apache2/htdocs/", 用来最终测试效果.

6. 查看Pod是否挂载PVC

# kubectl describe pv

Name:            pv-demo
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller=yes
StorageClass:
Status:          Bound
Claim:           default/pvc-demo
Reclaim Policy:  Recycle
Access Modes:    RWO
Capacity:        5Gi
Message:
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    kube-master
    Path:      /srv/pv-demo
    ReadOnly:  false
Events:        <none>

# kubectl describe pods

Name:         httpd-pod
Namespace:    default
Node:         kube-master/172.17.2.153
Start Time:   Fri, 23 Feb 2018 15:38:55 +0800
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           10.244.0.46
Containers:
  httpd-pod:
    Container ID:   docker://b7e5fd2732864934b732fdbd4bb24b3ccc8949c2e9d8832a36e271f2ee350b2b
    Image:          httpd
    Image ID:       docker-pullable://httpd@sha256:6e61d60e4142ea44e8e69b22f1e739d89e1dc8a2764182d7eecc83a5bb31181e
    Port:           <none>
    State:          Running
      Started:      Fri, 23 Feb 2018 15:38:59 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/local/apache2/htdocs from httpd-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bnkxx (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  httpd-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc-demo
    ReadOnly:   false
  default-token-bnkxx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bnkxx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                  Message
  ----    ------                 ----  ----                  -------
  Normal  Scheduled              58m   default-scheduler     Successfully assigned httpd-pod to kube-master
  Normal  SuccessfulMountVolume  58m   kubelet, kube-master  MountVolume.SetUp succeeded for volume "default-token-bnkxx"
  Normal  SuccessfulMountVolume  58m   kubelet, kube-master  MountVolume.SetUp succeeded for volume "pv-demo"
  Normal  Pulling                58m   kubelet, kube-master  pulling image "httpd"
  Normal  Pulled                 58m   kubelet, kube-master  Successfully pulled image "httpd"
  Normal  Created                58m   kubelet, kube-master  Created container
  Normal  Started                58m   kubelet, kube-master  Started container

通过返回打印的信息我们可以看到PVC已经成功mount到我们定义好的Pod的apache根目录下

7. 向Pod下apache根目录写入index.html

# kubectl exec -ti httpd-pod -- /bin/sh -c "echo 'This is a persistent volume from httpd-pod' > /usr/local/apache2/htdocs/index.html"

8.确认文件写入

# kubectl exec -ti httpd-pod -- cat /usr/local/apache2/htdocs/index.html 

This is a persistent volume from httpd-pod

9. 删除并重新创建Pod来验证数据是否会随Pod销毁而丢失.

# kubectl delete pod httpd-pod
# kubectl create -f pvpod.yaml
# kubectl exec -ti httpd-pod -- cat /usr/local/apache2/htdocs/index.html 

This is a persistent volume from httpd-pod

可以看到我们之前写入的index.html仍旧存储在PVC中, 证明其不会随Pod销毁而丢失.

10.查看Pod内网IP

# kubectl get pods -o wide

NAME        READY     STATUS    RESTARTS   AGE       IP            NODE
httpd-pod   1/1       Running   0          1m        10.244.0.46   kube-master

11.利用curl验证写入的index.html

# curl 10.244.0.46

This is a persistent volume from httpd-pod

这里我们成功在Pod下将PVC挂载到apache的家目录, 并返回HTML返回内容.

本文相关代码:

https://git.showerlee.com/showerlee/kube-deploy

大功告成...

Pod:

Pod是一组紧密关联的容器集合，它们共享PID、IPC、Network和UTS namespace，是Kubernetes调度的基本单位。Pod的设计理念是支持多个容器在一个Pod中共享网络和文件系统，可以通过进程间通信和文件共享这种简单高效的方式组合完成服务.

缺点: 不支持高并发, 高可用, 当Pod当机后无法自动恢复.

1.创建Pod

# vi pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: demo
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always

# kubectl create -f pod.yaml

2.查看Pod

# kubectl get pods

NAME    READY     STATUS    RESTARTS   AGE
demo    1/1       Running      0       8d

# kubectl describe pods

...

3.删除Pod

# kubectl delete pod demo

Replicaset:

Replicaset在继承Pod的所有特性的同时, 它可以利用预先创建好的模板定义副本数量并自动控制, 通过改变Pod副本数量实现Pod的扩容和缩容

缺点: 无法修改template模板, 也就无法发布新的镜像版本

1.创建Replicaset

# vi replicaset.yaml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: demo-rc
  labels:
    app: demo-rc
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-rc
  template:
    metadata:
      labels:
        app: demo-rc
    spec:
      containers:
      - name: httpd
        image: httpd
        imagePullPolicy: Always

# kubectl create -f replicaset.yaml

2.查看replicaset

# kubectl get replicaset

NAME      READY     STATUS    RESTARTS   AGE
demo-rc    1/1       Running      0       8d

# kubectl describe replicaset

...

3.删除replicaset

# kubectl delete replicaset demo-rc

Deployment

Deployment在继承Pod和Replicaset的所有特性的同时, 它可以实现对template模板进行实时滚动更新并具备我们线上的Application life circle的特性.

1.创建Deployment

# vi deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd-deployment
  labels:
    app: httpd-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpd-demo
  template:
    metadata:
      labels:
        app: httpd-demo
    spec:
      containers:
      - name: httpd
        image: httpd
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        env:
        - name: VERSION
          value: "v1"

# kubectl create -f deployment.yaml

2.查看Deployment

# kubectl get deployment

NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
httpd-deployment   2         2         2            2           8d

# kubectl get pods -o wide

NAME                               READY     STATUS    RESTARTS   AGE       IP            NODE
httpd-deployment-956697567-8mqch   1/1       Running   0          8d        10.244.0.36   kube-master
httpd-deployment-956697567-wcbs6   1/1       Running   0          8d        10.244.0.37   kube-master

# kubectl describe deployment

...

3.更新deployment

通过此命令可以呼出vi编辑器对模板进行编辑.

# kubectl edit -f deployment.yaml

通过此命令使当前编辑结果生效.

# kubectl apply -f deployment.yaml

再次查看可以看到老版本的deployment已经下架, 新版本的已经生效.

# kubectl get deployment

NAME                          DESIRED   CURRENT   READY     AGE
httpd-deployment-6b98d94474   0         0         0         1m
httpd-deployment-956697567    2         2         2         7m

4.扩容与缩容

可以修改replicas的赋值对deployment进行扩容与缩容

#  kubectl scale deployment/httpd-deployment --replicas=1

5.删除deployment

# kubectl delete deployment httpd-deployment

Lable

Label是attach到Pod的一对键/值对，用来传递用户定义的属性。比如，你可能创建了一个"tier"和“app”标签，通过Label（tier=frontend, app=myapp）来标记前端Pod容器，使用Label（tier=backend, app=myapp）标记后台Pod。然后可以使用Selectors选择带有特定Label的Pod，让具体某一个Pod或者Deployment去使用某一个Service实现特定的网络配置.

Service

Service是应用服务的抽象，通过labels为应用提供负载均衡和服务发现。匹配labels的Pod IP和端口列表组成endpoints，由kube-proxy负责将服务IP负载均衡到这些endpoints上。
每个Service都会自动分配一个cluster IP（仅在集群内部可访问的虚拟地址）和DNS名，其他容器可以通过该地址或DNS来访问服务，而不需要了解后端容器的运行。

1.更改NodePort限制

Kubernetes默认对外的NodePort限制范围为30000-32767, 这里如果要使用一些常用的端口(80, 8080, 443)需将这个范围放大.

# vi /etc/kubernetes/manifests/kube-apiserver.yaml

在--service-cluster-ip-range与insecure-port间添加如下node port配置

...
- --service-cluster-ip-range=10.96.0.0/12
- --service-node-port-range=0-32767
- --insecure-port=0
....

重启服务

# systemctl restart kubelet

2.创建Service

# vi svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: NodePort
  ports:
    - port: 80
      nodePort: 80
  selector:
    app: httpd-demo

# kubectl create -f svc.yaml

Tip: 如果要对某一Pod或deployment添加对外访问端口, 这里service添加的selector的键值需与之相对应.

3.查看开放端口

# kubectl get svc demo

NAME      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
demo      NodePort   10.100.96.157   <none>        80:80/TCP   1h

# kubectl describe service demo

Name:                     demo
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=httpd-demo
Type:                     NodePort
IP:                       10.100.96.157
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  80/TCP
Endpoints:                10.244.0.36:80,10.244.0.37:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

4.验证外网是否可以访问

本文相关代码:

https://git.showerlee.com/showerlee/kube-deploy

Finished...

最近研究了一下目前比较火的Kubernetes(k8s), 重点关注了下它最新的1.9版本, 这个版本较老的1.1版本的确简化了很多配置, 它利用kubeadm这个工具对全局进行批量化部署, 减轻了我们初学者起步的学习成本. 

目前主流的安装k8s系统平台有Centos7和ubuntu, 这里笔者因为对centos有常年的运维开发经验, 所以就选择前者.

另外官方已经在近期将kubernetes1.1版本的centos7配置从官网移除, 所以建议大家使用1.9版本完成所有的安装部署.

本文档推荐给大家的原因是因为目前网上基本上没有一个较为完整和正确率较高的k8s的安装文档, 笔者因此整合了网上零散的k8s资源, 给大家提供一个较为靠谱的离线安装k8s 1.9版本的安装范例.

什么是Kubernates?

Kubernetes 用于自动部署、扩展和管理容器化（containerized）应用程序的开源系统。它旨在提供“跨主机集群的自动部署、扩展以及运行应用程序容器的平台”。它支持一系列容器工具, 目前主流会使用Docker作为他的主流配置容器.

为什么要使用离线安装呢?

因为kubeadm默认要从google仓库下载镜像，但目前国内无法访问google仓库，所以这里从网上找到1.9的离线安装包，大家只需要将离线包的镜像导入到相应节点即可.

Let's start...

安装环境

Local Desktop: MacOS

Virtual Machine: Virtual Box

Virtual System: CentOS 7.4

Kubernetes: Kubernetes1.9

Docker: 17.03.2-ce

kube-master 10.110.16.10

kube-node-1 10.110.16.11

一. 系统环境配置

(Master Node都需要配置)

1.下载离线安装包

链接: https://pan.baidu.com/s/1c2O1gIW 密码: 9s92

下载到本地后, 上传到虚拟机root根目录

# tar jxvf k8s_images.tar.bz2

2. 安装依赖

Tip: 这里需要更新CentOS7内核到最新版本才能打开centos的路由功能以供k8s使用, 更新完毕需要重启系统使kernel新版本生效

# yum install policycoreutils-python libtool-ltdl libseccomp device-mapper-libs kernel ntpdate

# reboot

3.绑定本地host

# echo "10.110.16.10 kube-master" >> /etc/hosts

# echo "10.110.16.11 kube-node-1" >> /etc/hosts

4.添加kube-master到kube-node-1的秘钥认证(仅Master需要配置)

# ssh-keygen

# ssh-copy-id kube-node-1

5.关闭SELINUX和firewall

# vi /etc/sysconfig/selinux

...
SELINUX=disabled 
...

# setenforce 0

# systemctl stop firewalld  && systemctl disable firewalld

6.路由配置

# modprobe br_netfilter

# echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf

# echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf

# sysctl -p

7.关闭Swap分区

大家在安装CentOS7时如果配置了swap分区, 这里由于k8s禁止系统开启此功能, 需要提前进行关闭, 没有配置swap请无视.

查看swap分区目录

# swapon -s

Filename                                Type            Size    Used    Priority
/swapfile                               file    1996796 1214364 -1

关闭swap分区

# swapoff /swapfile

注释掉swap分区, 禁止开机启动swap

# vi /etc/fstab

...
#/dev/mapper/centos-swap swap                    swap    defaults        0 0
...

8.开启系统时间同步

# systemctl enable ntpdate && systemctl start ntpdate

二. Docker配置

(Master Node都需要配置)

1.安装docker-ce17.03

# rpm -ihv docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
# rpm -ivh docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm

2.修改Docker使用阿里的镜像

Tip: 访问https://cr.console.aliyun.com/#/accelerator注册账号, 在镜像加速器里获取个人的仓库地址, 并填入如下配置文件

# mkdir /etc/docker

# vi /etc/docker/daemon.json

...
{
  "registry-mirrors": ["https://xxxxxxx.mirror.aliyuncs.com"]
}
...

3.重启docker并设置开机启动
# systemctl daemon-reload
# systemctl restart docker
# systemctl enable docker

4.本地载入我们的离线镜像

# cd /root

# mv /root/k8s_images/docker_images/etcd-amd64\:v3.1.10.tar /root/k8s_images/docker_images/etcd-amd64_v3.1.10.tar

# vi load_image.sh

docker load < /root/k8s_images/docker_images/etcd-amd64_v3.1.10.tar
docker load </root/k8s_images/docker_images/flannel_v0.9.1-amd64.tar
docker load </root/k8s_images/docker_images/k8s-dns-dnsmasq-nanny-amd64_v1.14.7.tar
docker load </root/k8s_images/docker_images/k8s-dns-kube-dns-amd64_1.14.7.tar
docker load </root/k8s_images/docker_images/k8s-dns-sidecar-amd64_1.14.7.tar
docker load </root/k8s_images/docker_images/kube-apiserver-amd64_v1.9.0.tar
docker load </root/k8s_images/docker_images/kube-controller-manager-amd64_v1.9.0.tar
docker load </root/k8s_images/docker_images/kube-scheduler-amd64_v1.9.0.tar
docker load < /root/k8s_images/docker_images/kube-proxy-amd64_v1.9.0.tar
docker load </root/k8s_images/docker_images/pause-amd64_3.0.tar
docker load < /root/k8s_images/kubernetes-dashboard_v1.8.1.tar

# sh load_image.sh

三.Kubernetes配置

1.安装kubelet kubeadm kubectl(Master Node都需要配置)

# cd k8s_images
# rpm -ivh socat-1.7.3.2-2.el7.x86_64.rpm
# rpm -ivh kubernetes-cni-0.6.0-0.x86_64.rpm  kubelet-1.9.9-9.x86_64.rpm kubectl-1.9.0-0.x86_64.rpm
# rpm -ivh kubeadm-1.9.0-0.x86_64.rpm

2.重启kubelet并设置开机启动(Master Node都需要配置)

# systemctl enable kubelet && sudo systemctl start kubelet

3.修改docker配置文件中的Cgroup Driver参数与k8s一致(Master Node都需要配置)

# docker info |grep "Cgroup Driver"
如果"Cgroup Driver"value为 "cgroupfs", 将其值写入kubeadm配置文件
# vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

...
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver="cgroupfs"
...

# systemctl daemon-reload && systemctl restart kubelet

4.Master初始化(Master 需要配置)

# kubeadm reset

# kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.244.0.0/16

Tip: 这里的10.244.0.0/16与默认的配置文件网段一致, 如需调整请在后面的kube-flannel.yml文件做相应改变

如果一切没问题会output一个token返回命令

# kubeadm join --token 288f34.481c8faa5636966f 10.110.16.10:6443 --discovery-token-ca-cert-hash sha256:8036fac3b76e1a0dd189edaa8f7d36f2b51429dd0c0cf7ea0d78e7972d611002

24小时后这个token会失效, 需要重新生成, 使用如下命令进行生成.

# kubeadm token create

# kubeadm join --token "Your token code" "Your master ip address":6443

如果在24小时内忘记了，可以用如下命令获取.

# kubeadm token list

5.设置用户环境变量(Master 需要配置)

如果你使用的是root
# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
# source ~/.bash_profile

如果是非root用户

# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config

测试kubectl版本

# kubectl version

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

6.使用flannel组件进行k8s网络配置(Master 需要配置)

# wget https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

Tip: 这里的yml配置文件内的网段配置需与上面介绍的Master初始化的配置网段"--pod-network-cidr=10.244.0.0/16" 一致

# kubectl create -f kube-flannel.yml

7.查看所有pod详细信息(Master 需要配置)

# kubectl get pod --all-namespaces -o wide

# kubectl describe pods

8.添加Node(Node 需要配置)

从Master初始化输出中获取

# kubeadm join --token 288f34.481c8faa5636966f 10.110.16.10:6443 --discovery-token-ca-cert-hash sha256:8036fac3b76e1e0dd189edaa8f7d36f1b51429dd0c0cf7ea0d78e7972d611002

Tip:这里作为可选, 因为笔者将Master加入Node调度中, 让其同时充当Master和Node的角色, 如果资源有限只有一台测试机的小伙伴这步可以略过.

9.将Master加入Node schedual调度(Master 需要配置)

Tip: 默认Master不会加入Node调度, 这里使用如下命令开启这个限制.

# kubectl taint nodes kube-master node-role.kubernetes.io/master-

如果我们有多个node需要手动去做调度, 从而不让我们的pod进入该node调度列表, 可以使用如下命令:

禁用该node调度

# kubectl cordon kube-node-1

查看是否禁用

# kubectl get node kube-node-1

NAME          STATUS                     ROLES     AGE       VERSION
kube-node-1   Ready,SchedulingDisabled   node    11d       v1.9.0

解禁该node调度

# kubectl uncordon kube-node-1

查看是否解禁

# kubectl get nodes kube-node-1

NAME          STATUS  ROLES     AGE       VERSION
kube-node-1   Ready   node    11d       v1.9.0

10.测试K8s cluster(Master 需要配置)

我们利用k8s创建一个apache的网站实例, 镜像为httpd, 并设置2个副本. 

# kubectl run httpd-app --image=httpd --replicas=2

# kubectl get deployment

NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
httpd-app   2         2         2            2           2h

# kubectl get pods -o wide

NAME                         READY     STATUS    RESTARTS   AGE       IP            NODE
httpd-app-5fbccd7c6c-27nzv   1/1       Running   0          2h        10.244.0.13   kube-master
httpd-app-5fbccd7c6c-n9qs2   1/1       Running   0          2h        10.244.0.12   kube-master

11.测试网站实例(Master 需要配置)

# curl 10.244.0.13

<html><body><h1>It works!</h1></body></html>

# curl 10.244.0.12

<html><body><h1>It works!</h1></body></html>

12.安装kubernetes-dashboard(Master 需要配置)

Tip:默认kubernetes是没有图形管理界面, 我们这里通过添加一个dashboard容器去添加一个可管理的GUI界面

# cd /root/k8s_images

添加dashboard容器

# kubectl create -f kubernetes-dashboard.yaml

创建认证文件

# vi /etc/kubernetes/pki/basic_auth_file

#user,password,userid
admin,admin,2

配置dashboard basic认证

# vi /etc/kubernetes/manifests/kube-apiserver.yaml

添加一行:

- --basic_auth_file=/etc/kubernetes/pki/basic_auth_file

如下:

...
- --etcd-servers=http://127.0.0.1:2379
- --basic_auth_file=/etc/kubernetes/pki/basic_auth_file
image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.0
...

更新认证配置

# kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml (可不执行, 待验证)

# systemctl restart kubelet

将admin账号加入cluster-admin的role, 获取k8s最高系统权限

# kubectl create clusterrolebinding login-on-dashboard-with-cluster-admin --clusterrole=cluster-admin --user=admin

访问dashboard主页

https://10.110.16.10:32666

账号/密码: admin/admin

大功告成...

更多kubernetes组件介绍:

Kubernetes之Pod, Replicaset, Deployment, Label, Service

Kubernetes之Persistent Volume(持久化卷)

Kubernetes之Secrets与Config Maps

Kubernetes之Helm包管理