刚好最近闲下来想在把之前的CICD项目搞的更加完备, 这里就结合这块顺带把Jenkins github webhook给大家整理一个文档, 帮助有需求的小伙伴针对如何使用Jenkins插件仓库下的github plugin与我们常用的github代码进行webhook集成.

DEMO: https://github.com/showerlee/gradle-demo-with-docker

什么是Github Webhook?

顾名思义, 也就是Github项目下Webhook允许使用者触发一个CI系统(例如Jenkins)的URL，这个URL通常由github plugin提供，这个Webhook会侦听特定事件，如推送master push, PR或merge请求。如果项目推送了新代码,  Github会向这个Webhook URL发送包含此次commit的相关POST请求, 并通过Jenkins相关配置触发相应的jenkins job, 实现我们每次的代码提交都能够自动触发一次Jenkins代码构建, 真正打通了我们代码仓库与CI系统的壁垒, 使我们开发人员能够快速进行CICD自动化集成构建部署, 并且在后期的审计过程中, 能够通过每次commit追溯每一次构建内容. 实现commit与构建一一对应.

这里开始我们的具体配置:

1.在Github下添加Jenkins github plungin webhook URL

访问你的代码仓库如下路径

https://github.com/<org>/<repo>/settings/hooks/

点击右上角的Add webhook

按照如图添加我们的Jenkins URL地址, 假设你的Jenkins主页是jenkins.example.com, 这里URL即为如下地址(如有非80端口, 这里也要添加)

这里需要保证jenkins.example.com为可以访问的公网域名且对应的IP地址可公网解析.

本地虚拟机用户不适合此次webhook的教程, 因为github需要与你的jenkins主机通信, 这里本地虚拟机通常都在一个内网, 无法实现公网互通

配置好点击Add webhook保存

2.在Github下创建Personal access token

访问你的代码仓库如下路径

https://github.com/settings/tokens

按照如图配置你的token, 创建好后保存token内容

3. 安装配置Jenkins github plugin

访问http://jenkins.example.com/pluginManager/available 找到github plugin, 并进行安装.

进入http://jenkins.example.com/configure

找到github plugin区域, 按照如图进行配置

这里需要在Credentials下添加之前创建的Personal access token作为我们Github项目向Jenkins webhook URL发送POST请求的凭证.

点击Test connection确认连接正常

点击左下角Save保存Github配置

4.激活Jenkins job使用该插件触发webhook

这里我进入我的一个jenkins pipeline item, 按照如图激活github webhook.

这样子我们就成功的配置了Github到Jenkins的webhook.

这里我提供一个demo项目, 大家可以参考这个项目进行此次的配置.

https://github.com/showerlee/gradle-demo-with-docker

不过作为我们DevOps, 不是所有语言都能拿得起放得下(自黑), 在写自动化脚本过程中势必要跟我们的Jenkins本家groovy脚本语言打交道, 这玩意可是直接能够调用Jenkins API的敲门砖, 但是前提你能玩得转他的API doc, 本铲屎官给Jenkins铲屎已经够意思了, 还让我去研究如果优雅的铲屎, 不留一丝痕迹? 

这里多亏了Docker swam和JCasC, 他们组合起来就能把我们可爱的Jenkins小宠宠瞬间打回一只可以被随时替代的小牛, 我们可以不必一天在给Jenkins遛弯的过程, 怕他走丢啦, 怕他肚子吃坏啦, 怕他被人欺负了, 给他打一针JCasC, 扔进Docker Swarm牛圈, 瞬间变成一只可被随时打入冷宫的小牛, 想怎么玩就怎么玩, 玩坏了分分钟扔掉并直接给他再克隆一只, 是不是很神奇?

闲话不多说, 其实宠物和牛对应的就是可变基础设施与不可变基础设施, 具体的基础设施我们可以拿vm与docker做比较.

可变基础设施vs不可变基础设施:

可变基础设施: 服务器会不断更新和修改。使用此类基础架构的工程师和管理员可以通过SSH连接到他们的服务器，手动升级或降级软件包，逐个服务器地调整配置文件，以及将新代码直接部署到现有服务器上。换句话说，这些服务器是可变的; 它们可以在创建后进行更改。

不可变基础设施: 他们部署了服务器之后决不会被修改。如果需要以任何方式更新，修复或修改某些内容，则会根据具有相应更改的公共映像构建新服务器以替换旧服务器。经过验证后，它们就会投入使用，而旧的则会退役。

比起传统的虚拟机基础环境, 使用docker本身就能让我们节省资源的同时快速构建我们的应用, 他的持续集成、版本控制、可移植性、隔离性和安全性能够让我们摆脱传统的可变基础设施(宠物), 让我们不必去在对他创建后进行任何配置更改, 只需要把他当做牛一样, 所有的配置都在牛(镜像)使用前进行版本配置变更, 使用时直接扔到docker swarm, 做到召之即来挥之即去, 降低我们的运维成本.

这里的Jenkins CasC plugin对我们的Jenkins在不可变基础设施中又做了一次二次优化, 他让我们原本需要去在我们的自动化脚本内嵌入groovy语言变成了历史, 我们只需要按照插件的语法格式编写一个human readable的yaml文件并嵌入Jenkins docker启动, 他就能自动化帮我们完成装插件, 配slave节点, 配环境变量, 配CI工具, 配后台管理权限等工作, 真正做到快速版本迭代更新, 节省了我们大量版本更新时花费的人力物力.

OK, Let's rack and roll...

安装环境

system: CentOS 7.4 x64

Docker: 17.12.0-ce

Jenkins docker image: 2.164.1

Docker manager node:  192.168.0.102

Docker worker1 node:   192.168.0.102

Docker worker2 node:   192.168.0.104

一. 系统环境配置

1.关闭iptables和selinux

# su - root

# systemctl stop firewalld

# setenforce 0

# vi /etc/sysconfig/selinux

修改

SELINUX=disabled

2.在三台节点分别添加本地host DNS

# vi /etc/hosts

192.168.0.102  manage
192.168.0.103  worker1
192.168.0.104  worker2

3.更改对应系统主机名
Manager:
# hostnamectl set-hostname manager
Worker1:
# hostnamectl set-hostname worker1
Worker2
# hostnamectl set-hostname worker2

二. 安装Docker Engine

1. 下载yum镜像源

# wget https://download.docker.com/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker.repo

2. 安装docker engine

# yum install docker-ce –y

3.启动docker service 

# systemctl start docker
# systemctl enable docker

4.创建docker manager节点(manager)

# docker swarm init --advertise-addr 192.168.0.102

Swarm initialized: current node (viwovkb0bk0kxlk98r78apopo) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-3793hvb71g0a6ubkgq8zgk9w99hlusajtmj5aqr3n2wrhzzf8z-    1s38lymnir13hhso1qxt5pqru 192.168.0.102:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

5.加入worker节点到manager节点(worker1, worker2)

# docker swarm join --token SWMTKN-1-3793hvb71g0a6ubkgq8zgk9w99hlusajtmj5aqr3n2wrhzzf8z-1s38lymnir13hhso1qxt5pqru 192.168.0.102:2377

6.查看节点连接信息

# docker node ls

ID                            HOSTNAME           STATUS              AVAILABILITY        MANAGER STATUS
viwovkb0bk0kxlk98r78apopo *   manager            Ready               Active              Leader
yf6nb2er69pydlp6drijdfmwd     worker1            Ready               Active              
yyavdslji7ovmw5fd3v7l62g8     worker2            Ready               Active         

7.在docker swarm下创建http服务

# docker service create -p 80:80 --name webservice --replicas 5 httpd

ID                  NAME                IMAGE               NODE               DESIRED STATE       CURRENT STATE       ERROR           PORTS
xsa5wb0eg2ln        webservice.1        httpd:latest        manager            Running             Running about 10 minutes ago                                       
3sbscs7m9lnh        webservice.2        httpd:latest        worker2            Running             Running about 10 minutes ago                                      
yao2m5wi54kz        webservice.3        httpd:latest        worker2            Running             Running about 10 minutes ago                                 
dfg2mswa52sf        webservice.4        httpd:latest        worker1            Running             Running about 15 seconds ago                                 
kah1j5hs14as        webservice.5        httpd:latest        worker1            Running             Running about 15 seconds ago                                 

8.测试http站点

# curl 192.168.0.102

It works...

三.在docker swarm下构建Jenkins cluster with configuration as code

详见作者的github仓库

https://github.com/showerlee/docker-compose-with-JCasC

大功告成...

这里k8s官方给我们提供了两种比较主流的连接k8s API cluster的语言, 一种是GO, 另外一种就是我们DevOps比较主流的Python.

今天这里我将给大家介绍如何使用python client扩展包去编写脚本并远程连接k8s API cluster.

这样我们平时除了可以远程使用ssh+kubectl去与kubernetes进行命令行数据交互外, 同样可以在远程不依赖命令行直接连接k8s cluster API cluster, 与k8s进行python API交互, 方便我们后期利用python去对k8s进行自动化集成与二次开发.

这里我们不多说, 直接进入我们的runbook配置环节:

一.部署python client环境连接我们的k8s API cluster.

1.安装python3.6.5源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel -y

2.编译安装python3.6.5以及pip package manager

# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz --no-check-certificate

# tar xf Python-3.6.5.tar.xz

# cd Python-3.6.5

# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

3.安装virtualenv

# pip3.6 install --upgrade pip

# pip3.6 install virtualenv

4.配置加载virtualenv环境

# virtualenv -p /usr/local/bin/python3 .py3env

# source .py3env/bin/activate

5.安装kubernetes python client扩展包

# pip install kubernetes

二.在k8s master获取API cluster URL与token

我们需要在创建python脚本前, 在k8s主机上创建一个admin权限的service account, 并获取其token 用来作为我们的脚本凭证.

1.抓取Cluster URL地址

# APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")

# echo $APISERVER

2.创建k8s admin-user

# mkdir -p /kube/role

# cd /kube/role

在kube-system下创建admin-user

# vi CreateServiceAccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system

给admin-user赋予admin权限

# vi CreateServiceAccount.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

# kubectl create -f CreateServiceAccount.yaml
# kubectl create -f RoleBinding.yaml

3.获取admin-user token

# Token=$(kubectl describe secret $(kubectl get secret -n kube-system | grep ^admin-user | awk '{print $1}') -n kube-system | grep -E '^token'| awk '{print $2}')

# echo $Token

最后将token与APISERVER地址返回内容复制到python client主机上, 供脚本使用.

三. 在python client主机上编写脚本

1. 创建目录结构

# mkdir -p /kube/auth

# cd /kube/auth

# touch token.txt

2. 创建token.txt文件并将k8s主机上获取的Token字符串复制到该文件

这里我们获取的token会引入到我们的脚本下, 作为bearer authorization的api key与远程k8s API建立认证连接.

3. 编写python client脚本

# vi k8s_auth.py

#!/usr/bin/env python


from kubernetes import client, config


def main():
    # Define the barer token we are going to use to authenticate.
    # See here to create the token:
    # https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/

    with open('token.txt', 'r') as file:
        Token = file.read().strip('\n')

    APISERVER = 'https://10.110.16.14:6443'

    # Create a configuration object
    configuration = client.Configuration()

    # Specify the endpoint of your Kube cluster
    configuration.host = APISERVER

    # Security part.
    # In this simple example we are not going to verify the SSL certificate of
    # the remote cluster (for simplicity reason)
    configuration.verify_ssl = False

    # Nevertheless if you want to do it you can with these 2 parameters
    # configuration.verify_ssl=True
    # ssl_ca_cert is the filepath to the file that contains the certificate.
    # configuration.ssl_ca_cert="certificate"
    configuration.api_key = {"authorization": "Bearer " + Token}

    # configuration.api_key["authorization"] = "bearer " + Token
    # configuration.api_key_prefix['authorization'] = 'Bearer'
    # configuration.ssl_ca_cert = 'ca.crt'
    # Create a ApiClient with our config
    client.Configuration.set_default(configuration)

    # Do calls
    v1 = client.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = v1.list_pod_for_all_namespaces(watch=False)
    for i in ret.items:
        print("%s\t%s\t%s" %
              (i.status.pod_ip, i.metadata.namespace, i.metadata.name))


if __name__ == '__main__':
    main()

这个脚本通过抓取同目录的k8s token字符串, 以及我们之前获取到的api server地址,利用python kubernetes扩展模块连接我们的远程API, 最终实现打印我们k8s所有namespace下的所有pods.

实际效果与kubectl get pods --all-namespaces类似.

4. 修改权限并执行

# chmod 755 k8s_auth.py

# ./k8s_auth.py

Listing pods with their IPs:
...
10.244.1.3	default	db
10.244.1.2	default	kubernetes-downwardapi-volume-example
10.244.1.245	default	nginx1-7-deployment-667547b6d8-c5nsr
10.244.1.239	default	nginx1-7-deployment-667547b6d8-mjxq4
10.244.1.247	default	nginx1-8-deployment-d46768cf9-888v8
10.244.1.242	default	nginx1-8-deployment-d46768cf9-fglq2
10.110.16.14	kube-system	etcd-kube-master
10.110.16.14	kube-system	kube-apiserver-kube-master
10.110.16.14	kube-system	kube-controller-manager-kube-master
10.244.1.241	kube-system	kube-dns-6f4fd4bdf-qsxrj
10.110.16.14	kube-system	kube-flannel-ds-5sdlg
10.110.16.15	kube-system	kube-flannel-ds-qctv4
10.110.16.14	kube-system	kube-proxy-5nscq
10.110.16.15	kube-system	kube-proxy-6g7jc
10.110.16.14	kube-system	kube-scheduler-kube-master
10.244.1.246	kube-system	kubernetes-dashboard-58f5cb49c-6dxgn
10.244.1.240	kube-system	tiller-deploy-cbb85d8dc-f6rsv
10.110.16.15	kube-system	traefik-ingress-lb-765c44656f-fkzxb
10.244.1.243	spinnaker	kubelive-create-bucket-wxlv4
10.244.1.244	spinnaker	kubelive-delete-jobs-zhn75

这样我们就成功的编写完成python client脚本连接k8s API cluster, 将所有namespace下的pod的基本信息打印出来.

Finished...

traefik 是一款开源的反向代理与负载均衡工具。它最大的优点是能够与常见的微服务系统直接整合，可以实现自动化动态配置。目前支持 Docker, Swarm, Mesos/Marathon, Mesos, Kubernetes, Consul, Etcd, Zookeeper, BoltDB, Rest API 等等后端模型。

由于微服务架构以及 Docker 技术和 kubernetes 编排工具最近几年才开始逐渐流行，所以一开始的反向代理服务器比如 nginx、apache 并未提供其支持，所以才会出现 Ingress Controller 来做 kubernetes 和前端负载均衡器如 nginx 之间做衔接；即 Ingress Controller 的存在就是为了能跟 kubernetes 交互，又能写 nginx 配置，还能 reload 它，这是一种折中方案；而 traefik 天生就是提供了对 kubernetes 的支持，也就是说 traefik 本身就能跟 kubernetes API 交互，感知后端变化，因此可以得知:  traefik 完全已经取代了 Ingress Controller 作为与k8s交互的代理，在此给大家这里整体架构如下:

为什么选择 traefik？

Golang 编写，单文件部署，与系统无关，同时也提供小尺寸 Docker 镜像。
支持 Docker/Etcd 后端，天然连接我们的微服务集群。
内置 Web UI，管理相对方便。
自动配置 ACME(Let’s Encrypt) 证书功能。
性能尚可，我们也没有到压榨 LB 性能的阶段，易用性更重要。

除了这些以外，traefik 还有以下特点：
Restful API 支持。
支持后端健康状态检查，根据状态自动配置。
支持动态加载配置文件和 graceful 重启。
支持 WebSocket 和 HTTP/2。

目前网上很多资料虽然能够成功在k8s下搭建ingress+traefik实现cluster内网的代理功能, 但如何将这个反向代理IP暴露在我们外网, 也就是不依赖于AWS等公有云, 在我们on-prem(本地k8s网络环境)通过定义hostnetwork去实现将我们的node ip直接作为我们在cluster下创建的反向代理ip,  将是我们本次需要给大家介绍的内容.

我们接着上一篇Kubernates1.9+Docker17离线安装部署, 给大家展开如何实现k8s通过使用Ingress+Traefik实现集群反向代理功能.

1.获取Traefik支持K8s仓库脚本

# cd /kube/

# git clone https://github.com/containous/traefik.git

# cd traefik/examples/k8s/

# ll

total 36
-rw-r--r-- 1 root root  140 Aug 26 04:23 cheese-default-ingress.yaml
-rw-r--r-- 1 root root 1805 Aug 26 04:23 cheese-deployments.yaml
-rw-r--r-- 1 root root  519 Aug 26 04:23 cheese-ingress.yaml
-rw-r--r-- 1 root root  509 Aug 26 04:23 cheese-services.yaml
-rw-r--r-- 1 root root  504 Aug 26 04:23 cheeses-ingress.yaml
-rw-r--r-- 1 root root 1144 Aug 26 06:40 traefik-deployment.yaml
-rw-r--r-- 1 root root 1206 Aug 26 04:23 traefik-ds.yaml
-rw-r--r-- 1 root root  694 Aug 26 04:23 traefik-rbac.yaml
-rw-r--r-- 1 root root  466 Aug 26 04:40 ui.yaml

这个目录下就是示例 Traefik 启动所需要的 yaml 文件，Traefik 提供了适配各个类型服务编排的部署方式，kubernetes 启动方式支持 Deployment 和 DaemonSet，二选一都可以, 这里笔者选择Deployment

2. Traefik脚本配置

还记得我之前提到的hostnetwork吗? 

这里我们需要在部署之前将我们的traefik-deployment.yaml下添加一行 hostNetwork: true

这样子就能保证我们的Traefik反向代理的会将自己Endpoints IP配置成我们的NodeIP, 供我们与K8S同一网段的设备访问.

我们可以这么配置:

# vi traefik-deployment.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      hostNetwork: true
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
        - name: admin
          containerPort: 8080
        args:
        - --api
        - --kubernetes
        - --logLevel=INFO
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

# vi traefik-rbac.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

这样子我们就完成了K8S脚本的初始修改, 接下来我们就可以直接开始我们的Traefik安装

3. Traefik安装

# kubectl apply -f traefik-rbac.yaml

clusterrole "traefik-ingress-controller" created
clusterrolebinding "traefik-ingress-controller" created

# kubectl apply -f traefik-deployment.yaml

serviceaccount "traefik-ingress-controller" created
deployment "traefik-ingress-controller" created
service "traefik-ingress-service" created

# kubectl apply -f ui.yaml

service "traefik-web-ui" created
ingress "traefik-web-ui" created

# kubectl get pods --all-namespaces -o wide

NAME                                          READY     STATUS    RESTARTS   AGE       IP             NODE
…
traefik-ingress-controller-795ffb7d78-ppw94   1/1       Running   0          5m        10.110.16.15   kube-node1

可以看到我们的traefik-ingress pods成功的打开了我们的hostnetwork, 这里显示的是我们的node ip而不是pod内网 ip, 也就是我们可以直接通过访问node ip 10.110.16.15来连接traefik反向代理, 利用traefik来分配我们的K8S的容器连接.

# kubectl get service --all-namespaces

NAMESPACE      NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
default        frontend                    ClusterIP   10.110.87.1      <none>        80/TCP                        4h
default        kubernetes                  ClusterIP   10.96.0.1        <none>        443/TCP                       1d
default        my-nginx                    ClusterIP   10.102.200.83    <none>        80/TCP                        4h
kube-system    kube-dns                    ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP                 1d
kube-system    kubernetes-dashboard        NodePort    10.108.187.244   <none>        443:32666/TCP                 1d
kube-system    tiller-deploy               ClusterIP   10.111.251.3     <none>        44134/TCP                     1d
kube-system    traefik-ingress-service     NodePort    10.100.70.138    <none>        80:31087/TCP,8080:31017/TCP   13m
kube-system    traefik-web-ui              ClusterIP   10.96.54.167     <none>        80/TCP                        13m
newegg-nginx   newegg-nginx-newegg-nginx   NodePort    10.97.200.124    <none>        80:32239/TCP                  8h

我们可以通过两种方式去访问我们的traefik后台管理员界面.

1).通过Node Port方式

我们可以在与K8S node同网段的机器去访问其Node暴露的端口31017

2).直接通过ui.yaml配置traefik+ingress反向代理, 这样我们就可以通过host域名直接访问后台

# vi ui.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  rules:
  - host: traefik-ui.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

在本地MacOS系统添加traefik-ui.k8s解析到10.110.16.15

# echo "10.110.16.15 traefik-ui.k8s" >> /etc/hosts

我们可以同样利用traefik反向代理, 通过http域名的方式访问admin后台.

4.部署两个服务nginx1-7和nginx1-8，配置Traefik去负载这两个服务：

配置一个deployment类型的1.7版本nginx实例, 并利用service开启其内部80端口监听.

# vi nginx1-7.yaml

apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-7
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx1-7-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-7
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

配置一个deployment类型的1.8版本nginx实例, 并利用service开启其内部80端口监听.

# vi nginx1-8.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-nginx
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-8
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx1-8-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-8
    spec:
      containers:
      - name: nginx
        image: nginx:1.8
        ports:
        - containerPort: 80

通过利用Ingress绑定1.7与1.8nginx实例下的serviceName, 给其定义不同的域名, 实现traefik反向代理.

# vi traefik.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: default
spec:
  rules:
  - host: traefik.nginx.io
    http:
      paths:
      - path: /
        backend:
          serviceName: my-nginx
          servicePort: 80
  - host: traefik.frontend.io
    http:
      paths:
      - path: /
        backend:
          serviceName: frontend
          servicePort: 80

我们这里通过配置MacOS本地Host DNS, 将所有的DNS记录指向我们暴露在外网的traefik IP, 从而让traefik作为代理接管所有访问K8S内部实例的连接, 实际的原理和我们之前用过的apache, nginx基本一致.

# echo "10.110.16.15 traefik.nginx.io traefik.frontend.io" >> /etc/hosts

这样子我们就可以直接在MacOS浏览器下访问traefik.nginx.io与traefik.frontend.io从而实现traefik反向代理.

5.配置HTTPS访问traefik管理员后台

1).创建并配置https证书并保存到k8s secret下.

# mkdir -p /opt/k8s/ssl

# cd /opt/k8s/ssl

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=traefik-ui.k8s"

# kubectl create secret generic traefik-cert --from-file=tls.crt --from-file=tls.key -n kube-system

2).配置traefik反向代理并保存到k8s configmap下.

# mkdir -p /opt/k8s/conf/

# cd /opt/k8s/ssl/

# vi traefik-ui.toml

defaultEntryPoints = ["http","https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    regex = "^http://traefik-ui.k8s/(.*)"
    replacement = "https://traefik-ui.k8s/$1"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/opt/k8s/ssl/tls.crt"
      keyFile = "/opt/k8s/ssl/tls.key"

# kubectl create configmap traefik-ui-conf --from-file=traefik-ui.toml -n kube-system

这样子我们就成功配置traefik反向代理, 仅使http://traefik-ui.k8s会重定向到https://traefik-ui.k8s, 实现https加密访问.

3).创建traefik deployment实例并支持https反向代理.

# vi traefik-deployment-ssl.yaml 

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: traefik-ingress-lb
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      restartPolicy: Always
      serviceAccountName: traefik-ingress-controller
      volumes:
      - name: ssl
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-ui-conf
      containers:
      - image: traefik
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: "/opt/k8s/ssl"
          name: "ssl"
        - mountPath: "/opt/k8s/conf"
          name: "config"
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
        - name: admin
          containerPort: 8080
          hostPort: 8080
        args:
        - --configFile=/opt/k8s/conf/traefik-ui.toml
        - --web
        - --web.address=:8080
        - --kubernetes
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: https
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

这里我们在原来的traefik deployment实例基础上将之前配置的secret与configmap导入该deployment下, 并mount ssl与conf目录用来让traefik获取我们在k8s node目录下保存的相应SSL证书与traefik反向代理配置.

最后我们通过service类型打开我们的deployment 443端口.

4).创建traefik管理员界面的ingress, 从而实现https域名访问.

# vi ui-ssl.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  tls:
    - secretName: traefik-cert
  rules:
  - host: traefik-ui.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

最后我们访问https://traefik-ui.k8s 验证结果.

这样子我们就成功的开启了traefik后台admin界面的https加密连接...

更多traefik+ingress相关内容: https://docs.traefik.io/user-guide/kubernetes/

这里简单的介绍一下我们这个自动化流水线所使用到的工具:

Jenkins Pipeline: 目前国内外DevOps, CI/CD比较主流的一种将我们软件开发周期所涉及到的环节通过pipeline流水线完美的串联在一起的自动化部署框架. 它提出了一种pipeline as a code的概念, 意在将我们的所有软件开发周期中涉及到的所有步骤(版本控制 - 代码检查 - 编译 - 单元测试 - 打包 - 测试环境部署 - 集成测试 - 功能测试 - 生产环境部署)通过代码的方式推送给我们的Jenkins进行pipeline自动化部署, 实现将我们的自动化流水线的配置代码化, 并同样实现版本控制. 

目前Jenkins官方有两种pipeline的写法, 一种叫做Declarative Pipeline, 另外一种叫做Scripted Pipeline. 前者较为方便阅读, 适合初学者入门, 但相对对集成模块的兼容性以及pipeline逻辑编写的功能性较后者相对弱一些, 后者较为专业, 可以实现逻辑编写, 并支持很多主流集成模块, 推荐具有groovy脚本编写经验的人员使用.

具体内容详见: http://www.showerlee.com/archives/1972

Kubernetes: 目前这几年流行的一种容器化管理系统, 用于自动部署、扩展和管理容器化（containerized）应用程序的开源系统。它旨在提供“跨主机集群的自动部署、扩展以及运行应用程序容器的平台”。它支持一系列容器工具, 目前主流会使用Docker作为他的主流配置容器.使用它的原因也在于它将是目前DevOps领域的一个主流的Docker容器管理系统, 目前国内外很多公司都即将或者已经把自己的传统的虚拟机架构转向为Docker微服务架构, 这里我们非常有必要去学习如何基于k8s下去做自动化部署.

具体内容详见: http://www.showerlee.com/archives/2200

Helm: 这个应该算作k8s的一个功能性的工具, 它作为k8s的包管理工具, 非常方便的帮助我们整合k8s下零散的部署脚本为一个具体的项目包, 最终简化了Kubernetes部署应用的版本控制、打包、发布、删除、更新等操作.

具体内容详见: http://www.showerlee.com/archives/2455

这里本次自动化流水线实现的部署内容就是通过编写pipeline脚本, 将一个从Docker官方拿到的centos docker镜像容器, 进行二次安装配置, 打包为一个nignx静态网站容器, 发布到我们k8s下, 期间我们会对这个容器进行一些常规的功能测试, 从而模拟我们在k8s下实现自动化流水线交付的完整架构.

本次内容涉及到的代码可以参考我的github仓库:

https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes

Okay, Let's roll out...

安装环境

Local Desktop: MacOS

Virtual Machine: Virtual Box

Virtual System: CentOS 7.4

Jenkins: Jenkins 2.138

Kubernetes: Kubernetes 1.9

Docker: 17.03.2-ce

Helm: helm-v2.7.0

kube-master 10.110.16.14

kube-node-1 10.110.16.15

一. 系统环境配置

1.关闭SELINUX和firewall

# vi /etc/sysconfig/selinux

...
SELINUX=disabled 
...

# setenforce 0

# systemctl stop firewalld  && systemctl disable firewalld

2.安装k8s环境.

http://www.showerlee.com/archives/2200

3.安装helm环境.

http://www.showerlee.com/archives/2455

4.安装Jenkins环境.

http://www.showerlee.com/archives/1880

二. Jenkins Pipeline配置

1.将jenkins用户添加到默认docker用户组下, 从而保证jenkins可以直接访问/var/run/docker.sock

# usermod -a -G docker jenkins

2.更改全局安全配置, 保证pipeline可以直接调用helm

进入Jenkins --> Manage Jenkins --> Configure Global Security
在Authorization下选择Project-based Matrix Authorization Strategy

配置Anonymous User具有Read Jenkins Jobs的权限

如图:

3.配置jenkins用户加载kubectl环境变量.

# mkdir -p /home/jenkins/.kube

# cp -i /etc/kubernetes/admin.conf /home/jenkins/.kube/config

# chown jenkins:jenkins /home/jenkins/.kube/config

4.配置github与dockerhub的Jenkins账户凭证, 后面pipeline对应模块需要调用.

5.创建pipeline任务

#!groovy

def kubectlTest() {
    // Test that kubectl can correctly communication with the Kubernetes API
    echo "running kubectl test"
    sh "kubectl get nodes"

}

def helmLint(String chart_dir) {
    // lint helm chart
    sh "/usr/local/bin/helm lint ${chart_dir}"

}

def helmDeploy(Map args) {
    //configure helm client and confirm tiller process is installed

    if (args.dry_run) {
        println "Running dry-run deployment"

        sh "/usr/local/bin/helm upgrade --dry-run --debug --install ${args.name} ${args.chart_dir} --set ImageTag=${args.tag},Replicas=${args.replicas},Cpu=${args.cpu},Memory=${args.memory},DomainName=${args.name} --namespace=${args.name}"
    } else {
        println "Running deployment"
        sh "/usr/local/bin/helm upgrade --install ${args.name} ${args.chart_dir} --set ImageTag=${args.tag},Replicas=${args.replicas},Cpu=${args.cpu},Memory=${args.memory},DomainName=${args.name} --namespace=${args.name}"

        echo "Application ${args.name} successfully deployed. Use helm status ${args.name} to check"
    }
}



timeout(time: 2000, unit: 'SECONDS') {
    node {
        println "----------------------------------------------------------------------------"
        stage 'Check out pipeline from GitHub Repo'
        //git url: 'https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git'
        git branch: 'master',
            credentialsId: 'showerlee-github',
            url: 'https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git'

        // Setup the Docker Registry (Docker Hub) + Credentials 
        registry_url = "https://index.docker.io/v1/" // Docker Hub
        docker_creds_id = "showerlee-dockerhub" // name of the Jenkins Credentials ID

        def pwd = pwd()
        def chart_dir = "${pwd}/charts/newegg-nginx"

        // Add build tag version
        Properties props = new Properties()
        File propsFile = new File("${pwd}/promote.properties")
        props.load(propsFile.newDataInputStream())
        def build_tag_raw = props.getProperty('BUILD_TAG')
        float build_tag = Float.parseFloat(build_tag_raw)+0.1;
        println("Set current build_tag="+build_tag+" temporarily")

        //Set build_tag to index.html
        sh """
        echo "<h1>Welcome Newegg Nginx Test Version: ${build_tag}</h1>" > index.html
        """

        def inputFile = readFile('config.json')
        def config = new groovy.json.JsonSlurperClassic().parseText(inputFile)
        println "pipeline config ==> ${config}"
        println "----------------------------------------------------------------------------"
        
        stage 'Register DockerHub'
        echo "[INFO] Register Dockerhub"
        docker.withRegistry("${registry_url}", "${docker_creds_id}") {
        
            // Set up the container to build 
            maintainer_name = "showerlee"
            container_name = "nginx-test"
            println "----------------------------------------------------------------------------"

            stage "Build Nginx Container"
            echo "[INFO] Building Nginx with docker.build(${maintainer_name}/${container_name}:${build_tag})"
            container = docker.build("${maintainer_name}/${container_name}:${build_tag}", '.')
            println "----------------------------------------------------------------------------"
            try {
                
                // Start Testing
                stage "Spin up Nginx Container"
                echo "[INFO] Spin up Nginx Container"
                
                // Run the container with the env file, mounted volumes and the ports:
                docker.image("${maintainer_name}/${container_name}:${build_tag}").withRun("--name=${container_name}  -p 80:80 ")  { c ->
                       
                    // wait for the django server to be ready for testing
                    // the 'waitUntil' block needs to return true to stop waiting
                    // in the future this will be handy to specify waiting for a max interval: 
                    // https://issues.jenkins-ci.org/browse/JENKINS-29037
                    //
                    waitUntil {
                        sh """
                        set +x
                        ss -antup | grep :::80[^0-9] | grep LISTEN | wc -l | tr -d '\n' > /tmp/wait_results
                        set -x
                        """
                        wait_results = readFile '/tmp/wait_results'

                        echo "[INFO] Wait Results(${wait_results})"
                        if ("${wait_results}" == "1")
                        {
                            echo "[INFO] Nginx is listening on port 80"
                            sh "rm -f /tmp/wait_results"
                            return true
                        }
                        else
                        {
                            echo "[INFO] Nginx is not listening on port 80 yet"
                            return false
                        }
                    } // end of waitUntil
                    
                    // At this point Nginx is running
                    echo "[INFO] Docker Container is running"
                    input 'You can check the running container on docker build server now! Click Proceed to next stage...'    
                    // this pipeline is using 3 tests 
                    // by setting it to more than 3 you can test the error handling and see the pipeline Stage View error message
                    MAX_TESTS = 3
                    for (test_num = 1; test_num <= MAX_TESTS; test_num++) {     
                        println "----------------------------------------------------------------------------"   
                        echo "Running Test(${test_num})"
                    
                        expected_results = 0
                        if (test_num == 1 ) 
                        {
                            // Test we can download the home page from the running docker container
                            echo "[INFO] Check validation of home page"
                            sh """
                            set +x
                            docker exec -t ${container_name} curl -s http://localhost | grep Welcome | wc -l | tr -d '\n' > /tmp/test_results
                            set -x
                            """
                            expected_results = 1
                        }
                        else if (test_num == 2)
                        {
                            // Test if port 80 is exposed
                            echo "[INFO] Check if port 80 is exposed"
                            sh """
                            set +x
                            docker inspect --format '{{ (.NetworkSettings.Ports) }}' ${container_name}
                            docker inspect --format '{{ (.NetworkSettings.Ports) }}' ${container_name} | grep map | grep '80/tcp:' | wc -l | tr -d '\n' > /tmp/test_results
                            set -x
                            """
                            expected_results = 1
                        }
                        else if (test_num == 3)
                        {
                            // Test there's nothing established on the port since nginx is not running:
                            echo "[INFO] Check if nothing established from nginx container"
                            sh """
                            set +x
                            docker exec -t ${container_name} ss -apn | grep 80 | grep ESTABLISHED | wc -l | tr -d '\n' > /tmp/test_results
                            set -x
                            """
                            expected_results = 0
                        }
                        else
                        {
                            err_msg = "Missing Test(${test_num})"
                            echo "[ERROR] ${err_msg}"
                            currentBuild.result = 'FAILURE'
                            error "Failed to finish container testing with Message(${err_msg})"
                        }
                        
                        // Now validate the results match the expected results
                        stage "Test(${test_num}) - Validate Results"
                        test_results = readFile '/tmp/test_results'
                        echo "[INFO] Test(${test_num}) Results($test_results) == Expected(${expected_results})"
                        sh """
                        set +x
                        if [ \"${test_results}\" != \"${expected_results}\" ]; 
                        then 
                            echo \" --------------------- Test(${test_num}) Failed--------------------\"
                            echo \" - Test(${test_num}) Failed\"
                            exit 1
                        else 
                            echo \" - Test(${test_num}) Passed\"
                            exit 0
                        fi
                        set -x
                        """

                        echo "[INFO] Finished Running Test(${test_num})"
                    
                        // cleanup after the test run
                        sh "rm -f /tmp/test_results"
                        currentBuild.result = 'SUCCESS'
                    }
                }
                
            } catch (Exception err) {
                err_msg = "Test had Exception(${err})"
                currentBuild.result = 'FAILURE'
                error "FAILED - Stopping build for Error(${err_msg})"
            }
            println "----------------------------------------------------------------------------"
            stage "Push to DockerHub"
            input 'Do you approve to push?'
            container.push()
            currentBuild.result = 'SUCCESS'
            println "----------------------------------------------------------------------------"
            stage "Push properties to git repo"
            echo "Push current build_tag="+build_tag+" to git repo"
            withCredentials([usernamePassword(credentialsId: 'showerlee-github', passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
                sh """
                set +x
                git --version
                echo 'BUILD_TAG=${build_tag}' > ${pwd}/promote.properties
                git config --global user.email "showerlee@vip.qq.com"
                git config --global user.name "showerlee"
                git add ${pwd}/promote.properties ${pwd}/index.html
                git commit -m"Update docker tag to ${build_tag}"
                git push --set-upstream https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git master
                set -x
                """
            }
            println "----------------------------------------------------------------------------"
            
        }
        
        stage ('helm test') { 
            echo "[INFO] Start helm test"   
            // run helm chart linter
            echo "[INFO] Run helm chart linter"
            helmLint(chart_dir)

            // dry-run helm chart installation
            echo "[INFO] Dry-run helm chart installation"
            helmDeploy(
                dry_run       : true,
                name          : config.app.name,
                chart_dir     : chart_dir,
                tag           : build_tag,
                replicas      : config.app.replicas,
                cpu           : config.app.cpu,
                memory        : config.app.memory
            )
            println "----------------------------------------------------------------------------"
        }
        
        stage ('helm deploy') {
            input 'Do you approve to deploy?'
            echo "[INFO] Start helm deployment"
            // Deploy using Helm chart
            helmDeploy(
                dry_run       : false,
                name          : config.app.name,
                chart_dir     : chart_dir,
                tag           : build_tag,
                replicas      : config.app.replicas,
                cpu           : config.app.cpu,
                memory        : config.app.memory
            )
            echo "[INFO] Deployment Finished..."
        }
        
        ///////////////////////////////////////
        //
        // Coming Soon Feature Enhancements
        //
        // 1. Add Docker Compose testing as a new Pipeline item that is initiated after this one for "Integration" testing
        // 2. Make sure to set the Pipeline's "Throttle builds" to 1 because the docker containers will collide on resources like ports and names
        // 3. Should be able to parallelize the docker.withRegistry() methods to ensure the container is running on the slave
        // 4. After the tests finish (and before they start), clean up container images to prevent stale docker image builds from affecting the current test run
    }
}

6.执行pipeline任务构建

Started by user admin
Obtained Jenkinsfile from git https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git
[Pipeline] timeout
Timeout set to expire in 33 min
[Pipeline] {
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/kube-helm-pipeline@2
[Pipeline] {
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Check out pipeline from GitHub Repo)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Check out pipeline from GitHub Repo
Proceeding
[Pipeline] git
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git # timeout=10
Fetching upstream changes from https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials github credential
 > git fetch --tags --progress https://github.com/showerlee/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision ce4fc2593333f3ed89cd5654966919b0aa97628d (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ce4fc2593333f3ed89cd5654966919b0aa97628d
 > git branch -a -v --no-abbrev # timeout=10
 > git branch -D master # timeout=10
 > git checkout -b master ce4fc2593333f3ed89cd5654966919b0aa97628d
Commit message: "Update docker tag to 1.5"
 > git rev-list --no-walk ce4fc2593333f3ed89cd5654966919b0aa97628d # timeout=10
[Pipeline] pwd
[Pipeline] echo
Set current build_tag=1.6 temporarily
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ echo '<h1>Welcome Newegg Nginx Test Version: 1.6</h1>'
[Pipeline] readFile
[Pipeline] echo
pipeline config ==> [app:[memory:128Mi, replicas:3, name:newegg-nginx, cpu:10m], pipeline:[library:[branch:master], enabled:true]]
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Register DockerHub)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Register DockerHub
Proceeding
[Pipeline] echo
[INFO] Register Dockerhub
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withDockerRegistry
$ docker login -u showerlee -p ******** https://index.docker.io/v1/
Login Succeeded
[Pipeline] {
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Build Nginx Container)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Build Nginx Container
Proceeding
[Pipeline] echo
[INFO] Building Nginx with docker.build(showerlee/nginx-test:1.6)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker build -t showerlee/nginx-test:1.6 .
Sending build context to Docker daemon 2.164 MB

Step 1/6 : FROM centos:centos7
 ---> 5182e96772bf
Step 2/6 : MAINTAINER showerlee
 ---> Using cache
 ---> cc77ae5c175a
Step 3/6 : RUN yum -y update         && yum clean all         && yum install -y epel-release         && yum install -y nginx iproute
 ---> Using cache
 ---> 2c718c146ba5
Step 4/6 : EXPOSE 80
 ---> Using cache
 ---> 829b29a719e6
Step 5/6 : COPY index.html /usr/share/nginx/html/
 ---> Using cache
 ---> 22e805b84cee
Step 6/6 : CMD nginx -g daemon off;
 ---> Using cache
 ---> de02105d9033
Successfully built de02105d9033
[Pipeline] dockerFingerprintFrom
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Spin up Nginx Container)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Spin up Nginx Container
Proceeding
[Pipeline] echo
[INFO] Spin up Nginx Container
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker run -d --name=nginx-test -p 80:80 showerlee/nginx-test:1.6
[Pipeline] dockerFingerprintRun
[Pipeline] waitUntil
[Pipeline] {
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
[Pipeline] readFile
[Pipeline] echo
[INFO] Wait Results(1)
[Pipeline] echo
[INFO] Nginx is listening on port 80
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ rm -f /tmp/wait_results
[Pipeline] }
[Pipeline] // waitUntil
[Pipeline] echo
[INFO] Docker Container is running
[Pipeline] input
You can check the running container on docker build server now! Click Proceed to next stage...
Proceed or Abort
Approved by admin
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] echo
Running Test(1)
[Pipeline] echo
[INFO] Check validation of home page
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
[Pipeline] stage (Test(1) - Validate Results)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Test(1) - Validate Results
Proceeding
[Pipeline] readFile
[Pipeline] echo
[INFO] Test(1) Results(1) == Expected(1)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
 - Test(1) Passed
[Pipeline] echo
[INFO] Finished Running Test(1)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ rm -f /tmp/test_results
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] echo
Running Test(2)
[Pipeline] echo
[INFO] Check if port 80 is exposed
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
map[80/tcp:[{0.0.0.0 80}]]
[Pipeline] stage (Test(2) - Validate Results)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Test(2) - Validate Results
Proceeding
[Pipeline] readFile
[Pipeline] echo
[INFO] Test(2) Results(1) == Expected(1)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
 - Test(2) Passed
[Pipeline] echo
[INFO] Finished Running Test(2)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ rm -f /tmp/test_results
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] echo
Running Test(3)
[Pipeline] echo
[INFO] Check if nothing established from nginx container
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
[Pipeline] stage (Test(3) - Validate Results)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Test(3) - Validate Results
Proceeding
[Pipeline] readFile
[Pipeline] echo
[INFO] Test(3) Results(0) == Expected(0)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
 - Test(3) Passed
[Pipeline] echo
[INFO] Finished Running Test(3)
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ rm -f /tmp/test_results
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker stop 05a19a7c8519a04cc03029d7273a8e2ae3363e74a03b6d4167dfbec0bf6ee978
05a19a7c8519a04cc03029d7273a8e2ae3363e74a03b6d4167dfbec0bf6ee978
+ docker rm -f 05a19a7c8519a04cc03029d7273a8e2ae3363e74a03b6d4167dfbec0bf6ee978
05a19a7c8519a04cc03029d7273a8e2ae3363e74a03b6d4167dfbec0bf6ee978
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Push to DockerHub)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Push to DockerHub
Proceeding
[Pipeline] input
Do you approve to push?
Proceed or Abort
Approved by admin
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker tag showerlee/nginx-test:1.6 index.docker.io/showerlee/nginx-test:1.6
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ docker push index.docker.io/showerlee/nginx-test:1.6
The push refers to a repository [docker.io/showerlee/nginx-test]
aedd55edeb74: Preparing
85e9d4859663: Preparing
1d31b5806ba4: Preparing
85e9d4859663: Layer already exists
1d31b5806ba4: Layer already exists
aedd55edeb74: Retrying in 5 seconds
aedd55edeb74: Retrying in 4 seconds
aedd55edeb74: Retrying in 3 seconds
aedd55edeb74: Retrying in 2 seconds
aedd55edeb74: Retrying in 1 second
aedd55edeb74: Pushed
1.6: digest: sha256:8ee214129c880a0d759837daf30c31772797fa6709f11edd9e30db6891710de4 size: 948
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] stage (Push properties to git repo)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Push properties to git repo
Proceeding
[Pipeline] echo
Push current build_tag=1.6 to git repo
[Pipeline] withCredentials
[Pipeline] {
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ set +x
git version 1.8.3.1
[master cdbb62c] Update docker tag to 1.6
 2 files changed, 2 insertions(+), 2 deletions(-)
To https://****:****@github.com/****/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git
   ce4fc25..cdbb62c  master -> master
Branch master set up to track remote branch master from https://****:****@github.com/****/Jenkins-Pipeline-CI-CD-with-Helm-on-Kubernetes.git.
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] stage
[Pipeline] { (helm test)
[Pipeline] echo
[INFO] Start helm test
[Pipeline] echo
[INFO] Run helm chart linter
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ /usr/local/bin/helm lint /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx
==> Linting /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, no failures
[Pipeline] echo
[INFO] Dry-run helm chart installation
[Pipeline] echo
Running dry-run deployment
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ /usr/local/bin/helm upgrade --dry-run --debug --install newegg-nginx /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx --set ImageTag=1.6,Replicas=3,Cpu=10m,Memory=128Mi,DomainName=newegg-nginx --namespace=newegg-nginx
[debug] Created tunnel using local port: '39460'

[debug] SERVER: "localhost:39460"

Release "newegg-nginx" does not exist. Installing it now.
[debug] CHART PATH: /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx

NAME:   newegg-nginx
REVISION: 1
RELEASED: Sun Aug 26 01:38:02 2018
CHART: newegg-nginx-1.0.0
USER-SUPPLIED VALUES:
Cpu: 10m
DomainName: newegg-nginx
ImageTag: "1.6"
Memory: 128Mi
Replicas: 3

COMPUTED VALUES:
ContainerPort: 80
Cpu: 10m
DomainName: newegg-nginx
Image: showerlee/nginx-test
ImagePullPolicy: Always
ImageTag: "1.6"
Imagetag: latest
Memory: 128Mi
Replicas: 3
ServicePort: 80
ServiceType: NodePort

HOOKS:
MANIFEST:

---
# Source: newegg-nginx/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: newegg-nginx-newegg-nginx
  labels:    
    app: newegg-nginx-newegg-nginx
    version: 1.0.0
    release: newegg-nginx
spec:
  type: "NodePort"
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: newegg-nginx-newegg-nginx
---
# Source: newegg-nginx/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: newegg-nginx-newegg-nginx
  labels:    
    app: newegg-nginx-newegg-nginx
    version: 1.0.0
    release: newegg-nginx
spec:
  replicas: 3
  template:
    metadata:
      labels:        
        app: newegg-nginx-newegg-nginx
        version: 1.0.0
        release: newegg-nginx
    spec:
      containers:
        - name: newegg-nginx
          image: "showerlee/nginx-test:1.6"
          imagePullPolicy: "Always"
          ports:
            - containerPort: 80
              protocol: TCP
          resources:
            requests:
              cpu: "10m"
              memory: "128Mi"
---
# Source: newegg-nginx/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: newegg-nginx-newegg-nginx
  labels:    
    app: newegg-nginx-newegg-nginx
    version: 1.0.0
    release: newegg-nginx
spec:
  rules:
  - host: newegg-nginx.buyabs.corp
    http:
      paths:
      - path: /
        backend:
          serviceName: newegg-nginx-newegg-nginx
          servicePort: 80
[Pipeline] echo
----------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (helm deploy)
[Pipeline] input
Do you approve to deploy?
Proceed or Abort
Approved by admin
[Pipeline] echo
[INFO] Start helm deployment
[Pipeline] echo
Running deployment
[Pipeline] sh
[kube-helm-pipeline@2] Running shell script
+ /usr/local/bin/helm upgrade --install newegg-nginx /var/jenkins_home/workspace/kube-helm-pipeline@2/charts/newegg-nginx --set ImageTag=1.6,Replicas=3,Cpu=10m,Memory=128Mi,DomainName=newegg-nginx --namespace=newegg-nginx
Release "newegg-nginx" does not exist. Installing it now.
NAME:   newegg-nginx
LAST DEPLOYED: Sun Aug 26 01:38:24 2018
NAMESPACE: newegg-nginx
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Ingress
NAME                       HOSTS                     ADDRESS  PORTS  AGE
newegg-nginx-newegg-nginx  newegg-nginx.buyabs.corp  80       1m

==> v1/Pod(related)
NAME                                        READY  STATUS             RESTARTS  AGE
newegg-nginx-newegg-nginx-56c478c888-6n6rt  0/1    ContainerCreating  0         1m
newegg-nginx-newegg-nginx-56c478c888-hsbcp  0/1    ContainerCreating  0         1m
newegg-nginx-newegg-nginx-56c478c888-vdfgb  0/1    ContainerCreating  0         1m

==> v1/Service
NAME                       TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)       AGE
newegg-nginx-newegg-nginx  NodePort  10.97.200.124  <none>       80:32239/TCP  1m

==> v1beta1/Deployment
NAME                       DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
newegg-nginx-newegg-nginx  3        3        3           0          1m


[Pipeline] echo
Application newegg-nginx successfully deployed. Use helm status newegg-nginx to check
[Pipeline] echo
[INFO] Deployment Finished...
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline
Finished: SUCCESS

7.验证结果

这样我们就成功的利用Jenkins集成k8s+helm实现将我们的静态网站部署到我们的kubernetes集群中.

大功告成...

安装环境

Remote: CentOS 7.4 x64 (django.example.com)

Python: Python3.6.5

Django: Django 2.0.4

nWSGI:  uwsgi-2.0.15

Nginx:  nginx- 1.10.2-1.el6

一. 系统环境配置

1.关闭iptables和selinux

# su - root

# service iptables stop

# setenforce 0

# vi /etc/sysconfig/selinux

修改

SELINUX=disabled

2.添加本地host DNS

# vi /etc/hosts

127.0.0.1    django.example.com

二. Python配置

1.安装python3.6.5源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel libffi-devel -y

2.编译安装python3.6.5以及pip package manager

# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz --no-check-certificate

# tar xf Python-3.6.5.tar.xz

# cd Python-3.6.5

# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

3.安装virtualenv

# pip3.6 install --upgrade pip

# pip3.6 install virtualenv

三. Nginx配置

1. 安装nginx package

# yum install nginx -y

2.配置nginx with nWSGI

# vi /etc/nginx/conf.d/django.conf

server {
    listen  80;
    server_name  django.example.com;  

    charset utf-8;

    access_log  /var/log/nginx/django_access.log;
    error_log   /var/log/nginx/django_error.log;
    
    location = /favicon.ico { access_log off; log_not_found off; }

    location /static/ {
        root /usr/share/nginx/html/django.example.com;
    }

    client_max_body_size 20M;
    
    location / {
        include         uwsgi_params;
        uwsgi_pass      unix:/etc/uwsgi/uwsgi-django.sock;
        uwsgi_read_timeout 30s;
        uwsgi_send_timeout 30s;
    }

}

四. Django+uWSGI配置

1. uWSGI配置

# mkdir -p /etc/uwsgi

# vi /etc/uwsgi/uwsgi-django.ini

[uwsgi]
project = django.example.com
base = /data/www

chdir = %(base)/%(project)
home = %(base)/%(project)/.py3env
module = myproject.wsgi:application

pidfile = /tmp/uwsgi-master-django.pid

master = true
processes = 2
enable-threads = true

# use unix socket because it is more secure and faster than TCP socket
socket = /etc/uwsgi/uwsgi-django.sock
chmod-socket = 660
uid = nginx
gid = nginx

vacuum = true
die-on-term = true

logto = /var/log/nginx/uwsgi-django.log

2. 配置Django base folder

# cd /usr/share/nginx/html

# mkdir django.example.com

# cd django.example.com

# virtualenv -p /usr/local/bin/python3 .py3env

3. 开启virtualenv python3环境

# source .py3env/bin/activate

4. 在此环境安装Django相关模块

# pip install django uwsgi PyMySQL

5. 创建uWSGI启动脚本

# mkdir -p /etc/uwsgi/bin

# vi /etc/systemd/system/uwsgi-django.service

[Unit]
    Description=uWSGI instance to serve myproject

[Service]
    BASE=/data/www/django.example.com
    ENV=$BASE/.py3env
    ExecStartPre=-/usr/bin/bash -c 'chown -R nginx:nginx /etc/uwsgi'
    ExecStart=/usr/bin/bash -c 'source /usr/share/nginx/html/django.example.com/.py3env/bin/activate; uwsgi --ini /etc/uwsgi/uwsgi-django.ini'

[Install]
    WantedBy=multi-user.target

五. Django项目配置

1. 保证virtualenv python3环境开启

# cd /usr/share/nginx/html/django.example.com/

# source .py3env/bin/activate

2.创建一个Django项目

# django-admin startproject myproject .

3.添加static目录

# vi myproject/settings.py

末行添加:

STATIC_ROOT = os.path.join(BASE_DIR, "static/")

4.创建本地SQLlite文件

Tip:这里使用SQLlite代替其他数据库作为我们项目的DB

# ./manage.py makemigrations
# ./manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK

5.创建项目管理员账户

# ./manage.py createsuperuser

Username (leave blank to use 'root'): root
Email address: admin@admin.com
Password:
Password (again):
Superuser created successfully.

6.生成项目静态文件目录

# ./manage.py collectstatic

7.修改wsgi入口文件

# vi myproject/wsgi.py

import os
import sys
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
sys.path.append('/usr/share/nginx/html/django.example.com')

from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()

8.添加ALLOWED_HOSTS

# vi myproject/settings.py

Update:

ALLOWED_HOSTS = ['django.example.com']

9. 修改权限(可执行并保持与nginx启动user一致)

# chmod -R 755 /etc/uwsgi

# chown -R nginx:nginx /etc/uwsgi

# chmod -R 755 /usr/share/nginx/html/django.example.com

# chown -R nginx:nginx /usr/share/nginx/html/django.example.com

10.启动nginx+uwsgi

# systemctl restart nginx

# systemctl restart uwsgi-django

展示效果(保证Windows本地host文件能够解析django.example.com)

Finished...

Remote: CentOS 7.4 x64 (django.example.com)

Python: Python3.6.5

Apache: Apache 2.4.6

Mod_wsgi: 4.6.4

Django: Django 2.0.4

一. 系统环境配置

1.关闭iptables和selinux

# su - root

# systemctl stop firewalld

# setenforce 0

# vi /etc/sysconfig/selinux

修改

SELINUX=disabled

2.添加本地host DNS

# vi /etc/hosts

127.0.0.1    django.example.com

二. Python配置

1.安装python3.6.5源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel libffi-devel -y

2.编译安装python3.6.5以及pip package manager

# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz --no-check-certificate

# tar xf Python-3.6.5.tar.xz

# cd Python-3.6.5

# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

3.安装virtualenv

# pip3.6 install --upgrade pip

# pip3.6 install virtualenv

三. Django环境配置

1. 配置Django virtualenv

# mkdir -p /var/www/html/django

# cd /var/www/html/django

# virtualenv -p /usr/local/bin/python3.6 .py3env

2. 开启virtualenv python3环境

# source .py3env/bin/activate

3. 在此环境安装Django相关模块

# pip install django pymysql

四. Apache配置

1. 安装apache package

# yum install httpd httpd-devel -y

2.安装mod_wsgi for python3

Tip:这里其实是一个远古巨坑, 网上90%以上资料的会粗心的直接使用yum install mod_wsgi去安装apache mod_wsgi模块, 这样做其实最终mod模块会调用本地默认的python2的所有库文件, 无论你后面如何配置django入口文件, apache都不会使用我们配置的virutalenv下隔离的python3, 导致apache无法调用python3而报错. 这里小伙伴要注意哦.

# pip install mod_wsgi 

3.导出apache所需的mod_wsgi模块

# mod_wsgi-express install-module

LoadModule wsgi_module "/usr/lib64/httpd/modules/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"
WSGIPythonHome "/var/www/html/.py3env"

4.配置apache配置文件

# vi /etc/httpd/conf/httpd.conf

末行添加:

LoadModule wsgi_module "/usr/lib64/httpd/modules/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"

# vi /etc/httpd/conf.d/django.conf

WSGIPythonHome "/var/www/html/django/.py3env"

Listen 8080
<VirtualHost *:8080>

ServerName django.example.com

Alias /static /var/www/html/django/static
<Directory /var/www/html/django/static>
 Require all granted
</Directory>

<Directory /var/www/html/django/myproject>
  <Files wsgi.py>
    Require all granted
  </Files>
</Directory>

WSGIDaemonProcess myproject python-path=/var/www/html/django/.py3env/lib/python3.6/site-packages
WSGIScriptAlias / /var/www/html/django/myproject/wsgi.py

</VirtualHost>

5.重启apache并设置开机自启动

# systemctl restart httpd

# systemctl enable httpd

五. Django项目配置

1. 保证virtualenv python3环境开启

# source /var/www/html/django/.py3env/bin/activate

2.创建一个Django项目

# cd /var/www/html/django/

# django-admin startproject myproject .

3.添加static目录

# vi myproject/settings.py

末行添加:

STATIC_ROOT = os.path.join(BASE_DIR, "static/")

4.创建本地SQLlite文件

Tip:这里使用SQLlite代替其他数据库作为我们项目的DB

# ./manage.py makemigrations
# ./manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK

5.创建项目管理员账户

# ./manage.py createsuperuser

Username (leave blank to use 'root'): root
Email address: admin@admin.com
Password:
Password (again):
Superuser created successfully.

6.生成项目静态文件目录

# ./manage.py collectstatic

7.修改wsgi入口文件

# vi myproject/wsgi.py

import os
import sys
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
sys.path.append('/var/www/html/django')

from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()

8.添加ALLOWED_HOSTS

# vi myproject/settings.py

Update:

ALLOWED_HOSTS = ['django.example.com']

9.修改项目属主和权限

# chmod -R 755 /var/www/html

# chown -R apache:apache /var/www/html

查看最终目录下的生成的项目文件

# ls -l

-rwxr-xr-x 1 apache apache 38912 Apr 16 15:04 db.sqlite3
-rwxr-xr-x 1 apache apache   541 Apr 16 14:50 manage.py
drwxr-xr-x 3 apache apache  4096 Apr 16 15:21 myproject
drwxr-xr-x 3 apache apache  4096 Apr 16 15:05 static

最终浏览器访问django项目

Tip:保证windows本地添加django服务器的HOST域名

django测试页面

项目主页, 输入之前创建的管理员账号密码

项目后台

Finished...

最近研究了下kubernetes用的比较火的Helm, Helm作为一个包管理工具, 它把Kubernetes资源(比如deployments、services或 ingress等) 打包到一个chart中，方便我们将其chart保存到chart仓库用来存储和分享, Helm支持发布应用配置的版本管理, 使发布可配置, 它最终简化了Kubernetes部署应用的版本控制、打包、发布、删除、更新等操作。

其实Helm和我们的ansible playbook有一些类似的地方就是, 它支持变量预定义, 使我们每一个kube脚本将一些重复的配置使用变量代替, 方便我们对一个project release的管理和批量部署, 升级, 回滚等操作.

Let's roll out...

安装环境

Local Desktop: MacOS

Virtual Machine: Virtual Box

Virtual System: CentOS 7.4

Kubernetes: Kubernetes1.9

Docker: 17.03.2-ce

Helm: helm-v2.7.0

kube-master 10.110.16.10

kube-node-1 10.110.16.11

一. 系统环境配置

1.关闭SELINUX和firewall

# vi /etc/sysconfig/selinux

...
SELINUX=disabled 
...

# setenforce 0

# systemctl stop firewalld  && systemctl disable firewalld

2.安装k8s环境.

http://www.showerlee.com/archives/2200

二. Helm配置

1.Helm安装

# wget https://storage.googleapis.com/kubernetes-helm/helm-v2.7.0-linux-amd64.tar.gz

# tar -zxvf helm-v2.7.0-linux-amd64.tar.gz

# mv linux-amd64/helm /usr/local/bin/

2.添加tiller到k8s service account

# kubectl create serviceaccount --namespace kube-system tiller

# kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

3.使用阿里云tiller镜像以及tiller账户初始化helm, 将tiller部署到k8s deployment下.

# vi ~/.helm/repository/repositories.yaml

Tip: username, password为你的阿里云账号密码

apiVersion: v1
generated: 2018-04-13T23:48:19.490774427-04:00
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/stable-index.yaml
  certFile: ""
  keyFile: ""
  name: stable
  password: "password"
  url: https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
  username: "username"
- caFile: ""
  cache: /root/.helm/repository/cache/local-index.yaml
  certFile: ""
  keyFile: ""
  name: local
  password: ""
  url: http://127.0.0.1:8879/charts
  username: ""

#  helm init --service-account tiller --upgrade --tiller-image=registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.7.0

Tip: 这里helm可以理解为一个操作tiller服务的客户端, tiller作为部署到k8s下的一个deployment, 负责去将我们的chart脚本解析给k8s去做进一步的部署工作.

4.检查tiller是否部署到k8s

# kubectl get pods --namespace kube-system

NAME                                  READY     STATUS    RESTARTS   AGE
etcd-kube-master                      1/1       Running   0          26d
kube-apiserver-kube-master            1/1       Running   0          26d
kube-controller-manager-kube-master   1/1       Running   1          26d
kube-dns-6f4fd4bdf-54smn              3/3       Running   0          26d
kube-flannel-ds-gwl2z                 1/1       Running   0          26d
kube-flannel-ds-m754s                 1/1       Running   0          26d
kube-proxy-697qx                      1/1       Running   0          26d
kube-proxy-cvfd9                      1/1       Running   0          26d
kube-scheduler-kube-master            1/1       Running   1          26d
tiller-deploy-cf797bfbf-rnk4k         1/1       Running   0          1h

5.创建一个chart范例

# helm create helm-chart

# tree ./helm-chart

./helm-chart
├── charts
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── ingress.yaml
│   ├── NOTES.txt
│   └── service.yaml
└── values.yaml

Tip: 可以看到helm默认创建了一个chart表结构, 这里的templates下面放的大部分为k8s的部署脚本, values.yaml和chart.yaml为主要的参数文件存放一些变量供k8s yaml文件调用, 有需要的小伙伴可以将自己的k8s脚本与默认进行替换.

6.检查chart语法

# helm lint ./helm-chart

7.使用默认chart部署到k8s

# helm install --name example1 ./helm-chart --set service.type=NodePort

Tip: 这里 --name命名我们这个chart release的名称, --set service.type=NodePort为将我们的任意node的ip映射到我们部署的pod, 以供访问.

# helm install --name example1 ./helm-chart --set service.type=NodePort
NAME:   example1
LAST DEPLOYED: Sat Apr 14 01:08:16 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                 TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)       AGE
example1-helm-chart  NodePort  10.105.111.66  <none>       80:25146/TCP  0s

==> v1beta1/Deployment
NAME                 DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
example1-helm-chart  1        1        1           0          0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
example1-helm-chart-7975cbf9b7-86vx5  0/1    ContainerCreating  0         0s


NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services example1-helm-chart)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

我们可以使用上面的NOTES去访问我们的部署网站

# curl 10.110.16.10:25146

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

8.查看当前的部署列表

# helm ls

NAME    	REVISION	UPDATED                 	STATUS  	CHART           	NAMESPACE
example1	1       	Sat Apr 14 01:08:16 2018	DEPLOYED	helm-chart-0.1.0	default

# kubectl get deployment

NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
example1-helm-chart   1         1         1            1           4m

9.打包chart

# helm package ./helm-chart --debug

10.使用包去做release部署

# helm install --name example2 helm-chart-0.1.0.tgz --set service.type=NodePort

11.升级当前release

# helm upgrade example2 ./helm-chart

12.回滚当前release

# helm rollback example2 1

13.删除该release

# helm delete example2

# helm del --purge example2

14.查看release历史删除记录

Tip: 如果删除时未使用--purge参数可查看删除记录

# helm ls --deleted -d

NAME    	REVISION	UPDATED                 	STATUS 	CHART           	NAMESPACE
example2	2       	Sat Apr 14 00:14:54 2018	DELETED	helm-chart-0.1.0	default

这里作者就不继续介绍helm chart的一些语法结构了, 有需要的小伙伴可以直接访问Helm官方去查看相关文档

https://docs.helm.sh

Finished...

安装环境

System: CentOS 7.4

Kubernetes: Kubernetes1.9

Docker: 17.03.2-ce

kube-master 10.110.16.10

kube-node-1 10.110.16.11

一. NFS配置:

1. NFS依赖包安装

在Master与Node分别安装NFS组件

# yum install nfs-utils -y

Tip: 这里需保证nfs-utils安装到所有master和node中, 否则容器挂载NFS时会报错.

2. 为Master下mysql data和wordpress源码配置NFS共享目录

# systemctl enable nfs-server && systemctl start nfs-server

# mkdir -p /kube/mysql-db

# mkdir -p /kube/wordpress

# chown nfsnobody:nfsnobody /kube/mysql-db

# chown nfsnobody:nfsnobody /kube/wordpress

# chmod 755 /kube/mysql-db

# chmod 755 /kube/wordpress

# echo -e "/kube/mysql-db    kube-*(rw,sync,no_subtree_check,no_root_squash)" > /etc/exports

# echo -e "/kube/wordpress    kube-*(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports

Tip: 这里kube-*限制只有kube相关的server才能连接Master下NFS共享目录, no_root_squash参数保证wordpress-mysql pod在初始化mysql配置的时候向在其下挂载的/var/lib/mysql目录有写入权限

3.应用配置

# exportfs -a

二. Persistent volume配置

1.为mysql data与wordpress源码存储创建Persistent volume
# kubectl create -f mysql-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    app: mysql
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /kube/mysql-db
    server: kube-master

# kubectl create -f wordpress-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: wp-pv
  labels:
    app: wordpress
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /kube/wordpress
    server: kube-master

2.创建存放mysql data的PVC

# kubectl create -f mysql-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-pv-claim
  labels:
    app: mysql
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

3.创建存放wordpress源码的PVC
# kubectl create -f wordpress-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

查看绑定

# kubectl get pvc

NAME             STATUS    VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-pv-claim   Bound     mysql-pv   5Gi        RWO                           3m
wp-pv-claim      Bound     wp-pv      5Gi        RWO                           6s

三. Secret配置

1.创建mysql root password

# kubectl create secret generic mysql-pass --from-literal='password=countonme'

四. Deployment配置

1.部署mysql deployment with PVC
# kubectl create -f mysql-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

2.部署wordpress deployment with PVC

# kubectl create -f wordpress-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
    spec:
      containers:
      - image: wordpress:4.8-apache
        name: wordpress
        env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql
        - name: WORDPRESS_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          claimName: wp-pv-claim

3.Service配置

Tip: 这里我们开启了node IP的80端口的外部访问权限, 可以方便我们直接利用主机去访问虚拟机任意Node地址从而登录我们的Wordpress网站.

# kubectl create -f wp-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
      nodePort: 80
  selector:
    app: wordpress
    tier: frontend
  type: NodePort

Tip: 这里service定义的name: wordpress-mysql保证我们wordpress-deployment.yaml定义的如下环境变量可以作为有效的域名成功去访问我们的mysql容器, 保证网站服务器与数据库服务器的通讯.

env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql

五. 验证结果

1.访问wordpress主页

这里我们可以直接在浏览器访问任意node的IP地址从而进入wordpress主页

添加相关信息并初始化安装WordPress

完成安装

后台

主页

2.查看NFS主机对在容器里的mysql data与wordpress root dir的目录挂载.

有兴趣的小伙伴可以直接访问如下代码仓库去下载本文相关代码:

https://git.showerlee.com/showerlee/kube-deploy/tree/master/wordpress-mysql

Finished...

后记:

如果我们使用helm包管理去部署wordpress, 将大大简化我们的工作量.

这里我的代码仓库提供了wordpress chart部署脚本, 以下是详细的部署步骤:

Prerequisite:

Kubernetes cluster setup

http://www.showerlee.com/archives/2200

Helm setup

http://www.showerlee.com/archives/2455

Helm deployment:

# git clone git@git.showerlee.com:showerlee/kube-deploy.git

# cd kube-deploy

# kubectl create secret generic mysql-pass --from-literal='password=countonme'

# helm install --name wordpress-mysql ./wordpress-helm-chart --set service.type=NodePort

NAME:   wordpress-mysql
LAST DEPLOYED: Sat Apr 14 03:09:46 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/PersistentVolume
NAME      CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                   STORAGECLASS  REASON  AGE
mysql-pv  5Gi       RWO           Recycle         Bound   default/mysql-pv-claim  1s
wp-pv     5Gi       RWO           Recycle         Bound   default/wp-pv-claim     1s

==> v1/PersistentVolumeClaim
NAME            STATUS  VOLUME    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
mysql-pv-claim  Bound   mysql-pv  5Gi       RWO           1s
wp-pv-claim     Bound   wp-pv     5Gi       RWO           1s

==> v1/Service
NAME             TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)    AGE
wordpress-mysql  ClusterIP  None           <none>       3306/TCP   1s
wordpress        NodePort   10.110.14.233  <none>       80:80/TCP  1s

==> v1/Deployment
NAME             AGE
wordpress-mysql  1s
wordpress        1s


NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services wordpress-mysql-wordpress-helm-chart)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

Secrets是一个包含敏感数据的对象，例如我们常用的密码，令牌或密钥等,  我们编写yaml如果直接明文这些信息则会将我们的敏感信息暴漏在我们的脚本中; 所以将其放置在Secret对象中可以更好地控制它的使用方式，并降低意外暴露的风险。

Pod可以引用我们事先创建好的Secrets键值对到环境变量, 通过获取环境变量的键值对动态更新我们Pod的环境配置, 从而实现动态配置更新.

1. 创建一个secret

# kubectl create secret generic secret-demo --from-literal='password=countonme'

2. 查看创建好的secret

# kubectl get secret secret-demo

NAME          TYPE      DATA      AGE
secret-demo   Opaque    1         13s

3.创建一个Pod并引用这个secret

# vi secret-env-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    env:
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: secret-demo
          key: password

# kubectl create -f secret-env-pod.yaml

4.查看secret

# kubectl describe secret

5.查看变量是否引入Pod

# kubectl exec -ti httpd-pod env

PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=httpd-pod
TERM=xterm
PASSWORD=countonme
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
HTTPD_PREFIX=/usr/local/apache2
NGHTTP2_VERSION=1.18.1-1
OPENSSL_VERSION=1.0.2l-1~bpo8+1
HTTPD_VERSION=2.4.29
HTTPD_SHA256=777753a5a25568a2a27428b2214980564bc1c38c1abf9ccc7630b639991f7f00
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=
HOME=/root

可以看到Pod的环境变量里已经引入一组键值对PASSWORD=countonme

6.向Pod挂载目录写入secret文件.

# vi secret-vol-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod-secret-vol
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    volumeMounts:
    - name: secret
      mountPath: "/mnt"
      readOnly: true
  volumes:
  - name: secret
    secret:
      secretName: secret-demo

# kubectl create -f secret-vol-pod.yaml

# kubectl exec -it httpd-pod-secret-vol cat /mnt/password

countonme

可以看到该Pod下面有一个文件名为password, 内容为countonme的文本文件. 

Config Map

1.创建config map

# vi cfgmap-demo.yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  name: cfgmap-demo

# kubectl create -f cfgmap-demo.yaml

2.查看config map

# kubectl get configmap cfgmap-demo -o yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  creationTimestamp: 2018-02-24T07:11:01Z
  name: cfgmap-demo
  namespace: default
  resourceVersion: "1064654"
  selfLink: /api/v1/namespaces/default/configmaps/cfgmap-demo
  uid: de9248d1-1931-11e8-9e24-00163e0e24bf

3. 修改config map

# vi cfgmap-demo.yaml

添加一行键值对

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
  http_port: "80"
kind: ConfigMap
metadata:
  name: cfgmap-demo

更新config map

# kubectl replace -f cfgmap-demo.yaml

查看更新后的config map

# kubectl get configmap cfgmap-demo -o yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_port: "80"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  creationTimestamp: 2018-02-24T07:11:01Z
  name: cfgmap-demo
  namespace: default
  resourceVersion: "1065520"
  selfLink: /api/v1/namespaces/default/configmaps/cfgmap-demo
  uid: de9248d1-1931-11e8-9e24-00163e0e24bf

4.创建一个Pod并引用这个config map

# vi cfgmap-env-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: cfgmap-httpd-pod
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    envFrom:
    - configMapRef:
        name: cfgmap-demo

# kubectl create -f cfgmap-env-pod.yaml

5.查看config map的键值对是否引入Pod

# kubectl exec -ti cfgmap-httpd-pod env

PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=cfgmap-httpd-pod
TERM=xterm
db_port=3306
http_port=80
http_url=http://www.example.com
database=db.example.com
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
HTTPD_PREFIX=/usr/local/apache2
NGHTTP2_VERSION=1.18.1-1
OPENSSL_VERSION=1.0.2l-1~bpo8+1
HTTPD_VERSION=2.4.29
HTTPD_SHA256=777753a5a25568a2a27428b2214980564bc1c38c1abf9ccc7630b639991f7f00
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=
HOME=/root

可以看到我们Config map下的所有键值对已经成功引入Pod环境变量.

相关代码:

https://git.showerlee.com/showerlee/kube-deploy

Finished...