这里k8s官方给我们提供了两种比较主流的连接k8s API cluster的语言, 一种是GO, 另外一种就是我们DevOps比较主流的Python.

今天这里我将给大家介绍如何使用python client扩展包去编写脚本并远程连接k8s API cluster.

这样我们平时除了可以远程使用ssh+kubectl去与kubernetes进行命令行数据交互外, 同样可以在远程不依赖命令行直接连接k8s cluster API cluster, 与k8s进行python API交互, 方便我们后期利用python去对k8s进行自动化集成与二次开发.

这里我们不多说, 直接进入我们的runbook配置环节:

一.部署python client环境连接我们的k8s API cluster.

1.安装python3.6.5源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel -y

2.编译安装python3.6.5以及pip package manager

# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz --no-check-certificate

# tar xf Python-3.6.5.tar.xz

# cd Python-3.6.5

# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

3.安装virtualenv

# pip3.6 install --upgrade pip

# pip3.6 install virtualenv

4.配置加载virtualenv环境

# virtualenv -p /usr/local/bin/python3 .py3env

# source .py3env/bin/activate

5.安装kubernetes python client扩展包

# pip install kubernetes

二.在k8s master获取API cluster URL与token

我们需要在创建python脚本前, 在k8s主机上创建一个admin权限的service account, 并获取其token 用来作为我们的脚本凭证.

1.抓取Cluster URL地址

# APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")

# echo $APISERVER

2.创建k8s admin-user

# mkdir -p /kube/role

# cd /kube/role

在kube-system下创建admin-user

# vi CreateServiceAccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system

给admin-user赋予admin权限

# vi CreateServiceAccount.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

# kubectl create -f CreateServiceAccount.yaml
# kubectl create -f RoleBinding.yaml

3.获取admin-user token

# Token=$(kubectl describe secret $(kubectl get secret -n kube-system | grep ^admin-user | awk '{print $1}') -n kube-system | grep -E '^token'| awk '{print $2}')

# echo $Token

最后将token与APISERVER地址返回内容复制到python client主机上, 供脚本使用.

三. 在python client主机上编写脚本

1. 创建目录结构

# mkdir -p /kube/auth

# cd /kube/auth

# touch token.txt

2. 创建token.txt文件并将k8s主机上获取的Token字符串复制到该文件

这里我们获取的token会引入到我们的脚本下, 作为bearer authorization的api key与远程k8s API建立认证连接.

3. 编写python client脚本

# vi k8s_auth.py

#!/usr/bin/env python


from kubernetes import client, config


def main():
    # Define the barer token we are going to use to authenticate.
    # See here to create the token:
    # https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/

    with open('token.txt', 'r') as file:
        Token = file.read().strip('\n')

    APISERVER = 'https://10.110.16.14:6443'

    # Create a configuration object
    configuration = client.Configuration()

    # Specify the endpoint of your Kube cluster
    configuration.host = APISERVER

    # Security part.
    # In this simple example we are not going to verify the SSL certificate of
    # the remote cluster (for simplicity reason)
    configuration.verify_ssl = False

    # Nevertheless if you want to do it you can with these 2 parameters
    # configuration.verify_ssl=True
    # ssl_ca_cert is the filepath to the file that contains the certificate.
    # configuration.ssl_ca_cert="certificate"
    configuration.api_key = {"authorization": "Bearer " + Token}

    # configuration.api_key["authorization"] = "bearer " + Token
    # configuration.api_key_prefix['authorization'] = 'Bearer'
    # configuration.ssl_ca_cert = 'ca.crt'
    # Create a ApiClient with our config
    client.Configuration.set_default(configuration)

    # Do calls
    v1 = client.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = v1.list_pod_for_all_namespaces(watch=False)
    for i in ret.items:
        print("%s\t%s\t%s" %
              (i.status.pod_ip, i.metadata.namespace, i.metadata.name))


if __name__ == '__main__':
    main()

这个脚本通过抓取同目录的k8s token字符串, 以及我们之前获取到的api server地址,利用python kubernetes扩展模块连接我们的远程API, 最终实现打印我们k8s所有namespace下的所有pods.

实际效果与kubectl get pods --all-namespaces类似.

4. 修改权限并执行

# chmod 755 k8s_auth.py

# ./k8s_auth.py

Listing pods with their IPs:
...
10.244.1.3	default	db
10.244.1.2	default	kubernetes-downwardapi-volume-example
10.244.1.245	default	nginx1-7-deployment-667547b6d8-c5nsr
10.244.1.239	default	nginx1-7-deployment-667547b6d8-mjxq4
10.244.1.247	default	nginx1-8-deployment-d46768cf9-888v8
10.244.1.242	default	nginx1-8-deployment-d46768cf9-fglq2
10.110.16.14	kube-system	etcd-kube-master
10.110.16.14	kube-system	kube-apiserver-kube-master
10.110.16.14	kube-system	kube-controller-manager-kube-master
10.244.1.241	kube-system	kube-dns-6f4fd4bdf-qsxrj
10.110.16.14	kube-system	kube-flannel-ds-5sdlg
10.110.16.15	kube-system	kube-flannel-ds-qctv4
10.110.16.14	kube-system	kube-proxy-5nscq
10.110.16.15	kube-system	kube-proxy-6g7jc
10.110.16.14	kube-system	kube-scheduler-kube-master
10.244.1.246	kube-system	kubernetes-dashboard-58f5cb49c-6dxgn
10.244.1.240	kube-system	tiller-deploy-cbb85d8dc-f6rsv
10.110.16.15	kube-system	traefik-ingress-lb-765c44656f-fkzxb
10.244.1.243	spinnaker	kubelive-create-bucket-wxlv4
10.244.1.244	spinnaker	kubelive-delete-jobs-zhn75

这样我们就成功的编写完成python client脚本连接k8s API cluster, 将所有namespace下的pod的基本信息打印出来.

Finished...

最近研究了下kubernetes用的比较火的Helm, Helm作为一个包管理工具, 它把Kubernetes资源(比如deployments、services或 ingress等) 打包到一个chart中，方便我们将其chart保存到chart仓库用来存储和分享, Helm支持发布应用配置的版本管理, 使发布可配置, 它最终简化了Kubernetes部署应用的版本控制、打包、发布、删除、更新等操作。

其实Helm和我们的ansible playbook有一些类似的地方就是, 它支持变量预定义, 使我们每一个kube脚本将一些重复的配置使用变量代替, 方便我们对一个project release的管理和批量部署, 升级, 回滚等操作.

Let's roll out...

安装环境

Local Desktop: MacOS

Virtual Machine: Virtual Box

Virtual System: CentOS 7.4

Kubernetes: Kubernetes1.9

Docker: 17.03.2-ce

Helm: helm-v2.7.0

kube-master 10.110.16.10

kube-node-1 10.110.16.11

一. 系统环境配置

1.关闭SELINUX和firewall

# vi /etc/sysconfig/selinux

...
SELINUX=disabled 
...

# setenforce 0

# systemctl stop firewalld  && systemctl disable firewalld

2.安装k8s环境.

http://www.showerlee.com/archives/2200

二. Helm配置

1.Helm安装

# wget https://storage.googleapis.com/kubernetes-helm/helm-v2.7.0-linux-amd64.tar.gz

# tar -zxvf helm-v2.7.0-linux-amd64.tar.gz

# mv linux-amd64/helm /usr/local/bin/

2.添加tiller到k8s service account

# kubectl create serviceaccount --namespace kube-system tiller

# kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

3.使用阿里云tiller镜像以及tiller账户初始化helm, 将tiller部署到k8s deployment下.

# vi ~/.helm/repository/repositories.yaml

Tip: username, password为你的阿里云账号密码

apiVersion: v1
generated: 2018-04-13T23:48:19.490774427-04:00
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/stable-index.yaml
  certFile: ""
  keyFile: ""
  name: stable
  password: "password"
  url: https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
  username: "username"
- caFile: ""
  cache: /root/.helm/repository/cache/local-index.yaml
  certFile: ""
  keyFile: ""
  name: local
  password: ""
  url: http://127.0.0.1:8879/charts
  username: ""

#  helm init --service-account tiller --upgrade --tiller-image=registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.7.0

Tip: 这里helm可以理解为一个操作tiller服务的客户端, tiller作为部署到k8s下的一个deployment, 负责去将我们的chart脚本解析给k8s去做进一步的部署工作.

4.检查tiller是否部署到k8s

# kubectl get pods --namespace kube-system

NAME                                  READY     STATUS    RESTARTS   AGE
etcd-kube-master                      1/1       Running   0          26d
kube-apiserver-kube-master            1/1       Running   0          26d
kube-controller-manager-kube-master   1/1       Running   1          26d
kube-dns-6f4fd4bdf-54smn              3/3       Running   0          26d
kube-flannel-ds-gwl2z                 1/1       Running   0          26d
kube-flannel-ds-m754s                 1/1       Running   0          26d
kube-proxy-697qx                      1/1       Running   0          26d
kube-proxy-cvfd9                      1/1       Running   0          26d
kube-scheduler-kube-master            1/1       Running   1          26d
tiller-deploy-cf797bfbf-rnk4k         1/1       Running   0          1h

5.创建一个chart范例

# helm create helm-chart

# tree ./helm-chart

./helm-chart
├── charts
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── ingress.yaml
│   ├── NOTES.txt
│   └── service.yaml
└── values.yaml

Tip: 可以看到helm默认创建了一个chart表结构, 这里的templates下面放的大部分为k8s的部署脚本, values.yaml和chart.yaml为主要的参数文件存放一些变量供k8s yaml文件调用, 有需要的小伙伴可以将自己的k8s脚本与默认进行替换.

6.检查chart语法

# helm lint ./helm-chart

7.使用默认chart部署到k8s

# helm install --name example1 ./helm-chart --set service.type=NodePort

Tip: 这里 --name命名我们这个chart release的名称, --set service.type=NodePort为将我们的任意node的ip映射到我们部署的pod, 以供访问.

# helm install --name example1 ./helm-chart --set service.type=NodePort
NAME:   example1
LAST DEPLOYED: Sat Apr 14 01:08:16 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                 TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)       AGE
example1-helm-chart  NodePort  10.105.111.66  <none>       80:25146/TCP  0s

==> v1beta1/Deployment
NAME                 DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
example1-helm-chart  1        1        1           0          0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
example1-helm-chart-7975cbf9b7-86vx5  0/1    ContainerCreating  0         0s


NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services example1-helm-chart)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

我们可以使用上面的NOTES去访问我们的部署网站

# curl 10.110.16.10:25146

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

8.查看当前的部署列表

# helm ls

NAME    	REVISION	UPDATED                 	STATUS  	CHART           	NAMESPACE
example1	1       	Sat Apr 14 01:08:16 2018	DEPLOYED	helm-chart-0.1.0	default

# kubectl get deployment

NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
example1-helm-chart   1         1         1            1           4m

9.打包chart

# helm package ./helm-chart --debug

10.使用包去做release部署

# helm install --name example2 helm-chart-0.1.0.tgz --set service.type=NodePort

11.升级当前release

# helm upgrade example2 ./helm-chart

12.回滚当前release

# helm rollback example2 1

13.删除该release

# helm delete example2

# helm del --purge example2

14.查看release历史删除记录

Tip: 如果删除时未使用--purge参数可查看删除记录

# helm ls --deleted -d

NAME    	REVISION	UPDATED                 	STATUS 	CHART           	NAMESPACE
example2	2       	Sat Apr 14 00:14:54 2018	DELETED	helm-chart-0.1.0	default

这里作者就不继续介绍helm chart的一些语法结构了, 有需要的小伙伴可以直接访问Helm官方去查看相关文档

https://docs.helm.sh

Finished...

安装环境

System: CentOS 7.4

Kubernetes: Kubernetes1.9

Docker: 17.03.2-ce

kube-master 10.110.16.10

kube-node-1 10.110.16.11

一. NFS配置:

1. NFS依赖包安装

在Master与Node分别安装NFS组件

# yum install nfs-utils -y

Tip: 这里需保证nfs-utils安装到所有master和node中, 否则容器挂载NFS时会报错.

2. 为Master下mysql data和wordpress源码配置NFS共享目录

# systemctl enable nfs-server && systemctl start nfs-server

# mkdir -p /kube/mysql-db

# mkdir -p /kube/wordpress

# chown nfsnobody:nfsnobody /kube/mysql-db

# chown nfsnobody:nfsnobody /kube/wordpress

# chmod 755 /kube/mysql-db

# chmod 755 /kube/wordpress

# echo -e "/kube/mysql-db    kube-*(rw,sync,no_subtree_check,no_root_squash)" > /etc/exports

# echo -e "/kube/wordpress    kube-*(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports

Tip: 这里kube-*限制只有kube相关的server才能连接Master下NFS共享目录, no_root_squash参数保证wordpress-mysql pod在初始化mysql配置的时候向在其下挂载的/var/lib/mysql目录有写入权限

3.应用配置

# exportfs -a

二. Persistent volume配置

1.为mysql data与wordpress源码存储创建Persistent volume
# kubectl create -f mysql-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    app: mysql
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /kube/mysql-db
    server: kube-master

# kubectl create -f wordpress-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: wp-pv
  labels:
    app: wordpress
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /kube/wordpress
    server: kube-master

2.创建存放mysql data的PVC

# kubectl create -f mysql-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-pv-claim
  labels:
    app: mysql
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

3.创建存放wordpress源码的PVC
# kubectl create -f wordpress-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

查看绑定

# kubectl get pvc

NAME             STATUS    VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-pv-claim   Bound     mysql-pv   5Gi        RWO                           3m
wp-pv-claim      Bound     wp-pv      5Gi        RWO                           6s

三. Secret配置

1.创建mysql root password

# kubectl create secret generic mysql-pass --from-literal='password=countonme'

四. Deployment配置

1.部署mysql deployment with PVC
# kubectl create -f mysql-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

2.部署wordpress deployment with PVC

# kubectl create -f wordpress-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
    spec:
      containers:
      - image: wordpress:4.8-apache
        name: wordpress
        env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql
        - name: WORDPRESS_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          claimName: wp-pv-claim

3.Service配置

Tip: 这里我们开启了node IP的80端口的外部访问权限, 可以方便我们直接利用主机去访问虚拟机任意Node地址从而登录我们的Wordpress网站.

# kubectl create -f wp-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
      nodePort: 80
  selector:
    app: wordpress
    tier: frontend
  type: NodePort

Tip: 这里service定义的name: wordpress-mysql保证我们wordpress-deployment.yaml定义的如下环境变量可以作为有效的域名成功去访问我们的mysql容器, 保证网站服务器与数据库服务器的通讯.

env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql

五. 验证结果

1.访问wordpress主页

这里我们可以直接在浏览器访问任意node的IP地址从而进入wordpress主页

添加相关信息并初始化安装WordPress

完成安装

后台

主页

2.查看NFS主机对在容器里的mysql data与wordpress root dir的目录挂载.

有兴趣的小伙伴可以直接访问如下代码仓库去下载本文相关代码:

https://git.showerlee.com/showerlee/kube-deploy/tree/master/wordpress-mysql

Finished...

后记:

如果我们使用helm包管理去部署wordpress, 将大大简化我们的工作量.

这里我的代码仓库提供了wordpress chart部署脚本, 以下是详细的部署步骤:

Prerequisite:

Kubernetes cluster setup

http://www.showerlee.com/archives/2200

Helm setup

http://www.showerlee.com/archives/2455

Helm deployment:

# git clone git@git.showerlee.com:showerlee/kube-deploy.git

# cd kube-deploy

# kubectl create secret generic mysql-pass --from-literal='password=countonme'

# helm install --name wordpress-mysql ./wordpress-helm-chart --set service.type=NodePort

NAME:   wordpress-mysql
LAST DEPLOYED: Sat Apr 14 03:09:46 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/PersistentVolume
NAME      CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                   STORAGECLASS  REASON  AGE
mysql-pv  5Gi       RWO           Recycle         Bound   default/mysql-pv-claim  1s
wp-pv     5Gi       RWO           Recycle         Bound   default/wp-pv-claim     1s

==> v1/PersistentVolumeClaim
NAME            STATUS  VOLUME    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
mysql-pv-claim  Bound   mysql-pv  5Gi       RWO           1s
wp-pv-claim     Bound   wp-pv     5Gi       RWO           1s

==> v1/Service
NAME             TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)    AGE
wordpress-mysql  ClusterIP  None           <none>       3306/TCP   1s
wordpress        NodePort   10.110.14.233  <none>       80:80/TCP  1s

==> v1/Deployment
NAME             AGE
wordpress-mysql  1s
wordpress        1s


NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services wordpress-mysql-wordpress-helm-chart)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

Secrets是一个包含敏感数据的对象，例如我们常用的密码，令牌或密钥等,  我们编写yaml如果直接明文这些信息则会将我们的敏感信息暴漏在我们的脚本中; 所以将其放置在Secret对象中可以更好地控制它的使用方式，并降低意外暴露的风险。

Pod可以引用我们事先创建好的Secrets键值对到环境变量, 通过获取环境变量的键值对动态更新我们Pod的环境配置, 从而实现动态配置更新.

1. 创建一个secret

# kubectl create secret generic secret-demo --from-literal='password=countonme'

2. 查看创建好的secret

# kubectl get secret secret-demo

NAME          TYPE      DATA      AGE
secret-demo   Opaque    1         13s

3.创建一个Pod并引用这个secret

# vi secret-env-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    env:
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: secret-demo
          key: password

# kubectl create -f secret-env-pod.yaml

4.查看secret

# kubectl describe secret

5.查看变量是否引入Pod

# kubectl exec -ti httpd-pod env

PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=httpd-pod
TERM=xterm
PASSWORD=countonme
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
HTTPD_PREFIX=/usr/local/apache2
NGHTTP2_VERSION=1.18.1-1
OPENSSL_VERSION=1.0.2l-1~bpo8+1
HTTPD_VERSION=2.4.29
HTTPD_SHA256=777753a5a25568a2a27428b2214980564bc1c38c1abf9ccc7630b639991f7f00
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=
HOME=/root

可以看到Pod的环境变量里已经引入一组键值对PASSWORD=countonme

6.向Pod挂载目录写入secret文件.

# vi secret-vol-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod-secret-vol
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    volumeMounts:
    - name: secret
      mountPath: "/mnt"
      readOnly: true
  volumes:
  - name: secret
    secret:
      secretName: secret-demo

# kubectl create -f secret-vol-pod.yaml

# kubectl exec -it httpd-pod-secret-vol cat /mnt/password

countonme

可以看到该Pod下面有一个文件名为password, 内容为countonme的文本文件. 

Config Map

1.创建config map

# vi cfgmap-demo.yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  name: cfgmap-demo

# kubectl create -f cfgmap-demo.yaml

2.查看config map

# kubectl get configmap cfgmap-demo -o yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  creationTimestamp: 2018-02-24T07:11:01Z
  name: cfgmap-demo
  namespace: default
  resourceVersion: "1064654"
  selfLink: /api/v1/namespaces/default/configmaps/cfgmap-demo
  uid: de9248d1-1931-11e8-9e24-00163e0e24bf

3. 修改config map

# vi cfgmap-demo.yaml

添加一行键值对

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_url: http://www.example.com
  http_port: "80"
kind: ConfigMap
metadata:
  name: cfgmap-demo

更新config map

# kubectl replace -f cfgmap-demo.yaml

查看更新后的config map

# kubectl get configmap cfgmap-demo -o yaml

apiVersion: v1
data:
  database: db.example.com
  db_port: "3306"
  http_port: "80"
  http_url: http://www.example.com
kind: ConfigMap
metadata:
  creationTimestamp: 2018-02-24T07:11:01Z
  name: cfgmap-demo
  namespace: default
  resourceVersion: "1065520"
  selfLink: /api/v1/namespaces/default/configmaps/cfgmap-demo
  uid: de9248d1-1931-11e8-9e24-00163e0e24bf

4.创建一个Pod并引用这个config map

# vi cfgmap-env-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: cfgmap-httpd-pod
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always
    envFrom:
    - configMapRef:
        name: cfgmap-demo

# kubectl create -f cfgmap-env-pod.yaml

5.查看config map的键值对是否引入Pod

# kubectl exec -ti cfgmap-httpd-pod env

PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=cfgmap-httpd-pod
TERM=xterm
db_port=3306
http_port=80
http_url=http://www.example.com
database=db.example.com
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
HTTPD_PREFIX=/usr/local/apache2
NGHTTP2_VERSION=1.18.1-1
OPENSSL_VERSION=1.0.2l-1~bpo8+1
HTTPD_VERSION=2.4.29
HTTPD_SHA256=777753a5a25568a2a27428b2214980564bc1c38c1abf9ccc7630b639991f7f00
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=
HOME=/root

可以看到我们Config map下的所有键值对已经成功引入Pod环境变量.

相关代码:

https://git.showerlee.com/showerlee/kube-deploy

Finished...

持久化卷下PV和PVC概念:

Persistent Volume（PV）是由管理员设置的存储，它是群集的一部分。就像节点是集群中的资源一样，PV 也是集群中的资源。 PV 是 Volume 之类的卷插件，但具有独立于使用 PV 的 Pod 的生命周期。此 API 对象包含存储实现的细节，即 NFS、iSCSI 或特定于云供应商的存储系统

PersistentVolumeClaim（PVC）是用户存储的请求。它与 Pod 相似。Pod 消耗节点资源，PVC 消耗 PV 资源。Pod 可以请求特定级别的资源（CPU 和内存）。PVC声明可以请求特定的大小和访问模式（例如，可以以读/写一次或 只读多次模式挂载）

它和普通Volume的区别是什么呢？

普通Volume和使用它的Pod之间是一种静态绑定关系，在定义Pod的文件里，同时定义了它使用的Volume。Volume是Pod的附属品，我们无法单独创建一个Volume，因为它不是一个独立的K8S资源对象。

如何简单理解持久化卷?

我们需要首先创建一个独立的持久化卷(PV)资源对象, 然后创建一个与PV绑定的PVC存储请求, 这个请求会事先定义accessModes, resources等资源配置, 最终我们会在Pod中挂载定义好的PVC以供我们数据存储使用

Let's start...

一. NFS安装配置

我们这里利用NFS去实现k8s持久化卷的配置

1,安装NFS server

# yum install nfs-utils -y

2.启动NFS服务

# systemctl enable nfs-server

# systemctl start nfs-server

3.配置NFS共享目录

# mkdir /srv/pv-demo

# chown nfsnobody:nfsnobody /srv/pv-demo

# chmod 755 /srv/pv-demo

# echo -e "/srv/pv-demo    kube-master(rw,sync)" > /etc/export

4.生效共享目录

# exportfs -a

因为资源有限, 我们最终在Master上创建一个NFS共享目录/srv/pv-demo, 以供我们后面的持久化卷使用, 有富裕的小伙伴可以创建一台与kube-master同一网段的独立server去充当NFS服务器, 

二. Persistent Volume配置

1.创建Persistent Volume

# vi pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-demo
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /srv/pv-demo
    server: kube-master

Tip: 这里的定义卷的大小为5G, 使用的accessmodes为ReadWriteOnce, PVC policy为Recycle, NFS共享目录为我们之前在master创建好的/srv/pv-demo, server为我们定义好的本地host kube-master

# kubectl create -f pv.yaml

2. 查看PV

# kubectl get pv pv-demo

NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM              STORAGECLASS   REASON    AGE
pv-demo   5Gi        RWO            Recycle          Bound     default/pvc-demo                            1h

3.创建Persistent Volume Claim

Tip: 这里PVC可以理解为在PV请求的资源, 也就是说所有我们的数据都会保存在PVC里, 任何PVC的删除操作都会清除我们存储在这里的数据.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

# kubectl create -f pvc.yaml

4.查看PVC

# kubectl get pvc pvc-demo

NAME       STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-demo   Bound     pv-demo   5Gi        RWO                           1h

5.创建一个Pod并使用该PVC

# vi pvpod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod
spec:
  containers:
  - image: httpd
    name: httpd-pod
    imagePullPolicy: Always
    volumeMounts:
    - mountPath: "/usr/local/apache2/htdocs/"
      name: httpd-volume
  volumes:
    - name: httpd-volume
      persistentVolumeClaim:
        claimName: pvc-demo

Tip: 这里需要保证claimName的值与我们之前创建的PVC name一致.

# kubectl create -f pvpod.yaml

Tip: 这里我们将PVC挂载到Pod的Apache根目录"/usr/local/apache2/htdocs/", 用来最终测试效果.

6. 查看Pod是否挂载PVC

# kubectl describe pv

Name:            pv-demo
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller=yes
StorageClass:
Status:          Bound
Claim:           default/pvc-demo
Reclaim Policy:  Recycle
Access Modes:    RWO
Capacity:        5Gi
Message:
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    kube-master
    Path:      /srv/pv-demo
    ReadOnly:  false
Events:        <none>

# kubectl describe pods

Name:         httpd-pod
Namespace:    default
Node:         kube-master/172.17.2.153
Start Time:   Fri, 23 Feb 2018 15:38:55 +0800
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           10.244.0.46
Containers:
  httpd-pod:
    Container ID:   docker://b7e5fd2732864934b732fdbd4bb24b3ccc8949c2e9d8832a36e271f2ee350b2b
    Image:          httpd
    Image ID:       docker-pullable://httpd@sha256:6e61d60e4142ea44e8e69b22f1e739d89e1dc8a2764182d7eecc83a5bb31181e
    Port:           <none>
    State:          Running
      Started:      Fri, 23 Feb 2018 15:38:59 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/local/apache2/htdocs from httpd-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bnkxx (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  httpd-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc-demo
    ReadOnly:   false
  default-token-bnkxx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bnkxx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                  Message
  ----    ------                 ----  ----                  -------
  Normal  Scheduled              58m   default-scheduler     Successfully assigned httpd-pod to kube-master
  Normal  SuccessfulMountVolume  58m   kubelet, kube-master  MountVolume.SetUp succeeded for volume "default-token-bnkxx"
  Normal  SuccessfulMountVolume  58m   kubelet, kube-master  MountVolume.SetUp succeeded for volume "pv-demo"
  Normal  Pulling                58m   kubelet, kube-master  pulling image "httpd"
  Normal  Pulled                 58m   kubelet, kube-master  Successfully pulled image "httpd"
  Normal  Created                58m   kubelet, kube-master  Created container
  Normal  Started                58m   kubelet, kube-master  Started container

通过返回打印的信息我们可以看到PVC已经成功mount到我们定义好的Pod的apache根目录下

7. 向Pod下apache根目录写入index.html

# kubectl exec -ti httpd-pod -- /bin/sh -c "echo 'This is a persistent volume from httpd-pod' > /usr/local/apache2/htdocs/index.html"

8.确认文件写入

# kubectl exec -ti httpd-pod -- cat /usr/local/apache2/htdocs/index.html 

This is a persistent volume from httpd-pod

9. 删除并重新创建Pod来验证数据是否会随Pod销毁而丢失.

# kubectl delete pod httpd-pod
# kubectl create -f pvpod.yaml
# kubectl exec -ti httpd-pod -- cat /usr/local/apache2/htdocs/index.html 

This is a persistent volume from httpd-pod

可以看到我们之前写入的index.html仍旧存储在PVC中, 证明其不会随Pod销毁而丢失.

10.查看Pod内网IP

# kubectl get pods -o wide

NAME        READY     STATUS    RESTARTS   AGE       IP            NODE
httpd-pod   1/1       Running   0          1m        10.244.0.46   kube-master

11.利用curl验证写入的index.html

# curl 10.244.0.46

This is a persistent volume from httpd-pod

这里我们成功在Pod下将PVC挂载到apache的家目录, 并返回HTML返回内容.

本文相关代码:

https://git.showerlee.com/showerlee/kube-deploy

大功告成...

Pod:

Pod是一组紧密关联的容器集合，它们共享PID、IPC、Network和UTS namespace，是Kubernetes调度的基本单位。Pod的设计理念是支持多个容器在一个Pod中共享网络和文件系统，可以通过进程间通信和文件共享这种简单高效的方式组合完成服务.

缺点: 不支持高并发, 高可用, 当Pod当机后无法自动恢复.

1.创建Pod

# vi pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: demo
spec:
  containers:
  - image: httpd
    name: httpd
    imagePullPolicy: Always

# kubectl create -f pod.yaml

2.查看Pod

# kubectl get pods

NAME    READY     STATUS    RESTARTS   AGE
demo    1/1       Running      0       8d

# kubectl describe pods

...

3.删除Pod

# kubectl delete pod demo

Replicaset:

Replicaset在继承Pod的所有特性的同时, 它可以利用预先创建好的模板定义副本数量并自动控制, 通过改变Pod副本数量实现Pod的扩容和缩容

缺点: 无法修改template模板, 也就无法发布新的镜像版本

1.创建Replicaset

# vi replicaset.yaml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: demo-rc
  labels:
    app: demo-rc
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-rc
  template:
    metadata:
      labels:
        app: demo-rc
    spec:
      containers:
      - name: httpd
        image: httpd
        imagePullPolicy: Always

# kubectl create -f replicaset.yaml

2.查看replicaset

# kubectl get replicaset

NAME      READY     STATUS    RESTARTS   AGE
demo-rc    1/1       Running      0       8d

# kubectl describe replicaset

...

3.删除replicaset

# kubectl delete replicaset demo-rc

Deployment

Deployment在继承Pod和Replicaset的所有特性的同时, 它可以实现对template模板进行实时滚动更新并具备我们线上的Application life circle的特性.

1.创建Deployment

# vi deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd-deployment
  labels:
    app: httpd-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpd-demo
  template:
    metadata:
      labels:
        app: httpd-demo
    spec:
      containers:
      - name: httpd
        image: httpd
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        env:
        - name: VERSION
          value: "v1"

# kubectl create -f deployment.yaml

2.查看Deployment

# kubectl get deployment

NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
httpd-deployment   2         2         2            2           8d

# kubectl get pods -o wide

NAME                               READY     STATUS    RESTARTS   AGE       IP            NODE
httpd-deployment-956697567-8mqch   1/1       Running   0          8d        10.244.0.36   kube-master
httpd-deployment-956697567-wcbs6   1/1       Running   0          8d        10.244.0.37   kube-master

# kubectl describe deployment

...

3.更新deployment

通过此命令可以呼出vi编辑器对模板进行编辑.

# kubectl edit -f deployment.yaml

通过此命令使当前编辑结果生效.

# kubectl apply -f deployment.yaml

再次查看可以看到老版本的deployment已经下架, 新版本的已经生效.

# kubectl get deployment

NAME                          DESIRED   CURRENT   READY     AGE
httpd-deployment-6b98d94474   0         0         0         1m
httpd-deployment-956697567    2         2         2         7m

4.扩容与缩容

可以修改replicas的赋值对deployment进行扩容与缩容

#  kubectl scale deployment/httpd-deployment --replicas=1

5.删除deployment

# kubectl delete deployment httpd-deployment

Lable

Label是attach到Pod的一对键/值对，用来传递用户定义的属性。比如，你可能创建了一个"tier"和“app”标签，通过Label（tier=frontend, app=myapp）来标记前端Pod容器，使用Label（tier=backend, app=myapp）标记后台Pod。然后可以使用Selectors选择带有特定Label的Pod，让具体某一个Pod或者Deployment去使用某一个Service实现特定的网络配置.

Service

Service是应用服务的抽象，通过labels为应用提供负载均衡和服务发现。匹配labels的Pod IP和端口列表组成endpoints，由kube-proxy负责将服务IP负载均衡到这些endpoints上。
每个Service都会自动分配一个cluster IP（仅在集群内部可访问的虚拟地址）和DNS名，其他容器可以通过该地址或DNS来访问服务，而不需要了解后端容器的运行。

1.更改NodePort限制

Kubernetes默认对外的NodePort限制范围为30000-32767, 这里如果要使用一些常用的端口(80, 8080, 443)需将这个范围放大.

# vi /etc/kubernetes/manifests/kube-apiserver.yaml

在--service-cluster-ip-range与insecure-port间添加如下node port配置

...
- --service-cluster-ip-range=10.96.0.0/12
- --service-node-port-range=0-32767
- --insecure-port=0
....

重启服务

# systemctl restart kubelet

2.创建Service

# vi svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: NodePort
  ports:
    - port: 80
      nodePort: 80
  selector:
    app: httpd-demo

# kubectl create -f svc.yaml

Tip: 如果要对某一Pod或deployment添加对外访问端口, 这里service添加的selector的键值需与之相对应.

3.查看开放端口

# kubectl get svc demo

NAME      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
demo      NodePort   10.100.96.157   <none>        80:80/TCP   1h

# kubectl describe service demo

Name:                     demo
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=httpd-demo
Type:                     NodePort
IP:                       10.100.96.157
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  80/TCP
Endpoints:                10.244.0.36:80,10.244.0.37:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

4.验证外网是否可以访问

本文相关代码:

https://git.showerlee.com/showerlee/kube-deploy

Finished...

最近研究了一下目前比较火的Kubernetes(k8s), 重点关注了下它最新的1.9版本, 这个版本较老的1.1版本的确简化了很多配置, 它利用kubeadm这个工具对全局进行批量化部署, 减轻了我们初学者起步的学习成本. 

目前主流的安装k8s系统平台有Centos7和ubuntu, 这里笔者因为对centos有常年的运维开发经验, 所以就选择前者.

另外官方已经在近期将kubernetes1.1版本的centos7配置从官网移除, 所以建议大家使用1.9版本完成所有的安装部署.

本文档推荐给大家的原因是因为目前网上基本上没有一个较为完整和正确率较高的k8s的安装文档, 笔者因此整合了网上零散的k8s资源, 给大家提供一个较为靠谱的离线安装k8s 1.9版本的安装范例.

什么是Kubernates?

Kubernetes 用于自动部署、扩展和管理容器化（containerized）应用程序的开源系统。它旨在提供“跨主机集群的自动部署、扩展以及运行应用程序容器的平台”。它支持一系列容器工具, 目前主流会使用Docker作为他的主流配置容器.

为什么要使用离线安装呢?

因为kubeadm默认要从google仓库下载镜像，但目前国内无法访问google仓库，所以这里从网上找到1.9的离线安装包，大家只需要将离线包的镜像导入到相应节点即可.

Let's start...

安装环境

Local Desktop: MacOS

Virtual Machine: Virtual Box

Virtual System: CentOS 7.4

Kubernetes: Kubernetes1.9

Docker: 17.03.2-ce

kube-master 10.110.16.10

kube-node-1 10.110.16.11

一. 系统环境配置

(Master Node都需要配置)

1.下载离线安装包

链接: https://pan.baidu.com/s/1c2O1gIW 密码: 9s92

下载到本地后, 上传到虚拟机root根目录

# tar jxvf k8s_images.tar.bz2

2. 安装依赖

Tip: 这里需要更新CentOS7内核到最新版本才能打开centos的路由功能以供k8s使用, 更新完毕需要重启系统使kernel新版本生效

# yum install policycoreutils-python libtool-ltdl libseccomp device-mapper-libs kernel ntpdate

# reboot

3.绑定本地host

# echo "10.110.16.10 kube-master" >> /etc/hosts

# echo "10.110.16.11 kube-node-1" >> /etc/hosts

4.添加kube-master到kube-node-1的秘钥认证(仅Master需要配置)

# ssh-keygen

# ssh-copy-id kube-node-1

5.关闭SELINUX和firewall

# vi /etc/sysconfig/selinux

...
SELINUX=disabled 
...

# setenforce 0

# systemctl stop firewalld  && systemctl disable firewalld

6.路由配置

# modprobe br_netfilter

# echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf

# echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf

# sysctl -p

7.关闭Swap分区

大家在安装CentOS7时如果配置了swap分区, 这里由于k8s禁止系统开启此功能, 需要提前进行关闭, 没有配置swap请无视.

查看swap分区目录

# swapon -s

Filename                                Type            Size    Used    Priority
/swapfile                               file    1996796 1214364 -1

关闭swap分区

# swapoff /swapfile

注释掉swap分区, 禁止开机启动swap

# vi /etc/fstab

...
#/dev/mapper/centos-swap swap                    swap    defaults        0 0
...

8.开启系统时间同步

# systemctl enable ntpdate && systemctl start ntpdate

二. Docker配置

(Master Node都需要配置)

1.安装docker-ce17.03

# rpm -ihv docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
# rpm -ivh docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm

2.修改Docker使用阿里的镜像

Tip: 访问https://cr.console.aliyun.com/#/accelerator注册账号, 在镜像加速器里获取个人的仓库地址, 并填入如下配置文件

# mkdir /etc/docker

# vi /etc/docker/daemon.json

...
{
  "registry-mirrors": ["https://xxxxxxx.mirror.aliyuncs.com"]
}
...

3.重启docker并设置开机启动
# systemctl daemon-reload
# systemctl restart docker
# systemctl enable docker

4.本地载入我们的离线镜像

# cd /root

# mv /root/k8s_images/docker_images/etcd-amd64\:v3.1.10.tar /root/k8s_images/docker_images/etcd-amd64_v3.1.10.tar

# vi load_image.sh

docker load < /root/k8s_images/docker_images/etcd-amd64_v3.1.10.tar
docker load </root/k8s_images/docker_images/flannel_v0.9.1-amd64.tar
docker load </root/k8s_images/docker_images/k8s-dns-dnsmasq-nanny-amd64_v1.14.7.tar
docker load </root/k8s_images/docker_images/k8s-dns-kube-dns-amd64_1.14.7.tar
docker load </root/k8s_images/docker_images/k8s-dns-sidecar-amd64_1.14.7.tar
docker load </root/k8s_images/docker_images/kube-apiserver-amd64_v1.9.0.tar
docker load </root/k8s_images/docker_images/kube-controller-manager-amd64_v1.9.0.tar
docker load </root/k8s_images/docker_images/kube-scheduler-amd64_v1.9.0.tar
docker load < /root/k8s_images/docker_images/kube-proxy-amd64_v1.9.0.tar
docker load </root/k8s_images/docker_images/pause-amd64_3.0.tar
docker load < /root/k8s_images/kubernetes-dashboard_v1.8.1.tar

# sh load_image.sh

三.Kubernetes配置

1.安装kubelet kubeadm kubectl(Master Node都需要配置)

# cd k8s_images
# rpm -ivh socat-1.7.3.2-2.el7.x86_64.rpm
# rpm -ivh kubernetes-cni-0.6.0-0.x86_64.rpm  kubelet-1.9.9-9.x86_64.rpm kubectl-1.9.0-0.x86_64.rpm
# rpm -ivh kubeadm-1.9.0-0.x86_64.rpm

2.重启kubelet并设置开机启动(Master Node都需要配置)

# systemctl enable kubelet && sudo systemctl start kubelet

3.修改docker配置文件中的Cgroup Driver参数与k8s一致(Master Node都需要配置)

# docker info |grep "Cgroup Driver"
如果"Cgroup Driver"value为 "cgroupfs", 将其值写入kubeadm配置文件
# vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

...
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver="cgroupfs"
...

# systemctl daemon-reload && systemctl restart kubelet

4.Master初始化(Master 需要配置)

# kubeadm reset

# kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.244.0.0/16

Tip: 这里的10.244.0.0/16与默认的配置文件网段一致, 如需调整请在后面的kube-flannel.yml文件做相应改变

如果一切没问题会output一个token返回命令

# kubeadm join --token 288f34.481c8faa5636966f 10.110.16.10:6443 --discovery-token-ca-cert-hash sha256:8036fac3b76e1a0dd189edaa8f7d36f2b51429dd0c0cf7ea0d78e7972d611002

24小时后这个token会失效, 需要重新生成, 使用如下命令进行生成.

# kubeadm token create

# kubeadm join --token "Your token code" "Your master ip address":6443

如果在24小时内忘记了，可以用如下命令获取.

# kubeadm token list

5.设置用户环境变量(Master 需要配置)

如果你使用的是root
# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
# source ~/.bash_profile

如果是非root用户

# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config

测试kubectl版本

# kubectl version

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

6.使用flannel组件进行k8s网络配置(Master 需要配置)

# wget https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

Tip: 这里的yml配置文件内的网段配置需与上面介绍的Master初始化的配置网段"--pod-network-cidr=10.244.0.0/16" 一致

# kubectl create -f kube-flannel.yml

7.查看所有pod详细信息(Master 需要配置)

# kubectl get pod --all-namespaces -o wide

# kubectl describe pods

8.添加Node(Node 需要配置)

从Master初始化输出中获取

# kubeadm join --token 288f34.481c8faa5636966f 10.110.16.10:6443 --discovery-token-ca-cert-hash sha256:8036fac3b76e1e0dd189edaa8f7d36f1b51429dd0c0cf7ea0d78e7972d611002

Tip:这里作为可选, 因为笔者将Master加入Node调度中, 让其同时充当Master和Node的角色, 如果资源有限只有一台测试机的小伙伴这步可以略过.

9.将Master加入Node schedual调度(Master 需要配置)

Tip: 默认Master不会加入Node调度, 这里使用如下命令开启这个限制.

# kubectl taint nodes kube-master node-role.kubernetes.io/master-

如果我们有多个node需要手动去做调度, 从而不让我们的pod进入该node调度列表, 可以使用如下命令:

禁用该node调度

# kubectl cordon kube-node-1

查看是否禁用

# kubectl get node kube-node-1

NAME          STATUS                     ROLES     AGE       VERSION
kube-node-1   Ready,SchedulingDisabled   node    11d       v1.9.0

解禁该node调度

# kubectl uncordon kube-node-1

查看是否解禁

# kubectl get nodes kube-node-1

NAME          STATUS  ROLES     AGE       VERSION
kube-node-1   Ready   node    11d       v1.9.0

10.测试K8s cluster(Master 需要配置)

我们利用k8s创建一个apache的网站实例, 镜像为httpd, 并设置2个副本. 

# kubectl run httpd-app --image=httpd --replicas=2

# kubectl get deployment

NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
httpd-app   2         2         2            2           2h

# kubectl get pods -o wide

NAME                         READY     STATUS    RESTARTS   AGE       IP            NODE
httpd-app-5fbccd7c6c-27nzv   1/1       Running   0          2h        10.244.0.13   kube-master
httpd-app-5fbccd7c6c-n9qs2   1/1       Running   0          2h        10.244.0.12   kube-master

11.测试网站实例(Master 需要配置)

# curl 10.244.0.13

<html><body><h1>It works!</h1></body></html>

# curl 10.244.0.12

<html><body><h1>It works!</h1></body></html>

12.安装kubernetes-dashboard(Master 需要配置)

Tip:默认kubernetes是没有图形管理界面, 我们这里通过添加一个dashboard容器去添加一个可管理的GUI界面

# cd /root/k8s_images

添加dashboard容器

# kubectl create -f kubernetes-dashboard.yaml

创建认证文件

# vi /etc/kubernetes/pki/basic_auth_file

#user,password,userid
admin,admin,2

配置dashboard basic认证

# vi /etc/kubernetes/manifests/kube-apiserver.yaml

添加一行:

- --basic_auth_file=/etc/kubernetes/pki/basic_auth_file

如下:

...
- --etcd-servers=http://127.0.0.1:2379
- --basic_auth_file=/etc/kubernetes/pki/basic_auth_file
image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.0
...

更新认证配置

# kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml (可不执行, 待验证)

# systemctl restart kubelet

将admin账号加入cluster-admin的role, 获取k8s最高系统权限

# kubectl create clusterrolebinding login-on-dashboard-with-cluster-admin --clusterrole=cluster-admin --user=admin

访问dashboard主页

https://10.110.16.10:32666

账号/密码: admin/admin

大功告成...

更多kubernetes组件介绍:

Kubernetes之Pod, Replicaset, Deployment, Label, Service

Kubernetes之Persistent Volume(持久化卷)

Kubernetes之Secrets与Config Maps

Kubernetes之Helm包管理