欢迎访问www.showerlee.com, 您的支持就是我前进的动力.

Kubernetes之Ingress+Traefik

showerlee 2018-09-08 17:32 DevTools, Docker, Kubernetes 阅读 (18,113) 抢沙发

今天是一个值得庆祝的日子, 终于把研究了半年未果的k8s ingress反向代理通过traefik这个工具给试验出来了, 这里感谢我的甜宝宝对我的支持, 因为有你, 我才有动力继续前行.

traefik 是一款开源的反向代理与负载均衡工具。它最大的优点是能够与常见的微服务系统直接整合,可以实现自动化动态配置。目前支持 Docker, Swarm, Mesos/Marathon, Mesos, Kubernetes, Consul, Etcd, Zookeeper, BoltDB, Rest API 等等后端模型。

由于微服务架构以及 Docker 技术和 kubernetes 编排工具最近几年才开始逐渐流行,所以一开始的反向代理服务器比如 nginx、apache 并未提供其支持,所以才会出现 Ingress Controller 来做 kubernetes 和前端负载均衡器如 nginx 之间做衔接;即 Ingress Controller 的存在就是为了能跟 kubernetes 交互,又能写 nginx 配置,还能 reload 它,这是一种折中方案;而 traefik 天生就是提供了对 kubernetes 的支持,也就是说 traefik 本身就能跟 kubernetes API 交互,感知后端变化,因此可以得知:  traefik 完全已经取代了 Ingress Controller 作为与k8s交互的代理,在此给大家这里整体架构如下:

traefik

为什么选择 traefik?

Golang 编写,单文件部署,与系统无关,同时也提供小尺寸 Docker 镜像。
支持 Docker/Etcd 后端,天然连接我们的微服务集群。
内置 Web UI,管理相对方便。
自动配置 ACME(Let’s Encrypt) 证书功能。
性能尚可,我们也没有到压榨 LB 性能的阶段,易用性更重要。


除了这些以外,traefik 还有以下特点:
Restful API 支持。
支持后端健康状态检查,根据状态自动配置。
支持动态加载配置文件和 graceful 重启。
支持 WebSocket 和 HTTP/2。

目前网上很多资料虽然能够成功在k8s下搭建ingress+traefik实现cluster内网的代理功能, 但如何将这个反向代理IP暴露在我们外网, 也就是不依赖于AWS等公有云, 在我们on-prem(本地k8s网络环境)通过定义hostnetwork去实现将我们的node ip直接作为我们在cluster下创建的反向代理ip,  将是我们本次需要给大家介绍的内容.

我们接着上一篇Kubernates1.9+Docker17离线安装部署, 给大家展开如何实现k8s通过使用Ingress+Traefik实现集群反向代理功能.

1.获取Traefik支持K8s仓库脚本

# cd /kube/

# git clone https://github.com/containous/traefik.git

# cd traefik/examples/k8s/

# ll

total 36
-rw-r--r-- 1 root root  140 Aug 26 04:23 cheese-default-ingress.yaml
-rw-r--r-- 1 root root 1805 Aug 26 04:23 cheese-deployments.yaml
-rw-r--r-- 1 root root  519 Aug 26 04:23 cheese-ingress.yaml
-rw-r--r-- 1 root root  509 Aug 26 04:23 cheese-services.yaml
-rw-r--r-- 1 root root  504 Aug 26 04:23 cheeses-ingress.yaml
-rw-r--r-- 1 root root 1144 Aug 26 06:40 traefik-deployment.yaml
-rw-r--r-- 1 root root 1206 Aug 26 04:23 traefik-ds.yaml
-rw-r--r-- 1 root root  694 Aug 26 04:23 traefik-rbac.yaml
-rw-r--r-- 1 root root  466 Aug 26 04:40 ui.yaml

这个目录下就是示例 Traefik 启动所需要的 yaml 文件,Traefik 提供了适配各个类型服务编排的部署方式,kubernetes 启动方式支持 Deployment 和 DaemonSet,二选一都可以, 这里笔者选择Deployment


2. Traefik脚本配置

还记得我之前提到的hostnetwork吗? 

这里我们需要在部署之前将我们的traefik-deployment.yaml下添加一行 hostNetwork: true

这样子就能保证我们的Traefik反向代理的会将自己Endpoints IP配置成我们的NodeIP, 供我们与K8S同一网段的设备访问.

我们可以这么配置:

# vi traefik-deployment.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      hostNetwork: true
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
        - name: admin
          containerPort: 8080
        args:
        - --api
        - --kubernetes
        - --logLevel=INFO
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

# vi traefik-rbac.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

这样子我们就完成了K8S脚本的初始修改, 接下来我们就可以直接开始我们的Traefik安装

3. Traefik安装

# kubectl apply -f traefik-rbac.yaml

clusterrole "traefik-ingress-controller" created
clusterrolebinding "traefik-ingress-controller" created

# kubectl apply -f traefik-deployment.yaml

serviceaccount "traefik-ingress-controller" created
deployment "traefik-ingress-controller" created
service "traefik-ingress-service" created

# kubectl apply -f ui.yaml

service "traefik-web-ui" created
ingress "traefik-web-ui" created

# kubectl get pods --all-namespaces -o wide

NAME                                          READY     STATUS    RESTARTS   AGE       IP             NODE
…
traefik-ingress-controller-795ffb7d78-ppw94   1/1       Running   0          5m        10.110.16.15   kube-node1

可以看到我们的traefik-ingress pods成功的打开了我们的hostnetwork, 这里显示的是我们的node ip而不是pod内网 ip, 也就是我们可以直接通过访问node ip 10.110.16.15来连接traefik反向代理, 利用traefik来分配我们的K8S的容器连接.

# kubectl get service --all-namespaces

NAMESPACE      NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
default        frontend                    ClusterIP   10.110.87.1      <none>        80/TCP                        4h
default        kubernetes                  ClusterIP   10.96.0.1        <none>        443/TCP                       1d
default        my-nginx                    ClusterIP   10.102.200.83    <none>        80/TCP                        4h
kube-system    kube-dns                    ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP                 1d
kube-system    kubernetes-dashboard        NodePort    10.108.187.244   <none>        443:32666/TCP                 1d
kube-system    tiller-deploy               ClusterIP   10.111.251.3     <none>        44134/TCP                     1d
kube-system    traefik-ingress-service     NodePort    10.100.70.138    <none>        80:31087/TCP,8080:31017/TCP   13m
kube-system    traefik-web-ui              ClusterIP   10.96.54.167     <none>        80/TCP                        13m
newegg-nginx   newegg-nginx-newegg-nginx   NodePort    10.97.200.124    <none>        80:32239/TCP                  8h

我们可以通过两种方式去访问我们的traefik后台管理员界面.

1).通过Node Port方式

我们可以在与K8S node同网段的机器去访问其Node暴露的端口31017

2).直接通过ui.yaml配置traefik+ingress反向代理, 这样我们就可以通过host域名直接访问后台

# vi ui.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  rules:
  - host: traefik-ui.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

在本地MacOS系统添加traefik-ui.k8s解析到10.110.16.15

# echo "10.110.16.15 traefik-ui.k8s" >> /etc/hosts

我们可以同样利用traefik反向代理, 通过http域名的方式访问admin后台.

4.部署两个服务nginx1-7和nginx1-8,配置Traefik去负载这两个服务:

配置一个deployment类型的1.7版本nginx实例, 并利用service开启其内部80端口监听.

# vi nginx1-7.yaml

apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-7
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx1-7-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-7
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

配置一个deployment类型的1.8版本nginx实例, 并利用service开启其内部80端口监听.

vi nginx1-8.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-nginx
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-8
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx1-8-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-8
    spec:
      containers:
      - name: nginx
        image: nginx:1.8
        ports:
        - containerPort: 80

通过利用Ingress绑定1.7与1.8nginx实例下的serviceName, 给其定义不同的域名, 实现traefik反向代理.

# vi traefik.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: default
spec:
  rules:
  - host: traefik.nginx.io
    http:
      paths:
      - path: /
        backend:
          serviceName: my-nginx
          servicePort: 80
  - host: traefik.frontend.io
    http:
      paths:
      - path: /
        backend:
          serviceName: frontend
          servicePort: 80

我们这里通过配置MacOS本地Host DNS, 将所有的DNS记录指向我们暴露在外网的traefik IP, 从而让traefik作为代理接管所有访问K8S内部实例的连接, 实际的原理和我们之前用过的apache, nginx基本一致.

# echo "10.110.16.15 traefik.nginx.io traefik.frontend.io" >> /etc/hosts

这样子我们就可以直接在MacOS浏览器下访问traefik.nginx.io与traefik.frontend.io从而实现traefik反向代理.

5.配置HTTPS访问traefik管理员后台

1).创建并配置https证书并保存到k8s secret下.

# mkdir -p /opt/k8s/ssl

# cd /opt/k8s/ssl

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=traefik-ui.k8s"

# kubectl create secret generic traefik-cert --from-file=tls.crt --from-file=tls.key -n kube-system

2).配置traefik反向代理并保存到k8s configmap下.

# mkdir -p /opt/k8s/conf/

# cd /opt/k8s/ssl/

# vi traefik-ui.toml

defaultEntryPoints = ["http","https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    regex = "^http://traefik-ui.k8s/(.*)"
    replacement = "https://traefik-ui.k8s/$1"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/opt/k8s/ssl/tls.crt"
      keyFile = "/opt/k8s/ssl/tls.key"

# kubectl create configmap traefik-ui-conf --from-file=traefik-ui.toml -n kube-system

这样子我们就成功配置traefik反向代理, 仅使http://traefik-ui.k8s会重定向到https://traefik-ui.k8s, 实现https加密访问.

3).创建traefik deployment实例并支持https反向代理.

# vi traefik-deployment-ssl.yaml 

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: traefik-ingress-lb
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      restartPolicy: Always
      serviceAccountName: traefik-ingress-controller
      volumes:
      - name: ssl
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-ui-conf
      containers:
      - image: traefik
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: "/opt/k8s/ssl"
          name: "ssl"
        - mountPath: "/opt/k8s/conf"
          name: "config"
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
        - name: admin
          containerPort: 8080
          hostPort: 8080
        args:
        - --configFile=/opt/k8s/conf/traefik-ui.toml
        - --web
        - --web.address=:8080
        - --kubernetes
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: https
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

这里我们在原来的traefik deployment实例基础上将之前配置的secret与configmap导入该deployment下, 并mount ssl与conf目录用来让traefik获取我们在k8s node目录下保存的相应SSL证书与traefik反向代理配置.

最后我们通过service类型打开我们的deployment 443端口.

4).创建traefik管理员界面的ingress, 从而实现https域名访问.

# vi ui-ssl.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  tls:
    - secretName: traefik-cert
  rules:
  - host: traefik-ui.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

最后我们访问https://traefik-ui.k8s 验证结果.

这样子我们就成功的开启了traefik后台admin界面的https加密连接...

更多traefik+ingress相关内容: https://docs.traefik.io/user-guide/kubernetes/

正文部分到此结束
版权声明:除非注明,本文由(showerlee)原创,转载请保留文章出处!
本文链接:http://www.showerlee.com/archives/2701

继续浏览:

还没有评论,快来抢沙发!

发表评论

icon_wink.gif icon_neutral.gif icon_mad.gif icon_twisted.gif icon_smile.gif icon_eek.gif icon_sad.gif icon_rolleyes.gif icon_razz.gif icon_redface.gif icon_surprised.gif icon_mrgreen.gif icon_lol.gif icon_idea.gif icon_biggrin.gif icon_evil.gif icon_cry.gif icon_cool.gif icon_arrow.gif icon_confused.gif icon_question.gif icon_exclaim.gif