欢迎访问www.showerlee.com, 您的支持就是我前进的动力.

[Saltstack] CentOS下批量部署Nginx

showerlee 2015-09-25 13:27 DevOps, Saltstack 阅读 (2,805) 抢沙发

上一篇博文我介绍了如何快速搭建Salt环境, 以及利用相关pillar, state, grains模块进行基本的客户端部署操作.
本篇文档我们会详细介绍如何利用Salt来批量部署安装Nginx, 并自动化配置Nginx的相关属性.

Salt环境部署详见: http://www.showerlee.com/archives/1472

安装环境:

System: Centos 6.3

Salt master: salt-master.example.com 

Salt minion: salt-client01.example.com

Salt minion: salt-client02.example.com

一.  主控端配置

1. 配置master基本参数

# vi /etc/salt/master
添加:

nodegroups:
   webgroup1: 'salt-client01.example.com'
   webgroup2: 'salt-client02.example.com'

file_roots:
  base:
    - /srv/salt

pillar_roots:
  base:
    - /srv/pillar

Tip: 这里的nodegroups里的分组会应用到随后的pillar脚本下, 用来区分不同的salt-minion使用相应的pillar参数, 从而定制不同的minion使用不同的配置方案.

重启服务

# /etc/init.d/salt-master restart

2. 动态配置客户端系统连接数

使用python脚本编写grains_module, 实现动态配置被控主机连接数(CLI可用"limit -n"查询该值), 以便随后的Nginx配置文件中的worker_rlimit_nofile, worker_connections可动态调用脚本中max_open_file的参数值.

Tip: "ulimit -n" 是用来查询当前linux系统的最大打开文件数, 默认为1024

也可用"ulimit -a"来查看其他相关参数

# ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7819
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

暂时修改当前session的参数值

# ulimit -n 2048

永久修改需修改该参数值

# vi /etc/security/limits.conf

添加:

root soft  nofile 2048
root hard  nofile 2048

# vi /etc/pam.d/common-session

添加:

session required pam_limits.so

重启生效.

脚本具体配置如下:

# mkdir -p /srv/salt/_grains

# vi /srv/salt/_grains/nginx_config.py

#!/usr/bin/env python

import os,sys,commands  

# return Nginx config grains value  
def NginxGrains():  
    grains = {}  
    max_open_file=65536   
    try:  
        getulimit=commands.getstatusoutput('source /etc/profile && ulimit -n')  
    except Exception,e:  
        pass  
    if getulimit[0]==0:  
        max_open_file=int(getulimit[1])  
    grains['max_open_file'] = max_open_file  
    return grains

Tip: 该脚本会同步到远程后执行, 脚本实际就是获取并返回当前主机的最大打开数值, 最终返回值会赋予字典 grains['max_open_file']

同步grains模块:

# salt '*' saltutil.sync_all

刷新模块(让minion编译模块)

# salt '*' sys.reload_modules

验证max_open_file key的value

# salt '*' grains.item max_open_file 

Tip: 这里笔者测试更改客户端最大文件打开值时发现了一个问题, 无论客户端如何更改这个值, 在验证key value时终会显示最早的系统初始值1024, 翻阅了大量文档, 发现minion端会将所有服务端的推送保存在(/var/cache/salt/minion), 这里删除这个缓存目录并重启salt-minion, 让其生成新的缓存目录, 从新同步grains模块, 新的vaule就会生效.

# salt '*' cmd.run 'rm -rf /var/cache/salt/minion && /etc/init.d/salt-minion restart'

3. 配置pillar

1). 定义入口sls

# vi /srv/pillar/top.sls

base:
  webgroup1:
   - match: nodegroup
   - nginx.webserver1
  webgroup2:
   - match: nodegroup
   - nginx.webserver2

Tip: 这里定义webgroup1的所有minions会使用pillar/nginx/webserver1.sls脚本, webgroup2所有minions使用pillar/nginx/webserver2.sls脚本, match: nodegroup 定义webgroup1, webgroup2匹配nodegroup组.


2). 定义webserver1, webserver2 sls

# mkdir /srv/pillar/nginx

# vi /srv/pillar/nginx/webserver1.sls

nginx:
    hostname: webserver1
    name: nginx
    root: /www
    source: salt://nginx/nginx.conf
    file: /etc/nginx/nginx.conf
    user: root
    group: root
    mode: 644
    template: jinja
    status: installed

vi /srv/pillar/nginx/webserver2.sls

nginx:
    hostname: webserver2
    name: nginx
    root: /data
    source: salt://nginx/nginx.conf
    file: /etc/nginx/nginx.conf
    user: root
    group: root
    mode: 666
    template: jinja
    status: installed

3). 查看配置结果

# salt '*' pillar.data nginx

salt-client02.example.com:
    ----------
    nginx:
        ----------
        file:
            /etc/nginx/nginx.conf
        group:
            root
        hostname:
            webserver2
        mode:
            666
        name:
            nginx
        root:
            /data
        source:
            salt://nginx/nginx.conf
        status:
            installed
        template:
            jinja
        user:
            root
salt-client01.example.com:
    ----------
    nginx:
        ----------
        file:
            /etc/nginx/nginx.conf
        group:
            root
        hostname:
            webserver1
        mode:
            644
        name:
            nginx
        root:
            /www
        source:
            salt://nginx/nginx.conf
        status:
            installed
        template:
            jinja
        user:
            root

Tip: 可以通过字典 pillar['nginx']['root'] 在配置文件中调用


4. 配置state

1). 定义入口sls

# vi /srv/salt/top.sls

base:
 '*':
   - env
   - nginx.deploy

2). 定义env, nginx服务状态管理配置sls

通用系统环境配置

# vi /srv/salt/env.sls

policycoreutils:
    pkg.installed
 
policycoreutils-python:
    pkg.installed

{% if grains['selinux']['enabled'] %}
setselinux:
  selinux.mode:
    - name: 'permissive'
    - require:
      - pkg: policycoreutils
      - pkg: policycoreutils-python

/etc/sysconfig/selinux:
  file.replace:
    - pattern: 'SELINUX=\w+'
    - repl: 'SELINUX=disabled'
{% endif %}

iptables:
  pkg:
    - installed
  iptables.flush:
    - table: 'filter'
    - save: True
  cmd.run:
    - name: 'service iptables save'
    - require:
      - pkg: iptables

Tip: 使用setenforce需要安装2个python依赖包, 通过判断系统默认的SELINUX是否打开, 从而确定是否关闭SELINUX, 如果不作此判断, 系统SELINUX若关闭, 部署时会报错.


NGINX配置

# mkdir /srv/salt/nginx

# vi /srv/salt/nginx/deploy.sls

{% if 'nginx' in pillar %}
  {% set item = pillar['nginx'] %}
  {{ item['name'] }}:
    pkg: 
     - {{ item['status'] }}
    file.managed:  
     - source: {{ item['source'] }}  
     - name: {{ item['file'] }}  
     - user: {{ item['user'] }}  
     - group: {{ item['group'] }}  
     - mode: {{ item['mode'] }}  
     - template: {{ item['template'] }}  
  
    service.running:  
     - enable: True  
     - reload: True  
     - watch:  
       - file: {{ item['file'] }}  
       - pkg: {{ item['name'] }}

  {{ item['root'] }}:
    file.directory:
      - user: {{ item['name'] }}
      - group: {{ item['name'] }}
      - mode: 755
      - makedirs: True
      - recurse:
        - user
        - group
        - mode

{% endif %}

Tip: "source: {{ item['source'] }}"等价于"source: {{ pillar['nginx']['source'] }}"为配置模板文件位置 

      "-enable: True" 等价于 "chkconfig nginx on"

      "-reload True" 等价于 "service nginx reload", 若不加则默认执行"service nginx restart"

      "-wotch -file:" 检查 /etc/nginx/nginx.conf是否发生变化 

      "-watch -pkg" 确保nginx安装成功.

      "{{ item['root'] }}: file.directory:" 等价于mkdir /www


3). 定义Nginx配置文件(引用jinja模板)

# vi /srv/salt/nginx/nginx.conf

# For more information on configuration, see: 
{% set item = pillar['nginx'] %}

user              nginx;  
worker_processes  {{ grains['num_cpus'] }};  
{% if grains['num_cpus'] == 2 %}  
worker_cpu_affinity 01 10;  
{% elif grains['num_cpus'] == 4 %}  
worker_cpu_affinity 1000 0100 0010 0001;  
{% elif grains['num_cpus'] >= 8 %}  
worker_cpu_affinity 00000001 00000010 00000100 00001000 00010000 00100000 01000000 10000000;  
{% else %}  
worker_cpu_affinity 1000 0100 0010 0001;  
{% endif %}  
worker_rlimit_nofile {{ grains['max_open_file'] }};  
  
error_log  /var/log/nginx/error.log;  
#error_log  /var/log/nginx/error.log  notice;  
#error_log  /var/log/nginx/error.log  info;  
  
pid        /var/run/nginx.pid;  
  
events {  
    worker_connections  {{ grains['max_open_file'] }};  
}  
  
  
http {  
    include       /etc/nginx/mime.types;  
    default_type  application/octet-stream;  
  
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '  
                      '$status $body_bytes_sent "$http_referer" '  
                      '"$http_user_agent" "$http_x_forwarded_for"';  
  
    access_log  /var/log/nginx/access.log  main;  
  
    sendfile        on;  
    #tcp_nopush     on;  
  
    #keepalive_timeout  0;  
    keepalive_timeout  65;  
  
    #gzip  on;  
      
    # Load config files from the /etc/nginx/conf.d directory  
    # The default server is in conf.d/default.conf  
    #include /etc/nginx/conf.d/*.conf;  
    server {  
        listen       80 default_server;  
        server_name  _;  
  
        #charset koi8-r;  
  
        #access_log  logs/host.access.log  main;  
  
        location / {  
            root   {{ item['root'] }};  
            index  index.html index.htm;  
        }  
  
        error_page  404              /404.html;  
        location = /404.html {  
            root   /usr/share/nginx/html;  
        }  
  
        # redirect server error pages to the static page /50x.html  
        #  
        error_page   500 502 503 504  /50x.html;  
        location = /50x.html {  
            root   /usr/share/nginx/html;  
        }  
  
    }  
  
} 

Tip:

  • worker_processes参数采用grains['num_cpus'] 上报值(与设备CPU核数一致);
  • worker_cpu_affinity分配多核CPU根据当前设备核数进行匹配,分别为2\4\8\其它核;
  • worker_rlimit_nofile参数与grains['max_open_file'] 获取的系统ulimit -n一致;
  • worker_connections 参数理论上为grains['max_open_file'];
  • root参数为定制的pillar['nginx']['root']值。

4). 执行最终state配置, 将master的所有nginx配置部署到客户端

# salt '*' state.highstate

Tip: 可使用salt '*' -l debug state.highstate 查看部署过程中是否有错误

salt-client02.example.com:
----------
          ID: policycoreutils
    Function: pkg.installed
      Result: True
     Comment: Package policycoreutils is already installed.
     Started: 14:35:20.183439
    Duration: 1165.132 ms
     Changes:   
----------
          ID: policycoreutils-python
    Function: pkg.installed
      Result: True
     Comment: Package policycoreutils-python is already installed.
     Started: 14:35:21.348902
    Duration: 0.875 ms
     Changes:   
----------
          ID: iptables
    Function: pkg.installed
      Result: True
     Comment: Package iptables is already installed.
     Started: 14:35:21.349904
    Duration: 0.47 ms
     Changes:   
----------
          ID: iptables
    Function: iptables.flush
      Result: True
     Comment: Flush iptables rules in filter table  chain ipv4 family
     Started: 14:35:21.351883
    Duration: 16.006 ms
     Changes:   
              ----------
              locale:
                  iptables
----------
          ID: iptables
    Function: cmd.run
        Name: service iptables save
      Result: True
     Comment: Command "service iptables save" run
     Started: 14:35:21.371349
    Duration: 50.934 ms
     Changes:   
              ----------
              pid:
                  3352
              retcode:
                  0
              stderr:
              stdout:
                  iptables: Saving firewall rules to /etc/sysconfig/iptables: ?[60G[?[0;32m  OK  ?[0;39m]
----------
          ID: nginx
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: nginx
     Started: 14:35:21.422715
    Duration: 19128.275 ms
     Changes:   
              ----------
              nginx:
                  ----------
                  new:
                      1.0.15-12.el6
                  old:
----------
          ID: nginx
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: True
     Comment: File /etc/nginx/nginx.conf updated
     Started: 14:35:40.560042
    Duration: 40.855 ms
     Changes:   
              ----------
              diff:
                  ---  
                  +++  
                  @@ -1,42 +1,70 @@
                  -# For more information on configuration, see:
                  -#   * Official English Documentation: http://nginx.org/en/docs/
                  -#   * Official Russian Documentation: http://nginx.org/ru/docs/
                  -
                  -user              nginx;
                  -worker_processes  1;
                  -
                  -error_log  /var/log/nginx/error.log;
                  -#error_log  /var/log/nginx/error.log  notice;
                  -#error_log  /var/log/nginx/error.log  info;
                  -
                  -pid        /var/run/nginx.pid;
                  +# For more information on configuration, see: 
                   
                   
                  -events {
                  -    worker_connections  1024;
                  -}
                  -
                  -
                  -http {
                  -    include       /etc/nginx/mime.types;
                  -    default_type  application/octet-stream;
                  -
                  -    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  -                      '$status $body_bytes_sent "$http_referer" '
                  -                      '"$http_user_agent" "$http_x_forwarded_for"';
                  -
                  -    access_log  /var/log/nginx/access.log  main;
                  -
                  -    sendfile        on;
                  -    #tcp_nopush     on;
                  -
                  -    #keepalive_timeout  0;
                  -    keepalive_timeout  65;
                  -
                  -    #gzip  on;
                  -    
                  -    # Load config files from the /etc/nginx/conf.d directory
                  -    # The default server is in conf.d/default.conf
                  -    include /etc/nginx/conf.d/*.conf;
                  -
                  -}
                  +user              nginx;  
                  +worker_processes  1;  
                  +  
                  +worker_cpu_affinity 1000 0100 0010 0001;  
                  +  
                  +worker_rlimit_nofile 1024;  
                  +  
                  +error_log  /var/log/nginx/error.log;  
                  +#error_log  /var/log/nginx/error.log  notice;  
                  +#error_log  /var/log/nginx/error.log  info;  
                  +  
                  +pid        /var/run/nginx.pid;  
                  +  
                  +events {  
                  +    worker_connections  1024;  
                  +}  
                  +  
                  +  
                  +http {  
                  +    include       /etc/nginx/mime.types;  
                  +    default_type  application/octet-stream;  
                  +  
                  +    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '  
                  +                      '$status $body_bytes_sent "$http_referer" '  
                  +                      '"$http_user_agent" "$http_x_forwarded_for"';  
                  +  
                  +    access_log  /var/log/nginx/access.log  main;  
                  +  
                  +    sendfile        on;  
                  +    #tcp_nopush     on;  
                  +  
                  +    #keepalive_timeout  0;  
                  +    keepalive_timeout  65;  
                  +  
                  +    #gzip  on;  
                  +      
                  +    # Load config files from the /etc/nginx/conf.d directory  
                  +    # The default server is in conf.d/default.conf  
                  +    #include /etc/nginx/conf.d/*.conf;  
                  +    server {  
                  +        listen       80 default_server;  
                  +        server_name  _;  
                  +  
                  +        #charset koi8-r;  
                  +  
                  +        #access_log  logs/host.access.log  main;  
                  +  
                  +        location / {  
                  +            root   /data;  
                  +            index  index.html index.htm;  
                  +        }  
                  +  
                  +        error_page  404              /404.html;  
                  +        location = /404.html {  
                  +            root   /usr/share/nginx/html;  
                  +        }  
                  +  
                  +        # redirect server error pages to the static page /50x.html  
                  +        #  
                  +        error_page   500 502 503 504  /50x.html;  
                  +        location = /50x.html {  
                  +            root   /usr/share/nginx/html;  
                  +        }  
                  +  
                  +    }  
                  +  
                  +} 
              mode:
                  0666
----------
          ID: nginx
    Function: service.running
      Result: True
     Comment: Service nginx has been enabled, and is running
     Started: 14:35:40.608381
    Duration: 320.864 ms
     Changes:   
              ----------
              nginx:
                  True
----------
          ID: /data
    Function: file.directory
      Result: True
     Comment: Directory /data is in the correct state
     Started: 14:35:40.929607
    Duration: 2.265 ms
     Changes:   

Summary
------------
Succeeded: 9 (changed=5)
Failed:    0
------------
Total states run:     9
salt-client01.example.com:
----------
          ID: policycoreutils
    Function: pkg.installed
      Result: True
     Comment: Package policycoreutils is already installed.
     Started: 18:14:48.158376
    Duration: 1211.452 ms
     Changes:   
----------
          ID: policycoreutils-python
    Function: pkg.installed
      Result: True
     Comment: Package policycoreutils-python is already installed.
     Started: 18:14:49.370171
    Duration: 0.787 ms
     Changes:   
----------
          ID: setselinux
    Function: selinux.mode
        Name: permissive
      Result: True
     Comment: SELinux is already in Permissive mode
     Started: 18:14:49.372833
    Duration: 0.652 ms
     Changes:   
----------
          ID: /etc/sysconfig/selinux
    Function: file.replace
      Result: True
     Comment: No changes needed to be made
     Started: 18:14:49.377284
    Duration: 2.507 ms
     Changes:   
----------
          ID: iptables
    Function: pkg.installed
      Result: True
     Comment: Package iptables is already installed.
     Started: 18:14:49.379941
    Duration: 0.597 ms
     Changes:   
----------
          ID: iptables
    Function: iptables.flush
      Result: True
     Comment: Flush iptables rules in filter table  chain ipv4 family
     Started: 18:14:49.381900
    Duration: 21.393 ms
     Changes:   
              ----------
              locale:
                  iptables
----------
          ID: iptables
    Function: cmd.run
        Name: service iptables save
      Result: True
     Comment: Command "service iptables save" run
     Started: 18:14:49.406906
    Duration: 342.078 ms
     Changes:   
              ----------
              pid:
                  3124
              retcode:
                  0
              stderr:
              stdout:
                  iptables: Saving firewall rules to /etc/sysconfig/iptables: ?[60G[?[0;32m  OK  ?[0;39m]
----------
          ID: nginx
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: nginx
     Started: 18:14:49.749415
    Duration: 23737.719 ms
     Changes:   
              ----------
              nginx:
                  ----------
                  new:
                      1.0.15-12.el6
                  old:
----------
          ID: nginx
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: True
     Comment: File /etc/nginx/nginx.conf updated
     Started: 18:15:13.490651
    Duration: 52.469 ms
     Changes:   
              ----------
              diff:
                  ---  
                  +++  
                  @@ -1,42 +1,70 @@
                  -# For more information on configuration, see:
                  -#   * Official English Documentation: http://nginx.org/en/docs/
                  -#   * Official Russian Documentation: http://nginx.org/ru/docs/
                  -
                  -user              nginx;
                  -worker_processes  1;
                  -
                  -error_log  /var/log/nginx/error.log;
                  -#error_log  /var/log/nginx/error.log  notice;
                  -#error_log  /var/log/nginx/error.log  info;
                  -
                  -pid        /var/run/nginx.pid;
                  +# For more information on configuration, see: 
                   
                   
                  -events {
                  -    worker_connections  1024;
                  -}
                  -
                  -
                  -http {
                  -    include       /etc/nginx/mime.types;
                  -    default_type  application/octet-stream;
                  -
                  -    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  -                      '$status $body_bytes_sent "$http_referer" '
                  -                      '"$http_user_agent" "$http_x_forwarded_for"';
                  -
                  -    access_log  /var/log/nginx/access.log  main;
                  -
                  -    sendfile        on;
                  -    #tcp_nopush     on;
                  -
                  -    #keepalive_timeout  0;
                  -    keepalive_timeout  65;
                  -
                  -    #gzip  on;
                  -    
                  -    # Load config files from the /etc/nginx/conf.d directory
                  -    # The default server is in conf.d/default.conf
                  -    include /etc/nginx/conf.d/*.conf;
                  -
                  -}
                  +user              nginx;  
                  +worker_processes  1;  
                  +  
                  +worker_cpu_affinity 1000 0100 0010 0001;  
                  +  
                  +worker_rlimit_nofile 1024;  
                  +  
                  +error_log  /var/log/nginx/error.log;  
                  +#error_log  /var/log/nginx/error.log  notice;  
                  +#error_log  /var/log/nginx/error.log  info;  
                  +  
                  +pid        /var/run/nginx.pid;  
                  +  
                  +events {  
                  +    worker_connections  1024;  
                  +}  
                  +  
                  +  
                  +http {  
                  +    include       /etc/nginx/mime.types;  
                  +    default_type  application/octet-stream;  
                  +  
                  +    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '  
                  +                      '$status $body_bytes_sent "$http_referer" '  
                  +                      '"$http_user_agent" "$http_x_forwarded_for"';  
                  +  
                  +    access_log  /var/log/nginx/access.log  main;  
                  +  
                  +    sendfile        on;  
                  +    #tcp_nopush     on;  
                  +  
                  +    #keepalive_timeout  0;  
                  +    keepalive_timeout  65;  
                  +  
                  +    #gzip  on;  
                  +      
                  +    # Load config files from the /etc/nginx/conf.d directory  
                  +    # The default server is in conf.d/default.conf  
                  +    #include /etc/nginx/conf.d/*.conf;  
                  +    server {  
                  +        listen       80 default_server;  
                  +        server_name  _;  
                  +  
                  +        #charset koi8-r;  
                  +  
                  +        #access_log  logs/host.access.log  main;  
                  +  
                  +        location / {  
                  +            root   /www;  
                  +            index  index.html index.htm;  
                  +        }  
                  +  
                  +        error_page  404              /404.html;  
                  +        location = /404.html {  
                  +            root   /usr/share/nginx/html;  
                  +        }  
                  +  
                  +        # redirect server error pages to the static page /50x.html  
                  +        #  
                  +        error_page   500 502 503 504  /50x.html;  
                  +        location = /50x.html {  
                  +            root   /usr/share/nginx/html;  
                  +        }  
                  +  
                  +    }  
                  +  
                  +} 
----------
          ID: nginx
    Function: service.running
      Result: True
     Comment: Service nginx has been enabled, and is running
     Started: 18:15:13.550329
    Duration: 424.431 ms
     Changes:   
              ----------
              nginx:
                  True
----------
          ID: /www
    Function: file.directory
      Result: True
     Comment: Directory /www is in the correct state
     Started: 18:15:13.975177
    Duration: 2.586 ms
     Changes:   

Summary
-------------
Succeeded: 11 (changed=5)
Failed:     0
-------------
Total states run:     11

Tip: 笔者为了验证效果, 事先手动关闭了client02的SELINUX配置, 这里可以看到client02跳过SELINUX配置项, client01则正常执行.

这样我们就可以从返回信息查看Nginx的配置文件参数是否正确调用, 以及最终是否部署成功.

最终/srv目录下的树状结构

# cd /srv && tree .

.
|-- pillar
|   |-- nginx
|   |   |-- webserver1.sls
|   |   `-- webserver2.sls
|   `-- top.sls
`-- salt
    |-- env.sls
    |-- _grains
    |   `-- nginx_config.py
    |-- nginx
    |   |-- deploy.sls
    |   `-- nginx.conf
    |-- nginx.sls.bak
    `-- top.sls

5 directories, 9 files

至此, 一个模拟生产环境的WEB服务配置集中管理部署平台已经搭建并测试完成, 大家可以拓展思路, 利用该平台扩展到其他应用业务当中.

正文部分到此结束
版权声明:除非注明,本文由(showerlee)原创,转载请保留文章出处!
本文链接:http://www.showerlee.com/archives/1538

继续浏览:Saltstack

还没有评论,快来抢沙发!

发表评论

icon_wink.gif icon_neutral.gif icon_mad.gif icon_twisted.gif icon_smile.gif icon_eek.gif icon_sad.gif icon_rolleyes.gif icon_razz.gif icon_redface.gif icon_surprised.gif icon_mrgreen.gif icon_lol.gif icon_idea.gif icon_biggrin.gif icon_evil.gif icon_cry.gif icon_cool.gif icon_arrow.gif icon_confused.gif icon_question.gif icon_exclaim.gif