安装环境

Remote: CentOS 7.4 x64 (django.example.com)

Python: Python3.6.5

Django: Django 2.0.4

nWSGI:  uwsgi-2.0.15

Nginx:  nginx- 1.10.2-1.el6

一. 系统环境配置

1.关闭iptables和selinux

# su - root

# service iptables stop

# setenforce 0

# vi /etc/sysconfig/selinux

修改

SELINUX=disabled

2.添加本地host DNS

# vi /etc/hosts

127.0.0.1    django.example.com

二. Python配置

1.安装python3.6.5源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel libffi-devel -y

2.编译安装python3.6.5以及pip package manager

# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz --no-check-certificate

# tar xf Python-3.6.5.tar.xz

# cd Python-3.6.5

# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

3.安装virtualenv

# pip3.6 install --upgrade pip

# pip3.6 install virtualenv

三. Nginx配置

1. 安装nginx package

# yum install nginx -y

2.配置nginx with nWSGI

# vi /etc/nginx/conf.d/django.conf

server {
    listen  80;
    server_name  django.example.com;  

    charset utf-8;

    access_log  /var/log/nginx/django_access.log;
    error_log   /var/log/nginx/django_error.log;
    
    location = /favicon.ico { access_log off; log_not_found off; }

    location /static/ {
        root /usr/share/nginx/html/django.example.com;
    }

    client_max_body_size 20M;
    
    location / {
        include         uwsgi_params;
        uwsgi_pass      unix:/etc/uwsgi/uwsgi-django.sock;
        uwsgi_read_timeout 30s;
        uwsgi_send_timeout 30s;
    }

}

四. Django+uWSGI配置

1. uWSGI配置

# mkdir -p /etc/uwsgi

# vi /etc/uwsgi/uwsgi-django.ini

[uwsgi]
project = django.example.com
base = /data/www

chdir = %(base)/%(project)
home = %(base)/%(project)/.py3env
module = myproject.wsgi:application

pidfile = /tmp/uwsgi-master-django.pid

master = true
processes = 2
enable-threads = true

# use unix socket because it is more secure and faster than TCP socket
socket = /etc/uwsgi/uwsgi-django.sock
chmod-socket = 660
uid = nginx
gid = nginx

vacuum = true
die-on-term = true

logto = /var/log/nginx/uwsgi-django.log

2. 配置Django base folder

# cd /usr/share/nginx/html

# mkdir django.example.com

# cd django.example.com

# virtualenv -p /usr/local/bin/python3 .py3env

3. 开启virtualenv python3环境

# source .py3env/bin/activate

4. 在此环境安装Django相关模块

# pip install django uwsgi PyMySQL

5. 创建uWSGI启动脚本

# mkdir -p /etc/uwsgi/bin

# vi /etc/systemd/system/uwsgi-django.service

[Unit]
    Description=uWSGI instance to serve myproject

[Service]
    BASE=/data/www/django.example.com
    ENV=$BASE/.py3env
    ExecStartPre=-/usr/bin/bash -c 'chown -R nginx:nginx /etc/uwsgi'
    ExecStart=/usr/bin/bash -c 'source /usr/share/nginx/html/django.example.com/.py3env/bin/activate; uwsgi --ini /etc/uwsgi/uwsgi-django.ini'

[Install]
    WantedBy=multi-user.target

五. Django项目配置

1. 保证virtualenv python3环境开启

# cd /usr/share/nginx/html/django.example.com/

# source .py3env/bin/activate

2.创建一个Django项目

# django-admin startproject myproject .

3.添加static目录

# vi myproject/settings.py

末行添加:

STATIC_ROOT = os.path.join(BASE_DIR, "static/")

4.创建本地SQLlite文件

Tip:这里使用SQLlite代替其他数据库作为我们项目的DB

# ./manage.py makemigrations
# ./manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK

5.创建项目管理员账户

# ./manage.py createsuperuser

Username (leave blank to use 'root'): root
Email address: admin@admin.com
Password:
Password (again):
Superuser created successfully.

6.生成项目静态文件目录

# ./manage.py collectstatic

7.修改wsgi入口文件

# vi myproject/wsgi.py

import os
import sys
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
sys.path.append('/usr/share/nginx/html/django.example.com')

from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()

8.添加ALLOWED_HOSTS

# vi myproject/settings.py

Update:

ALLOWED_HOSTS = ['django.example.com']

9. 修改权限(可执行并保持与nginx启动user一致)

# chmod -R 755 /etc/uwsgi

# chown -R nginx:nginx /etc/uwsgi

# chmod -R 755 /usr/share/nginx/html/django.example.com

# chown -R nginx:nginx /usr/share/nginx/html/django.example.com

10.启动nginx+uwsgi

# systemctl restart nginx

# systemctl restart uwsgi-django

展示效果(保证Windows本地host文件能够解析django.example.com)

Finished...

Remote: CentOS 7.4 x64 (django.example.com)

Python: Python3.6.5

Apache: Apache 2.4.6

Mod_wsgi: 4.6.4

Django: Django 2.0.4

一. 系统环境配置

1.关闭iptables和selinux

# su - root

# systemctl stop firewalld

# setenforce 0

# vi /etc/sysconfig/selinux

修改

SELINUX=disabled

2.添加本地host DNS

# vi /etc/hosts

127.0.0.1    django.example.com

二. Python配置

1.安装python3.6.5源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel libffi-devel -y

2.编译安装python3.6.5以及pip package manager

# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz --no-check-certificate

# tar xf Python-3.6.5.tar.xz

# cd Python-3.6.5

# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

3.安装virtualenv

# pip3.6 install --upgrade pip

# pip3.6 install virtualenv

三. Django环境配置

1. 配置Django virtualenv

# mkdir -p /var/www/html/django

# cd /var/www/html/django

# virtualenv -p /usr/local/bin/python3.6 .py3env

2. 开启virtualenv python3环境

# source .py3env/bin/activate

3. 在此环境安装Django相关模块

# pip install django pymysql

四. Apache配置

1. 安装apache package

# yum install httpd httpd-devel -y

2.安装mod_wsgi for python3

Tip:这里其实是一个远古巨坑, 网上90%以上资料的会粗心的直接使用yum install mod_wsgi去安装apache mod_wsgi模块, 这样做其实最终mod模块会调用本地默认的python2的所有库文件, 无论你后面如何配置django入口文件, apache都不会使用我们配置的virutalenv下隔离的python3, 导致apache无法调用python3而报错. 这里小伙伴要注意哦.

# pip install mod_wsgi 

3.导出apache所需的mod_wsgi模块

# mod_wsgi-express install-module

LoadModule wsgi_module "/usr/lib64/httpd/modules/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"
WSGIPythonHome "/var/www/html/.py3env"

4.配置apache配置文件

# vi /etc/httpd/conf/httpd.conf

末行添加:

LoadModule wsgi_module "/usr/lib64/httpd/modules/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"

# vi /etc/httpd/conf.d/django.conf

WSGIPythonHome "/var/www/html/django/.py3env"

Listen 8080
<VirtualHost *:8080>

ServerName django.example.com

Alias /static /var/www/html/django/static
<Directory /var/www/html/django/static>
 Require all granted
</Directory>

<Directory /var/www/html/django/myproject>
  <Files wsgi.py>
    Require all granted
  </Files>
</Directory>

WSGIDaemonProcess myproject python-path=/var/www/html/django/.py3env/lib/python3.6/site-packages
WSGIScriptAlias / /var/www/html/django/myproject/wsgi.py

</VirtualHost>

5.重启apache并设置开机自启动

# systemctl restart httpd

# systemctl enable httpd

五. Django项目配置

1. 保证virtualenv python3环境开启

# source /var/www/html/django/.py3env/bin/activate

2.创建一个Django项目

# cd /var/www/html/django/

# django-admin startproject myproject .

3.添加static目录

# vi myproject/settings.py

末行添加:

STATIC_ROOT = os.path.join(BASE_DIR, "static/")

4.创建本地SQLlite文件

Tip:这里使用SQLlite代替其他数据库作为我们项目的DB

# ./manage.py makemigrations
# ./manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK

5.创建项目管理员账户

# ./manage.py createsuperuser

Username (leave blank to use 'root'): root
Email address: admin@admin.com
Password:
Password (again):
Superuser created successfully.

6.生成项目静态文件目录

# ./manage.py collectstatic

7.修改wsgi入口文件

# vi myproject/wsgi.py

import os
import sys
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
sys.path.append('/var/www/html/django')

from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()

8.添加ALLOWED_HOSTS

# vi myproject/settings.py

Update:

ALLOWED_HOSTS = ['django.example.com']

9.修改项目属主和权限

# chmod -R 755 /var/www/html

# chown -R apache:apache /var/www/html

查看最终目录下的生成的项目文件

# ls -l

-rwxr-xr-x 1 apache apache 38912 Apr 16 15:04 db.sqlite3
-rwxr-xr-x 1 apache apache   541 Apr 16 14:50 manage.py
drwxr-xr-x 3 apache apache  4096 Apr 16 15:21 myproject
drwxr-xr-x 3 apache apache  4096 Apr 16 15:05 static

最终浏览器访问django项目

Tip:保证windows本地添加django服务器的HOST域名

django测试页面

项目主页, 输入之前创建的管理员账号密码

项目后台

Finished...

1. 随机执行方法

# vi ~/multipro1.py

import multiprocessing

def spawn(num):
    print('Sprawned! {}.'.format(num))


if __name__ == "__main__":
    for i in range(50):
        p = multiprocessing.Process(target=spawn, args=(i,))

        p.start()

2. 按顺序执行方法

# vi ~/multipro2.py

import multiprocessing


def spawn(num):
    print('Sprawned! {}.'.format(num))


if __name__ == "__main__":
    for i in range(50):
        p = multiprocessing.Process(target=spawn, args=(i,))

        p.start()

        # Waiting to finish One by one
        p.join()

Tip: 这里if __name__ == "__main__":的作用:

1. 当直接执行该脚本时, 内建变量__name__的值被赋予"__main__", 所以按照if逻辑, python解释器可以继续执行接下来的代码.

2. 当其他脚本去调用(import)该脚本时, __name__的值被赋予当前脚本名"multipro1", 而不是"__main__", 所以按照if逻辑, 保证if下面的代码不会被其他脚本调用.

所以在我们日常编写脚本的时候, 可以推荐按照如下结构, 在自己使用正常的同时, 保证在其他python脚本在调用test.py中的test1(), test2(), test3()的同时, 不会去执行if条件下的方法.

# vi test.py

def test1():
    ....

def test2():
    ....

def test3():
    ....

if __name__ == "__main__":
    ....

3. 多线程处理返回值.

# vi multipro3.py

from multiprocessing import pool

def job(num):
	return num * 2

if __name__ == '__main__':
	p = Pool(processes=20)
	data = p.map(job, range(5))
	data2 = p.map(job, [5, 2])
	p.close()

4.多线程爬取随机网站URL

# vi ~/multipro4.py

from multiprocessing import Pool
import bs4 as bs
import random
import requests
import string

# Return Four digits domain URL
def random_starting_url():
    starting = ''.join(random.SystemRandom().choice(
        string.ascii_lowercase) for _ in range(4))
    url = ''.join(['http://', starting, '.com'])
    return url


# Correct URL if it is relative
def handle_local_links(url, link):
    if link.startswith('/'):
        return ''.join([url, link])
    else:
        return link


# Get URL from "a" tag of 'body' tag
def get_links(url):
    try:
        # Request the URL
        resp = requests.get(url)
        # return the HTML
        soup = bs.BeautifulSoup(resp.text, 'lxml')
        # Get 'body' tag of the HTML
        body = soup.body
        # Get URL from "a" tag of 'body' tag
        links = [link.get('href') for link in body.find_all('a')]
        # Correct URL if it is relative
        links = [handle_local_links(url, link) for link in links]
        # Encoding the URL to ascii
        links = [str(link.encode('ascii')) for link in links]

        return links

    except TypeError as e:
        print(e)
        print('Got a TypeError, probably got a None that we tried to iterate over')
        return([])
    except IndexError as e:
        print(e)
        print('No valid link found, return a empty list')
        return([])
    except AttributeError as e:
        print(e)
        print('Likely got None for links, so we are throwing this')
        return([])
    except Exception as e:
        print(str(e))
        return([])


def main():
    # CPU process
    process = 5
    # The site intends to scrap
    site = 3
    p = Pool(processes=process)
    # Get random URL list
    parse_us = [random_starting_url() for _ in range(site)]
    # Multiprocessing the URL and parse the result
    data = p.map(get_links, parse_us)
    # Get each URL in a list
    data = [url for url_list in data for url in url_list]
    p.close()
    # Write to txt file
    with open('urls.txt', 'w') as f:
        f.write(str(data))


if __name__ == '__main__':
    main()

Beautiful Soup主要从解析好的HTML源码中抓取我们所需要的关键内容

Pandas与Beautiful Soup类似, 不过它侧重去抓取源码中的表格信息

pyqt5这里的作用是模拟浏览器去解析源码中的Javasript, 并最终抓取JS实际的返回值.

这里我在我的Flask env下创建了一个测试页面, 用这些模块进行一些简单的页面爬虫测试.

http://flask.showerlee.com/scrapingtest/

安装环境

OS:       Windows 7 x64   
Python:   Python3.6.2
Git Bash: Git-2.15.1.2-64-bit

一. 环境配置:

1. 安装并运行Git bash

2. 安装并测试python版本

# python -V

Python 3.6.2

3. 安装相关爬虫模块

# python -m pip install beautifulsoup4 lxml pandas html5lib pyqt5 selenum

二. Beautiful Soup演示

# vi ~/scrap1.py

Tip: 这里首先去调用io和sys模块是为了改变默认的标准输出为utf-8, 这么做是为了保证无论我们抓取的源页面是什么格式, 都不会在用BS解析时报UnicodeEncodeError.

import io
import sys
import bs4 as bs
import urllib.request

# 改变标准输出的默认编码为utf-8
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf8')

# 获取该页面编码并解码成utf-8
sauce = urllib.request.urlopen(
    'http://flask.showerlee.com/scrapingtest/').read().decode('utf-8')

# 利用BS抓取页面源代码,并利用lxml规范格式
soup = bs.BeautifulSoup(sauce, 'lxml')

# 获取页面源代码
print(soup)

# 获取titile标签源代码
print(soup.title)

# 获取titile标签name
print(soup.title.name)

# 获取titile标签字符
print(soup.title.string)
print(soup.title.text)

# 获取第一个p标签源代码
print(soup.p)

# 获取p标签源代码
print(soup.find_all('p'))

# 获取p标签所有内容
for paragraph in soup.find_all('p'):
    print(paragraph.text)

# 获取页面所有内容
print(soup.get_text())

# 获取a标签所有内容
for url in soup.find_all('a'):
    print(url.text)

# 获取a标签所有href链接
for url in soup.find_all('a'):
    print(url.get('href'))

# 获取nav标签源代码
nav = soup.nav
print(nav)

# 获取nav标签URL
for url in nav.find_all('a'):
    print(url.get('href'))

# 获取body标签内容
body = soup.body
for paragraph in body.find_all('p'):
    print(paragraph.text)

# 获取div标签下body下的内容
for div in soup.find_all('div', class_='body'):
    print(div.text)

# 获取table标签源代码
table = soup.table
# table = soup.find('table')
print(table)

# 获取table每行内容
table_rows = table.find_all('tr')

for tr in table_rows:
    td = tr.find_all('td')
    row = [i.text for i in td]
    print(row)

	

# python scrap1.py

......

三. Pandas演示

# vi ~/scrap2.py

import pandas as pd

dfs = pd.read_html(
    'http://flask.showerlee.com/scrapingtest/', header=0)

for df in dfs:
    print(df)

# python scrap2.py

  Program Name  Internet Points    Kittens?
0       Python        932914021  Definitely
1       Pascal              532    Unlikely
2         Lisp             1522   Uncertain
3           D#               12    Possibly
4        Cobol                3         No.
5      Fortran            52124        Yes.
6      Haskell               24        lol.

四. pyqt5演示

这里我们首先不解析JS, 直接利用Beautiful Soup去抓取p标签下class=jstest的内容

# vi ~/scrap3.py

import io
import sys
import bs4 as bs
import urllib.request

# 改变标准输出的默认编码为utf-8
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf8')

# 获取该页面编码并解码成utf-8
sauce = urllib.request.urlopen(
    'http://flask.showerlee.com/scrapingtest/').read().decode('utf-8')

# 利用BS抓取页面源代码,并利用lxml规范格式
soup = bs.BeautifulSoup(sauce, 'lxml')

js_test = soup.find('p', class_='jstest')

print(js_test.text)

# python scrap3.py

No js loaded

可以看到实际抓取的为未被JS处理的标签内容

这里利用pyqt5去抓取p标签下class=jstest的内容

# vi ~/scrap4.py

import bs4 as bs
import sys
from PyQt5.QtWebEngineWidgets import QWebEnginePage
from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QUrl


class Page(QWebEnginePage):
    def __init__(self, url):
        self.app = QApplication(sys.argv)
        QWebEnginePage.__init__(self)
        self.html = ''
        self.loadFinished.connect(self._on_load_finished)
        self.load(QUrl(url))
        self.app.exec_()

    def _on_load_finished(self):
        self.html = self.toHtml(self.Callable)
        # print('Load finished')

    def Callable(self, html_str):
        self.html = html_str
        self.app.quit()


def main():
    page = Page('http://flask.showerlee.com/scrapingtest/')
    soup = bs.BeautifulSoup(page.html, 'html.parser')
    js_test = soup.find('p', class_='jstest')
    print(js_test.text)


if __name__ == '__main__':
    main()

# python scrap4.py

js loaded successfully

JS解析成功.

四. selenum演示

首先我们需要从官网下载chrome driver, 并放到脚本同路径的driver目录里.

这里需要查找匹配你当前chrome浏览器版本的driver版本. 这边我的chrome版本为62.0, 所以选择driver版本为2.35.

# vi selenum.py

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup
import time, sys, io

sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf8')

def scrape():

	chromedriver = r".\driver\chromedriver.exe"
	URL = "http://flask.showerlee.com/scrapingtest/"

	try:
		driver = webdriver.Chrome(chromedriver)
		driver.set_window_position(-10000, 0)
		driver.get(URL)
		time.sleep(10)
		result = driver.execute_script("return document.body.innerHTML").encode('utf-8')
	except TimeoutException as e:
		print(e)

	soup = BeautifulSoup(result, "lxml")

	print(soup)

	driver.close()

scrape()

# python selenum.py

...

更多文档:

https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/

http://pandas.pydata.org/pandas-docs/stable/

http://pyqt.sourceforge.net/Docs/PyQt5/

https://sites.google.com/a/chromium.org/chromedriver/downloads

https://chromedriver.storage.googleapis.com/index.html

最近在业余时间去学习python web开发, 并用Flask做了一个BLOG小程序放到我的个人仓库和Github仓库, 这里把自己的文档分享给大家, 仅供参考

https://github.com/showerlee/Flaskdev

https://git.showerlee.com/showerlee/Flaskdev

安装环境

OS(virtualbox): CentOS 6.9 x64 (flask.example.com)

OS(localhost):  Windows 7 x64   

Python: Python3.4/Python2.6.6

Flask: Flask 0.12.x

nWSGI:  uwsgi-2.0.15

Nginx:  nginx- 1.10.2-1.el6

一. 系统环境配置

1.关闭iptables和selinux

# su - root

# service iptables stop

# setenforce 0

# vi /etc/sysconfig/selinux

修改

SELINUX=disabled

2.添加本地host DNS

# vi /etc/hosts

127.0.0.1    flask.example.com

二. Python配置

1.安装python3.4源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel -y

2.编译安装python3.4以及pip package manager

# wget https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tar.xz --no-check-certificate

# tar xf Python-3.4.3.tar.xz

# cd Python-3.4.3

# ./configure --prefix=/usr/local --with-ensurepip=install

# make && make install

3.安装virtualenv

# pip install --upgrade pip

# pip install virtualenv

三. Nginx配置

1. 安装nginx package

# yum install nginx -y

2.配置nginx with nWSGI

# vi /etc/nginx/conf.d/flask.conf

server {
    listen	 80;
    server_name  flask.example.com;  

    charset utf-8;

    access_log  /var/log/nginx/flask_access.log  main;
    error_log   /var/log/nginx/flask_error.log;


    location / { 
        try_files $uri @yourapplication; 
        client_max_body_size 32M;
    }
    location @yourapplication {
        include uwsgi_params;
        uwsgi_pass unix:/tmp/uwsgi.sock;       
        uwsgi_read_timeout 30s;
        uwsgi_send_timeout 30s;
    }
}

四. Flask+uWSGI配置

1. uWSGI配置

# mkdir -p /etc/uwsgi

# vi /etc/uwsgi/uwsgi.ini

[uwsgi]
base = /usr/share/nginx/html/flask
 
python-path = %(base)
app = hello
module = %(app)
callable = app
 
pidfile = /tmp/uwsgi-master.pid
touch-reload = /etc/uwsgi/bin/reload
 
master = true
processes = 5
enable-threads = true
 
limit-as = 512
 
# use unix socket because it is more secure and faster than TCP socket
socket = /tmp/uwsgi.sock
chmod-socket = 660
uid = nginx
gid = nginx
 
vacuum = true
die-on-term = true
emperor = true
 
logto = /var/log/nginx/uwsgi.log

2. 配置Flask base folder

# cd /usr/share/nginx/html

# mkdir flask

# cd flask

# virtualenv -p /usr/local/bin/python3 .py3env

3. 开启virtualenv python3环境

# source .py3env/bin/activate

4. 在此环境安装Flask相关模块

# pip install flask uwsgi PyMySQL flask-wtf passlib

5. 写入Flask测试文件

# vi hello.py

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

6. 创建uWSGI启动脚本

# mkdir -p /etc/uwsgi/bin

# vi /etc/uwsgi/bin/uwsgi.sh

#!/bin/bash
 
# Get function from functions library
. /etc/init.d/functions
 
BASE=/usr/share/nginx/html/flask
UWSGI=$BASE/.py3env/bin
INI=/etc/uwsgi
 
start() {
  echo -n "Starting server ..." 
  touch $INI/bin/reload
  $UWSGI/uwsgi --ini $INI/uwsgi.ini >/dev/null 2>&1 &
  success $"Started"
  echo
}
 
reload() {  
  $UWSGI/uwsgi --reload /tmp/uwsgi-master.pid
  success $ "Reloaded"
  echo
}
 
stop() {
  echo -n "Stopping server ..."
  $UWSGI/uwsgi --stop /tmp/uwsgi-master.pid
  success $ "Stopped"
  echo
}
 
### main logic ###
case "$1" in
  start) 
    start
    ;;
  stop)
    stop
    ;;
  reload)
    reload
    ;;
  restart)
    stop
    start
    ;;
  *)
    echo $"Usage: $0 {start|stop|restart|reload}"
    exit 1
 
esac
exit 0

7. 修改权限(可执行并保持与nginx启动user一致)

# chmod -R 755 /etc/uwsgi

# chown -R nginx:nginx /etc/uwsgi

# chmod -R 755 /usr/share/nginx/html/flask

# chown -R nginx:nginx /usr/share/nginx/html/flask

8.启动nginx+uwsgi

# service nginx start

# /etc/uwsgi/bin/uwsgi.sh start

展示效果(保证Windows本地host文件能够解析flask.example.com)

Finished...

Trouble shooting:

1.查看uwsgi log, 发现循环输出如下warning.

# tail -f /var/log/nginx/uwsgi.log

chdir(): Permission denied [core/emperor.c line 1499]
chdir(): Permission denied [core/emperor.c line 1499]
chdir(): Permission denied [core/emperor.c line 1499]
chdir(): Permission denied [core/emperor.c line 1499]
...

Solution:

# cd /usr/share/nginx/html

# source .py3env/bin/activate

# python hellp.py

 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 126-450-249

若无报错提示并显示如上输出, ctrl+C 强制退出. 并重启服务, 该warning即停止.

Tip:  该方法亦可测试你的主方法文件是否有语法格式错误.

# /etc/uwsgi/bin/uwsgi.sh start

该错误会偶尔在开发过程中添加新方法后出现, 暂不清楚原因, 有经验的朋友可以留言.

My Flask APP for the reference:

https://git.showerlee.com/showerlee/Flaskdev

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel wget xz -y

2.编译安装python2.7

# cd ~ 

# wget --no-check-certificate https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tar.xz

# tar xvfJ Python-2.7.9.tar.xz 

# cd Python-2.7.9

# ./configure --prefix=/usr/local LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

# ln -s /usr/local/bin/python2.7 /usr/local/bin/python

退出SSH并重新登录

# python -V

Python 2.7.9

# which python

/usr/local/bin/python

3.安装python setuptools

# python ez_setup.py

# which easy_install

/usr/local/bin/easy_install

# /usr/local/bin/easy_install-2.7 pip

# ll /usr/local/bin

total 6136
-rwxr-xr-x 1 root root     101 Oct 29 02:34 2to3
-rwxr-xr-x 1 root root     323 Oct 29 02:42 easy_install
-rwxr-xr-x 1 root root     331 Oct 29 02:42 easy_install-2.7
-rwxr-xr-x 1 root root      99 Oct 29 02:34 idle
-rwxr-xr-x 1 root root     287 Oct 29 02:48 pip
-rwxr-xr-x 1 root root     289 Oct 29 02:48 pip2
-rwxr-xr-x 1 root root     293 Oct 29 02:48 pip2.7
-rwxr-xr-x 1 root root      84 Oct 29 02:34 pydoc
lrwxrwxrwx 1 root root      24 Oct 29 02:36 python -> /usr/local/bin/python2.7
-rwxr-xr-x 1 root root 6224569 Oct 29 02:34 python2.7
-rwxr-xr-x 1 root root    1687 Oct 29 02:34 python2.7-config
-rwxr-xr-x 1 root root   18547 Oct 29 02:34 smtpd.py

升级完毕

# vi diff.py

#!/usr/bin/env python

import difflib

text1 = '''text1:
This mudule provides classes and fuctions for comparing sequences.
including HTML and context and unified diffs.
difflib document v7.4
add string
'''
text1_lines = text1.splitlines()

text2 = '''text2:
This mudule provides classes and fuctions for Comparing sequences.
including HTML and  context and unified diffs.
difflib document v7.5
''' 
text2_lines = text2.splitlines()

#d = difflib.Differ()
#diff = d.compare(text1_lines, text2_lines)

#print '\n'.join(list(diff))

d = difflib.HtmlDiff()
print d.make_file(text1_lines, text2_lines)

运行脚本并重定向为一个html静态页面

# chmod 755 diff.py

# ./diff.py  > diff.html

浏览器打开该页面

如图:

2. 比对两个配置文件的差异.

# vi diff02.py

#!/usr/bin/env python

import difflib
import sys

try:
	textfile1 = sys.argv[1]
	textfile2 = sys.argv[2]
except Exception, e:
	print "Error: " + str(e)
	print "Usage: %s filename1 filename2" %sys.argv[0]
	sys.exit()

def readfile(filename):
	try:
		with open(filename, 'rb') as fileHandle:
			text = fileHandle.read().splitlines()
		return text
	except IOError as error:
		print ('Read file Error:' + str(error))
		sys.exit()

if not textfile1 or not textfile2:
	print "Usage: %s filename1 filename2" %sys.argv[0]
	sys.exit()

text1_lines = readfile(textfile1)
text2_lines = readfile(textfile2)

d = difflib.HtmlDiff()

print d.make_file(text1_lines, text2_lines)

# ./diff02.py nginx.conf.v1 nginx.conf.v2 >diff02.html

QQ邮箱 POP3端口: 995 SMTP端口: 587

密码使用QQ邮箱独立密码

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import smtplib  
import email.encoders
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
import getpass

# Build a new attach instance
msg = MIMEMultipart()

# Mail configuration
Mail_subject = 'Python test mail'
Mail_content = 'Send with pic attachment'
Recipient_list = ['XXX@126.com', 'XXX@qq.com']
SMTP_server = "smtp.qq.com"  
SMTP_port = "587"
Username = raw_input('Please input your Username:')  
Password = getpass.getpass("Please input your Password: ")   
mail_postfix = "qq.com"  

# Mail attachment
def attach(file_path, file_name, type, postfix):
    with open(file_path + "/" + file_name, 'rb') as f:
        # 设置附件的MIME和文件名，这里是png类型:
        mime = MIMEBase(type, postfix, filename = file_name)
        # 加上必要的头信息:
        mime.add_header('Content-Disposition', 'attachment', filename = file_name)
        mime.add_header('Content-ID', '<0>')
        mime.add_header('X-Attachment-Id', '0')
        # 把附件的内容读进来:
        mime.set_payload(f.read())
        # 用Base64编码:
        email.encoders.encode_base64(mime)
        # 添加到MIMEMultipart:
        msg.attach(mime)

  
def send_mail(recipient, title, content): 
    # Mail info
    author = "%s<%s@%s>" %(Username, Username, mail_postfix)
    msg['Subject'] = title  
    msg['From'] = author  
    msg['To'] = ";".join(recipient)

    # Send attachment
    msg.attach(MIMEText(content, 'plain', 'utf-8'))
    attach('/Users/XXX/Pictures/com.tencent.ScreenCapture','QQ20150827-1.png', 'image', 'png')
    attach('/Users/XXX/Work/Python','test01.py', 'txt', 'py')
    attach('/Users/XXX/Work/Python','test.zip', 'zip', 'zip')

    try:  
        server = smtplib.SMTP(SMTP_server, SMTP_port)
        server.starttls()
        server.set_debuglevel(1)   
        server.login(Username, Password)  
        server.sendmail(author, recipient, msg.as_string())  
        server.quit()  
        return True  
    except Exception, e:
        print str(e)
        return False


if __name__ == '__main__': 
    if send_mail(Recipient_list, Mail_subject, Mail_content): 
        print "Sent Successfully"  
    else: 
        print "Sent Failure" 

QQ邮箱利用的是加密STMP, 需要加密版本的童鞋请关注随后的更新.

TIPS: 我的本地环境是MAC系统, Windows环境需要修改相应的字符编码.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import smtplib  
import email.encoders
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
import getpass

# Build a new attach instance
msg = MIMEMultipart()

# Mail configuration
Mail_subject = 'Python test mail'
Mail_content = 'Send with attachments'
Recipient_list = ['XXX@126.com', 'XXX@qq.com']
SMTP_server = "smtp.126.com"  
Username = raw_input('Please input your Username:')  
Password = getpass.getpass("Please input your Password: ")   
mail_postfix = "126.com"  

# Mail attachment
def attach(file_path, file_name, type, postfix):
    with open(file_path + "/" + file_name, 'rb') as f:
        # 设置附件的MIME和文件名，这里是png类型:
        mime = MIMEBase(type, postfix, filename = file_name)
        # 加上必要的头信息:
        mime.add_header('Content-Disposition', 'attachment', filename = file_name)
        mime.add_header('Content-ID', '<0>')
        mime.add_header('X-Attachment-Id', '0')
        # 把附件的内容读进来:
        mime.set_payload(f.read())
        # 用Base64编码:
        email.encoders.encode_base64(mime)
        # 添加到MIMEMultipart:
        msg.attach(mime)

  
def send_mail(recipient, title, content): 
    # Mail info
    author = "%s<%s@%s>" %(Username, Username, mail_postfix)
    msg['Subject'] = title  
    msg['From'] = author  
    msg['To'] = ";".join(recipient)

    # send attachment
    msg.attach(MIMEText(content, 'plain', 'utf-8'))
    attach('/Users/XXX/Pictures/com.tencent.ScreenCapture','QQ20150827-1.png', 'image', 'png')
    attach('/Users/XXX/Work/Python','test01.py', 'txt', 'py')
    attach('/Users/XXX/Work/Python','test.zip', 'zip', 'zip')

    try:  
        server = smtplib.SMTP()
        server.set_debuglevel(1) 
        server.connect(SMTP_server)  
        server.login(Username, Password)  
        server.sendmail(author, recipient, msg.as_string())  
        server.quit()  
        return True  
    except Exception, e:
        print str(e)
        return False


if __name__ == '__main__': 
    if send_mail(Recipient_list, Mail_subject, Mail_content): 
        print "Sent Successfully"  
    else: 
        print "Sent Failure" 

  正则表达式使用反斜杠" \ "来代表特殊形式或用作转义字符，这里跟Python的语法冲突，因此，Python用" \\\\ "表示正则表达式中的" \ "，因为正则表达式中如果要匹配" \ "，需要用\来转义，变成" \\ "，而Python语法中又需要对字符串中每一个\进行转义，所以就变成了" \\\\ "。
上面的写法是不是觉得很麻烦，为了使正则表达式具有更好的可读性，Python特别设计了原始字符串(raw string)，需要提醒你的是，在写文件路径的时候就不要使用raw string了，这里存在陷阱。raw string就是用'r'作为字符串的前缀，如 r"\n"：表示两个字符"\"和"n"，而不是换行符了。Python中写正则表达式时推荐使用这种形式。

二、正则表达式元字符说明：

.    匹配除换行符以外的任意字符

^    匹配字符串的开始

$    匹配字符串的结束

[]   用来匹配一个指定的字符类别

？   匹配前一个字符字符0次到1次

*    匹配前一个字符0次到无穷次

+    匹配前一个字符1次到无穷次

{}   对于前一个字符重复m次

{m，n} 对前一个字符重复为m到n次

\d   匹配数字，相当于[0-9]

\D   匹配任何非数字字符，相当于[^0-9]

\s   匹配任意的空白符，相当于[ fv]

\S   匹配任何非空白字符，相当于[^ fv]

\w   匹配任何字母数字字符，相当于[a-zA-Z0-9_]

\W   匹配任何非字母数字字符，相当于[^a-zA-Z0-9_]

\b   匹配单词的开始或结束

三、导入正则表达式模块

3.1、导入正则表达式模块

>>> import  re

3.2、查看正则表达式模块方法

>>> dir(re)

['DEBUG', 'DOTALL', 'I', 'IGNORECASE', 'L', 'LOCALE', 'M', 'MULTILINE', 'S', 'Scanner', 'T', 'TEMPLATE', 'U', 'UNICODE', 'VERBOSE', 'X', '_MAXCACHE', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__version__', '_alphanum', '_cache', '_cache_repl', '_compile', '_compile_repl', '_expand', '_pattern_type', '_pickle', '_subx', 'compile', 'copy_reg', 'error', 'escape', 'findall', 'finditer', 'match', 'purge', 'search', 'split', 'sre_compile', 'sre_parse', 'sub', 'subn', 'sys', 'template']
>>>

四、常用的正则表达式处理函数

4.1、re.search

     re.search 函数会在字符串内查找模式匹配，只到找到第一个匹配然后返回，如果字符串没有匹配，则返回None。

    提示：当我们不会用模块方法的时候用help

>>> help(re.search)

search(pattern, string, flags=0)

    第一个参数：规则

    第二个参数：表示要匹配的字符串

    第三个参数：标致位，用于控制正则表达式的匹配方式

    实例：下面的例子kuangl 

>>> name="Hello,My name is kuangl,nice to meet you..."

>>> k=re.search(r'k(uan)gl',name)

>>> if k:

...     print k.group(0),k.group(1)

... else:

...     print "Sorry,not search!"

...

kuangl uan

4.2、re.match

    re.match 尝试从字符串的开始匹配一个模式，也等于说是匹配第一个单词

>>> help(re.match)

match(pattern, string, flags=0)

    第一个参数：规则

    第二个参数：表示要匹配的字符串

    第三个参数：标致位，用于控制正则表达式的匹配方式

    实例：下面的例子匹配Hello单词

>>> name="Hello,My name is kuangl,nice to meet you..."

>>> k=re.match(r"(\H....)",name)

>>> if k:

...      print k.group(0),'\n',k.group(1)

... else:

...     print "Sorry,not match!"

...

Hello

Hello

>>>

     re.match与re.search的区别：re.match只匹配字符串的开始，如果字符串开始不符合正则表达式，则匹配失败，函数返回None；而re.search匹配整个字串，直到找到一个匹配。

4.3、re.findall

    re.findall 在目标字符串查找符合规则的字符串

>>> help(re.findall)

findall(pattern, string, flags=0)

   第一个参数：规则

   第二个参数：目标字符串

   但三个参数：后面还可以跟一个规则选择项

   返回的结果是一个列表，建中存放的是符合规则的字符串，如果没有符合规则的字符串呗找到，就会返回一个空值。

   实例：查找邮件账号

>>> mail='<user01@mail.com> <user02@mail.com> user04@mail.com' #第3个故意没有尖括号

>>> re.findall(r'(\w+@m....[a-z]{3})',mail)

['user01@mail.com', 'user02@mail.com', 'user04@mail.com']

4.4、re.sub

   re.sub 用于替换字符串的匹配项

>>> help(re.sub)

sub(pattern, repl, string, count=0)

   第一个参数：规则

   第二个参数：替换后的字符串

   第三个参数：字符串

   第四个参数：替换个数。默认为0，表示每个匹配项都替换

   实例：将空白处替换成-

>>> test="Hi, nice to meet you where are you from?"

>>> re.sub(r'\s','-',test)

'Hi,-nice-to-meet-you-where-are-you-from?'

>>> re.sub(r'\s','-',test,5)                      #替换至第5个

'Hi,-nice-to-meet-you-where are you from?'

>>>

4.5、re.split

   re.split 用于来分割字符串

>>> help(re.split)

split(pattern, string, maxsplit=0)

   第一个参数：规则

   第二个参数：字符串

   第三个参数：最大分割字符串，默认为0，表示每个匹配项都分割

   实例：分割所有的字符串

>>> test="Hi, nice to meet you where are you from?"

>>> re.split(r"\s+",test)

['Hi,', 'nice', 'to', 'meet', 'you', 'where', 'are', 'you', 'from?']

>>> re.split(r"\s+",test,3)                  #分割前三个

['Hi,', 'nice', 'to', 'meet you where are you from?']

>>>

4.6、re.compile

   re.compile 可以把正则表达式编译成一个正则对象

>>> help(re.compile)

compile(pattern, flags=0)

   第一个参数：规则

   第二个参数：标志位 

   实例：

>>> test="Hi, nice to meet you where are you from?"

>>> k=re.compile(r'\w*o\w*') #匹配带o的字符串

>>> dir(k)

['__copy__', '__deepcopy__', 'findall', 'finditer', 'match', 'scanner', 'search', 'split', 'sub', 'subn']

>>> print k.findall(test)     #显示所有包涵o的字符串

['to', 'you', 'you', 'from']

>>> print k.sub(lambda m: '[' + m.group(0) + ']',test)  # 将字符串中含有o的单词用[]括起来

Hi, nice [to] meet [you] where are [you] [from]?

>>>

五、用urllib2、re、os 模块下载文件的脚本

#!/usr/bin/env python

import urllib2
import re
import os
URL='http://image.baidu.com/channel/wallpaper'
read=urllib2.urlopen(URL).read()
pat =  re.compile(r'src="http://.+?.js">')
urls=re.findall(pat,read)
for i in urls:
    url= i.replace('src="','').replace('">','')
try:
    iread=urllib2.urlopen(url).read()
    name=os.path.basename(url)
    with open(name,'wb') as jsname:
    jsname.write(iread)
except:
    print url,"url error"