Remote: CentOS 7.4 x64 (django.example.com)

Python: Python3.6.5

Apache: Apache 2.4.6

Mod_wsgi: 4.6.4

Django: Django 2.0.4

一. 系统环境配置

1.关闭iptables和selinux

# su - root

# systemctl stop firewalld

# setenforce 0

# vi /etc/sysconfig/selinux

修改

SELINUX=disabled

2.添加本地host DNS

# vi /etc/hosts

127.0.0.1    django.example.com

二. Python配置

1.安装python3.6.5源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel libffi-devel -y

2.编译安装python3.6.5以及pip package manager

# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz --no-check-certificate

# tar xf Python-3.6.5.tar.xz

# cd Python-3.6.5

# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

# make && make altinstall

3.安装virtualenv

# pip3.6 install --upgrade pip

# pip3.6 install virtualenv

三. Django环境配置

1. 配置Django virtualenv

# mkdir -p /var/www/html/django

# cd /var/www/html/django

# virtualenv -p /usr/local/bin/python3.6 .py3env

2. 开启virtualenv python3环境

# source .py3env/bin/activate

3. 在此环境安装Django相关模块

# pip install django pymysql

四. Apache配置

1. 安装apache package

# yum install httpd httpd-devel -y

2.安装mod_wsgi for python3

Tip:这里其实是一个远古巨坑, 网上90%以上资料的会粗心的直接使用yum install mod_wsgi去安装apache mod_wsgi模块, 这样做其实最终mod模块会调用本地默认的python2的所有库文件, 无论你后面如何配置django入口文件, apache都不会使用我们配置的virutalenv下隔离的python3, 导致apache无法调用python3而报错. 这里小伙伴要注意哦.

# pip install mod_wsgi 

3.导出apache所需的mod_wsgi模块

# mod_wsgi-express install-module

LoadModule wsgi_module "/usr/lib64/httpd/modules/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"
WSGIPythonHome "/var/www/html/.py3env"

4.配置apache配置文件

# vi /etc/httpd/conf/httpd.conf

末行添加:

LoadModule wsgi_module "/usr/lib64/httpd/modules/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"

# vi /etc/httpd/conf.d/django.conf

WSGIPythonHome "/var/www/html/django/.py3env"

Listen 8080
<VirtualHost *:8080>

ServerName django.example.com

Alias /static /var/www/html/django/static
<Directory /var/www/html/django/static>
 Require all granted
</Directory>

<Directory /var/www/html/django/myproject>
  <Files wsgi.py>
    Require all granted
  </Files>
</Directory>

WSGIDaemonProcess myproject python-path=/var/www/html/django/.py3env/lib/python3.6/site-packages
WSGIScriptAlias / /var/www/html/django/myproject/wsgi.py

</VirtualHost>

5.重启apache并设置开机自启动

# systemctl restart httpd

# systemctl enable httpd

五. Django项目配置

1. 保证virtualenv python3环境开启

# source /var/www/html/django/.py3env/bin/activate

2.创建一个Django项目

# cd /var/www/html/django/

# django-admin startproject myproject .

3.添加static目录

# vi myproject/settings.py

末行添加:

STATIC_ROOT = os.path.join(BASE_DIR, "static/")

4.创建本地SQLlite文件

Tip:这里使用SQLlite代替其他数据库作为我们项目的DB

# ./manage.py makemigrations
# ./manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK

5.创建项目管理员账户

# ./manage.py createsuperuser

Username (leave blank to use 'root'): root
Email address: admin@admin.com
Password:
Password (again):
Superuser created successfully.

6.生成项目静态文件目录

# ./manage.py collectstatic

7.修改wsgi入口文件

# vi myproject/wsgi.py

import os
import sys
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
sys.path.append('/var/www/html/django')

from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()

8.添加ALLOWED_HOSTS

# vi myproject/settings.py

Update:

ALLOWED_HOSTS = ['django.example.com']

9.修改项目属主和权限

# chmod -R 755 /var/www/html

# chown -R apache:apache /var/www/html

查看最终目录下的生成的项目文件

# ls -l

-rwxr-xr-x 1 apache apache 38912 Apr 16 15:04 db.sqlite3
-rwxr-xr-x 1 apache apache   541 Apr 16 14:50 manage.py
drwxr-xr-x 3 apache apache  4096 Apr 16 15:21 myproject
drwxr-xr-x 3 apache apache  4096 Apr 16 15:05 static

最终浏览器访问django项目

Tip:保证windows本地添加django服务器的HOST域名

django测试页面

项目主页, 输入之前创建的管理员账号密码

项目后台

Finished...

1. 随机执行方法

# vi ~/multipro1.py

import multiprocessing

def spawn(num):
    print('Sprawned! {}.'.format(num))


if __name__ == "__main__":
    for i in range(50):
        p = multiprocessing.Process(target=spawn, args=(i,))

        p.start()

2. 按顺序执行方法

# vi ~/multipro2.py

import multiprocessing


def spawn(num):
    print('Sprawned! {}.'.format(num))


if __name__ == "__main__":
    for i in range(50):
        p = multiprocessing.Process(target=spawn, args=(i,))

        p.start()

        # Waiting to finish One by one
        p.join()

Tip: 这里if __name__ == "__main__":的作用:

1. 当直接执行该脚本时, 内建变量__name__的值被赋予"__main__", 所以按照if逻辑, python解释器可以继续执行接下来的代码.

2. 当其他脚本去调用(import)该脚本时, __name__的值被赋予当前脚本名"multipro1", 而不是"__main__", 所以按照if逻辑, 保证if下面的代码不会被其他脚本调用.

所以在我们日常编写脚本的时候, 可以推荐按照如下结构, 在自己使用正常的同时, 保证在其他python脚本在调用test.py中的test1(), test2(), test3()的同时, 不会去执行if条件下的方法.

# vi test.py

def test1():
    ....

def test2():
    ....

def test3():
    ....

if __name__ == "__main__":
    ....

3. 多线程处理返回值.

# vi multipro3.py

from multiprocessing import pool

def job(num):
	return num * 2

if __name__ == '__main__':
	p = Pool(processes=20)
	data = p.map(job, range(5))
	data2 = p.map(job, [5, 2])
	p.close()

4.多线程爬取随机网站URL

# vi ~/multipro4.py

from multiprocessing import Pool
import bs4 as bs
import random
import requests
import string

# Return Four digits domain URL
def random_starting_url():
    starting = ''.join(random.SystemRandom().choice(
        string.ascii_lowercase) for _ in range(4))
    url = ''.join(['http://', starting, '.com'])
    return url


# Correct URL if it is relative
def handle_local_links(url, link):
    if link.startswith('/'):
        return ''.join([url, link])
    else:
        return link


# Get URL from "a" tag of 'body' tag
def get_links(url):
    try:
        # Request the URL
        resp = requests.get(url)
        # return the HTML
        soup = bs.BeautifulSoup(resp.text, 'lxml')
        # Get 'body' tag of the HTML
        body = soup.body
        # Get URL from "a" tag of 'body' tag
        links = [link.get('href') for link in body.find_all('a')]
        # Correct URL if it is relative
        links = [handle_local_links(url, link) for link in links]
        # Encoding the URL to ascii
        links = [str(link.encode('ascii')) for link in links]

        return links

    except TypeError as e:
        print(e)
        print('Got a TypeError, probably got a None that we tried to iterate over')
        return([])
    except IndexError as e:
        print(e)
        print('No valid link found, return a empty list')
        return([])
    except AttributeError as e:
        print(e)
        print('Likely got None for links, so we are throwing this')
        return([])
    except Exception as e:
        print(str(e))
        return([])


def main():
    # CPU process
    process = 5
    # The site intends to scrap
    site = 3
    p = Pool(processes=process)
    # Get random URL list
    parse_us = [random_starting_url() for _ in range(site)]
    # Multiprocessing the URL and parse the result
    data = p.map(get_links, parse_us)
    # Get each URL in a list
    data = [url for url_list in data for url in url_list]
    p.close()
    # Write to txt file
    with open('urls.txt', 'w') as f:
        f.write(str(data))


if __name__ == '__main__':
    main()

Beautiful Soup主要从解析好的HTML源码中抓取我们所需要的关键内容

Pandas与Beautiful Soup类似, 不过它侧重去抓取源码中的表格信息

pyqt5这里的作用是模拟浏览器去解析源码中的Javasript, 并最终抓取JS实际的返回值.

这里我在我的Flask env下创建了一个测试页面, 用这些模块进行一些简单的页面爬虫测试.

http://flask.showerlee.com/scrapingtest/

安装环境

OS:       Windows 7 x64   
Python:   Python3.6.2
Git Bash: Git-2.15.1.2-64-bit

一. 环境配置:

1. 安装并运行Git bash

2. 安装并测试python版本

# python -V

Python 3.6.2

3. 安装相关爬虫模块

# python -m pip install beautifulsoup4 lxml pandas html5lib pyqt5 selenum

二. Beautiful Soup演示

# vi ~/scrap1.py

Tip: 这里首先去调用io和sys模块是为了改变默认的标准输出为utf-8, 这么做是为了保证无论我们抓取的源页面是什么格式, 都不会在用BS解析时报UnicodeEncodeError.

import io
import sys
import bs4 as bs
import urllib.request

# 改变标准输出的默认编码为utf-8
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf8')

# 获取该页面编码并解码成utf-8
sauce = urllib.request.urlopen(
    'http://flask.showerlee.com/scrapingtest/').read().decode('utf-8')

# 利用BS抓取页面源代码,并利用lxml规范格式
soup = bs.BeautifulSoup(sauce, 'lxml')

# 获取页面源代码
print(soup)

# 获取titile标签源代码
print(soup.title)

# 获取titile标签name
print(soup.title.name)

# 获取titile标签字符
print(soup.title.string)
print(soup.title.text)

# 获取第一个p标签源代码
print(soup.p)

# 获取p标签源代码
print(soup.find_all('p'))

# 获取p标签所有内容
for paragraph in soup.find_all('p'):
    print(paragraph.text)

# 获取页面所有内容
print(soup.get_text())

# 获取a标签所有内容
for url in soup.find_all('a'):
    print(url.text)

# 获取a标签所有href链接
for url in soup.find_all('a'):
    print(url.get('href'))

# 获取nav标签源代码
nav = soup.nav
print(nav)

# 获取nav标签URL
for url in nav.find_all('a'):
    print(url.get('href'))

# 获取body标签内容
body = soup.body
for paragraph in body.find_all('p'):
    print(paragraph.text)

# 获取div标签下body下的内容
for div in soup.find_all('div', class_='body'):
    print(div.text)

# 获取table标签源代码
table = soup.table
# table = soup.find('table')
print(table)

# 获取table每行内容
table_rows = table.find_all('tr')

for tr in table_rows:
    td = tr.find_all('td')
    row = [i.text for i in td]
    print(row)

	

# python scrap1.py

......

三. Pandas演示

# vi ~/scrap2.py

import pandas as pd

dfs = pd.read_html(
    'http://flask.showerlee.com/scrapingtest/', header=0)

for df in dfs:
    print(df)

# python scrap2.py

  Program Name  Internet Points    Kittens?
0       Python        932914021  Definitely
1       Pascal              532    Unlikely
2         Lisp             1522   Uncertain
3           D#               12    Possibly
4        Cobol                3         No.
5      Fortran            52124        Yes.
6      Haskell               24        lol.

四. pyqt5演示

这里我们首先不解析JS, 直接利用Beautiful Soup去抓取p标签下class=jstest的内容

# vi ~/scrap3.py

import io
import sys
import bs4 as bs
import urllib.request

# 改变标准输出的默认编码为utf-8
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf8')

# 获取该页面编码并解码成utf-8
sauce = urllib.request.urlopen(
    'http://flask.showerlee.com/scrapingtest/').read().decode('utf-8')

# 利用BS抓取页面源代码,并利用lxml规范格式
soup = bs.BeautifulSoup(sauce, 'lxml')

js_test = soup.find('p', class_='jstest')

print(js_test.text)

# python scrap3.py

No js loaded

可以看到实际抓取的为未被JS处理的标签内容

这里利用pyqt5去抓取p标签下class=jstest的内容

# vi ~/scrap4.py

import bs4 as bs
import sys
from PyQt5.QtWebEngineWidgets import QWebEnginePage
from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QUrl


class Page(QWebEnginePage):
    def __init__(self, url):
        self.app = QApplication(sys.argv)
        QWebEnginePage.__init__(self)
        self.html = ''
        self.loadFinished.connect(self._on_load_finished)
        self.load(QUrl(url))
        self.app.exec_()

    def _on_load_finished(self):
        self.html = self.toHtml(self.Callable)
        # print('Load finished')

    def Callable(self, html_str):
        self.html = html_str
        self.app.quit()


def main():
    page = Page('http://flask.showerlee.com/scrapingtest/')
    soup = bs.BeautifulSoup(page.html, 'html.parser')
    js_test = soup.find('p', class_='jstest')
    print(js_test.text)


if __name__ == '__main__':
    main()

# python scrap4.py

js loaded successfully

JS解析成功.

四. selenum演示

首先我们需要从官网下载chrome driver, 并放到脚本同路径的driver目录里.

这里需要查找匹配你当前chrome浏览器版本的driver版本. 这边我的chrome版本为62.0, 所以选择driver版本为2.35.

# vi selenum.py

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup
import time, sys, io

sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf8')

def scrape():

	chromedriver = r".\driver\chromedriver.exe"
	URL = "http://flask.showerlee.com/scrapingtest/"

	try:
		driver = webdriver.Chrome(chromedriver)
		driver.set_window_position(-10000, 0)
		driver.get(URL)
		time.sleep(10)
		result = driver.execute_script("return document.body.innerHTML").encode('utf-8')
	except TimeoutException as e:
		print(e)

	soup = BeautifulSoup(result, "lxml")

	print(soup)

	driver.close()

scrape()

# python selenum.py

...

更多文档:

https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/

http://pandas.pydata.org/pandas-docs/stable/

http://pyqt.sourceforge.net/Docs/PyQt5/

https://sites.google.com/a/chromium.org/chromedriver/downloads

https://chromedriver.storage.googleapis.com/index.html

最近在业余时间去学习python web开发, 并用Flask做了一个BLOG小程序放到我的个人仓库和Github仓库, 这里把自己的文档分享给大家, 仅供参考

https://github.com/showerlee/Flaskdev

https://git.showerlee.com/showerlee/Flaskdev

安装环境

OS(virtualbox): CentOS 6.9 x64 (flask.example.com)

OS(localhost):  Windows 7 x64   

Python: Python3.4/Python2.6.6

Flask: Flask 0.12.x

nWSGI:  uwsgi-2.0.15

Nginx:  nginx- 1.10.2-1.el6

一. 系统环境配置

1.关闭iptables和selinux

# su - root

# service iptables stop

# setenforce 0

# vi /etc/sysconfig/selinux

修改

SELINUX=disabled

2.添加本地host DNS

# vi /etc/hosts

127.0.0.1    flask.example.com

二. Python配置

1.安装python3.4源及依赖包

# yum install epel-release -y

# yum groupinstall "Development tools" -y

# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel zx-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel -y

2.编译安装python3.4以及pip package manager

# wget https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tar.xz --no-check-certificate

# tar xf Python-3.4.3.tar.xz

# cd Python-3.4.3

# ./configure --prefix=/usr/local --with-ensurepip=install

# make && make install

3.安装virtualenv

# pip install --upgrade pip

# pip install virtualenv

三. Nginx配置

1. 安装nginx package

# yum install nginx -y

2.配置nginx with nWSGI

# vi /etc/nginx/conf.d/flask.conf

server {
    listen	 80;
    server_name  flask.example.com;  

    charset utf-8;

    access_log  /var/log/nginx/flask_access.log  main;
    error_log   /var/log/nginx/flask_error.log;


    location / { 
        try_files $uri @yourapplication; 
        client_max_body_size 32M;
    }
    location @yourapplication {
        include uwsgi_params;
        uwsgi_pass unix:/tmp/uwsgi.sock;       
        uwsgi_read_timeout 30s;
        uwsgi_send_timeout 30s;
    }
}

四. Flask+uWSGI配置

1. uWSGI配置

# mkdir -p /etc/uwsgi

# vi /etc/uwsgi/uwsgi.ini

[uwsgi]
base = /usr/share/nginx/html/flask
 
python-path = %(base)
app = hello
module = %(app)
callable = app
 
pidfile = /tmp/uwsgi-master.pid
touch-reload = /etc/uwsgi/bin/reload
 
master = true
processes = 5
enable-threads = true
 
limit-as = 512
 
# use unix socket because it is more secure and faster than TCP socket
socket = /tmp/uwsgi.sock
chmod-socket = 660
uid = nginx
gid = nginx
 
vacuum = true
die-on-term = true
emperor = true
 
logto = /var/log/nginx/uwsgi.log

2. 配置Flask base folder

# cd /usr/share/nginx/html

# mkdir flask

# cd flask

# virtualenv -p /usr/local/bin/python3 .py3env

3. 开启virtualenv python3环境

# source .py3env/bin/activate

4. 在此环境安装Flask相关模块

# pip install flask uwsgi PyMySQL flask-wtf passlib

5. 写入Flask测试文件

# vi hello.py

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

6. 创建uWSGI启动脚本

# mkdir -p /etc/uwsgi/bin

# vi /etc/uwsgi/bin/uwsgi.sh

#!/bin/bash
 
# Get function from functions library
. /etc/init.d/functions
 
BASE=/usr/share/nginx/html/flask
UWSGI=$BASE/.py3env/bin
INI=/etc/uwsgi
 
start() {
  echo -n "Starting server ..." 
  touch $INI/bin/reload
  $UWSGI/uwsgi --ini $INI/uwsgi.ini >/dev/null 2>&1 &
  success $"Started"
  echo
}
 
reload() {  
  $UWSGI/uwsgi --reload /tmp/uwsgi-master.pid
  success $ "Reloaded"
  echo
}
 
stop() {
  echo -n "Stopping server ..."
  $UWSGI/uwsgi --stop /tmp/uwsgi-master.pid
  success $ "Stopped"
  echo
}
 
### main logic ###
case "$1" in
  start) 
    start
    ;;
  stop)
    stop
    ;;
  reload)
    reload
    ;;
  restart)
    stop
    start
    ;;
  *)
    echo $"Usage: $0 {start|stop|restart|reload}"
    exit 1
 
esac
exit 0

7. 修改权限(可执行并保持与nginx启动user一致)

# chmod -R 755 /etc/uwsgi

# chown -R nginx:nginx /etc/uwsgi

# chmod -R 755 /usr/share/nginx/html/flask

# chown -R nginx:nginx /usr/share/nginx/html/flask

8.启动nginx+uwsgi

# service nginx start

# /etc/uwsgi/bin/uwsgi.sh start

展示效果(保证Windows本地host文件能够解析flask.example.com)

Finished...

Trouble shooting:

1.查看uwsgi log, 发现循环输出如下warning.

# tail -f /var/log/nginx/uwsgi.log

chdir(): Permission denied [core/emperor.c line 1499]
chdir(): Permission denied [core/emperor.c line 1499]
chdir(): Permission denied [core/emperor.c line 1499]
chdir(): Permission denied [core/emperor.c line 1499]
...

Solution:

# cd /usr/share/nginx/html

# source .py3env/bin/activate

# python hellp.py

 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 126-450-249

若无报错提示并显示如上输出, ctrl+C 强制退出. 并重启服务, 该warning即停止.

Tip:  该方法亦可测试你的主方法文件是否有语法格式错误.

# /etc/uwsgi/bin/uwsgi.sh start

该错误会偶尔在开发过程中添加新方法后出现, 暂不清楚原因, 有经验的朋友可以留言.

My Flask APP for the reference:

https://git.showerlee.com/showerlee/Flaskdev