Web_Scraping Techniques

web_scraping_package.py

from bs4 import BeautifulSoup
import requests
session = requests.Session()
headers = {
'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
}

import sys
sys.path

获取python的path

['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/christopher/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']

这里我们把

web_scraping_package.py

放到

/home/christopher/.local/lib/python3.6/site-packages

目录下

以后就可以直接import

from web_scraping_package import session, headers, BeautifulSoup

就不用再写一大串导入文件了。