商业爬虫学习笔记day4

一.获取登录后页面信息的两种方法

1.第一种方法：

人为把有效cookies加到请求头中，代码如下

import urllib.request

# 确定url
url = "https://weibo.com/u/5811151623/home"

# 添加请求头
headers = {
    "User-Agent": " Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36",
    "Cookie": "SINAGLOBAL=1056235723769.1433.1526891663625; Ugrow-G0=1ac418838b431e81ff2d99457147068c; login_sid_t=92c3e5d6461a4a8852629513dba4eb99; cross_origin_proto=SSL; TC-V5-G0=8dc78264df14e433a87ecb460ff08bfe; wb_view_log=1536*8641.25; _s_tentry=passport.weibo.com; Apache=5865173766078.335.1563513209889; ULV=1563513209896:3:1:1:5865173766078.335.1563513209889:1538661475261; SCF=AtThUzjOQECs-GvkBXwLmxzpM3TCF5NC7qGqelck97jPb-KYiDq0-erPkD5uDCEcHbOaMMDc8-xeDffKoiS-W7g.; SUHB=0IIF8F3k8nZAPY; un=310955019@qq.com; TC-Page-G0=52dad2141fc02c292fc30606953e43ef|1563513247|1563513247; wb_view_log_5811151623=1536*8641.25; webim_unReadCount=%7B%22time%22%3A1563513248700%2C%22dm_pub_total%22%3A13%2C%22chat_group_client%22%3A0%2C%22allcountNum%22%3A19%2C%22msgbox%22%3A0%7D; SUBP=0033WrSXqPxfM72wWs9jqgMF55529P9D9W5i-1qlF8XW2aLLs530P9DB5JpV2hepeh.fSK5p1-WpMC4odcXt; SUB=_2AkMqbd6cdcPxrAZZmfkUyWrkbo5H-jyZuLdqAn7uJhMyAxh77g4KqSVutBF-XGCSCDwAF4qC498CuN9Q_vjFiwsA; UOR=www.hejizhan.com,vdisk.weibo.com,login.sina.com.cn"
}
# 创建请求对象
request = urllib.request.Request(url, headers=headers)
# 发送请求
response = urllib.request.urlopen(request)
# 读取数据并保存数据
data = response.read()
with open("01cookies.html", "wb") as f:
    f.write(data)

2. 第二种方法

（1）代码登录（post请求），登录成功后得到有效cookie

（2）自动带着cookie去请求

注 cookiejar能自动保存这个cookie 若不用此包的话则每次获取cookie都需要登录，获取cookie（request.get_header('Cookie')），添加cookie("Cookie":相应的cookie)，很繁琐。

代码如下

import urllib.request
from http import cookiejar
from urllib import parse

# 登录之前的 登录页的网址https://www.yaozh.com/login/
# 找登录 参数

# 1. 代码登录
# 1.1 登录的网址
login_url = 'https://www.yaozh.com/login'
# 1.2 登录的参数
login_form_data = {
    "username": "xiaomaoera12",
    "pwd": "lina081012",
    "formhash": "CE3ADF28C5",
    "backurl": "https%3A%2F%2Fwww.yaozh.com%2F"

}
# 1.3 发送登录请求POST
cook_jar = cookiejar.CookieJar()
# 定义有添加  cook 功能的 处理器
cook_hanlder = urllib.request.HTTPCookieProcessor(cook_jar)
# 根据处理器 生成 opener
opener = urllib.request.build_opener(cook_hanlder)

# 带着参数 发送post请求
# 添加请求头
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36'
}
# 1.参数转译（其是字典）; 2. post请求的 data要求是bytes
login_str = parse.urlencode(login_form_data).encode('utf-8')

login_request = urllib.request.Request(login_url, headers=headers, data=login_str)
# 如果登录成功, cookjar自动保存cookie
opener.open(login_request)

# 2. 代码带着cooke去访问 个人中心
center_url = 'https://www.yaozh.com/member/'
center_request = urllib.request.Request(center_url, headers=headers)
response = opener.open(center_url)
# bytes -->str
data = response.read().decode()

with open('02cook.html', 'w') as f:
    f.write(data)

二. 异常

HTTPError与URLError

HTTPError继承自URLError

捕获异常代码

try:
    response = urllib.request.urlopen(url)
except urllib.request.HTTPError as e:
    print(e.code)

三.requests包

安装方法pip install requests

1. get

（1）具体用法如下代码

import requests

url = "http://www.baidu.com/"
response = requests.get(url)
data = response.content
print(type(data))      # 得到<class 'bytes'>

data = response.text
print(type(data))   # 得到<class 'str'>

content得到的数据类型是字节，而text一般会猜一个编码类型（可以直接去看content,text的源码），有可能会错，所以优先使用content

（2）获取一些具体信息

a. 获取请求头

request_headers = response.request.headers

b. 获取响应头

response_headers = self.response.headers

c. 获取响应状态码

code = response.status_code

d. 请求的cookie以及响应的cookie

request_cookie = response.request._cookies  # 请求的cookie
response_cookie = response.cookies   # 响应的cookie

（3）使用requests包时，不需要对包含中文的url转译（requests自动帮我们转了），使用python自带的urllib时需要自己转译，如下

new_url = urllib.parse.quote(url, safe=string.printable)

相应的请求代码

# url = 'https://www.baidu.com/s?wd=美女'
url = 'https://www.baidu.com/s'

params = {
    'wd':"美女"
}
headers = {
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36'
}
response = requests.get(url,headers=headers, params=params)
data = response.content
with open('baidu.html', 'wb') as f:
    f.write(data)

(4) jason

当网站返回的不是html而是标准的jason时，想获取里面信息时，转化成jason时，获取数据更方便import requests

import requests
import json

url = 'https://api.github.com/user'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36'
}
# 这个 网址 返回的内容不是html 而是标准的json
response = requests.get(url, headers=headers)

# str
data = response.content.decode()
print(type(data))   # 得到数据类型为字符串（jason字符串）
print(data) 下面是返回的内容   
'''

{
  "message": "Requires authentication",
  "documentation_url": "https://developer.github.com/v3/users/#get-the-authenticated-user"
}
'''

# str-- dict 
data_dict = json.loads(data)  
# json() 自动将json字符串 转换成Python dict list 
data = response.json() 
print(data['message'])