User Agent 介绍
User Agent 的本质
一个特殊字符串头
User Agent 的作用
使得服务器能够识别客户使用的操作系统及版本、CPU 类型、浏览器及版本、浏览器渲染引擎、浏览器语言、浏览器插件等
查看浏览器 UA 的方法
查看 Scrapy 爬虫 UA 的方法
scrapy shell 网址
Scrapy 设置随机 UA
编写 UserAgentMiddleware 类
建立 user-agent 池
在每次发送 request 之前从 agent 池中随机选取一项设置 request 的 User-Agent
1 class UserAgentMiddleware(object):
2 def __init__(self):
3 self.user_agent_list = [
4 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "
5 "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
6 "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "
7 "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
8 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "
9 "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
10 "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "
11 "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
12 "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "
13 "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
14 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "
15 "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
16 "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "
17 "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
18 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
19 "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
20 "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "
21 "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
22 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "
23 "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
24 "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
25 "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
26 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
27 "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
28 "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
29 "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
30 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
31 "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
32 "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "
33 "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
34 "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
35 "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
36 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "
37 "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
38 "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "
39 "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
40 ]
41
42 def process_request(self, request, spider):
43 user_agent = random.choice(self.user_agent_list)
44 request.headers['User-Agent'] = user_agent
45 print('user_agent:%s'%user_agent)
设置 settings.py 文件设置相应的配置和属性
在 DOWNLOADER_MIDDLEWARES 下配置中间件
1 DOWNLOADER_MIDDLEWARES = {
2 # 'sohu.middlewares.SohuDownloaderMiddleware': 543,
3 'sohu.middlewares.UserAgentMiddleware': 300,
4 }