12-UA池和代理池

一、UA池和代理池

1、UA池

scrapy的下载中间件：

下载中间件（Downloader Middlewares）位于scrapy引擎和下载器之间的一层组件。

作用：

（1）引擎将请求传递给下载器过程中，下载中间件可以对请求进行一系列处理。比如设置请求的 User-Agent，设置代理等

（2）在下载器完成将Response传递给引擎中，下载中间件可以对响应进行一系列处理。比如进行gzip解压等。

我们主要使用下载中间件处理请求，一般会对请求设置随机的User-Agent ，设置随机的代理。目的在于防止爬取网站的反爬虫策略。

UA池作用：尽可能多的将scrapy框架中的请求伪装成不同类型的浏览器以及不同电脑的身份。

- 操作流程：

1.下载中间件中拦截请求

2.将拦截到的请求的请求头信息中的UA进行伪装

3.在配置文件中开启下载中间件

代码展示：（只需要看proccess_request方法）

# -*- coding: utf-8 -*-

# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals
from scrapy.http import HtmlResponse
import random
from time import sleep

class WangyixinwenDownloaderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.
    user_agent_list = [
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "
        "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
        "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "
        "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "
        "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "
        "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "
        "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "
        "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "
        "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
    ]
    # 代理池
    PROXY_http = [
        '153.180.102.104:80',
        '195.208.131.189:56055',
    ]
    PROXY_https = [
        '120.83.49.90:9000',
        '95.189.112.214:35508',
    ]

    def process_request(self, request, spider):
        """
        在这里可以设置UA，如果要设置IP就需要等自己IP出现问题再前去process_exception中设置代理IP
        但最后一定要返回request对象
        :param request:
        :param spider:
        :return:
        """
        # ua = random.choice(self.user_agent_list)#随机选择一个元素（反爬策略（即不同浏览器和机型））出来
        # request.headers["User-Agent"] = ua
        #return request#将修正的request对象重新进行发送
        return None

    def process_response(self, request, response, spider):
        #刚才响应体缺失，因此从这里我们应该重新返回新的响应体
        #这里要用到爬虫程序中的urls，判断url是否在里面，在urls里面的就会出现响应缺失，
        # 、因此需要返回新的响应体
        if request.url in spider.urls:
            #响应缺失是因为是动态加载数据，因此我们配合selenium使用
            #在这里实例化selenium的话会被实例化多次，然而selenium只需要实例化一次，
            #这个时候我们可以将selenium放在实例化一次的爬虫程序开始的时候,实例化完成引入
            sleep(2)
            bro = spider.bro.get(url=request.url)#浏览器中发送请求
            sleep(1)
            spider.bro.execute_script("window.scrollTo(0,document.body.scrollHeight)")
            sleep(1.5)
            spider.bro.execute_script("window.scrollTo(0,document.body.scrollHeight)")
            sleep(0.7)
            spider.bro.execute_script("window.scrollTo(0,document.body.scrollHeight)")
            sleep(1)
            spider.bro.execute_script("window.scrollTo(0,document.body.scrollHeight)")
            #发送到请求我们需要获取浏览器当前页面的源码数据 获取数据之前需要翻滚页面
            page_text = spider.bro.page_source
            #改动返回响应对象  scrapy提供了一个库url=spider.bro.current_url, body=page_text, encoding='utf-8', request=request
            new_response = HtmlResponse(url=request.url,body=page_text,encoding="utf-8",request=request)
            return new_response
        else:
            return response
        #提交完新的响应体之后，去设置将下载中间件打开

    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        #如果抛出异常，那么就换代理IP
        # 代理ip的设定
        # if request.url.split(':')[0] == 'http':
        #     request.meta['proxy'] = random.choice(self.PROXY_http)
        # else:
        #     request.meta['proxy'] = random.choice(self.PROXY_https)
        # # 将修正后的请求对象重新进行请求发送
        # return request
        pass

然后去配置文件开启中间件settings.py:

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
   'Wangyixinwen.middlewares.WangyixinwenDownloaderMiddleware': 543,
}

2、代理池

作用：尽可能多的将scrapy框架中的请求的IP设置成不同的，当服务器禁止自己电脑的ip的时候将自动调用process_exception中的ip代理设置

操作流程：

1.在下载中间件中拦截请求

2.将拦截到的请求的IP修改成某一代理IP

3.在配置文件中开启下载中间件

 def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        #如果抛出异常，那么就换代理IP
        # 代理ip的设定
         if request.url.split(':')[0] == 'http':
             request.meta['proxy'] = random.choice(self.PROXY_http)
         else:
             request.meta['proxy'] = random.choice(self.PROXY_https)
         # 将修正后的请求对象重新进行请求发送
         return request

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
   'Wangyixinwen.middlewares.WangyixinwenDownloaderMiddleware': 543,
}

12-UA池和代理池

一、UA池和代理池

1、UA池

2、代理池

拦截请求直接在上面中间件代码的process_exception中，配置文件也在上面。