在Scrapy中实现随机UA，代理IP，出错重试

在这篇文章中，我们使用Scrapy中的下载中间件来实现标题里的三个功能

• 参考：Scrapy文档 - downloader middleware

注意所有的下载中间件都需要在settings.py中启用：

DOWNLOADER_MIDDLEWARES = {
    '项目名称.middlewares.下载中间件类名': 启动优先级（数字）,
}

默认的启动优先级见：downloader-middlewares-base

随机UA

Scrapy文档 - UserAgentMiddleware
参考 - Scrapy设置代理Proxy
实现：

from fake_useragent import UserAgent


class RandomUAMiddleware(object):
    def __init__(self):
        self.user_agent = UserAgent().random

    def process_request(self, request, spider):
        request.headers['User-Agent'] = self.user_agent

测试：

    def start_requests(self):
        yield scrapy.Request(url='https://httpbin.org/get', callback=self.parse)

    def parse(self, response):
        # from scrapy.shell import inspect_response
        # inspect_response(response, self)
        from scrapy.selector import Selector
        s = Selector(text=response.text)
        ua = s.re(r'"User-Agent": "(.*)"')
        self.logger.error("UA", ua)

代理IP

Scrapy文档 - HttpProxyMiddleware

实现：

class ProxyMiddleware(object):

    def process_request(self, request, spider):
        proxy = requests.get(URL).text  # URL是自建代理池的web接口
        request.meta["proxy"] = proxy

测试：

    def start_requests(self):
        yield scrapy.Request(url='https://httpbin.org/get', callback=self.parse)

    def parse(self, response):
        # from scrapy.shell import inspect_response
        # inspect_response(response, self)
        from scrapy.selector import Selector
        s = Selector(text=response.text)
        origin = s.re(r'"origin": "(.*)"')
        self.logger.error("origin", origin)

出错重试与代理切换

Scrapy文档 - DownloadTimeoutMiddleware
Scrapy文档 - RetryMiddleware