scrapy框架修改单个爬虫的配置,包括下载延时,下载超时设置

在一个框架里面有多个爬虫时,每个爬虫的需求不相同,例如,延时的时间,所以可以在这里配置一下custom_settings = {},大括号里面写需要修改的配置,然后就能把settings里面的配置给覆盖了
例如:
custom_settings = {
        "ITEM_PIPELINES": {
            'taskspider.pipelines.CommonPipeline': 300,
        }
    }
    custom_settings = {
        'DOWNLOAD_DELAY': 1.25,
     'DOWNLOAD_TIMEOUT':60,
}

  DOWNLOAD_DELAY是下载延时的意思,就是下载网页(html)的间隔时间,

  DOWNLOAD_TIMEOUT是超时时间限制,就是如果60s还没有把网页(html)下载了,那么就会放弃这个网页,例如pycharm运行爬虫时的提示:“(failed 1 times):User timeout caused connection failure: Getting http://shop.lelai.com/product/detail?id=634&sid=306 took longer than 60.0 seconds..”

原文地址:https://www.cnblogs.com/qiaoer1993/p/10736279.html