0安装和创建爬虫项目

创建项目

scrapy  startproject projectName
cd projectName
scrapy genspider [爬虫名字] [爬虫域名] 爬虫名字不能和projectName重名

 运行项目

scrapy crawl [爬虫名字]

或者用配置文件的方式

from scrapy import  cmdline
cmdline.execute(['scrapy','crawl','dalian_spider'])  
爬虫名字

setting.py设置

20
ROBOTSTXT_OBEY = False

40
DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'
}