首先创建项目test1，这里不能不直接用test作文件名，会出现错误，
scrapy startproject test1
然后在目录/～/test1/test1/spiders 中创建文件spider.py
1 import scrapy 2 class MySpider(scrapy.Spider): 3 name = "myspider" 4 start_urls = [ # 5 "http://example.com", 6 "http://example.org", 7 "http://example.net" 8 ] 9 def parse(self , response): 10 if ".org" in response.url: 11 from scrapy.shell import inspect_response 12 inspect_response(response , self)
在使用scrapy命令及引用时，首字母不大写， 但在书上都是Scrapy且会出现错误：未找到命令
然后在test1项目目录下，运行爬虫
scrapy crawl myspider
同时shell也会被调用
2020-04-09 18:43:24 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: test1) 2020-04-09 18:43:24 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.6.9 (default, Nov 7 2019, 10:44:02) - [GCC 8.3.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Linux-5.0.0-27-generic-x86_64-with-Ubuntu-18.04-bionic 2020-04-09 18:43:24 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor 2020-04-09 18:43:24 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'test1', 'NEWSPIDER_MODULE': 'test1.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['test1.spiders']} 2020-04-09 18:43:24 [scrapy.extensions.telnet] INFO: Telnet Password: 9a20daab62ac5a9e 2020-04-09 18:43:24 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats'] 2020-04-09 18:43:24 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-04-09 18:43:24 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-04-09 18:43:24 [scrapy.middleware] INFO: Enabled item pipelines: [] 2020-04-09 18:43:24 [scrapy.core.engine] INFO: Spider opened 2020-04-09 18:43:24 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2020-04-09 18:43:24 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024 2020-04-09 18:43:24 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://example.com/robots.txt> (referer: None) 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 12 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 13 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 14 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 18 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 19 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 20 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 22 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 23 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 25 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 27 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 29 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 31 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 32 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 43 without any user agent to enforce it on. 2020-04-09 18:43:24 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://example.org/robots.txt> (referer: None) 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 12 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 13 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 14 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 18 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 19 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 20 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 22 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 23 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 25 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 27 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 29 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 31 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 32 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 43 without any user agent to enforce it on. 2020-04-09 18:43:24 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://example.net/robots.txt> (referer: None) 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 12 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 13 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 14 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 18 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 19 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 20 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 22 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 23 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 25 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 27 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 29 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 31 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 32 without any user agent to enforce it on. 2020-04-09 18:43:24 [protego] DEBUG: Rule at line 43 without any user agent to enforce it on. 2020-04-09 18:43:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.com> (referer: None) 2020-04-09 18:43:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.org> (referer: None) 2020-04-09 18:43:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.net> (referer: None) [s] Available Scrapy objects: [s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc) [s] crawler <scrapy.crawler.Crawler object at 0x7fe120f3ac88> [s] item {} [s] request <GET http://example.org> [s] response <200 http://example.org> [s] settings <scrapy.settings.Settings object at 0x7fe120c83e80> [s] spider <MySpider 'myspider' at 0x7fe12091f5c0> [s] Useful shortcuts: [s] shelp() Shell help (print this help) [s] view(response) View response in a browser
In [1]: response.url 查看地址 Out[1]: 'http://example.org'