Scrapy的安装与使用

一、安装scrapy

1.Linux环境

yum install python34-devel epel-release libxslt-devel libxml2-devel openssl-devel
pip install Scrapy -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

此处可能会有报错：

Download error on https://pypi.python.org/simple/incremental/: [Errno 97] Address family not supported by protocol -- Some packages may not be found!
    Couldn't find index page for 'incremental' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno 97] Address family not supported by protocol -- Some packages may not be found!
    No local packages or working download links found for incremental>=16.10.1
......
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-bkql5v24/Twisted/

解决方法：

先安装incremental
pip install --upgrade incremental  -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com
再安装scrapy就不会有报错了！

2.Windows环境

pip install Scrapy

二、使用scrapy

1.Windows使用

执行scrapy crawl xxxx报错：

ModuleNotFoundError: No module named 'win32api'

解决方法：

下载Python访问windows系统API的第三方库，库的名称叫pywin32，可以从网上直接下载，下载链接：http://sourceforge.net/projects/pywin32/files%2Fpywin32/ （下载适合你的Python版本）

未完待续。。。