爬虫发送post请求（requests和scrapy）

一、scrapy框架的

cookies = {'zgb_abtest':'xxxx','wsKey':'xxxxxx'}
headers = {
    'Referer': 'xxxxx',
    'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
    'Host': 'xxxxx',
    'Connection': 'Keep-Alive',
    'Accept-Encoding': 'gzip'
}

data = {'keyword': response.meta['cat_name'],
        'pageNum': str(i),
        'pageSize': '20',
        'frontCatalogId': response.meta['cat_id'],
        }
request = scrapy.FormRequest(url=url_list,cookies=self.cookie,headers=self.headers,formdata=data, callback=self.get_goods_lists)

二、requests的

data = {'keyword':'生肖茅台酒',
        'pageNum':'2',
        'pageSize':'20',
        'frontCatalogId':'1172214559477501952',
        }
headers = {
    'cookie':'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36'
}
response = requests.post(url=url,data=data,headers=headers)

总结：
scrapy框架的cookie和User-Agent要单独写出来，cookie和headers写在spider里面，User-Agent写在setting里面；
requests的cookie和User-Agent写在headers里面