1. 用requests库和BeautifulSoup库,爬取校园新闻首页新闻的标题、链接、正文。
import requests newsurl = 'http://news.gzcc.cn/html/xiaoyuanxinwen/' res = requests.get(newsurl) # 返回response对象 res.encoding = 'utf-8' from bs4 import BeautifulSoup soup = BeautifulSoup(res.text, 'html.parser') for news in soup.select("li"): if(len(news.select('.news-list-info'))>0): print(news.select('a')[0].attrs['href']) print(news.select('.news-list-title')[0].text) a = news.select('a')[0].attrs['href'] rq = requests.get(a) rq.encoding="utf-8" soup2 = BeautifulSoup(rq.text, 'html.parser') print(soup2.select("#content")[0].text)
效果显示如下图所示:
2. 分析字符串,获取每篇新闻的发布时间,作者,来源,摄影等信息。
show = soup2.select(".show-info")[0].text.split() dt = show[0].lstrip('发布时间:') + " " + show[1] writer = show[2].lstrip('作者:') sh = show[3].lstrip('审核:') ly = show[4].lstrip('来源:') print(dt," ", writer," ", sh, " ",ly)
效果显示如下图所示:
3. 将其中的发布时间由str转换成datetime类型。
from datetime import datetime da = datetime.strptime(dt,"%Y-%m-%d %H:%M:%S") print(type(da)) print(da,writer,sh,ly) break;
效果显示如下图所示: