Beautifulsoup

Beautiful Soup：解析HTML页面信息标记与提取方法

获取网页源代码

import requests
from bs4 import BeautifulSoup

kv = {'user-agent':'Mozilla/5.0'}
url = "https://python123.io/ws/demo.html"
r = requests.get(url,headers = kv)
print(r.status_code)
demo = r.text
soup = BeautifulSoup(demo,"html.parser")#解析
print(soup.prettify())

200
<html><head><title>This is a python demo page</title></head>
<body>
The demo python introduces several python courses.
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.
</body></html>

<html>
<head>
<title>
This is a python demo page
</title>
</head>
<body>


The demo python introduces several python courses.



Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
Basic Python
</a>
and
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
Advanced Python
</a>
.

</body>
</html>

BeautifulSoup的使用

BeautifulSoup库解析器

BeautifulSoup类的基本元素

https://python123.io/ws/demo.html

import requests
from bs4 import BeautifulSoup
kv = {'user-agent':'Mozilla/5.0'}
url = "https://python123.io/ws/demo.html"
r = requests.get(url,headers = kv)
print(r.status_code)
demo = r.text
soup = BeautifulSoup(demo,"html.parser")
print(soup.title)
tag = soup.a#只能返回一个a标签
print(tag)

200
<title>This is a python demo page</title>
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>

print(soup.a.name)
print(soup.a.parent.name)
print(soup.a.parent.parent.name)

a
p
body

print(tag.attrs['href'])
print(type(tag.attrs))字典
print(type(tag))

{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}
http://www.icourse163.org/course/BIT-268001
<class 'dict'>
<class 'bs4.element.Tag'>

tag = soup.a
print(tag)
print(tag.string)
tag1 = soup.p
print(tag1)
print(tag1.string)
tag2  = soup.b
print(tag2)
print(tag2.string)

<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
Basic Python
The demo python introduces several python courses.
The demo python introduces several python courses.
The demo python introduces several python courses.
The demo python introduces several python courses.

print(type(tag2.string))

soup = BeautifulSoup("<b><!--This is a comment--></b><p>This is not a comment</p>","html.parser")
print(soup.b.string)
print(type(soup.b.string))
print(soup.p.string)
print(type(soup.p.string))

This is a comment
<class 'bs4.element.Comment'>
This is not a comment
<class 'bs4.element.NavigableString'>

基于bs4库的HTML内容遍历方法

上述的标签树如下

三种遍历方式

下行遍历

soup = BeautifulSoup(demo,"html.parser")
print(soup.head)
print(soup.head.contents)
print(soup.body.contents)返回列表
print(len(soup.body.contents))
print(soup.body.contents[1])

<head><title>This is a python demo page</title></head>
[<title>This is a python demo page</title>]
[' ', The demo python introduces several python courses., ' ', Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the foll
owing courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>., ' ']
5
The demo python introduces several python courses.

for child in soup.body.children:
      print(child) # 遍历儿子节点

The demo python introduces several python courses.

Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.

for child in soup.body.descendants:
     print(child) # 遍历子孙节点

The demo python introduces several python courses.
The demo python introduces several python courses.
The demo python introduces several python courses.

<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
Basic Python
and
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>
Advanced Python
.


上行遍历

for parent in soup.a.parents: # 遍历soup的a标签的先辈标签
   if parent is None:
       print(parent)
   else:
       print(parent.name)

p
body
html
[document]

soup = BeautifulSoup(demo,"html.parser")
tag = soup.a
print(tag)
print(tag.parent)

<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.

强调：soup.html的parent是它本身 soup.parent是空的

for parent in soup.a.parents: # 遍历soup的a标签的先辈标签
   if parent is None:
       print( parent)
   else:
       print(parent.name)

p
body
html
[document]

平行遍历

平行遍历发生在同一父节点的各节点间

标签间的NavigableString也会构成标签树的节点，那么某个节点的父节点、子节点或者平行标签都有可能是NavigableString类型的

soup = BeautifulSoup(demo,"html.parser")
tag = soup.a
print(tag.next_sibling)
print(tag.next_sibling.next_sibling)
print(tag.previous_sibling)
print(tag.previous_sibling.previous_sibling)

print(tag.parent)

and
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>
.

Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:

基于bs4库的HTML格式输出

soup = BeautifulSoup(demo,"html.parser")
print(soup.prettify())#在每个标签后面加了一个换行符，便于美观的输出

bs4的编码默认都为utf-8编码

soup = BeautifulSoup("<p>你好</p>","html.parser")
print(soup.p.string)
print(soup.p.prettify())