BeautifulSoup

BeautifulSoup解析器
解析器 使用方法 条件
bs4的HTML解析器 BeautifulSoup(mk,'html.parser') 安装bs库
lxml的HTML解析器 BeautifulSoup(mk,'lxml') pip install lxml
lxml的xml解析器 BeautifulSoup(mk,'xml') pip install lxml
html5lib的解析器 BeautifulSoup(mk,'html5lib') pip install html5lib
BeautifulSoup基本元素
Tag 标签，最基本的信息组织单元，分别用<>和</>标明开头和结尾
Name 标签的名字，<p></p>的名字是'p',格式:<tag>.name
Attributes 标签的属性，字典形式组织，格式：<tag>.attrs
NavigableString 标签内非属性字符串，<>...</>中字符串，格式：<tag>.string
comment 标签内字符串的注释部分，一种特殊的comment（注释）类型

示例：

# url = 'http://python123.io/ws/demo.html'   网页HTML代码如下：
<html><head><title>This is a python demo page</title></head>
<body>
<p class="title"><b>The demo python introduces several python courses.</b></p>
<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p>
</body></html>

 a标签的名字:

from bs4 import BeautifulSoup
import requests

url = 'http://python123.io/ws/demo.html'
demo = requests.get(url)
html = demo.text

soup = BeautifulSoup(html,'html.parser')
tag = soup.a

print(tag.name) 
运行结果：
a

a标签的父亲标签名字：

from bs4 import BeautifulSoup
import requests

url = 'http://python123.io/ws/demo.html'
demo = requests.get(url)
html = demo.text

soup = BeautifulSoup(html,'html.parser')
tag = soup.a.parent

print(tag.name)
运行结果:
p

a标签的属性：

from bs4 import BeautifulSoup
import requests

url = 'http://python123.io/ws/demo.html'
demo = requests.get(url)
html = demo.text

soup = BeautifulSoup(html,'html.parser')
tag = soup.a

print(tag.attrs)　　　　　　　　　　# 打印a标签的属性，得到字典
print(tag.attrs['class'])　　　　 # 打印a标签的class属性值
print(tag.attrs['href'])         # 打印a标签的href属性值
print(type(tag.attrs))　　　　　　# 打印a标签属性类型
print(type(tag]))　　　　　　　　　　# 打印标签类型
print(tag.string)　　　　　　　　　　# 打印a标签的string
运行结果：

{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}

['py1']

['http://www.icourse163.org/course/BIT-268001']

<class 'dict'>

<class 'bs4.element.Tag'>

Basic Python

标签树的下行遍历
属性 说明
.contents 子节点的列表，将<tag>所有儿子节点存入列表
.children 子节点的迭代类型，与.contents类似，用于循环遍历儿子节点
.descendants 子孙节点的迭代类型，包含所有子孙节点，用于遍历循环

示例：

from bs4 import BeautifulSoup
import requests

url = 'http://python123.io/ws/demo.html'
demo = requests.get(url)
html = demo.text

soup = BeautifulSoup(html,'html.parser')
print(soup.head)                    # 获取head标签节点
print('')
print(soup.head.contents)           # 获取head的所有节点
print('')
print(soup.body.contents)           # 获取body的所有节点
print('')
print(str(len(soup.body.contents))) # 获取body的所有节点数量
print('')
print(soup.body.contents[1])        # 获取body标签中的第2个节点
运行结果：

<head><title>This is a python demo page</title></head>



[<title>This is a python demo page</title>]




['
', <p class="title"><b>The demo python introduces several python courses.</b></p>, '
', <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>, '
']




5




<p class="title"><b>The demo python introduces several python courses.</b></p>


 




标签数的下行遍历







for child in soup.body.children:
    print(child)




遍历儿子节点








for child in soup.body.descendants:
    print(child)




遍历子孙节点






标签树的上行遍历



.parent
节点的父亲标签




.parents
节点的先辈标签的迭代类型，用于循环先辈节点









from bs4 import BeautifulSoup
import requests

url = 'http://python123.io/ws/demo.html'
demo = requests.get(url)
html = demo.text

soup = BeautifulSoup(html,'html.parser')
''' 标签树上行遍历'''
for parent in soup.a.parents:
    if parent is None:
        print(parent)
    else:
        print(parent.name)

运行结果：
p
body
html
[document]





标签树的平行遍历（必须发生在同一个父亲节点下）



属性
说明




.next_sibling
返回按照HTML文本顺序的下一个平行节点标签




.previous_sibling
返回按照HTML文本舒徐的上一个平行节点标签




.next_siblings
迭代类型，返回按照HTML文本顺序的后续所有平行节点标签




.previous_siblings
迭代类型，返回按照HTML文本顺序的前续所有平行节点标签






标签树的平行遍历







for sibling in next_siblings:
    print(siblink)




遍历后续节点








for sibling in previous_siblings:
    print(siblink)




遍历前续节点














from bs4 import BeautifulSoup
import requests

url = 'https://python123.io/ws/demo.html'
r = requests.get(url)
getHTML = r.text
soup = BeautifulSoup(getHTML,"html.parser")
print(soup.prettify())






D:python_workvenvScriptspython.exe D:/python_work/test.py
<html>
 <head>
  <title>
   This is a python demo page
  </title>
 </head>
 <body>
  <p class="title">
   <b>
    The demo python introduces several python courses.
   </b>
  </p>
  <p class="course">
   Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
   <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
    Basic Python
   </a>
   and
   <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
    Advanced Python
   </a>
   .
  </p>
 </body>
</html>

Process finished with exit code 0







from bs4 import BeautifulSoup
import requests

url = 'https://python123.io/ws/demo.html'
r = requests.get(url)
getHTML = r.text
soup = BeautifulSoup(getHTML,"html.parser")

# 提取a标签的属性href的值
for link in soup.find_all('a'):　　
    print(link.get('href'))


运行结果：
D:python_workvenvScriptspython.exe D:/python_work/test.py
http://www.icourse163.org/course/BIT-268001
http://www.icourse163.org/course/BIT-1001870001

Process finished with exit code 0








        

	      

	    

	  

	  

	    
      
        【推广】
        免费学中医，健康全家人
      
	  

    

    

          原文地址：https://www.cnblogs.com/leisurelyRD/p/12364834.html