《python网络数据采集》读后感 第八章：自然语言处理

1. 概括数据：

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
import string
import operator

def cleanInput(input):
    input = re.sub('
+', " ", input).lower()
    input = re.sub('[[0-9]*]', "", input)
    input = re.sub(' +', " ", input)
    input = bytes(input, "UTF-8")
    input = input.decode("ascii", "ignore")
    cleanInput = []
    input = input.split(' ')
    for item in input:
        item = item.strip(string.punctuation)
        if len(item) > 1 or (item.lower() == 'a' or item.lower() == 'i'):
            cleanInput.append(item)
    return cleanInput

def ngrams(input, n):
    input = cleanInput(input)
    output = {}
    for i in range(len(input)-n+1):
        ngramTemp = " ".join(input[i:i+n])
        if ngramTemp not in output:
                 output[ngramTemp] = 0
        output[ngramTemp] += 1
    return output
    content = str(
           urlopen("http://pythonscraping.com/files/inaugurationSpeech.txt").read(),
                   'utf-8')
    ngrams = ngrams(content, 2)
    sortedNGrams = sorted(ngrams.items(), key = operator.itemgetter(1), reverse=True)
    print(sortedNGrams)

def isCommon(ngram):
    commonWords = ["the", "be", "and", "of", "a", "in", "to", "have", "it",
             "i", "that", "for", "you", "he", "with", "on", "do", "say", "this",
             "they", "is", "an", "at", "but","we", "his", "from", "that", "not",
             "by", "she", "or", "as", "what", "go", "their","can", "who", "get",
             "if", "would", "her", "all", "my", "make", "about", "know", "will",
             "as", "up", "one", "time", "has", "been", "there", "year", "so",
             "think", "when", "which", "them", "some", "me", "people", "take",
             "out", "into", "just", "see", "him", "your", "come", "could", "now",
             "than", "like", "other", "how", "then", "its", "our", "two", "more",
             "these", "want", "way", "look", "first", "also", "new", "because",
             "day", "more", "use", "no", "man", "find", "here", "thing", "give",
             "many", "well"]
    for word in ngram:
        if word in commonWords:
            return True
    return False

2. 马尔可夫模型：

对一个天气系统建立马尔可夫模型：

在这个天气系统模型中，如果今天是晴天，那么明天有 70% 的可能是晴天，20% 的可能 多云，10% 的可能下雨。如果今天是下雨天，那么明天有 50% 的可能也下雨，25% 的可 能是晴天，25% 的可能是多云。

3. 广度优先搜索算法：

广度优先搜索算法的思路是优先搜寻直接连接到起始页的所有链接(而不是找到一个链接 就纵向深入搜索)。如果这些链接不包含目标页面(你想要找的词条)，就对第二层的链 接——连接到起始页的页面的所有链接——进行搜索。这个过程不断重复，直到达到搜索 深度限制或者找到目标页面为止。

4.自然语言工具包：

自然语言工具包(Natural Language Toolkit，NLTK)就是这样一个 Python 库，用于识别和 标记英语文本中各个词的词性(parts of speech)。