网络爬虫速成指南（二）网页解析(基于模板)

网页解析技术：
1 xpath教程
2 正则表达式教程

xpath是将html加载为DOM树解析，简单，易维护。
通常我用正则作为辅助抽取，用xpath定位后，再从定位的数据中用正则抽取。

xpath的类库：
.net 方向主要用到HtmlAgilityPack
java 方向主要用到HtmlCleaner(得翻墙） jsoup

以下是本人封装好的：HtmlParser
使用示例：

            HtmlParser<HotelInfo> parser = new HtmlParser<HotelInfo>();
            ParseConfig config = new ParseConfig("save_xc.xml");
            String html = File.ReadAllText("xiecheng.txt", Encoding.GetEncoding("GBK"));
            HotelInfo entity = parser.GetEntity(html, config);

模板样例1：

<?xml version="1.0" encoding="utf-8"?>
<template>
   <page>
       <save root=".">
           <field>
               <name>Title</name>
               <xpath>//div[@id='J_Article_Wrap']//h1</xpath>
           </field>
           <field>
               <name>PubTime</name>
               <xpath>//*[@id='pub_date']</xpath>
               <regex>
                   <pattern>(d+)年(d+)月(d+)日</pattern>
                   <format>{0}-{1}-{2}</format>
               </regex>
           </field>
           <field>
               <name>Article</name>
               <xpath>//*[@id="artibody"]</xpath>
           </field>
       </save>
   </page>
</template>
模板样例2：

<?xml version="1.0" encoding="utf-8"?>
<template>
   <page>
       <save_m root="//tr[@id]">
           <field>
               <name>Price</name>
               <xpath>./td[@class='price']</xpath>
           </field>
       </save_m>
   </page>
</template>

模板样例3：

<?xml version="1.0" encoding="utf-8"?>
<template>
   <page>
       <save root=".">
           <field>
               <name>Name</name>
               <xpath>//h1</xpath>
           </field>
           <field>
               <name>EngName</name>
               <xpath>//div[@class='name']/h2</xpath>
           </field>
           <field>
               <name>Star</name>
               <xpath>//div[@class='grade']/span/@title</xpath>
           </field>
           <field>
               <name>Address</name>
               <xpath>//div[@class='adress']</xpath>
           </field>
           <field>
               <name>Description</name>
               <xpath>//*[@id="htlDes"]</xpath>
           </field>
           <field>
               <name>Facility</name>
               <xpath position='outerhtml'>//div[@class="htl_info_table "]</xpath>
           </field>
           <field>
               <name>Policy</name>
               <xpath position='outerhtml'>//div[@class='detail_main']/div[@class="htl_info_table"]</xpath>
           </field>
           <field>
               <name>Traffic</name>
               <xpath position='outerhtml'>//div[@class='transSub'][1]/div[@class="htl_info_table"]</xpath>
           </field>
           <field>
               <name>Nearby</name>
               <xpath position='outerhtml'>//div[@class='transSub'][2]/div[@class="htl_info_table"]</xpath>
           </field>
       </save>
   </page>
</template>