VIPS: a VIsion based Page Segmentation Algorithm

VIPS: a VIsion based Page Segmentation Algorithm

VIPS: a VIsion based Page Segmentation Algorithm

Introduction

The VIsion-based Page Segmentation (VIPS) algorithm aims to extract the semantic structure of a web page based on its visual presentation. Such semantic structure is a tree structure; each node in the tree corresponds to a block. Each node will be assigned a value (Degree of Coherence) to indicate how coherent of the content in the block based on visual perception, the bigger is the DoC value, the more coherent is the block. The VIPS algo-rithm makes full use of page layout structure. It first extracts all the suitable blocks from the html DOM tree, and then it finds the separators between these blocks. Here, separators denote the hori-zontal or vertical lines in a web page that visually cross with no blocks. Based on these separators, the semantic tree of the web page is constructed. Thus, a web page can be represented as a set of blocks (leaf nodes of the semantic tree). Compared with DOM based methods, the segments obtained by VIPS are much more semantically aggregated. Noisy information, such as navigation, advertisement, and decoration can be easily removed because they are often placed in certain positions of a page. Contents with different topics are distinguished as separate blocks.

 

Paper List

Original Paper

Applications using VIPS

If you find the VIPS algoirthm useful, we appreciate it very much if you can cite our following works:

@Inproceedings{CHWM04
author = "Deng Cai and Xiaofei He and Ji-Rong Wen and Wei-Ying Ma",
title = "Block-level link analysis",
booktitle = "Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR'04)",
pages = {440--447},
year = "2004"}

@Inproceedings{CYWM04
author = "Deng Cai and Shipeng Yu and Ji-Rong Wen and Wei-Ying Ma",
title = "Block-based web search",
booktitle = "Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR'04)",
pages = {456--463},
year = "2004"}

@Inproceedings{YCWM03,
author = "Shipeng Yu and Deng Cai and Ji-Rong Wen and Wei-Ying Ma",
title = "Improving Pseudo-Relevance Feedback in Web Information Retrieval Using Web Page Segmentation",
booktitle = "Twelfth International World Wide Web Conference (WWW2003)",
year = "2003"}

@Inproceedings{CYWM03,
author = "Deng Cai and Shipeng Yu and Ji-Rong Wen and Wei-Ying Ma",
title = "Extracting Content Structure for Web Pages based on Visual Representation",
booktitle = "Fifth Asia Pacific Web Conference (APWeb2003)",
year = "2003"}

 

Demo

Copyright Notice: All these programs can only be used for research.

VIPS dll (The VIPS DLL is always under development. All versions are downloadable here.)

  • VIPS dll (pageanalyzer.dll) (release date: 03/26/2008. One bug fixed. Thanks Ankur Gupta for pointing out the bug.)

     

  • VIPS dll (pageanalyzer.dll) (release date: 01/16/2006. Some people requested for the HTML source code output, I added it. Also I changed some interfaces so you need to rebuild your program if you want to use this new dll. Meanwhile, please download the newest demo.)

     

  • VIPS Demo (release date: 01/16/2006) (You should download VIPS dll and register it first! This demo can only work on the new VIPS dll)

     

  • VIPS dll (pageanalyzer.dll) (release date: 03/20/2005, some bugs fixed)

     

  • VIPS dll (pageanalyzer.dll) (release date: 08/20/2004)

     

  • VIPS Demo (release date: 08/20/2004) (You should download VIPS dll and register it first!)

How to use VIPS dll.

  • You should familiar with how to host a webbrowser(Internet Explorer) in your program. Some articles in MSDN are very useful.

     

  • A more powerful example of using VIPS dll in VS2003 (release date: 01/25/2006)
    (This example provides source code on how to process batch job using VIPS dll. The framework of this example is based on MFCbrowser, which is a demo project in MSDN. You only need to focus on the MFCbrowserView.cpp and MFCbrowserView.h. I added some comments and hopefully these two files are self explained. Email me if you still have any questions.)

     

  • A example of using VIPS dll in VC6.0 (release date: 08/20/2004) (You should download VIPS dll and register it first!)

Notice: we are currently working to enhance the VIPS algorithm, any suggestions or problems can be send to dengcai2 AT cs DOT uiuc DOT edu.

【推广】 免费学中医，健康全家人
原文地址：https://www.cnblogs.com/lexus/p/3600470.html