采用基于Java的开源搜索结果聚合引擎,Carrot2 2.0 中的后缀树算法
Carrot2 可以自动的把搜索结果归类到相应的语义类别中,这个功能是通过Carrot2一个现成的组件完成的,除此之外Carrot2 还包括了很多其他的搜索结果聚合聚类算法。
因为没有做中文分词,也没有中文的Stopword,所以我们用英文测试,实现代码
1
SnippetTokenizer snippetTokenizer = new SnippetTokenizer();
2
List<DocReference> documentReferences = new ArrayList<DocReference>();
3
List<TokenizedDocument> documents = new ArrayList<TokenizedDocument>();
4
TokenizedDocument doc = null;
5
DocReference documentReference = null;
6
7
//从搜索引擎google获取100篇数据
8
{
9
String url = "http://www.google.com/search?as_q=phone&num=100&hl=en&newwindow=1&btnG=Google+Search&as_epq=&as_oq=&as_eq=&lr=&as_ft=i&as_filetype=&as_qdr=all&as_nlo=&as_nhi=&as_occt=any&as_dt=i&as_sitesearch=&as_rights=&safe=images";
10
byte[] pageHtml = HttpUtil.getPage(url);
11
if(pageHtml == null ) return ;
12
try {
13
String strHtml = new String(pageHtml, "utf-8");
14
String[][] result = StringUtil.splitByReg(strHtml,"<td class=j>(.*?)<br>");
15
16
if(result != null)
17
{ for(int i=0;i<result.length;i++)
18
{
19
for(int j=0;j<result[i].length;j++)
20
{
21
doc = snippetTokenizer
22
.tokenize(new RawDocumentSnippet(i+"sen"+j,result[i][j].replaceAll("<[^<>]+>",""), "en"));
23
documentReference = new DocReference(doc);
24
documentReferences.add(documentReference);
25
documents.add(doc);
26
}
27
}
28
}
29
} catch (UnsupportedEncodingException e) {
30
e.printStackTrace();
31
}
32
}
33![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
34
35
//构建后缀树
36
final STCEngine stcEngine = new STCEngine(documentReferences);
37
stcEngine.createSuffixTree();
38
HashMap<String,String> defaults = new HashMap<String,String>();
39
defaults.put("lsi.threshold.clusterAssignment", "0.150");
40
defaults.put("lsi.threshold.candidateCluster", "0.775");
41
final StcParameters params = StcParameters.fromMap(defaults);
42
stcEngine.createBaseClusters(params);
43
stcEngine.createMergedClusters(params);
44![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
45
final List clusters = stcEngine.getClusters();
46
int max = params.getMaxClusters();
47![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
48
// Convert STC's clusters to the format required by local interfaces.
49
final List rawClusters = new ArrayList();
50
for (Iterator i = clusters.iterator(); i.hasNext() && (max > 0); max--)
51
{
52
final MergedCluster b = (MergedCluster) i.next();
53
final RawClusterBase rawCluster = new RawClusterBase();
54![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
55
int maxPhr = 3; // TODO: This should be a configuration parameter moved to STCEngine perhaps.
56
final List phrases = b.getDescriptionPhrases();
57
for (Iterator j = phrases.iterator(); j.hasNext() && (maxPhr > 0); maxPhr--)
58
{
59
Phrase p = (Phrase) j.next();
60
rawCluster.addLabel(p.userFriendlyTerms().trim());
61
}
62![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
63
for (Iterator j = b.getDocuments().iterator(); j.hasNext();)
64
{
65
final int docIndex = ((Integer) j.next()).intValue();
66
final TokenizedDocument tokenizedDoc = (TokenizedDocument) documents.get(docIndex);
67
final RawDocument rawDoc = (RawDocument) tokenizedDoc.getProperty(TokenizedDocument.PROPERTY_RAW_DOCUMENT);
68
rawCluster.addDocument(rawDoc);
69
}
70![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
71
rawClusters.add(rawCluster);
72
}
73
74
//得到结果,输出
75
for (Iterator iter = rawClusters.iterator(); iter.hasNext();)
76
{
77
RawCluster cluster = (RawCluster) iter.next();
78
final List phrases = cluster.getClusterDescription();
79
for(int i=0;i<phrases.size();i++)
80
System.out.print("#"+phrases.get(i));
81
System.out.println();
82
83
}
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
2
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
3
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
4
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
5
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
6
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
7
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
8
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
9
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
10
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
11
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
12
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
13
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
14
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
15
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
16
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
17
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
18
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
19
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
20
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
21
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
22
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
23
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
24
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
25
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
26
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
27
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
28
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
29
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
30
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
31
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
32
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
33
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
34
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
35
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
36
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
37
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
38
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
39
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
40
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
41
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
42
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
43
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
44
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
45
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
46
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
47
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
48
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
49
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
50
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
51
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
52
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
53
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
54
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
55
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
56
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
57
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
58
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
59
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
60
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
61
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
62
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
63
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
64
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
65
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
66
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
67
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
68
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
69
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
70
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
71
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
72
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
73
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
74
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
75
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
76
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
77
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
78
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
79
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
80
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
81
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
82
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
83
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
下面是输出聚类phone的结果,还不错
#phone
#Phone Number
#yellow pages
#mobile phone
#cell phone
#Phone Book
#area code
#Business
#services
#Wireless
#people
#directory
#telephone
#address
#online