首先介绍一下 CoreSeek/Sphinx的发布包
indexer: 用于创建全文索引;
search: 一个简单的命令行(CLI) 的测试程序，用于测试全文索引;
searchd: 一个守护进程，其他软件（例如WEB程序）可以通过这个守护进程进行全文检索;
sphinxapi: 一系列searchd 的客户端API 库，用于流行的Web脚本开发语言(PHP, Python, Perl, Ruby, Java).
spelldump: 一个简单的命令行工具，用于从 ispell 或者 MySpell (OpenOffice内置绑定) 格式的字典中提取词条。当使用 wordforms时可用这些词条对索引进行定制.
indextool: 工具程序，用来转储关于索引的多项调试信息。此工具是从版本Coreseek 3.1(Sphinx 0.9.9-rc2)开始加入的。
mmseg: 工具程序和库，Coreseek用于提供中文分词和词典处理。
在Sphinx+LibMMSeg搭建中文全文搜索引擎_安装配置中安装试验了Sphinx 的使用，但是还有几方面的问题有待处理。
- 动态增量索引
- 区段查询
- 实时索引
- 匹配到的字段在前端界面表现
本篇主要是对动态增量更新的一些研究
在利用 Sphinx 做搜索引擎的时候，一般他的索引建立构成有如下几个部分：
- 固定不变的主索引
- 增量索引重建
- 索引数据合并
- 在实际操作中，需要为增量索引的建立创建辅助表，这样才可以记住最后建立索引的记录ID,做实际的增量部分的索引建立。
CREATE TABLE `sph_counter` ( `counter_id` int(11) NOT NULL, `max_doc_id` int(11) NOT NULL, PRIMARY KEY (`counter_id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8
- 在主索引的数据源中作如下方式的取数据设置
source src { # data source type. mandatory, no default value # known types are mysql, pgsql, mssql, xmlpipe, xmlpipe2, odbc type = mysql sql_host = localhost sql_user = root sql_pass = xxxxx sql_db = test sql_port = 3306 # optional, default is 3306 sql_query_pre = SET NAMES utf8 sql_query_pre = SET SESSION query_cache_type=OFF sql_query_pre = REPLACE INTO sph_counter SELECT 1, MAX(id) FROM cn sql_query = SELECT id,title,content from cn where id<=(SELECT max_doc_id FROM sph_counter WHERE counter_id=1) sql_query_info = SELECT * FROM cn WHERE id=$id }
- 在增量索引的数据源中作如下方式的取数据设置，需要注意的是sql_query_pre要和主索引数量相同，不然查询结果不是想要的内容
#表示增量数据源 source moresrc : src { sql_query_pre = SET NAMES utf8 sql_query_pre = SET SESSION query_cache_type=OFF sql_query = SELECT id,title,content from cn where id>(SELECT max_doc_id FROM sph_counter WHERE counter_id=1) #sql_ranged_throttle = 100 }
- 主索引index定义配置
index src { source = src path = /usr/local/coreseek/var/data/test1 docinfo = extern mlock = 0 morphology = none # 启用中文分词功能source 数据源中需要 设置读取的数据编码字符集为UTF-8，否则无 法正确处理；如果是xml，则正确输出为UTF-8编码格式即可；如果是MySQL，则设置读取数据输出 字符集为UTF-8即可 charset_type = zh_cn.utf-8 # 中文分词词库位置 charset_dictpath =/usr/local/mmseg/etc/ }
- 增量索引index定义配置
index moresrc : src { source = moresrc path = /usr/local/coreseek/var/data/moresrc morphology = stem_en } indexer { # memory limit, in bytes, kiloytes (16384K) or megabytes (256M) # optional, default is 32M, max is 2047M, recommended is 256M to 1024M mem_limit = 32M }
- 创建更新全部索引
root@timeless-HP-Pavilion-g4-Notebook-PC:/usr/local/coreseek/bin# /usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/csft.increment.conf --all --rotate Coreseek Fulltext 4.1 [ Sphinx 2.0.2-dev (r2922)] Copyright (c) 2007-2011, Beijing Choice Software Technologies Inc (http://www.coreseek.com) using config file '/usr/local/coreseek/etc/csft.increment.conf'... indexing index 'src'... WARNING: Attribute count is 0: switching to none docinfo collected 6 docs, 0.0 MB sorted 0.0 Mhits, 100.0% done total 6 docs, 293 bytes total 0.011 sec, 25970 bytes/sec, 531.82 docs/sec indexing index 'moresrc'... WARNING: Attribute count is 0: switching to none docinfo collected 0 docs, 0.0 MB total 0 docs, 0 bytes total 0.001 sec, 0 bytes/sec, 0.00 docs/sec total 3 reads, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg total 11 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg rotating indices: succesfully sent SIGHUP to searchd (pid=13100).
- 更新增量索引
数据库中添加数据 然后运行，更新增量索引 记得 执行 indeser search 还有启动searchd时候要 要指定-c 配置文件。
root@timeless-HP-Pavilion-g4-Notebook-PC:/usr/local/coreseek/bin# /usr/local/coreseek/bin/indexer moresrc -c /usr/local/coreseek/etc/csft.increment.conf --rotate Coreseek Fulltext 4.1 [ Sphinx 2.0.2-dev (r2922)] Copyright (c) 2007-2011, Beijing Choice Software Technologies Inc (http://www.coreseek.com) using config file '/usr/local/coreseek/etc/csft.increment.conf'... indexing index 'moresrc'... WARNING: Attribute count is 0: switching to none docinfo collected 1 docs, 0.0 MB sorted 0.0 Mhits, 100.0% done total 1 docs, 30 bytes total 0.008 sec, 3457 bytes/sec, 115.24 docs/sec total 2 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg total 6 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg rotating indices: succesfully sent SIGHUP to searchd (pid=13100).
- 合并增量索引到主索引
root@timeless-HP-Pavilion-g4-Notebook-PC:/usr/local/coreseek/bin# /usr/local/coreseek/bin/indexer --merge src moresrc -c /usr/local/coreseek/etc/csft.increment.conf --rotate Coreseek Fulltext 4.1 [ Sphinx 2.0.2-dev (r2922)] Copyright (c) 2007-2011, Beijing Choice Software Technologies Inc (http://www.coreseek.com) using config file '/usr/local/coreseek/etc/csft.increment.conf'... read 0.0 of 0.0 MB, 100.0% done merged 0.0 Kwords merged in 0.000 sec total 40 reads, 0.000 sec, 6.4 kb/call avg, 0.0 msec/call avg total 5 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg rotating indices: succesfully sent SIGHUP to searchd (pid=13100).
- 测试数据如下所示
root@timeless-HP-Pavilion-g4-Notebook-PC:/usr/local/coreseek/bin# ./search -c /usr/local/coreseek/etc/csft.increment.conf 测试 Coreseek Fulltext 4.1 [ Sphinx 2.0.2-dev (r2922)] Copyright (c) 2007-2011, Beijing Choice Software Technologies Inc (http://www.coreseek.com) using config file '/usr/local/coreseek/etc/csft.increment.conf'... index 'src': query '测试 ': returned 3 matches of 3 total in 0.000 sec displaying matches: 1. document=2, weight=2637 id=2 title=????? content=?? ?????? ???? ?? addtime=0 2. document=7, weight=2627 id=7 title=?? content=???????? addtime=0 3. document=1, weight=1571 id=1 title=??? content=???????? ?????? ???? addtime=1444444444 words: 1. '测试': 3 documents, 14 hits index 'moresrc': query '测试 ': returned 1 matches of 1 total in 0.000 sec displaying matches: 1. document=7, weight=2500 id=7 title=?? content=???????? addtime=0 words: 1. '测试': 1 documents, 5 hits
通过测试数据看到 index moresrc 增量索引 查询到 “测试” 关键字 5 次 合并之后的住索引变为 14个匹配。 之前是 9个 匹配。