分词器2.0

一、normalization

1、图例

 2、Kibana

 二、字符过滤器

1、html过滤器

2、字符过滤器

3、正则过滤器

 三、令牌过滤器

1、同义词

文本指定同义词替换

PUT /test_index
{
  "settings": {
    "analysis": {
      "filter": {
        "my_synonym":{
          "type":"synonym_graph",
          "synonyms_path":"analysis/synonym.txt"    此处为自定义近义词文本
        }
      },
      "analyzer": {
        "my_analyzer":{
          "tokenizer":"ik_max_word",
          "filter":["my_synonym"]
        }
      }
    }
  }
}
GET test_index/_analyze
{
  "analyzer": "my_analyzer",
  "text":["大G，霸道，daG"]
}

 代码中自定义同义词

PUT /test_index
{
  "settings": {
    "analysis": {
      "filter": {
        "my_synonym":{
          "type":"synonym",
          "synonyms":["赵，钱，孙，李=>吴","周=>王"]
        }
      },
      "analyzer": {
        "my_analyzer":{
          "tokenizer":"standard",
          "filter":["my_synonym"]
        }
      }
    }
  }
}
GET test_index/_analyze
{
  "analyzer": "my_analyzer",
  "text":["赵，钱，孙，李","周"]
}

2、大小写

GET test_index/_analyze
{
  "tokenizer": "standard",
  "filter": ["lowercase"],
  "text":["ASADASD FDF GFDD ASDASDQW"]
}
GET test_index/_analyze
{
  "tokenizer": "standard",
  "filter": ["uppercase"],
  "text":["sadasd fdsfsg xzczxcxzc"]
}

3、停用词

PUT /test_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "type": "standard",
          "stopwords":["me"]   将指定的词语在结果中不展示
        }
      }
    }
  }
}
GET test_index/_analyze
{
  "analyzer": "my_analyzer",
  "text":["Teacher me and you in the china"]
}

 四、自定义分词器

PUT custom_analysis
{
  "settings":{
    "analysis":{
      "char_filter": {
        "my_char_filter":{
          "type":"mapping",
          "mappings":["& ==> and","| => or"]
        },
        "html_strip_char_filter":{
          "type":"html_strip",
          "escaped_tags":["a"]
        }
      },
      "filter": {
        "my_stopword":{
          "type":"stop",
          "stopwords":["is","in","the","a","at","for"]
        }
      },
      "tokenizer": {
        "my_tokenizer":{
          "type":"pattern",
          "pattern":"[ ,.!?]"
        }
      },
      "analyzer": {
        "my_analyzer":{
          "type":"custom",     type为custom使得es辨认出为自定义分词器
          "char_filter":["my_char_filter","html_strip_char_filter"],
          "filter":["my_stopword","lowercase"],
          "tokenizer":"my_tokenizer"
        }
      }
    }
  }
}
GET custom_analysis/_analyze
{
  "analyzer":"my_analyzer",
  "text":["What is ,<a>as.df</a> . ss<p> in ? &</p> | is ! in the a at for"]
}

 五、中文分词器

pluginsikconfig目录下文件描述：

自定义ik分词器词库：在IKAnalyzer.cfg.xml文件中添加新增文件

 六、热更新

由来：ik分词器在词库中添加词汇后必须重启才会生效，但是生产环境如果频繁重启会出问题，这时就需要热更新

　　　有了热更新，当我们在频繁扩展ik分词器后就不需要频繁重启es

1、基于远程词库文件

　　请求url：localhost:9200/api/hotWord?wordlib=（等于1时为新增ik词库，等于2时为停用词）

　　代码：先在application.yml文件中配置端口号

 2、基于数据库

 需要修改ik分词器源码