Elasticsearch07-分词器

分词器作用

1.切词
2.提升召回率：能搜索到的结果的比率

分析器

character filter：分词之前预处理（过滤无用字符、标签等，转换一些&=>and 《Elasticsearch》=> Elasticsearch

# html标标签解析
PUT index_my_char_filter_html
{
  "settings": {
    "analysis": {
      "char_filter": {
        "my_char_filter": {
          "type": "html_strip",
          "escaped_tags": ["a"] # a标签不用解析，escaped_tags：需要保留的html标签
        }
      },
      "analyzer": {
        "my_analyzerr_html": {
          "tokenizer": "keyword",
          "char_filter": ["my_char_filter"]
        }
      }
    }
  }
}

验证：
POST index_my_char_filter_html/_analyze
{
  "analyzer": "my_analyzerr_html",
  "text": "<p>I&apos;m so <a>happy</a>!</p>"
}



#filter mapping
PUT index_my_char_filter_mapping
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "keyword",
          "char_filter": [
            "analyzer_mapping"
          ]
        }
      },
      "char_filter": {
        "analyzer_mapping": {
          "type": "mapping",
          "mappings": [
            "٠ => 0",
            "١ => 1",
            "٢ => 2",
            "٣ => 3",
            "٤ => 4",
            "٥ => 5",
            "٦ => 6",
            "٧ => 7",
            "٨ => 8",
            "٩ => 9"
          ]
        }
      }
    }
  }
}


验证：
POST index_my_char_filter_mapping/_analyze
{
  "analyzer": "my_analyzer",
  "text": "My license plate is ٢٥٠١٥"
}


#正则
PUT my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "standard",
          "char_filter": ["my_char_filter"]
        }
      },
      "char_filter": {
        "my_char_filter": {
          "type": "pattern_replace",
          "pattern": "(\d+)-(?=\d)",
          "replacement": "$1_"
        }
      }
    }
  }
}

验证:
POST my_index/_analyze
{
  "analyzer": "my_analyzer",
  "text": "My credit card is 123-456-789"
}

tokenizer（分词器）：分词

GET _analyze
{
  "tokenizer" : "standard", #默认分词器
  "filter" : ["lowercase"],
  "text" : "THE Quick FoX JUMPs"
}

token filter：停用词、时态转换、大小写转换、同义词转换、语气词处理等。比如：has=>have him=>he apples=>apple the/oh/a=>干掉

GET /_analyze
{
  "tokenizer": "standard",
  "filter": [
    {
      "type": "condition",
      "filter": [ "lowercase" ],
      "script": {
        "source": "token.getTerm().length() < 5"
      }
    }
  ],
  "text": "THE QUICK BROWN FOX"
}

#停用词 stopwords token filter
PUT /index_stopword
{
  "settings": {
    "analysis": {
      "analyzer": {
        "index_stopword_analyzer":{
          "type":"standard",
          "stopwords":"_english_"
        }
      }
    }
  }
}

GET index_stopword/_analyze
{
  "analyzer": "index_stopword_analyzer",
  "text": "Teacher Ma is in the restroom"
}

分词器：analyzer

standard：默认分词器，中文支持的不理想，会逐字拆分。

GET _analyze
{
  "tokenizer" : "standard",
  "text" : "THE Quick FoX JUMPs"
}

Pattern Tokenizer：以正则匹配分隔符，把文本拆分成若干词项。

Simple Pattern Tokenizer：以正则匹配词项，速度比Pattern Tokenizer快。

GET /index_stopword/_analyze
{
  "text": "江山如此多娇,小姐姐哪里可以撩",
  "analyzer": "simple"
}

whitespace：以空白符分隔

自定义分词器

PUT /index_custom
{
  "settings": {
    "analysis": {
      "char_filter": {
        "test_char_filter": {
          "type": "mapping",
          "mappings": [
            "& => and",
            "| => or"
          ]
        }
      },
      "filter": {
        "test_stopwords": {
          "type": "stop",
          "stopwords": ["is","in","at","the","a","for"]
        }
      },
      "tokenizer": {
        "punctuation": { 
          "type": "pattern",
          "pattern": "[ .,!?]"
        }
      },
      "analyzer": {
        "index_custom_analyzer": {
          "type": "custom",
          "char_filter": [
            "html_strip",
            "test_char_filter"
          ],
          "tokenizer": "standard",
          "filter": ["lowercase","test_stopwords"]
        }
      }
    }
  }
}

GET /index_custom/_analyze
{
  "text": "Teacher ma & zhang also thinks [mother's friends] is good | nice!!!",
  "analyzer": "index_custom_analyzer"
}

创建mapping时候指定分词器

#创建mapping时候指定分词器
PUT /index_create_mapping_analuzerc/
{
  "mappings": {
    "properties": {
      "content": {
        "type": "text",
        "analyzer": "standard"
      }
    }
  }
}


#创建mapping字段时候指定分词器
PUT /index_create_mapping_analuzerb/_mapping
{
  "properties": {
    "column": {
      "type": "text",
     "analyzer": "ik_max_word"
    }
  }
}

中文分词器

PUT index_ik_max
{
  "settings": {
    "analysis": {
      "analyzer": {
        "default": {
          "type": "ik_max_word"
        }
      }
    }
  }
}
PUT /index_ik_smart
{
  "mappings": {
      "properties": {
        "text": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_smart"
        }
    }
  }
}

GET /index_ik_max/_analyze
{
  "text": "中华人民共和国国歌",
  "analyzer": "ik_max_word"
}
GET /index_ik_smart/_analyze
{
  "text": "中华人民共和国国歌",
  "analyzer": "ik_smart"
}

安装Ik分词器

1.https://github.com/medcl/elasticsearch-analysis-ik/releases
2.Releases -> Assets -> 下载zip文件
3.es的/plugin创建/ik目录
4.将文件解压到ik目录下
5.将mysql驱动包放到ik目录下
6.重新启动

热加载ik分词器

1.下载https://github.com/medcl/elasticsearch-analysis-ik
2.打包
3.获取jar包: /target/releases/elasticsearch-analysis-ik-7.6.2.zip
4.es的/plugin创建/ik目录
5.将文件elasticsearch-analysis-ik-7.6.2.zip解压在ik目录下
6.将mysql驱动包放到ik目录下
7.重新启动

备注： 修改代码目录：dic/Dictionary initial方法，这里面有对应的主词库、停用词修改代码。
原理： 
1.es启动的时候启动maven项目
2.maven项目中写个定时器，定时去mysql中查询数据
3.将查询的数据临时加载进内存，也就是es启动时候加载这些主词、停用词文件代码的地方

ik分词器注意点

1.es版本、kibana版本、ik分词版本你要一致
2.三个最新版本不同步，一般不要下载最新版本