使用java操作elasticsearch(1)

1.安装elasticsearch

　　这儿用的是5.6.9的版本，下载安装过程较为简单,在官网上下载好后解压到文件夹。需要注意的是在elasticsearch-5.6.9config下的elasticsearch.yml文件夹中

　　配置data与log的地址，自己创建文件夹后如下配置

 1 # ----------------------------------- Paths ------------------------------------
 2 #
 3 # Path to directory where to store the data (separate multiple locations by comma):
 4 #
 5 path.data: D:/java/soft/elasticsearch/dev/data
 6 #
 7 # Path to log files:
 8 #
 9 path.logs: D:/java/soft/elasticsearch/dev/log
10 #

　启动elasticsearch 点击elasticksearch.bat即可启动注意电脑上要安装1.8以上的jdk

出现以下消息则说明启动成功

上面红圈的9300端口则是与es进行数据交互时的端口，下面的9200则是查看数据的端口，ip为安装es的ip

启动项目后，访问http://ip:9200 ip是安装elasticsearch服务的机器ip，如果出现以下信息即代表安装成功

 1 {
 2     "name": "kSs9lbx",
 3     "cluster_name": "elasticsearch",
 4     "cluster_uuid": "Wu6n-WzvQuecDEMWspm_aA",
 5     "version": {
 6         "number": "5.6.9",
 7         "build_hash": "877a590",
 8         "build_date": "2018-04-12T16:25:14.838Z",
 9         "build_snapshot": false,
10         "lucene_version": "6.6.1"
11     },
12     "tagline": "You Know, for Search"
13 }

2.创建java的springboot工程

　　2.1pom.xml文件依赖

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.2.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <java.version>1.8</java.version>
    <fastjson.version>1.2.31</fastjson.version>
    <jackson.version>2.8.7</jackson.version>
</properties>

<dependencies>
    <!-- set jetty server -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-tomcat</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-configuration-processor</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-jetty</artifactId>
    </dependency>
    <!-- aop -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-aop</artifactId>
    </dependency>
    <!-- json处理 -->
    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>fastjson</artifactId>
        <version>${fastjson.version}</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/net.sourceforge.javacsv/javacsv -->
    <dependency>
        <groupId>net.sourceforge.javacsv</groupId>
        <artifactId>javacsv</artifactId>
        <version>2.0</version>
    </dependency>
    <!-- jackson json begin -->
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.module</groupId>
        <artifactId>jackson-module-jaxb-annotations</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <!-- jackson json end -->
    <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.3.2</version>
    </dependency>
    <!-- druid datasource -->
    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>druid</artifactId>
        <version>1.0.27</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-validation</artifactId>
    </dependency>
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.4</version>
    </dependency>
    <dependency>
        <groupId>com.github.theborakompanioni</groupId>
        <artifactId>thymeleaf-extras-shiro</artifactId>
        <version>${thymeleaf-shiro.version}</version>
    </dependency>
    <!-- mysql驱动 -->
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <scope>runtime</scope>
    </dependency>
    <!-- spring boot 开发工具 -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <optional>true</optional>
    </dependency>
    <!-- 测试 -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <!--elasticsearrch-->
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>5.6.9</version>
    </dependency>

    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>transport</artifactId>
        <version>5.6.9</version>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>

</dependencies>

<repositories>
    <repository>
        <id>aliyun</id>
        <name>aliyun</name>
        <url>http://maven.aliyun.com/nexus/content/groups/public</url>
    </repository>
</repositories>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <includeSystemScope>true</includeSystemScope>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.18.1</version>
            <configuration>
                <skipTests>true</skipTests>
            </configuration>
        </plugin>
    </plugins>
</build>

　　2.2application.yml文件配置

server:
  port: 9999
es:
  host: 127.0.0.1
  port: 9300
spring:
  datasource:
    initialize: false #false时不进行数据库表结构和数据初始化
    username: root
    #password: root
    password: root
    url: jdbc:mysql://localhost:3306/estest?createDatabaseIfNotExist=true&autoReconnect=true&useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true
    type: com.alibaba.druid.pool.DruidDataSource
    #连接池配置
    driverClassName: com.mysql.jdbc.Driver
    # 初始化大小，最小，最大
    initialSize: 5
    minIdle: 5
    maxActive: 50
    # 配置获取连接等待超时的时间
    maxWait: 60000
    # 配置间隔多久才进行一次检测，检测需要关闭的空闲连接，单位是毫秒
    timeBetweenEvictionRunsMillis: 60000
    # 配置一个连接在池中最小生存的时间，单位是毫秒
    minEvictableIdleTimeMillis: 300000
    validationQuery: SELECT 1 FROM DUAL
    testWhileIdle: true
    testOnBorrow: false
    testOnReturn: false
    # 打开PSCache，并且指定每个连接上PSCache的大小
    poolPreparedStatements: true
    maxPoolPreparedStatementPerConnectionSize: 20
    # 配置监控统计拦截的filters，去掉后监控界面sql无法统计，'wall'用于防火墙
    filters: stat,log4j
    # 通过connectProperties属性来打开mergeSql功能；慢SQL记录
    connectionProperties: druid.stat.mergeSql=true;druid.stat.slowSqlMillis=5000

　　2.3 创建elasticsearch的配置类

 1 package spakt.demo.config;
 2 
 3 import org.elasticsearch.client.transport.TransportClient;
 4 import org.elasticsearch.common.settings.Settings;
 5 import org.elasticsearch.common.transport.InetSocketTransportAddress;
 6 import org.elasticsearch.transport.client.PreBuiltTransportClient;
 7 import org.springframework.beans.factory.annotation.Value;
 8 import org.springframework.context.annotation.Bean;
 9 import org.springframework.context.annotation.Configuration;
10 
11 import java.net.InetAddress;
12 import java.net.UnknownHostException;
13 
14 @Configuration
15 public class EsConfiguration {
16 
17 
18     private String clusterName = "elasticsearch";
19     @Value("${es.host}")
20     private String esHost;
21 
22     @Value("${es.port}")
23     private String esPort;
24 
25     @Bean
26     public TransportClient getTransportClient() throws UnknownHostException {
27 
28         Settings sets = Settings.builder().put("cluster.name", clusterName) //集群名字
29                 .put("client.transport.sniff", true)
30                 .build();
31         InetSocketTransportAddress inet = new InetSocketTransportAddress(InetAddress.getByName(esHost), Integer.parseInt(esPort));
32         TransportClient transportClient = new PreBuiltTransportClient(sets).addTransportAddress(inet);
33         return transportClient;
34     }
35 }

　　这儿注意如果是es6.x以后的版本，在引入es包之后，还要引入netty插件包，这儿以6.7.2为例

1         <!-- https://mvnrepository.com/artifact/org.elasticsearch.plugin/transport-netty4-client -->
2         <dependency>
3             <groupId>org.elasticsearch.plugin</groupId>
4             <artifactId>transport-netty4-client</artifactId>
5             <version>6.7.2</version>
6         </dependency>

　　并且上面的配置中InetSocketTransportAddress 修改为TransportAddress即可具体如下

1 　　 @Bean
2     public TransportClient initTransportClient() throws UnknownHostException {
3         Settings build = Settings.builder().put("cluster.name", clusterName).put("client.transport.sniff", true).build();
4         TransportAddress transportAddress = new TransportAddress(InetAddress.getByName(address),port);
5         TransportClient transportClient = new PreBuiltTransportClient(build).addTransportAddress(transportAddress);
6         return transportClient;
7     }

　　2.4 创建测试的controller类包含添加，查询，删除方法

  1 package spakt.demo.elasticsearch;
  2 
  3 import org.apache.commons.lang3.StringUtils;
  4 import org.elasticsearch.action.bulk.BulkRequestBuilder;
  5 import org.elasticsearch.action.bulk.BulkResponse;
  6 import org.elasticsearch.action.delete.DeleteRequestBuilder;
  7 import org.elasticsearch.action.index.IndexRequestBuilder;
  8 import org.elasticsearch.action.index.IndexResponse;
  9 import org.elasticsearch.action.search.SearchRequestBuilder;
 10 import org.elasticsearch.action.search.SearchResponse;
 11 import org.elasticsearch.client.transport.TransportClient;
 12 import org.elasticsearch.common.text.Text;
 13 import org.elasticsearch.index.query.BoolQueryBuilder;
 14 import org.elasticsearch.index.query.QueryBuilders;
 15 import org.elasticsearch.rest.RestStatus;
 16 import org.elasticsearch.search.SearchHit;
 17 import org.elasticsearch.search.SearchHits;
 18 import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
 19 import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
 20 import org.springframework.beans.factory.annotation.Autowired;
 21 import org.springframework.jdbc.core.JdbcTemplate;
 22 import org.springframework.stereotype.Controller;
 23 import org.springframework.web.bind.annotation.RequestMapping;
 24 import org.springframework.web.bind.annotation.ResponseBody;
 25 import spakt.demo.bean.Prisoner;
 26 
 27 import javax.annotation.Resource;
 28 import java.text.DecimalFormat;
 29 import java.util.*;
 30 
 31 /**
 32  *
 33  */
 34 @Controller
 35 public class EsController {
 36 
 37 //    @Resource(name = "jdbc2")
 38     @Autowired
 39     private JdbcTemplate jdbcTemplate;
 40 
 41     @Autowired
 42     private TransportClient transportClient;
 43 
 44     /**
 45      * 插入数据方法
 46      *
 47      * @return
 48      */
 49     @RequestMapping("/es")
 50     @ResponseBody
 51     public Object addEs() {
 52 
 53         //要插入的数据
 54         List<Map<String, Object>> prisonerListList = jdbcTemplate.
 55                 queryForList("select id,name,fact,family_address as address from da_prisoner ");
 56         //批处理类
 57         BulkRequestBuilder bulkRequestBuilder = transportClient.prepareBulk();
 58         int count = 0;
 59         for (Map<String, Object> prisoner : prisonerListList) {
 60 
 61             //插入方法
 62             //jyplatform 代表index   理解为数据库中的库
 63             //prisoner   代表type   理解为数据库总的表
 64             //UUID  个人理解是为数据库中的主键
 65             String uuid = UUID.randomUUID().toString().replace("-", "");
 66             IndexRequestBuilder indexRequestBuilder = transportClient.
 67                     prepareIndex("jyplatform", "prisoner",uuid)
 68                     .setSource(prisoner);
 69             //单个执行
 70             //IndexResponse indexResponse = indexRequestBuilder.get();
 71 
 72             //批量执行 将添加器加入批处理中
 73             bulkRequestBuilder.add(indexRequestBuilder);
 74             if (count % 5000 == 0) {
 75                 //每隔5000条执行一次批量插入的操作
 76                 BulkResponse bulkItemResponses = bulkRequestBuilder.execute().actionGet();
 77                 System.err.println(bulkItemResponses.getTook());
 78                 float num = (float) count / prisonerListList.size();
 79                 DecimalFormat df = new DecimalFormat("0.00");
 80                 String s = df.format(num * 100);
 81                 System.err.println("================================" + s + "%");
 82             }
 83             count++;
 84         }
 85         return prisonerListList;
 86     }
 87 
 88     /**
 89      * 查询数据
 90      *
 91      * @return
 92      */
 93     @RequestMapping("/get")
 94     @ResponseBody
 95     public Object getMap(String content) {
 96         List<Prisoner> prisonerList = new ArrayList<>();
 97         // 构造搜索条件
 98         BoolQueryBuilder query = QueryBuilders.boolQuery();
 99         //fact 即是field  理解为表中的字段
100         query.filter(QueryBuilders.matchPhraseQuery("fact", content));
101         //开始查询
102         SearchRequestBuilder searchRequestBuilder = transportClient
103                 .prepareSearch("jyplatform")   //确定index
104                 .setTypes("prisoner")   //确定type
105                 .setFrom(0)  //从多少条开始  分页使用
106                 .setSize(10000) //查多少条  分页使用 size-from>10000会报异常
107                 .setQuery(query); //加入查询条件
108         //结果高亮显示  
109         HighlightBuilder highlightBuilder = new HighlightBuilder();
110         highlightBuilder.field("fact");
111         highlightBuilder.preTags("<span style="color:red">");
112         highlightBuilder.postTags("</span>");
113         searchRequestBuilder.highlighter(highlightBuilder);
114         //获取查询结果
115         SearchResponse searchResponse = searchRequestBuilder.get();
116         if (searchResponse.status() != RestStatus.OK) {
117             System.err.println("查询结果错误");
118             return "error";
119         }
120         //得到查询的结果
121         SearchHits hits = searchResponse.getHits();
122         for (SearchHit hit : hits) {
123             //进行高亮处理
124             String fact = "";
125             Map<String, HighlightField> highlightFields = hit.getHighlightFields();
126             HighlightField field = highlightFields.get("fact");
127             if(field!=null){
128                 Text[] fragments = field.fragments();
129                 for (Text fragment : fragments) {
130                     fact+=fragment.toString();
131                 }
132             }
133 
134             prisonerList.add(new Prisoner(hit.getSource().get("id").toString(),
135                                           hit.getSource().get("name").toString(),
136                                           StringUtils.isBlank(fact)?hit.getSource().get("fact").toString():fact,
137                                           hit.getSource().get("address").toString()));
138         }
139         return prisonerList;
140     }
141 
142     /**
143      * 删除数据
144      */
145     @RequestMapping("/delete")
146     @ResponseBody
147     public String delete() {
148         //开始查询
149         SearchRequestBuilder searchRequestBuilder = transportClient
150                 .prepareSearch("jyplatform") //确定index
151                 .setTypes("prisoner") //确定type
152                 .setFrom(0)  // begin
153                 .setSize(10000); //size  size-from>10000会报异常
154         //获取查询结果
155         SearchHits hits = searchRequestBuilder.get().getHits();
156         //批量操作
157         BulkRequestBuilder bulkRequestBuilder = transportClient.prepareBulk();
158         for (SearchHit hit : hits) {
159             //删除方法
160             //jyplatform 代表index   理解为数据库中的库
161             //prisoner   代表type   理解为数据库总的表
162             // hit.getId()  这条数据的id
163             DeleteRequestBuilder deleteRequestBuilder = transportClient.
164                     prepareDelete("jyplatform", "prisoner", hit.getId());
165             //批量删除
166 //                bulkRequestBuilder.add(deleteRequestBuilder);
167 //                if(i%5000 ==0){
168 //                    bulkRequestBuilder.execute();
169 //                }
170             //单个操作
171             deleteRequestBuilder.get();
172         }
173         return "OK";
174     }
175 
176 
177 }

　　2.5 创建启动类启动测试

 1 package spakt.demo;
 2 
 3 import org.springframework.boot.SpringApplication;
 4 import org.springframework.boot.autoconfigure.SpringBootApplication;
 5 import org.springframework.context.annotation.ImportResource;
 6 
 7 @SpringBootApplication
 8 @ImportResource(value = "classpath:spring-*.xml")
 9 public class DemoApplication {
10 
11     public static void main(String[] args) {
12         SpringApplication.run(DemoApplication.class, args);
13     }
14 }

总结

　　目前只学习了下初步的使用，复杂的应用待后面继续挖掘。集群也，还有分词器一大堆东西待待填坑

使用java操作elasticsearch(1)

1.安装elasticsearch

2.创建java的springboot工程

2.1pom.xml文件依赖

2.2application.yml文件配置

2.3 创建elasticsearch的配置类

2.4 创建测试的controller类 包含添加，查询，删除方法

2.5 创建启动类启动测试

总结

　　2.1pom.xml文件依赖

　　2.2application.yml文件配置

　　2.3 创建elasticsearch的配置类

　　2.4 创建测试的controller类包含添加，查询，删除方法

　　2.5 创建启动类启动测试