Golang ETCD包的安装使用 golang安装etcd第三方库clientv3 报错解决

视频地址

https://www.bilibili.com/video/BV19J41147uT?p=7

课件资料

https://www.liwenzhou.com/posts/Go/go_etcd/

ETCD

日志收集项目

为什么要自己写不用ELK?

ELK: 部署的时候麻烦每一个filebeat都需要配置一个配置文件

使用etcd来管理被收集的日志项。

项目的架构

上节课项目进度

kafka：消息队列
tailf：从文件里读日志
go-ini：解析配置文件

今日内容

etcd

etcd介绍

详见群文件：etcd.pdf

http://www.5lmh.com/数据库操作/go操作etcd/操作etcd.html

安装etcd

详见：https://docs.qq.com/doc/DTndrQXdXYUxUU09O

Ubuntu(linux系统)安装etcd

下载：etcd-v3.3.18-linux-amd64
解压文件即可使用，无需安装

命令行操作etcd

注意一定要使用V3版本的API

启动服务端

在命令行中

进入解压出来的目录中/opt/etcd-v3.3.18-linux-amd64

root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# su root  //切换到root账号
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcd   //启动服务端
2020-05-19 07:14:33.832992 I | etcdmain: etcd Version: 3.3.18
2020-05-19 07:14:33.833045 I | etcdmain: Git SHA: 3c8740a79
2020-05-19 07:14:33.833054 I | etcdmain: Go Version: go1.12.9
2020-05-19 07:14:33.833063 I | etcdmain: Go OS/Arch: linux/amd64
2020-05-19 07:14:33.833072 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2020-05-19 07:14:33.833085 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2020-05-19 07:14:33.833443 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2020-05-19 07:14:33.836644 I | embed: listening for peers on http://localhost:2380
2020-05-19 07:14:33.836736 I | embed: listening for client requests on localhost:2379
2020-05-19 07:14:33.849127 I | etcdserver: name = default
2020-05-19 07:14:33.849150 I | etcdserver: data dir = default.etcd
2020-05-19 07:14:33.849165 I | etcdserver: member dir = default.etcd/member
2020-05-19 07:14:33.849174 I | etcdserver: heartbeat = 100ms
2020-05-19 07:14:33.849184 I | etcdserver: election = 1000ms
2020-05-19 07:14:33.849194 I | etcdserver: snapshot count = 100000
2020-05-19 07:14:33.849228 I | etcdserver: advertise client URLs = http://localhost:2379
2020-05-19 07:14:33.851066 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 8
2020-05-19 07:14:33.851102 I | raft: 8e9e05c52164694d became follower at term 2
2020-05-19 07:14:33.851117 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 8, applied: 0, lastindex: 8, lastterm: 2]
2020-05-19 07:14:33.867515 W | auth: simple token is not cryptographically signed
2020-05-19 07:14:33.871849 I | etcdserver: starting server... [version: 3.3.18, cluster version: to_be_decided]
2020-05-19 07:14:33.878161 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2020-05-19 07:14:33.878282 N | etcdserver/membership: set the initial cluster version to 3.3
2020-05-19 07:14:33.878340 I | etcdserver/api: enabled capabilities for version 3.3
2020-05-19 07:14:35.352464 I | raft: 8e9e05c52164694d is starting a new election at term 2
2020-05-19 07:14:35.352492 I | raft: 8e9e05c52164694d became candidate at term 3
2020-05-19 07:14:35.352510 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3
2020-05-19 07:14:35.352526 I | raft: 8e9e05c52164694d became leader at term 3
2020-05-19 07:14:35.352541 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3
2020-05-19 07:14:35.352898 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2020-05-19 07:14:35.352917 I | embed: ready to serve client requests
2020-05-19 07:14:35.353073 E | etcdmain: forgot to set Type=notify in systemd service file?
2020-05-19 07:14:35.353526 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

启动成功后，此时已经是后台运行了，可以关闭黑窗口

客户端操作

再开启一个命令行窗口

先设置环境变量：

Windows执行：

SET ETCDCTL_API=3

Mac或者Linux 执行

export ETCDCTL_API=3

如果不设置上面的环境变量，会报下面的错误：

root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcdctl --endpoints=http://127.0.0.1:2379 put name lisi
No help topic for 'put'

命令行操作：使用自带的etcdctl 命令操作
进入解压出来的/opt/etcd-v3.3.18-linux-amd64目录下

PUT：

./etcdctl --endpoints=http://127.0.0.1:2379 put key value

GET：

./etcdctl --endpoints=http://127.0.0.1:2379 GET key

DEL:

./etcdctl --endpoints=http://127.0.0.1:2379 DEL key

haima@haima-PC:/opt/etcd-v3.3.18-linux-amd64$ su
密码：
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcdctl --endpoints=http://127.0.0.1:2379 put name lisi
No help topic for 'put'
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# export ETCDCTL_API=3
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcdctl --endpoints=http://127.0.0.1:2379 put name lisi
OK
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcdctl --endpoints=http://127.0.0.1:2379 get name
name
lisi

go操作etcd

go mod设置代理

SET GOPROXY=https://goproxy.cn // Windows
export GOPROXY=https://goproxy.cn // Mac&Linux

PUT/GET

新建 etcd_put/main.go

package main

import (
	"context"
	"fmt"
	"go.etcd.io/etcd/clientv3"
	"time"
)

// etcd client put/get demo
// use etcd/clientv3
func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"127.0.0.1:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		// handle error!
		fmt.Printf("connect to etcd failed, err:%v
", err)
		return
	}
	fmt.Println("connect to etcd success")

	defer cli.Close()

	// put
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	//value := `[{"path":"c:/tmp/nginx.log","topic":"web.log"},{"path":"d:/xxx/redis.log","topic":"redis.log"}]`
	value := `[{"path":"c:/tmp/nginx.log","topic":"web.log"},{"path":"d:/xxx/redis.log","topic":"redis.log"},{"path":"d:/xxx/mysql.log","topic":"mysql.log"}]`
	_, err = cli.Put(ctx, "/logagent/collect_config", value)
	//_, err = cli.Put(ctx, "baodelu", "dsb")
	cancel()
	if err != nil {
		fmt.Printf("put to etcd failed, err:%v
", err)
		return
	}

	// get
	ctx, cancel = context.WithTimeout(context.Background(), time.Second)
	resp, err := cli.Get(ctx, "/logagent/collect_config")
	cancel()
	if err != nil {
		fmt.Printf("get from etcd failed, err:%v
", err)
		return
	}
	for _, ev := range resp.Kvs {
		fmt.Printf("%s:%s
", ev.Key, ev.Value)
	}
}

先运行服务端，
再启动go代码
首次运行go mod报错

go mod init
go mod tidy

go: haimait/learn/etcd/etcd_put imports
        go.etcd.io/etcd/clientv3 tested by
        go.etcd.io/etcd/clientv3.test imports
        github.com/coreos/etcd/auth imports
        github.com/coreos/etcd/mvcc/backend imports
        github.com/coreos/bbolt: github.com/coreos/bbolt@v1.3.4: parsing go.mod:
        module declares its path as: go.etcd.io/bbolt
                but was required as: github.com/coreos/bbolt

我的go版本是（go version 可以查看版本）

go version go1.14.2 linux/amd64

报错原因：
go.mod中更改下etcd的版本号，估计是etcd版本用的"google.golang.org/grpc/resolver"版本比较新;

解决方法：
在go.mod里写上

module haimait/learn/etcd/etcd_watch

go 1.14
require (
	github.com/coreos/bbolt v1.3.3 // indirect
	github.com/coreos/etcd v3.3.21+incompatible // indirect
        go.etcd.io/etcd v3.3.13+incompatible    
)

再次执行

haima@haima-PC:/media/haima/34E401CC64DD0E28/site/go/src/haimait/learn/etcd/etcd_put$ go mod tidy
go: finding module for package google.golang.org/grpc/keepalive
go: finding module for package golang.org/x/crypto/bcrypt
go: finding module for package google.golang.org/grpc/grpclog
go: finding module for package google.golang.org/grpc/health/grpc_health_v1
go: finding module for package google.golang.org/grpc/metadata
go: finding module for package google.golang.org/grpc/credentials
go: finding module for package google.golang.org/grpc/codes
go: finding module for package github.com/gogo/protobuf/gogoproto
go: finding module for package google.golang.org/grpc/peer
go: finding module for package github.com/dgrijalva/jwt-go
go: finding module for package github.com/coreos/pkg/capnslog
go: finding module for package github.com/prometheus/client_golang/prometheus/promhttp
go: finding module for package google.golang.org/genproto/googleapis/api/annotations
go: finding module for package google.golang.org/grpc/connectivity
go: finding module for package github.com/golang/protobuf/proto
go: finding module for package google.golang.org/grpc/resolver/dns
go: finding module for package google.golang.org/grpc/resolver
go: finding module for package github.com/json-iterator/go
go: finding module for package go.uber.org/zap/zapcore
go: finding module for package go.uber.org/zap
go: finding module for package golang.org/x/time/rate
go: finding module for package golang.org/x/net/trace
go: finding module for package github.com/xiang90/probing
go: finding module for package github.com/grpc-ecosystem/go-grpc-prometheus
go: finding module for package golang.org/x/net/context
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/utilities
go: finding module for package github.com/coreos/go-semver/semver
go: finding module for package github.com/jonboulle/clockwork
go: finding module for package github.com/modern-go/reflect2
go: finding module for package sigs.k8s.io/yaml
go: finding module for package google.golang.org/grpc/resolver/passthrough
go: finding module for package github.com/soheilhy/cmux
go: finding module for package google.golang.org/grpc/balancer
go: finding module for package github.com/coreos/go-systemd/journal
go: finding module for package google.golang.org/grpc
go: finding module for package github.com/google/btree
go: finding module for package google.golang.org/grpc/health
go: finding module for package github.com/tmc/grpc-websocket-proxy/wsproxy
go: finding module for package github.com/dustin/go-humanize
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware
go: finding module for package github.com/prometheus/client_golang/prometheus
go: finding module for package github.com/prometheus/client_model/go
go: finding module for package github.com/gogo/protobuf/proto
go: finding module for package google.golang.org/grpc/status
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/runtime
go: finding module for package golang.org/x/sys/unix
go: finding module for package go.etcd.io/bbolt
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/google/uuid
go: finding module for package github.com/golang/groupcache/lru
go: found google.golang.org/grpc in google.golang.org/grpc v1.29.1
go: found github.com/gogo/protobuf/gogoproto in github.com/gogo/protobuf v1.3.1
go: found github.com/golang/protobuf/proto in github.com/golang/protobuf v1.4.2
go: found google.golang.org/genproto/googleapis/api/annotations in google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587
go: found github.com/grpc-ecosystem/go-grpc-prometheus in github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: found github.com/prometheus/client_golang/prometheus/promhttp in github.com/prometheus/client_golang v1.6.0
go: found golang.org/x/crypto/bcrypt in golang.org/x/crypto v0.0.0-20200510223506-06a226fb4e37
go: found github.com/coreos/pkg/capnslog in github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
go: found github.com/dgrijalva/jwt-go in github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: found github.com/google/uuid in github.com/google/uuid v1.1.1
go: found go.uber.org/zap in go.uber.org/zap v1.15.0
go: found github.com/soheilhy/cmux in github.com/soheilhy/cmux v0.1.4
go: found github.com/coreos/go-systemd/journal in github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf
go: found github.com/coreos/go-semver/semver in github.com/coreos/go-semver v0.3.0
go: found github.com/json-iterator/go in github.com/json-iterator/go v1.1.9
go: found github.com/modern-go/reflect2 in github.com/modern-go/reflect2 v1.0.1
go: found github.com/grpc-ecosystem/grpc-gateway/runtime in github.com/grpc-ecosystem/grpc-gateway v1.14.5
go: found github.com/tmc/grpc-websocket-proxy/wsproxy in github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966
go: found golang.org/x/net/trace in golang.org/x/net v0.0.0-20200513185701-a91f0712d120
go: found sigs.k8s.io/yaml in sigs.k8s.io/yaml v1.2.0
go: found github.com/jonboulle/clockwork in github.com/jonboulle/clockwork v0.1.0
go: found github.com/dustin/go-humanize in github.com/dustin/go-humanize v1.0.0
go: found github.com/grpc-ecosystem/go-grpc-middleware in github.com/grpc-ecosystem/go-grpc-middleware v1.2.0
go: found golang.org/x/time/rate in golang.org/x/time v0.0.0-20200416051211-89c76fbcd5d1
go: found github.com/xiang90/probing in github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: found golang.org/x/sys/unix in golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9
go: found github.com/google/btree in github.com/google/btree v1.0.0
go: found github.com/golang/groupcache/lru in github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: found go.etcd.io/bbolt in go.etcd.io/bbolt v1.3.4
go: found github.com/prometheus/client_model/go in github.com/prometheus/client_model v0.2.0
go: finding module for package github.com/gorilla/websocket
go: found github.com/gorilla/websocket in github.com/gorilla/websocket v1.4.2

haima@haima-PC:/media/haima/34E401CC64DD0E28/site/go/src/haimait/learn/etcd/etcd_put$ go run main.go
connect to etcd success
/logagent/collect_config:[{"path":"c:/tmp/nginx.log","topic":"web.log"},{"path":"d:/xxx/redis.log","topic":"redis.log"},{"path":"d:/xxx/mysql.log","topic":"mysql.log"}]

如果还是解决不了，请参考下面连接

https://my.oschina.net/u/2321997/blog/4258724/print

watch

非常重要
新建 etcd_watch/main.go

package main

import (
	"context"
	"fmt"
	"go.etcd.io/etcd/clientv3"
	"time"
)
// etcd watch


func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints: []string{"127.0.0.1:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		// handle error!
		fmt.Printf("connect to etcd failed, err:%v
", err)
		return
	}
	fmt.Println("connect to etcd success")

	defer cli.Close()
	// watch
	// 派一个哨兵 一直监视着 /logagent/collect_config 这个key的变化（新增、修改、删除）
	ch := cli.Watch(context.Background(), "/logagent/collect_config")
	// 从通道尝试取值(监视的信息)
	for wresp := range ch{
		for _, evt := range wresp.Events{
			fmt.Printf("Type:%v key:%v value:%v
", evt.Type, string(evt.Kv.Key), string(evt.Kv.Value))
		}
	}
}

运行代码

haima@haima-PC:/media/haima/34E401CC64DD0E28/site/go/src/haimait/learn/etcd/etcd_watch$ go run main.go
connect to etcd success
Type:PUT key:/logagent/collect_config value:[{"path":"c:/tmp/nginx1111.log","topic":"web.log"},{"path":"d:/xxx/redis.log","topic":"redis.log"},{"path":"d:/xxx/mysql.log","topic":"mysql.log"}]

可以看到，当key /logagent/collect_config 的值变化时，会自动监听到

我的代码目录

.
├── etcd_put
│   ├── go.mod
│   ├── go.sum
│   └── main.go
└── etcd_watch
├── go.mod
├── go.sum
└── main.go

使用etcd优化日志收集项目

本周任务

Raft协议
1. 选举
2. 日志复制机制
3. 异常处理（脑裂）
4. zookeeper的zad协议的区别

Golang ETCD包的安装使用 golang安装etcd第三方库clientv3 报错 解决