confluent介绍
https://www.cnblogs.com/dadadechengzi/p/9506964.html
kafka connect:
https://www.cnblogs.com/dadadechengzi/p/9506216.html
官网文档:
https://docs.confluent.io/current/
kudu sink:
https://docs.confluent.io/current/connect/kafka-connect-kudu/sink-connector/index.html#
部署:
https://blog.csdn.net/zzq900503/article/details/83419245 GG
https://www.jianshu.com/p/87ae3a822f2f
官网安装:
https://docs.confluent.io/current/quickstart/cos-quickstart.html#cos-quickstart
日志：
tail -500 /data1/confluent-5.2.2/logs/connect/kafka-connect.log |grep error
阿里：
/mnt/disk1/confluent-5.2.2/logs/connect/kafka-connect.log
日志是作业的日志,也是服务的日志
错误:
1/ [2020-02-18 11:34:16,624] WARN [Producer clientId=producer-3] Error connecting to node ali-bi-kafka-1:9092 (id: 101 rack: null) (org.apache.kafka.clients.NetworkClient)
连接kafka错误,是因为在/etc/hosts中没有注册内网域名:
cat /etc/hosts
10.52.5.208 ali-bi-kafka-1
10.52.5.209 ali-bi-kafka-2
10.52.5.210 ali-bi-kafka-3
confluent status
confluent start
confluent stop
平台自带监控平台: 适用期30天
Control Center
https://docs.confluent.io/current/control-center/installation/install-apache-kafka.html
Confluent的组件
Confluent Platform 包括更多的工具和服务，使构建和管理数据流平台更加容易。
Confluent Control Center（闭源）。管理和监控Kafka最全面的GUI驱动系统。
Confluent Kafka Connectors（开源）。连接SQL数据库/Hadoop/Hive
Confluent Kafka Clients（开源）。对于其他编程语言，包括C/C++,Python
Confluent Kafka REST Proxy（开源）。允许一些系统通过HTTP和kafka之间发送和接收消息。
Confluent Schema Registry（开源）。帮助确定每一个应用使用正确的schema当写数据或者读数据到kafka中。
confluent:
安装JDK
https://www.cnblogs.com/hongfeng2019/p/11331688.html
下载部署
https://docs.confluent.io/current/installation/installing_cp/zip-tar.html
curl -O http://packages.confluent.io/archive/5.3/confluent-5.3.1-2.12.tar.gz
tar xzf confluent-5.3.1-2.12.tar.gz -C /data1/
添加环境变量:
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64
JRE_HOME=$JAVA_HOME/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:/data1/confluent-5.2.2/bin
source /etc/profile
可能还配zK,不用kafka就不需要
启动
confluent status
测试:
curl localhost:8083/connectors
[root@ali-gateway-confluent-1 system]# curl -X GET localhost:8083/connectors/
["opay-account-request-record-binlog-oss-sink","opay-account-request-record-mysql-binlog-source","opay-account-user-record-mysql-binlog-source","oride-db-mysql-binlog-source","opay-account-user-record-binlog-oss-sink","oride-db-binlog-oss-sink","oride-mysql-binlog-source-old-version","opay-bigorder-user-order-mysql-binlog
#如果场景是指用connector,做一个systemctl启动服务
cd /usr/lib/systemd/system/
ll | grep confluent
[root@bigdata-confluent-1 system]# cat confluent-kafka-connect.service
[Unit]
Description=Apache Kafka Connect - distributed
Documentation=http://docs.confluent.io/
After=network.target
[Service]
Type=simple
Environment="KAFKA_HEAP_OPTS=-Xms1G -Xmx6G" #调整连接kafka的heap大小
ExecStart=/data1/confluent-5.2.2/bin/connect-distributed /data1/confluent-5.2.2/etc/schema-registry/connect-avro-distributed.properties
TimeoutStopSec=30
Restart=always
[Install]
WantedBy=multi-user.target
---------------
systemctl daemon-reload
systemctl enable confluent-kafka-connect.service
systemctl star confluent-kafka-connect
/data1/confluent-5.2.2/etc/schema-registry
vim connect-avro-distributed.properties
bootstrap.servers=152.32.140.94:9092,152.32.140.94:19092,152.32.140.143:9092,152.32.140.143:19092
/data1/confluent-5.2.2/bin/connect-distributed /data1/confluent-5.2.2/etc/schema-registry/connect-avro-distributed.properties
公司：
ali-gateway-confluent-binglog-1-10.52.5.218上：
scp -r confluent-5.2.2/ root@10.52.5.227:/mnt/disk1/
#配置文件:
#cluster模式,都指向同一kafka就是一个集群,元数据都在kafka里,即消费的是kafka
#配置文件:
vim /mnt/disk1/confluent-5.2.2/etc/schema-registry/connect-avro-distributed.properties
bootstrap.servers=10.52.5.208:9092,10.52.5.209:9092,10.52.5.210:9092
#confluent中kudu的默认jar包
[root@ali-gateway-confluent-1 confluent-5.4.0]# ll /mnt/disk1/confluent-hup/share/confluent-hub-components/confluentinc-kafka-connect-kudu/lib
total 22220
-rw-r--r-- 1 root root 3482 Feb 4 10:22 animal-sniffer-annotations-1.14.jar
-rw-r--r-- 1 root root 20437 Feb 4 10:22 audience-annotations-0.5.0.jar
-rw-r--r-- 1 root root 31547 Feb 4 10:22 checker-compat-qual-2.0.0.jar
-rw-r--r-- 1 root root 51605 Feb 4 10:22 confluent-licensing-new-5.3.2-ce.jar
-rw-r--r-- 1 root root 883740 Feb 4 10:22 confluent-serializers-new-5.3.2-ce.jar
-rw-r--r-- 1 root root 12322 Feb 4 10:22 connect-licensing-extensions-0.4.4.jar
-rw-r--r-- 1 root root 62470 Feb 4 10:22 connect-utils-0.1.8.jar
-rw-r--r-- 1 root root 13704 Feb 4 10:22 error_prone_annotations-2.1.3.jar
-rw-r--r-- 1 root root 241622 Feb 4 10:22 gson-2.8.5.jar
-rw-r--r-- 1 root root 2719791 Feb 4 10:22 guava-24.0-jre.jar
-rw-r--r-- 1 root root 13348778 Feb 4 10:22 ImpalaJDBC4-2.6.4.1005.jar
-rw-r--r-- 1 root root 8782 Feb 4 10:22 j2objc-annotations-1.1.jar
-rw-r--r-- 1 root root 125632 Feb 4 10:22 jaxb-api-2.3.0.jar
-rw-r--r-- 1 root root 87325 Feb 4 10:22 jline-0.9.94.jar
-rw-r--r-- 1 root root 259680 Feb 4 10:22 jose4j-0.6.4.jar
-rw-r--r-- 1 root root 19936 Feb 4 10:22 jsr305-3.0.2.jar
-rw-r--r-- 1 root root 223667 Feb 4 10:22 kafka-connect-jdbc-5.3.1.jar
-rw-r--r-- 1 root root 258429 Feb 4 10:22 kafka-connect-kudu-1.0.1.jar
-rw-r--r-- 1 root root 489884 Feb 4 10:22 log4j-1.2.17.jar
-rw-r--r-- 1 root root 82123 Feb 4 10:22 metrics-core-2.2.0.jar
-rw-r--r-- 1 root root 1292696 Feb 4 10:22 netty-3.10.6.Final.jar
-rw-r--r-- 1 root root 1383882 Feb 4 10:22 protobuf-java-3.4.0.jar
-rw-r--r-- 1 root root 68545 Feb 4 10:22 protobuf-java-util-3.4.0.jar
-rw-r--r-- 1 root root 12189 Feb 4 10:22 slf4j-log4j12-1.7.26.jar
-rw-r--r-- 1 root root 15120 Feb 4 10:22 spotbugs-annotations-3.1.8.jar
-rw-r--r-- 1 root root 74798 Feb 4 10:22 zkclient-0.10.jar
-rw-r--r-- 1 root root 911603 Feb 4 10:22 zookeeper-3.4.14.jar