若代理设置了 auto.create.topics.enable=true,这样还未创建topic就往kafka发送消息时,
会自动创建一个 ${num.partitions}个分区和{default.replication.factor}个副本的主题
创建主题
1.创建一个名称为“kafka-action”的主题,该主题有两个副本,3个分区:
[root@docp-n bin]# ./kafka-topics.sh --create --zookeeper docp-n:2181 --replication-factor 2 --partitions 3 --topic kafka-action Created topic "kafka-action".
*docp-n:2181 zookeeper所在的ip
zookeeper 必传参数,多个zookeeper用 ‘,’分开。
partitions 用于设置主题分区数,每个线程处理一个分区数据
replication-factor 用于设置主题副本数,每个副本分布在不通节点,不能超过总结点数。如你只有一个节点,但是创建时指定副本数为2,就会报错。
创建主题时,还可以通过config参数设置主题几倍的配置以覆盖默认配置
例如:
kafka-topics.sh --create --zookeeper server-1:2181,server-2:2181 --replacation-factor 2 --partitions 3 --topic config-test --config max.messege.bytes=404800
通过zookeeper客户端可以看到覆盖的相关配置。get /config/topics/config-test
[root@haha bin]# /data/app/zookeeper/bin/zkCli.sh -server localhost:2181 [zk: localhost:2181(CONNECTED) 2] get /brokers/topics/test2 {"version":1,"partitions":{"1":[5],"0":[5]}} cZxid = 0x25a ctime = Thu Nov 01 15:25:29 CST 2018 mZxid = 0x25a mtime = Thu Nov 01 15:25:29 CST 2018 pZxid = 0x25d cversion = 1 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 44 numChildren = 1
可以登录zookeeper客户端查看所创建topic元数据信息,“kafka-action”
[root@docp-n bin]# /data/app/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181 Connecting to docp-n:2181 Welcome to ZooKeeper! JLine support is enabled [zk: docp-n:2181(CONNECTING) 0] WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: docp-n:2181(CONNECTED) 0] ls /brokers/topics/kafka-action/partitions [0, 1, 2]
2.删除kafka主题
(1)手动删除log.dir目录下的分区目录,并删除zookeeper客户端对应的节点,/broker/topics/ && /config/topics
(2) 执行 kafka-topics.sh 脚本删除,如果想通过脚本彻底删除,需要修改启动时加载的配置文件 server.propers中
delete-topic-enable=true 默认为false,否则并未真正删除
[root@docp-n bin]# ./kafka-topics.sh --delete --zookeeper docp-n:2181 --topic test Topic test is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true.
注意控制台打印的消息
3.删除主题
[root@docp-n bin]# ../bin/kafka-topics.sh --delete --zookeeper 127.0.0.1:2181 --topic test Topic test is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true.
4.查看主题
(1)查看所有主题
[root@docp-n bin]# ./kafka-topics.sh --list --zookeeper 127.0.0.1:2181 1ab42b8d882347ef81a19914440bc308 286a9b791be54ca7bda4b65530fde8b4 2dfb690e5a514398879894a56f081520 414a2a3e915b4fd6811a77e6cadf75be 87a9ba31db734577bef86143e95a8b44 __consumer_offsets b1fe8eb154f5493e90e939dff8031e71 f53def0af82344b796529ba7ab79fe66 hello_topic kafka-action log-format test
(2) 查看某个特定主题信息,不指定topic则查询所有
通过 --describe
[root@docp-n bin]# ./kafka-topics.sh --describe --zookeeper 127.0.0.1:2181 --topic kafka-action Topic:kafka-action PartitionCount:3 ReplicationFactor:2 Configs: Topic: kafka-action Partition: 0 Leader: 1 Replicas: 1,5 Isr: 1,5 Topic: kafka-action Partition: 1 Leader: 5 Replicas: 5,0 Isr: 5,0 Topic: kafka-action Partition: 2 Leader: 0 Replicas: 0,1 Isr: 0,1
(3)查看正在同步的主题
通过 --describe 和 under-replicated-partitions命令组合查看 under-replacation状态
[root@docp-n bin]# ./kafka-topics.sh --describe --zookeeper 127.0.0.1:2181 --under-replicated-partitions
(4)查看主题覆盖的配置
通过 describe 和 topics-with-overrides 组合
[root@docp-n bin]# ./kafka-topics.sh --describe --zookeeper 127.0.0.1:2181 --topics-with-override Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer