kafka Increasing replication factor

一开始创建Topic的时候

/root/kafka_2.10-0.10.0.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 16 --topic oplog

现在需求是增加replication-factor为3,使用kafka自带的脚本kafka-reassign-partitions.sh
首先看一下增加replication-factor前的Topic详情:

[root@template ~]# /root/kafka_2.10-0.10.0.0/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic oplog
Topic:oplog    PartitionCount:16    ReplicationFactor:1    Configs:
    Topic: oplog    Partition: 0    Leader: 3    Replicas: 3    Isr: 3
    Topic: oplog    Partition: 1    Leader: 1    Replicas: 1    Isr: 1
    Topic: oplog    Partition: 2    Leader: 2    Replicas: 2    Isr: 2
    Topic: oplog    Partition: 3    Leader: 3    Replicas: 3    Isr: 3
    Topic: oplog    Partition: 4    Leader: 1    Replicas: 1    Isr: 1
    Topic: oplog    Partition: 5    Leader: 2    Replicas: 2    Isr: 2
    Topic: oplog    Partition: 6    Leader: 3    Replicas: 3    Isr: 3
    Topic: oplog    Partition: 7    Leader: 1    Replicas: 1    Isr: 1
    Topic: oplog    Partition: 8    Leader: 2    Replicas: 2    Isr: 2
    Topic: oplog    Partition: 9    Leader: 3    Replicas: 3    Isr: 3
    Topic: oplog    Partition: 10    Leader: 1    Replicas: 1    Isr: 1
    Topic: oplog    Partition: 11    Leader: 2    Replicas: 2    Isr: 2
    Topic: oplog    Partition: 12    Leader: 3    Replicas: 3    Isr: 3
    Topic: oplog    Partition: 13    Leader: 1    Replicas: 1    Isr: 1
    Topic: oplog    Partition: 14    Leader: 2    Replicas: 2    Isr: 2
    Topic: oplog    Partition: 15    Leader: 3    Replicas: 3    Isr: 3

先将分区partitions 0 增加两个备份。

[root@template bin]# vim increase-replication-factor.json
{"version":1,
 "partitions":[{"topic":"oplog","partition":0,"replicas":[1,2,3]}]}
[root@template bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute

然后可以–verify来检查修改后的状态

[root@template bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [oplog,0] completed successfully

同样可以使用describe查看

[root@template bin]# /root/kafka_2.10-0.10.0.0/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic oplog
Topic:oplog    PartitionCount:16    ReplicationFactor:3    Configs:
    Topic: oplog    Partition: 0    Leader: 3    Replicas: 1,2,3    Isr: 3,1,2

然后修改increase-replication-factor.json

{"version":1,
 "partitions":[{"topic":"oplog","partition":1,"replicas":[1,2,3]},
               {"topic":"oplog","partition":2,"replicas":[1,2,3]},
               {"topic":"oplog","partition":3,"replicas":[1,2,3]},
               {"topic":"oplog","partition":4,"replicas":[1,2,3]},
               {"topic":"oplog","partition":5,"replicas":[1,2,3]},
               {"topic":"oplog","partition":6,"replicas":[1,2,3]},
               {"topic":"oplog","partition":7,"replicas":[1,2,3]},
               {"topic":"oplog","partition":8,"replicas":[1,2,3]},
               {"topic":"oplog","partition":9,"replicas":[1,2,3]},
               {"topic":"oplog","partition":10,"replicas":[1,2,3]},
               {"topic":"oplog","partition":11,"replicas":[1,2,3]},
               {"topic":"oplog","partition":12,"replicas":[1,2,3]},
               {"topic":"oplog","partition":13,"replicas":[1,2,3]},
               {"topic":"oplog","partition":14,"replicas":[1,2,3]},
               {"topic":"oplog","partition":15,"replicas":[1,2,3]}]}
[root@template bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
[root@template bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [oplog,14] completed successfully
Reassignment of partition [oplog,10] completed successfully
Reassignment of partition [oplog,3] completed successfully
Reassignment of partition [oplog,1] completed successfully
Reassignment of partition [oplog,11] completed successfully
Reassignment of partition [oplog,6] completed successfully
Reassignment of partition [oplog,2] completed successfully
Reassignment of partition [oplog,5] completed successfully
Reassignment of partition [oplog,9] completed successfully
Reassignment of partition [oplog,4] completed successfully
Reassignment of partition [oplog,12] completed successfully
Reassignment of partition [oplog,7] completed successfully
Reassignment of partition [oplog,13] completed successfully
Reassignment of partition [oplog,15] completed successfully
Reassignment of partition [oplog,8] completed successfully