Skip to content

TopicRebalance

Jimmy Bonds edited this page May 29, 2019 · 2 revisions

Rebalancing Topics after Cluster Expansion:

Rebalancing a topic from a 5 node cluster to a 9 node cluster:

  • Describe the existing topic, examine current Leaders and Replicas:
# > kafkactl describe topic -x my.topic.one

TOPIC         PART  OFFSET  LEADER  REPLICAS  ISRs     OFFLINE
my.topic.one  0     565060  5       [5 2 3]   [3 2 5]  []
my.topic.one  1     565346  4       [4 3 5]   [5 3 4]  []
my.topic.one  2     564768  1       [1 5 4]   [5 4 1]  []
my.topic.one  3     566025  2       [2 4 1]   [1 2 4]  []
my.topic.one  4     565027  3       [3 1 2]   [1 3 2]  []
my.topic.one  5     564926  5       [5 3 4]   [5 4 3]  []
my.topic.one  6     564827  4       [4 5 1]   [5 1 4]  []
my.topic.one  7     565435  1       [1 4 2]   [4 2 1]  []
my.topic.one  8     564624  2       [2 1 3]   [1 3 2]  []
my.topic.one  9     564809  3       [3 2 5]   [5 2 3]  []
my.topic.one  10    564917  5       [5 4 1]   [5 4 1]  []
my.topic.one  11    566616  4       [4 1 2]   [1 4 2]  []
my.topic.one  12    566058  1       [1 2 3]   [2 1 3]  []
my.topic.one  13    564785  2       [2 3 5]   [5 2 3]  []
my.topic.one  14    564372  3       [3 5 4]   [5 4 3]  []

  • Initiate the rebalance across all available nodes. The new targeted Brokers will be output upon success:
# > kafkactl admin rebalance -x my.topic.one

TOPIC         PARTITION  BROKERS
my.topic.one  0          [1 8 9]
my.topic.one  1          [2 7 5]
my.topic.one  2          [3 8 9]
my.topic.one  3          [4 7 2]
my.topic.one  4          [5 4 1]
my.topic.one  5          [6 3 2]
my.topic.one  6          [7 6 1]
my.topic.one  7          [8 3 4]
my.topic.one  8          [9 6 7]
my.topic.one  9          [1 9 8]
my.topic.one  10         [2 5 7]
my.topic.one  11         [3 9 8]
my.topic.one  12         [4 6 3]
my.topic.one  13         [5 4 1]
my.topic.one  14         [6 5 2]

  • Describe the topic again and note the activity. Get broker details and note the current migrating replicas::
# > kafkactl describe topic -x my.topic.one

TOPIC         PART  OFFSET  LEADER  REPLICAS       ISRs       OFFLINE
my.topic.one  0     565068  5       [5 1 9 2 3 8]  [3 2 5]    []
my.topic.one  1     565355  4       [5 2 7 3 4]    [5 3 4 7]  []
my.topic.one  2     564783  1       [5 1 9 3 8 4]  [5 4 1]    []
my.topic.one  3     566029  2       [4 7 2 1]      [1 2 4]    []
my.topic.one  4     565037  3       [5 1 2 3 4]    [1 3 2]    []
my.topic.one  5     564935  5       [5 6 2 3 4]    [5 4 3]    []
my.topic.one  6     564834  4       [5 1 6 7 4]    [5 1 4]    []
my.topic.one  7     565445  1       [1 2 3 8 4]    [4 2 1]    []
my.topic.one  8     564632  2       [1 6 9 2 7 3]  [1 3 2]    []
my.topic.one  9     564820  3       [5 1 9 2 3 8]  [5 2 3]    []
my.topic.one  10    564922  5       [5 1 2 7 4]    [5 4 1 7]  []
my.topic.one  11    566625  4       [1 9 2 3 8 4]  [1 4 2]    []
my.topic.one  12    566062  1       [1 6 2 3 4]    [2 1 3]    []
my.topic.one  13    564796  2       [5 1 2 3 4]    [5 2 3]    []
my.topic.one  14    564378  3       [5 6 2 3 4]    [5 4 3]    []


# > kafkactl get brokers

BROKER                    ID  GRPs  LDR.REPLICAS  PEER.REPLICAS  TOTAL.REPLICAS  MIGRATING.REPLICAS  OVERLOAD
broker01.example.io:9092  1   4     5820          11652          17472           0                   55
broker02.example.io:9092  2   7     5815          11642          17457           16                  78
broker03.example.io:9092  3   9     5830          11672          17502           0                   78
broker04.example.io:9092  4   6     5434          10880          16314           13                  81
broker05.example.io:9092  5   7     5793          11598          17391           11                  77
broker06.example.io:9092  6   12    454           918            1372            62                  0
broker07.example.io:9092  7   8     460           930            1390            68                  1
broker08.example.io:9092  8   7     467           944            1411            72                  1
broker09.example.io:9092  9   8     507           1024           1531            36                  1

  • Once complete, describe the topic for the results and new Partition Leaders. Note a Preferred Replica Election is needed:
# > kafkactl describe topic -x my.topic.one

TOPIC         PART  OFFSET  LEADER  REPLICAS  ISRs     OFFLINE
my.topic.one  0     565072  1       [1 8 9]   [1 9 8]  []
my.topic.one  1     565357  2       [2 7 5]   [5 2 7]  []
my.topic.one  2     564786  3       [3 8 9]   [9 3 8]  []
my.topic.one  3     566031  2       [4 7 2]   [2 4 7]  []
my.topic.one  4     565042  5       [5 4 1]   [5 1 4]  []
my.topic.one  5     564936  6       [6 3 2]   [6 2 3]  []
my.topic.one  6     564835  7       [7 6 1]   [1 6 7]  []
my.topic.one  7     565448  8       [8 3 4]   [3 8 4]  []
my.topic.one  8     564632  9       [9 6 7]   [6 9 7]  []
my.topic.one  9     564823  1       [1 9 8]   [1 9 8]  []
my.topic.one  10    564924  5       [2 5 7]   [5 2 7]  []
my.topic.one  11    566628  3       [3 9 8]   [9 3 8]  []
my.topic.one  12    566064  4       [4 6 3]   [6 3 4]  []
my.topic.one  13    564804  5       [5 4 1]   [5 1 4]  []
my.topic.one  14    564381  6       [6 5 2]   [5 6 2]  []

  • Run a Preferred Replica Election and describe the topic once again for final results:
# > kafkactl admin set pre
Preferred Replica Election Successfully Sent.

TOPICS.NEED.PRE  PARTITIONS.NEED.PRE
1                2


# > kafkactl describe topic my.topic.one

TOPIC         PART  OFFSET  LEADER  REPLICAS  ISRs     OFFLINE
my.topic.one  0     566201  1       [1 8 9]   [1 9 8]  []
my.topic.one  1     566497  2       [2 7 5]   [5 2 7]  []
my.topic.one  2     565955  3       [3 8 9]   [9 3 8]  []
my.topic.one  3     567207  4       [4 7 2]   [2 4 7]  []
my.topic.one  4     566177  5       [5 4 1]   [5 1 4]  []
my.topic.one  5     566053  6       [6 3 2]   [6 2 3]  []
my.topic.one  6     565987  7       [7 6 1]   [1 6 7]  []
my.topic.one  7     566636  8       [8 3 4]   [3 8 4]  []
my.topic.one  8     565797  9       [9 6 7]   [6 9 7]  []
my.topic.one  9     566000  1       [1 9 8]   [1 9 8]  []
my.topic.one  10    566041  2       [2 5 7]   [5 2 7]  []
my.topic.one  11    567756  3       [3 9 8]   [9 3 8]  []
my.topic.one  12    567261  4       [4 6 3]   [6 3 4]  []
my.topic.one  13    566031  5       [5 4 1]   [5 1 4]  []
my.topic.one  14    565536  6       [6 5 2]   [5 6 2]  []
Clone this wiki locally