IMPORTANT: This bug is fixed in versions 2.1.2, 2.2.2, 2.3.0 and newer.
One of those bugs that you normally don’t come across… until you do. Officially, Kafka topic names can be up to 249 characters. If you try to create a topic with a name longer than that, it will reject it:
$ kafka-topics --bootstrap-server localhost:9092 --create --partitions 1 --replication-factor 1 --topic [topic name with 250 characters]
Error while executing topic command : org.apache.kafka.common.errors.InvalidTopicException: Topic name is illegal, it can't be longer than 249 characters, topic name: [topic name with 250 characters]
Pretty self-explanatory, right? Well, it’s not that simple…
249 characters? Actually… 209
While officially advertised as 249, in reality, you should not create topics with names longer than 209 characters. Why? One of the comments in the Jira ticket report explains it easily:
This limit ends up not being quite right since topic deletion ends up renaming the directory to the form topic-partition.uniqueId-delete as can be seen in LogManager.asyncDelete.
If we do the math:
- 249 –
- 1 character for the “.”
- 32 characters for the uniqueId (a.k.a. UUID)
- 7 characters for “-delete”
The result is 209. That is the longest your topic name can be if you don’t want to hit the bug.
What if I use a topic name longer than 209?
Well, terrible things… if you try to delete the topic. Otherwise nothing. But if you do try to delete the topic, the first error that you will encounter is this:
ERROR Error while renaming dir for [your long topic name here]-0 in log dir /var/lib/kafka/data (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: /var/lib/kafka/data/[your long topic name here]-0 -> /var/lib/kafka/data/[your long topic name here]-0.abb0d0dcdfd94e2b9222335d8bcaebcd-delete: File name too long
What is worse, your broker will keep failing upon restart, trying to remove the folder that is still pending to get deleted.
Get me out of this trap!
There is hope though if you can use a bit of SSH and connect to the broker that is failing (or maybe all of them are failing because they were all hosting partitions for the deleted topic).
Go to the directory where Kafka stores its data (log.dirs=/var/lib/kafka/data normally) and remove the folders for the partitions associated with the topic you want to get rid of.
Once that is deleted, it is time to mangle a bit with Zookeeper data. Be very careful here, if you make a mistake you will probably lose data (and maybe the whole cluster).
Log in into Zookeeper using zookeeper-shell and execute a couple of delete instructions to get rid of all traces of the faulty topic.
$ zookeeper-shell localhost:2181
rmr /admin/delete_topics/[your long topic name here]
rmr /brokers/topics/[your long topic name here]
rmr /config/topics/[your long topic name here]
quit
Now, restart your broker(s) and they should have completely forgotten that this topic ever existed…
What about AWS Kafka (MSK)?
As of this writing, MSK supports three versions:
- 1.1.1: this version is just “too old” (published in July, 2018) and I wouldn’t recommend using it. It suffers this bug too.
- 2.1.1: definitively suffers this bug (personally experienced it…).
- 2.3.1: is not affected by this bug.
Therefore, my recommendation is to run Kafka 2.3.1 and take advantage of this and other bug fixes and features.
