Cleanup Kafka logs

The following steps illustrates the way to cleanup Kafka logs.

Instructions

Backup list of log file names and its disk consumption data (optional)

$ kubectl exec -it kafka-v2-0 -- du -h /opt/kafka-data/logs |tee backup_0.logs
$ kubectl exec -it kafka-v2-1 -- du -h /opt/kafka-data/logs |tee backup_1.logs
$ kubectl exec -it kafka-v2-2 -- du -h /opt/kafka-data/logs |tee backup_2.logs

2. Cleanup logs

 $ kubectl exec -it kafka-v2-0 -- rm -rf /opt/kafka-data/logs/* -n kafka-cluster
 $ kubectl exec -it kafka-v2-1 -- rm -rf /opt/kafka-data/logs/* -n kafka-cluster
 $ kubectl exec -it kafka-v2-2 -- rm -rf /opt/kafka-data/logs/* -n kafka-cluster

3. If the pod is in crashlookbackoff state, and the storage is full, use the following workaround:

Make a copy of the pod manifest

$ kubectl get statefulsets kafka-v2 -n kafka-cluster -oyaml > manifest.yaml

Scale down the kafka statefulset replica count to zero

Make following changes to the copy of the statefulsets manifest file
- Modify command line from:

To

Apply this statefulsets manifest and scale up statefulsets replica count to 3, the pod should be in a running state now and follow [step 2].
Again scale down the kafka statefulset replica count to zero

Make following changes to the copy of the statefulsets manifest file
- Modify command line from:

To

Apply this statefulsets manifest and scale up statefulsets replica count to 3