I am trying to test Zookeeper to Kraft migration on my machine that later needs to done on dev env. I have created ZK and kafka images using podman and started them. Now one step in the migration is to enable migration on Kraft controllers and then on the ZK based Kafka brokers.

So to enable migration we have to add this flag zookeeper.metadata.migration.enable=true to brokers under the server.properties file and then restart them for the migration to begin. To make the change I exec into the container open the files and added this flag and saved it, but as soon as I stop and start it again, the change that I did is not present.

Now for migration, changes to server.properties needs to be on both Kraft controller and ZK-based broker a few times during the whole migration process like this flag. So will I have to rebuild the image again and again after I make changes to the configs ? or is there a way to make the changes persistent upon container restart..

I am trying to test Zookeeper to Kraft migration on my machine that later needs to done on dev env. I have created ZK and kafka images using podman and started them. Now one step in the migration is to enable migration on Kraft controllers and then on the ZK based Kafka brokers.

So to enable migration we have to add this flag zookeeper.metadata.migration.enable=true to brokers under the server.properties file and then restart them for the migration to begin. To make the change I exec into the container open the files and added this flag and saved it, but as soon as I stop and start it again, the change that I did is not present.

Now for migration, changes to server.properties needs to be on both Kraft controller and ZK-based broker a few times during the whole migration process like this flag. So will I have to rebuild the image again and again after I make changes to the configs ? or is there a way to make the changes persistent upon container restart..

• java
• docker
• apache-kafka
• docker-container
• podman

• 1 Where does the configuration file come from? docker exec has exactly the problem you describe here, that when the container is deleted and recreated any changes you make manually are lost. One typical approach is to bind-mount the configuration file from the host instead, so you can edit it normally and it will survive restarts, but you need a copy of it on the host to start with. – David Maze Commented yesterday
• @DavidMaze so this file server.properties is already present and comes with kafka's source code. So as I understand, ig I could create this file on my local system and volume mount it in the container... and after this if i lets say change this flag from true to false and restart the container I could see the changes being reflected, right ? – BloodFury Commented yesterday
• @BloodFury check out the edited answer, option1 – aran Commented yesterday

1 Answer 1

To make the change I exec into the container open the files and added this flag and saved it, but as soon as I stop and start it again, the change that I did is not present.

2 options...

1 - Mount server.properties

Mount a volume so it reads the server.properties of your local filesystem. All changes in the local server.properties will be read by further containers that also mount the configuration directory.

That way you can modify whatever you want without changing the image.

Let's say you save the server.properties file in /home/user/kafka/config/server.properties

Simple example:

docker run -v /home/user/kafka/config:/opt/kafka/config imagen:tag

This will copy everything inside your local kafka config, including server.properties, to the path the kafka container uses to read the configuration.

----

If you require it inside the docker-compose:

services:
  kafka:
    image: imagen:tag
    volumes:
      - ./config:/opt/kafka/config

Where ./config contains the kafka properties you want to share to the container.

-----

If you have a Dockerfile, the option is to copy the files inside the container, must you'd have to create a new image each time:

COPY /path/to/your/local/config /opt/kafka/config

2 - Commit changes (not recommended, look at comments)

In order for the changes to persist, after exitting the container you just modified you must call commit :

docker commit <currentKafkaContainer> <your_image>:tag

This will create an image with your modifications.

For example, after changing live the container just:

docker commit current_container newimage:beta

And then run the newimage:beta, which will have the changes stored. More about this here.