I'm currently running a kafka cluster on k8s. I'm running in KRaft mode. The version im running is 3.7.2. i have a persistent volume to persist all of the logs and metadata. When i try to upgrade my kafka cluster to 3.9.0 or 4.0.0 my controllers are stuck on the following:

[2025-04-03 19:30:53,997] INFO [RaftManager id=1] Node -2 disconnected. (.apache.kafka.clients.NetworkClient)
[2025-04-03 19:30:53,997] WARN [RaftManager id=1] Connection to node -2 (localhost/127.0.0.1:9093) could not be established. Node may not be available. (.apache.kafka.clients.NetworkClient)
[2025-04-03 19:30:54,014] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (.apache.kafka.image.loader.MetadataLoader)
[2025-04-03 19:30:54,115] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (.apache.kafka.image.loader.MetadataLoader)
[2025-04-03 19:30:54,215] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (.apache.kafka.image.loader.MetadataLoader)
[2025-04-03 19:30:54,315] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (.apache.kafka.image.loader.MetadataLoader)
[2025-04-03 19:30:54,415] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (.apache.kafka.image.loader.MetadataLoader)

From what i understand i need to upgrade my controllers first and then my brokers. Im not sure why its stuck on this phase and what could be the problem. When i change the image back to 3.7.2 my controllers start successfully and fast.