I'm using an autopilot cluster and have some workloads that can't be stopped under 10 minutes. I would like to set a termination grace period to X hours/days to prevent any interruption.
When I set manually a terminationGracePeriodSeconds over 600 seconds, it's automatically set back to 600 with a warning: Warning: autopilot-default-resources-mutator:The max supported TerminationGracePeriodSeconds is 600 seconds. Defaulting down from configured 1800 seconds to 600 seconds. deployment.apps/extended-pods configured
In Google documentation I saw this annotation to use to prevent this behavior cluster-autoscaler.kubernetes.io/safe-to-evict: "false" introduced in July, 2023 and documented here
But, when I'm using this annotation, I sill have the warning and the update of the TerminationGracePeriodSeconds.
Am I missing something?
I'm using an autopilot cluster and have some workloads that can't be stopped under 10 minutes. I would like to set a termination grace period to X hours/days to prevent any interruption.
When I set manually a terminationGracePeriodSeconds over 600 seconds, it's automatically set back to 600 with a warning: Warning: autopilot-default-resources-mutator:The max supported TerminationGracePeriodSeconds is 600 seconds. Defaulting down from configured 1800 seconds to 600 seconds. deployment.apps/extended-pods configured
In Google documentation I saw this annotation to use to prevent this behavior cluster-autoscaler.kubernetes.io/safe-to-evict: "false" introduced in July, 2023 and documented here
But, when I'm using this annotation, I sill have the warning and the update of the TerminationGracePeriodSeconds.
Am I missing something?
Share Improve this question edited Nov 20, 2024 at 8:46 jps 22.6k16 gold badges88 silver badges106 bronze badges asked Nov 20, 2024 at 8:43 AirmanbzhAirmanbzh 6451 gold badge5 silver badges11 bronze badges 1- By workload do you mean jobs or pods? Can't you implement graceful shutdowns and have the graceful shutdown hook to gracefully cleanup the workload within 10 mins and let the next pod or job to pick it up? – M B Commented Nov 20, 2024 at 11:33
1 Answer
Reset to default 0As per this official document
In GKE For Autopilot clusters therterminationGracePeriodSeconds is limited to 10 minutes (600 seconds) for most Pods except for Spot Pods, which are limited to 25 seconds.
The Annotation which you applied prevents the cluster Autoscaler from evicting a pod during Scale down. But it does not override the termination grace period enforced by Autopilot.
As per this kodekloud blog You can also use a preStop hook in your pods lifecycle to cleanup tasks before termination.
Still you require a longer grace period to consider requesting extended run time of pods through google cloud support to inquire about specific feature requests for your use case.