Recently I worked on a project for which we noticed a lot of database connection errors when a new version of the pod was deployed on Kubernetes.
At first we thought that the during the shutdown the application cleaned up the database connection pool too soon, but it appeared the application had database connection errors before the connection pool was cleaned up.
Because the shutdown of the application took some time, we had defined in the deployment description a
terminationGracePeriodSeconds of 30 seconds. We use an istio sidecar in the pod and it appeared that the istio pod has its own graceful shutdown setting, which is independent of the one of the application container.
When you don’t define a graceful shutdown period for istio, the default is 5 seconds, so while the application was still shutting down, after 5 seconds all external connectivity of the application was broken, leading to errors.
After setting the graceful shutdown period of the istio side car to the same value as the
terminationGracePeriodSeconds of the application container, the issue was solved.
To change the graceful shutdown period of the istio container to e.g. 30 seconds, add the following annotions to your template:
proxy.istio.io/config: | terminationDrainDuration: 30s
Note that the value of
proxy.istio.io/config needs to be in YAML format. For that reason, leaving out the pipe symbol (|) wouldn’t work: it needs to be interpreted as a string.