Category: Uncategorized

Recently I worked on a project for which we noticed a lot of database connection errors when a new version of the pod was deployed on Kubernetes.

At first we thought that the during the shutdown the application cleaned up the database connection pool too soon, but it appeared the application had database connection errors before the connection pool was cleaned up.

Because the shutdown of the application took some time, we had defined in the deployment description a terminationGracePeriodSeconds of 30 seconds. We use an istio sidecar in the pod and it appeared that the istio pod has its own graceful shutdown setting, which is independent of the one of the application container.

When you don’t define a graceful shutdown period for istio, the default is 5 seconds, so while the application was still shutting down, after 5 seconds all external connectivity of the application was broken, leading to errors.

After setting the graceful shutdown period of the istio side car to the same value as the terminationGracePeriodSeconds of the application container, the issue was solved.

To change the graceful shutdown period of the istio container to e.g. 30 seconds, add the following annotions to your template:

proxy.istio.io/config: |
          terminationDrainDuration: 30s

Note that the value of proxy.istio.io/config needs to be in YAML format. For that reason, leaving out the pipe symbol (|) wouldn’t work: it needs to be interpreted as a string.

When you use Mockito to mock an object and add behavior, you sometimes need to combine variable arguments with fixed arguments. When you do that in this way

when(client.execute("myId1", any[String])

You get as error Invalid use of argument matchers with the explanation that you should use eq as a matcher for the fixed argument.

However, eq defines referential equality in Scala, so you should use ArgumentMatchers.eq instead or use e.g. import org.mockito.ArgumentMatchers.{eq => eqTo} and use eqTo instead of eq. So the example above would become

import org.mockito.ArgumentMatchers.{eq => eqTo}

…

when(client.execute(eqTo("myId1"), any[String])