Kubernetes too old resource version

Question

I'm working on a operator which create watch for different k8s resources. Every now and then I can see below exception in the logs and application just stop. What is causing this issue and how can I fix this ?

io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 29309228 (33284573)
    at [email protected]/io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:263)
    at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
    at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
    at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

Solution

I'm from Fabric8 Kubernetes Client team. I think it's standard behavior of Kubernetes to give 410 after some time during watch. It's usually client's responsibility to handle it. In the context of a watch, it will return HTTP_GONE when you ask to see changes for a resourceVersion that is too old - i.e. when it can no longer tell you what has changed since that version, since too many things have changed. In that case, you'll need to start again, by not specifying a resourceVersion in which case the watch will send you the current state of the thing you are watching and then send updates from that point.

Fabric8 does not handle it with plain watch. But it is handling it in SharedInformer API, see ReflectorWatcher. I would recommend using informer API when writing operators since it's better than plain list and watch. Here is a simple example of Using SharedInformer API:

try (KubernetesClient client = new DefaultKubernetesClient()) {
  SharedInformerFactory sharedInformerFactory = client.informers();
  SharedIndexInformer<Pod> podInformer = sharedInformerFactory.sharedIndexInformerFor(Pod.class, PodList.class, 30 * 1000L);
  podInformer.addEventHandler(new ResourceEventHandler<Pod>() {
    @Override
    public void onAdd(Pod pod) {
      // Handle Creation
    }

    @Override
    public void onUpdate(Pod oldPod, Pod newPod) {
      // Handle update
    }

    @Override
    public void onDelete(Pod pod, boolean deletedFinalStateUnknown) {
      // Handle deletion
    }
  });
  sharedInformerFactory.startAllRegisteredInformers();
}

You can find a full demo of a simple operator using Fabric8 SharedInformer API here: PodSet Operator In Java

Comments

[hudi]

nice example. What about how to watch just configMaps/secrets ? In spring boot this handle org.springframework.cloud.kubernetes.config.reload.EventBase‌​dConfigurationChange‌​Detector. Problem is that they dont handle this situation when resource is too old and status is GONE and they just close watcher and config map is no longer updated in application

[Rohan Kumar]

I think this should work if you replace Pod with ConfigMap

[hudi]

nope it is not working in spring boot application. I tried: stackoverflow.com/questions/63719631/… . There is no mechanism how to propagate this new object in spring boot application

[beatrice]

for me nothing has been changed. Sometimes I get this: too old resource version: 21382351 (21382351)