Watcher loop has timeouts

[anekdoti]

Our operator gets sometimes timeouts in the watcher loop. It is running in our Kubernetes cluster (so not locally) so I would have expected requests to the Kubernetes API to be fine. What might be reasons for that?

[2021-09-27 07:26:36.0638264 [KubeOps.Operator.Kubernetes.ResourceWatcher] [Error] There was an error while watching the resource ""{{OUR-CUSTOM-RESOURCE-NAME}}"".
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.Http2Connection.Http2Stream.<>c.<WaitForDataAsync>b__68_0(Object s)
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()
--- End of stack trace from previous location ---
   at System.Net.Http.Http2Connection.Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at k8s.WatcherDelegatingHandler.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal()
   at k8s.WatcherDelegatingHandler.PeekableStreamReader.PeekLineAsync()
   at k8s.WatcherDelegatingHandler.LineSeparatedHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at k8s.Kubernetes.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.WatcherExt.<>c__DisplayClass0_0`2.<<Watch>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)

[buehler]

Yes, this sometimes happens. Do you have a special config in the Kubernetes API? (maybe other defaults to watchers?)
The watcher should restart immediately when it encounters an error.

[anekdoti]

Thank you :-)