Observability interface for worker health

[rcaudy]

Formalizing some side discussions I've had with a few devs. Right now, we have various "best-effort" mechanisms for notifying GUI users that the worker driving their REPL has terminated.

1. Do we need an API for monitoring worker health (e.g heap usage)?
2. Should we log shutdown messaging or "liveness" heartbeats to a file that can be externally observed (bind mount)? In deference to terminology proposed by others, I might call this a "tombstone" service.
3. Should we introduce a side-car process that reports on the health of a worker from within an application, with some persistence in order to enable clearer communication on GUI refresh?

I think we definitely want to introduce a standard session-status listener in all API clients:
Proposed callbacks:

1. WorkerTerminationNotification: terminal state; worker has gone to the giant virtual private cloud in the sky
2. ConnectionInterrupted - recoverable state; we aren't getting heartbeat or other request responses in timely fashion
3. ConnectionLost - terminal state; follows ConnectionInterrupted, we're giving up due to timeout or other considerations
4. ConnectionReestablished - happy state; follows ConnectionInterrupted, things look OK again

[rcaudy]

@nbauernfeind is adding support for worker termination notifications from the worker itself now. I understand that @niloc132 has added sufficient support that we can monitor "connection health" over the GRPC API already. We're missing integration with the Java, C++, and Python client APIs; presumable @devinrsmith @kosak @jmao-denver will need issues for this if we agree on an a design.

[devinrsmith]

I don't think we can necessarily rely on WorkerTerminationNotification being delivered in exceptional circumstances. But I think it's probably a good message.

[jcferretti]

@devinrsmith I think ConnectionLost is trying to be a catch for cases where the termination notification didn't happen. MIA versus KIA so to speak.

[rcaudy]

Cristian's reply captures my intent.

[devinrsmith]

Worker may not be the best name?

[rcaudy]

Agree. @nbauernfeind do you like "server" ? Or just drop that word from the name?

[nbauernfeind]

I do like server. I went with TerminationNotification -- and dropped the adjective alltogether.

[jcferretti]

The 4 status above make me unsure what is the status for "it started and nothing bad has happened [yet]" (connection restablished sounds wrong for that)

[rcaudy]

They were intended to be callbacks on status change, not statuses themselves.

[nbauernfeind]

The PR for the server-side, and jsapi-impl, TerminationNotification is here: #1210