feature: Add health-check (liveness and readiness) probes for Zebra

[gustavovalverde]

Motivation

If we're deploying some/several Zebra nodes, and we need to continuously confirm if those nodes are running and behaving as expected, we should have a way to validate the nodes are live and ready, which would be defined as:

• Readiness: Zebra is up, it's able to connect to other nodes, and respond to requests. If
• Liveness: The node is running, and it's synced and/or progressing with the sync. Ideally we should have states for this:
  • Ready to sync
  • Syncing
  • Synced

This would make nodes management easier, and more automatic. Here is an explanation on how this would work with Kubernetes, but this applies to most container orchestration tools.

Specifications

Any solution should consider this endpoints will be publicly available and thus their security implications should be thoroughly evaluated.

Requirements and examples from other projects:

• An implementation using a custom script (which I'd like to avoid): https://github.com/blockchain-etl/blockchain-kubernetes/blob/dev/charts/parity/templates/statefulset.yaml#L75-L80
• Similar requirement for Geth: Liveness and readiness probes for geth golemfactory/concent-deployment#44

Complex Code or Requirements

• If using the RPC endpoints, then we should enable those in our nodes, and not allow public access to them, but just from our infrastructure. For example, we might be able to ping getblockchaininfo (sync progress) or getinfo (liveness) instead; and we could add more fields.
• This wouldn't be a solution for everyone, but it would be a starting point

Testing

No response

Related Work

This was previously requested and partially done:

• Deploy zebrad containers via managed instance group #479