This is a demo for the Medium blog post How to Design a Self-Healing, Dynamic-Size Raft Cluster in Go. It shows how to use the API of Hashicorp's Raft library in Go to automatise adding and removing Raft peers to the cluster. The code here attempts to minimise the risk of a cluster failure - this is when a majority of nodes in the cluster are dead/unreachable.
- Start the cluster:
docker-compose up - Kill any of the Raft containers (not all)
- Watch the cluster self-heal
- Change
NumberReplicasin desired_state.json - Watch the cluster scale
NOTE: The api-server has the Docker socket mounted to it. This means it can create containers by itself.
- The leader asks the api-server for a desired state
- The leader continuously checks the cluster state to check whether it has diverged from the desired state
- The leader asks the api-server to add/remove containers to reach the desired state
- New followers ask the api-server who is in the cluster
- New followers ask around who the leader is
- New followers ask the leader whether they can join the cluster
- Dying followers ask the leader whether it can remove them from the cluster