Architecture
When a master crashes, MHA recovers rest slaves as below.
Fig: Steps for recovery
Basic algorithms are described in the slides presented at the MySQL Conference and Expo 2011, especially from page no.13 to no.34.
In relay log files on slaves, master's binary log positions are written at "end_log_pos" sections (example). By comparing the latest end_log_pos between slaves, we can identify which relay log events are not sent to each slave. MHA internally recovers slaves (fixes consistency issues) by using this mechanism. In addition to basic algorithms covered in the slides at the MySQL Conf 2011, MHA does some optimizations and enhancements, such as generating differential relay logs very quickly (indenpendent from relay log file size), making recovery work with row based formats, etc.
MHA consists of MHA Manager and MHA Node as below.
Fig: MHA components
MHA Manager has manager programs such as monitoring MySQL master, controlling master failover, etc.
MHA Node has failover helper scripts such as parsing MySQL binary/relay logs, identifying relay log position from which relay logs should be applied to other slaves, applying events to the target slave, etc. MHA Node runs on each MySQL server.
When MHA Manager does failover, MHA manager connects MHA Node via SSH and executes MHA Node commands when needed.
MHA has a couple of extention points. For example, MHA can call any custom script to update master's ip address (updating global catalog database that manages master's ip address, updating virtual ip, etc). This is because how to manage IP address depends on users' environments and MHA does not want to force one approach.
Current extension points are as below. MHA Manager package includes sample scripts.
- secondary_check_script: For checking master availability from multiple network routes
- master_ip_failover_script: For updating master's ip address used from applications
- shutdown_script: For forcing shutdown the master
- report_script: For sending reports
- init_conf_load_script: For loading initial configuration parameters
- master_ip_online_change_script: For updating master ip address. This is not used for master failover, but used for online master switch