MHA 구성 init

DBMS/MHA/README.md

+[![logo](https://www.hongsnet.net/images/logo.gif)](https://www.hongsnet.net)
+
+# MHA 개요
+
+- **Yoshinori Matsunobu**에 의해 2011년 07월 23일 MHA 0.50이 Release 되었다.
+- MHA는 최소한의 Down Time으로 Master를 장애조치 하고, Slave를 새로운 Master로 선출하여 서비스 가동이 정상적으로 수행되도록 하는 Auto Failover 솔루션이다.
+- 각 노드(Master 및 Slave)를 자동으로 전환하며, Master와 Slave의 데이터를 동일하게 유지한다.
+- 자동 Master 모니터와 Failover를 지원한다.
+- 대화형 Master Failover 및 비대화형 Master Failover를 지원하며, 수동으로 장애 조치가 가능하다.
+- 기존 **MySQL 5.0 이후부터 사용이 가능**하며, `DBMS의 성능에 전혀 영향을 주지 않는다.`
+
+
+# MHA 주요 기능
+
+- MySQL 5.6 GTID 지원
+- MySQL 5.6 Multi-Thread Slave 지원
+- MySQL 5.6 Binlog checksum 지원
+- mysqlbinlog streaming host 지원
+- mysqlbinlog 위치 지원
+- ping_type= Select / Connect 외에 Insert 추가
+- master_ip_online_change_script에 --orig_master_is_new_slave, --orig_master_ssh_user and --new_master_ssh_user option 추가
+
+
+# MHA Pre-Requirement
+
+- Master 서버와 Slave 서버는 **사전에 Replication 구성(연결)이 되어있어야 한다.**
+- **모든 서버**는 공개 SSH 키를 사용하여 서로 접속할 수 있어야한다.
+- **모든 서버**는 서로의 MySQL 서버에 접속할 수 있어야한다.
+- **모든 서버**는 동일한 복제용 유저 및 비밀번호를 가져야한다.
+- MySQL 버전은 **5.0 이상이어야한다.**
+- `마스터 후보(candidate_master=1)`는 바이너리 로그를 사용하도록 설정해야한다.
+- **바이너리로그 변수**는 모든 서버에서 동일하게 설정해야한다(replicate-wild, binlog-do-db…).
+
+
+# MHA 시스템 환경구성
+
+| NO | 구분 | IP주소 | 마스터 후보 | 비고 |
+| ------ | ------ | ------ | ------ | ------ |
+| 1 | **MASTER** | 180.180.180.226, **Active** | `O` |  |
+| 2 | **MASTER** | 180.180.180.231, **Active Backup** | `O` |  |
+| 3 | SLAVE | 180.180.180.242, Slave01 | `X` |  |
+| 4 | SLAVE | 180.180.180.243, Slave02 | `X` |  |
+| 5 | SLAVE | 180.180.180.237, Slave03 | `X` |  |
+| 6 | MHA | 180.180.180.238, Slave01 | `X` | Manager |
+| 7 | VIP | 180.180.180.239, Slave01 | `X` | 서비스 VIP |
+
+
+# MHA 시스템 구성
+
+* [ **ALL** ]  mha 사용자 생성 및 sudo 설정
+* [ **MASTER** ] MHA DB 생성
+* [ **ALL** ] SSH RSA-Key 설정
+* [ **ALL** ] MHA Manager / Node 패키지 설치
+* [ **MANAGER** ] MHA Manager 설정 및 스크립트 작성
+* [ **MANAGER** ] SSH / Replication 검증
+
+
+- [ **ALL** ]  mha 사용자 생성 및 sudo 설정
+
+```bash
+# useradd -g mysql mhauser -d /MHA
+# echo ghdwngkstjqj |passwd --stdin mhauser
+
+* Debian/Ubuntu의 경우 --stdin 옵션이 지원되지않으므로, 다음과 같이 수행해야 한다.
+
+# adduser --home /MHA mhauser
+...패스워드를 설정할 것!
+
+# cat /etc/sudoers
+...중략
+
+# Defaults specification
+Cmnd_Alias  MHA = /usr/sbin/ip, /usr/sbin/ifconfig, /usr/sbin/arping <= 추가
+...중략
+
+root    ALL=(ALL)       ALL
+mhauser ALL=(ALL)       NOPASSWD:MHA <= 추가
+
+* Debian/Ubuntu의 경우 ip,ifconfig 명령의 경로가 아래와 같다.
+
+Cmnd_Alias  MHA = /sbin/ip, /sbin/ifconfig, /usr/sbin/arping <= 추가
+```
+
+`!중요` : mha 사용자 계정을 root 유저로 하는 것은 보안적으로 권장되지 않는다.
+
+
+mhauser의 $HOMEDIR 및 기본 디렉토리를 생성한다.
+
+```bash
+# mkdir -p /MHA/conf
+# mkdir -p /MHA/scripts
+# mkdir -p /MHA/logs
+# chown -R mhauser.mysql /MHA
+```
+
+- [ **MASTER** ] MHA DB 생성
+
+Replication에서 사용되어야하는 유저의 DB를 생성한다.
+
+```bash
+MariaDB [(none)]> use mysql;
+Reading table information for completion of table and column names
+You can turn off this feature to get a quicker startup with -A
+
+Database changed
+
+MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.226' identified by '패스워드';
+Query OK, 0 rows affected (0.003 sec)
+
+MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.231' identified by '패스워드';
+Query OK, 0 rows affected (0.001 sec)
+
+MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.242' identified by '패스워드';
+Query OK, 0 rows affected (0.001 sec)
+
+MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.243' identified by '패스워드';
+Query OK, 0 rows affected (0.001 sec)
+
+MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.237' identified by '패스워드';
+Query OK, 0 rows affected (0.001 sec)
+
+MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.238' identified by '패스워드';
+Query OK, 0 rows affected (0.001 sec)
+```
+
+
+- [ **ALL** ] SSH RSA-Key 설정
+
+`!중요` : 이 설정은 특정 서버를 기준으로 rsa-key를 생성하는것이 아닌, 전체 서버가 양방향으로 패스워드없이 접속이 가능하도록 수행하여야 한다.
+
+```bash
+$ id
+mhauser
+
+$ ssh-keygen -t rsa
+Generating public/private rsa key pair.
+...중략
+엔터 3번 입력
+
+$ ssh-copy-id -p 22222 mhauser@현재서버를 제외한 모든 IP
+```
+
+
+- [ **ALL** ] MHA Manager / Node 패키지 설치
+
+`!중요` : MHA 0.57 이하 버전의 경우 Replication 체크 및 Auto Failover 기능이 정상적으로 동작하지 않는다. 따라서 최소 0.57 이상의 버전을 사용해야 한다.
+
+
+1.  **MHA Manager** 패키지 설처
+
+**!참고** : Manager 패키지를 설치하기 전에 **사전에 Node 패키지 설치해야** 한다.
+
+패키지를 다음과 같이 다운로드 한다.
+
+```bash
+# wget http://pds.hongsnet.net:8888/database/mysql/mha4mysql-manager-0.57.tar.gz
+# wget http://pds.hongsnet.net:8888/database/mysql/mha4mysql-node-0.57.tar.gz
+```
+
+다음과 같이 **Linux 배포판에 따라 패키지를 사전에 먼저 설치해야 한다.**
+
+```bash
+
+* Debian 계열
+
+# apt-get install libmodule-install-perl libconfig-tiny-perl # apt install liblog-dispatch-perl libparallel-forkmanager-perl
+
+-------------------------------------------------------------------------------
+
+# redhat 계열
+
+# yum install epel-release
+# yum install perl-Compress-Raw-Bzip2 perl-Compress-Raw-Zlib perl-DBD-MySQL perl-DBI perl-Data-Dumper perl-IO-Compress perl-Net-Daemon perl-PlRPC perl-CPAN perl-core perl-ExtUtils-Manifest
+
+```
+
+`먼저 Node 패키지를 설치`한다.
+
+```bash
+# perl Makefile.PL
+*** Module::AutoInstall version 1.06
+*** Checking for Perl dependencies...
+[Core Features]
+- DBI        ...loaded. (1.636)
+- DBD::mysql ...loaded. (4.041)
+*** Module::AutoInstall configuration finished.
+Checking if your kit is complete...
+Looks good
+Generating a Unix-style Makefile
+Writing Makefile for mha4mysql::node
+Writing MYMETA.yml and MYMETA.json
+```
+
+```bash
+# make
+cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
+cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
+cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
+cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
+cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
+cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
+cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
+cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
+cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
+cp bin/purge_relay_logs blib/script/purge_relay_logs
+"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
+cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
+"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
+cp bin/save_binary_logs blib/script/save_binary_logs
+"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
+cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
+"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
+Manifying 4 pod documents
+
+# make install
+Manifying 4 pod documents
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogManager.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogHeaderParser.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderXid.pm
+Installing /usr/local/share/perl/5.24.1/MHA/SlaveUtil.pm
+Installing /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinder.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFindManager.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderElp.pm
+Installing /usr/local/share/perl/5.24.1/MHA/NodeConst.pm
+Installing /usr/local/man/man1/purge_relay_logs.1p
+Installing /usr/local/man/man1/filter_mysqlbinlog.1p
+Installing /usr/local/man/man1/save_binary_logs.1p
+Installing /usr/local/man/man1/apply_diff_relay_logs.1p
+Installing /usr/local/bin/purge_relay_logs
+Installing /usr/local/bin/apply_diff_relay_logs
+Installing /usr/local/bin/save_binary_logs
+Installing /usr/local/bin/filter_mysqlbinlog
+Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/perllocal.pod
+```
+
+**위와 같이 Node 패키지의 설치가 완료되었으면**, 아래와 같이 Manager 패키지를 설치한다.
+
+```bash
+# perl Makefile.PL
+*** Module::AutoInstall version 1.06
+*** Checking for Perl dependencies...
+[Core Features]
+- DBI                   ...loaded. (1.636)
+- DBD::mysql            ...loaded. (4.041)
+- Time::HiRes           ...loaded. (1.9733)
+- Config::Tiny          ...loaded. (2.23)
+- Log::Dispatch         ...loaded. (2.58)
+- Parallel::ForkManager ...loaded. (1.19)
+- MHA::NodeConst        ...loaded. (0.57)
+*** Module::AutoInstall configuration finished.
+Checking if your kit is complete...
+Looks good
+Generating a Unix-style Makefile
+Writing Makefile for mha4mysql::manager
+Writing MYMETA.yml and MYMETA.json
+```
+
+다음과 같이 make; make install을 수행하면 설치가 완료된다.
+
+```bash
+# make
+cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
+cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
+cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
+cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
+cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
+cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
+cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
+cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
+cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
+cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
+/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
+cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
+/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
+cp bin/purge_relay_logs blib/script/purge_relay_logs
+/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
+cp bin/save_binary_logs blib/script/save_binary_logs
+/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
+Manifying blib/man1/filter_mysqlbinlog.1
+Manifying blib/man1/apply_diff_relay_logs.1
+Manifying blib/man1/purge_relay_logs.1
+Manifying blib/man1/save_binary_logs.1
+
+# make install
+Installing /usr/local/share/perl5/MHA/BinlogManager.pm
+Installing /usr/local/share/perl5/MHA/BinlogPosFindManager.pm
+Installing /usr/local/share/perl5/MHA/BinlogPosFinderXid.pm
+Installing /usr/local/share/perl5/MHA/BinlogHeaderParser.pm
+Installing /usr/local/share/perl5/MHA/BinlogPosFinder.pm
+Installing /usr/local/share/perl5/MHA/NodeUtil.pm
+Installing /usr/local/share/perl5/MHA/BinlogPosFinderElp.pm
+Installing /usr/local/share/perl5/MHA/SlaveUtil.pm
+Installing /usr/local/share/perl5/MHA/NodeConst.pm
+Installing /usr/local/share/man/man1/filter_mysqlbinlog.1
+Installing /usr/local/share/man/man1/apply_diff_relay_logs.1
+Installing /usr/local/share/man/man1/purge_relay_logs.1
+Installing /usr/local/share/man/man1/save_binary_logs.1
+Installing /usr/local/bin/filter_mysqlbinlog
+Installing /usr/local/bin/apply_diff_relay_logs
+Installing /usr/local/bin/purge_relay_logs
+Installing /usr/local/bin/save_binary_logs
+Appending installation info to /usr/lib64/perl5/perllocal.pod
+```
+
+
+2.  **MHA Node** 패키지 설치
+
+```bash
+# perl Makefile.PL
+*** Module::AutoInstall version 1.06
+*** Checking for Perl dependencies...
+[Core Features]
+- DBI        ...loaded. (1.636)
+- DBD::mysql ...loaded. (4.041)
+*** Module::AutoInstall configuration finished.
+Checking if your kit is complete...
+Looks good
+Generating a Unix-style Makefile
+Writing Makefile for mha4mysql::node
+Writing MYMETA.yml and MYMETA.json
+```
+
+```bash
+# make
+cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
+cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
+cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
+cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
+cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
+cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
+cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
+cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
+cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
+cp bin/purge_relay_logs blib/script/purge_relay_logs
+"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
+cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
+"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
+cp bin/save_binary_logs blib/script/save_binary_logs
+"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
+cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
+"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
+Manifying 4 pod documents
+
+# make install
+Manifying 4 pod documents
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogManager.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogHeaderParser.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderXid.pm
+Installing /usr/local/share/perl/5.24.1/MHA/SlaveUtil.pm
+Installing /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinder.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFindManager.pm
+Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderElp.pm
+Installing /usr/local/share/perl/5.24.1/MHA/NodeConst.pm
+Installing /usr/local/man/man1/purge_relay_logs.1p
+Installing /usr/local/man/man1/filter_mysqlbinlog.1p
+Installing /usr/local/man/man1/save_binary_logs.1p
+Installing /usr/local/man/man1/apply_diff_relay_logs.1p
+Installing /usr/local/bin/purge_relay_logs
+Installing /usr/local/bin/apply_diff_relay_logs
+Installing /usr/local/bin/save_binary_logs
+Installing /usr/local/bin/filter_mysqlbinlog
+Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/perllocal.pod
+```
+
+- [ **MANAGER** ] MHA Manager 설정 및 스크립트 작성
+
+```bash
+# cat /MHA/conf/mha.cnf
+[server default]
+
+user=mhauser
+password=패스워드
+
+ssh_user=mhauser
+ssh_port=ssh_포트번호
+
+repl_user=mhauser
+repl_password=패스워드
+
+# working directory on the manager
+manager_workdir=/MHA/log
+
+# manager log file
+manager_log=/MHA/log/manager.log
+
+master_ip_failover_script=/usr/local/bin/master_ip_failover
+master_ip_online_change_script=/usr/local/bin/master_ip_online_change
+
+ping_interval=3
+
+[server1]
+hostname=180.180.180.226
+master_binlog_dir=/var/lib/mysql
+candidate_master=1
+port=3306
+
+[server2]
+hostname=180.180.180.231
+master_binlog_dir=/var/lib/mysql
+candidate_master=1
+port=3306
+
+[server3]
+hostname=180.180.180.242
+master_binlog_dir=/var/lib/mysql
+no_master=1
+port=3306
+
+[server4]
+hostname=180.180.180.243
+master_binlog_dir=/var/lib/mysql
+no_master=1
+port=3306
+
+[server5]
+hostname=180.180.180.237
+master_binlog_dir=/var/lib/mysql
+no_master=1
+port=3306
+```
+
+> ** 설정파라미터 설명 
+
+- user, password : DB계정 및 암호
+- ssh_user : ssh용 접속 계정
+- repl_user,repl_password : replication 계정 및 암호
+- manager_workdir : 매니저서버의 작업 공간
+- manager_log : 매니저서버의 로그파일 위치
+- **master_ip_failover_script** : 자동으로 페일오버를 수행하는 스크립트
+- **master_ip_online_change_script** : 수동으로 페일오버를 수행하는 스크립트로 변수값을 명시해야 한다.
+- **candidate_master = 1** : 장애 발생시 master가 될 지를 결정하는 설정 (후보 설정)
+- no_master = 1 : master의 후보가 아님을 설정
+
+
+**3. master_ip_failover 스크립트 작성**
+
+**!참고** : 이 스크립트는 Master 시스템의 이슈발생 시 Master 후보군 서버로의 서비스 이관(Fail-Over)을 담당하는 스크립트 이다.
+
+```bash
+# chmod 755 /usr/local/bin/master_ip_failover
+```
+> 첨부된 master_ip_failover 스크립트 참조
+
+
+**4. master_ip_online_change**
+
+**!참고** : 이 스크립트는 Master 시스템이 유지보수 등에 대한 이슈로 인해 수동 Fail-Over를 담당하는 스크립트 이다.
+
+```bash
+# chmod 755 /usr/local/bin/master_ip_online_change
+```
+> 첨부된 master_ip_online_chage 스크립트 참조
+
+
+- [ **MANAGER** ] SSH / Replication 검증
+
+
+**1. 전체 구성 시스템의 SSH 접속 테스트 수행**
+
+```bash
+# su - mhauser
+
+$ masterha_check_ssh --conf=/MHA/conf/mha.cnf
+Tue Dec  1 00:26:34 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
+Tue Dec  1 00:26:34 2020 - [info] Reading application default configuration from /MHA/conf/mha.cnf..
+Tue Dec  1 00:26:34 2020 - [info] Reading server configuration from /MHA/conf/mha.cnf..
+Tue Dec  1 00:26:34 2020 - [info] Starting SSH connection tests..
+...중략
+
+Tue Dec  1 00:26:39 2020 - [info] All SSH connection tests passed successfully.
+```
+
+**2. Replication 구성테스트 수행**
+
+```bash
+# su - mhauser
+
+$ masterha_check_repl --conf=/MHA/conf/mha.cnf
+Tue Dec  1 00:31:38 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
+Tue Dec  1 00:31:38 2020 - [info] Reading application default configuration from /MHA/conf/mha.cnf..
+Tue Dec  1 00:31:38 2020 - [info] Reading server configuration from /MHA/conf/mha.cnf..
+Tue Dec  1 00:31:38 2020 - [info] MHA::MasterMonitor version 0.57.
+...중략
+
+IN SCRIPT TEST====sudo /usr/sbin/ifconfig eth1:0 down==sudo /usr/sbin/ifconfig eth1:0 180.180.180.239 netmask 255.255.255.0 broadcast 180.180.180.255 up===
+
+Checking the Status of the script.. OK
+Tue Dec  1 01:17:41 2020 - [info]  OK.
+Tue Dec  1 01:17:41 2020 - [warning] shutdown_script is not defined.
+Tue Dec  1 01:17:41 2020 - [info] Got exit code 0 (Not master dead).
+
+MySQL Replication Health is OK.
+```
+
+
+# ISSUED
+
+* [ **Issue 1** ] : [error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. Redundant argument in sprintf at /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm line 184.
+
+> MariaDB Version 차이에 따른 이슈로써 다음과 같이 Perl Module의 내용을 수정한다.
+
+```bash
+# cat /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
+...중략
+
+sub parse_mysql_version($) {
+my $str = shift;
+#2020-12-29
+($str) = $str =~ m/^[^-]*/g; <= 추가
+my $result = sprintf( '%03d%03d%03d', $str =~ m/(\d+)/g );
+return $result;
+}
+
+sub parse_mysql_major_version($) {
+my $str = shift;
+#2020-12-29
+($str) = $str =~ m/^[^-]*/g; <= 추가
+#my $result = sprintf( '%03d%03d', $str =~ m/(\d+)/g ); <= 주석
+my $result = sprintf( '%03d%03d%03d', $str =~ m/(\d+)/g ); <= 변경
+return $result;
+}
+```
+
+* [ **Issue 2** ] : mysqlbinlog: unknown variable 'default-character-set=utf8mb4'
+mysqlbinlog version command failed with rc 7:0, please verify PATH, LD_LIBRARY_PATH, and client options
+at /usr/local/bin/apply_diff_relay_logs line 493.
+[error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln208] Slaves settings check failed!
+[error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln416] Slave configuration failed.
+
+> my.cnf 파일에 잘못된 캐릭터 셋 파라미터가 설정되어서 발생된 에러로써 다음과 같이 주석처리하거나 제거해야 한다.
+
+```bash
+# cat my.cnf
+...중략
+
+[client]
+port            = 3306
+socket          = /var/run/mysqld/mysqld.sock
+#default-character-set=utf8mb4 <= 이 파라미터가 원인이다. 주석처리를 해야한다.
+```
+
+

DBMS/MHA/master_ip_failover

+#!/usr/bin/env perl
+# use strict;
+# use warnings FATAL => 'all';
+use Getopt::Long;
+
+my (
+        $command,
+        $ssh_user,
+        $orig_master_host,
+        $orig_master_ip,
+        $orig_master_port,
+        $new_master_host,
+        $new_master_ip,
+        $new_master_port,
+        $new_master_user,
+        $new_master_password
+);
+
+my $vip = '180.180.180.239';
+my $netmask = '255.255.255.0';
+my $broadcast = '180.180.180.255';
+my $key = "0";
+my $ssh_start_vip   = "sudo /usr/sbin/ifconfig eth1:$key $vip netmask $netmask broadcast $broadcast up";
+my $ssh_stop_vip    = "sudo /usr/sbin/ifconfig eth1:$key down";
+my $ssh_mac_refresh = "sudo /usr/sbin/arping -c 4 -A -I eth1:$key $vip";
+
+GetOptions(
+        'command=s'          => \$command,
+        'ssh_user=s'         => \$ssh_user,
+        'orig_master_host=s' => \$orig_master_host,
+        'orig_master_ip=s'   => \$orig_master_ip,
+        'orig_master_port=i' => \$orig_master_port,
+        'new_master_host=s'  => \$new_master_host,
+        'new_master_ip=s'    => \$new_master_ip,
+        'new_master_port=i'  => \$new_master_port,
+        'new_master_user=s'  => \$new_master_user,
+        'new_master_password=s' => \$new_master_password
+);
+
+exit &main();
+
+sub main {
+
+ print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
+
+ if ( $command eq "stop" || $command eq "stopssh" ) {
+
+        print $command;
+
+     # $orig_master_host, $orig_master_ip, $orig_master_port are passed.
+     # If you manage master ip address at global catalog database,
+     # invalidate orig_master_ip here.
+     my $exit_code = 1;
+     eval {
+         print "Disabling the VIP on old master: $orig_master_host \n";
+         &stop_vip();
+         $exit_code = 0;
+     };
+     if ($@) {
+         warn "Got Error: $@\n";
+         exit $exit_code;
+     }
+     exit $exit_code;
+ }
+ elsif ( $command eq "start" ) {
+
+     # all arguments are passed.
+     # If you manage master ip address at global catalog database,
+     # activate new_master_ip here.
+     # You can also grant write access (create user, set read_only=0, etc) here.
+     my $exit_code = 10;
+     eval {
+         print "Enabling the VIP - $vip on the new master - $new_master_host \n";
+         &start_vip();
+         $exit_code = 0;
+     };
+     if ($@) {
+         warn $@;
+         exit $exit_code;
+     }
+     exit $exit_code;
+ }
+ elsif ( $command eq "status" ) {
+     print "Checking the Status of the script.. OK \n";
+        `ssh -p ssh_포트번호 $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;
+     exit 0;
+ }
+ else {
+     &usage();
+     exit 1;
+ }
+}
+
+# A simple system call that enable the VIP on the new master
+sub start_vip() {
+        `ssh -p ssh_포트번호 $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
+        `ssh -p ssh_포트번호 $ssh_user\@$new_master_host \" $ssh_mac_refresh\"`;
+}
+# A simple system call that disable the VIP on the old_master
+sub stop_vip() {
+        `ssh -p ssh_포트번호 $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
+}
+
+sub usage {
+ print
+ "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
+}

DBMS/MHA/master_ip_online_change

+#!/usr/bin/env perl
+
+#  Copyright (C) 2011 DeNA Co.,Ltd.
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; either version 2 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#   along with this program; if not, write to the Free Software
+#  Foundation, Inc.,
+#  51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+
+## Note: This is a sample script and is not complete. Modify the script based on your environment.
+
+use strict;
+use warnings FATAL => 'all';
+
+use Getopt::Long;
+use MHA::DBHelper;
+use MHA::NodeUtil;
+use Time::HiRes qw( sleep gettimeofday tv_interval );
+use Data::Dumper;
+
+my $_tstart;
+my $_running_interval = 0.1;
+
+my (
+  $command,              $orig_master_is_new_slave, $orig_master_host,
+  $orig_master_ip,       $orig_master_port,         $orig_master_user,
+  $orig_master_password, $orig_master_ssh_user,     $new_master_host,
+  $new_master_ip,        $new_master_port,          $new_master_user,
+  $new_master_password,  $new_master_ssh_user,      $orig_master_ssh_port,
+  $new_master_ssh_port,
+);
+
+my $vip = '180.180.180.239';
+my $netmask = '255.255.255.0';
+my $broadcast = '180.180.180.255';
+my $key = "0";
+my $ssh_start_vip   = "sudo /usr/sbin/ifconfig eth1:$key $vip netmask $netmask broadcast $broadcast up";
+my $ssh_stop_vip    = "sudo /usr/sbin/ifconfig eth1:$key down";
+my $ssh_mac_refresh = "sudo /usr/sbin/arping -c 4 -A -I eth1:$key $vip";
+
+GetOptions(
+  'command=s'                => \$command,
+  'orig_master_is_new_slave' => \$orig_master_is_new_slave,
+  'orig_master_host=s'       => \$orig_master_host,
+  'orig_master_ip=s'         => \$orig_master_ip,
+  'orig_master_port=i'       => \$orig_master_port,
+  'orig_master_user=s'       => \$orig_master_user,
+  'orig_master_password=s'   => \$orig_master_password,
+  'orig_master_ssh_user=s'   => \$orig_master_ssh_user,
+  'new_master_host=s'        => \$new_master_host,
+  'new_master_ip=s'          => \$new_master_ip,
+  'new_master_port=i'        => \$new_master_port,
+  'new_master_user=s'        => \$new_master_user,
+  'new_master_password=s'    => \$new_master_password,
+  'new_master_ssh_user=s'    => \$new_master_ssh_user,
+  'orig_master_ssh_port=i'   => \$orig_master_ssh_port,
+  'new_master_ssh_port=i'    => \$new_master_ssh_port,
+);
+
+exit &main();
+
+sub current_time_us {
+  my ( $sec, $microsec ) = gettimeofday();
+  my $curdate = localtime($sec);
+  return $curdate . " " . sprintf( "%06d", $microsec );
+}
+
+sub sleep_until {
+  my $elapsed = tv_interval($_tstart);
+  if ( $_running_interval > $elapsed ) {
+    sleep( $_running_interval - $elapsed );
+  }
+}
+
+sub get_threads_util {
+  my $dbh                    = shift;
+  my $my_connection_id       = shift;
+  my $running_time_threshold = shift;
+  my $type                   = shift;
+  $running_time_threshold = 0 unless ($running_time_threshold);
+  $type                   = 0 unless ($type);
+  my @threads;
+
+  my $sth = $dbh->prepare("SHOW PROCESSLIST");
+  $sth->execute();
+
+  while ( my $ref = $sth->fetchrow_hashref() ) {
+    my $id         = $ref->{Id};
+    my $user       = $ref->{User};
+    my $host       = $ref->{Host};
+    my $command    = $ref->{Command};
+    my $state      = $ref->{State};
+    my $query_time = $ref->{Time};
+    my $info       = $ref->{Info};
+    $info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);
+    next if ( $my_connection_id == $id );
+    next if ( defined($query_time) && $query_time < $running_time_threshold );
+    next if ( defined($command)    && $command eq "Binlog Dump" );
+    next if ( defined($user)       && $user eq "system user" );
+    next
+      if ( defined($command)
+      && $command eq "Sleep"
+      && defined($query_time)
+      && $query_time >= 1 );
+
+    if ( $type >= 1 ) {
+      next if ( defined($command) && $command eq "Sleep" );
+      next if ( defined($command) && $command eq "Connect" );
+    }
+
+    if ( $type >= 2 ) {
+      next if ( defined($info) && $info =~ m/^select/i );
+      next if ( defined($info) && $info =~ m/^show/i );
+    }
+
+    push @threads, $ref;
+  }
+  return @threads;
+}
+
+# A simple system call that enable the VIP on the new master
+sub start_vip() {
+        `ssh -p ssh_포트번호 $new_master_ssh_user\@$new_master_host \" $ssh_start_vip \"`;
+        `ssh -p ssh_포트번호 $new_master_ssh_user\@$new_master_host \" $ssh_mac_refresh\"`;
+}
+# A simple system call that disable the VIP on the old_master
+sub stop_vip() {
+        `ssh -p ssh_포트번호 $orig_master_ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
+}
+
+sub main {
+  if ( $command eq "stop" ) {
+    ## Gracefully killing connections on the current master
+    # 1. Set read_only= 1 on the new master
+    # 2. DROP USER so that no app user can establish new connections
+    # 3. Set read_only= 1 on the current master
+    # 4. Kill current queries
+    # * Any database access failure will result in script die.
+    my $exit_code = 1;
+    eval {
+      ## Setting read_only=1 on the new master (to avoid accident)
+      my $new_master_handler = new MHA::DBHelper();
+
+      # args: hostname, port, user, password, raise_error(die_on_error)_or_not
+      $new_master_handler->connect( $new_master_ip, $new_master_port,
+        $new_master_user, $new_master_password, 1 );
+      print current_time_us() . " Set read_only on the new master.. ";
+      $new_master_handler->enable_read_only();
+      if ( $new_master_handler->is_read_only() ) {
+        print "ok.\n";
+      }
+      else {
+        die "Failed!\n";
+      }
+      $new_master_handler->disconnect();
+
+      # Connecting to the orig master, die if any database error happens
+      my $orig_master_handler = new MHA::DBHelper();
+      $orig_master_handler->connect( $orig_master_ip, $orig_master_port,
+        $orig_master_user, $orig_master_password, 1 );
+
+      ## Drop application user so that nobody can connect. Disabling per-session binlog beforehand
+      $orig_master_handler->disable_log_bin_local();
+      print current_time_us() . " Drpping app user on the orig master..\n";
+      #FIXME_xxx_drop_app_user($orig_master_handler);
+
+      ## Waiting for N * 100 milliseconds so that current connections can exit
+      my $time_until_read_only = 15;
+      $_tstart = [gettimeofday];
+      my @threads = get_threads_util( $orig_master_handler->{dbh},
+        $orig_master_handler->{connection_id} );
+      while ( $time_until_read_only > 0 && $#threads >= 0 ) {
+        if ( $time_until_read_only % 5 == 0 ) {
+          printf
+"%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n",
+            current_time_us(), $#threads + 1, $time_until_read_only * 100;
+          if ( $#threads < 5 ) {
+            print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
+              foreach (@threads);
+          }
+        }
+        sleep_until();
+        $_tstart = [gettimeofday];
+        $time_until_read_only--;
+        @threads = get_threads_util( $orig_master_handler->{dbh},
+          $orig_master_handler->{connection_id} );
+      }
+
+      ## Setting read_only=1 on the current master so that nobody(except SUPER) can write
+      print current_time_us() . " Set read_only=1 on the orig master.. ";
+      $orig_master_handler->enable_read_only();
+      if ( $orig_master_handler->is_read_only() ) {
+        print "ok.\n";
+      }
+      else {
+        die "Failed!\n";
+      }
+
+      ## Waiting for M * 100 milliseconds so that current update queries can complete
+      my $time_until_kill_threads = 5;
+      @threads = get_threads_util( $orig_master_handler->{dbh},
+        $orig_master_handler->{connection_id} );
+      while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {
+        if ( $time_until_kill_threads % 5 == 0 ) {
+          printf
+"%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n",
+            current_time_us(), $#threads + 1, $time_until_kill_threads * 100;
+          if ( $#threads < 5 ) {
+            print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
+              foreach (@threads);
+          }
+        }
+        sleep_until();
+        $_tstart = [gettimeofday];
+        $time_until_kill_threads--;
+        @threads = get_threads_util( $orig_master_handler->{dbh},
+          $orig_master_handler->{connection_id} );
+      }
+
+      ## Terminating all threads
+      print current_time_us() . " Killing all application threads..\n";
+      $orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );
+      print current_time_us() . " done.\n";
+      $orig_master_handler->enable_log_bin_local();
+      $orig_master_handler->disconnect();
+
+      ## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK
+      &stop_vip();
+      $exit_code = 0;
+    };
+    if ($@) {
+      warn "Got Error: $@\n";
+      exit $exit_code;
+    }
+    exit $exit_code;
+  }
+  elsif ( $command eq "start" ) {
+    ## Activating master ip on the new master
+    # 1. Create app user with write privileges
+    # 2. Moving backup script if needed
+    # 3. Register new master's ip to the catalog database
+
+# We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery.
+# If exit code is 0 or 10, MHA does not abort
+    my $exit_code = 10;
+    eval {
+      my $new_master_handler = new MHA::DBHelper();
+
+      # args: hostname, port, user, password, raise_error_or_not
+      $new_master_handler->connect( $new_master_ip, $new_master_port,
+        $new_master_user, $new_master_password, 1 );
+
+      ## Set read_only=0 on the new master
+      $new_master_handler->disable_log_bin_local();
+      print current_time_us() . " Set read_only=0 on the new master.\n";
+      $new_master_handler->disable_read_only();
+
+      ## Creating an app user on the new master
+      print current_time_us() . " Creating app user on the new master..\n";
+      #FIXME_xxx_create_app_user($new_master_handler);
+      $new_master_handler->enable_log_bin_local();
+      $new_master_handler->disconnect();
+
+      ## Update master ip on the catalog database, etc
+      &start_vip();
+      $exit_code = 0;
+    };
+    if ($@) {
+      warn "Got Error: $@\n";
+      exit $exit_code;
+    }
+    exit $exit_code;
+  }
+  elsif ( $command eq "status" ) {
+
+    # do nothing
+    exit 0;
+  }
+  else {
+    &usage();
+    exit 1;
+  }
+}
+
+sub usage {
+  print
+"Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
+  die;
+}