logo

MHA 개요

  • Yoshinori Matsunobu에 의해 2011년 07월 23일 MHA 0.50이 Release 되었다.
  • MHA는 최소한의 Down Time으로 Master를 장애조치 하고, Slave를 새로운 Master로 선출하여 서비스 가동이 정상적으로 수행되도록 하는 Auto Failover 솔루션이다.
  • 각 노드(Master 및 Slave)를 자동으로 전환하며, Master와 Slave의 데이터를 동일하게 유지한다.
  • 자동 Master 모니터와 Failover를 지원한다.
  • 대화형 Master Failover 및 비대화형 Master Failover를 지원하며, 수동으로 장애 조치가 가능하다.
  • 기존 MySQL 5.0 이후부터 사용이 가능하며, DBMS의 성능에 전혀 영향을 주지 않는다.

MHA 주요 기능

  • MySQL 5.6 GTID 지원
  • MySQL 5.6 Multi-Thread Slave 지원
  • MySQL 5.6 Binlog checksum 지원
  • mysqlbinlog streaming host 지원
  • mysqlbinlog 위치 지원
  • ping_type= Select / Connect 외에 Insert 추가
  • master_ip_online_change_script에 --orig_master_is_new_slave, --orig_master_ssh_user and --new_master_ssh_user option 추가

MHA Pre-Requirement

  • Master 서버와 Slave 서버는 사전에 Replication 구성(연결)이 되어있어야 한다.
  • 모든 서버는 공개 SSH 키를 사용하여 서로 접속할 수 있어야한다.
  • 모든 서버는 서로의 MySQL 서버에 접속할 수 있어야한다.
  • 모든 서버는 동일한 복제용 유저 및 비밀번호를 가져야한다.
  • MySQL 버전은 5.0 이상이어야한다.
  • 마스터 후보(candidate_master=1)는 바이너리 로그를 사용하도록 설정해야한다.
  • 바이너리로그 변수는 모든 서버에서 동일하게 설정해야한다(replicate-wild, binlog-do-db…).

MHA 시스템 환경구성

NO 구분 IP주소 마스터 후보 비고
1 MASTER 180.180.180.226, Active O
2 MASTER 180.180.180.231, Active Backup O
3 SLAVE 180.180.180.242, Slave01 X
4 SLAVE 180.180.180.243, Slave02 X
5 SLAVE 180.180.180.237, Slave03 X
6 MHA 180.180.180.238, Slave01 X Manager
7 VIP 180.180.180.239, Slave01 X 서비스 VIP

MHA 시스템 구성

  • [ ALL ] mha 사용자 생성 및 sudo 설정
  • [ MASTER ] MHA DB 생성
  • [ ALL ] SSH RSA-Key 설정
  • [ ALL ] MHA Manager / Node 패키지 설치
  • [ MANAGER ] MHA Manager 설정 및 스크립트 작성
  • [ MANAGER ] SSH / Replication 검증
  • [ ALL ] mha 사용자 생성 및 sudo 설정
# useradd -g mysql mhauser -d /MHA
# echo ghdwngkstjqj |passwd --stdin mhauser

* Debian/Ubuntu의 경우 --stdin 옵션이 지원되지않으므로, 다음과 같이 수행해야 한다.

# adduser --home /MHA mhauser
...패스워드를 설정할 것!

# cat /etc/sudoers
...중략

# Defaults specification
Cmnd_Alias  MHA = /usr/sbin/ip, /usr/sbin/ifconfig, /usr/sbin/arping <= 추가
...중략

root    ALL=(ALL)       ALL
mhauser ALL=(ALL)       NOPASSWD:MHA <= 추가

* Debian/Ubuntu의 경우 ip,ifconfig 명령의 경로가 아래와 같다.

Cmnd_Alias  MHA = /sbin/ip, /sbin/ifconfig, /usr/sbin/arping <= 추가

!중요 : mha 사용자 계정을 root 유저로 하는 것은 보안적으로 권장되지 않는다.

mhauser의 $HOMEDIR 및 기본 디렉토리를 생성한다.

# mkdir -p /MHA/conf
# mkdir -p /MHA/scripts
# mkdir -p /MHA/logs
# chown -R mhauser.mysql /MHA
  • [ MASTER ] MHA DB 생성

Replication에서 사용되어야하는 유저의 DB를 생성한다.

MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.226' identified by '패스워드';
Query OK, 0 rows affected (0.003 sec)

MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.231' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)

MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.242' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)

MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.243' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)

MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.237' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)

MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.238' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)
  • [ ALL ] SSH RSA-Key 설정

!중요 : 이 설정은 특정 서버를 기준으로 rsa-key를 생성하는것이 아닌, 전체 서버가 양방향으로 패스워드없이 접속이 가능하도록 수행하여야 한다.

$ id
mhauser

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
...중략
엔터 3번 입력

$ ssh-copy-id -p 22222 mhauser@현재서버를 제외한 모든 IP
  • [ ALL ] MHA Manager / Node 패키지 설치

!중요 : MHA 0.57 이하 버전의 경우 Replication 체크 및 Auto Failover 기능이 정상적으로 동작하지 않는다. 따라서 최소 0.57 이상의 버전을 사용해야 한다.

  1. MHA Manager 패키지 설처

!참고 : Manager 패키지를 설치하기 전에 사전에 Node 패키지 설치해야 한다.

패키지를 다음과 같이 다운로드 한다.

# wget http://pds.hongsnet.net:8888/database/mysql/mha4mysql-manager-0.57.tar.gz
# wget http://pds.hongsnet.net:8888/database/mysql/mha4mysql-node-0.57.tar.gz

다음과 같이 Linux 배포판에 따라 패키지를 사전에 먼저 설치해야 한다.


* Debian 계열

# apt-get install libmodule-install-perl libconfig-tiny-perl # apt install liblog-dispatch-perl libparallel-forkmanager-perl

-------------------------------------------------------------------------------

# redhat 계열

# yum install epel-release
# yum install perl-Compress-Raw-Bzip2 perl-Compress-Raw-Zlib perl-DBD-MySQL perl-DBI perl-Data-Dumper perl-IO-Compress perl-Net-Daemon perl-PlRPC perl-CPAN perl-core perl-ExtUtils-Manifest

먼저 Node 패키지를 설치한다.

# perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI        ...loaded. (1.636)
- DBD::mysql ...loaded. (4.041)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for mha4mysql::node
Writing MYMETA.yml and MYMETA.json
# make
cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
cp bin/purge_relay_logs blib/script/purge_relay_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
cp bin/save_binary_logs blib/script/save_binary_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
Manifying 4 pod documents

# make install
Manifying 4 pod documents
Installing /usr/local/share/perl/5.24.1/MHA/BinlogManager.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogHeaderParser.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderXid.pm
Installing /usr/local/share/perl/5.24.1/MHA/SlaveUtil.pm
Installing /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinder.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFindManager.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderElp.pm
Installing /usr/local/share/perl/5.24.1/MHA/NodeConst.pm
Installing /usr/local/man/man1/purge_relay_logs.1p
Installing /usr/local/man/man1/filter_mysqlbinlog.1p
Installing /usr/local/man/man1/save_binary_logs.1p
Installing /usr/local/man/man1/apply_diff_relay_logs.1p
Installing /usr/local/bin/purge_relay_logs
Installing /usr/local/bin/apply_diff_relay_logs
Installing /usr/local/bin/save_binary_logs
Installing /usr/local/bin/filter_mysqlbinlog
Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/perllocal.pod

위와 같이 Node 패키지의 설치가 완료되었으면, 아래와 같이 Manager 패키지를 설치한다.

# perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI                   ...loaded. (1.636)
- DBD::mysql            ...loaded. (4.041)
- Time::HiRes           ...loaded. (1.9733)
- Config::Tiny          ...loaded. (2.23)
- Log::Dispatch         ...loaded. (2.58)
- Parallel::ForkManager ...loaded. (1.19)
- MHA::NodeConst        ...loaded. (0.57)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for mha4mysql::manager
Writing MYMETA.yml and MYMETA.json

다음과 같이 make; make install을 수행하면 설치가 완료된다.

# make
cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
cp bin/purge_relay_logs blib/script/purge_relay_logs
/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
cp bin/save_binary_logs blib/script/save_binary_logs
/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
Manifying blib/man1/filter_mysqlbinlog.1
Manifying blib/man1/apply_diff_relay_logs.1
Manifying blib/man1/purge_relay_logs.1
Manifying blib/man1/save_binary_logs.1

# make install
Installing /usr/local/share/perl5/MHA/BinlogManager.pm
Installing /usr/local/share/perl5/MHA/BinlogPosFindManager.pm
Installing /usr/local/share/perl5/MHA/BinlogPosFinderXid.pm
Installing /usr/local/share/perl5/MHA/BinlogHeaderParser.pm
Installing /usr/local/share/perl5/MHA/BinlogPosFinder.pm
Installing /usr/local/share/perl5/MHA/NodeUtil.pm
Installing /usr/local/share/perl5/MHA/BinlogPosFinderElp.pm
Installing /usr/local/share/perl5/MHA/SlaveUtil.pm
Installing /usr/local/share/perl5/MHA/NodeConst.pm
Installing /usr/local/share/man/man1/filter_mysqlbinlog.1
Installing /usr/local/share/man/man1/apply_diff_relay_logs.1
Installing /usr/local/share/man/man1/purge_relay_logs.1
Installing /usr/local/share/man/man1/save_binary_logs.1
Installing /usr/local/bin/filter_mysqlbinlog
Installing /usr/local/bin/apply_diff_relay_logs
Installing /usr/local/bin/purge_relay_logs
Installing /usr/local/bin/save_binary_logs
Appending installation info to /usr/lib64/perl5/perllocal.pod
  1. MHA Node 패키지 설치
# perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI        ...loaded. (1.636)
- DBD::mysql ...loaded. (4.041)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for mha4mysql::node
Writing MYMETA.yml and MYMETA.json
# make
cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
cp bin/purge_relay_logs blib/script/purge_relay_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
cp bin/save_binary_logs blib/script/save_binary_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
Manifying 4 pod documents

# make install
Manifying 4 pod documents
Installing /usr/local/share/perl/5.24.1/MHA/BinlogManager.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogHeaderParser.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderXid.pm
Installing /usr/local/share/perl/5.24.1/MHA/SlaveUtil.pm
Installing /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinder.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFindManager.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderElp.pm
Installing /usr/local/share/perl/5.24.1/MHA/NodeConst.pm
Installing /usr/local/man/man1/purge_relay_logs.1p
Installing /usr/local/man/man1/filter_mysqlbinlog.1p
Installing /usr/local/man/man1/save_binary_logs.1p
Installing /usr/local/man/man1/apply_diff_relay_logs.1p
Installing /usr/local/bin/purge_relay_logs
Installing /usr/local/bin/apply_diff_relay_logs
Installing /usr/local/bin/save_binary_logs
Installing /usr/local/bin/filter_mysqlbinlog
Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/perllocal.pod
  • [ MANAGER ] MHA Manager 설정 및 스크립트 작성
# cat /MHA/conf/mha.cnf
[server default]

user=mhauser
password=패스워드

ssh_user=mhauser
ssh_port=ssh_포트번호

repl_user=mhauser
repl_password=패스워드

# working directory on the manager
manager_workdir=/MHA/log

# manager log file
manager_log=/MHA/log/manager.log

master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change

ping_interval=3

[server1]
hostname=180.180.180.226
master_binlog_dir=/var/lib/mysql
candidate_master=1
port=3306

[server2]
hostname=180.180.180.231
master_binlog_dir=/var/lib/mysql
candidate_master=1
port=3306

[server3]
hostname=180.180.180.242
master_binlog_dir=/var/lib/mysql
no_master=1
port=3306

[server4]
hostname=180.180.180.243
master_binlog_dir=/var/lib/mysql
no_master=1
port=3306

[server5]
hostname=180.180.180.237
master_binlog_dir=/var/lib/mysql
no_master=1
port=3306

** 설정파라미터 설명

  • user, password : DB계정 및 암호
  • ssh_user : ssh용 접속 계정
  • repl_user,repl_password : replication 계정 및 암호
  • manager_workdir : 매니저서버의 작업 공간
  • manager_log : 매니저서버의 로그파일 위치
  • master_ip_failover_script : 자동으로 페일오버를 수행하는 스크립트
  • master_ip_online_change_script : 수동으로 페일오버를 수행하는 스크립트로 변수값을 명시해야 한다.
  • candidate_master = 1 : 장애 발생시 master가 될 지를 결정하는 설정 (후보 설정)
  • no_master = 1 : master의 후보가 아님을 설정

3. master_ip_failover 스크립트 작성

!참고 : 이 스크립트는 Master 시스템의 이슈발생 시 Master 후보군 서버로의 서비스 이관(Fail-Over)을 담당하는 스크립트 이다.

# chmod 755 /usr/local/bin/master_ip_failover

첨부된 master_ip_failover 스크립트 참조

4. master_ip_online_change

!참고 : 이 스크립트는 Master 시스템이 유지보수 등에 대한 이슈로 인해 수동 Fail-Over를 담당하는 스크립트 이다.

# chmod 755 /usr/local/bin/master_ip_online_change

첨부된 master_ip_online_chage 스크립트 참조

  • [ MANAGER ] SSH / Replication 검증

1. 전체 구성 시스템의 SSH 접속 테스트 수행

# su - mhauser

$ masterha_check_ssh --conf=/MHA/conf/mha.cnf
Tue Dec  1 00:26:34 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Dec  1 00:26:34 2020 - [info] Reading application default configuration from /MHA/conf/mha.cnf..
Tue Dec  1 00:26:34 2020 - [info] Reading server configuration from /MHA/conf/mha.cnf..
Tue Dec  1 00:26:34 2020 - [info] Starting SSH connection tests..
...중략

Tue Dec  1 00:26:39 2020 - [info] All SSH connection tests passed successfully.

2. Replication 구성테스트 수행

# su - mhauser

$ masterha_check_repl --conf=/MHA/conf/mha.cnf
Tue Dec  1 00:31:38 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Dec  1 00:31:38 2020 - [info] Reading application default configuration from /MHA/conf/mha.cnf..
Tue Dec  1 00:31:38 2020 - [info] Reading server configuration from /MHA/conf/mha.cnf..
Tue Dec  1 00:31:38 2020 - [info] MHA::MasterMonitor version 0.57.
...중략

IN SCRIPT TEST====sudo /usr/sbin/ifconfig eth1:0 down==sudo /usr/sbin/ifconfig eth1:0 180.180.180.239 netmask 255.255.255.0 broadcast 180.180.180.255 up===

Checking the Status of the script.. OK
Tue Dec  1 01:17:41 2020 - [info]  OK.
Tue Dec  1 01:17:41 2020 - [warning] shutdown_script is not defined.
Tue Dec  1 01:17:41 2020 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

ISSUED

  • [ Issue 1 ] : [error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. Redundant argument in sprintf at /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm line 184.

MariaDB Version 차이에 따른 이슈로써 다음과 같이 Perl Module의 내용을 수정한다.

# cat /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
...중략

sub parse_mysql_version($) {
my $str = shift;
#2020-12-29
($str) = $str =~ m/^[^-]*/g; <= 추가
my $result = sprintf( '%03d%03d%03d', $str =~ m/(\d+)/g );
return $result;
}

sub parse_mysql_major_version($) {
my $str = shift;
#2020-12-29
($str) = $str =~ m/^[^-]*/g; <= 추가
#my $result = sprintf( '%03d%03d', $str =~ m/(\d+)/g ); <= 주석
my $result = sprintf( '%03d%03d%03d', $str =~ m/(\d+)/g ); <= 변경
return $result;
}
  • [ Issue 2 ] : mysqlbinlog: unknown variable 'default-character-set=utf8mb4' mysqlbinlog version command failed with rc 7:0, please verify PATH, LD_LIBRARY_PATH, and client options at /usr/local/bin/apply_diff_relay_logs line 493. [error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln208] Slaves settings check failed! [error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln416] Slave configuration failed.

my.cnf 파일에 잘못된 캐릭터 셋 파라미터가 설정되어서 발생된 에러로써 다음과 같이 주석처리하거나 제거해야 한다.

# cat my.cnf
...중략

[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
#default-character-set=utf8mb4 <= 이 파라미터가 원인이다. 주석처리를 해야한다.