MHA 개요
- Yoshinori Matsunobu에 의해 2011년 07월 23일 MHA 0.50이 Release 되었다.
- MHA는 최소한의 Down Time으로 Master를 장애조치 하고, Slave를 새로운 Master로 선출하여 서비스 가동이 정상적으로 수행되도록 하는 Auto Failover 솔루션이다.
- 각 노드(Master 및 Slave)를 자동으로 전환하며, Master와 Slave의 데이터를 동일하게 유지한다.
- 자동 Master 모니터와 Failover를 지원한다.
- 대화형 Master Failover 및 비대화형 Master Failover를 지원하며, 수동으로 장애 조치가 가능하다.
- 기존 MySQL 5.0 이후부터 사용이 가능하며,
DBMS의 성능에 전혀 영향을 주지 않는다.
MHA 주요 기능
- MySQL 5.6 GTID 지원
- MySQL 5.6 Multi-Thread Slave 지원
- MySQL 5.6 Binlog checksum 지원
- mysqlbinlog streaming host 지원
- mysqlbinlog 위치 지원
- ping_type= Select / Connect 외에 Insert 추가
- master_ip_online_change_script에 --orig_master_is_new_slave, --orig_master_ssh_user and --new_master_ssh_user option 추가
MHA Pre-Requirement
- Master 서버와 Slave 서버는 사전에 Replication 구성(연결)이 되어있어야 한다.
- 모든 서버는 공개 SSH 키를 사용하여 서로 접속할 수 있어야한다.
- 모든 서버는 서로의 MySQL 서버에 접속할 수 있어야한다.
- 모든 서버는 동일한 복제용 유저 및 비밀번호를 가져야한다.
- MySQL 버전은 5.0 이상이어야한다.
-
마스터 후보(candidate_master=1)
는 바이너리 로그를 사용하도록 설정해야한다. - 바이너리로그 변수는 모든 서버에서 동일하게 설정해야한다(replicate-wild, binlog-do-db…).
MHA 시스템 환경구성
NO | 구분 | IP주소 | 마스터 후보 | 비고 |
---|---|---|---|---|
1 | MASTER | 180.180.180.226, Active | O |
|
2 | MASTER | 180.180.180.231, Active Backup | O |
|
3 | SLAVE | 180.180.180.242, Slave01 | X |
|
4 | SLAVE | 180.180.180.243, Slave02 | X |
|
5 | SLAVE | 180.180.180.237, Slave03 | X |
|
6 | MHA | 180.180.180.238, Slave01 | X |
Manager |
7 | VIP | 180.180.180.239, Slave01 | X |
서비스 VIP |
MHA 시스템 구성
- [ ALL ] mha 사용자 생성 및 sudo 설정
- [ MASTER ] MHA DB 생성
- [ ALL ] SSH RSA-Key 설정
- [ ALL ] MHA Manager / Node 패키지 설치
- [ MANAGER ] MHA Manager 설정 및 스크립트 작성
- [ MANAGER ] SSH / Replication 검증
- [ ALL ] mha 사용자 생성 및 sudo 설정
# useradd -g mysql mhauser -d /MHA
# echo ghdwngkstjqj |passwd --stdin mhauser
* Debian/Ubuntu의 경우 --stdin 옵션이 지원되지않으므로, 다음과 같이 수행해야 한다.
# adduser --home /MHA mhauser
...패스워드를 설정할 것!
# cat /etc/sudoers
...중략
# Defaults specification
Cmnd_Alias MHA = /usr/sbin/ip, /usr/sbin/ifconfig, /usr/sbin/arping <= 추가
...중략
root ALL=(ALL) ALL
mhauser ALL=(ALL) NOPASSWD:MHA <= 추가
* Debian/Ubuntu의 경우 ip,ifconfig 명령의 경로가 아래와 같다.
Cmnd_Alias MHA = /sbin/ip, /sbin/ifconfig, /usr/sbin/arping <= 추가
!중요
: mha 사용자 계정을 root 유저로 하는 것은 보안적으로 권장되지 않는다.
mhauser의 $HOMEDIR 및 기본 디렉토리를 생성한다.
# mkdir -p /MHA/conf
# mkdir -p /MHA/scripts
# mkdir -p /MHA/logs
# chown -R mhauser.mysql /MHA
- [ MASTER ] MHA DB 생성
Replication에서 사용되어야하는 유저의 DB를 생성한다.
MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.226' identified by '패스워드';
Query OK, 0 rows affected (0.003 sec)
MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.231' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)
MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.242' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)
MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.243' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)
MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.237' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)
MariaDB [mysql]> grant all privileges on *.* to 'mhauser'@'180.180.180.238' identified by '패스워드';
Query OK, 0 rows affected (0.001 sec)
- [ ALL ] SSH RSA-Key 설정
!중요
: 이 설정은 특정 서버를 기준으로 rsa-key를 생성하는것이 아닌, 전체 서버가 양방향으로 패스워드없이 접속이 가능하도록 수행하여야 한다.
$ id
mhauser
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
...중략
엔터 3번 입력
$ ssh-copy-id -p 22222 mhauser@현재서버를 제외한 모든 IP
- [ ALL ] MHA Manager / Node 패키지 설치
!중요
: MHA 0.57 이하 버전의 경우 Replication 체크 및 Auto Failover 기능이 정상적으로 동작하지 않는다. 따라서 최소 0.57 이상의 버전을 사용해야 한다.
- MHA Manager 패키지 설처
!참고 : Manager 패키지를 설치하기 전에 사전에 Node 패키지 설치해야 한다.
패키지를 다음과 같이 다운로드 한다.
# wget http://pds.hongsnet.net:8888/database/mysql/mha4mysql-manager-0.57.tar.gz
# wget http://pds.hongsnet.net:8888/database/mysql/mha4mysql-node-0.57.tar.gz
다음과 같이 Linux 배포판에 따라 패키지를 사전에 먼저 설치해야 한다.
* Debian 계열
# apt-get install libmodule-install-perl libconfig-tiny-perl # apt install liblog-dispatch-perl libparallel-forkmanager-perl
-------------------------------------------------------------------------------
# redhat 계열
# yum install epel-release
# yum install perl-Compress-Raw-Bzip2 perl-Compress-Raw-Zlib perl-DBD-MySQL perl-DBI perl-Data-Dumper perl-IO-Compress perl-Net-Daemon perl-PlRPC perl-CPAN perl-core perl-ExtUtils-Manifest
먼저 Node 패키지를 설치
한다.
# perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI ...loaded. (1.636)
- DBD::mysql ...loaded. (4.041)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for mha4mysql::node
Writing MYMETA.yml and MYMETA.json
# make
cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
cp bin/purge_relay_logs blib/script/purge_relay_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
cp bin/save_binary_logs blib/script/save_binary_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
Manifying 4 pod documents
# make install
Manifying 4 pod documents
Installing /usr/local/share/perl/5.24.1/MHA/BinlogManager.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogHeaderParser.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderXid.pm
Installing /usr/local/share/perl/5.24.1/MHA/SlaveUtil.pm
Installing /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinder.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFindManager.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderElp.pm
Installing /usr/local/share/perl/5.24.1/MHA/NodeConst.pm
Installing /usr/local/man/man1/purge_relay_logs.1p
Installing /usr/local/man/man1/filter_mysqlbinlog.1p
Installing /usr/local/man/man1/save_binary_logs.1p
Installing /usr/local/man/man1/apply_diff_relay_logs.1p
Installing /usr/local/bin/purge_relay_logs
Installing /usr/local/bin/apply_diff_relay_logs
Installing /usr/local/bin/save_binary_logs
Installing /usr/local/bin/filter_mysqlbinlog
Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/perllocal.pod
위와 같이 Node 패키지의 설치가 완료되었으면, 아래와 같이 Manager 패키지를 설치한다.
# perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI ...loaded. (1.636)
- DBD::mysql ...loaded. (4.041)
- Time::HiRes ...loaded. (1.9733)
- Config::Tiny ...loaded. (2.23)
- Log::Dispatch ...loaded. (2.58)
- Parallel::ForkManager ...loaded. (1.19)
- MHA::NodeConst ...loaded. (0.57)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for mha4mysql::manager
Writing MYMETA.yml and MYMETA.json
다음과 같이 make; make install을 수행하면 설치가 완료된다.
# make
cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
cp bin/purge_relay_logs blib/script/purge_relay_logs
/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
cp bin/save_binary_logs blib/script/save_binary_logs
/usr/bin/perl "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
Manifying blib/man1/filter_mysqlbinlog.1
Manifying blib/man1/apply_diff_relay_logs.1
Manifying blib/man1/purge_relay_logs.1
Manifying blib/man1/save_binary_logs.1
# make install
Installing /usr/local/share/perl5/MHA/BinlogManager.pm
Installing /usr/local/share/perl5/MHA/BinlogPosFindManager.pm
Installing /usr/local/share/perl5/MHA/BinlogPosFinderXid.pm
Installing /usr/local/share/perl5/MHA/BinlogHeaderParser.pm
Installing /usr/local/share/perl5/MHA/BinlogPosFinder.pm
Installing /usr/local/share/perl5/MHA/NodeUtil.pm
Installing /usr/local/share/perl5/MHA/BinlogPosFinderElp.pm
Installing /usr/local/share/perl5/MHA/SlaveUtil.pm
Installing /usr/local/share/perl5/MHA/NodeConst.pm
Installing /usr/local/share/man/man1/filter_mysqlbinlog.1
Installing /usr/local/share/man/man1/apply_diff_relay_logs.1
Installing /usr/local/share/man/man1/purge_relay_logs.1
Installing /usr/local/share/man/man1/save_binary_logs.1
Installing /usr/local/bin/filter_mysqlbinlog
Installing /usr/local/bin/apply_diff_relay_logs
Installing /usr/local/bin/purge_relay_logs
Installing /usr/local/bin/save_binary_logs
Appending installation info to /usr/lib64/perl5/perllocal.pod
- MHA Node 패키지 설치
# perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI ...loaded. (1.636)
- DBD::mysql ...loaded. (4.041)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for mha4mysql::node
Writing MYMETA.yml and MYMETA.json
# make
cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm
cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm
cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm
cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm
cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm
cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm
cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm
cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm
cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm
cp bin/purge_relay_logs blib/script/purge_relay_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs
cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog
cp bin/save_binary_logs blib/script/save_binary_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs
cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs
"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs
Manifying 4 pod documents
# make install
Manifying 4 pod documents
Installing /usr/local/share/perl/5.24.1/MHA/BinlogManager.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogHeaderParser.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderXid.pm
Installing /usr/local/share/perl/5.24.1/MHA/SlaveUtil.pm
Installing /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinder.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFindManager.pm
Installing /usr/local/share/perl/5.24.1/MHA/BinlogPosFinderElp.pm
Installing /usr/local/share/perl/5.24.1/MHA/NodeConst.pm
Installing /usr/local/man/man1/purge_relay_logs.1p
Installing /usr/local/man/man1/filter_mysqlbinlog.1p
Installing /usr/local/man/man1/save_binary_logs.1p
Installing /usr/local/man/man1/apply_diff_relay_logs.1p
Installing /usr/local/bin/purge_relay_logs
Installing /usr/local/bin/apply_diff_relay_logs
Installing /usr/local/bin/save_binary_logs
Installing /usr/local/bin/filter_mysqlbinlog
Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/perllocal.pod
- [ MANAGER ] MHA Manager 설정 및 스크립트 작성
# cat /MHA/conf/mha.cnf
[server default]
user=mhauser
password=패스워드
ssh_user=mhauser
ssh_port=ssh_포트번호
repl_user=mhauser
repl_password=패스워드
# working directory on the manager
manager_workdir=/MHA/log
# manager log file
manager_log=/MHA/log/manager.log
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
ping_interval=3
[server1]
hostname=180.180.180.226
master_binlog_dir=/var/lib/mysql
candidate_master=1
port=3306
[server2]
hostname=180.180.180.231
master_binlog_dir=/var/lib/mysql
candidate_master=1
port=3306
[server3]
hostname=180.180.180.242
master_binlog_dir=/var/lib/mysql
no_master=1
port=3306
[server4]
hostname=180.180.180.243
master_binlog_dir=/var/lib/mysql
no_master=1
port=3306
[server5]
hostname=180.180.180.237
master_binlog_dir=/var/lib/mysql
no_master=1
port=3306
** 설정파라미터 설명
- user, password : DB계정 및 암호
- ssh_user : ssh용 접속 계정
- repl_user,repl_password : replication 계정 및 암호
- manager_workdir : 매니저서버의 작업 공간
- manager_log : 매니저서버의 로그파일 위치
- master_ip_failover_script : 자동으로 페일오버를 수행하는 스크립트
- master_ip_online_change_script : 수동으로 페일오버를 수행하는 스크립트로 변수값을 명시해야 한다.
- candidate_master = 1 : 장애 발생시 master가 될 지를 결정하는 설정 (후보 설정)
- no_master = 1 : master의 후보가 아님을 설정
3. master_ip_failover 스크립트 작성
!참고 : 이 스크립트는 Master 시스템의 이슈발생 시 Master 후보군 서버로의 서비스 이관(Fail-Over)을 담당하는 스크립트 이다.
# chmod 755 /usr/local/bin/master_ip_failover
첨부된 master_ip_failover 스크립트 참조
4. master_ip_online_change
!참고 : 이 스크립트는 Master 시스템이 유지보수 등에 대한 이슈로 인해 수동 Fail-Over를 담당하는 스크립트 이다.
# chmod 755 /usr/local/bin/master_ip_online_change
첨부된 master_ip_online_chage 스크립트 참조
- [ MANAGER ] SSH / Replication 검증
1. 전체 구성 시스템의 SSH 접속 테스트 수행
# su - mhauser
$ masterha_check_ssh --conf=/MHA/conf/mha.cnf
Tue Dec 1 00:26:34 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Dec 1 00:26:34 2020 - [info] Reading application default configuration from /MHA/conf/mha.cnf..
Tue Dec 1 00:26:34 2020 - [info] Reading server configuration from /MHA/conf/mha.cnf..
Tue Dec 1 00:26:34 2020 - [info] Starting SSH connection tests..
...중략
Tue Dec 1 00:26:39 2020 - [info] All SSH connection tests passed successfully.
2. Replication 구성테스트 수행
# su - mhauser
$ masterha_check_repl --conf=/MHA/conf/mha.cnf
Tue Dec 1 00:31:38 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Dec 1 00:31:38 2020 - [info] Reading application default configuration from /MHA/conf/mha.cnf..
Tue Dec 1 00:31:38 2020 - [info] Reading server configuration from /MHA/conf/mha.cnf..
Tue Dec 1 00:31:38 2020 - [info] MHA::MasterMonitor version 0.57.
...중략
IN SCRIPT TEST====sudo /usr/sbin/ifconfig eth1:0 down==sudo /usr/sbin/ifconfig eth1:0 180.180.180.239 netmask 255.255.255.0 broadcast 180.180.180.255 up===
Checking the Status of the script.. OK
Tue Dec 1 01:17:41 2020 - [info] OK.
Tue Dec 1 01:17:41 2020 - [warning] shutdown_script is not defined.
Tue Dec 1 01:17:41 2020 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
ISSUED
- [ Issue 1 ] : [error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. Redundant argument in sprintf at /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm line 184.
MariaDB Version 차이에 따른 이슈로써 다음과 같이 Perl Module의 내용을 수정한다.
# cat /usr/local/share/perl/5.24.1/MHA/NodeUtil.pm
...중략
sub parse_mysql_version($) {
my $str = shift;
#2020-12-29
($str) = $str =~ m/^[^-]*/g; <= 추가
my $result = sprintf( '%03d%03d%03d', $str =~ m/(\d+)/g );
return $result;
}
sub parse_mysql_major_version($) {
my $str = shift;
#2020-12-29
($str) = $str =~ m/^[^-]*/g; <= 추가
#my $result = sprintf( '%03d%03d', $str =~ m/(\d+)/g ); <= 주석
my $result = sprintf( '%03d%03d%03d', $str =~ m/(\d+)/g ); <= 변경
return $result;
}
- [ Issue 2 ] : mysqlbinlog: unknown variable 'default-character-set=utf8mb4' mysqlbinlog version command failed with rc 7:0, please verify PATH, LD_LIBRARY_PATH, and client options at /usr/local/bin/apply_diff_relay_logs line 493. [error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln208] Slaves settings check failed! [error][/usr/local/share/perl/5.24.1/MHA/MasterMonitor.pm, ln416] Slave configuration failed.
my.cnf 파일에 잘못된 캐릭터 셋 파라미터가 설정되어서 발생된 에러로써 다음과 같이 주석처리하거나 제거해야 한다.
# cat my.cnf
...중략
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
#default-character-set=utf8mb4 <= 이 파라미터가 원인이다. 주석처리를 해야한다.