Skip to content

J
joohanhong
Commit d0ce9280 authored by nuxer's avatar nuxer
Browse files
Options

web-proxy 클러스터구성

parent 0efc9567
Hide whitespace changes
Inline Side-by-side
Showing with 608 additions and 0 deletions
INFRA/PROXY/README.md 0 → 100644
View file @ d0ce9280
[![logo](https://www.hongsnet.net/images/logo.gif)](https://www.hongsnet.net)
# Pacemaker를 이용한 WEB Proxy 이중화 구성
> KVM 가상화를 이용한 MASTER / SLAVE 구성을 다룬다.
# 구성 환경
- KVM Virtual Machine
- CentOS 7
- Pacemaker / Corosync
- Heartbeat Network
## Pacemaker 설치 및 구성
Pacemaker는 Heartbeat Network으로 구성되기 때문에 다음과 같이 `/etc/hosts` 파일에 IP를 설정한다.
```bash
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.242 adminproxy-master
192.168.200.243 adminproxy-slave
```
아래와 같이 모든 Node에 Pacemaker 패키지를 설치한다.
```bash
[ALL]# yum install pcs fence-agents-all
```
> 위의 패키지는 CentOS 기본(Base) 패키지에서 제공한다.
설치를 마무리 한다.
```bash
[ALL]# systemctl start pcsd; systemctl enable pcsd
[ALL]# echo 패스워드 | passwd --stdin hacluster
[MASTER]# pcs cluster adminproxy-master adminproxy-slave
[MASTER]# pcs cluster setup --start --name ADMIN-PROXY adminproxy-master adminproxy-slave
[MASTER]# pcs cluster enable --all
[MASTER]# pcs property set stonith-enabled=false
[MASTER]# pcs property set no-quorum-policy=ignore
```
`!중요` : 클러스터 Quorum 및 Fencing 설정
> 다음의 클러스터 구성은 2 Node로 구성되기 때문에 정족수(Quorum) 설정을 비활성화하여야 한다. 또한 Fence Device(IPMI 등)가 구성되지 않은 경우에도 설정(stonith)을 비활성화 해야 한다. 만약 `이 설정이 무시될 경우 클러스터가 구성되지 않는다.`
## 클러스터 리소스 구성 및 용도
* [ ] 서비스 / Private VIP 리소스 구성 => **서비스 VIP 사용**
* [ ] NGINX / PHP-FPM 설치 및 리소스 구성 => **WEB Proxy 서비스**
* [ ] 클러스터 제약조건 설정
## 서비스 / Private VIP 구성
```bash
[MASTER]# pcs resource create PU-VIP IPaddr2 ip=61.100.0.XX cidr_netmask=32 nic=eth0 op monitor interval=10s timeout=30s on-fail=standby
[MASTER]# pcs resource create PR-VIP IPaddr2 ip=172.24.0.XX cidr_netmask=32 nic=eth1 op monitor interval=10s timeout=30s on-fail=standby
```
> `서비스 VIP(61.100.0.XX)를 설정`하고, 시스템의 nic는 eth0, 모니터링 주기는 10초, 최종 timeout은 30초로 설정한다. 만약 현재 Owner의 장애가 발생하면, Standby로 Role을 이관한다.
## NGINX / PHP-FPM 설치 및 리소스 구성
**Nginx**의 경우 `CentOS Base Repository를 통해 설치가 불가능`하므로, 다음과 같이 내부 저장소의 Repository를 설정한다.
```bash
[ALL]# cat /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://yum.hongsnet.net:8889/nginx/centos7/$basearch
gpgcheck=0
enabled=1
```
```bash
[ALL]# yum install nginx
```
**PHP-FPM**의 경우는 `최신버전의 PHP-FPM`을 CentOS Base Repository를 통한 설치가 불가능하므로, 다음과 같이 내부 저장소의 Repository를 설정한다.
```bash
[ALL]# cat /etc/yum.repos.d/hongs.repo
[remi]
name = Remi Safe
baseurl = http://yum.hongsnet.net:8889/centos/7/remi/$basearch
gpgcheck = 0
enabled = 1
[centos7-extra]
name = CentOS7 Extras(EPEL)
baseurl = http://yum.hongsnet.net:8889/centos/7/extra/$basearch
gpgcheck = 0
enabled = 1
```
PHP-FPM 설치는 일반적인 WEB 환경이 모두 지원되도록, 다음과 같이 각종 라이브러리를 설치한다.
```bash
[ALL]# yum install php74-php-fpm php74 php74-php-pdo php74-php-xml php74-php-cli php74-php-pdo-firebird \
php74-php-mysqlnd php74-php-process php74-php-pear php74-php-pecl-mysql \
php74-php-pecl-interbase php74-php-xmlrpc php74-php-odbc php74-php-pspell php74-php-gmp \
php74-php-ldap php74-php-pecl-zip php74-php-intl php74-php-dbg php74-php-enchant \
php74-php-tidy php74-php-pecl-mcrypt php74-php-opcache php74-php-bcmath \
php74-php-dba php74-php-soap php74-php-imap php74-php-gd php74-php-mbstring \
php74-php-pecl-recode
```
`!중요` : 설치 후 nginx / php-fpm 데몬은 pacemaker가 제어해야하기 때문에 리부팅 후 실행되지않게 비활성처리가 되어야 함
```bash
[ALL]# systemctl disable nginx; systemctl disable php74-php-fpm
```
다음과 같이 클러스터 리소스로 haproxy 데몬을 등록한다.
```bash
[MASTER]# pcs resource create NGINX systemd:nginx
```
## php-fpm 설정
php-fpm을 설치하면 기본적으로 TCP 방식으로 구성되는데, 만약 이를 소켓방식으로 변경하려면 다음과 같이 변경한다.
```bash
[ALL]# cat /etc/opt/remi/php74/php-fpm.d/www.conf
...중략
;listen = 127.0.0.1:9000
listen = /var/opt/remi/php70/run/php-fpm/php-fpm.socket
```
## nginx 설정
Nginx를 php-fpm과 연동하려면, 다음과 같이 설정해야 한다.
```bash
# cat /etc/nginx/conf.d/default.conf
server {
listen 80;
server_name 61.100.0.185;
root /var/www/html;
...중략
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
location ~ \.php$ {
#fastcgi_pass unix:/var/opt/remi/php70/run/php-fpm/php-fpm.socket;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
```
## nginx 및 php-fpm 리소스 구성
다음과 같이 클러스터 리소스로 nginx / php-fpm 데몬을 등록한다.
```bash
[MASTER]# pcs resource create NGINX systemd:nginx
[MASTER]# pcs resource create PHP-FPM systemd:php74-php-fpm
```
## 클러스터 제약조건 설정
> 위와 같이 서비스가 등록되면, MASTER / SLVAE의 구분없이, 서로 리소스를 실행하려고 한다.
- 클러스터 리소스 시작순서 설정
* [ **STEP 1** ] : 서비스 Public IP / Private IP 실행
```bash
[MASTER]# pcs constraint order start PU-VIP then start PR-VIP
```
> PU(Public IP)-VIP 리소스를 실행한 후 PR-VIP 리소스를 실행한다.
* [ **STEP 2** ] : Private IP / NGINX 실행
```bash
[MASTER]# pcs constraint order start PR-VIP then start NGINX
```
> PR(Private IP)-VIP 리소스를 실행한 후 NGINX 리소스를 실행한다.
* [ **STEP 3** ] : php-fpm를 실행
```bash
[MASTER]# pcs constraint order start NGINX then start PHP-FPM
```
* [ **STEP 4** ] : 중요 클러스터 리소스의 우선순위 설정
```bash
[MASTER]# pcs constraint colocation add PU-VIP with PR-VIP
[MASTER]# pcs constraint colocation add PR-VIP with NGINX
[MASTER]# pcs constraint colocation add NGINX with PHP-FPM
```
> 이 설정은 클러스터의 모든 리소스가 Owner Node에서 실행됨을 의미한다.
## 클러스터 구성내역
```bash
# pcs status
Cluster name: ADMIN-PROXY
Stack: corosync
Current DC: adminproxy-master (version 1.1.23-1.el7_9.1-9acf116022) - partition with quorum
Last updated: Tue Mar 2 11:31:35 2021
Last change: Mon Mar 1 01:45:26 2021 by root via crm_resource on adminproxy-master
2 nodes configured
4 resource instances configured
Online: [ adminproxy-master adminproxy-slave ]
Full list of resources:
PU-VIP (ocf::heartbeat:IPaddr2): Started adminproxy-master
PR-VIP (ocf::heartbeat:IPaddr2): Started adminproxy-master
NGINX (systemd:nginx): Started adminproxy-master
PHP-FPM (systemd:php74-php-fpm): Started adminproxy-master
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
```
다음은 리소스의 제약조건에 대한 결과이다.
```bash
# pcs resource --full
Resource: PU-VIP (class=ocf provider=heartbeat type=IPaddr2)
Attributes: cidr_netmask=32 ip=61.100.0.231 nic=eth0
Operations: monitor interval=10s on-fail=standby timeout=30s (PU-VIP-monitor-interval-10s)
start interval=0s timeout=20s (PU-VIP-start-interval-0s)
stop interval=0s timeout=20s (PU-VIP-stop-interval-0s)
Resource: PR-VIP (class=ocf provider=heartbeat type=IPaddr2)
Attributes: cidr_netmask=32 ip=192.168.200.231 nic=eth1
Operations: monitor interval=10s on-fail=standby timeout=30s (PR-VIP-monitor-interval-10s)
start interval=0s timeout=20s (PR-VIP-start-interval-0s)
stop interval=0s timeout=20s (PR-VIP-stop-interval-0s)
Resource: NGINX (class=systemd type=nginx)
Operations: monitor interval=60 timeout=100 (NGINX-monitor-interval-60)
start interval=0s timeout=100 (NGINX-start-interval-0s)
stop interval=0s timeout=100 (NGINX-stop-interval-0s)
Resource: PHP-FPM (class=systemd type=php74-php-fpm)
Operations: monitor interval=60 timeout=100 (PHP-FPM-monitor-interval-60)
start interval=0s timeout=100 (PHP-FPM-start-interval-0s)
stop interval=0s timeout=100 (PHP-FPM-stop-interval-0s)
```
# 클러스터 관리명령
* [ ] 클러스터 리소스 CleanUP 수행
클러스터에 에러가 발생할 경우 다음과 같이 `CleanUP`을 수행해야 처리할수 있다.
```bash
[OWNER]# pcs resource cleanup 리소스명
```
단, 시스템 구성이나 데몬에 이상이 발생되면, 에러는 Fix되지 않고 계속 발생될 것이다. 이는 기 발생된 에러에 대한 CleanUP 처리이다.
* [ ] 클러스터 리소스 Relocate 수행
현재 운용되는 클러스터의 유지보수로 인해 Slave Node로의 `On-demand 절체(relocate)`를 수행할 수 있다.
```bash
[OWNER]# pcs resource relocate 리소스명
```
# Extras
다음의 설정은 WEB Proxy로 사용되고 있는 nginx의 설정이다. **아래의 설정들은 Back-End IP주소와 포트를 변경한 후 서비스에 맞게 즉시 적용하여 사용할 수 있다.**
## **elk**
```bash
# cat /etc/nginx/conf.d/elk-demo.conf
upstream elk_demo_backend {
server 172.16.0.228:5601;
}
server {
listen 443 ssl http2;
server_name elk-demo.hongsnet.net;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/conf.d/.htpasswd_demo;
ssl_certificate /etc/letsencrypt/live/hongsnet.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/hongsnet.net/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/hongsnet.net/chain.pem;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;
ssl_stapling on;
ssl_stapling_verify on;
resolver 61.100.0.136 8.8.4.4 valid=300s;
resolver_timeout 30s;
add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
access_log /var/log/nginx/elk-demo.hongsnet.net-access.log;
error_log /var/log/nginx/elk-demo.hongsnet.net-error.log;
location ~ /api/v[0-9]+/(users/)?websocket$ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 50M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_pass http://elk_demo_backend;
}
location / {
proxy_http_version 1.1;
client_max_body_size 50M;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_pass http://elk_demo_backend;
}
}
```
## **jenkins**
```bash
# cat /etc/nginx/conf.d/jenkins.conf
server {
listen 80;
server_name jenkins.freehongs.net;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location / {
return 301 https://$server_name$request_uri;
}
}
server {
listen 443 ssl http2;
server_name jenkins.freehongs.net;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
ssl_certificate /etc/letsencrypt_freehongs/live/freehongs.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt_freehongs/live/freehongs.net/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt_freehongs/live/freehongs.net/chain.pem;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Required for new HTTP-based CLI
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_buffering off; # Required for HTTP-based CLI to work over SSL
proxy_pass http://172.16.0.158:8888;
proxy_read_timeout 90;
proxy_redirect http://172.16.0.158:8888 jenkins.freehongs.net;
# workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
add_header 'X-SSH-Endpoint' 'jenkins.freehongs.tld:50022' always;
}
location /asynchPeople {
return 401;
}
}
```
## **mattermost**
```bash
# cat /etc/nginx/conf.d/mattermost.conf
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;
upstream mattermost_backend {
server 172.24.0.151:8888;
keepalive 32;
}
server {
listen 443 ssl http2;
server_name chat.hongsnet.net;
ssl_certificate /etc/letsencrypt/live/hongsnet.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/hongsnet.net/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/hongsnet.net/chain.pem;
http2_push_preload on; # Enable HTTP/2 Server Push
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;
ssl_stapling on;
ssl_stapling_verify on;
resolver 61.100.0.136 8.8.4.4 valid=300s;
resolver_timeout 30s;
add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
access_log /var/log/nginx/chat.hongsnet.net-access.log;
error_log /var/log/nginx/chat.hongsnet.net-error.log;
location ~ /api/v[0-9]+/(users/)?websocket$ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 50M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_pass http://mattermost_backend;
}
location / {
proxy_http_version 1.1;
client_max_body_size 50M;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_pass http://mattermost_backend;
}
}
```
## **portainer**
```bash
# cat /etc/nginx/conf.d/port135.conf
server {
listen 80;
server_name port135.hongsnet.net;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location / {
return 301 https://$server_name$request_uri;
}
}
server {
listen 443 ssl http2;
server_name port135.hongsnet.net;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
ssl_certificate /etc/letsencrypt/live/hongsnet.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/hongsnet.net/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/hongsnet.net/chain.pem;
#include ssl-params.conf;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Required for new HTTP-based CLI
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_buffering off; # Required for HTTP-based CLI to work over SSL
proxy_pass http://192.168.200.62:9000;
proxy_read_timeout 90;
proxy_redirect http://192.168.200.62:9000 port135.hongsnet.net;
}
location /api/websocket/ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_pass http://192.168.200.62:9000/api/websocket/;
}
}
```
## **grafana**
```bash
# cat /etc/nginx/conf.d/grafana.conf
server {
listen 80;
server_name grafana.hongsnet.net;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location / {
return 301 https://$server_name$request_uri;
}
}
server {
listen 443 ssl http2;
server_name grafana.hongsnet.net;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
ssl_certificate /etc/letsencrypt/live/hongsnet.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/hongsnet.net/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/hongsnet.net/chain.pem;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Required for new HTTP-based CLI
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_buffering off; # Required for HTTP-based CLI to work over SSL
proxy_pass http://172.24.0.245:3000;
proxy_read_timeout 90;
proxy_redirect http://172.24.0.245:3000 grafana.hongsnet.net;
}
}
```
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment