Skip to content

J
joohanhong
Switch branch/tag
History Find file
Name
Last commit
 Last update
..
images ci/cd, k8s init
Dockerfile swarm directory move
README.md k8s ci_cd init
README.md

logo

www.hongsnet.net Container Deploy (Deprecated)

source_overview

구성 요소

  • Git and GitLab (gitlab-ci.yml)
  • gitlab-runner
  • Docker Swarm Cluster

GitLab Runner Overview

Runner는 yaml을 파일을 수행한다. 특정 프로젝트에 국한되거나, 여러 프로젝트에서 사용할 수 있도록 제공된다. 여기서 모든 여러 프로젝트에서 사용할 수 있는 Runner를 Shared Runner이라고 하며, 특정 프로젝트에 국한되서 사용되는 Runner를 Specific Runner라고 한다.

홍쓰넷 컨테이너 배포에는 Specific Runner를 사용한다.

Container Deploy Overview (Deprecated)

  • STEP 1 : STAG 시스템에 먼저 Docker Image를 배포한 후 검증한다.

사전에 devel 브랜치로 checkout을 수행한다.

  • STEP 2 : STAG 시스템에서 검증이 완료되면, 다음과 같이 두 개의 Stage가 수행된다.
    • push : GitLAB의 Container Registry에 Container Image를 PUSH 한다.
    • deploy : Docker Swarm Manager 서버에 Image 배포를 명령한다. 단, 모든 WORKER Node가 한 번에 업데이트되는 것이 아니라 한 대씩(--update-parallelism 1) 업데이트를 수행하게 된다.

www.hongsnet.net Dockerfile 

# cat Dockerfile
FROM registry.hongsnet.net/joohan.hong/docker/centos:7.6.1810
MAINTAINER Hongs <master@hongsnet.net>

ENV TZ=Asia/Seoul
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\
    yum -y install epel-release

ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8

RUN yum -y install glibc glibc-common freetype-devel libjpeg-devel openssl-devel net-tools vi vim iproute vsftpd wget postfix cronie crontabs supervisor

COPY config/libmcrypt-config /usr/local/bin/libmcrypt-config
COPY config/mm-config /usr/local/bin/mm-config
COPY config/mcrypt.h /usr/local/include/mcrypt.h
COPY config/mm.h /usr/local/include/mm.h
COPY config/libmm.so.14.0.22 /usr/local/lib/libmm.so.14.0.22
COPY config/libmm.so.14 /usr/local/lib/libmm.so.14
COPY config/libmm.so /usr/local/lib/libmm.so
COPY config/libmhash.a /usr/local/lib/libmhash.a
COPY config/libmhash.la /usr/local/lib/libmhash.la
COPY config/libmhash.so /usr/local/lib/libmhash.so
COPY config/libmhash.so.2 /usr/local/lib/libmhash.so.2
COPY config/libmhash.so.2.0.1 /usr/local/lib/libmhash.so.2.0.1
COPY config/libmcrypt.la /usr/local/lib/libmcrypt.la
COPY config/libmcrypt.so /usr/local/lib/libmcrypt.so
COPY config/libmcrypt.so.4 /usr/local/lib/libmcrypt.so.4
COPY config/libmcrypt.so.4.4.7 /usr/local/lib/libmcrypt.so.4.4.7

RUN ln -s /usr/lib64/libpng.so  /usr/lib/libpng.so &&\
    ln -s /usr/lib64/libjpeg.so /usr/lib/libjpeg.so

COPY config/bashrc /root/.bashrc

COPY config/main.cf /etc/postfix/main.cf
COPY config/master.cf /etc/postfix/master.cf

COPY binary/mysql /usr/local/mysql
COPY binary/apache /usr/local/apache
RUN chmod 701 /usr/local/apache/htdocs
COPY binary/php /usr/local/php
COPY binary/curl /usr/local/curl
RUN mkdir /usr/lib64/curl_backup
RUN mv /usr/lib64/libcurl.so* /usr/lib64/curl_backup/
RUN cp -rf /usr/local/curl/lib/libcurl.so* /usr/lib64/

RUN rpm -ivh http://pds.hongsnet.net:8888/packages/filebeat-7.9.3-x86_64.rpm
COPY config/filebeat.yml /etc/filebeat/filebeat.yml

COPY config/supervisord.conf /etc/supervisor/supervisord.conf
COPY config/hongsnet.conf /etc/supervisor/conf.d/hongsnet.conf

EXPOSE 80

RUN echo "hongsnet, 2020-01-20(5b23e123)" > /root/release_image

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/supervisord.conf"]

다음은 배포에 대한 정책을 수행하는 .gitlab-ci.yml 파일에 대한 내역이다.

GitLAB .gitlab-ci.yml 파일의 작성

# cat .gitlab-ci.yml
stages:
  - devel
  - push
  - deploy

variables:
  GIT_STRATEGY: clone
  GIT_SSL_NO_VERIFY: "1"
  GITLAB_TOKEN: "XXXXXX"
  BUILD_IMAGE: ${CI_PROJECT_DIR}/hongsnet
  BUILD_SHA: ${CI_COMMIT_SHA}

before_script:
  #- export RELEASE_DATE=`date +"%Y%m%d"`

push:
  stage: push
  only:
    - deploy
  script:
   - docker login -u juhanida21@nate.com -p $GITLAB_TOKEN registry.hongsnet.net
   - /usr/bin/docker tag hongsnet-dev:$BUILD_SHA registry.hongsnet.net/joohan.hong/docker/hongsnet:latest
   - /usr/bin/docker push registry.hongsnet.net/joohan.hong/docker/hongsnet:latest
  tags:
   - deploy

deploy:
  stage: deploy
  only:
    - deploy
  script:
   - docker login -u juhanida21@nate.com -p $GITLAB_TOKEN registry.hongsnet.net
   - docker service update --image registry.hongsnet.net/joohan.hong/docker/hongsnet:latest --update-parallelism 1 web_hongsnet
  tags:
   - backup

build:
  stage: devel
  only:
    - devel
  script:
   - cd /DATA/DOCKERS/hongsnet-dev; docker-compose down
   - cd ${BUILD_IMAGE}
   - docker build --rm -t hongsnet-dev:$BUILD_SHA .
   - (if [[ `docker images -f "dangling=true" -q` != "" ]]; then docker rmi $(docker images -f "dangling=true" -q); fi);
   - (rm -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml; cp -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml_init /DATA/DOCKERS/hongsnet-dev/docker-compose.yml);
   - (sed 's/init/$BUILD_SHA/g' -i /DATA/DOCKERS/hongsnet-dev/docker-compose.yml);
   - cd /DATA/DOCKERS/hongsnet-dev; docker-compose up -d
  tags:
   - devel

컨테이너 배포의 핵심은 gitlab-runner가 Manager Node에서 다음과 같이 명령하면서 업데이트가 이루어진다.

docker service update --image registry.hongsnet.net/joohan.hong/docker/hongsnet:latest --update-parallelism 1 web_hongsnet

컨테이너 배포 절차

  • STEP 1 : devel 브랜치를 Checkout 한다.
# git checkout devel

# git branch
* devel
  deploy
  master
  • STEP 2 : 컨테이너 이미지에 대한 작업을 수행한다.

Working...

컨테이너 이미지 생성 시 Git의 Commit SHA(BUILD_SHA 변수) 값으로 컨테이너의 Tagging을 지정한다.

(rm -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml; cp -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml_init /DATA/DOCKERS/hongsnet-dev/docker-compose.yml);

수행된 docker-compose.yml 파일내역은 다음과 같다.

# cat docker-compose.yml_init
version: '3'
services:
  hongsnet:
       image: hongsnet-dev:init
       hostname: 'dev.hongsnet.net'
       volumes:
        - "/DATA/SERVICE:/home"
        - "/WEB_DATA/EDU/Data:/home/edu/public_html/HongsBoard/Data"
        - "/WEB_DATA/EDU/Web_editor/EDU:/home/edu/public_html/HongsBoard/Web_editor/EDU"
        - "/WEB_DATA/EDU/Web_editor/FILE:/home/edu/public_html/HongsBoard/Web_editor/FILE"
        - "/WEB_DATA/HONGSNET/Data:/home/hongsnet/public_html/Data"
        - "/WEB_DATA/HONGSNET/Web_editor/FILE:/home/hongsnet/public_html/Web_editor/FILE"
        - "/WEB_DATA/NEWHONGSYSTEM/Data:/home/newhongsystem/public_html/Data"
        - "/WEB_DATA/NEWHONGSYSTEM/Web_editor/FILE:/home/newhongsystem/public_html/Web_editor/FILE"
       ports:
        - "80:80"

!참고 .gitlab-ci.yml 파일에 정의된 내역을 보면 다음과 같이 Git Commit SHA(BUILD_SHA) 값으로 변경하는 명령이 존재한다.

- (sed 's/init/$BUILD_SHA/g' -i /DATA/DOCKERS/hongsnet-dev/docker-compose.yml);

이는 Commit 된 SHA(BUILD_SHA) 값을 향후 참조하기 위한 용도로 사용된다.

container_dev

  • STEP 3 : STAG 시스템에서 Docker Image의 상태를 확인한다.
# docker ps
CONTAINER ID   IMAGE                                                   COMMAND                  CREATED         STATUS         PORTS                              NAMES
29bc33723517   hongsnet-dev:d7f4a64cc6098bd61af337e0c822bc53c5ed35b7   "/usr/bin/supervisor…"   8 minutes ago   Up 8 minutes   0.0.0.0:80->80/tcp                 hongsnet-dev_hongsnet_1
0779415724e2   portainer/portainer-ce:latest                           "/portainer"             4 weeks ago     Up 4 weeks     8000/tcp, 0.0.0.0:9000->9000/tcp   portainer
  • STEP 4 : Stag 단계에서 문제가 없다면, 다음과 같이 배포하고, 실제 서비스에 적용한다.

먼저 deploy 브랜치로 Checkout 한다.

# git checkout deploy
Switched to branch 'deploy'

!중요 Stag 과정 중에 Commit 된 $BUILD_SHA 변수를 /root/release_image 파일에 업데이트함으로써 증적을 기록한다.

# cat Dockerfile
...중략
RUN echo "hongsnet, 2020-01-20(d7f4a64cc6098bd61af337e0c822bc53c5ed35b7)" > /root/release_image

이제 add/commit/push를 진행하면 된다.

# git add hongsnet/Dockerfile
# git commit -m "2020-01-20(d7f4a64cc6098bd61af337e0c822bc53c5ed35b7)"
# git push -u origin deploy

Swarm Manager Review

  • docker-stack.yml
# cat docker-stack.yml
version: '3'
services:
  hongsnet:
       image: registry.hongsnet.net/joohan.hong/docker/hongsnet:latest
       volumes:
        - HOME:/home
        - EDU_DATA:/home/edu/public_html/HongsBoard/Data
        - EDU_EDITOR:/home/edu/public_html/HongsBoard/Web_editor/EDU"
        - EDU_FILE:/home/edu/public_html/HongsBoard/Web_editor/FILE"
        - HONGS_DATA:/home/hongsnet/public_html/Data"
        - HONGS_EDITOR:/home/hongsnet/public_html/Web_editor/FILE"
        - NEWSYSTEM_DATA:/home/newhongsystem/public_html/Data"
        - NEWSYSTEM_EDITOR:/home/newhongsystem/public_html/Web_editor/FILE"
       ports:
        - "80:80"
       deploy:
        mode: global
        placement:
          constraints: [node.hostname != TB2-DOCKER]
        update_config:
          parallelism: 5
          delay: 10s
        restart_policy:
          condition: on-failure
          max_attempts: 3
          window: 120s

volumes:
  HOME:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_SRC/home"
  EDU_DATA:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/EDU/Data"
  EDU_EDITOR:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/EDU/Web_editor/EDU"
  EDU_FILE:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/EDU/Web_editor/FILE"
  HONGS_DATA:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/HONGSNET/Data"
  HONGS_EDITOR:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/HONGSNET/Web_editor/FILE"
  NEWSYSTEM_DATA:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/NEWHONGSYSTEM/Data"
  NEWSYSTEM_EDITOR:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/NEWHONGSYSTEM/Web_editor/FILE"

global mode를 사용한다. 이는 사전에 의도된 구성이며, 내역은 다음과 같다.

deploy:
   #mode: replicated
   #replicas: 3
   mode: global

또한 Manager(Leader) Node는 다음과 같이 제외한다.

placement:
    constraints: [node.hostname != TB2-DOCKER]
  • Docker Swarm Node status
# docker node ls
ID                            HOSTNAME     STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
bb6yuipq9ja8llzi25owyz1a2     TB2          Ready     Active         Reachable        20.10.2
5mtd5wiql2cqnh5088p8036by *   TB2-DOCKER   Ready     Active         Leader           20.10.2
m1rybwgn3facxoklcuj5j20tg     TB3          Ready     Active         Reachable        20.10.2
t3zbiuhkpam480yfqgc78tzgn     TB3-DOCKER   Ready     Active                          20.10.2

Swarm 서비스의 상태는 다음과 같다.

# docker service ls
ID             NAME                   MODE      REPLICAS   IMAGE                                                      PORTS
xodct3yxupq6   monitor_alertmanager   global    1/1        prom/alertmanager:latest                                   *:9093->9093/tcp
zlay4qoq8gg7   monitor_cadvisor       global    4/4        google/cadvisor:latest                                     *:8080->8080/tcp
pfljlqixrepi   monitor_grafana        global    1/1        grafana/grafana:latest                                     *:3000->3000/tcp
1kakkg4asokp   monitor_prometheus     global    1/1        prom/prometheus:latest                                     *:9090->9090/tcp
hjsvav9409zy   web_hongsnet           global    3/3        registry.hongsnet.net/joohan.hong/docker/hongsnet:latest   *:80->80/tcp

web_hongsnet 이 서비스에 대한 컨테이너 이다.