Skip to content

J
joohanhong
Switch branch/tag
History Find file
Name
Last commit
 Last update
..
binary swarm -> k8s, docker binary,config move
config swarm -> k8s, docker binary,config move
images k8s ci_cd update
.gitkeep k8s, ci_cd init
README.md k8s ci_cd update
README.md

logo

www.hongsnet.net Container Deploy (Current)

source_overview

구성 요소

  • Git and GitLab (gitlab-ci.yml)
  • gitlab-runner
  • docker-compose (STAG 시스템에서 사용)
  • K8s

GitLab Runner Overview

Runner는 yaml을 파일을 수행한다. 특정 프로젝트에 국한되거나, 여러 프로젝트에서 사용할 수 있도록 제공된다. 여기서 모든 여러 프로젝트에서 사용할 수 있는 Runner를 Shared Runner이라고 하며, 특정 프로젝트에 국한되서 사용되는 Runner를 Specific Runner라고 한다.

홍쓰넷 컨테이너 배포에는 Specific Runner를 사용한다.

Container Deploy Overview (Current)

  • STEP 1 : STAG 시스템에 먼저 Docker Image를 배포한 후 검증한다.

사전에 devel 브랜치로 checkout을 수행한다.

  • STEP 2 : STAG 시스템에서 검증이 완료되면, 다음과 같이 두 가지 Stage가 수행된다.
    • push : GitLAB의 Container Registry에 Container Image를 PUSH 한다.
    • deploy : K8s의 Master Node에 Image 배포를 명령한다.

www.hongsnet.net Dockerfile 

# cat Dockerfile
FROM registry.hongsnet.net/joohan.hong/docker/centos:7.6.1810
MAINTAINER Hongs <master@hongsnet.net>

ENV TZ=Asia/Seoul
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\
    yum -y install epel-release

ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8

RUN yum -y install glibc glibc-common freetype-devel libjpeg-devel openssl-devel net-tools vi vim iproute vsftpd wget postfix cronie crontabs supervisor

COPY config/libmcrypt-config /usr/local/bin/libmcrypt-config
COPY config/mm-config /usr/local/bin/mm-config
COPY config/mcrypt.h /usr/local/include/mcrypt.h
COPY config/mm.h /usr/local/include/mm.h
COPY config/libmm.so.14.0.22 /usr/local/lib/libmm.so.14.0.22
COPY config/libmm.so.14 /usr/local/lib/libmm.so.14
COPY config/libmm.so /usr/local/lib/libmm.so
COPY config/libmhash.a /usr/local/lib/libmhash.a
COPY config/libmhash.la /usr/local/lib/libmhash.la
COPY config/libmhash.so /usr/local/lib/libmhash.so
COPY config/libmhash.so.2 /usr/local/lib/libmhash.so.2
COPY config/libmhash.so.2.0.1 /usr/local/lib/libmhash.so.2.0.1
COPY config/libmcrypt.la /usr/local/lib/libmcrypt.la
COPY config/libmcrypt.so /usr/local/lib/libmcrypt.so
COPY config/libmcrypt.so.4 /usr/local/lib/libmcrypt.so.4
COPY config/libmcrypt.so.4.4.7 /usr/local/lib/libmcrypt.so.4.4.7

RUN ln -s /usr/lib64/libpng.so  /usr/lib/libpng.so &&\
    ln -s /usr/lib64/libjpeg.so /usr/lib/libjpeg.so

COPY config/bashrc /root/.bashrc

COPY config/main.cf /etc/postfix/main.cf
COPY config/master.cf /etc/postfix/master.cf

COPY binary/mysql /usr/local/mysql
COPY binary/apache /usr/local/apache
RUN chmod 701 /usr/local/apache/htdocs
COPY binary/php /usr/local/php
COPY binary/curl /usr/local/curl
RUN mkdir /usr/lib64/curl_backup
RUN mv /usr/lib64/libcurl.so* /usr/lib64/curl_backup/
RUN cp -rf /usr/local/curl/lib/libcurl.so* /usr/lib64/

RUN rpm -ivh http://pds.hongsnet.net:8888/packages/filebeat-7.9.3-x86_64.rpm
COPY config/filebeat.yml /etc/filebeat/filebeat.yml

COPY config/supervisord.conf /etc/supervisor/supervisord.conf
COPY config/hongsnet.conf /etc/supervisor/conf.d/hongsnet.conf

EXPOSE 80

RUN echo "hongsnet, 2020-12-20(2283829f44f6e5b7b17393ff6861553c13170a9d)" > /root/release_image

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/supervisord.conf"]

다음은 배포에 대한 정책을 수행하는 .gitlab-ci.yml 파일에 대한 내역이다.

GitLAB .gitlab-ci.yml 파일의 작성

# cat .gitlab-ci.yml
stages:
  - devel
  - push
  - deploy

variables:
  GIT_STRATEGY: clone
  GIT_SSL_NO_VERIFY: "1"
  GITLAB_TOKEN: "XXXXXX"
  BUILD_IMAGE: ${CI_PROJECT_DIR}/hongsnet
  BUILD_SHA: ${CI_COMMIT_SHA}
  BUILD_SHA_RESULT: "2283829f44f6e5b7b17393ff6861553c13170a9d" #deploy tag, requirement
  BUILD_VERSION: "20201220_v1"

before_script:
  #- export RELEASE_DATE=`date +"%Y%m%d"`

push:
  stage: push
  only:
    - deploy
  script:
   - /usr/bin/docker login -u juhanida21@nate.com -p $GITLAB_TOKEN registry.hongsnet.net
   - /usr/bin/docker tag hongsnet-dev:$BUILD_SHA_RESULT registry.hongsnet.net/joohan.hong/docker/hongsnet:$BUILD_VERSION
   - /usr/bin/docker push registry.hongsnet.net/joohan.hong/docker/hongsnet:$BUILD_VERSION
  tags:
   - deploy

deploy:
  stage: deploy
  only:
    - deploy
  script:
   - /usr/bin/docker login -u juhanida21@nate.com -p $GITLAB_TOKEN registry.hongsnet.net
   - /usr/bin/kubectl set image deployment/hongsnet-web hongsnet-web=registry.hongsnet.net/joohan.hong/docker/hongsnet:$BUILD_VERSION
   - /usr/bin/kubectl rollout history deployment/hongsnet-web
   - /usr/bin/kubectl rollout status deployment/hongsnet-web
  tags:
   - backup

build:
  stage: devel
  only:
    - devel
  script:
   - cd /DATA/DOCKERS/hongsnet-dev; docker-compose down
   - cd ${BUILD_IMAGE}
   - /usr/bin/docker build --rm -t hongsnet-dev:$BUILD_SHA .
   - (if [[ `docker images -f "dangling=true" -q` != "" ]]; then docker rmi $(docker images -f "dangling=true" -q); fi);
   - (rm -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml; cp -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml_init /DATA/DOCKERS/hongsnet-dev/docker-compose.yml);
   - (sed 's/init/$BUILD_SHA/g' -i /DATA/DOCKERS/hongsnet-dev/docker-compose.yml);
   - cd /DATA/DOCKERS/hongsnet-dev; docker-compose up -d
  tags:
   - devel

!중요 : K8s의 경우 Container Image Tag를 latest로 배포하는 것을 권장하지 않는다.

배포과정 : Image set -> Rolling Update history 출력 -> Rolling Update watching -> 배포완료

컨테이너 배포 절차

  • STEP 1 : devel 브랜치를 Checkout 한다.
# git checkout devel

# git branch
* devel
  deploy
  master
  • STEP 2 : 컨테이너 이미지에 대한 작업을 수행한다.

Working...

컨테이너 이미지 생성 시 Git의 Commit SHA(BUILD_SHA) 값으로 컨테이너의 Tagging을 지정한다.

(rm -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml; cp -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml_init /DATA/DOCKERS/hongsnet-dev/docker-compose.yml);

docker-compose.yml 파일내역은 다음과 같다.

# cat docker-compose.yml_init
version: '3'
services:
  hongsnet:
       image: hongsnet-dev:init
       hostname: 'dev.hongsnet.net'
       volumes:
        - "/DATA/SERVICE:/home"
        - "/WEB_DATA/EDU/Data:/home/edu/public_html/HongsBoard/Data"
        - "/WEB_DATA/EDU/Web_editor/EDU:/home/edu/public_html/HongsBoard/Web_editor/EDU"
        - "/WEB_DATA/EDU/Web_editor/FILE:/home/edu/public_html/HongsBoard/Web_editor/FILE"
        - "/WEB_DATA/HONGSNET/Data:/home/hongsnet/public_html/Data"
        - "/WEB_DATA/HONGSNET/Web_editor/FILE:/home/hongsnet/public_html/Web_editor/FILE"
        - "/WEB_DATA/NEWHONGSYSTEM/Data:/home/newhongsystem/public_html/Data"
        - "/WEB_DATA/NEWHONGSYSTEM/Web_editor/FILE:/home/newhongsystem/public_html/Web_editor/FILE"
       ports:
        - "80:80"

!참고 .gitlab-ci.yml 파일에 정의된 내역을 보면 다음과 같이 Git Commit SHA(BUILD_SHA) 값으로 변경하는 명령이 존재한다.

- (sed 's/init/$BUILD_SHA/g' -i /DATA/DOCKERS/hongsnet-dev/docker-compose.yml);

이는 Commit 된 SHA 값을 향후 참조하기 위한 용도로 사용된다.

  • STEP 3 : STAG 시스템에서 Docker Image의 상태를 확인한다.
# docker ps
CONTAINER ID   IMAGE                                                   COMMAND                  CREATED         STATUS         PORTS                              NAMES
29bc33723517   hongsnet-dev:2283829f44f6e5b7b17393ff6861553c13170a9d   "/usr/bin/supervisor…"   8 minutes ago   Up 8 minutes   0.0.0.0:80->80/tcp                 hongsnet-dev_hongsnet_1
0779415724e2   portainer/portainer-ce:latest                           "/portainer"             4 weeks ago     Up 4 weeks     8000/tcp, 0.0.0.0:9000->9000/tcp   portainer
  • STEP 4 : Stag 단계에서 문제가 없다면, 다음과 같이 배포하고, 실제 서비스에 적용한다.

먼저 deploy 브랜치로 Checkout 한다.

# git checkout deploy
Switched to branch 'deploy'

!중요 Stag 과정 중에 Commit 된 $BUILD_SHA_RESULT 변수를 /root/release_image 파일에 업데이트함으로써 증적을 기록한다.

# cat Dockerfile
...중략
RUN echo "hongsnet, 2020-12-20(2283829f44f6e5b7b17393ff6861553c13170a9d)" > /root/release_image

이제 add/commit/push를 진행하면 된다.

# git add hongsnet/Dockerfile
# git commit -m "2020-12-20(2283829f44f6e5b7b17393ff6861553c13170a9d)"
# git push -u origin deploy

K8s Deploy Review

  • Image Pull Policy
imagePullPolicy: IfNotPresent

기본 풀(pull) 정책은 IfNotPresent이며, 이것은 kubelet이 이미 존재하는 이미지에 대한 Pull을 생략하게 한다.

  • Gitlab Pipeline 결과

container_dev

  • Pod Describe 결과
# kubectl describe pod hongsnet-web |grep 20201220
    Image:          registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1
  Normal  Pulling    8m14s  kubelet            Pulling image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1"
  Normal  Pulled     8m13s  kubelet            Successfully pulled image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" in 585.920698ms
    Image:          registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1
  Normal  Pulling    8m4s  kubelet            Pulling image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1"
  Normal  Pulled     8m2s  kubelet            Successfully pulled image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" in 2.254472117s
    Image:          registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1
  Normal  Pulled     7m52s  kubelet            Container image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" already present on machine
    Image:          registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1
  Normal  Pulling    8m9s   kubelet            Pulling image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1"
  Normal  Pulled     8m8s   kubelet            Successfully pulled image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" in 543.885153ms
    Image:          registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1
  Normal  Pulled     7m56s  kubelet            Container image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" already present on machine
  • K8s Dashboard 확인

container_dev

  • Pod Update 확인

container_update