[![logo](https://www.hongsnet.net/images/logo.gif)](https://www.hongsnet.net) # www.hongsnet.net Container Deploy (`Current`) ![source_overview](./images/hongsnet-container-deploy.png) ## 구성 요소 * Git and GitLab (gitlab-ci.yml) * gitlab-runner * docker-compose (**STAG** 시스템에서 사용) * **K8s** ## GitLab Runner Overview Runner는 yaml을 파일을 수행한다. 특정 프로젝트에 국한되거나, 여러 프로젝트에서 사용할 수 있도록 제공된다. 여기서 모든 여러 프로젝트에서 사용할 수 있는 Runner를 **Shared Runner**이라고 하며, 특정 프로젝트에 국한되서 사용되는 Runner를 **Specific Runner**라고 한다. 홍쓰넷 컨테이너 배포에는 `Specific Runner`를 사용한다. ## Container Deploy Overview (`Current`) - **STEP 1** : STAG 시스템에 먼저 Docker Image를 배포한 후 검증한다. > 사전에 devel 브랜치로 checkout을 수행한다. - **STEP 2** : **STAG** 시스템에서 검증이 완료되면, 다음과 같이 두 가지 Stage가 수행된다. - **push** : GitLAB의 Container Registry에 Container Image를 PUSH 한다. - **deploy** : K8s의 Master Node에 Image 배포를 명령한다. ## www.hongsnet.net **Dockerfile** ```bash # cat Dockerfile FROM registry.hongsnet.net/joohan.hong/docker/centos:7.6.1810 MAINTAINER Hongs ENV TZ=Asia/Seoul RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\ yum -y install epel-release ENV LANG en_US.UTF-8 ENV LC_ALL en_US.UTF-8 RUN yum -y install glibc glibc-common freetype-devel libjpeg-devel openssl-devel net-tools vi vim iproute vsftpd wget postfix cronie crontabs supervisor COPY config/libmcrypt-config /usr/local/bin/libmcrypt-config COPY config/mm-config /usr/local/bin/mm-config COPY config/mcrypt.h /usr/local/include/mcrypt.h COPY config/mm.h /usr/local/include/mm.h COPY config/libmm.so.14.0.22 /usr/local/lib/libmm.so.14.0.22 COPY config/libmm.so.14 /usr/local/lib/libmm.so.14 COPY config/libmm.so /usr/local/lib/libmm.so COPY config/libmhash.a /usr/local/lib/libmhash.a COPY config/libmhash.la /usr/local/lib/libmhash.la COPY config/libmhash.so /usr/local/lib/libmhash.so COPY config/libmhash.so.2 /usr/local/lib/libmhash.so.2 COPY config/libmhash.so.2.0.1 /usr/local/lib/libmhash.so.2.0.1 COPY config/libmcrypt.la /usr/local/lib/libmcrypt.la COPY config/libmcrypt.so /usr/local/lib/libmcrypt.so COPY config/libmcrypt.so.4 /usr/local/lib/libmcrypt.so.4 COPY config/libmcrypt.so.4.4.7 /usr/local/lib/libmcrypt.so.4.4.7 RUN ln -s /usr/lib64/libpng.so /usr/lib/libpng.so &&\ ln -s /usr/lib64/libjpeg.so /usr/lib/libjpeg.so COPY config/bashrc /root/.bashrc COPY config/main.cf /etc/postfix/main.cf COPY config/master.cf /etc/postfix/master.cf COPY binary/mysql /usr/local/mysql COPY binary/apache /usr/local/apache RUN chmod 701 /usr/local/apache/htdocs COPY binary/php /usr/local/php COPY binary/curl /usr/local/curl RUN mkdir /usr/lib64/curl_backup RUN mv /usr/lib64/libcurl.so* /usr/lib64/curl_backup/ RUN cp -rf /usr/local/curl/lib/libcurl.so* /usr/lib64/ RUN rpm -ivh http://pds.hongsnet.net:8888/packages/filebeat-7.9.3-x86_64.rpm COPY config/filebeat.yml /etc/filebeat/filebeat.yml COPY config/supervisord.conf /etc/supervisor/supervisord.conf COPY config/hongsnet.conf /etc/supervisor/conf.d/hongsnet.conf EXPOSE 80 RUN echo "hongsnet, 2020-12-20(2283829f44f6e5b7b17393ff6861553c13170a9d)" > /root/release_image CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/supervisord.conf"] ``` 다음은 배포에 대한 정책을 수행하는 **.gitlab-ci.yml** 파일에 대한 내역이다. ## GitLAB .gitlab-ci.yml 파일의 작성 ```python # cat .gitlab-ci.yml stages: - devel - push - deploy variables: GIT_STRATEGY: clone GIT_SSL_NO_VERIFY: "1" GITLAB_TOKEN: "XXXXXX" BUILD_IMAGE: ${CI_PROJECT_DIR}/hongsnet BUILD_SHA: ${CI_COMMIT_SHA} BUILD_SHA_RESULT: "2283829f44f6e5b7b17393ff6861553c13170a9d" #deploy tag, requirement BUILD_VERSION: "20201220_v1" before_script: #- export RELEASE_DATE=`date +"%Y%m%d"` push: stage: push only: - deploy script: - /usr/bin/docker login -u juhanida21@nate.com -p $GITLAB_TOKEN registry.hongsnet.net - /usr/bin/docker tag hongsnet-dev:$BUILD_SHA_RESULT registry.hongsnet.net/joohan.hong/docker/hongsnet:$BUILD_VERSION - /usr/bin/docker push registry.hongsnet.net/joohan.hong/docker/hongsnet:$BUILD_VERSION tags: - deploy deploy: stage: deploy only: - deploy script: - /usr/bin/docker login -u juhanida21@nate.com -p $GITLAB_TOKEN registry.hongsnet.net - /usr/bin/kubectl set image deployment/hongsnet-web hongsnet-web=registry.hongsnet.net/joohan.hong/docker/hongsnet:$BUILD_VERSION - /usr/bin/kubectl rollout history deployment/hongsnet-web - /usr/bin/kubectl rollout status deployment/hongsnet-web tags: - backup build: stage: devel only: - devel script: - cd /DATA/DOCKERS/hongsnet-dev; docker-compose down - cd ${BUILD_IMAGE} - /usr/bin/docker build --rm -t hongsnet-dev:$BUILD_SHA . - (if [[ `docker images -f "dangling=true" -q` != "" ]]; then docker rmi $(docker images -f "dangling=true" -q); fi); - (rm -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml; cp -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml_init /DATA/DOCKERS/hongsnet-dev/docker-compose.yml); - (sed 's/init/$BUILD_SHA/g' -i /DATA/DOCKERS/hongsnet-dev/docker-compose.yml); - cd /DATA/DOCKERS/hongsnet-dev; docker-compose up -d tags: - devel ``` > !중요 : **K8s**의 경우 Container Image Tag를 `latest`로 배포하는 것을 권장하지 않는다. > 배포과정 : **Image set** -> Rolling Update history 출력 -> Rolling Update watching -> 배포완료 ## 컨테이너 배포 절차 - **STEP 1** : devel 브랜치를 Checkout 한다. ```bash # git checkout devel # git branch * devel deploy master ``` - **STEP 2** : 컨테이너 이미지에 대한 작업을 수행한다. > Working... 컨테이너 이미지 생성 시 **Git의 Commit SHA(BUILD_SHA) 값으로 컨테이너의 Tagging을 지정**한다. ```python (rm -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml; cp -rf /DATA/DOCKERS/hongsnet-dev/docker-compose.yml_init /DATA/DOCKERS/hongsnet-dev/docker-compose.yml); ``` **docker-compose.yml** 파일내역은 다음과 같다. ```bash # cat docker-compose.yml_init version: '3' services: hongsnet: image: hongsnet-dev:init hostname: 'dev.hongsnet.net' volumes: - "/DATA/SERVICE:/home" - "/WEB_DATA/EDU/Data:/home/edu/public_html/HongsBoard/Data" - "/WEB_DATA/EDU/Web_editor/EDU:/home/edu/public_html/HongsBoard/Web_editor/EDU" - "/WEB_DATA/EDU/Web_editor/FILE:/home/edu/public_html/HongsBoard/Web_editor/FILE" - "/WEB_DATA/HONGSNET/Data:/home/hongsnet/public_html/Data" - "/WEB_DATA/HONGSNET/Web_editor/FILE:/home/hongsnet/public_html/Web_editor/FILE" - "/WEB_DATA/NEWHONGSYSTEM/Data:/home/newhongsystem/public_html/Data" - "/WEB_DATA/NEWHONGSYSTEM/Web_editor/FILE:/home/newhongsystem/public_html/Web_editor/FILE" ports: - "80:80" ``` **!참고** .gitlab-ci.yml 파일에 정의된 내역을 보면 다음과 같이 **Git Commit SHA(BUILD_SHA) 값**으로 변경하는 명령이 존재한다. ```python - (sed 's/init/$BUILD_SHA/g' -i /DATA/DOCKERS/hongsnet-dev/docker-compose.yml); ``` 이는 `Commit 된 SHA 값`을 향후 **참조하기 위한 용도**로 사용된다. - **STEP 3** : **STAG** 시스템에서 Docker Image의 상태를 확인한다. ```bash # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 29bc33723517 hongsnet-dev:2283829f44f6e5b7b17393ff6861553c13170a9d "/usr/bin/supervisor…" 8 minutes ago Up 8 minutes 0.0.0.0:80->80/tcp hongsnet-dev_hongsnet_1 0779415724e2 portainer/portainer-ce:latest "/portainer" 4 weeks ago Up 4 weeks 8000/tcp, 0.0.0.0:9000->9000/tcp portainer ``` - **STEP 4** : Stag 단계에서 문제가 없다면, 다음과 같이 배포하고, **실제 서비스에 적용**한다. 먼저 deploy 브랜치로 Checkout 한다. ```bash # git checkout deploy Switched to branch 'deploy' ``` **!중요** Stag 과정 중에 Commit 된 $BUILD_SHA_RESULT 변수를 /root/release_image 파일에 업데이트함으로써 증적을 기록한다. ```bash # cat Dockerfile ...중략 RUN echo "hongsnet, 2020-12-20(2283829f44f6e5b7b17393ff6861553c13170a9d)" > /root/release_image ``` 이제 add/commit/push를 진행하면 된다. ```bash # git add hongsnet/Dockerfile # git commit -m "2020-12-20(2283829f44f6e5b7b17393ff6861553c13170a9d)" # git push -u origin deploy ``` # K8s Deploy Review - **Image Pull Policy** ```python imagePullPolicy: IfNotPresent ``` > 기본 풀(pull) 정책은 IfNotPresent이며, 이것은 kubelet이 **이미 존재하는 이미지에 대한 Pull을 생략**하게 한다. - **Gitlab Pipeline 결과** ![container_dev](./images/gitlab_ci_cd_container.png) - **Pod Describe 결과** ```bash # kubectl describe pod hongsnet-web |grep 20201220 Image: registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1 Normal Pulling 8m14s kubelet Pulling image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" Normal Pulled 8m13s kubelet Successfully pulled image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" in 585.920698ms Image: registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1 Normal Pulling 8m4s kubelet Pulling image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" Normal Pulled 8m2s kubelet Successfully pulled image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" in 2.254472117s Image: registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1 Normal Pulled 7m52s kubelet Container image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" already present on machine Image: registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1 Normal Pulling 8m9s kubelet Pulling image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" Normal Pulled 8m8s kubelet Successfully pulled image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" in 543.885153ms Image: registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1 Normal Pulled 7m56s kubelet Container image "registry.hongsnet.net/joohan.hong/docker/hongsnet:20201220_v1" already present on machine ``` - **K8s Dashboard 확인** ![container_dev](./images/k8s-dashboard-ci_cd.png) - **Pod Update 확인** ![container_update](./images/k8s-dashboard-console.png)