Docker Storage 검토

컨테이너는 원격 NFS 서버로의 마운트를 수행하여, 서비스 한다.

Overview

NFS Storage Overview

Host에 다음과 같이 Infiniband(HCA)가 설치되어 있으며, 전용 Infiniband Switch로 연결되어 있다.

# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x248a070300607bc0
        System image GUID: 0x248a070300607bc3
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 6
                LMC: 0
                SM lid: 1
                Capability mask: 0x02514868
                Port GUID: 0x248a070300607bc1
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02514868
                Port GUID: 0x248a070300607bc2
                Link layer: InfiniBand

IP 대역은 192.192.0.0/16 대역이 설정되어 있다.

# ifconfig ib0
ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044
        inet6 fe80::268a:703:60:7bc1  prefixlen 64  scopeid 0x20<link>
        inet 192.192.0.254  netmask 255.255.0.0  broadcast 192.192.255.255
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
        infiniband 80:00:02:08:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)
        RX packets 100840202  bytes 37952388228 (35.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 83620153  bytes 34850877380 (32.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Bandwidth는 40 Gbps 이며, IP 정보는 다음과 같다.

# ethtool ib0
Settings for ib0:
        Supported ports: [ ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 255
        Transceiver: internal
        Auto-negotiation: on

NFS Export 현황은 다음과 같다.

NFS Configure

# exportfs
/WEB_DATA       192.192.0.0/16
/WEB_SRC        192.192.0.0/16

Swarm Deploy(use docker-compose)

# cat docker-stack.yml
version: '3'
services:
  hongsnet:
       image: registry.hongsnet.net/joohan.hong/docker/hongsnet:latest
       volumes:
        - HOME:/home
        - EDU_DATA:/home/edu/public_html/HongsBoard/Data
        - EDU_EDITOR:/home/edu/public_html/HongsBoard/Web_editor/EDU"
        - EDU_FILE:/home/edu/public_html/HongsBoard/Web_editor/FILE"
        - HONGS_DATA:/home/hongsnet/public_html/Data"
        - HONGS_EDITOR:/home/hongsnet/public_html/Web_editor/FILE"
        - NEWSYSTEM_DATA:/home/newhongsystem/public_html/Data"
        - NEWSYSTEM_EDITOR:/home/newhongsystem/public_html/Web_editor/FILE"
       ports:
        - "80:80"
        deploy:
          mode: replicated
          replicas: 5
          placement:
            constraints: [node.hostname != TB2-DOCKER]
          update_config:
            parallelism: 5
            delay: 10s
          restart_policy:
            condition: on-failure
            max_attempts: 3
            window: 120s

volumes:
  HOME:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_SRC/home"
  EDU_DATA:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/EDU/Data"
  EDU_EDITOR:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/EDU/Web_editor/EDU"
  EDU_FILE:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/EDU/Web_editor/FILE"
  HONGS_DATA:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/HONGSNET/Data"
  HONGS_EDITOR:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/HONGSNET/Web_editor/FILE"
  NEWSYSTEM_DATA:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/NEWHONGSYSTEM/Data"
  NEWSYSTEM_EDITOR:
   driver_opts:
     type: "nfs"
     o: "addr=192.192.0.254,nolock,soft,rw"
     device: ":/WEB_DATA/NEWHONGSYSTEM/Web_editor/FILE"