[![logo](https://www.hongsnet.net/images/logo.gif)](https://www.hongsnet.net)

# 개요

ELK Stack을 이용한 www.hongsnet.net 웹 로그분석을 운영 한다.


# ELK Stack 구성 참조

- [ELK Stack 구성내역 바로가기](https://gitlab.hongsnet.net/joohan.hong/joohanhong/tree/master/LOG)

- [ELK WEB Dashboard 바로가기](https://elk-demo.hongsnet.net/app/dashboards#/view/80c8bb60-55c1-11eb-b165-4db2745e2382?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:''),timeRestore:!f,title:'www.hongsnet.net%20%EC%9A%B4%EC%98%81%ED%98%84%ED%99%A9',viewMode:view))

> ELK DEMO의 **로그인 정보**는 포트폴리오 참조

# www.hongsnet.net WEB Dashboard Overview

![elk_stack_hongnset_web1](./images/elk_hongsnet_web1.png)
![elk_stack_hongnset_web2](./images/elk_hongsnet_web2.png)
![elk_stack_hongnset_web3](./images/elk_hongsnet_web3.png)
![elk_stack_hongnset_web4](./images/elk_hongsnet_web4.png)



# www.hongsnet.net WEB Log 구성

![elk_stack_hongnset](./images/elk-stack-hongsnet.png)


- **Apache Access_Logs 설정**

httpd.conf 파일에 LogFormat을 설정한다.

```bash
SetEnvIf Request_URI \.gif do_not_log
SetEnvIf Request_URI \.jpg do_not_log
SetEnvIf Request_URI \.png do_not_log
SetEnvIf Request_URI \.bmp do_not_log
SetEnvIf Request_URI \.swf do_not_log
SetEnvIf Request_URI \.js do_not_log
SetEnvIf Request_URI \.css do_not_log
LogFormat "%v %{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" proxy
```

위의 설정 중 **do_not_log** 는 image/js/css/swf 파일에 대한 로그이므로, 이를 남기지 않는다는 설정이다.


- **Logstash 설정**

```bash
# cat /etc/logstash/conf.d/logstash.conf
input {
  beats {
    port => 5444
    host => "0.0.0.0"
    client_inactivity_timeout => "1200"
  }

  udp {
    port => 514
    host => "0.0.0.0"
    type => "syslog"
  }
}

filter {

        if "apache_access" in [tags] {
          grok {
             match => { "message" => [ "%{URIHOST:[vhost]} %{IPORHOST:[clientip]} - - \[%{HTTPDATE:[timestamp]}\] \"%{WORD:[method]} %{DATA:[request]} HTTP/%{NUMBER:[http_version]}\" %{NUMBER:[response]} (?:%{NUMBER:[bytes]}|-) (\"%{DATA:referrer}\") ?(\"%{DATA:user-agent}\")?" ] }
          }
          useragent {
             source => "user-agent"
             prefix => "agents_"
          }
          mutate {
             remove_field => ["ecs","ident","auth"]
             #remove_field => ["agent","agentest","event","ecs","fileset","build","user_name"]
             convert => {"status" => "integer"
                      "bytes" => "integer"
                      "request_time" => "float"
                      "geoip.city_name" => "string"
                      "vhost" => "string"
                      "agents_name" => "string"
                      "agents_os" => "string"
                      "agents_os_name" => "string"
                      "agents_device" => "string"
                      "user_agent" => "string"}
             }
             geoip {
               source => "clientip"
             }

        }else if "apache_error" in [tags] {
            grok {
              patterns_dir => [ "/etc/logstash/conf.d/patterns" ]
              match => { "message" => "%{APACHE_ERROR_LOG}"}
            }
            geoip {
              source => "clientip"
           }
        }

}

filter {

    if "osmessages" in [tags] {
         grok {
             match => ["message", "Error updating SMART data: Error sending ATA command CHECK"]
             add_tag => "HDD_SMART_CHECK_ERROR"
         }

    }
    else if "secure" in [tags] {
         geoip {
             source => "sshd_client_ip"
         }
    }
}

output {
      elasticsearch {
        hosts => ["localhost:9200", "172.16.0.228:9200"]
        manage_template => false
        index => "logstash-%{+YYYY.MM.dd}"
      }
}

```


# 핵심 내역

아파치의 로그를 Filebeat가 전송(ship)하면, Logstash에서는 다음의 설정을 기준으로 로그를 정제하여, ElasticSearch에 전송하는 것이다.

- **Log 정제**

```plaintext
match => { "message" => [ "%{URIHOST:[vhost]} %{IPORHOST:[clientip]} - - \[%{HTTPDATE:[timestamp]}\] \"%{WORD:[method]} %{DATA:[request]} HTTP/%{NUMBER:[http_version]}\" %{NUMBER:[response]} (?:%{NUMBER:[bytes]}|-) (\"%{DATA:referrer}\") ?(\"%{DATA:user-agent}\")?" ] }
```

- **ElasticSearch 전송**

```plaintext
elasticsearch {
        hosts => ["localhost:9200", "172.16.0.228:9200"]
        manage_template => false
        index => "logstash-%{+YYYY.MM.dd}"
      }
```