elk web, init

LOG/WEB/README.md

+[![logo](https://www.hongsnet.net/images/logo.gif)](https://www.hongsnet.net)
+
+# 개요
+
+ELK Stack을 이용한 www.hongsnet.net 웹 로그분석을 운영 한다.
+
+
+# ELK Stack 구성 참조
+
+[바로가기](https://gitlab.hongsnet.net/joohan.hong/joohanhong/tree/master/LOG)
+
+
+# www.hongsnet.net WEB Log 구성
+
+![elk_stack_hongnset](./images/elk-stack-hongsnet.png)
+
+
+- **Apache Access_Logs 설정**
+
+httpd.conf 파일에 LogFormat을 설정한다.
+
+```bash
+SetEnvIf Request_URI \.gif do_not_log
+SetEnvIf Request_URI \.jpg do_not_log
+SetEnvIf Request_URI \.png do_not_log
+SetEnvIf Request_URI \.bmp do_not_log
+SetEnvIf Request_URI \.swf do_not_log
+SetEnvIf Request_URI \.js do_not_log
+SetEnvIf Request_URI \.css do_not_log
+LogFormat "%v %{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" proxy
+```
+
+위의 설정 중 **do_not_log** 는 image/js/css/swf 파일에 대한 로그이므로, 이를 남기지 않는다는 설정이다.
+
+
+- **Logstash 설정**
+
+```bash
+# cat /etc/logstash/conf.d/logstash.conf
+input {
+  beats {
+    port => 5444
+    host => "0.0.0.0"
+    client_inactivity_timeout => "1200"
+  }
+
+  udp {
+    port => 514
+    host => "0.0.0.0"
+    type => "syslog"
+  }
+}
+
+filter {
+
+        if "apache_access" in [tags] {
+          grok {
+             match => { "message" => [ "%{URIHOST:[vhost]} %{IPORHOST:[clientip]} - - \[%{HTTPDATE:[timestamp]}\] \"%{WORD:[method]} %{DATA:[request]} HTTP/%{NUMBER:[http_version]}\" %{NUMBER:[response]} (?:%{NUMBER:[bytes]}|-) (\"%{DATA:referrer}\") ?(\"%{DATA:user-agent}\")?" ] }
+          }
+          useragent {
+             source => "user-agent"
+             prefix => "agents_"
+          }
+          mutate {
+             remove_field => ["ecs","ident","auth"]
+             #remove_field => ["agent","agentest","event","ecs","fileset","build","user_name"]
+             convert => {"status" => "integer"
+                      "bytes" => "integer"
+                      "request_time" => "float"
+                      "geoip.city_name" => "string"
+                      "vhost" => "string"
+                      "agents_name" => "string"
+                      "agents_os" => "string"
+                      "agents_os_name" => "string"
+                      "agents_device" => "string"
+                      "user_agent" => "string"}
+             }
+             geoip {
+               source => "clientip"
+             }
+
+        }else if "apache_error" in [tags] {
+            grok {
+              patterns_dir => [ "/etc/logstash/conf.d/patterns" ]
+              match => { "message" => "%{APACHE_ERROR_LOG}"}
+            }
+            geoip {
+              source => "clientip"
+           }
+        }
+
+}
+
+filter {
+
+    if "osmessages" in [tags] {
+         grok {
+             match => ["message", "Error updating SMART data: Error sending ATA command CHECK"]
+             add_tag => "HDD_SMART_CHECK_ERROR"
+         }
+
+    }
+    else if "secure" in [tags] {
+         geoip {
+             source => "sshd_client_ip"
+         }
+    }
+}
+
+output {
+      elasticsearch {
+        hosts => ["localhost:9200", "172.16.0.228:9200"]
+        manage_template => false
+        index => "logstash-%{+YYYY.MM.dd}"
+      }
+}
+
+```
+
+
+# 핵심 내역
+
+아파치의 로그를 Filebeat가 전송(ship)하면, Logstash에서는 다음의 설정을 기준으로 로그를 정제하여, ElasticSearch에 전송하는 것이다.
+
+- **Log 정제**
+
+```plaintext
+match => { "message" => [ "%{URIHOST:[vhost]} %{IPORHOST:[clientip]} - - \[%{HTTPDATE:[timestamp]}\] \"%{WORD:[method]} %{DATA:[request]} HTTP/%{NUMBER:[http_version]}\" %{NUMBER:[response]} (?:%{NUMBER:[bytes]}|-) (\"%{DATA:referrer}\") ?(\"%{DATA:user-agent}\")?" ] }
+```
+
+- **ElasticSearch 전송**
+
+```plaintext
+elasticsearch {
+        hosts => ["localhost:9200", "172.16.0.228:9200"]
+        manage_template => false
+        index => "logstash-%{+YYYY.MM.dd}"
+      }
+```