Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
J
joohanhong
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
JooHan Hong
joohanhong
Commits
2000aca7
Commit
2000aca7
authored
Mar 30, 2021
by
JooHan Hong
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
prometheus node_down 중간저장
parent
6cd0daac
Pipeline
#5313
passed with stages
in 46 seconds
Changes
1
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
42 additions
and
4 deletions
+42
-4
README.md
DOCKER/PROMETHEUS/ITEM/NODES/RESULT/DOWN/README.md
+42
-4
No files found.
DOCKER/PROMETHEUS/ITEM/NODES/RESULT/DOWN/README.md
View file @
2000aca7
...
...
@@ -5,9 +5,9 @@
> Node 장애 시 Alert을 수신하도록 구성하고 검증한다.
#
Verify
#
Configuration
3분 동안 Node의
Down
이 감지된 경우 Alert을 발생시키는 Rule
3분 동안 Node의
`Down`
이 감지된 경우 Alert을 발생시키는 Rule
-
**결과**
...
...
@@ -15,8 +15,46 @@
-
**검증 과정**
*
[
**STEP 1**
]
: Prometheus의 Graph 메뉴의 Expression에서 수식에 대한 검증을 진행한다.
![
node_down_verify
](
../../images/node_down_verify.png
)
*
[
**STEP 2**
]
: Prometheus Configmap 파일에 ITEM을 추가한다.
```
bash
# cat prometheus-config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
namespace: monitoring
data:
prometheus.rules: |-
groups
:
- name: Node
(
Host
)
Down alerts
rules:
- alert: Node down
expr
: up
{
job
=
"node_exporter"
}
==
0
for
: 3m
labels:
severity: fatal
annotations:
title:
"Node {{
$labels
.instance }} is down"
description:
"Failed to scrape {{
$labels
.job }} on {{
$labels
.instance }} for more than 3 minutes. Node seems down."
...하략
```
*
[
**STEP 3**
]
: kubectl 툴을 이용하여 Replace 처리하고, curl을 이용하여 POST
**reload**
Payload를 전송한다.
```
bash
# kubectl replace -f prometheus-config-map.yaml
# curl -X POST http://172.24.0.222:30003/-/reload
```
> 또는 prometheus를 재시작하면 즉시 적용된다.
# Verify
# Completed
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment