Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
J
joohanhong
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
JooHan Hong
joohanhong
Commits
28bc810d
Commit
28bc810d
authored
4 years ago
by
JooHan Hong
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
memory low, 1차완료
parent
1b0183dd
master
No related merge requests found
Pipeline
#5322
passed with stages
in 49 seconds
Changes
5
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
5 changed files
with
79 additions
and
0 deletions
+79
-0
README.md
DOCKER/PROMETHEUS/ITEM/NODES/RESULT/MEMORYLOW/README.md
+79
-0
memory_free.png
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free.png
+0
-0
memory_free_alert.png
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_alert.png
+0
-0
memory_free_resolved.png
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_resolved.png
+0
-0
memory_free_verify.png
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_verify.png
+0
-0
No files found.
DOCKER/PROMETHEUS/ITEM/NODES/RESULT/MEMORYLOW/README.md
0 → 100644
View file @
28bc810d
[

](https://www.hongsnet.net)
# Node Memory 부족 검증
> 특정 Metric 값을 기준으로 메모리의 가용량을 모니터링 한다.
# Configuration
**10초**
동안 Node의 메모리가용량이
`10`
% 이하일 경우가 감지된 경우 Alert을 발생시키는 Rule
-
**결과**

-
**검증 과정**
*
[
**STEP 1**
]
: Prometheus의 Graph 메뉴의 Expression에서 수식에 대한 검증을 진행한다.

> 대상 Nodes에 현재 가용할 수 있는 메모리의 비율 값이 쿼리 된다.
*
[
**STEP 2**
]
: Prometheus Configmap 파일에 ITEM을 추가한다.
```
bash
# cat prometheus-config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
namespace: monitoring
data:
prometheus.rules: |-
groups
:
- name: Node Memory 10% Under Free
rules:
- alert: HostOutOfMemory 10% Under Free
expr
:
(
node_memory_MemAvailable_bytes /
(
node_memory_MemTotal_bytes
))
*
100 <
=
10
for
: 10s
annotations:
summary:
"Host out of memory instance {{
$labels
.instance }}"
description:
"{{
$labels
.instance }} has more than 10% Under of its memory free."
...하략
```
*
[
**STEP 3**
]
: kubectl 툴을 이용하여 Replace 처리하고, curl을 이용하여 POST
**reload**
Payload를 전송한다.
```
bash
# kubectl replace -f prometheus-config-map.yaml
# curl -X POST http://172.24.0.222:30003/-/reload
```
> 또는 prometheus를 재시작하면 즉시 적용된다.
# Verify
*
[
**STEP 1**
]
: 검증을 위해 10% 미만 기준을
**90**
%로 변경한다.
```
python
expr
:
(
node_memory_MemAvailable_bytes
/
(
node_memory_MemTotal_bytes
))
*
100
<=
90
```
*
[
**STEP 2**
]
: Alert Manager를 통한 Alert 발송을 확인한다. 90%로 변경되었기 때문에
**모든 Node가 해당된 상태**
이다.

*
[
**STEP 3**
]
: Resolved를 검증하기 위해 다시 기존 10%로 변경한다.
```
python
expr
:
(
node_memory_MemAvailable_bytes
/
(
node_memory_MemTotal_bytes
))
*
100
<=
10
```
*
[
**STEP 4**
]
: Alert Manager를 통한
`Resolved`
Alert 발송을 확인한다.

This diff is collapsed.
Click to expand it.
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free.png
0 → 100644
View file @
28bc810d
12 KB
This diff is collapsed.
Click to expand it.
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_alert.png
0 → 100644
View file @
28bc810d
56.7 KB
This diff is collapsed.
Click to expand it.
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_resolved.png
0 → 100644
View file @
28bc810d
57.3 KB
This diff is collapsed.
Click to expand it.
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_verify.png
0 → 100644
View file @
28bc810d
22 KB
This diff is collapsed.
Click to expand it.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment