Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
J
joohanhong
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
JooHan Hong
joohanhong
Commits
28bc810d
Commit
28bc810d
authored
Mar 31, 2021
by
JooHan Hong
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
memory low, 1차완료
parent
1b0183dd
Pipeline
#5322
passed with stages
in 49 seconds
Changes
5
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
5 changed files
with
79 additions
and
0 deletions
+79
-0
README.md
DOCKER/PROMETHEUS/ITEM/NODES/RESULT/MEMORYLOW/README.md
+79
-0
memory_free.png
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free.png
+0
-0
memory_free_alert.png
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_alert.png
+0
-0
memory_free_resolved.png
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_resolved.png
+0
-0
memory_free_verify.png
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_verify.png
+0
-0
No files found.
DOCKER/PROMETHEUS/ITEM/NODES/RESULT/MEMORYLOW/README.md
0 → 100644
View file @
28bc810d
[
![logo
](
https://www.hongsnet.net/images/logo.gif
)
](https://www.hongsnet.net)
# Node Memory 부족 검증
> 특정 Metric 값을 기준으로 메모리의 가용량을 모니터링 한다.
# Configuration
**10초**
동안 Node의 메모리가용량이
`10`
% 이하일 경우가 감지된 경우 Alert을 발생시키는 Rule
-
**결과**
![
memory_free
](
../../images/memory_free.png
)
-
**검증 과정**
*
[
**STEP 1**
]
: Prometheus의 Graph 메뉴의 Expression에서 수식에 대한 검증을 진행한다.
![
memory_free_verify
](
../../images/memory_free_verify.png
)
> 대상 Nodes에 현재 가용할 수 있는 메모리의 비율 값이 쿼리 된다.
*
[
**STEP 2**
]
: Prometheus Configmap 파일에 ITEM을 추가한다.
```
bash
# cat prometheus-config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
namespace: monitoring
data:
prometheus.rules: |-
groups
:
- name: Node Memory 10% Under Free
rules:
- alert: HostOutOfMemory 10% Under Free
expr
:
(
node_memory_MemAvailable_bytes /
(
node_memory_MemTotal_bytes
))
*
100 <
=
10
for
: 10s
annotations:
summary:
"Host out of memory instance {{
$labels
.instance }}"
description:
"{{
$labels
.instance }} has more than 10% Under of its memory free."
...하략
```
*
[
**STEP 3**
]
: kubectl 툴을 이용하여 Replace 처리하고, curl을 이용하여 POST
**reload**
Payload를 전송한다.
```
bash
# kubectl replace -f prometheus-config-map.yaml
# curl -X POST http://172.24.0.222:30003/-/reload
```
> 또는 prometheus를 재시작하면 즉시 적용된다.
# Verify
*
[
**STEP 1**
]
: 검증을 위해 10% 미만 기준을
**90**
%로 변경한다.
```
python
expr
:
(
node_memory_MemAvailable_bytes
/
(
node_memory_MemTotal_bytes
))
*
100
<=
90
```
*
[
**STEP 2**
]
: Alert Manager를 통한 Alert 발송을 확인한다. 90%로 변경되었기 때문에
**모든 Node가 해당된 상태**
이다.
![
memory_free_alert
](
../../images/memory_free_alert.png
)
*
[
**STEP 3**
]
: Resolved를 검증하기 위해 다시 기존 10%로 변경한다.
```
python
expr
:
(
node_memory_MemAvailable_bytes
/
(
node_memory_MemTotal_bytes
))
*
100
<=
10
```
*
[
**STEP 4**
]
: Alert Manager를 통한
`Resolved`
Alert 발송을 확인한다.
![
memory_free_resolved
](
../../images/memory_free_resolved.png
)
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free.png
0 → 100644
View file @
28bc810d
12 KB
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_alert.png
0 → 100644
View file @
28bc810d
56.7 KB
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_resolved.png
0 → 100644
View file @
28bc810d
57.3 KB
DOCKER/PROMETHEUS/ITEM/NODES/images/memory_free_verify.png
0 → 100644
View file @
28bc810d
22 KB
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment