pub-solar/infra
6
1
Fork 3
Code Issues 20 Pull requests 9 Actions Packages Projects 1 Releases Wiki Activity

alertmanager: alert on high load only after 20m #255

Merged
teutat3s merged 3 commits from alerts-tweak-load into main 2024-11-12 14:47:54 +00:00
Conversation 0 Commits 3 Files changed 1 +10 -10
teutat3s commented 2024-11-07 11:23:49 +00:00
Owner
Copy link

This should avoid load alerts by delite.

Also disable alert on high memory page faults, which is non actionable. We still monitor high memory usage and OOM kills.

Enable e2e_dead_man_switch.

This should avoid load alerts by delite. Also disable alert on high memory page faults, which is non actionable. We still monitor high memory usage and OOM kills. Enable `e2e_dead_man_switch`.
teutat3s added 1 commit 2024-11-07 11:23:49 +00:00
alertmanager: alert on high load only after 20m
All checks were successful
Flake checks / Check (pull_request) Successful in 2m8s
Details
afe52ca6af
teutat3s requested review from b12f 2024-11-07 11:23:55 +00:00
teutat3s requested review from hensoko 2024-11-07 11:23:55 +00:00
teutat3s requested review from axeman 2024-11-07 11:23:56 +00:00
teutat3s added 2 commits 2024-11-12 12:41:50 +00:00 
This alert is non actionable, we still monitor high memory usage.
alertmanager: enable e2e_dead_man_switch
All checks were successful
Flake checks / Check (pull_request) Successful in 23m13s
Details
ab85ba751a
hensoko approved these changes 2024-11-12 14:45:59 +00:00
modules/prometheus/alert-rules.nix
@ -234,10 +234,10 @@ lib.mapAttrsToList
};
*/
host_memory_under_memory_pressure = {
hensoko commented 2024-11-12 14:38:40 +00:00
Owner
Copy link

why is this check disabled?

why is this check disabled?
hensoko marked this conversation as resolved
teutat3s merged commit 73333537a5 into main 2024-11-12 14:47:54 +00:00
teutat3s deleted branch alerts-tweak-load 2024-11-12 14:47:54 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
b12f
axeman
hensoko
Labels
Clear labels   
Bug

   
Docs

   
Feature request

   
Good for newcomers

   
Infra

   
Privacy

   
Refactoring or migration

   
Security
No labels
Bug
Docs
Feature request
Good for newcomers
Infra
Privacy
Refactoring or migration
Security
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: pub-solar/infra#255
No description provided.
Delete branch "alerts-tweak-load"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?