Add alertmanager config #169

pinpox · 2024-04-26T22:15:30Z

pinpox commented

2024-04-26 22:15:30 +00:00

Added config for prometheus alertmanager
Added alerting rules

There are a few TODO's open, I marked them with comments accordingly so they can be easily grepped

- Added config for prometheus alertmanager - Added alerting rules There are a few TODO's open, I marked them with comments accordingly so they can be easily grepped

pinpox added 1 commit 2024-04-26 22:15:30 +00:00

Add alertmanager config

Flake checks / Check (pull_request) Failing after 4m2s

Details

8e66bea9c8

pinpox force-pushed main from e504392cf8 to a66c6ada59

2024-04-26 22:26:22 +00:00

Compare

pinpox added 1 commit 2024-04-26 22:26:55 +00:00

Autoformat dns.tf

Flake checks / Check (pull_request) Failing after 3m55s

Details

a98cfc82e5

b12f reviewed 2024-04-26 23:29:01 +00:00

hosts/flora-6/apps/prometheus.nix

					
				@ -68,0 +80,4 @@

				    alertmanager = {

				      enable = true;

				      # port = 9093; # Default

				      webExternalUrl = "https://alerts.pub.solar"; # TODO use a proper url?

b12f commented

2024-04-26 23:29:01 +00:00

You're missing the nginx config in front of the alertmanager

pinpox commented

2024-04-26 23:38:34 +00:00

fixed by d58209ef93

fixed by d58209ef93385735e00084fb02015918985dc06d

pinpox marked this conversation as resolved

pinpox added 1 commit 2024-04-26 23:38:10 +00:00

Add reverseproxy for alerts.pub.solar

Flake checks / Check (pull_request) Failing after 4m3s

Details

d58209ef93

~~teutat3s referenced this pull request 2024-04-27 13:52:42 +00:00~~

Resource alerts #168

teutat3s reviewed 2024-04-27 16:12:05 +00:00

teutat3s left a comment

Thanks a lot for this contribution! Would you be fine with us taking this to a branch and adding our fixes on top?

hosts/flora-6/apps/caddy.nix

					
				@ -40,0 +41,4 @@

				        logFormat = lib.mkForce ''

				          output discard

				        '';

				        extraConfig = ''

teutat3s commented

2024-04-27 16:11:14 +00:00

Here we'd want to use the bind directive to only listen on the WireGuard IP, exposing alertmanager only internally.

bind 10.7.6.2
tls internal

and then

reverse_proxy :${toString config.services.prometheus.alertmanager.port}

To have valid Let's Encrypt certificates, we'd need to configure DNS challenge, but that could be a future task IMO.

Here we'd want to use the `bind` directive to only listen on the WireGuard IP, exposing alertmanager only internally. ``` bind 10.7.6.2 tls internal ``` and then ``` reverse_proxy :${toString config.services.prometheus.alertmanager.port} ``` To have valid Let's Encrypt certificates, we'd need to configure DNS challenge, but that could be a future task IMO.

pinpox commented

2024-04-27 19:13:30 +00:00

Sure feel free to reuse as you wish. I might be able to join tomorrow if you are still hakking on it