pub-solar/infra
6
1
Fork 3
Code Issues 16 Pull requests 7 Actions Packages Projects 1 Releases Wiki Activity

loki, prometheus, promtail should connect via wireguard #200

Merged
teutat3s merged 7 commits from loki-prometheus-via-wireguard into main 2024-06-05 00:04:40 +00:00
Conversation 0 Commits 7 Files changed 11 +83 -169
teutat3s commented 2024-06-01 15:09:20 +00:00
Owner
Copy link

This changes the behaviour that was implemented before we had wireguard connections between hosts. Previously, HTTPS + basic auth was used to push logs to loki and scrape metrics using prometheus. With the wireguard VPN between hosts, this is no longer necessary. Instead, we bind to the wireguard interface wg-ssh (or its IPv4+IPv6 IP addresses) and use HTTP tunnelled via wireguard.

Also add new *.wg.pub.solar DNS A + AAAA records to get the wireguard IPs of our hosts.

This changes the behaviour that was implemented before we had wireguard connections between hosts. Previously, HTTPS + basic auth was used to push logs to loki and scrape metrics using prometheus. With the wireguard VPN between hosts, this is no longer necessary. Instead, we bind to the wireguard interface `wg-ssh` (or its IPv4+IPv6 IP addresses) and use HTTP tunnelled via wireguard. Also add new *.wg.pub.solar DNS `A` + `AAAA` records to get the wireguard IPs of our hosts.
teutat3s added 3 commits 2024-06-01 15:09:21 +00:00 
wireguard to secure connections
networking: remove nachtigall + flora-6 from /etc/hosts
Some checks failed
Flake checks / Check (pull_request) Failing after 4s
Details
04b05a9c2d 
to reduce suprises when DNS records are different depending on the host.

For wireguard IPs, we should use the *.wg.pub.solar DNS records instead.
teutat3s requested review from b12f 2024-06-01 15:09:26 +00:00
teutat3s requested review from hensoko 2024-06-01 15:09:26 +00:00
teutat3s requested review from axeman 2024-06-01 15:09:27 +00:00
teutat3s added 1 commit 2024-06-01 15:24:09 +00:00
docs: update unlocking ZFS pool
All checks were successful
Flake checks / Check (pull_request) Successful in 3m52s
Details
62add8e095
b12f reviewed 2024-06-02 18:47:07 +00:00
modules/core/networking.nix Outdated
@ -27,11 +27,6 @@
# Don't expose SSH via public interfaces
networking.firewall.interfaces.wg-ssh.allowedTCPPorts = [ 22 ];
networking.hosts = {
b12f commented 2024-06-02 18:47:07 +00:00
Owner
Copy link

Is there a reason to move this into the global DNS? we can connect to other nodes in the network without DNS being available, so it might be nice to also have their local names without a DNS server being available.

Is there a reason to move this into the global DNS? we can connect to other nodes in the network [without DNS being available](https://git.pub.solar/pub-solar/infra/src/branch/main/hosts/nachtigall/wireguard.nix#L24), so it might be nice to also have their local names without a DNS server being available.
teutat3s commented 2024-06-03 08:26:42 +00:00
Author
Owner
Copy link

As the commit message states, the goal is to avoid surprises. I find it very confusing to have the same DNS name resolve to different IP addresses depending what host you're on. It would be my preference to remove this for clarity.

As the commit message states, the goal is to avoid surprises. I find it very confusing to have the same DNS name resolve to different IP addresses depending what host you're on. It would be my preference to remove this for clarity.
hensoko commented 2024-06-03 09:48:56 +00:00
Owner
Copy link

But to get this straight: We have the internal "wg.pub.solar" hosts that would could additionally set statically via networking.hosts right? @teutat3s

I tend to agree with @b12f about a possible fuckup because of broken DNS but the middle ground here would be to not have context based in relation to the host your on but cached dns entries. It creates a bit of additional maintenance work as we have to update both but that could be ruled out by mentioning it in the docs and hanging people for not RTFM.

But to get this straight: We have the internal "wg.pub.solar" hosts that would could additionally set statically via `networking.hosts` right? @teutat3s I tend to agree with @b12f about a possible fuckup because of broken DNS but the middle ground here would be to not have context based in relation to the host your on but cached dns entries. It creates a bit of additional maintenance work as we have to update both but that could be ruled out by mentioning it in the docs and hanging people for not RTFM.
teutat3s commented 2024-06-03 10:00:01 +00:00
Author
Owner
Copy link

I have no objections to setting *.wg.pub.solar in /etc/hosts as well if you're suggesting that. Not sure if I understood your comment correctly, @hensoko.

I have no objections to setting `*.wg.pub.solar` in `/etc/hosts` as well if you're suggesting that. Not sure if I understood your comment correctly, @hensoko.
teutat3s commented 2024-06-03 10:31:20 +00:00
Author
Owner
Copy link

Better like this? 56f692740e

Better like this? https://git.pub.solar/pub-solar/infra/commit/56f692740e31c183b7624f9d81f4ebf1237b08ec
hensoko commented 2024-06-03 12:31:25 +00:00
Owner
Copy link

Yessss. @b12f also happy?

Yessss. @b12f also happy?
teutat3s marked this conversation as resolved
teutat3s force-pushed loki-prometheus-via-wireguard from 62add8e095 to 8f1b932fdc 2024-06-03 10:30:15 +00:00 Compare
teutat3s added 1 commit 2024-06-03 10:34:19 +00:00
networking: add internal IPv6 wireguard IPs to /etc/hosts
All checks were successful
Flake checks / Check (pull_request) Successful in 3m8s
Details
61ea0ad7c2
hensoko approved these changes 2024-06-03 12:31:10 +00:00
hensoko approved these changes 2024-06-03 12:31:35 +00:00
b12f approved these changes 2024-06-04 07:20:31 +00:00
teutat3s added 2 commits 2024-06-05 00:00:19 +00:00
nginx: use square brackets for IPv6 address
All checks were successful
Flake checks / Check (pull_request) Successful in 4m0s
Details
e93a56e594
teutat3s merged commit 10ed117dfe into main 2024-06-05 00:04:40 +00:00
teutat3s deleted branch loki-prometheus-via-wireguard 2024-06-05 00:04:40 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
axeman
hensoko
b12f
Labels
Clear labels   
Bug

   
Docs

   
Feature request

   
Good for newcomers

   
Infra

   
Privacy

   
Refactoring or migration

   
Security
No labels
Bug
Docs
Feature request
Good for newcomers
Infra
Privacy
Refactoring or migration
Security
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: pub-solar/infra#200
No description provided.
Delete branch "loki-prometheus-via-wireguard"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?