From 015327b5e35b4ca1eac106ab2b14566777950b24 Mon Sep 17 00:00:00 2001 From: Earl Warren Date: Sun, 29 Sep 2024 17:52:24 +0200 Subject: [PATCH] hetzner{05,06} k8s node & network configuration --- README.md | 181 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) diff --git a/README.md b/README.md index 54844fb..aa339ce 100644 --- a/README.md +++ b/README.md @@ -343,6 +343,14 @@ iface enp5s0.4000 inet static The IP address ends with the same number as the hardware (hetzner02 => .2). +#### vSwitch DRBD + +The vSwitch on VLAN 4000 is for DRBD exclusively + +#### vSwitch k8s + +The vSwitch on VLAN 4002 is for the k8s control plane + ### DRBD DRBD is [configured](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#p-work) like in the following example with hetzner02 as the primary and hetzner03 as the secondary: @@ -724,6 +732,179 @@ stream { Dedicated to https://codeberg.org/forgejo-contrib/forgejo-helm and running from an ephemeral disk +### hetzner{05,06} + +https://hetzner05.forgejo.org & https://hetzner06.forgejo.org run on [EX44](https://www.hetzner.com/dedicated-rootserver/ex44) Hetzner hardware. + + +#### LXC + +```sh +lxc-helpers.sh lxc_install_lxc_inside 10.47.3 fc11 +``` + +#### Disk partitioning + +- First disk + - OS + - non precious data such as the LXC containers with runners. +- Second disk + - a partition configured with DRBD + +#### Root filesystem backups + +- `hetzner05:/etc/cron.daily/backup-hetzner06` + `rsync -aHSv --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.6:/ /srv/backups/hetzner06/` +- `hetzner06:/etc/cron.daily/backup-hetzner05` + `rsync -aHSv --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.5:/ /srv/backups/hetzner05/` + +#### Failover IP addresses + +The failover IP addresses are configured on all hosts. Their routing is managed by the [Hetzner k8s controller](https://github.com/cbeneke/hcloud-fip-controller). + +``` +auto enp5s0 +iface enp5s0 inet static +... + up ip addr add 188.40.16.47/32 dev enp5s0 + +iface enp5s0 inet6 static +... + up ip addr add 2a01:4f8:fff2:48::2/64 dev enp5s0 +``` + +#### k8s node + +```sh +$ cat /etc/rancher/k3s/config.yaml.d/config.yaml +# https://docs.k3s.io/installation/network-options +# the ipv4-ranges are default in k3s +# the fd-range is a private ipv6-range +cluster-cidr: 10.42.0.0/16,fd01::/48 +service-cidr: 10.43.0.0/16,fd02::/112 + +disable: + - servicelb +``` + +The `10.88.1.5` and `fd01::5` IPs are assigned to the interface with VLAN 4002. + +``` +auto enp5s0.4002 +iface enp5s0.4002 inet static + address 10.88.1.5 + netmask 255.255.0.0 + vlan-raw-device enp5s0 + mtu 1400 + up ip addr add fd01::5/48 dev enp5s0.4002 +``` + +```sh +sudo apt-get install curl +curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--write-kubeconfig-mode=644 --node-ip=10.88.1.5,fd01::5' sh - +curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash - +``` + +### k8s networking + +`k3s` is configured to use [metallb](https://metallb.universe.tf) instead of the default load balancer because it does not allow for a public IP different from the `k8s` node IP. + +[metallb](https://metallb.universe.tf). + +``` +helm install metallb metallb/metallb +cat > metallb.yaml < clusterissuer.yml < traefik.yml <