hetzner{05,06} k8s node & network configuration

README.md

@@ -343,6 +343,14 @@ iface enp5s0.4000 inet static
 
 The IP address ends with the same number as the hardware (hetzner02 => .2).
 
+#### vSwitch DRBD
+
+The vSwitch on VLAN 4000 is for DRBD exclusively
+
+#### vSwitch k8s
+
+The vSwitch on VLAN 4002 is for the k8s control plane
+
 ### DRBD
 
 DRBD is [configured](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#p-work) like in the following example with hetzner02 as the primary and hetzner03 as the secondary:
@@ -724,6 +732,179 @@ stream {
 
   Dedicated to https://codeberg.org/forgejo-contrib/forgejo-helm and running from an ephemeral disk
 
+### hetzner{05,06}
+
+https://hetzner05.forgejo.org & https://hetzner06.forgejo.org run on [EX44](https://www.hetzner.com/dedicated-rootserver/ex44) Hetzner hardware.
+
+
+#### LXC
+
+```sh
+lxc-helpers.sh lxc_install_lxc_inside 10.47.3 fc11
+```
+
+#### Disk partitioning
+
+- First disk
+  - OS
+  - non precious data such as the LXC containers with runners.
+- Second disk
+  - a partition configured with DRBD
+
+#### Root filesystem backups
+
+- `hetzner05:/etc/cron.daily/backup-hetzner06`
+  `rsync -aHSv --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.6:/ /srv/backups/hetzner06/`
+- `hetzner06:/etc/cron.daily/backup-hetzner05`
+  `rsync -aHSv --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.5:/ /srv/backups/hetzner05/`
+
+#### Failover IP addresses
+
+The failover IP addresses are configured on all hosts. Their routing is  managed by the [Hetzner k8s controller](https://github.com/cbeneke/hcloud-fip-controller).
+
+```
+auto enp5s0
+iface enp5s0 inet static
+...
+  up ip addr add 188.40.16.47/32 dev enp5s0
+
+iface enp5s0 inet6 static
+...
+  up ip addr add 2a01:4f8:fff2:48::2/64 dev enp5s0
+```
+
+#### k8s node
+
+```sh
+$ cat /etc/rancher/k3s/config.yaml.d/config.yaml
+# https://docs.k3s.io/installation/network-options
+# the ipv4-ranges are default in k3s
+# the fd-range is a private ipv6-range
+cluster-cidr: 10.42.0.0/16,fd01::/48
+service-cidr: 10.43.0.0/16,fd02::/112
+
+disable:
+ - servicelb
+```
+
+The `10.88.1.5` and `fd01::5` IPs are assigned to the interface with VLAN 4002.
+
+```
+auto enp5s0.4002
+iface enp5s0.4002 inet static
+  address 10.88.1.5
+  netmask 255.255.0.0
+  vlan-raw-device enp5s0
+  mtu 1400
+  up ip addr add fd01::5/48 dev enp5s0.4002
+```
+
+```sh
+sudo apt-get install curl
+curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--write-kubeconfig-mode=644 --node-ip=10.88.1.5,fd01::5' sh -
+curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -
+```
+
+### k8s networking
+
+`k3s` is configured to use [metallb](https://metallb.universe.tf) instead of the default load balancer because it does not allow for a public IP different from the `k8s` node IP.
+
+[metallb](https://metallb.universe.tf).
+
+```
+helm install metallb metallb/metallb
+cat > metallb.yaml <<EOF
+apiVersion: metallb.io/v1beta1
+kind: IPAddressPool
+metadata:
+  name: first-pool
+spec:
+  addresses:
+   - 188.40.16.47/32
+   - 2a01:4f8:fff2:48::0/64
+EOF
+kubectl apply --server-side=true -f metallb.yml
+```
+
+[cert-manager](https://cert-manager.io/).
+
+```
+helm install mycertmanager --set installCRDs=true oci://registry-1.docker.io/bitnamicharts/cert-manager
+cat > clusterissuer.yml <<EOF
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-http
+spec:
+  acme:
+    email: contact@forgejo.org
+    server: https://acme-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-http
+    solvers:
+    - http01:
+        ingress:
+          class: traefik
+EOF
+kubectl apply --server-side=true -f clusterissuer.yml
+```
+
+[traefik](https://traefik.io/) requests with [annotations](https://github.com/traefik/traefik-helm-chart/blob/7a13fc8a61a6ad30fcec32eec497dab9d8aea686/traefik/values.yaml#L736) specific IPs from `metalldb`.
+
+```
+cat > traefik.yml <<EOF
+apiVersion: helm.cattle.io/v1
+kind: HelmChartConfig
+metadata:
+  name: traefik
+  namespace: kube-system
+spec:
+  valuesContent: |-
+    service:
+      annotations:
+        metallb.universe.tf/allow-shared-ip: "key-to-share-188-40-16-47"
+        metallb.universe.tf/loadBalancerIPs: 188.40.16.47,2a01:4f8:fff2:48::2
+EOF
+kubectl apply --server-side=true -f traefik.yml
+```
+
+[forgejo](https://code.forgejo.org/forgejo-helm/forgejo-helm) configuration in [ingress](https://code.forgejo.org/forgejo-helm/forgejo-helm#ingress) for the reverse proxy (`traefik`) to route the domain and for the ACME issuer (`cert-manager`) to obtain a certificate. And in [service](https://code.forgejo.org/forgejo-helm/forgejo-helm#service) for the `ssh` port to be bound to the desired IPs of the load balancer (`metallb`).
+
+```
+ingress:
+  enabled: true
+  annotations:
+    # https://cert-manager.io/docs/usage/ingress/#supported-annotations
+    # https://github.com/cert-manager/cert-manager/issues/2239
+    cert-manager.io/cluster-issuer: letsencrypt-http
+    cert-manager.io/private-key-algorithm: ECDSA
+    cert-manager.io/private-key-size: 384
+    kubernetes.io/ingress.class: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+  tls:
+    - hosts:
+        - t1.forgejo.org
+      secretName: tls-forgejo-t1-ingress-http
+  hosts:
+    - host: t1.forgejo.org
+      paths:
+        - path: /
+          pathType: Prefix
+
+service:
+  http:
+    type: ClusterIP
+    ipFamilyPolicy: PreferDualStack
+    port: 3000
+  ssh:
+    type: LoadBalancer
+    annotations:
+      metallb.universe.tf/loadBalancerIPs: 188.40.16.47,2a01:4f8:fff2:48::2
+      metallb.universe.tf/allow-shared-ip: "key-to-share-188-40-16-47"
+    ipFamilyPolicy: PreferDualStack
+    port: 2222
+```
+
 ## Uberspace
 
 The website https://forgejo.org is hosted at