diff --git a/k8s.md b/k8s.md index 7f69d49..25caf17 100644 --- a/k8s.md +++ b/k8s.md @@ -1,4 +1,8 @@ -#### Imaging +# K8S node + +Installing a K8S node using [scripts from the k3s-host](k3s-host) directory. + +## Imaging Using installimage from the rescue instance. @@ -25,7 +29,7 @@ Debian user. - `mkdir -p /home/debian/.ssh ; cp -a .ssh/authorized_keys /home/debian/.ssh ; chown -R debian /home/debian/.ssh` - in `/etc/sudoers` edit `%sudo ALL=(ALL:ALL) NOPASSWD:ALL` -#### Install helpers +## Install helpers Each node is identifed by the last digit of the hostname. @@ -52,11 +56,11 @@ Variables that must be set depending on the role of the node. The other variables depend on the setup. -#### Firewall +## Firewall `./setup.sh setup_ufw` -#### DRBD +## DRBD DRBD is [configured](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#p-work) with: @@ -71,7 +75,7 @@ sudo drbdadm new-current-uuid --clear-bitmap r1/0 sudo mount /precious ``` -#### NFS +## NFS `./setup.sh setup_nfs` @@ -81,7 +85,7 @@ On the node that has the DRBD volume `/precious` mounted, set the IP of the NFS sudo ip addr add 10.53.101.100/24 dev enp5s0.4001 ``` -#### K8S +## K8S For the first node `./setup.sh setup_k8s`. For nodes joining the cluster `./setup.sh setup_k8s 6` where `hetzner06` is an existing node. @@ -94,7 +98,7 @@ For the first node `./setup.sh setup_k8s`. For nodes joining the cluster `./setu - NFS storage class `./setup.sh setup_k8s_nfs` -#### Forgejo +## Forgejo [forgejo](https://code.forgejo.org/forgejo-helm/forgejo-helm) configuration in [ingress](https://code.forgejo.org/forgejo-helm/forgejo-helm#ingress) for the reverse proxy (`traefik`) to route the domain and for the ACME issuer (`cert-manager`) to obtain a certificate. And in [service](https://code.forgejo.org/forgejo-helm/forgejo-helm#service) for the `ssh` port to be bound to the desired IPs of the load balancer (`metallb`). @@ -102,38 +106,38 @@ For the first node `./setup.sh setup_k8s`. For nodes joining the cluster `./setu ingress: enabled: true annotations: - # https://cert-manager.io/docs/usage/ingress/#supported-annotations - # https://github.com/cert-manager/cert-manager/issues/2239 - cert-manager.io/cluster-issuer: letsencrypt-http - cert-manager.io/private-key-algorithm: ECDSA - cert-manager.io/private-key-size: 384 - kubernetes.io/ingress.class: traefik - traefik.ingress.kubernetes.io/router.entrypoints: websecure + # https://cert-manager.io/docs/usage/ingress/#supported-annotations + # https://github.com/cert-manager/cert-manager/issues/2239 + cert-manager.io/cluster-issuer: letsencrypt-http + cert-manager.io/private-key-algorithm: ECDSA + cert-manager.io/private-key-size: 384 + kubernetes.io/ingress.class: traefik + traefik.ingress.kubernetes.io/router.entrypoints: websecure tls: - - hosts: - - t1.forgejo.org - secretName: tls-forgejo-t1-ingress-http + - hosts: + - t1.forgejo.org + secretName: tls-forgejo-t1-ingress-http hosts: - - host: t1.forgejo.org - paths: - - path: / - pathType: Prefix + - host: t1.forgejo.org + paths: + - path: / + pathType: Prefix service: http: - type: ClusterIP - ipFamilyPolicy: PreferDualStack - port: 3000 + type: ClusterIP + ipFamilyPolicy: PreferDualStack + port: 3000 ssh: - type: LoadBalancer - annotations: - metallb.universe.tf/loadBalancerIPs: 188.40.16.47,2a01:4f8:fff2:48::2 - metallb.universe.tf/allow-shared-ip: "key-to-share-failover" - ipFamilyPolicy: PreferDualStack - port: 2222 + type: LoadBalancer + annotations: + metallb.universe.tf/loadBalancerIPs: 188.40.16.47,2a01:4f8:fff2:48::2 + metallb.universe.tf/allow-shared-ip: "key-to-share-failover" + ipFamilyPolicy: PreferDualStack + port: 2222 ``` -### K8S NFS storage creation +# K8S NFS storage creation Define the 20GB `forgejo-data` pvc owned by user id 1000. @@ -150,27 +154,26 @@ persistence: claimName: forgejo-data ``` -## Disaster recovery and maintenance + Disaster recovery and maintenance -### When a machine or disk is scheduled for replacement. +# When a machine or disk is scheduled for replacement. * `kubectl drain hetzner05` # evacuate all the pods out of the node to be shutdown * `kubectl taint nodes hetzner05 key1=value1:NoSchedule` # prevent any pod from being created there (metallb speaker won't be drained, for instance) * `kubectl delete node hetzner05` # let the cluster know it no longer exists so a new one by the same name can replace it -### Routing the failover IP +# Routing the failover IP When the machine to which the failover IP (failover.forgejo.org) is routed is unavailable or to be shutdown, to the [Hetzner server panel](https://robot.hetzner.com/server), to the IPs tab and change the route of the failover IP to another node. All nodes are configured with the failover IP, there is nothing else to do. -### Manual boot operations +# Manual boot operations -#### On the machine that runs the NFS server +## On the machine that runs the NFS server * `sudo drbdadm primary r1` # Switch the DRBD to primary * `sudo mount /precious` # DRBD volume shared via NFS * `sudo ip addr add 10.53.101.100/24 dev enp5s0.4001` # add NFS server IP -#### On the other machines +## On the other machines * `sudo ip addr del 10.53.101.100/24 dev enp5s0.4001` # remove NFS server IP -