forgejo/infrastructure-documentation
1
0
Fork 0
mirror of https://code.forgejo.org/infrastructure/documentation synced 2024-11-28 21:31:10 +00:00
Code Issues Activity

k8s: reference the scripts and add an intro

Browse source
This commit is contained in:
Earl Warren 2024-10-20 11:31:03 +02:00
parent 731d2931be
commit e75adbbb82
No known key found for this signature in database
GPG key ID: 0579CB2928A78A00
1 changed files with 41 additions and 38 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

79
k8s.md
View file

@ -1,4 +1,8 @@
#### Imaging # K8S node
Installing a K8S node using [scripts from the k3s-host](k3s-host) directory.
## Imaging
Using installimage from the rescue instance. Using installimage from the rescue instance.
@ -25,7 +29,7 @@ Debian user.
- `mkdir -p /home/debian/.ssh ; cp -a .ssh/authorized_keys /home/debian/.ssh ; chown -R debian /home/debian/.ssh` - `mkdir -p /home/debian/.ssh ; cp -a .ssh/authorized_keys /home/debian/.ssh ; chown -R debian /home/debian/.ssh`
- in `/etc/sudoers` edit `%sudo ALL=(ALL:ALL) NOPASSWD:ALL` - in `/etc/sudoers` edit `%sudo ALL=(ALL:ALL) NOPASSWD:ALL`
#### Install helpers ## Install helpers
Each node is identifed by the last digit of the hostname. Each node is identifed by the last digit of the hostname.
@ -52,11 +56,11 @@ Variables that must be set depending on the role of the node.
The other variables depend on the setup. The other variables depend on the setup.
#### Firewall ## Firewall
`./setup.sh setup_ufw` `./setup.sh setup_ufw`
#### DRBD ## DRBD
DRBD is [configured](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#p-work) with: DRBD is [configured](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#p-work) with:
@ -71,7 +75,7 @@ sudo drbdadm new-current-uuid --clear-bitmap r1/0
sudo mount /precious sudo mount /precious
``` ```
#### NFS ## NFS
`./setup.sh setup_nfs` `./setup.sh setup_nfs`
@ -81,7 +85,7 @@ On the node that has the DRBD volume `/precious` mounted, set the IP of the NFS
sudo ip addr add 10.53.101.100/24 dev enp5s0.4001 sudo ip addr add 10.53.101.100/24 dev enp5s0.4001
``` ```
#### K8S ## K8S
For the first node `./setup.sh setup_k8s`. For nodes joining the cluster `./setup.sh setup_k8s 6` where `hetzner06` is an existing node. For the first node `./setup.sh setup_k8s`. For nodes joining the cluster `./setup.sh setup_k8s 6` where `hetzner06` is an existing node.
@ -94,7 +98,7 @@ For the first node `./setup.sh setup_k8s`. For nodes joining the cluster `./setu
- NFS storage class - NFS storage class
`./setup.sh setup_k8s_nfs` `./setup.sh setup_k8s_nfs`
#### Forgejo ## Forgejo
[forgejo](https://code.forgejo.org/forgejo-helm/forgejo-helm) configuration in [ingress](https://code.forgejo.org/forgejo-helm/forgejo-helm#ingress) for the reverse proxy (`traefik`) to route the domain and for the ACME issuer (`cert-manager`) to obtain a certificate. And in [service](https://code.forgejo.org/forgejo-helm/forgejo-helm#service) for the `ssh` port to be bound to the desired IPs of the load balancer (`metallb`). [forgejo](https://code.forgejo.org/forgejo-helm/forgejo-helm) configuration in [ingress](https://code.forgejo.org/forgejo-helm/forgejo-helm#ingress) for the reverse proxy (`traefik`) to route the domain and for the ACME issuer (`cert-manager`) to obtain a certificate. And in [service](https://code.forgejo.org/forgejo-helm/forgejo-helm#service) for the `ssh` port to be bound to the desired IPs of the load balancer (`metallb`).
@ -102,38 +106,38 @@ For the first node `./setup.sh setup_k8s`. For nodes joining the cluster `./setu
ingress: ingress:
enabled: true enabled: true
annotations: annotations:
# https://cert-manager.io/docs/usage/ingress/#supported-annotations # https://cert-manager.io/docs/usage/ingress/#supported-annotations
# https://github.com/cert-manager/cert-manager/issues/2239 # https://github.com/cert-manager/cert-manager/issues/2239
cert-manager.io/cluster-issuer: letsencrypt-http cert-manager.io/cluster-issuer: letsencrypt-http
cert-manager.io/private-key-algorithm: ECDSA cert-manager.io/private-key-algorithm: ECDSA
cert-manager.io/private-key-size: 384 cert-manager.io/private-key-size: 384
kubernetes.io/ingress.class: traefik kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: websecure traefik.ingress.kubernetes.io/router.entrypoints: websecure
tls: tls:
- hosts: - hosts:
- t1.forgejo.org - t1.forgejo.org
secretName: tls-forgejo-t1-ingress-http secretName: tls-forgejo-t1-ingress-http
hosts: hosts:
- host: t1.forgejo.org - host: t1.forgejo.org
paths: paths:
- path: / - path: /
pathType: Prefix pathType: Prefix
service: service:
http: http:
type: ClusterIP type: ClusterIP
ipFamilyPolicy: PreferDualStack ipFamilyPolicy: PreferDualStack
port: 3000 port: 3000
ssh: ssh:
type: LoadBalancer type: LoadBalancer
annotations: annotations:
metallb.universe.tf/loadBalancerIPs: 188.40.16.47,2a01:4f8:fff2:48::2 metallb.universe.tf/loadBalancerIPs: 188.40.16.47,2a01:4f8:fff2:48::2
metallb.universe.tf/allow-shared-ip: "key-to-share-failover" metallb.universe.tf/allow-shared-ip: "key-to-share-failover"
ipFamilyPolicy: PreferDualStack ipFamilyPolicy: PreferDualStack
port: 2222 port: 2222
``` ```
### K8S NFS storage creation # K8S NFS storage creation
Define the 20GB `forgejo-data` pvc owned by user id 1000. Define the 20GB `forgejo-data` pvc owned by user id 1000.
@ -150,27 +154,26 @@ persistence:
claimName: forgejo-data claimName: forgejo-data
``` ```
## Disaster recovery and maintenance Disaster recovery and maintenance
### When a machine or disk is scheduled for replacement. # When a machine or disk is scheduled for replacement.
* `kubectl drain hetzner05` # evacuate all the pods out of the node to be shutdown * `kubectl drain hetzner05` # evacuate all the pods out of the node to be shutdown
* `kubectl taint nodes hetzner05 key1=value1:NoSchedule` # prevent any pod from being created there (metallb speaker won't be drained, for instance) * `kubectl taint nodes hetzner05 key1=value1:NoSchedule` # prevent any pod from being created there (metallb speaker won't be drained, for instance)
* `kubectl delete node hetzner05` # let the cluster know it no longer exists so a new one by the same name can replace it * `kubectl delete node hetzner05` # let the cluster know it no longer exists so a new one by the same name can replace it
### Routing the failover IP # Routing the failover IP
When the machine to which the failover IP (failover.forgejo.org) is routed is unavailable or to be shutdown, to the [Hetzner server panel](https://robot.hetzner.com/server), to the IPs tab and change the route of the failover IP to another node. All nodes are configured with the failover IP, there is nothing else to do. When the machine to which the failover IP (failover.forgejo.org) is routed is unavailable or to be shutdown, to the [Hetzner server panel](https://robot.hetzner.com/server), to the IPs tab and change the route of the failover IP to another node. All nodes are configured with the failover IP, there is nothing else to do.
### Manual boot operations # Manual boot operations
#### On the machine that runs the NFS server ## On the machine that runs the NFS server
* `sudo drbdadm primary r1` # Switch the DRBD to primary * `sudo drbdadm primary r1` # Switch the DRBD to primary
* `sudo mount /precious` # DRBD volume shared via NFS * `sudo mount /precious` # DRBD volume shared via NFS
* `sudo ip addr add 10.53.101.100/24 dev enp5s0.4001` # add NFS server IP * `sudo ip addr add 10.53.101.100/24 dev enp5s0.4001` # add NFS server IP
#### On the other machines ## On the other machines
* `sudo ip addr del 10.53.101.100/24 dev enp5s0.4001` # remove NFS server IP * `sudo ip addr del 10.53.101.100/24 dev enp5s0.4001` # remove NFS server IP