forgejo/infrastructure-documentation
1
0
Fork 0
mirror of https://code.forgejo.org/infrastructure/documentation synced 2024-11-22 03:21:10 +00:00
Code Issues Activity

maintenance and disaster recovery for k8s from the TOC

Browse source
This commit is contained in:
Earl Warren 2024-10-20 11:47:52 +02:00
parent 2f652df670
commit 85b658a645
No known key found for this signature in database
GPG key ID: 0579CB2928A78A00
3 changed files with 24 additions and 24 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
README.md
View file

@ -4,6 +4,7 @@ There is a [dedicated chatroom](https://matrix.to/#/#forgejo-ci:matrix.org). A m
## Table of content
- Setting up a new [K8S/DRBD/NFS k8s node](k8s.md)
- Maintenance and disaster recovery of a [K8S/DRBD/NFS k8s node](k8s-maintenance.md)
- Setting up a new [LXC/DRBD Host](lxc.md)
- Managing services with a [LXC/DRBD/nginx stack](drbd-nginx-lxc.md)
- Installing a [Forgejo instance in the K8S cluster](k8s-forgejo.md)

23
k8s-maintenance.md Normal file
View file

@ -0,0 +1,23 @@
# Disaster recovery and maintenance
## When a machine or disk is scheduled for replacement.
* `kubectl drain hetzner05` # evacuate all the pods out of the node to be shutdown
* `kubectl taint nodes hetzner05 key1=value1:NoSchedule` # prevent any pod from being created there (metallb speaker won't be drained, for instance)
* `kubectl delete node hetzner05` # let the cluster know it no longer exists so a new one by the same name can replace it
## Routing the failover IP
When the machine to which the failover IP (failover.forgejo.org) is routed is unavailable or to be shutdown, to the [Hetzner server panel](https://robot.hetzner.com/server), to the IPs tab and change the route of the failover IP to another node. All nodes are configured with the failover IP, there is nothing else to do.
## Manual boot operations
### On the machine that runs the NFS server
* `sudo drbdadm primary r1` # Switch the DRBD to primary
* `sudo mount /precious` # DRBD volume shared via NFS
* `sudo ip addr add 10.53.101.100/24 dev enp5s0.4001` # add NFS server IP
### On the other machines
* `sudo ip addr del 10.53.101.100/24 dev enp5s0.4001` # remove NFS server IP

24
k8s.md
View file

@ -105,27 +105,3 @@ Define the 20GB `forgejo-data` pvc owned by user id 1000.
```sh
./setup.sh setup_k8s_pvc forgejo-data 20Gi 1000
```
# Disaster recovery and maintenance
## When a machine or disk is scheduled for replacement.
* `kubectl drain hetzner05` # evacuate all the pods out of the node to be shutdown
* `kubectl taint nodes hetzner05 key1=value1:NoSchedule` # prevent any pod from being created there (metallb speaker won't be drained, for instance)
* `kubectl delete node hetzner05` # let the cluster know it no longer exists so a new one by the same name can replace it
## Routing the failover IP
When the machine to which the failover IP (failover.forgejo.org) is routed is unavailable or to be shutdown, to the [Hetzner server panel](https://robot.hetzner.com/server), to the IPs tab and change the route of the failover IP to another node. All nodes are configured with the failover IP, there is nothing else to do.
## Manual boot operations
### On the machine that runs the NFS server
* `sudo drbdadm primary r1` # Switch the DRBD to primary
* `sudo mount /precious` # DRBD volume shared via NFS
* `sudo ip addr add 10.53.101.100/24 dev enp5s0.4001` # add NFS server IP
### On the other machines
* `sudo ip addr del 10.53.101.100/24 dev enp5s0.4001` # remove NFS server IP