infrastructure-documentation/README.md at 8f0057787a0f4fc06f8696b96731b43447143eb1

mirror of https://code.forgejo.org/infrastructure/documentation synced 2024-11-21 19:11:11 +00:00

maintenance and disaster recovery for k8s from the TOC

2024-10-20 11:47:52 +02:00

10 KiB

Raw Blame History

The resources used by the infrastructure are in the https://code.forgejo.org/infrastructure/ organization. There is a dedicated chatroom. A mirror of this repository is available at https://git.pub.solar/forgejo/infrastructure-documentation.

Table of content

Setting up a new K8S/DRBD/NFS k8s node
Maintenance and disaster recovery of a K8S/DRBD/NFS k8s node
Setting up a new LXC/DRBD Host
Managing services with a LXC/DRBD/nginx stack
Installing a Forgejo instance in the K8S cluster
Installing a Forgejo runner in an LXC container
Managing the Octopuce host

hetzner{01,04}

https://hetzner{01,04}.forgejo.org run on EX101 Hetzner hardware.

LXC

lxc-helpers.sh lxc_install_lxc_inside 10.41.13 fc29

Disk partitioning

First disk
- OS
- a partition mounted on /srv where non precious data goes such as the LXC containers with runners.
Second disk
- configured with DRBD for precious data.

Root filesystem backups

hetzner01:/etc/cron.daily/backup-hetzner04 rsync -aHS --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /precious --exclude /srv --exclude /var/lib/lxc 10.53.100.4:/ /srv/backups/hetzner04/ >& /var/log/$(basename $0).log
hetzner04:/etc/cron.daily/backup-hetzner01 rsync -aHS --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /precious --exclude /srv --exclude /var/lib/lxc 10.53.100.1:/ /srv/backups/hetzner01/ >& /var/log/$(basename $0).log

LXC containers

runner-lxc-helpers (hetzner01)

Dedicated to Forgejo runners for the https://code.forgejo.org/forgejo/lxc-helpers project.
- K8S enabled
- code.forgejo.org/forgejo/lxc-helpers/config*.yml
forgejo-runners (hetzner01)

Dedicated to Forgejo runners for the https://codeberg.org/forgejo organization.
- Docker enabled
- codeberg.org/forgejo/config*.yml
runner01-lxc (hetzner01)

Dedicated to Forgejo runners for https://code.forgejo.org.
- Docker and LXC enabled 10.194.201 fc35
- code.forgejo.org/forgejo/config*.yml
- code.forgejo.org/actions/config*.yml
- code.forgejo.org/forgejo-integration/config*.yml
- code.forgejo.org/forgejo-contrib/config*.yml
- code.forgejo.org/f3/config*.yml
- code.forgejo.org/forgefriends/config*.yml
forgejo-v9 (hetzner04) same as forgejo-v8
forgejo-v8 (hetzner04)

Dedicated to https://v8.next.forgejo.org, see https://code.forgejo.org/infrastructure/k8s
- K8S enabled
- K8S wakeup-on-logs script /etc/wakeup-on-logs/forgejo-v8
- Values file
- nginx forwarding of SSH streams in /etc/nginx/modules-enabled/next.forgejo.org.conf
```
stream {

  # v8 ip's
  upstream v8 {
    least_conn;
    server 10.41.13.27:2222;
  }

  # v8 definition
  server {
    listen 2080; # the port to listen on this server
    listen [::]:2080;
    proxy_pass v8; # forward traffic to this upstream group
  }
}
```
forgefriends-forum (hetzner04)

Dedicated to https://forum.forgefriends.org
- Docker enabled
forgefriends-gitlab (hetzner04)

Dedicated to https://lab.forgefriends.org
- Docker enabled
forgefriends-cloud (hetzner04)

Dedicated to https://cloud.forgefriends.org
- Docker enabled
gna-forgejo (hetzner04)

Dedicated to https://forgejo.gna.org
- Docker enabled
gna-forum (hetzner04)

Dedicated to https://forum.gna.org
- Docker enabled

hetzner{02,03}

https://hetzner02.forgejo.org & https://hetzner03.forgejo.org run on EX44 Hetzner hardware.

LXC

lxc-helpers.sh lxc_install_lxc_inside 10.6.83 fc16

Disk partitioning

First disk
- OS
- a partition configured with DRBD for precious data mounted on /var/lib/lxc
Second disk
- non precious data such as the LXC containers with runners.

Root filesystem backups

hetzner03:/etc/cron.daily/backup-hetzner02 rsync -aHS --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.2:/ /srv/backups/hetzner02/
hetzner02:/etc/cron.daily/backup-hetzner03 rsync -aHS --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.3:/ /srv/backups/hetzner03/

Public IP addresses

The public IP addresses attached to the hosts are not failover IPs that can be moved from one host to the next. The DNS entry needs to be updated if the primary hosts changes.

When additional IP addresses are attached to the server, they are added to /etc/network/interfaces like ipv4 65.21.67.71 and ipv6 2a01:4f9:3081:51ec::102 below.

auto enp5s0
iface enp5s0 inet static
  address 65.21.67.73
  netmask 255.255.255.192
  gateway 65.21.67.65
  # route 65.21.67.64/26 via 65.21.67.65
  up route add -net 65.21.67.64 netmask 255.255.255.192 gw 65.21.67.65 dev enp5s0
  # BEGIN code.forgejo.org
  up ip addr add 65.21.67.71/32 dev enp5s0
  up nft -f /home/debian/code.nftables
  down ip addr del 65.21.67.71/32 dev enp5s0
  # END code.forgejo.org

iface enp5s0 inet6 static
  address 2a01:4f9:3081:51ec::2
  netmask 64
  gateway fe80::1
  # BEGIN code.forgejo.org
  up ip -6 addr add 2a01:4f9:3081:51ec::102/64 dev enp5s0
  down ip -6 addr del 2a01:4f9:3081:51ec::102/64 dev enp5s0
  # END code.forgejo.org

For port forwarding to work, the LXC host must not bind them. For instance the ssh server configuration at /etc/ssh/sshd_config should not bind all IP but only a specific one.

Port 22
AddressFamily inet
ListenAddress 65.21.67.73
#ListenAddress ::

Port forwarding

Forwarding a port to an LXC container can be done with nginx streeam for the public IP of code.forgejo.org (65.21.67.71 & 2a01:4f9:3081:51ec::102) to the private IP (10.6.83.195) of the code LXC container in /etc/nginx/modules-enabled/ssh.conf:

stream {

  # code.forgejo.org ip's
  upstream codessh {
    least_conn;
    server 10.6.83.195:22;
  }

  # code.forgejo.org definition
  server {
    listen 65.21.67.71:22; # the port to listen on this server
    listen [2a01:4f9:3081:51ec::102]:22;
    proxy_pass codessh; # forward traffic to this upstream group
    proxy_timeout 3s;
    proxy_connect_timeout 3s;
  }
}

302 redirects

On hetzner02
- try.next.forgejo.org redirects to v(latest stable).next.forgejo.org
- dev.next.forgejo.org redirects to v(latest dev).next.forgejo.org

Containers

forgejo-code on hetzner02

Dedicated to https://code.forgejo.org
- Docker enabled
- upgrades checklist:
  - ssh -t debian@hetzner02.forgejo.org lxc-helpers.sh lxc_container_run forgejo-code -- sudo --user debian bash
```
emacs /home/debian/run-forgejo.sh # change the `image=`
docker stop forgejo
```
  - ssh -t debian@hetzner02.forgejo.org sudo /etc/cron.daily/backup-forgejo-code
  - ssh -t debian@hetzner02.forgejo.org lxc-helpers.sh lxc_container_run forgejo-code -- sudo --user debian bash
```
docker rm forgejo
bash -x /home/debian/run-forgejo.sh
docker logs -n 200 -f forgejo
```
- Rotating 30 days backups happen daily /etc/cron.daily/forgejo-code-backup.sh
- Add code.forgejo.org to the forgejo.org SPF record

forgejo-next on hetzner02

Dedicated to https://next.forgejo.org

Docker enabled
/etc/cron.hourly/forgejo-upgrade runs /home/debian/run-forgejo.sh > /home/debian/run-forgejo-$(date +%d).log
When a new major version is published (8.0 for instance) run-forgejo.sh must be updated with it

Reset everything

docker stop forgejo
docker rm forgejo
sudo rm -fr /srv/forgejo.old
sudo mv /srv/forgejo /srv/forgejo.old
bash -x /home/debian/run-forgejo.sh

/home/debian/next.nftables

add table ip next;
flush table ip next;
add chain ip next prerouting {
  type nat hook prerouting priority 0;
  policy accept;
  ip daddr 65.21.67.65 tcp dport { 2020 } dnat to 10.6.83.213;
};

Add to iface enp5s0 inet static in /etc/network/interfaces
```
up nft -f /home/debian/next.nftables
```

- `/etc/nginx/sites-available/next.forgejo.org` same as `/etc/nginx/sites-available/code.forgejo.org`

forgejo-v7 on hetzner02

Dedicated to https://v7.next.forgejo.org

Docker enabled
/etc/cron.hourly/forgejo-upgrade runs /home/debian/run-forgejo.sh > /home/debian/run-forgejo-$(date +%d).log

Reset everything

docker stop forgejo
docker rm forgejo
sudo rm -fr /srv/forgejo.old
sudo mv /srv/forgejo /srv/forgejo.old
bash -x /home/debian/run-forgejo.sh

/home/debian/v7.nftables

add table ip v7;
flush table ip v7;
add chain ip v7 prerouting {
  type nat hook prerouting priority 0;
  policy accept;
  ip daddr 65.21.67.65 tcp dport { 2070 } dnat to 10.6.83.179;
};

Add to iface enp5s0 inet static in /etc/network/interfaces
```
up nft -f /home/debian/v7.nftables
```

- `/etc/nginx/sites-available/v7.forgejo.org` same as `/etc/nginx/sites-available/code.forgejo.org`

static-pages on hetzner02

See the static pages documenation for more information.
- Unprivileged
runner-forgejo-helm on hetzner03

Dedicated to https://codeberg.org/forgejo-contrib/forgejo-helm and running from an ephemeral disk

hetzner{05,06}

https://hetzner05.forgejo.org & https://hetzner06.forgejo.org run on EX44 Hetzner hardware.

Nodes of a k8s cluster.

Uberspace

The website https://forgejo.org is hosted at https://uberspace.de/. The https://codeberg.org/forgejo/website/ CI has credentials to push HTML pages there.

10 KiB Raw Blame History

Table of content

hetzner{01,04}

LXC

Disk partitioning

Root filesystem backups

LXC containers

hetzner{02,03}

LXC

Disk partitioning

Root filesystem backups

Public IP addresses

Port forwarding

302 redirects

Containers

hetzner{05,06}

Uberspace

10 KiB

Raw Blame History