.gitignore | ||
drbd-nginx-lxc.md | ||
LICENSE | ||
lxc.md | ||
octopuce.md | ||
README.md | ||
runner-lxc.md |
The resources used by the infrastructure are in the https://code.forgejo.org/infrastructure/ organization. There is a dedicated chatroom. A mirror of this repository is available at https://git.pub.solar/forgejo/infrastructure-documentation.
Table of content
- K8S cluster files and documentation
- Setting up a new LXC/DRBD Host
- Managing services with a LXC/DRBD/nginx stack
- Installing a Forgejo runner in an LXC container
- Managing the Octopuce host
hetzner{05,06}
https://hetzner05.forgejo.org & https://hetzner06.forgejo.org run on EX44 Hetzner hardware.
They are dedicated to the long running K8S cluster files and documentation. It is also where some legacy machines that are no longer actively used are preserved, to be revived when/if they are needed. They are in the /precious/lxc
directory. The root file system are archived in /precious/hetzner
in case they contain something worth keeping.
-
forgefriends-forum
(hetzner06)Dedicated to https://forum.forgefriends.org
- Docker enabled
-
forgefriends-gitlab
(hetzner06)Dedicated to https://lab.forgefriends.org
- Docker enabled
-
forgefriends-cloud
(hetzner06)Dedicated to https://cloud.forgefriends.org
- Docker enabled
-
gna-forgejo
(hetzner06)Dedicated to https://forgejo.gna.org
- Docker enabled
-
gna-forum
(hetzner06)Dedicated to https://forum.gna.org
- Docker enabled
hetzner{02,03}
https://hetzner02.forgejo.org & https://hetzner03.forgejo.org run on EX44 Hetzner hardware.
LXC
lxc-helpers.sh lxc_install_lxc_inside 10.6.83 fc16
Disk partitioning
- First disk
- OS
- a partition configured with DRBD for precious data mounted on /var/lib/lxc
- Second disk
- non precious data such as the LXC containers with runners.
Root filesystem backups
hetzner03:/etc/cron.daily/backup-hetzner02
rsync -aHS --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.2:/ /srv/backups/hetzner02/
hetzner02:/etc/cron.daily/backup-hetzner03
rsync -aHS --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.3:/ /srv/backups/hetzner03/
Public IP addresses
The public IP addresses attached to the hosts are not failover IPs that can be moved from one host to the next. The DNS entry needs to be updated if the primary hosts changes.
When additional IP addresses are attached to the server, they are added to /etc/network/interfaces
like
ipv4 65.21.67.71 and ipv6 2a01:4f9:3081:51ec::102 below.
auto enp5s0
iface enp5s0 inet static
address 65.21.67.73
netmask 255.255.255.192
gateway 65.21.67.65
# route 65.21.67.64/26 via 65.21.67.65
up route add -net 65.21.67.64 netmask 255.255.255.192 gw 65.21.67.65 dev enp5s0
# BEGIN code.forgejo.org
up ip addr add 65.21.67.71/32 dev enp5s0
up nft -f /home/debian/code.nftables
down ip addr del 65.21.67.71/32 dev enp5s0
# END code.forgejo.org
iface enp5s0 inet6 static
address 2a01:4f9:3081:51ec::2
netmask 64
gateway fe80::1
# BEGIN code.forgejo.org
up ip -6 addr add 2a01:4f9:3081:51ec::102/64 dev enp5s0
down ip -6 addr del 2a01:4f9:3081:51ec::102/64 dev enp5s0
# END code.forgejo.org
For port forwarding to work, the LXC host must not bind them. For instance the ssh server configuration at /etc/ssh/sshd_config
should not bind all IP but only a specific one.
Port 22
AddressFamily inet
ListenAddress 65.21.67.73
#ListenAddress ::
Port forwarding
Forwarding a port to an LXC container can be done with nginx streeam for the public IP of code.forgejo.org (65.21.67.71 & 2a01:4f9:3081:51ec::102) to the private IP (10.6.83.195) of the code
LXC container in /etc/nginx/modules-enabled/ssh.conf
:
stream {
# code.forgejo.org ip's
upstream codessh {
least_conn;
server 10.6.83.195:22;
}
# code.forgejo.org definition
server {
listen 65.21.67.71:22; # the port to listen on this server
listen [2a01:4f9:3081:51ec::102]:22;
proxy_pass codessh; # forward traffic to this upstream group
proxy_timeout 3s;
proxy_connect_timeout 3s;
}
}
302 redirects
- On hetzner02
- try.next.forgejo.org redirects to v(latest stable).next.forgejo.org
- dev.next.forgejo.org redirects to v(latest dev).next.forgejo.org
Containers
-
forgejo-code
on hetzner02Dedicated to https://code.forgejo.org
- Docker enabled
- upgrades checklist:
ssh -t debian@hetzner02.forgejo.org lxc-helpers.sh lxc_container_run forgejo-code -- sudo --user debian bash
emacs /home/debian/run-forgejo.sh # change the `image=` docker stop forgejo
ssh -t debian@hetzner02.forgejo.org sudo /etc/cron.daily/backup-forgejo-code
ssh -t debian@hetzner02.forgejo.org lxc-helpers.sh lxc_container_run forgejo-code -- sudo --user debian bash
docker rm forgejo bash -x /home/debian/run-forgejo.sh docker logs -n 200 -f forgejo
- Rotating 30 days backups happen daily
/etc/cron.daily/forgejo-code-backup.sh
- Add code.forgejo.org to the forgejo.org SPF record
-
forgejo-next
on hetzner02Dedicated to https://next.forgejo.org
- Docker enabled
/etc/cron.hourly/forgejo-upgrade
runs/home/debian/run-forgejo.sh > /home/debian/run-forgejo-$(date +%d).log
- When a new major version is published (8.0 for instance)
run-forgejo.sh
must be updated with it - Reset everything
docker stop forgejo docker rm forgejo sudo rm -fr /srv/forgejo.old sudo mv /srv/forgejo /srv/forgejo.old bash -x /home/debian/run-forgejo.sh
/home/debian/next.nftables
add table ip next; flush table ip next; add chain ip next prerouting { type nat hook prerouting priority 0; policy accept; ip daddr 65.21.67.65 tcp dport { 2020 } dnat to 10.6.83.213; };
- Add to
iface enp5s0 inet static
in/etc/network/interfaces
up nft -f /home/debian/next.nftables
- `/etc/nginx/sites-available/next.forgejo.org` same as `/etc/nginx/sites-available/code.forgejo.org`
-
forgejo-v7
on hetzner02Dedicated to https://v7.next.forgejo.org
- Docker enabled
/etc/cron.hourly/forgejo-upgrade
runs/home/debian/run-forgejo.sh > /home/debian/run-forgejo-$(date +%d).log
- Reset everything
docker stop forgejo docker rm forgejo sudo rm -fr /srv/forgejo.old sudo mv /srv/forgejo /srv/forgejo.old bash -x /home/debian/run-forgejo.sh
/home/debian/v7.nftables
add table ip v7; flush table ip v7; add chain ip v7 prerouting { type nat hook prerouting priority 0; policy accept; ip daddr 65.21.67.65 tcp dport { 2070 } dnat to 10.6.83.179; };
- Add to
iface enp5s0 inet static
in/etc/network/interfaces
up nft -f /home/debian/v7.nftables
- `/etc/nginx/sites-available/v7.forgejo.org` same as `/etc/nginx/sites-available/code.forgejo.org`
-
static-pages
on hetzner02See the static pages documenation for more information.
- Unprivileged
-
runner-forgejo-helm
on hetzner03Dedicated to https://codeberg.org/forgejo-contrib/forgejo-helm and running from an ephemeral disk
Uberspace
The website https://forgejo.org is hosted at https://uberspace.de/. The https://codeberg.org/forgejo/website/ CI has credentials to push HTML pages there.