4.6 KiB
LXC Hosts
All LXC hosts are setup with lxc-helpers.
name=forgejo-host
lxc-helpers.sh lxc_container_run $name -- sudo --user debian bash
See https://github.com/mikesart/inotify-info. Running multiple LXC containers will quickly use the default limit (128 on bookworm).
echo fs.inotify.max_user_instances=8192 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
Unprivileged
name=forgejo-host
lxc-helpers.sh lxc_container_create --config "unprivileged" $name
echo "lxc.start.auto = 1" | sudo tee -a /var/lib/lxc/$name/config
lxc-helpers.sh lxc_container_start $name
lxc-helpers.sh lxc_container_user_install $name $(id -u) $USER
Docker enabled
name=forgejo-host
lxc-helpers.sh lxc_container_create --config "docker" $name
echo "lxc.start.auto = 1" | sudo tee -a /var/lib/lxc/$name/config
lxc-helpers.sh lxc_container_start $name
lxc-helpers.sh lxc_install_docker $name
lxc-helpers.sh lxc_container_user_install $name $(id -u) $USER
K8S enabled
name=forgejo-host
lxc-helpers.sh lxc_container_create --config "k8s" $name
echo "lxc.start.auto = 1" | sudo tee -a /var/lib/lxc/$name/config
lxc-helpers.sh lxc_container_start $name
lxc-helpers.sh lxc_container_user_install $name $(id -u) $USER
Docker and LXC enabled
name=forgejo-host
ipv4=10.85.12
ipv6=fc33
lxc-helpers.sh lxc_container_create --config "docker lxc" $name
echo "lxc.start.auto = 1" | sudo tee -a /var/lib/lxc/$name/config
lxc-helpers.sh lxc_container_start $name
lxc-helpers.sh lxc_install_docker $name
lxc-helpers.sh lxc_install_lxc $name $ipv4 $ipv6
lxc-helpers.sh lxc_container_user_install $name $(id -u) $USER
Hetzner
All hardware machines are running Debian GNU/linux bookworm. They are LXC hosts setup with lxc-helpers.
NOTE: only use EX101 with a ASRockRack W680D4U-1L motherboard.
vSwitch
A vSwitch is assigned via the Robot console on all servers for backend communications and configured in /etc/network/interfaces for each of them with something like:
auto enp5s0.4000
iface enp5s0.4000 inet static
address 10.53.100.2
netmask 255.255.255.0
vlan-raw-device enp5s0
mtu 1400
The IP address ends with the same number as the hardware (hetzner02 => .2).
vSwitch DRBD
The vSwitch on VLAN 4000 is for DRBD exclusively
DRBD
DRBD is configured like in the following example with hetzner02 as the primary and hetzner03 as the secondary:
$ apt-get install drbd-utils
$ cat /etc/drbd.d/r0.res
resource r0 {
net {
# A : write completion is determined when data is written to the local disk and the local TCP transmission buffer
# B : write completion is determined when data is written to the local disk and remote buffer cache
# C : write completion is determined when data is written to both the local disk and the remote disk
protocol C;
cram-hmac-alg sha1;
# any secret key for authentication among nodes
shared-secret "***";
}
disk {
resync-rate 1000M;
}
on hetzner02 {
address 10.53.100.2:7788;
volume 0 {
# device name
device /dev/drbd0;
# specify disk to be used for device above
disk /dev/nvme0n1p5;
# where to create metadata
# specify the block device name when using a different disk
meta-disk internal;
}
}
on hetzner03 {
address 10.53.100.3:7788;
volume 0 {
device /dev/drbd0;
disk /dev/nvme1n1p5;
meta-disk internal;
}
}
}
$ sudo drbdadm create-md r0
$ sudo systemctl enable drbd
$ sudo systemctl start drbd
On hetzner02 (the primary), pretend all is in sync to save the initial bitmap sync since there is actually no data at all.
sudo drbdadm new-current-uuid --clear-bitmap r0/0
The DRBD device is mounted on /var/lib/lxc in /etc/fstab there is a noauto line:
/dev/drbd0 /var/lib/lxc ext4 noauto,defaults 0 0
To prevent split brain situations a manual step is required at boot time, on the machine that is going to be the primary.
sudo drbdadm primary r0
sudo drbdsetup status
sudo mount /var/lib/lxc
sudo lxc-autostart start
sudo lxc-ls -f
sudo drbdsetup status