mirror of
https://code.forgejo.org/infrastructure/documentation
synced 2024-11-15 09:31:52 +00:00
Merge pull request 'hetzner{05,06} k8s node & network configuration' (#13) from earl-warren/documentation:wip-k8s-network into main
Reviewed-on: https://code.forgejo.org/infrastructure/documentation/pulls/13
This commit is contained in:
earl-warren
commit
b6a17d46ab
181
README.md
181
README.md
|
|
@ -343,6 +343,14 @@ iface enp5s0.4000 inet static
|
||||||
|
|
||||||
The IP address ends with the same number as the hardware (hetzner02 => .2).
|
The IP address ends with the same number as the hardware (hetzner02 => .2).
|
||||||
|
|
||||||
|
#### vSwitch DRBD
|
||||||
|
|
||||||
|
The vSwitch on VLAN 4000 is for DRBD exclusively
|
||||||
|
|
||||||
|
#### vSwitch k8s
|
||||||
|
|
||||||
|
The vSwitch on VLAN 4002 is for the k8s control plane
|
||||||
|
|
||||||
### DRBD
|
### DRBD
|
||||||
|
|
||||||
DRBD is [configured](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#p-work) like in the following example with hetzner02 as the primary and hetzner03 as the secondary:
|
DRBD is [configured](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#p-work) like in the following example with hetzner02 as the primary and hetzner03 as the secondary:
|
||||||
|
|
@ -724,6 +732,179 @@ stream {
|
||||||
|
|
||||||
Dedicated to https://codeberg.org/forgejo-contrib/forgejo-helm and running from an ephemeral disk
|
Dedicated to https://codeberg.org/forgejo-contrib/forgejo-helm and running from an ephemeral disk
|
||||||
|
|
||||||
|
### hetzner{05,06}
|
||||||
|
|
||||||
|
https://hetzner05.forgejo.org & https://hetzner06.forgejo.org run on [EX44](https://www.hetzner.com/dedicated-rootserver/ex44) Hetzner hardware.
|
||||||
|
|
||||||
|
|
||||||
|
#### LXC
|
||||||
|
|
||||||
|
```sh
|
||||||
|
lxc-helpers.sh lxc_install_lxc_inside 10.47.3 fc11
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Disk partitioning
|
||||||
|
|
||||||
|
- First disk
|
||||||
|
- OS
|
||||||
|
- non precious data such as the LXC containers with runners.
|
||||||
|
- Second disk
|
||||||
|
- a partition configured with DRBD
|
||||||
|
|
||||||
|
#### Root filesystem backups
|
||||||
|
|
||||||
|
- `hetzner05:/etc/cron.daily/backup-hetzner06`
|
||||||
|
`rsync -aHSv --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.6:/ /srv/backups/hetzner06/`
|
||||||
|
- `hetzner06:/etc/cron.daily/backup-hetzner05`
|
||||||
|
`rsync -aHSv --delete-excluded --delete --numeric-ids --exclude /proc --exclude /dev --exclude /sys --exclude /srv --exclude /var/lib/lxc 10.53.100.5:/ /srv/backups/hetzner05/`
|
||||||
|
|
||||||
|
#### Failover IP addresses
|
||||||
|
|
||||||
|
The failover IP addresses are configured on all hosts. Their routing is managed by the [Hetzner k8s controller](https://github.com/cbeneke/hcloud-fip-controller).
|
||||||
|
|
||||||
|
```
|
||||||
|
auto enp5s0
|
||||||
|
iface enp5s0 inet static
|
||||||
|
...
|
||||||
|
up ip addr add 188.40.16.47/32 dev enp5s0
|
||||||
|
|
||||||
|
iface enp5s0 inet6 static
|
||||||
|
...
|
||||||
|
up ip addr add 2a01:4f8:fff2:48::2/64 dev enp5s0
|
||||||
|
```
|
||||||
|
|
||||||
|
#### k8s node
|
||||||
|
|
||||||
|
```sh
|
||||||
|
$ cat /etc/rancher/k3s/config.yaml.d/config.yaml
|
||||||
|
# https://docs.k3s.io/installation/network-options
|
||||||
|
# the ipv4-ranges are default in k3s
|
||||||
|
# the fd-range is a private ipv6-range
|
||||||
|
cluster-cidr: 10.42.0.0/16,fd01::/48
|
||||||
|
service-cidr: 10.43.0.0/16,fd02::/112
|
||||||
|
|
||||||
|
disable:
|
||||||
|
- servicelb
|
||||||
|
```
|
||||||
|
|
||||||
|
The `10.88.1.5` and `fd01::5` IPs are assigned to the interface with VLAN 4002.
|
||||||
|
|
||||||
|
```
|
||||||
|
auto enp5s0.4002
|
||||||
|
iface enp5s0.4002 inet static
|
||||||
|
address 10.88.1.5
|
||||||
|
netmask 255.255.0.0
|
||||||
|
vlan-raw-device enp5s0
|
||||||
|
mtu 1400
|
||||||
|
up ip addr add fd01::5/48 dev enp5s0.4002
|
||||||
|
```
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo apt-get install curl
|
||||||
|
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--write-kubeconfig-mode=644 --node-ip=10.88.1.5,fd01::5' sh -
|
||||||
|
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -
|
||||||
|
```
|
||||||
|
|
||||||
|
### k8s networking
|
||||||
|
|
||||||
|
`k3s` is configured to use [metallb](https://metallb.universe.tf) instead of the default load balancer because it does not allow for a public IP different from the `k8s` node IP.
|
||||||
|
|
||||||
|
[metallb](https://metallb.universe.tf).
|
||||||
|
|
||||||
|
```
|
||||||
|
helm install metallb metallb/metallb
|
||||||
|
cat > metallb.yaml <<EOF
|
||||||
|
apiVersion: metallb.io/v1beta1
|
||||||
|
kind: IPAddressPool
|
||||||
|
metadata:
|
||||||
|
name: first-pool
|
||||||
|
spec:
|
||||||
|
addresses:
|
||||||
|
- 188.40.16.47/32
|
||||||
|
- 2a01:4f8:fff2:48::0/64
|
||||||
|
EOF
|
||||||
|
kubectl apply --server-side=true -f metallb.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
[cert-manager](https://cert-manager.io/).
|
||||||
|
|
||||||
|
```
|
||||||
|
helm install mycertmanager --set installCRDs=true oci://registry-1.docker.io/bitnamicharts/cert-manager
|
||||||
|
cat > clusterissuer.yml <<EOF
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: ClusterIssuer
|
||||||
|
metadata:
|
||||||
|
name: letsencrypt-http
|
||||||
|
spec:
|
||||||
|
acme:
|
||||||
|
email: contact@forgejo.org
|
||||||
|
server: https://acme-v02.api.letsencrypt.org/directory
|
||||||
|
privateKeySecretRef:
|
||||||
|
name: letsencrypt-http
|
||||||
|
solvers:
|
||||||
|
- http01:
|
||||||
|
ingress:
|
||||||
|
class: traefik
|
||||||
|
EOF
|
||||||
|
kubectl apply --server-side=true -f clusterissuer.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
[traefik](https://traefik.io/) requests with [annotations](https://github.com/traefik/traefik-helm-chart/blob/7a13fc8a61a6ad30fcec32eec497dab9d8aea686/traefik/values.yaml#L736) specific IPs from `metalldb`.
|
||||||
|
|
||||||
|
```
|
||||||
|
cat > traefik.yml <<EOF
|
||||||
|
apiVersion: helm.cattle.io/v1
|
||||||
|
kind: HelmChartConfig
|
||||||
|
metadata:
|
||||||
|
name: traefik
|
||||||
|
namespace: kube-system
|
||||||
|
spec:
|
||||||
|
valuesContent: |-
|
||||||
|
service:
|
||||||
|
annotations:
|
||||||
|
metallb.universe.tf/allow-shared-ip: "key-to-share-188-40-16-47"
|
||||||
|
metallb.universe.tf/loadBalancerIPs: 188.40.16.47,2a01:4f8:fff2:48::2
|
||||||
|
EOF
|
||||||
|
kubectl apply --server-side=true -f traefik.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
[forgejo](https://code.forgejo.org/forgejo-helm/forgejo-helm) configuration in [ingress](https://code.forgejo.org/forgejo-helm/forgejo-helm#ingress) for the reverse proxy (`traefik`) to route the domain and for the ACME issuer (`cert-manager`) to obtain a certificate. And in [service](https://code.forgejo.org/forgejo-helm/forgejo-helm#service) for the `ssh` port to be bound to the desired IPs of the load balancer (`metallb`).
|
||||||
|
|
||||||
|
```
|
||||||
|
ingress:
|
||||||
|
enabled: true
|
||||||
|
annotations:
|
||||||
|
# https://cert-manager.io/docs/usage/ingress/#supported-annotations
|
||||||
|
# https://github.com/cert-manager/cert-manager/issues/2239
|
||||||
|
cert-manager.io/cluster-issuer: letsencrypt-http
|
||||||
|
cert-manager.io/private-key-algorithm: ECDSA
|
||||||
|
cert-manager.io/private-key-size: 384
|
||||||
|
kubernetes.io/ingress.class: traefik
|
||||||
|
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||||
|
tls:
|
||||||
|
- hosts:
|
||||||
|
- t1.forgejo.org
|
||||||
|
secretName: tls-forgejo-t1-ingress-http
|
||||||
|
hosts:
|
||||||
|
- host: t1.forgejo.org
|
||||||
|
paths:
|
||||||
|
- path: /
|
||||||
|
pathType: Prefix
|
||||||
|
|
||||||
|
service:
|
||||||
|
http:
|
||||||
|
type: ClusterIP
|
||||||
|
ipFamilyPolicy: PreferDualStack
|
||||||
|
port: 3000
|
||||||
|
ssh:
|
||||||
|
type: LoadBalancer
|
||||||
|
annotations:
|
||||||
|
metallb.universe.tf/loadBalancerIPs: 188.40.16.47,2a01:4f8:fff2:48::2
|
||||||
|
metallb.universe.tf/allow-shared-ip: "key-to-share-188-40-16-47"
|
||||||
|
ipFamilyPolicy: PreferDualStack
|
||||||
|
port: 2222
|
||||||
|
```
|
||||||
|
|
||||||
## Uberspace
|
## Uberspace
|
||||||
|
|
||||||
The website https://forgejo.org is hosted at
|
The website https://forgejo.org is hosted at
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue