forgejo/infrastructure-documentation
1
0
Fork 0
mirror of https://code.forgejo.org/infrastructure/documentation synced 2024-11-15 09:31:52 +00:00
Code Issues Activity

Merge pull request 'k8s: the ipv6 range for nodes must not conflict with cluster/service' (#23) from earl-warren/documentation:wip-k8s-ipv6 into main

Browse source 
Reviewed-on: https://code.forgejo.org/infrastructure/documentation/pulls/23
This commit is contained in:
earl-warren 2024-10-06 10:40:51 +00:00
parent aeda8706d3 f70e35d1e8
commit 98bf0f94a9
1 changed files with 32 additions and 52 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

84
README.md
View file

@ -799,7 +799,7 @@ iface enp5s0 inet6 static
#### k8s node #### k8s node
The `10.88.1.5` and `fd01::5` IPs are assigned to the interface with VLAN 4002. The `10.88.1.5` and `fe88::5` IPs are assigned to the interface with VLAN 4002.
``` ```
auto enp5s0.4002 auto enp5s0.4002
@ -808,50 +808,28 @@ iface enp5s0.4002 inet static
netmask 255.255.0.0 netmask 255.255.0.0
vlan-raw-device enp5s0 vlan-raw-device enp5s0
mtu 1400 mtu 1400
up ip addr add fd01::5/48 dev enp5s0.4002 up ip addr add fe88::5/48 dev enp5s0.4002
``` ```
### k8s first server node ### k8s first server node
```sh
$ cat /etc/rancher/k3s/config.yaml.d/config.yaml
# https://docs.k3s.io/installation/network-options
# the ipv4-ranges are default in k3s
# the fd-range is a private ipv6-range
cluster-cidr: 10.42.0.0/16,fd01::/48
service-cidr: 10.43.0.0/16,fd02::/112
disable:
- servicelb
```
```sh ```sh
sudo apt-get install curl sudo apt-get install curl
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode=644 --cluster-init --node-ip=10.88.1.5,fd01::5" sh - master_node_ip=10.88.1.5,fe88::5
curl -fL https://get.k3s.io | sh -s - server --cluster-init --disable=servicelb --write-kubeconfig-mode=644 --node-ip=$master_node_ip --cluster-cidr=10.42.0.0/16,fd01::/48 --service-cidr=10.43.0.0/16,fd02::/112
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash - curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -
``` ```
### k8s second server node ### k8s second server node
```sh
$ cat /etc/rancher/k3s/config.yaml.d/config.yaml
# https://docs.k3s.io/installation/network-options
# the ipv4-ranges are default in k3s
# the fd-range is a private ipv6-range
cluster-cidr: 10.42.0.0/16,fd01::/48
service-cidr: 10.43.0.0/16,fd02::/112
disable:
- servicelb
```
The token is found on the first node in the `/var/lib/rancher/k3s/server/token` file. The token is found on the first node in the `/var/lib/rancher/k3s/server/token` file.
```sh ```sh
sudo apt-get install curl sudo apt-get install curl
master_node_ip=10.88.1.5 token=???
second_server_ip=10.88.1.6,fd01::6 master_ip=10.88.1.5
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode=644 --token $token --server https://$master_node_ip:6443 --node-ip=$second_server_ip" sh - second_node_ip=10.88.1.6,fe88::6
curl -fL https://get.k3s.io | sh -s - server --token $token --server https://$master_ip:6443 --cluster-init --disable=servicelb --write-kubeconfig-mode=644 --node-ip=$second_node_ip --cluster-cidr=10.42.0.0/16,fd01::/48 --service-cidr=10.43.0.0/16,fd02::/112
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash - curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -
``` ```
@ -862,9 +840,9 @@ curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 |
The token is found on one of the master nodes in the `/var/lib/rancher/k3s/server/token` file. The token is found on one of the master nodes in the `/var/lib/rancher/k3s/server/token` file.
```sh ```sh
master_node_ip=10.88.1.5 master_ip=10.88.1.5
etcd_node_ip=10.88.1.3,fd01::3 etcd_node_ip=10.88.1.3,fe88::3
curl -fL https://get.k3s.io | sh -s - server --token "$token" --server https://$master_node_ip:6443 --cluster-init --disable-apiserver --disable-controller-manager --disable-scheduler --write-kubeconfig-mode=644 --node-ip=$etcd_node_ip --cluster-cidr=10.42.0.0/16,fd01::/48 --service-cidr=10.43.0.0/16,fd02::/112 curl -sfL https://get.k3s.io | sh -s - server --token "$token" --server https://$master_ip:6443 --cluster-init --disable-apiserver --disable-controller-manager --disable-scheduler --write-kubeconfig-mode=644 --node-ip=$etcd_node_ip --cluster-cidr=10.42.0.0/16,fd01::/48 --service-cidr=10.43.0.0/16,fd02::/112
``` ```
It should not be allowed to schedule pods but for some reason it is. Working around this with: It should not be allowed to schedule pods but for some reason it is. Working around this with:
@ -875,29 +853,11 @@ kubectl taint nodes hetzner03 key1=value1:NoSchedule
### k8s networking ### k8s networking
`k3s` is configured to use [metallb](https://metallb.universe.tf) instead of the default load balancer because it does not allow for a public IP different from the `k8s` node IP.
[metallb](https://metallb.universe.tf).
```
helm install metallb metallb/metallb
cat > metallb.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
spec:
addresses:
- 188.40.16.47/32
- 2a01:4f8:fff2:48::0/64
EOF
kubectl apply --server-side=true -f metallb.yml
```
[cert-manager](https://cert-manager.io/). [cert-manager](https://cert-manager.io/).
``` ```
helm install mycertmanager --set installCRDs=true oci://registry-1.docker.io/bitnamicharts/cert-manager helm install mycertmanager --set installCRDs=true oci://registry-1.docker.io/bitnamicharts/cert-manager
# wait a few seconds
cat > clusterissuer.yml <<EOF cat > clusterissuer.yml <<EOF
apiVersion: cert-manager.io/v1 apiVersion: cert-manager.io/v1
kind: ClusterIssuer kind: ClusterIssuer
@ -917,6 +877,26 @@ EOF
kubectl apply --server-side=true -f clusterissuer.yml kubectl apply --server-side=true -f clusterissuer.yml
``` ```
`k3s` is configured to use [metallb](https://metallb.universe.tf) instead of the default load balancer because it does not allow for a public IP different from the `k8s` node IP.
[metallb](https://metallb.universe.tf).
```
helm install metallb metallb/metallb
# wait a few seconds
cat > metallb.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
spec:
addresses:
- 188.40.16.47/32
- 2a01:4f8:fff2:48::0/64
EOF
kubectl apply --server-side=true -f metallb.yml
```
[traefik](https://traefik.io/) requests with [annotations](https://github.com/traefik/traefik-helm-chart/blob/7a13fc8a61a6ad30fcec32eec497dab9d8aea686/traefik/values.yaml#L736) specific IPs from `metalldb`. [traefik](https://traefik.io/) requests with [annotations](https://github.com/traefik/traefik-helm-chart/blob/7a13fc8a61a6ad30fcec32eec497dab9d8aea686/traefik/values.yaml#L736) specific IPs from `metalldb`.
``` ```