blackbox-exporter: scrape every 60m, add URLs

style: fix formatting
monitoring: scrape blackbox every 5m, alert 21 days
2024-11-06 21:54:16 +01:00 · 2024-11-06 21:31:41 +01:00 · 2024-11-06 21:31:41 +01:00 · 2024-11-06 21:31:41 +01:00 · 2024-11-06 21:31:40 +01:00 · 2024-11-06 21:31:40 +01:00
59 changed files with 1837 additions and 749 deletions
--- a/.forgejo/workflows/check.yml
+++ b/.forgejo/workflows/check.yml
@ -18,7 +18,20 @@ jobs:
          # Prevent cache garbage collection by creating GC roots
          mkdir -p /var/lib/gitea-runner/tankstelle/.local/state/nix/results

-          sed -i 's/virtualisation.cores .*/virtualisation.cores = 16;/' tests/keycloak.nix
-          sed -i 's/virtualisation.memorySize .*/virtualisation.memorySize = 16384;/' tests/keycloak.nix
-          # 1 eval-worker needs about 13GB of memory
-          nix --accept-flake-config --access-tokens '' develop --command nix-fast-build --no-nom --skip-cached --systems "x86_64-linux" --max-jobs 10 --eval-workers 2 --out-link /var/lib/gitea-runner/tankstelle/.local/state/nix/results/nix-fast-build
+          for target in $(nix flake show --json --all-systems | jq --raw-output '
+            .["nixosConfigurations"] |
+            to_entries[] |
+            .key'
+          ); do
+            nix --print-build-logs --verbose --accept-flake-config --access-tokens '' \
+              build --out-link /var/lib/gitea-runner/tankstelle/.local/state/nix/results/"$target" ".#nixosConfigurations.${target}.config.system.build.toplevel"
+          done
+
+          for check in $(nix flake show --json --all-systems | jq --raw-output '
+            .checks."x86_64-linux" |
+            to_entries[] |
+            .key'
+          ); do
+            nix --print-build-logs --verbose --accept-flake-config --access-tokens '' \
+              build --out-link /var/lib/gitea-runner/tankstelle/.local/state/nix/results/"$check" ".#checks.x86_64-linux.${check}"
+          done
--- a/docs/administrative-access.md
+++ b/docs/administrative-access.md
@ -28,18 +28,18 @@ People with admin access to the infrastructure are added to [`logins/admins.nix`
 SSH is not reachable from the open internet. Instead, SSH Port 22 is protected by a wireguard VPN network. Thus, to get root access on the servers, at least two pieces of information have to be added to the admins config:

 1. **SSH Public key**: self-explanatory. Add your public key to your user attrset under `sshPubKeys`.
-2. **Wireguard device**: each wireguard device has two parts: the public key and the IP addresses it should have in the wireguard network. The pub.solar wireguard network uses the subnets `10.7.6.0/24` and `fd00:fae:fae:fae:fae::/80`. To add your device, it's best to choose a free number between 200 and 255 and use that in both the ipv4 and ipv6 ranges: `10.7.6.<ip-address>/32` `fd00:fae:fae:fae:fae:<ip-address>::/96`. For more information on how to generate keypairs, see [the NixOS Wireguard docs](https://nixos.wiki/wiki/WireGuard#Generate_keypair).
+2. **Wireguard device**: each wireguard device has two parts: the public key and the IP addresses it should have in the wireguard network. The pub.solar wireguard network is spaced under `10.7.6.0/24` and `fd00:fae:fae:fae:fae::/80`. To add your device, it's best to choose a free number between 200 and 255 and use that in both the ipv4 and ipv6 ranges: `10.7.6.<ip-address>/32` `fd00:fae:fae:fae:fae:<ip-address>::/96`. For more information on how to generate keypairs, see [the NixOS Wireguard docs](https://nixos.wiki/wiki/WireGuard#Generate_keypair).

 One can access our hosts using this domain scheme:

 ```
-ssh <unix-username>@<hostname>.wg.pub.solar
+ssh barkeeper@<hostname>.wg.pub.solar
 ```

 So, for example for `nachtigall`:

 ```
-ssh teutat3s@nachtigall.wg.pub.solar
+ssh barkeeper@nachtigall.wg.pub.solar
 ```

 Example NixOS snippet for WireGuard client config
--- a/docs/deletion-request.md
+++ b/docs/deletion-request.md
@ -34,13 +34,7 @@ Docs: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server
 ### Mastodon

 ```
-mkdir /tmp/tootctl
-sudo chown mastodon /tmp/tootctl
-cd /tmp/tootctl
-
 sudo -u mastodon mastodon-tootctl accounts delete --email <mail-address>
-
-rm -r /tmp/tootctl
 ```

 Docs: https://docs.joinmastodon.org/admin/tootctl/#accounts-delete
--- a/docs/deploying.md
+++ b/docs/deploying.md
@ -7,29 +7,16 @@ be manually deployed.
 To deploy, make sure you have a [working development shell](./development-shell.md).
 Then, run `deploy-rs` with the hostname of the server you want to deploy:

-### Dry-run
-
-Use `--dry-activate` to show a diff of updated packages and all services that
-would be restarted by the update. This will also put all files in place without
-switching to the new generation, enabling a quick switch to the new config at a
-later moment.
-
 For nachtigall.pub.solar:

 ```
-deploy --targets '.#nachtigall' --ssh-user <unix-username> --magic-rollback false --auto-rollback false --keep-result --result-path ./results --dry-activate
-```
-
-After reviewing the changes, apply the update with:
-
-```
-deploy --targets '.#nachtigall' --ssh-user <unix-username> --magic-rollback false --auto-rollback false --keep-result --result-path ./results
+deploy --targets '.#nachtigall' --magic-rollback false --auto-rollback false --keep-result --result-path ./results
 ```

 For metronom.pub.solar (aarch64-linux):

 ```
-deploy --targets '.#metronom' --ssh-user <unix-username> --magic-rollback false --auto-rollback false --keep-result --result-path ./results --remote-build
+deploy --targets '.#metronom' --magic-rollback false --auto-rollback false --keep-result --result-path ./results --remote-build
 ```

 Usually we skip all rollback functionality, but if you want to deploy a change
@ -38,6 +25,9 @@ that might lock you out, e.g. to SSH, it might make sense to set these to `true`
 To skip flake checks, e.g. because you already ran them manually before
 deployment, add the flag `--skip-checks` at the end of the command.

+`--dry-activate` can be used to only put all files in place without switching,
+to enable switching to the new config quickly at a later moment.
+
 We use `--keep-result --result-path ./results` to keep the last `result`
 symlink of each `deploy` from being garbage collected. That way, we keep builds
 cached in the Nix store. This is optional and both flags can be removed if disk
--- a/docs/garage.md
+++ b/docs/garage.md
@ -8,7 +8,7 @@ Requirements:
 - [Setup WireGuard](./administrative-access.md#ssh-access) for hosts: `trinkgenossin`, optionally: `delite`, `blue-shell`

 ```
-ssh <unix-username>@trinkgenossin.wg.pub.solar
+ssh barkeeper@trinkgenossin.wg.pub.solar
 ```

 ```
@ -58,7 +58,7 @@ Further reading:
 ### Notes on manual setup steps

 ```
-ssh <unix-username>@trinkgenossin.wg.pub.solar
+ssh barkeeper@trinkgenossin.wg.pub.solar

 # Add a few spaces to avoid leaking the secret to the shell history
    export GARAGE_RPC_SECRET=<secret-in-keepass>
--- a/docs/keycloak/delete-unverified-accounts.md
+++ b/docs/keycloak/delete-unverified-accounts.md
@ -12,7 +12,7 @@ Run following after SSH'ing to `nachtigall`.
 Credentials for the following command are in keepass. Create a keycloak
 config/credentials file at `/tmp/kcadm.config`:

-```bash
+```
 sudo --user keycloak kcadm.sh config credentials \
  --config /tmp/kcadm.config \
  --server https://auth.pub.solar \
@ -22,7 +22,7 @@ sudo --user keycloak kcadm.sh config credentials \

 Get list of accounts without a verified email address:

-```bash
+```
 sudo --user keycloak kcadm.sh get \
  --config /tmp/kcadm.config \
  users \
@ -35,7 +35,7 @@ Review list of accounts, especially check `createdTimestamp` if any accounts
 were created in the past 2 days. If so, delete those from the
 `/tmp/keycloak-unverified-accounts` file.

-```bash
+```
 createdTimestamps=( $( nix run nixpkgs#jq -- -r '.[].createdTimestamp' < /tmp/keycloak-unverified-accounts ) )

 # timestamps are in nanoseconds since epoch, so we need to strip the last three digits
@ -46,17 +46,17 @@ vim /tmp/keycloak-unverified-accounts

 Check how many accounts are going to be deleted:

-```bash
+```
 jq -r '.[].id' < /tmp/keycloak-unverified-accounts | wc -l
 ```

-```bash
+```
 jq -r '.[].id' < /tmp/keycloak-unverified-accounts > /tmp/keycloak-unverified-account-ids
 ```

 Final check before deletion (dry-run):

-```bash
+```
 for id in $(cat /tmp/keycloak-unverified-account-ids)
  do
    echo sudo --user keycloak kcadm.sh delete \
@ -68,7 +68,7 @@ for id in $(cat /tmp/keycloak-unverified-account-ids)

 THIS WILL DELETE ACCOUNTS:

-```bash
+```
 for id in $(cat /tmp/keycloak-unverified-account-ids)
  do
    sudo --user keycloak kcadm.sh delete \
@ -77,9 +77,3 @@ for id in $(cat /tmp/keycloak-unverified-account-ids)
      --realm pub.solar
  done
 ```
-
-Delete the temp files:
-
-```bash
-sudo rm /tmp/kcadm.config /tmp/keycloak-unverified-accounts /tmp/keycloak-unverified-account-ids
-```
--- a/docs/matrix-suspend-account.md
+++ b/docs/matrix-suspend-account.md
@ -1,27 +0,0 @@
-# Matrix account suspension
-
-> Unlike [account locking](https://spec.matrix.org/v1.12/client-server-api/#account-locking),
-> [suspension](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3823-code-for-account-suspension.md)
-> allows the user to have a (largely) readonly view of their account.
-> Homeserver administrators and moderators may use this functionality to
-> temporarily deactivate an account, or place conditions on the account's
-> experience. Critically, like locking, account suspension is reversible, unlike
-> the deactivation mechanism currently available in Matrix - a destructive,
-> irreversible, action.
-
-Required:
-
- `matrix-synapse admin token`
- [SSH access to host `nachtigall`](./administrative-access.md#ssh-access)
-
-## Suspending an account
-
-```bash
-curl --header "Authorization: Bearer <admin-access-token>" --request PUT http://127.0.0.1:8008/_synapse/admin/v1/suspend/@<username>:pub.solar --data '{"suspend": true}'
-```
-
-## Unsuspending an account
-
-```bash
-curl --header "Authorization: Bearer <admin-access-token>" --request PUT http://127.0.0.1:8008/_synapse/admin/v1/suspend/@<username>:pub.solar --data '{"suspend": false}'
-```
--- a/docs/nix-flake-updates.md
+++ b/docs/nix-flake-updates.md
@ -41,7 +41,3 @@ wrapped-ruby-mastodon-gems: 4.2.1 → 4.2.3
 zfs-kernel: 2.2.1-6.1.64 → 2.2.2-6.1.66
 zfs-user: 2.2.1 → 2.2.2
 ```
-
-### Deploying updates
-
-See [deploying.md](./deploying.md).
--- a/flake.lock
+++ b/flake.lock
@ -14,11 +14,11 @@
        "systems": "systems"
      },
      "locked": {
-        "lastModified": 1736955230,
-        "narHash": "sha256-uenf8fv2eG5bKM8C/UvFaiJMZ4IpUFaQxk9OH5t/1gA=",
+        "lastModified": 1723293904,
+        "narHash": "sha256-b+uqzj+Wa6xgMS9aNbX4I+sXeb5biPDi39VgvSFqFvU=",
        "owner": "ryantm",
        "repo": "agenix",
-        "rev": "e600439ec4c273cf11e06fe4d9d906fb98fa097c",
+        "rev": "f6291c5935fdc4e0bef208cfc0dcab7e3f7a1c41",
        "type": "github"
      },
      "original": {
@ -94,11 +94,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1738765162,
-        "narHash": "sha256-3Z40qHaFScWUCVQrGc4Y+RdoPsh1R/wIh+AN4cTXP0I=",
+        "lastModified": 1729712798,
+        "narHash": "sha256-a+Aakkb+amHw4biOZ0iMo8xYl37uUL48YEXIC5PYJ/8=",
        "owner": "nix-community",
        "repo": "disko",
-        "rev": "ff3568858c54bd306e9e1f2886f0f781df307dff",
+        "rev": "09a776702b004fdf9c41a024e1299d575ee18a7d",
        "type": "github"
      },
      "original": {
@ -185,11 +185,11 @@
        "nixpkgs-lib": "nixpkgs-lib"
      },
      "locked": {
-        "lastModified": 1738453229,
-        "narHash": "sha256-7H9XgNiGLKN1G1CgRh0vUL4AheZSYzPm+zmZ7vxbJdo=",
+        "lastModified": 1727826117,
+        "narHash": "sha256-K5ZLCyfO/Zj9mPFldf3iwS6oZStJcU4tSpiXTMYaaL0=",
        "owner": "hercules-ci",
        "repo": "flake-parts",
-        "rev": "32ea77a06711b758da0ad9bd6a844c5740a87abd",
+        "rev": "3d04084d54bedc3d6b8b736c70ef449225c361b1",
        "type": "github"
      },
      "original": {
@ -236,16 +236,16 @@
    },
    "fork": {
      "locked": {
-        "lastModified": 1738846146,
-        "narHash": "sha256-cIPiBEspPXQxju2AUZK9kjh6oqea+HkPFqmGv7yUztM=",
+        "lastModified": 1729963002,
+        "narHash": "sha256-2zrYfd/qdfExU5zVwvH80uJnKc/dMeK6zp3O1UtW2Mo=",
        "owner": "teutat3s",
        "repo": "nixpkgs",
-        "rev": "e370f40b129e47b08562524ab4f053a172a94273",
+        "rev": "005faaacbeede0296dec5c844f508027ab8a3ff6",
        "type": "github"
      },
      "original": {
        "owner": "teutat3s",
-        "ref": "init-matrix-authentication-service-module-0.13.0",
+        "ref": "init-matrix-authentication-service-module",
        "repo": "nixpkgs",
        "type": "github"
      }
@ -257,16 +257,16 @@
        ]
      },
      "locked": {
-        "lastModified": 1736373539,
-        "narHash": "sha256-dinzAqCjenWDxuy+MqUQq0I4zUSfaCvN9rzuCmgMZJY=",
+        "lastModified": 1726989464,
+        "narHash": "sha256-Vl+WVTJwutXkimwGprnEtXc/s/s8sMuXzqXaspIGlwM=",
        "owner": "nix-community",
        "repo": "home-manager",
-        "rev": "bd65bc3cde04c16755955630b344bc9e35272c56",
+        "rev": "2f23fa308a7c067e52dfcc30a0758f47043ec176",
        "type": "github"
      },
      "original": {
        "owner": "nix-community",
-        "ref": "release-24.11",
+        "ref": "release-24.05",
        "repo": "home-manager",
        "type": "github"
      }
@ -280,11 +280,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1738012343,
-        "narHash": "sha256-agMgWwVxXII+RtCqok8ROjzpKJung/5N5f2BVDmMC5Q=",
+        "lastModified": 1730041422,
+        "narHash": "sha256-aEz5/yUJN/PSEXwPBuKMs2FbAmz68fDIQ9B0tVRVmTo=",
        "ref": "main",
-        "rev": "4ffd7bc8ea032991756c5e8e8a37b039789045bc",
-        "revCount": 38,
+        "rev": "09f7b1ed16c99f5fb5c5f9a2a73ccc9ff0645b35",
+        "revCount": 32,
        "type": "git",
        "url": "https://git.pub.solar/pub-solar/keycloak-theme"
      },
@ -298,11 +298,11 @@
      "flake": false,
      "locked": {
        "dir": "web",
-        "lastModified": 1733177811,
-        "narHash": "sha256-1n7bPSCRw7keTCIu4tJGnUlkoId6H1+dPsTPzKo3Rrk=",
+        "lastModified": 1718796561,
+        "narHash": "sha256-RKAAHve17lrJokgAPkM2k/E+f9djencwwg3Xcd70Yfw=",
        "owner": "maunium",
        "repo": "stickerpicker",
-        "rev": "89d3aece041c85ebe5a1ad4e620388af5227cbb0",
+        "rev": "333567f481e60443360aa7199d481e1a45b3a523",
        "type": "github"
      },
      "original": {
@ -320,11 +320,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1739034224,
-        "narHash": "sha256-Mj/8jDzh1KNmUhWqEeVlW3hO9MZkxqioJGnmR7rivaE=",
+        "lastModified": 1729757100,
+        "narHash": "sha256-x+8uGaX66V5+fUBHY23Q/OQyibQ38nISzxgj7A7Jqds=",
        "owner": "lnl7",
        "repo": "nix-darwin",
-        "rev": "0b6f96a6b9efcfa8d3cc8023008bcbcd1b9bc1a4",
+        "rev": "04193f188e4144d7047f83ad1de81d6034d175cd",
        "type": "github"
      },
      "original": {
@ -336,30 +336,30 @@
    },
    "nixpkgs": {
      "locked": {
-        "lastModified": 1739055578,
-        "narHash": "sha256-2MhC2Bgd06uI1A0vkdNUyDYsMD0SLNGKtD8600mZ69A=",
+        "lastModified": 1729449015,
+        "narHash": "sha256-Gf04dXB0n4q0A9G5nTGH3zuMGr6jtJppqdeljxua1fo=",
        "owner": "nixos",
        "repo": "nixpkgs",
-        "rev": "a45fa362d887f4d4a7157d95c28ca9ce2899b70e",
+        "rev": "89172919243df199fe237ba0f776c3e3e3d72367",
        "type": "github"
      },
      "original": {
        "owner": "nixos",
-        "ref": "nixos-24.11",
+        "ref": "nixos-24.05",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "nixpkgs-lib": {
      "locked": {
-        "lastModified": 1738452942,
-        "narHash": "sha256-vJzFZGaCpnmo7I6i416HaBLpC+hvcURh/BQwROcGIp8=",
+        "lastModified": 1727825735,
+        "narHash": "sha256-0xHYkMkeLVQAMa7gvkddbPqpxph+hDzdu1XdGPJR+Os=",
        "type": "tarball",
-        "url": "https://github.com/NixOS/nixpkgs/archive/072a6db25e947df2f31aab9eccd0ab75d5b2da11.tar.gz"
+        "url": "https://github.com/NixOS/nixpkgs/archive/fb192fec7cc7a4c26d51779e9bab07ce6fa5597a.tar.gz"
      },
      "original": {
        "type": "tarball",
-        "url": "https://github.com/NixOS/nixpkgs/archive/072a6db25e947df2f31aab9eccd0ab75d5b2da11.tar.gz"
+        "url": "https://github.com/NixOS/nixpkgs/archive/fb192fec7cc7a4c26d51779e9bab07ce6fa5597a.tar.gz"
      }
    },
    "root": {
@ -387,21 +387,22 @@
        "nixpkgs": [
          "unstable"
        ],
-        "nixpkgs-24_11": [
+        "nixpkgs-24_05": [
          "nixpkgs"
-        ]
+        ],
+        "utils": "utils_2"
      },
      "locked": {
-        "lastModified": 1734884447,
-        "narHash": "sha256-HA9fAmGNGf0cOYrhgoa+B6BxNVqGAYXfLyx8zIS0ZBY=",
+        "lastModified": 1718084203,
+        "narHash": "sha256-Cx1xoVfSMv1XDLgKg08CUd1EoTYWB45VmB9XIQzhmzI=",
        "owner": "simple-nixos-mailserver",
        "repo": "nixos-mailserver",
-        "rev": "63209b1def2c9fc891ad271f474a3464a5833294",
+        "rev": "29916981e7b3b5782dc5085ad18490113f8ff63b",
        "type": "gitlab"
      },
      "original": {
        "owner": "simple-nixos-mailserver",
-        "ref": "nixos-24.11",
+        "ref": "nixos-24.05",
        "repo": "nixos-mailserver",
        "type": "gitlab"
      }
@ -466,13 +467,28 @@
        "type": "github"
      }
    },
+    "systems_5": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
+      }
+    },
    "unstable": {
      "locked": {
-        "lastModified": 1739020877,
-        "narHash": "sha256-mIvECo/NNdJJ/bXjNqIh8yeoSjVLAuDuTUzAo7dzs8Y=",
+        "lastModified": 1729665710,
+        "narHash": "sha256-AlcmCXJZPIlO5dmFzV3V2XF6x/OpNWUV8Y/FMPGd8Z4=",
        "owner": "nixos",
        "repo": "nixpkgs",
-        "rev": "a79cfe0ebd24952b580b1cf08cd906354996d547",
+        "rev": "2768c7d042a37de65bb1b5b3268fc987e534c49d",
        "type": "github"
      },
      "original": {
@ -499,6 +515,24 @@
        "repo": "flake-utils",
        "type": "github"
      }
+    },
+    "utils_2": {
+      "inputs": {
+        "systems": "systems_5"
+      },
+      "locked": {
+        "lastModified": 1709126324,
+        "narHash": "sha256-q6EQdSeUZOG26WelxqkmR7kArjgWCdw5sfJVHPH/7j8=",
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "rev": "d465f4819400de7c8d874d50b982301f28a84605",
+        "type": "github"
+      },
+      "original": {
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "type": "github"
+      }
    }
  },
  "root": "root",
--- a/flake.nix
+++ b/flake.nix
@ -1,14 +1,14 @@
 {
  inputs = {
    # Track channels with commits tested and built by hydra
-    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.11";
+    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";
    unstable.url = "github:nixos/nixpkgs/nixos-unstable";
-    fork.url = "github:teutat3s/nixpkgs/init-matrix-authentication-service-module-0.13.0";
+    fork.url = "github:teutat3s/nixpkgs/init-matrix-authentication-service-module";

    nix-darwin.url = "github:lnl7/nix-darwin/master";
    nix-darwin.inputs.nixpkgs.follows = "nixpkgs";

-    home-manager.url = "github:nix-community/home-manager/release-24.11";
+    home-manager.url = "github:nix-community/home-manager/release-24.05";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";

    flake-parts.url = "github:hercules-ci/flake-parts";
@ -37,8 +37,8 @@
    element-stickers.inputs.maunium-stickerpicker.follows = "maunium-stickerpicker";
    element-stickers.inputs.nixpkgs.follows = "nixpkgs";

-    simple-nixos-mailserver.url = "gitlab:simple-nixos-mailserver/nixos-mailserver/nixos-24.11";
-    simple-nixos-mailserver.inputs.nixpkgs-24_11.follows = "nixpkgs";
+    simple-nixos-mailserver.url = "gitlab:simple-nixos-mailserver/nixos-mailserver/nixos-24.05";
+    simple-nixos-mailserver.inputs.nixpkgs-24_05.follows = "nixpkgs";
    simple-nixos-mailserver.inputs.nixpkgs.follows = "unstable";
  };

@ -73,28 +73,11 @@
              overlays = [ inputs.agenix.overlays.default ];
            };
            unstable = import inputs.unstable { inherit system; };
+            master = import inputs.master { inherit system; };
          };

          checks =
            let
-              machinesPerSystem = {
-                aarch64-linux = [
-                  "metronom"
-                ];
-                x86_64-linux = [
-                  "blue-shell"
-                  "delite"
-                  "nachtigall"
-                  "tankstelle"
-                  "trinkgenossin"
-                  "underground"
-                ];
-              };
-              nixosMachines = inputs.nixpkgs.lib.mapAttrs' (n: inputs.nixpkgs.lib.nameValuePair "nixos-${n}") (
-                inputs.nixpkgs.lib.genAttrs (machinesPerSystem.${system} or [ ]) (
-                  name: self.nixosConfigurations.${name}.config.system.build.toplevel
-                )
-              );
              nixos-lib = import (inputs.nixpkgs + "/nixos/lib") { };
              testDir = builtins.attrNames (builtins.readDir ./tests);
              testFiles = builtins.filter (n: builtins.match "^.*.nix$" n != null) testDir;
@ -111,13 +94,12 @@
                  }
                );
              }) testFiles
-            )
-            // nixosMachines;
+            );

          devShells.default = pkgs.mkShell {
            buildInputs = with pkgs; [
              deploy-rs
-              nix-fast-build
+              nixpkgs-fmt
              agenix
              age-plugin-yubikey
              cachix
@ -126,9 +108,9 @@
              nvfetcher
              shellcheck
              shfmt
-              treefmt2
+              inputs.unstable.legacyPackages.${system}.treefmt2
              nixos-generators
-              opentofu
+              inputs.unstable.legacyPackages.${system}.opentofu
              terraform-backend-git
              terraform-ls
              jq
@ -138,43 +120,56 @@
          devShells.ci = pkgs.mkShell { buildInputs = with pkgs; [ nodejs ]; };
        };

-      flake = {
-        nixosModules = builtins.listToAttrs (
-          map (x: {
-            name = x;
-            value = import (./modules + "/${x}");
-          }) (builtins.attrNames (builtins.readDir ./modules))
-        );
+      flake =
+        let
+          username = "barkeeper";
+        in
+        {
+          inherit username;

-        checks = builtins.mapAttrs (
-          system: deployLib: deployLib.deployChecks self.deploy
-        ) inputs.deploy-rs.lib;
+          nixosModules = builtins.listToAttrs (
+            map (x: {
+              name = x;
+              value = import (./modules + "/${x}");
+            }) (builtins.attrNames (builtins.readDir ./modules))
+          );

-        formatter."x86_64-linux" = inputs.nixpkgs.legacyPackages."x86_64-linux".nixfmt-rfc-style;
+          checks = builtins.mapAttrs (
+            system: deployLib: deployLib.deployChecks self.deploy
+          ) inputs.deploy-rs.lib;

-        deploy.nodes = self.lib.deploy.mkDeployNodes self.nixosConfigurations {
-          nachtigall = {
-            hostname = "nachtigall.wg.pub.solar";
-          };
-          metronom = {
-            hostname = "metronom.wg.pub.solar";
-          };
-          tankstelle = {
-            hostname = "tankstelle.wg.pub.solar";
-          };
-          underground = {
-            hostname = "80.244.242.3";
-          };
-          trinkgenossin = {
-            hostname = "trinkgenossin.wg.pub.solar";
-          };
-          delite = {
-            hostname = "delite.wg.pub.solar";
-          };
-          blue-shell = {
-            hostname = "blue-shell.wg.pub.solar";
+          formatter."x86_64-linux" = inputs.unstable.legacyPackages."x86_64-linux".nixfmt-rfc-style;
+
+          deploy.nodes = self.lib.deploy.mkDeployNodes self.nixosConfigurations {
+            nachtigall = {
+              hostname = "nachtigall.wg.pub.solar";
+              sshUser = username;
+            };
+            metronom = {
+              hostname = "metronom.wg.pub.solar";
+              sshUser = username;
+            };
+            tankstelle = {
+              hostname = "tankstelle.wg.pub.solar";
+              sshUser = username;
+            };
+            underground = {
+              hostname = "80.244.242.3";
+              sshUser = username;
+            };
+            trinkgenossin = {
+              hostname = "trinkgenossin.wg.pub.solar";
+              sshUser = username;
+            };
+            delite = {
+              hostname = "delite.wg.pub.solar";
+              sshUser = username;
+            };
+            blue-shell = {
+              hostname = "blue-shell.wg.pub.solar";
+              sshUser = username;
+            };
          };
        };
-      };
    };
 }
--- a/hosts/blue-shell/wireguard.nix
+++ b/hosts/blue-shell/wireguard.nix
@ -22,7 +22,7 @@ in
        "${wireguardIPv6}/96"
      ];
      privateKeyFile = config.age.secrets.wg-private-key.path;
-      peers = flake.self.logins.wireguardDevices ++ [
+      peers = flake.self.logins.admins.wireguardDevices ++ [
        {
          # trinkgenossin.pub.solar
          publicKey = "QWgHovHxtqiQhnHLouSWiT6GIoQDmuvnThYL5c/rvU4=";
--- a/hosts/delite/wireguard.nix
+++ b/hosts/delite/wireguard.nix
@ -22,7 +22,7 @@ in
        "${wireguardIPv6}/96"
      ];
      privateKeyFile = config.age.secrets.wg-private-key.path;
-      peers = flake.self.logins.wireguardDevices ++ [
+      peers = flake.self.logins.admins.wireguardDevices ++ [
        {
          # trinkgenossin.pub.solar
          publicKey = "QWgHovHxtqiQhnHLouSWiT6GIoQDmuvnThYL5c/rvU4=";
--- a/hosts/metronom/wireguard.nix
+++ b/hosts/metronom/wireguard.nix
@ -18,7 +18,7 @@
        "fd00:fae:fae:fae:fae:3::/96"
      ];
      privateKeyFile = config.age.secrets.wg-private-key.path;
-      peers = flake.self.logins.wireguardDevices ++ [
+      peers = flake.self.logins.admins.wireguardDevices ++ [
        {
          # nachtigall.pub.solar
          endpoint = "138.201.80.102:51820";
--- a/hosts/nachtigall/configuration.nix
+++ b/hosts/nachtigall/configuration.nix
@ -73,24 +73,22 @@
    owner = "matrix-synapse";
  };

+  age.secrets."matrix-synapse-sliding-sync-secret" = {
+    file = "${flake.self}/secrets/matrix-synapse-sliding-sync-secret.age";
+    mode = "400";
+    owner = "matrix-synapse";
+  };
+
  age.secrets."matrix-authentication-service-secret-config.yml" = {
    file = "${flake.self}/secrets/matrix-authentication-service-secret-config.yml.age";
    mode = "400";
    owner = "matrix-authentication-service";
  };

-  # matrix-appservice-irc
-  age.secrets."matrix-appservice-irc-mediaproxy-signing-key" = {
-    file = "${flake.self}/secrets/matrix-appservice-irc-mediaproxy-signing-key.jwk.age";
-    mode = "400";
-    owner = "matrix-appservice-irc";
-  };
-
  pub-solar-os.matrix = {
    enable = true;
-    appservice-irc.mediaproxy.signingKeyPath =
-      config.age.secrets."matrix-appservice-irc-mediaproxy-signing-key".path;
    synapse = {
+      sliding-sync.enable = false;
      signing_key_path = config.age.secrets."matrix-synapse-signing-key".path;
      extra-config-files = [
        config.age.secrets."matrix-synapse-secret-config.yaml".path
--- a/hosts/nachtigall/default.nix
+++ b/hosts/nachtigall/default.nix
@ -9,10 +9,12 @@
    ./networking.nix
    ./wireguard.nix
    ./backups.nix
-    "${flake.inputs.fork}/nixos/modules/services/matrix/matrix-authentication-service.nix"
+    "${flake.inputs.fork}/nixos/modules/services//matrix/matrix-authentication-service.nix"
+    "${flake.inputs.unstable}/nixos/modules/services/web-apps/mastodon.nix"
  ];

  disabledModules = [
-    "services/matrix/matrix-authentication-service.nix"
+    "services/matrix/matrix-authentication-service.nix "
+    "services/web-apps/mastodon.nix"
  ];
 }
--- a/hosts/nachtigall/wireguard.nix
+++ b/hosts/nachtigall/wireguard.nix
@ -18,7 +18,7 @@
        "fd00:fae:fae:fae:fae:1::/96"
      ];
      privateKeyFile = config.age.secrets.wg-private-key.path;
-      peers = flake.self.logins.wireguardDevices ++ [
+      peers = flake.self.logins.admins.wireguardDevices ++ [
        {
          # tankstelle.pub.solar
          endpoint = "80.244.242.5:51820";
--- a/hosts/tankstelle/configuration.nix
+++ b/hosts/tankstelle/configuration.nix
@ -10,9 +10,6 @@
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

-  # kernel same-page merging
-  hardware.ksm.enable = true;
-
  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

  system.stateVersion = "23.11";
--- a/hosts/tankstelle/wireguard.nix
+++ b/hosts/tankstelle/wireguard.nix
@ -18,7 +18,7 @@
        "fd00:fae:fae:fae:fae:4::/96"
      ];
      privateKeyFile = config.age.secrets.wg-private-key.path;
-      peers = flake.self.logins.wireguardDevices ++ [
+      peers = flake.self.logins.admins.wireguardDevices ++ [
        {
          # nachtigall.pub.solar
          endpoint = "138.201.80.102:51820";
--- a/hosts/trinkgenossin/wireguard.nix
+++ b/hosts/trinkgenossin/wireguard.nix
@ -22,7 +22,7 @@ in
        "${wireguardIPv6}/96"
      ];
      privateKeyFile = config.age.secrets.wg-private-key.path;
-      peers = flake.self.logins.wireguardDevices ++ [
+      peers = flake.self.logins.admins.wireguardDevices ++ [
        {
          # nachtigall.pub.solar
          endpoint = "138.201.80.102:51820";
--- a/hosts/underground/configuration.nix
+++ b/hosts/underground/configuration.nix
@ -42,17 +42,8 @@
    owner = "matrix-authentication-service";
  };

-  # matrix-appservice-irc
-  age.secrets."matrix-appservice-irc-mediaproxy-signing-key" = {
-    file = "${flake.self}/secrets/staging-matrix-appservice-irc-mediaproxy-signing-key.jwk.age";
-    mode = "400";
-    owner = "matrix-appservice-irc";
-  };
-
  pub-solar-os.matrix = {
    enable = true;
-    appservice-irc.mediaproxy.signingKeyPath =
-      config.age.secrets."matrix-appservice-irc-mediaproxy-signing-key".path;
    synapse = {
      extra-config-files = [
        config.age.secrets."staging-matrix-synapse-secret-config.yaml".path
--- a/hosts/underground/default.nix
+++ b/hosts/underground/default.nix
@ -7,10 +7,10 @@
    ./configuration.nix

    ./networking.nix
-    "${flake.inputs.fork}/nixos/modules/services/matrix/matrix-authentication-service.nix"
+    "${flake.inputs.fork}/nixos/modules/services//matrix/matrix-authentication-service.nix"
  ];

  disabledModules = [
-    "services/matrix/matrix-authentication-service.nix"
+    "services/matrix/matrix-authentication-service.nix "
  ];
 }
--- a/logins/admins.nix
+++ b/logins/admins.nix
@ -38,22 +38,6 @@
          "fd00:fae:fae:fae:fae:200::/96"
        ];
      }
-      {
-        # chocolatebar
-        publicKey = "AS9w0zDUFLcH6IiF6T1vsyZPWPJ3p5fKsjIsM2AoZz8=";
-        allowedIPs = [
-          "10.7.6.205/32"
-          "fd00:fae:fae:fae:fae:205::/96"
-        ];
-      }
-      {
-        # biolimo
-        publicKey = "gnLq6KikFVVGxLxPW+3ZnreokEKLDoso+cUepPOZsBA=";
-        allowedIPs = [
-          "10.7.6.206/32"
-          "fd00:fae:fae:fae:fae:206::/96"
-        ];
-      }
    ];
  };

--- a/logins/default.nix
+++ b/logins/default.nix
@ -6,16 +6,19 @@ in
 {
  flake = {
    logins = {
-      admins = admins;
-      wireguardDevices = lib.lists.foldl (
-        wireguardDevices: adminConfig:
-        wireguardDevices ++ (if adminConfig ? "wireguardDevices" then adminConfig.wireguardDevices else [ ])
-      ) [ ] (lib.attrsets.attrValues admins);
-      sshPubKeys = lib.lists.foldl (
-        sshPubKeys: adminConfig:
-        sshPubKeys
-        ++ (if adminConfig ? "sshPubKeys" then lib.attrsets.attrValues adminConfig.sshPubKeys else [ ])
-      ) [ ] (lib.attrsets.attrValues admins);
+      admins =
+        lib.lists.foldl
+          (logins: adminConfig: {
+            sshPubKeys = logins.sshPubKeys ++ (lib.attrsets.attrValues adminConfig.sshPubKeys);
+            wireguardDevices =
+              logins.wireguardDevices
+              ++ (if adminConfig ? "wireguardDevices" then adminConfig.wireguardDevices else [ ]);
+          })
+          {
+            sshPubKeys = [ ];
+            wireguardDevices = [ ];
+          }
+          (lib.attrsets.attrValues admins);
      robots.sshPubKeys = lib.attrsets.attrValues robots;
    };
  };
--- a/logins/robots.nix
+++ b/logins/robots.nix
@ -1,8 +1,6 @@
 {
  # Used for restic backups to droppie, a server run by @b12f
-  "root@droppie" =
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZQSephFJU0NMbVbhwvVJ2/m6jcPYo1IsWCsoarqKin root@droppie";
+  "root@droppie" = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZQSephFJU0NMbVbhwvVJ2/m6jcPYo1IsWCsoarqKin root@droppie";

-  "hakkonaut" =
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGP5MvCwNRtCcP1pSDrn0XZTNlpOqYnjHDm9/OI4hECW hakkonaut";
+  "hakkonaut" = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGP5MvCwNRtCcP1pSDrn0XZTNlpOqYnjHDm9/OI4hECW hakkonaut";
 }
--- a/modules/backups/default.nix
+++ b/modules/backups/default.nix
@ -283,10 +283,8 @@ in

    # Used for pub-solar-os.backups.repos.storagebox
    programs.ssh.knownHosts = {
-      "u377325.your-storagebox.de".publicKey =
-        "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5EB5p/5Hp3hGW1oHok+PIOH9Pbn7cnUiGmUEBrCVjnAw+HrKyN8bYVV0dIGllswYXwkG/+bgiBlE6IVIBAq+JwVWu1Sss3KarHY3OvFJUXZoZyRRg/Gc/+LRCE7lyKpwWQ70dbelGRyyJFH36eNv6ySXoUYtGkwlU5IVaHPApOxe4LHPZa/qhSRbPo2hwoh0orCtgejRebNtW5nlx00DNFgsvn8Svz2cIYLxsPVzKgUxs8Zxsxgn+Q/UvR7uq4AbAhyBMLxv7DjJ1pc7PJocuTno2Rw9uMZi1gkjbnmiOh6TTXIEWbnroyIhwc8555uto9melEUmWNQ+C+PwAK+MPw==";
-      "[u377325.your-storagebox.de]:23".publicKey =
-        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs";
+      "u377325.your-storagebox.de".publicKey = "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5EB5p/5Hp3hGW1oHok+PIOH9Pbn7cnUiGmUEBrCVjnAw+HrKyN8bYVV0dIGllswYXwkG/+bgiBlE6IVIBAq+JwVWu1Sss3KarHY3OvFJUXZoZyRRg/Gc/+LRCE7lyKpwWQ70dbelGRyyJFH36eNv6ySXoUYtGkwlU5IVaHPApOxe4LHPZa/qhSRbPo2hwoh0orCtgejRebNtW5nlx00DNFgsvn8Svz2cIYLxsPVzKgUxs8Zxsxgn+Q/UvR7uq4AbAhyBMLxv7DjJ1pc7PJocuTno2Rw9uMZi1gkjbnmiOh6TTXIEWbnroyIhwc8555uto9melEUmWNQ+C+PwAK+MPw==";
+      "[u377325.your-storagebox.de]:23".publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs";
    };
  };
 }
--- a/modules/core/default.nix
+++ b/modules/core/default.nix
@ -54,5 +54,9 @@
    };

    time.timeZone = "Etc/UTC";
+
+    home-manager.users.${config.pub-solar-os.authentication.username} = {
+      home.stateVersion = "23.05";
+    };
  };
 }
--- a/modules/core/nix.nix
+++ b/modules/core/nix.nix
@ -11,17 +11,6 @@
    permittedInsecurePackages = [ "olm-3.2.16" ];
  };

-  system.activationScripts.diff-closures = {
-    text = ''
-      if [[ -e /run/current-system ]]; then
-        ${config.nix.package}/bin/nix store diff-closures \
-          /run/current-system "$systemConfig" \
-          --extra-experimental-features nix-command
-      fi
-    '';
-    supportsDryActivation = true;
-  };
-
  nix = {
    # Use default version alias for nix package
    package = pkgs.nix;
--- a/modules/core/terminal-tooling.nix
+++ b/modules/core/terminal-tooling.nix
@ -1,33 +1,19 @@
-{ flake, lib, ... }:
+{ flake, config, ... }:
 {
-  home-manager.users = (
-    lib.attrsets.foldlAttrs (
-      acc: name: value:
-      acc
-      // {
-        ${name} = {
-          programs.git.enable = true;
-          programs.starship.enable = true;
-          programs.bash = {
-            enable = true;
-            historyControl = [
-              "ignoredups"
-              "ignorespace"
-            ];
-          };
-          programs.neovim = {
-            enable = true;
-            vimAlias = true;
-            viAlias = true;
-            defaultEditor = true;
-            # configure = {
-            #   packages.myVimPackages = with pkgs.vimPlugins; {
-            #     start = [vim-nix vim-surrund rainbow];
-            #   };
-            # };
-          };
-        };
-      }
-    ) { } flake.self.logins.admins
-  );
+  home-manager.users.${config.pub-solar-os.authentication.username} = {
+    programs.git.enable = true;
+    programs.starship.enable = true;
+    programs.bash.enable = true;
+    programs.neovim = {
+      enable = true;
+      vimAlias = true;
+      viAlias = true;
+      defaultEditor = true;
+      # configure = {
+      #   packages.myVimPackages = with pkgs.vimPlugins; {
+      #     start = [vim-nix vim-surrund rainbow];
+      #   };
+      # };
+    };
+  };
 }
--- a/modules/core/users.nix
+++ b/modules/core/users.nix
@ -11,6 +11,18 @@
      inherit (lib) mkOption types;
    in
    {
+      username = mkOption {
+        description = "Username for the adminstrative user";
+        type = types.str;
+        default = flake.self.username;
+      };
+
+      sshPubKeys = mkOption {
+        description = "SSH Keys that should have administrative root access";
+        type = types.listOf types.str;
+        default = flake.self.logins.admins.sshPubKeys;
+      };
+
      root.initialHashedPassword = mkOption {
        description = "Hashed password of the root account";
        type = types.str;
@ -31,60 +43,36 @@
    };

  config = {
-    users.users =
-      (lib.attrsets.foldlAttrs (
-        acc: name: value:
-        acc
-        // {
-          ${name} = {
-            name = name;
-            group = name;
-            extraGroups = [
-              "wheel"
-              "docker"
-            ];
-            isNormalUser = true;
-            openssh.authorizedKeys.keys = lib.attrsets.attrValues value.sshPubKeys;
-          };
-        }
-      ) { } flake.self.logins.admins)
-      // {
-        # TODO: Remove when we stop locking ourselves out.
-        root.openssh.authorizedKeys.keys = flake.self.logins.sshPubKeys;
-        root.initialHashedPassword = config.pub-solar-os.authentication.root.initialHashedPassword;
+    users.users.${config.pub-solar-os.authentication.username} = {
+      name = config.pub-solar-os.authentication.username;
+      group = config.pub-solar-os.authentication.username;
+      extraGroups = [
+        "wheel"
+        "docker"
+      ];
+      isNormalUser = true;
+      openssh.authorizedKeys.keys = config.pub-solar-os.authentication.sshPubKeys;
+    };
+    users.groups.${config.pub-solar-os.authentication.username} = { };

-        ${config.pub-solar-os.authentication.robot.username} = {
-          description = "CI and automation user";
-          home = "/home/${config.pub-solar-os.authentication.robot.username}";
-          createHome = true;
-          useDefaultShell = true;
-          uid = 998;
-          group = "${config.pub-solar-os.authentication.robot.username}";
-          isSystemUser = true;
-          openssh.authorizedKeys.keys = config.pub-solar-os.authentication.robot.sshPubKeys;
-        };
-      };
+    # TODO: Remove when we stop locking ourselves out.
+    users.users.root.openssh.authorizedKeys.keys = config.pub-solar-os.authentication.sshPubKeys;

-    home-manager.users = (
-      lib.attrsets.foldlAttrs (
-        acc: name: value:
-        acc
-        // {
-          ${name} = {
-            home.stateVersion = "23.05";
-          };
-        }
-      ) { } flake.self.logins.admins
-    );
+    users.users.${config.pub-solar-os.authentication.robot.username} = {
+      description = "CI and automation user";
+      home = "/home/${config.pub-solar-os.authentication.robot.username}";
+      createHome = true;
+      useDefaultShell = true;
+      uid = 998;
+      group = "${config.pub-solar-os.authentication.robot.username}";
+      isSystemUser = true;
+      openssh.authorizedKeys.keys = config.pub-solar-os.authentication.robot.sshPubKeys;
+    };

-    users.groups =
-      (lib.attrsets.foldlAttrs (
-        acc: name: value:
-        acc // { "${name}" = { }; }
-      ) { } flake.self.logins.admins)
-      // {
-        ${config.pub-solar-os.authentication.robot.username} = { };
-      };
+    users.groups.${config.pub-solar-os.authentication.robot.username} = { };
+
+    users.users.root.initialHashedPassword =
+      config.pub-solar-os.authentication.root.initialHashedPassword;

    security.sudo.wheelNeedsPassword = false;
  };
--- a/modules/coturn/default.nix
+++ b/modules/coturn/default.nix
@ -5,8 +5,9 @@
  ...
 }:
 {
-  age.secrets."coturn-static-auth-secret" = {
-    file = "${flake.self}/secrets/coturn-static-auth-secret.age";
+  age.secrets."nachtigall-coturn-static-auth-secret" = {
+    file = "${flake.self}/secrets/nachtigall-coturn-static-auth-secret.age";
+    path = "/run/agenix/coturn-static-auth-secret";
    mode = "400";
    owner = "turnserver";
  };
@ -18,7 +19,7 @@
    min-port = 49000;
    max-port = 50000;
    use-auth-secret = true;
-    static-auth-secret-file = config.age.secrets."coturn-static-auth-secret".path;
+    static-auth-secret-file = config.age.secrets."nachtigall-coturn-static-auth-secret".path;
    realm = "turn.${config.pub-solar-os.networking.domain}";
    cert = "${config.security.acme.certs.${realm}.directory}/full.pem";
    pkey = "${config.security.acme.certs.${realm}.directory}/key.pem";
--- a/modules/forgejo/default.nix
+++ b/modules/forgejo/default.nix
@ -65,7 +65,6 @@

  services.forgejo = {
    enable = true;
-    package = pkgs.forgejo;
    user = "gitea";
    group = "gitea";
    database = {
@ -76,7 +75,7 @@
    };
    stateDir = "/var/lib/forgejo";
    lfs.enable = true;
-    secrets.mailer.PASSWD = config.age.secrets.forgejo-mailer-password.path;
+    mailerPasswordFile = config.age.secrets.forgejo-mailer-password.path;
    settings = {
      DEFAULT.APP_NAME = "pub.solar git server";

@ -142,12 +141,6 @@
        LOGIN_REMEMBER_DAYS = 365;
      };

-      # See https://docs.gitea.com/administration/config-cheat-sheet#migrations-migrations
-      migrations = {
-        # This allows migrations from the same forgejo instance
-        ALLOW_LOCALNETWORKS = true;
-      };
-
      # https://forgejo.org/docs/next/admin/config-cheat-sheet/#indexer-indexer
      indexer = {
        REPO_INDEXER_ENABLED = true;
--- a/modules/grafana/default.nix
+++ b/modules/grafana/default.nix
@ -28,6 +28,11 @@
      group = "grafana";
      user = "grafana";
    };
+    "grafana-dashboards/blackbox-exporter_rev3.json" = {
+      source = ./grafana-dashboards/blackbox-exporter_rev3.json;
+      group = "grafana";
+      user = "grafana";
+    };
    "grafana-dashboards/synapse.json" = {
      source = ./grafana-dashboards/synapse.json;
      group = "grafana";
@ -43,8 +48,7 @@
  services.nginx.virtualHosts."grafana.${config.pub-solar-os.networking.domain}" = {
    enableACME = true;
    forceSSL = true;
-    locations."/".proxyPass =
-      "http://127.0.0.1:${toString config.services.grafana.settings.server.http_port}";
+    locations."/".proxyPass = "http://127.0.0.1:${toString config.services.grafana.settings.server.http_port}";
  };

  services.grafana = {
--- a/modules/grafana/grafana-dashboards/blackbox-exporter_rev3.json
+++ b/modules/grafana/grafana-dashboards/blackbox-exporter_rev3.json
--- a/modules/keycloak/default.nix
+++ b/modules/keycloak/default.nix
@ -50,8 +50,7 @@
        hostname = "auth.${config.pub-solar-os.networking.domain}";
        http-host = "127.0.0.1";
        http-port = 8080;
-        proxy-headers = "xforwarded";
-        http-enabled = true;
+        proxy = "edge";
      };
      themes = {
        "pub.solar" =
--- a/modules/matrix-irc/default.nix
+++ b/modules/matrix-irc/default.nix
@ -16,128 +16,110 @@ let
  synapseClientPort = "${toString listenerWithClient.port}";
 in
 {
-  options.pub-solar-os = {
-    matrix.appservice-irc.mediaproxy = {
-      signingKeyPath = lib.mkOption {
-        description = "Path to file containing the IRC appservice mediaproxy signing key";
-        type = lib.types.str;
-        default = "/var/lib/matrix-appservice-irc/media-signingkey.jwk";
+  services.matrix-appservice-irc = {
+    enable = true;
+    localpart = "irc_bot";
+    port = 8010;
+    registrationUrl = "http://localhost:8010";
+    settings = {
+      homeserver = {
+        domain = "${config.pub-solar-os.networking.domain}";
+        url = "http://127.0.0.1:${synapseClientPort}";
+        media_url = "https://matrix.${config.pub-solar-os.networking.domain}";
+        enablePresence = false;
      };
-    };
-  };
-  config = {
-    services.matrix-appservice-irc = {
-      enable = true;
-      localpart = "irc_bot";
-      port = 8010;
-      registrationUrl = "http://localhost:8010";
-      settings = {
-        homeserver = {
-          domain = "${config.pub-solar-os.networking.domain}";
-          url = "http://127.0.0.1:${synapseClientPort}";
-          enablePresence = false;
+      ircService = {
+        ident = {
+          address = "::";
+          enabled = false;
+          port = 1113;
        };
-        ircService = {
-          ident = {
-            address = "::";
-            enabled = false;
-            port = 1113;
-          };
-          logging = {
-            # set to debug for debugging
-            level = "warn";
-            maxFiles = 5;
-            toCosole = true;
-          };
-          matrixHandler = {
-            eventCacheSize = 4096;
-          };
-          mediaProxy = {
-            signingKeyPath = config.pub-solar-os.matrix.appservice-irc.mediaproxy.signingKeyPath;
-            # keep media for 2 weeks
-            ttlSeconds = 1209600;
-            bindPort = 11111;
-            publicUrl = "https:///matrix.${config.pub-solar-os.networking.domain}/media";
-          };
-          metrics = {
-            enabled = true;
-            remoteUserAgeBuckets = [
-              "1h"
-              "1d"
-              "1w"
-            ];
-          };
-          provisioning = {
-            enabled = false;
-            requestTimeoutSeconds = 300;
-          };
-          servers =
-            let
-              commonConfig = {
-                allowExpiredCerts = false;
-                botConfig = {
-                  enabled = false;
-                  joinChannelsIfNoUsers = false;
-                  nick = "MatrixBot";
-                };
-                dynamicChannels = {
-                  createAlias = true;
-                  enabled = true;
-                  federate = true;
-                  joinRule = "public";
-                  published = true;
-                };
-                ircClients = {
-                  allowNickChanges = true;
-                  concurrentReconnectLimit = 50;
-                  idleTimeout = 10800;
-                  lineLimit = 3;
-                  maxClients = 30;
-                  nickTemplate = "$DISPLAY[m]";
-                  reconnectIntervalMs = 5000;
-                };
-                matrixClients = {
-                  joinAttempts = -1;
-                };
-                membershipLists = {
-                  enabled = true;
-                  floodDelayMs = 10000;
-                  global = {
-                    ircToMatrix = {
-                      incremental = true;
-                      initial = true;
-                    };
-                    matrixToIrc = {
-                      incremental = true;
-                      initial = true;
-                    };
+        logging = {
+          level = "debug";
+          maxFiles = 5;
+          toCosole = true;
+        };
+        matrixHandler = {
+          eventCacheSize = 4096;
+        };
+        metrics = {
+          enabled = true;
+          remoteUserAgeBuckets = [
+            "1h"
+            "1d"
+            "1w"
+          ];
+        };
+        provisioning = {
+          enabled = false;
+          requestTimeoutSeconds = 300;
+        };
+        servers =
+          let
+            commonConfig = {
+              allowExpiredCerts = false;
+              botConfig = {
+                enabled = false;
+                joinChannelsIfNoUsers = false;
+                nick = "MatrixBot";
+              };
+              dynamicChannels = {
+                createAlias = true;
+                enabled = true;
+                federate = true;
+                joinRule = "public";
+                published = true;
+              };
+              ircClients = {
+                allowNickChanges = true;
+                concurrentReconnectLimit = 50;
+                idleTimeout = 10800;
+                lineLimit = 3;
+                maxClients = 30;
+                nickTemplate = "$DISPLAY[m]";
+                reconnectIntervalMs = 5000;
+              };
+              matrixClients = {
+                joinAttempts = -1;
+              };
+              membershipLists = {
+                enabled = true;
+                floodDelayMs = 10000;
+                global = {
+                  ircToMatrix = {
+                    incremental = true;
+                    initial = true;
+                  };
+                  matrixToIrc = {
+                    incremental = true;
+                    initial = true;
                  };
                };
-                port = 6697;
-                privateMessages = {
-                  enabled = true;
-                  federate = true;
-                };
-                sasl = false;
-                sendConnectionMessages = true;
-                ssl = true;
              };
-            in
-            {
-              "irc.libera.chat" = lib.attrsets.recursiveUpdate commonConfig {
-                name = "libera";
-                dynamicChannels.groupId = "+libera.chat:localhost";
-                dynamicChannels.aliasTemplate = "#_libera_$CHANNEL";
-                matrixClients.displayName = "$NICK (LIBERA-IRC)";
-              };
-              "irc.scratch-network.net" = lib.attrsets.recursiveUpdate commonConfig {
-                name = "scratch";
-                matrixClients.displayName = "$NICK (SCRATCH-IRC)";
-                dynamicChannels.aliasTemplate = "#_scratch_$CHANNEL";
-                dynamicChannels.groupId = "+scratch-network.net:localhost";
+              port = 6697;
+              privateMessages = {
+                enabled = true;
+                federate = true;
              };
+              sasl = false;
+              sendConnectionMessages = true;
+              ssl = true;
            };
-        };
+          in
+          {
+            "irc.libera.chat" = lib.attrsets.recursiveUpdate commonConfig {
+              name = "libera";
+              dynamicChannels.groupId = "+libera.chat:localhost";
+              dynamicChannels.aliasTemplate = "#_libera_$CHANNEL";
+              matrixClients.displayName = "$NICK (LIBERA-IRC)";
+            };
+            "irc.scratch-network.net" = lib.attrsets.recursiveUpdate commonConfig {
+              name = "scratch";
+              matrixClients.displayName = "$NICK (SCRATCH-IRC)";
+              dynamicChannels.aliasTemplate = "#_scratch_$CHANNEL";
+              dynamicChannels.groupId = "+scratch-network.net:localhost";
+            };
+          };
      };
    };
  };
--- a/modules/matrix/default.nix
+++ b/modules/matrix/default.nix
@ -32,6 +32,11 @@ in
          type = lib.types.str;
          default = "${config.services.matrix-synapse.dataDir}/homeserver.signing.key";
        };
+
+        sliding-sync.enable = lib.mkEnableOption {
+          description = "Whether to enable a sliding-sync proxy, no longer needed with synapse version 1.114+";
+          default = false;
+        };
      };

      matrix-authentication-service = {
@ -119,17 +124,6 @@ in
        enable_room_list_search = true;
        encryption_enabled_by_default_for_room_type = "off";
        event_cache_size = "100K";
-
-        # https://github.com/element-hq/synapse/issues/11203
-        # No YAML deep-merge, so this needs to be in secret extraConfigFiles
-        # together with msc3861
-        #experimental_features = {
-        #  # Room summary API
-        #  msc3266_enabled = true;
-        #  # Rendezvous server for QR Code generation
-        #  msc4108_enabled = true;
-        #};
-
        federation_rr_transactions_per_room_per_second = 50;
        federation_client_minimum_tls_version = "1.2";
        forget_rooms_on_leave = true;
@ -289,9 +283,7 @@ in
      createDatabase = true;
      extraConfigFiles = config.pub-solar-os.matrix.matrix-authentication-service.extra-config-files;

-      # https://element-hq.github.io/matrix-authentication-service/reference/configuration.html
      settings = {
-        account.email_change_allowed = false;
        http.public_base = "https://mas.${config.pub-solar-os.networking.domain}";
        http.issuer = "https://mas.${config.pub-solar-os.networking.domain}";
        http.listeners = [
@ -334,13 +326,24 @@ in
      };
    };

+    services.matrix-sliding-sync = {
+      enable = config.pub-solar-os.matrix.synapse.sliding-sync.enable;
+      settings = {
+        SYNCV3_SERVER = "https://${publicDomain}";
+        SYNCV3_BINDADDR = "127.0.0.1:8011";
+        # The bind addr for Prometheus metrics, which will be accessible at
+        # /metrics at this address
+        SYNCV3_PROM = "127.0.0.1:9100";
+      };
+      environmentFile = config.age.secrets."matrix-synapse-sliding-sync-secret".path;
+    };
+
    pub-solar-os.backups.restic.matrix-synapse = {
      paths = [
        "/var/lib/matrix-synapse"
        "/var/lib/matrix-appservice-irc"
        "/var/lib/mautrix-telegram"
        "/tmp/matrix-synapse-backup.sql"
-        "/tmp/matrix-authentication-service-backup.sql"
      ];
      timerConfig = {
        OnCalendar = "*-*-* 05:00:00 Etc/UTC";
@ -348,11 +351,9 @@ in
      initialize = true;
      backupPrepareCommand = ''
        ${pkgs.sudo}/bin/sudo -u postgres ${pkgs.postgresql}/bin/pg_dump -d matrix > /tmp/matrix-synapse-backup.sql
-        ${pkgs.sudo}/bin/sudo -u postgres ${pkgs.postgresql}/bin/pg_dump -d matrix-authentication-service > /tmp/matrix-authentication-service-backup.sql
      '';
      backupCleanupCommand = ''
        rm /tmp/matrix-synapse-backup.sql
-        rm /tmp/matrix-authentication-service-backup.sql
      '';
      pruneOpts = [
        "--keep-daily 7"
--- a/modules/mediawiki/default.nix
+++ b/modules/mediawiki/default.nix
@ -139,10 +139,6 @@ let
      // https://www.mediawiki.org/wiki/Extension:PluggableAuth#Configuration
      $wgPluggableAuth_EnableAutoLogin = false;
      $wgPluggableAuth_ButtonLabel = 'Login with pub.solar ID';
-      // Avoid getting logged out after 30 minutes
-      // https://www.mediawiki.org/wiki/Topic:W4be4h6t63vf3y8p
-      // https://www.mediawiki.org/wiki/Manual:$wgRememberMe
-      $wgRememberMe = 'always';

      // https://www.mediawiki.org/wiki/Extension:OpenID_Connect#Keycloak
      $wgPluggableAuth_Config[] = [
@ -215,7 +211,7 @@ in
      backend = "docker";

      containers."mediawiki" = {
-        image = "git.pub.solar/pub-solar/mediawiki-oidc-docker:1.43.0";
+        image = "git.pub.solar/pub-solar/mediawiki-oidc-docker:1.42.1";
        user = "1000:${builtins.toString gid}";
        autoStart = true;

--- a/modules/nextcloud/default.nix
+++ b/modules/nextcloud/default.nix
@ -2,7 +2,6 @@
  config,
  pkgs,
  flake,
-  lib,
  ...
 }:
 {
@ -23,226 +22,118 @@
    forceSSL = true;
  };

-  services.nextcloud =
-    let
-      exiftool_1270 = pkgs.perlPackages.buildPerlPackage rec {
-        # NOTE nextcloud-memories needs this specific version of exiftool
-        pname = "Image-ExifTool";
-        version = "12.70";
-        src = pkgs.fetchFromGitHub {
-          owner = "exiftool";
-          repo = "exiftool";
-          rev = version;
-          hash = "sha256-YMWYPI2SDi3s4KCpSNwovemS5MDj5W9ai0sOkvMa8Zg=";
-        };
-        nativeBuildInputs = lib.optional pkgs.stdenv.hostPlatform.isDarwin pkgs.shortenPerlShebang;
-        postInstall = lib.optionalString pkgs.stdenv.hostPlatform.isDarwin ''
-          shortenPerlShebang $out/bin/exiftool
-        '';
-      };
-    in
-    {
-      hostName = "cloud.${config.pub-solar-os.networking.domain}";
-      home = "/var/lib/nextcloud";
+  services.nextcloud = {
+    hostName = "cloud.${config.pub-solar-os.networking.domain}";
+    home = "/var/lib/nextcloud";

-      enable = true;
-      # When updating package, remember to update nextcloud30Packages in
-      # services.nextcloud.extraApps
-      package = pkgs.nextcloud30;
-      https = true;
-      secretFile = config.age.secrets."nextcloud-secrets".path; # secret
-      maxUploadSize = "1G";
-
-      configureRedis = true;
-
-      notify_push = {
-        enable = true;
-        bendDomainToLocalhost = true;
-      };
-
-      config = {
-        adminuser = "admin";
-        adminpassFile = config.age.secrets."nextcloud-admin-pass".path;
-        dbuser = "nextcloud";
-        dbtype = "pgsql";
-        dbname = "nextcloud";
-      };
-
-      settings = {
-        overwrite.cli.url = "http://cloud.${config.pub-solar-os.networking.domain}";
-        overwriteprotocol = "https";
-
-        installed = true;
-        default_phone_region = "+49";
-        mail_sendmailmode = "smtp";
-        mail_from_address = "nextcloud";
-        mail_smtpmode = "smtp";
-        mail_smtpauthtype = "PLAIN";
-        mail_domain = "pub.solar";
-        mail_smtpname = "admins@pub.solar";
-        mail_smtpsecure = "ssl";
-        mail_smtpauth = true;
-        mail_smtphost = "mail.pub.solar";
-        mail_smtpport = "465";
-
-        # This is to allow connections to collabora and keycloak, among other services
-        # running on the same host
-        #
-        # https://docs.nextcloud.com/server/stable/admin_manual/configuration_server/config_sample_php_parameters.html?highlight=allow_local_remote_servers%20true
-        # https://github.com/ONLYOFFICE/onlyoffice-nextcloud/issues/293
-        allow_local_remote_servers = true;
-
-        enable_previews = true;
-        jpeg_quality = 60;
-        enabledPreviewProviders = [
-          "OC\\Preview\\PNG"
-          "OC\\Preview\\JPEG"
-          "OC\\Preview\\GIF"
-          "OC\\Preview\\BMP"
-          "OC\\Preview\\HEIC"
-          "OC\\Preview\\TIFF"
-          "OC\\Preview\\XBitmap"
-          "OC\\Preview\\SVG"
-          "OC\\Preview\\WebP"
-          "OC\\Preview\\Font"
-          "OC\\Preview\\Movie"
-          "OC\\Preview\\ImaginaryPDF"
-          "OC\\Preview\\MP3"
-          "OC\\Preview\\OpenDocument"
-          "OC\\Preview\\Krita"
-          "OC\\Preview\\TXT"
-          "OC\\Preview\\MarkDown"
-          "OC\\Preview\\Imaginary"
-        ];
-        preview_imaginary_url = "http://127.0.0.1:${toString config.services.imaginary.port}/";
-        preview_max_filesize_image = 128; # MB
-        preview_max_memory = 512; # MB
-        preview_max_x = 2048; # px
-        preview_max_y = 2048; # px
-        preview_max_scale_factor = 1;
-        "preview_ffmpeg_path" = lib.getExe pkgs.ffmpeg-headless;
-
-        "memories.exiftool_no_local" = false;
-        "memories.exiftool" = "${exiftool_1270}/bin/exiftool";
-        "memories.vod.ffmpeg" = lib.getExe pkgs.ffmpeg;
-        "memories.vod.ffprobe" = lib.getExe' pkgs.ffmpeg-headless "ffprobe";
-
-        auth.bruteforce.protection.enabled = true;
-        trashbin_retention_obligation = "auto,7";
-        skeletondirectory = "./nextcloud-skeleton";
-        defaultapp = "file";
-        activity_expire_days = "14";
-        integrity.check.disabled = false;
-        updater.release.channel = "stable";
-        loglevel = 2;
-        debug = false;
-        maintenance_window_start = "1";
-        # maintenance = false;
-        app_install_overwrite = [
-          "pdfdraw"
-          "integration_whiteboard"
-        ];
-        htaccess.RewriteBase = "/";
-        theme = "";
-        simpleSignUpLink.shown = false;
-      };
-
-      phpOptions = {
-        "opcache.interned_strings_buffer" = "32";
-        "opcache.max_accelerated_files" = "16229";
-        "opcache.memory_consumption" = "256";
-        # https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#enable-php-opcache
-        "opcache.revalidate_freq" = "60";
-        # https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#:~:text=opcache.jit%20%3D%201255%20opcache.jit_buffer_size%20%3D%20128m
-        "opcache.jit" = "1255";
-        "opcache.jit_buffer_size" = "128M";
-      };
-
-      # Calculated with 4GiB RAM, 80MiB process size available on
-      # https://spot13.com/pmcalculator/
-      poolSettings = {
-        pm = "dynamic";
-        "pm.max_children" = "52";
-        "pm.max_requests" = "500";
-        "pm.max_spare_servers" = "39";
-        "pm.min_spare_servers" = "13";
-        "pm.start_servers" = "13";
-      };
-
-      caching.redis = true;
-      appstoreEnable = true;
-      autoUpdateApps.enable = true;
-      extraApps = {
-        inherit (pkgs.nextcloud30Packages.apps) memories previewgenerator recognize;
-      };
-      database.createLocally = true;
-    };
-
-  # https://docs.nextcloud.com/server/30/admin_manual/installation/server_tuning.html#previews
-  services.imaginary = {
    enable = true;
-    address = "127.0.0.1";
-    settings.return-size = true;
-  };
+    package = pkgs.nextcloud29;
+    https = true;
+    secretFile = config.age.secrets."nextcloud-secrets".path; # secret
+    maxUploadSize = "1G";

-  systemd = {
-    services =
-      let
-        occ = "/run/current-system/sw/bin/nextcloud-occ";
-      in
-      {
-        nextcloud-cron-preview-generator = {
-          environment.NEXTCLOUD_CONFIG_DIR = "${config.services.nextcloud.home}/config";
-          serviceConfig = {
-            ExecStart = "${occ} preview:pre-generate";
-            Type = "oneshot";
-            User = "nextcloud";
-          };
-        };
+    configureRedis = true;

-        nextcloud-preview-generator-setup = {
-          wantedBy = [ "multi-user.target" ];
-          requires = [ "phpfpm-nextcloud.service" ];
-          after = [ "phpfpm-nextcloud.service" ];
-          environment.NEXTCLOUD_CONFIG_DIR = "${config.services.nextcloud.home}/config";
-          script = # bash
-            ''
-              # check with:
-              # for size in squareSizes widthSizes heightSizes; do echo -n "$size: "; nextcloud-occ config:app:get previewgenerator $size; done
-
-              # extra commands run for preview generator:
-              # 32   icon file list
-              # 64   icon file list android app, photos app
-              # 96   nextcloud client VFS windows file preview
-              # 256  file app grid view, many requests
-              # 512  photos app tags
-              ${occ} config:app:set --value="32 64 96 256 512" previewgenerator squareSizes
-
-              # 341 hover in maps app
-              # 1920 files/photos app when viewing picture
-              ${occ} config:app:set --value="341 1920" previewgenerator widthSizes
-
-              # 256 hover in maps app
-              # 1080 files/photos app when viewing picture
-              ${occ} config:app:set --value="256 1080" previewgenerator heightSizes
-            '';
-          serviceConfig = {
-            Type = "oneshot";
-            User = "nextcloud";
-          };
-        };
-      };
-    timers.nextcloud-cron-preview-generator = {
-      after = [ "nextcloud-setup.service" ];
-      timerConfig = {
-        OnCalendar = "*:0/10";
-        OnUnitActiveSec = "9m";
-        Persistent = true;
-        RandomizedDelaySec = 60;
-        Unit = "nextcloud-cron-preview-generator.service";
-      };
-      wantedBy = [ "timers.target" ];
+    notify_push = {
+      enable = true;
+      bendDomainToLocalhost = true;
    };
+
+    config = {
+      adminuser = "admin";
+      adminpassFile = config.age.secrets."nextcloud-admin-pass".path;
+      dbuser = "nextcloud";
+      dbtype = "pgsql";
+      dbname = "nextcloud";
+      dbtableprefix = "oc_";
+    };
+
+    settings = {
+      overwrite.cli.url = "http://cloud.${config.pub-solar-os.networking.domain}";
+      overwriteprotocol = "https";
+
+      installed = true;
+      default_phone_region = "+49";
+      mail_sendmailmode = "smtp";
+      mail_from_address = "nextcloud";
+      mail_smtpmode = "smtp";
+      mail_smtpauthtype = "PLAIN";
+      mail_domain = "pub.solar";
+      mail_smtpname = "admins@pub.solar";
+      mail_smtpsecure = "ssl";
+      mail_smtpauth = true;
+      mail_smtphost = "mail.pub.solar";
+      mail_smtpport = "465";
+
+      # This is to allow connections to collabora and keycloak, among other services
+      # running on the same host
+      #
+      # https://docs.nextcloud.com/server/stable/admin_manual/configuration_server/config_sample_php_parameters.html?highlight=allow_local_remote_servers%20true
+      # https://github.com/ONLYOFFICE/onlyoffice-nextcloud/issues/293
+      allow_local_remote_servers = true;
+
+      enable_previews = true;
+      enabledPreviewProviders = [
+        "OC\\Preview\\PNG"
+        "OC\\Preview\\JPEG"
+        "OC\\Preview\\GIF"
+        "OC\\Preview\\BMP"
+        "OC\\Preview\\XBitmap"
+        "OC\\Preview\\Movie"
+        "OC\\Preview\\PDF"
+        "OC\\Preview\\MP3"
+        "OC\\Preview\\TXT"
+        "OC\\Preview\\MarkDown"
+      ];
+      preview_max_x = "1024";
+      preview_max_y = "768";
+      preview_max_scale_factor = "1";
+
+      auth.bruteforce.protection.enabled = true;
+      trashbin_retention_obligation = "auto,7";
+      skeletondirectory = "./nextcloud-skeleton";
+      defaultapp = "file";
+      activity_expire_days = "14";
+      integrity.check.disabled = false;
+      updater.release.channel = "stable";
+      loglevel = 2;
+      debug = false;
+      maintenance_window_start = "1";
+      # maintenance = false;
+      app_install_overwrite = [
+        "pdfdraw"
+        "integration_whiteboard"
+      ];
+      htaccess.RewriteBase = "/";
+      theme = "";
+      simpleSignUpLink.shown = false;
+    };
+
+    phpOptions = {
+      "opcache.interned_strings_buffer" = "32";
+      "opcache.max_accelerated_files" = "16229";
+      "opcache.memory_consumption" = "256";
+      # https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#enable-php-opcache
+      "opcache.revalidate_freq" = "60";
+      # https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#:~:text=opcache.jit%20%3D%201255%20opcache.jit_buffer_size%20%3D%20128m
+      "opcache.jit" = "1255";
+      "opcache.jit_buffer_size" = "128M";
+    };
+
+    # Calculated with 4GiB RAM, 80MiB process size available on
+    # https://spot13.com/pmcalculator/
+    poolSettings = {
+      pm = "dynamic";
+      "pm.max_children" = "52";
+      "pm.max_requests" = "500";
+      "pm.max_spare_servers" = "39";
+      "pm.min_spare_servers" = "13";
+      "pm.start_servers" = "13";
+    };
+
+    caching.redis = true;
+    autoUpdateApps.enable = true;
+    database.createLocally = true;
  };

  services.restic.backups.nextcloud-storagebox = {
--- a/modules/nginx-matrix/default.nix
+++ b/modules/nginx-matrix/default.nix
@ -24,6 +24,14 @@ let
      secure_backup_required = false;
      secure_backup_setup_methods = [ ];
    };
+    "m.integrations" = {
+      managers = [
+        {
+          api_url = "https://dimension.${domain}/api/v1/scalar";
+          ui_url = "https://dimension.${domain}/element";
+        }
+      ];
+    };
  };
  wellKnownServer = domain: { "m.server" = "matrix.${domain}:8448"; };
  wellKnownSupport = {
@ -120,13 +128,6 @@ in
          extraConfig = commonHeaders;
        };

-        # For IRC appservice media proxy
-        "/media" = {
-          priority = 100;
-          proxyPass = "http://127.0.0.1:${toString (config.services.matrix-appservice-irc.settings.ircService.mediaProxy.bindPort)}";
-          extraConfig = commonHeaders;
-        };
-
        # Forward to the auth service
        "~ ^/_matrix/client/(.*)/(login|logout|refresh)" = {
          priority = 100;
--- a/modules/nginx-matrix/element-client-config.nix
+++ b/modules/nginx-matrix/element-client-config.nix
@ -50,15 +50,4 @@
    # FUTUREWORK: Replace with pub.solar logo
    auth_header_logo_url = "themes/element/img/logos/element-logo.svg";
  };
-  # Enable Element Call Beta
-  features = {
-    feature_video_rooms = true;
-    feature_group_calls = true;
-    feature_element_call_video_rooms = true;
-  };
-  element_call = {
-    url = "https://call.element.io";
-    participant_limit = 50;
-    brand = "Element Call";
-  };
 }
--- a/modules/prometheus/alert-rules.nix
+++ b/modules/prometheus/alert-rules.nix
@ -142,8 +142,8 @@ lib.mapAttrsToList

    cpu_using_90percent = {
      condition = ''100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) >= 90'';
-      time = "20m";
-      description = "{{$labels.instance}} is running with cpu usage > 90% for at least 20 minutes: {{$value}}";
+      time = "10m";
+      description = "{{$labels.instance}} is running with cpu usage > 90% for at least 10 minutes: {{$value}}";
    };

    reboot = {
@ -198,10 +198,10 @@ lib.mapAttrsToList
      description = "{{$labels.instance}}: healtcheck {{$labels.job}} fails!";
      };
    */
-    #cert_expiry = {
-    #  condition = "(probe_ssl_earliest_cert_expiry - time())/(3600*24) < 30";
-    #  description = "{{$labels.instance}}: The TLS certificate will expire in less than 30 days: {{$value}}s";
-    #};
+    cert_expiry = {
+      condition = "(probe_ssl_earliest_cert_expiry - time())/(3600*24) < 21";
+      description = "{{$labels.instance}}: The TLS certificate will expire in less than 21 days: {{$value}}s";
+    };

    # ignore devices that disabled S.M.A.R.T (example if attached via USB)

@ -234,10 +234,10 @@ lib.mapAttrsToList
      };
    */

-    #host_memory_under_memory_pressure = {
-    #  condition = "rate(node_vmstat_pgmajfault[1m]) > 1000";
-    #  description = "{{$labels.instance}}: The node is under heavy memory pressure. High rate of major page faults: {{$value}}";
-    #};
+    host_memory_under_memory_pressure = {
+      condition = "rate(node_vmstat_pgmajfault[1m]) > 1000";
+      description = "{{$labels.instance}}: The node is under heavy memory pressure. High rate of major page faults: {{$value}}";
+    };

    # ext4_errors = {
    #   condition = "ext4_errors_value > 0";
--- a/modules/prometheus/default.nix
+++ b/modules/prometheus/default.nix
@ -5,6 +5,22 @@
  flake,
  ...
 }:
+let
+  blackboxTargets = [
+    "https://pub.solar"
+    "https://chat.pub.solar"
+    "https://cloud.pub.solar"
+    "https://collabora.pub.solar"
+    "https://git.pub.solar"
+    "https://grafana.pub.solar"
+    "https://list.pub.solar"
+    "https://mastodon.pub.solar"
+    "https://obs-portal.pub.solar"
+    "https://stream.pub.solar"
+    "https://wiki.pub.solar"
+    "https://www.pub.solar"
+  ];
+in
 {
  age.secrets.alertmanager-envfile = {
    file = "${flake.self}/secrets/alertmanager-envfile.age";
@ -39,6 +55,32 @@
    enable = true;
    port = 9001;
    exporters = {
+      blackbox = {
+        enable = true;
+        # Default port is 9115
+        openFirewall = false;
+
+        configFile = pkgs.writeTextFile {
+          name = "blackbox-exporter-config";
+          text = ''
+            modules:
+              http_2xx:
+                prober: http
+                timeout: 5s
+                http:
+                  valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
+                  valid_status_codes: []  # Defaults to 2xx
+                  method: GET
+                  no_follow_redirects: false
+                  fail_if_ssl: false
+                  fail_if_not_ssl: false
+                  tls_config:
+                    insecure_skip_verify: false
+                  preferred_ip_protocol: "ip4" # defaults to "ip6"
+                  ip_protocol_fallback: true  # fallback to "ip6"
+          '';
+        };
+      };
      node = {
        enable = true;
        enabledCollectors = [ "systemd" ];
@ -50,6 +92,30 @@
      scrape_timeout = "9s";
    };
    scrapeConfigs = [
+      {
+        job_name = "blackbox";
+        scrape_interval = "60m";
+        metrics_path = "/probe";
+        params = {
+          module = [ "http_2xx" ];
+        };
+        static_configs = [ { targets = blackboxTargets; } ];
+
+        relabel_configs = [
+          {
+            source_labels = [ "__address__" ];
+            target_label = "__param_target";
+          }
+          {
+            source_labels = [ "__param_target" ];
+            target_label = "instance";
+          }
+          {
+            target_label = "__address__";
+            replacement = "127.0.0.1:9115"; # The blackbox exporter's real hostname:port.
+          }
+        ];
+      }
      {
        job_name = "node-exporter";
        static_configs = [
--- a/modules/unlock-luks-on-boot/default.nix
+++ b/modules/unlock-luks-on-boot/default.nix
@ -10,7 +10,7 @@

      # Please create this manually the first time.
      hostKeys = [ "/etc/secrets/initrd/ssh_host_ed25519_key" ];
-      authorizedKeys = flake.self.logins.sshPubKeys;
+      authorizedKeys = config.pub-solar-os.authentication.sshPubKeys;
    };
    postCommands = ''
      # Automatically ask for the password on SSH login
--- a/modules/unlock-zfs-on-boot/default.nix
+++ b/modules/unlock-zfs-on-boot/default.nix
@ -11,7 +11,7 @@

      # Please create this manually the first time.
      hostKeys = [ "/etc/secrets/initrd/ssh_host_ed25519_key" ];
-      authorizedKeys = flake.self.logins.sshPubKeys;
+      authorizedKeys = config.pub-solar-os.authentication.sshPubKeys;
    };
    # this will automatically load the zfs password prompt on login
    # and kill the other prompt so boot can continue
--- a/overlays/default.nix
+++ b/overlays/default.nix
@ -1,7 +1,7 @@
-{ inputs, ... }:
+{ self, inputs, ... }:
 {
  flake = {
-    nixosModules = {
+    nixosModules = rec {
      overlays = (
        { ... }:
        {
@ -12,11 +12,12 @@
                unstable = import inputs.unstable { system = prev.system; };
              in
              {
-                matrix-authentication-service = unstable.matrix-authentication-service;
                element-themes = prev.callPackage ./pkgs/element-themes { inherit (inputs) element-themes; };
                element-stickerpicker = prev.callPackage ./pkgs/element-stickerpicker {
                  inherit (inputs) element-stickers maunium-stickerpicker;
                };
+                mastodon = unstable.mastodon;
+                matrix-authentication-service = unstable.matrix-authentication-service;
              }
            )
          ];
--- a/secrets/drone-db-secrets.age
+++ b/secrets/drone-db-secrets.age
@ -0,0 +1,43 @@
+age-encryption.org/v1
+-> ssh-ed25519 Y0ZZaw 5slOxDM4xGALMpYxFm1WBX4Sds7itgPBMIiY97d7Egk
+mZNzn4I6obUHAdox6eVR4H82EZagZ1IrCcq0CDtK44Y
+-> ssh-ed25519 uYcDNw w5lzhmA8wIMXihKF25d5jx4/Cc5BFE3Lw6ad60b0wBg
+v9z03cpts6oVlcTQ48hMw8rjWHp1JUOov2qCUjFN4bs
+-> ssh-rsa f5THog
+A93Usdjf6yJFLFqDiy6UUIJ4faBgQXIvk0pZlABlj9M5n7fSf9uzR6sSih4HNCvm
+sMkZ4wKyQHJnUB4Uc2jGrdcWqpmP5MLYHhj74Rxsi6heZuCRf94KH7sE/03A958w
+jAV4v9z4EqmkvWLNQi/hxMVMs5A61Vs63WIX/TA6vhL8Yrn0FeIKlRZYUVIeMu42
+pbEuLWeIzbUioAuEA1ZKV5VDx+6ack8TS/Dj5bTNEnzFWpjnHjO0/GeQU+aaQZTe
+Zy325TcRosT0V7PIh0tDQZKKRpOH/e9LnDkd8NIjyfEsGdDYaP1EVOYVxPCqUDAh
+A0kV1kkTiBzaXDkuakc+HDCIxtYXLWthsmbD+vI3D7FlTl0CY4fOP0wwO/0rS5Yp
+KDuxjz89II1H4+ZvlcPUihyW7OEj4d+NwFQy+7Qq0Y9Ii0NONXNsnx17FKXJwOMo
+NKyLo097FvHV7k8F9wv9mmZboRulDAoRyDngeO0+SJA90uJass04DuiZvK+g3Hry
+xVzbkk59j9EQqUogopW/oSeSbUP0pvcKOahGcSIW8vmadDTgnN7zzqf3fq+dJ2TM
+QD2IXAwvoTFBE+9DnPOtptk1X1D2umZuAWTzGAseXOImrPFZ+bEr5MV3qLGlg7sL
+yA7Mvbp4diVdH5aePzeBefhxrYphz+yfCbELFTYam9g
+-> ssh-rsa kFDS0A
+Ng0FhTDjASWJkrlNh+UZxU/dU/wfmoV1/fwTv6Xg69k/2qU9lk0oR6e5xAimvX6u
+h7rKAYt3zSRIFveGczPCflC1nycG9wLSpaoJghav+q+muoDQ/fbSKSgHFXITC7Me
+f/wblyWvJsUQbjxSW3g6/8EGz6FvpTnycPtD2vbRj+Ctq72GPA2ZWg/OC4jAUlDs
+r6X0Ql2jwWzy3Y12v0mPknlBezN8cIfjBmoNOWokUeGJIBjujlS7loA1yif09BLg
+PTSLCY1YH3QYcm6lCXK0HaNcMjSSk/ZK9D0wROriF9PBbkpWgg5NlIrqGaeqPN9z
+QwRR2DvhuCa1br57F36Y2LKGphYjmhWAtzCyQ0h9YQ+AzEy9uFCbK0IFyyeVl/fN
+HBGgxacJBcEGsNV3mbJvh6dn1348eex0GgaQEf1B/lu/y66WHbmSqVyUDfWkqEz
+IytAC43VT2rKgg+B5u0d/JhLDLwXTp7iVDy52ul1n7keJHk8t1GDaufAXbWqalQ4
+vuyxs6ghSIXUi27IZrYblg/OEPFTBfcoMXkmCgyx5a+eK+DhnBazWjy5j+vgp2so
+ZQRQurbG02qpZasTwBM3iy4ZklX/uFjsKnk0c/YcmK4YcMviHcQQjdjKruEE93u+
+Za1KE+qZGLkhFCd9O3ZPMtEjRjpN10XIs5ylKQ9MKU4
+-> piv-p256 vRzPNw AiNjNIR0OGHBu5Qn+bvn+Lk5VnpI2BQ3eJ3+2/FTJfZC
+elT3acRVdmtBl0qC5YbvfntxkJrsZwEJqlF6aN5hhWw
+-> piv-p256 zqq/iw AjIzSibkqG+YcP894QekM61Wsty6MaKBghlWapHfU0Jn
+HyXBp8DxtnNsfuzZq13bwgma5CzLTf3UB5Eht6XUwe8
+-> ssh-ed25519 YFSOsg WRBQZZYM+X26hfoH4zvNWQulZvVWP/Ha5OgkUmGK/Q4
+5Hw4ZDNawn5YRC673Op/sbpexOKeL3gez2B7oZxUKhA
+-> ssh-ed25519 iHV63A wyr8R4DlqLAu0XypddVoFimK2ZMncWaa+KWV7vMEQm8
+puV3g1t5AbnEgC0S1U4ft1evB7KuNppEi1g/AtxHgWE
+-> ssh-ed25519 BVsyTA 0N3iyyGqTCRAHHcK7QfN5xRttorc2E2GL0RDTIVIBU4
+Bph0OujqmXzi9IswduX9Mbh+yRdPKOwCf3fBv2zUzqI
+-> ssh-ed25519 +3V2lQ 0p90VtsxWyGFaeeoTISIxQRyeKVk0HoGGq71tjpIPjg
+sRf73Tp3BJ0DsTnJO2xVGyCKjaX7C7oydXj+39dKMUg
+--- +/HCG0s/x+c03NG5qrgliJ+5EXXI6UnuJz5XDv2aphY
+ÞšÂ<>™Ý@»=£L¬“7*®„ÐFq<46>UÒ*ûUê¿‰»È$e=þLgJ|*1Ï½BÚE ZG—_Ü5ê²ð—²ŽíÂ,òöÛi<C39B>_'¸d7
Ý3Ú“Nä3ãç¡*»ðªê<C2AA>£ŽáŽòqýŸ‰Oy#¶([l³†pÄf¼õ¾¥ö
--- a/secrets/drone-secrets.age
+++ b/secrets/drone-secrets.age
--- a/secrets/forgejo-actions-runner-token.age
+++ b/secrets/forgejo-actions-runner-token.age
@ -0,0 +1,43 @@
+age-encryption.org/v1
+-> ssh-ed25519 Y0ZZaw tm4AmC8yXPgR82lgsQR4VZn4xfGiK8o6fIn8pKPY6XQ
+IDnsYVD8noh2HdPNvjY/M5G+meR5rwvVI5SVN/cHEVM
+-> ssh-ed25519 uYcDNw ZB7GyOvD8S8XLqE1AeMXWTPcJnvEntWbZ7TGg8CJVxc
+Bfb/+P2DEiKI9ZWH105rLAYQXTUwWftWtudUGnVtjSU
+-> ssh-rsa f5THog
+Q3Hqks6BMGmP8TXnUkbblO8btrVdls7AUdxDW8e/w5biis/4awZVBHuZCLpiasM7
+7RWXcUep2VVyfCMb+8tedaf5a1MpGPDkZvdbxhfDVWZakh7vsEnth+gK2QsV0h8e
+eIgfDMA9J6DHXNCr4EYSf22PxY12KPqGqsMpVBhOZQuXoJwJy0ob3jbJEOYfPlu6
+V+TLYQNGQ2UQ+A7zjpUrGz1L+u7rUejY4Rv1BmCakg1bLEs8oSDmIVmsuVmFPqOI
+wqQJvnYlRAdioVQZwYCiqJech2QJ0ZhtC/ZeVp4c6TgFwB1ottxiAI8l7Bz1nFzW
+/E85qU3Jkh1tcNcLsVHj8+tnxwn1SSQ3xQxcOT9l1Po98sNapK7mwd/xx3pJ1hH8
+5YUQAtG80e5YmOBxkabVI4s612wACfK9JrIdL+uyIIzGeNHgoimjx7GuOCQ2ut6L
+gj/+Rcv3e0ERkNaXTXGkcdrsMTt45lGmyxUgxz6lbHgtqq+r+BHogiQ9cdPKwXuK
+wom4AvaMOBKCxtB4qVsuNHRd0I8OaA9Ab9SUvHCRvzCkJRHP4qc0zRJif7Rk4qRl
+rAGYwVxq3DRk2HHCQCYC26VqLU6B5LuAAqOipVhOeTfbgaSGD6Wfrt+XBBPmWB7o
+i9zDrk5GKehsPeDKgjh9uVd1y+IBHcWoYBxR5WPTYnA
+-> ssh-rsa kFDS0A
+HS3y8/5wAej0jv0sQqhdGwWd79vUwkrLkoKKPmqo9HlJaO70Cr1bnAIdyA8PBphs
+NIRjIcdClbUBuelZudzuhHuEzH8/JMAVwgHoIiUxviEIr6JpJVBagtvSHp1nfDdn
+x/hkpt2isSYb6fVzYgewqkdD3tv9XEm/WR2JmzlfaNTV9N9x/HNJYy5iYoTWRxKr
+e8R7txdmgRaDDxpbkJdWBcoV9HVgytTMtvBkqGViWzaFDopb9nDlfN/C7/BkCp6H
+9b65JqznpIChoJV7+sK5SEw8VcFj7ikIHzREWscEn8XBb7Kth2iVukaEPM+BgGZz
+Irk0IdkSb/XmQFwsOLnQViwUjkFXGXwHyMdHIcU2qEzZ4PN0PgEengILt9vqWJs0
+qHxrA7sKiC1D8S0i1+Dn+DiI//1s5Bbmp1jk626tH6fNKqSOlpwM47IGArTxCAFd
+NMinIBnR47DUCXWheirsWF6yP7kwX7vOW1dR4UVJJnVPKkgjklwCZvJiNAo0Soo+
+95zuugaeobsJ+qz2Pv+l8BGYriOFpRIAu7YTy9yY7mqHwC5MoeY0G8eNg5UmEzFz
+JsEbKPsZcsMg4WdywzLU2aufK4M/Cd7lVPGZzuZ8hJHBF/EvTFov7L/HK5VnhZar
+CYtILdyiVvmMZ7dhEARG1GG988W9wMendikmKpM4dTA
+-> piv-p256 vRzPNw AuCJRxHGmvv25VTHpnbfMLyLIj8K+daFD97wwHvFAqHl
+m4lPR+5h3+xmdL0OBfmNoxSM/O5Ca+2lVRLwITUtVmQ
+-> piv-p256 zqq/iw ApNbp/6seWw6gCj/QWKLYlmuHaqdaSKVI+Hup1fKAO2O
+xpNXgDXMVFe15eS+L9lGaI0Ip2F13SuhjCTQLDtBIr0
+-> ssh-ed25519 YFSOsg v9BtvUZh5HIvN7nsnErVrHbWTwRhWpj/SlHoiiJSIR0
+ol9z9juHfOCuZsSpuRl/zGFuJg5RzpKK1YnX+VDLDTA
+-> ssh-ed25519 iHV63A yfa9P22C7+wCMqtRRSyiOhcFnLWPI50jMWxWpLarMT8
+VpU+Uv/20JJGkTWTATiF5JImMsDKwyHMj+Wp0mMC/MQ
+-> ssh-ed25519 BVsyTA dUj/mHSyOm61h2ETa4tSX7Cyf+KRN8AMXCVKwMbJTE0
+E1EEPqVQSqlxSAi7DaMlaS1Az1D5XsasrPrIdwylSAE
+-> ssh-ed25519 +3V2lQ AWBlXeuJq76UgQR0xQVPMN0NUq/mqa2iDlIDBBp/Wjo
+XiliN4FB5YQ7qmTyV3AIbvoCY8UoGS6Vi5LpVWrH9kg
+--- V4Tp84/WJUM+/l+eEjlypE4Lx47BtkGFpEnNIdIB35E
+ô¡qzîL¡çzYNŽv×0[¿AöKN<ÛÀ<C39B>ö¦÷:,D<>¢S¹:!$`Õw©^‡àŽhi†O'(ž7!=ÿC(ÿï>´ûDÐ¾ÙFÐ=J
--- a/secrets/matrix-appservice-irc-mediaproxy-signing-key.jwk.age
+++ b/secrets/matrix-appservice-irc-mediaproxy-signing-key.jwk.age
--- a/secrets/matrix-synapse-secret-config.yaml.age
+++ b/secrets/matrix-synapse-secret-config.yaml.age
--- a/secrets/matrix-synapse-sliding-sync-secret.age
+++ b/secrets/matrix-synapse-sliding-sync-secret.age
@ -0,0 +1,45 @@
+age-encryption.org/v1
+-> ssh-ed25519 iDKjwg GPTqfaZZC6ze7BUkT1uF4VslvE29BFKm0+AlJk+DKQQ
+GxI7erqw8p3GrCArh5vZOiTmYh40DVisCphNyFhNTqM
+-> ssh-ed25519 uYcDNw oo52Nh9BCO5NNF0YyzracKfvMifSiREsxyQqiRZ6WTs
+JvqwRX5yOMtEYgWyc7dIQs85wDghMRHQCIi6t5QxIwo
+-> ssh-rsa f5THog
+w+B5hc0E9u1fFWNNPaTtPmJfPJWUBbRwHYK/T69g2ORNfaBYynl0LL4vSUs8o9Gw
+rwBY+cLpth6e4tS819H5C7HtvT47KR3KF8JLxVjA2mbVO83+BnWFjThjYB452CdI
+KZvQQPhkSH/43YF6pjxnQjNWB/wroScyjGVtUamcij7YHxt71z0AAnyqE5PgWEc6
+6/ao5gLfTKhcWpxkTTz8LHn05s9IppXywDrvpwtJaU8LKgJT2H6Epsaci348lG+I
+tAZYODhQqP+yKl92DZbuQQCjxH5CJfhdBs2ZR63hQPj9OrIFRjLg4V+1gdcxzAuz
+9FwwIeLq3uxWXPdwTRR8RUsHEGhKMcVty4PkW0vlt+VwZrZBhdz3k+ApVG7Jvclz
+MPZYLzKC0DiODqPuA23ye6suFRCHXYfq3ZyCIIN6wOci0X0crSr9ZXW4M8R7aWaZ
+XDeZRaUgvd54WI0HZhVWBvJQyswgUXf+/RkS4aI8IgnNV801x12h+mTdWX9BC/cD
+YRIWBnGkfTX4WM4OEE2VEgqSDuKl/90o2LFIquIIJULVd2Vs5C2S8FhJcsT7+HmL
+TFWnLeIfGbw7RDUeH0c/Bbg9NK11SZF0/VdRZcBQ/zIXBMBlL1EZsH1HfIfhKISN
+PyHFB5kfmuVIBhDXgtDdgjKfDmQL9/9Aq1U4ZMBcUKA
+-> ssh-rsa kFDS0A
+KysKtr7wrKKJ8w+Dj7qjJstyXtKIw9weFi9oVwJkMvy2utn+JARs7puh7KC27TXC
+slZJrHf4vx+y8qSjRS0W4z8CPl8/auiYOilepT9JoxwGUP7J/nTr5SCofgWcdZm5
+FtgHoCcABjzcF+mrKUofuqrx6oYSDCS0JkV2tClQI6ybXnjRwIIicLmBN9UDHCuU
+9ZOesYp5XrJyBoD3Zv51b19xJyOfuWAUQvlNPRH2TpgvisutpESU/o869z5AMn4Z
+BfDD/0oR1ALbk/sB3r13Xi6oJZAB2AbggoQRlwvPeWc3MdS+bFNV2o2ue0ov6Fkd
+U5C/GnJVlyE0cv9I+YvxtLT6T/Gf/yoUZGfB7xD5QkHpMIEmKxUYqGNBB/NcnFMY
+Tal8jMDtZDEk+uk0MahE7GsL6Z3xrkKTevG+Rr3j+beFYie2RJbNwwUyQ1lL3EoA
+Rx1AMk+nYlvxVHiciYJNh9nffgAXXwO255IkWvYzmuPBEP1LmqadA4fQPf5Rgj3u
+DuOX3hJ+rIyRIoDXOZio3SDf+bb380xCxF+7efJ27Ep0sFviAq5qKeptbyt51Dp8
+tlbeYAylhVbV9Zgd+EozwE7Btlfqt3sbUij/0Iy+BdOYSPLmvx3oKybpipZ0i3fo
+KR/bZHlMKF1Ipd5L7zEwh5aTjImuomoyRyZG3NWdv44
+-> piv-p256 vRzPNw A7FwWUuml/VyHcOmha3R/DOg1RvnRXcwjaJJH/sgmsBR
+CP1/qY8sHbR7nkFl1T5HPsjYLRPDCSR01DEJaim96o
+-> piv-p256 zqq/iw AgYhaJWqe+QbVCHkXsU7AQhWhte/fjwVbOgmHVRPHsEE
+7jNmDI62i/9RakJhbo3MP0qMgXYGlhAW9BKo8HLWQYc
+-> ssh-ed25519 YFSOsg cGPMyhqcd20TDBeMkSDJ8hQ/vE9cuDgVi1hfcwAKVjw
+U9GRSr607w5oUGr0rC6XqdWMD65JidY/Ri3Ex1dmGXI
+-> ssh-ed25519 iHV63A cW7bblsvL1TwI6lp8KjPfUwB5EzWilLhc6Z2geE3SQw
+PzBdZ/LXA7iGI7ZjErredqC7ehHsr5MCY3qENv0nZI8
+-> ssh-ed25519 BVsyTA AGDqp6Rrp2vStBU9+eJMGf5O4SZQIASE63n8vbf8PEs
+SFakjoivQrFkSUBGZ9sISKVhAxNOpc2RxugiBTSK9/k
+-> ssh-ed25519 +3V2lQ MmMv45CQFAdgkV/B7InOY22iXzvIU8TY41SV5Jxx7RQ
+vNIRE5wSXVzy4miZLV90T1TEOhOjYQT12GWtZpsTxJ8
+--- EBBXvYr1OpETpgXOsUfJn6h1e4rXF+olz6DbhDUWCcw
+.Œ
+ösÊ~¦—åHÍ¯k‘	2 ¦À9<C380>¶§Bz¤¨?°3ëþTÇJ`§gº¦P°çioÙÜr<C39C>Š€" ØÝöwÒs-K€6©òšfÝ
+ß0XÎŒvù‘Xª&£8
š¶gÐ=ÄzrH¥jh>
--- a/secrets/nachtigall-coturn-static-auth-secret.age
+++ b/secrets/nachtigall-coturn-static-auth-secret.age
--- a/secrets/restic-password-droppie.age
+++ b/secrets/restic-password-droppie.age
@ -0,0 +1,29 @@
+age-encryption.org/v1
+-> ssh-ed25519 iDKjwg tSrbeVPpTxjeuCg6JiOdeIcaGWq/PUOyXrt+ZNIWmzI
+fz4B2nMu36G1DYPyNiu30Txx4cOLGyUreuBLqktORhY
+-> ssh-ed25519 uYcDNw zySKp8qUnCKOCbIP17bWYvJ/TsKxLl/v/AEsfVQpHTc
+SYqaUEXJ7mSA2wY6I3i/hirEiqvXVnWH1NYtXbaMzbw
+-> ssh-rsa kFDS0A
+MiyJjq7nn9B+PF9Lf6vD496FZUnUy5zI9H2kunJLhmYnr+7ZJaPGI+MW0cuPjsmM
+1XhmUC72zA9WTiuReGijl9GhIyU0D6vPx+ezQrx18dUGRJEqnvk5C75SCUuPGtUi
+DamPxJc781mnDSyW91MgRrqlBVwhfJ9Zmws3+/j8LODWb0T9Jvy0ywmFX4ilapr9
+6dqbNmwBZkfh2CKB9LtMG/DuSNicgHscpt4Mt7EogJRdgBrGIcA1u8BXUOwf+R8G
+Ya8VzoQV0iVhGtbff3g/6/0+UX9KEZBQc/aiT2ItrgUmgMyjAZo13Dj3pD2wPdfa
+t346symp7bcYxRea/U0hfmPdnixIKgBLTUZkzrMFza26QOvnSwiMxWY+//Vmbp3z
+0YCsyOQTQJVB+Q0pYE0+9EVI9bInP0LxHhM7mIdJroscRtY1YphjZUlFmcUXk0iC
+5g3/wdLr8KdT4aZ93m4WVgkBUzg8chhfMHb9uFvpHT9xeZHrV3u30TuJpmmuqMXK
+JD3+9xRzNVHwvfZuhD8B3vkK2tYO7pTxJytkJtdolx/uSju8SCm9F4lwt5y8rQd4
+7676QS0IWS1WItcpLrsgk9nWZUNzy78gBc39jtJUOsFDqJSdAWACq/jgYpnx6Bja
+5aZonRAp3fyCEavUpS4AQUJZGVx9EFp9LqXeUiIkCJU
+-> ssh-ed25519 YFSOsg vJ7xw4zDSKBFuv8FT6ZnB83uGwHDnDsEwsvQ6urZkEk
+gOkuwjOX8U/qmPuGrPPb+FYy6PuJjrgCNFYrukkaIPA
+-> ssh-ed25519 iHV63A FzZRHQB39iDTLm4y1QLWtfDi3jZn6i59nuSlf0d5mww
+5kiyKN7KFTv8pJgRAZTUMgR2+TInByFotDhAw307P6g
+-> ssh-ed25519 BVsyTA rTjBaEI6AwwmpK9Q8elVJaSN60deXOMEmZPrsH9O1nY
+HikNbHzx3BEJ3hn4YPFr2p/cnSu3qr9cvgdKCxN6xY8
+-> h|2u%UF\-grease Gcp2NG
+y6OjgzrOYEVgrY2+fv8a1xrVuoHQIPncGuGeLI0zHh4xf1qbzD2vKYp+W1fRBE8d
+/yPEGUe/T/ZjO+F4oF266HUR23wancFeFoGpfJgWQVS2oc0Z4aMuNebf/+Kw087W
+
+--- GJBLNxBoH7vI1mkcCmfbm9UiG4xuwIOWS7IaJKuO6cc
+C<EFBFBD>3=¼,¦{-TçÄéÈµ	™-ˆ]äÌ·»æq=‡“û
Òú–=~}Î1®˜ lß„¿fúì_[žU_å›A€G.ÒdŽêÖrëa`×åUyÉPt+Ì½òö‡¯!<
--- a/secrets/secrets.nix
+++ b/secrets/secrets.nix
@ -67,14 +67,12 @@ in
  "matrix-mautrix-telegram-env-file.age".publicKeys = nachtigallKeys ++ adminKeys;
  "matrix-synapse-signing-key.age".publicKeys = nachtigallKeys ++ adminKeys;
  "matrix-synapse-secret-config.yaml.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "matrix-synapse-sliding-sync-secret.age".publicKeys = nachtigallKeys ++ adminKeys;
  "matrix-authentication-service-secret-config.yml.age".publicKeys = nachtigallKeys ++ adminKeys;
-  "matrix-appservice-irc-mediaproxy-signing-key.jwk.age".publicKeys = nachtigallKeys ++ adminKeys;

  "staging-matrix-synapse-secret-config.yaml.age".publicKeys = undergroundKeys ++ adminKeys;
  "staging-matrix-authentication-service-secret-config.yml.age".publicKeys =
    undergroundKeys ++ adminKeys;
-  "staging-matrix-appservice-irc-mediaproxy-signing-key.jwk.age".publicKeys =
-    undergroundKeys ++ adminKeys;

  "nextcloud-secrets.age".publicKeys = nachtigallKeys ++ adminKeys;
  "nextcloud-admin-pass.age".publicKeys = nachtigallKeys ++ adminKeys;
@ -94,7 +92,7 @@ in
  "mediawiki-oidc-client-secret.age".publicKeys = nachtigallKeys ++ adminKeys;
  "mediawiki-secret-key.age".publicKeys = nachtigallKeys ++ adminKeys;

-  "coturn-static-auth-secret.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "nachtigall-coturn-static-auth-secret.age".publicKeys = nachtigallKeys ++ adminKeys;

  "grafana-admin-password.age".publicKeys = trinkgenossinKeys ++ adminKeys;
  "grafana-keycloak-client-secret.age".publicKeys = trinkgenossinKeys ++ adminKeys;
--- a/secrets/staging-matrix-appservice-irc-mediaproxy-signing-key.jwk.age
+++ b/secrets/staging-matrix-appservice-irc-mediaproxy-signing-key.jwk.age
--- a/secrets/staging-matrix-synapse-secret-config.yaml.age
+++ b/secrets/staging-matrix-synapse-secret-config.yaml.age
--- a/tests/keycloak.nix
+++ b/tests/keycloak.nix
@ -46,7 +46,6 @@ in

      systemd.tmpfiles.rules = [ "f /tmp/dbf 1777 root root 10d password" ];

-      virtualisation.cores = 1;
      virtualisation.memorySize = 4096;

      pub-solar-os.auth = {
@ -67,7 +66,7 @@ in
  testScript =
    { nodes, ... }:
    let
-      user = nodes.client.users.users.b12f;
+      user = nodes.client.users.users.${nodes.client.pub-solar-os.authentication.username};
      #uid = toString user.uid;
      bus = "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u ${user.name})/bus";
      gdbus = "${bus} gdbus";
@ -82,8 +81,6 @@ in
      nachtigall.succeed("ping 127.0.0.1 -c 2")
      nachtigall.wait_for_unit("nginx.service")
      nachtigall.wait_for_unit("keycloak.service")
-      nachtigall.wait_for_open_port(8080)
-      nachtigall.wait_for_open_port(443)
      nachtigall.wait_until_succeeds("curl http://127.0.0.1:8080/")
      nachtigall.wait_until_succeeds("curl https://auth.test.pub.solar/")

--- a/tests/support/client.nix
+++ b/tests/support/client.nix
@ -11,7 +11,7 @@
  services.xserver.displayManager.gdm.enable = true;
  services.xserver.desktopManager.gnome.enable = true;
  services.xserver.displayManager.autoLogin.enable = true;
-  services.xserver.displayManager.autoLogin.user = "b12f";
+  services.xserver.displayManager.autoLogin.user = config.pub-solar-os.authentication.username;

  systemd.user.services = {
    "org.gnome.Shell@wayland" = {
--- a/tests/website.nix
+++ b/tests/website.nix
@ -31,7 +31,6 @@
        ./support/global.nix
      ];

-      virtualisation.cores = 1;
      virtualisation.memorySize = 4096;

      networking.interfaces.eth0.ipv4.addresses = [
@ -49,14 +48,12 @@
    acme_server.wait_for_unit("system.slice")
    acme_server.wait_for_unit("step-ca.service")
    acme_server.succeed("ping ca.test.pub.solar -c 2")
-    acme_server.wait_for_open_port(443)
    acme_server.wait_until_succeeds("curl 127.0.0.1:443")

    nachtigall.wait_for_unit("system.slice")
    nachtigall.succeed("ping test.pub.solar -c 2")
    nachtigall.succeed("ping ca.test.pub.solar -c 2")
    nachtigall.wait_for_unit("nginx.service")
-    nachtigall.wait_for_open_port(443, "test.pub.solar")
    nachtigall.wait_until_succeeds("curl https://test.pub.solar/")
  '';
 }
Author	SHA1	Message	Date
teutat3s	10cc5e914b	blackbox-exporter: scrape every 60m, add URLs All checks were successful Flake checks / Check (pull_request) Successful in 24m13s Details	2024-11-06 21:54:16 +01:00
teutat3s	a9c54dc496	style: fix formatting	2024-11-06 21:31:41 +01:00
teutat3s	84e2b22771	monitoring: scrape blackbox every 5m, alert 21 days before certificate expiry	2024-11-06 21:31:41 +01:00
teutat3s	d2806156c9	grafana: add blackbox exporter dashboard	2024-11-06 21:31:41 +01:00
Pablo Ovelleiro Corral	15783ba0e8	fix formatting	2024-11-06 21:31:40 +01:00
Pablo Ovelleiro Corral	6548edee9a	Add blackbox-exporter config	2024-11-06 21:31:40 +01:00