obs-portal: init obs-portal on nachtigall

This follows the official installation instructions at https://github.com/openbikesensor/portal/blob/main/docs/production-deployment.md

Unfortunately, the postgres database needs to have postgis enabled, so
we'll have to start a second instance. To stay close to the official
deployment instructions, this is running in docker.

The secrets were taken from the old installation instance. During
initial installation, we'll need to import data from the old instance
into this one, which might take a while.

.forgejo/workflows/check.yml

@@ -10,7 +10,7 @@ jobs:
       - name: Check out repository code
         uses: https://code.forgejo.org/actions/checkout@v4
 
-      - uses: https://github.com/nixbuild/nix-quick-install-action@v26
+      - uses: https://github.com/nixbuild/nix-quick-install-action@v27
         with:
           load_nixConfig: false
           nix_conf: |
@@ -24,23 +24,22 @@ jobs:
           echo "hash=$(md5sum flake.lock | awk '{print $1}')" >> $GITHUB_OUTPUT
 
       - name: Restore and cache Nix store
-        uses: https://github.com/nix-community/cache-nix-action@v4
+        uses: https://github.com/nix-community/cache-nix-action@v5
         id: nix-store-cache
         with:
-          key: cache-${{ runner.os }}-nix-store-${{ steps.flake-lock-hash.outputs.hash }}
-          restore-keys: |
+          primary-key: cache-${{ runner.os }}-nix-store-${{ steps.flake-lock-hash.outputs.hash }}
+          restore-prefixes-first-match: |
             cache-${{ runner.os }}-nix-store-
 
           gc-linux: true
           gc-max-store-size-linux: 10000000000
 
-          purge-caches: true
-          purge-keys: cache-${{ runner.os }}-nix-store-
-          purge-created: true
-          purge-created-max-age: 42
+          purge: true
+          purge-prefixes: cache-${{ runner.os }}-nix-store-
+          purge-created: 42
 
       - name: Prepare cachix
-        uses: https://github.com/cachix/cachix-action@v12
+        uses: https://github.com/cachix/cachix-action@v14
         with:
           name: pub-solar
           authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'

docs/administrative-access.md

@@ -0,0 +1,37 @@
+# Adminstrative access
+
+People with admin access to the infrastructure are added to [`logins/admins.nix`](../logins/admins.nix). This is a attrset with the following structure:
+
+```
+{
+    <username> = {
+        sshPubKeys = {
+            <name> = <pubkey-string>;
+        };
+
+        wireguardDevices = [
+            {
+                publicKey = <pubkey-string>;
+                allowedIPs = [ "10.7.6.<ip-address>/32" "fd00:fae:fae:fae:fae:<ip-address>::/96" ];
+            }
+        }];
+
+        secretEncryptionKeys = {
+            <name> = <encryption-key-string>;
+        };
+    };
+}
+```
+
+# SSH Access
+
+SSH is not reachable from the open internet. Instead, SSH Port 22 is protected by a wireguard VPN network. Thus, to get root access on the servers, at least two pieces of information have to be added to the admins config:
+
+1. **SSH Public key**: self-explanatory. Add your public key to your user attrset under `sshPubKeys`.
+2. **Wireguard device**: each wireguard device has two parts: the public key and the IP addresses it should have in the wireguard network. The pub.solar wireguard network is spaced under `10.7.6.0/24` and `fd00:fae:fae:fae:fae::/80`. To add your device, it's best to choose a free number between 200 and 255 and use that in both the ipv4 and ipv6 ranges: `10.7.6.<ip-address>/32` `fd00:fae:fae:fae:fae:<ip-address>::/96`. For more information on how to generate keypairs, see [the NixOS Wireguard docs](https://nixos.wiki/wiki/WireGuard#Generate_keypair).
+
+# Secret encryption
+
+Deployment secrets are added to the repository in encrypted files. To be able to work with these encrypted files, your public key(s) will have to be added to your user attrset under `secretEncryptionKeys`.
+
+See also the docs on [working with secrets](./secrets.md).

docs/deploying.md

@@ -1,20 +1,32 @@
 # Deploying new versions
 
-We use [deploy-rs](https://github.com/serokell/deploy-rs) to deploy changes. Currently this process is not automated, so configuration changes will have to be manually deployed.
+We use [deploy-rs](https://github.com/serokell/deploy-rs) to deploy changes.
+Currently this process is not automated, so configuration changes will have to
+be manually deployed.
 
-To deploy, make sure you have a [working development shell](./development-shell.md). Then, run `deploy-rs` with the hostname of the server you want to deploy:
+To deploy, make sure you have a [working development shell](./development-shell.md).
+Then, run `deploy-rs` with the hostname of the server you want to deploy:
 
 For nachtigall.pub.solar:
 ```
-deploy '.#nachtigall'
+deploy --targets '.#nachtigall' --magic-rollback false --auto-rollback false
 ```
 
 For flora-6.pub.solar:
 ```
-deploy '.#flora-6'
+deploy --targets '.#flora-6' --magic-rollback false --auto-rollback false
 ```
 
-You'll need to have SSH Access to the boxes to be able to do this.
+Usually we skip all rollback functionality, but if you want to deploy a change
+that might lock you out, e.g. to SSH, it might make sense to set these to `true`.
 
-### SSH access
-Ensure your SSH public key is in place [here](./public-keys/admins.nix) and was deployed by someone with access.
+To skip flake checks, e.g. because you already ran them manually before
+deployment, add the flag `--skip-checks` at the end of the command.
+
+`--dry-activate` can be used to only put all files in place without switching,
+to enable switching to the new config quickly at a later moment.
+
+You'll need to have SSH Access to the boxes to be able to run `deploy`.
+
+### Getting SSH access
+See [administrative-access.md](./administrative-access.md).

docs/secrets.md

@@ -1 +1,5 @@
 # Working with secrets
+
+Secrets are handled with [agenix](https://github.com/ryantm/agenix). To be able to view secrets, your public key will have to be added to the admins config. See [Administrative Access](./administrative-access.md) on how to do this.
+
+For a comprehensive tutorial, see [the agenix repository](https://github.com/ryantm/agenix?tab=readme-ov-file#tutorial).

docs/ssh.md

@@ -1,3 +0,0 @@
-# SSH Access
-
-SSH Access is granted by adding a public key to [`public-keys/admins.nix`](../public-keys/admins.nix). This change will then have to be deployed to all hosts by an existing key. The keys will also grant access to the initrd SSH Server to enable remote unlock.

flake.lock

@@ -14,11 +14,11 @@
         "systems": "systems"
       },
       "locked": {
-        "lastModified": 1707830867,
-        "narHash": "sha256-PAdwm5QqdlwIqGrfzzvzZubM+FXtilekQ/FA0cI49/o=",
+        "lastModified": 1712079060,
+        "narHash": "sha256-/JdiT9t+zzjChc5qQiF+jhrVhRt8figYH29rZO7pFe4=",
         "owner": "ryantm",
         "repo": "agenix",
-        "rev": "8cb01a0e717311680e0cbca06a76cbceba6f3ed6",
+        "rev": "1381a759b205dff7a6818733118d02253340fd5e",
         "type": "github"
       },
       "original": {
@@ -36,11 +36,11 @@
         "utils": "utils"
       },
       "locked": {
-        "lastModified": 1708091384,
-        "narHash": "sha256-dTGGw2y8wvfjr+J9CjQbfdulOq72hUG17HXVNxpH1yE=",
+        "lastModified": 1711973905,
+        "narHash": "sha256-UFKME/N1pbUtn+2Aqnk+agUt8CekbpuqwzljivfIme8=",
         "owner": "serokell",
         "repo": "deploy-rs",
-        "rev": "0a0187794ac7f7a1e62cda3dabf8dc041f868790",
+        "rev": "88b3059b020da69cbe16526b8d639bd5e0b51c8b",
         "type": "github"
       },
       "original": {
@@ -109,11 +109,11 @@
         "nixpkgs-lib": "nixpkgs-lib"
       },
       "locked": {
-        "lastModified": 1709336216,
-        "narHash": "sha256-Dt/wOWeW6Sqm11Yh+2+t0dfEWxoMxGBvv3JpIocFl9E=",
+        "lastModified": 1712014858,
+        "narHash": "sha256-sB4SWl2lX95bExY2gMFG5HIzvva5AVMJd4Igm+GpZNw=",
         "owner": "hercules-ci",
         "repo": "flake-parts",
-        "rev": "f7b3c975cf067e56e7cda6cb098ebe3fb4d74ca2",
+        "rev": "9126214d0a59633752a136528f5f3b9aa8565b7d",
         "type": "github"
       },
       "original": {
@@ -180,11 +180,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1706981411,
-        "narHash": "sha256-cLbLPTL1CDmETVh4p0nQtvoF+FSEjsnJTFpTxhXywhQ=",
+        "lastModified": 1712386041,
+        "narHash": "sha256-dA82pOMQNnCJMAsPG7AXG35VmCSMZsJHTFlTHizpKWQ=",
         "owner": "nix-community",
         "repo": "home-manager",
-        "rev": "652fda4ca6dafeb090943422c34ae9145787af37",
+        "rev": "d6bb9f934f2870e5cbc5b94c79e9db22246141ff",
         "type": "github"
       },
       "original": {
@@ -224,11 +224,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1710717205,
-        "narHash": "sha256-Wf3gHh5uV6W1TV/A8X8QJf99a5ypDSugY4sNtdJDe0A=",
+        "lastModified": 1713543876,
+        "narHash": "sha256-olEWxacm1xZhAtpq+ZkEyQgR4zgfE7ddpNtZNvubi3g=",
         "owner": "lnl7",
         "repo": "nix-darwin",
-        "rev": "bcc8afd06e237df060c85bad6af7128e05fd61a3",
+        "rev": "9e7c20ffd056e406ddd0276ee9d89f09c5e5f4ed",
         "type": "github"
       },
       "original": {
@@ -240,11 +240,11 @@
     },
     "nixos-flake": {
       "locked": {
-        "lastModified": 1710867744,
-        "narHash": "sha256-wtAZ+zhW5kMkiOomEA27p+T3M5ZNgY6U4zB+03/EJDs=",
+        "lastModified": 1711376798,
+        "narHash": "sha256-37wawZGSX/dD1rn7TwFJhUdpozC2VPEQXetpfpK/D+w=",
         "owner": "srid",
         "repo": "nixos-flake",
-        "rev": "05f9464e282dee5a706273f50344a8201d8980b5",
+        "rev": "7b19503e7f8c7cc0884fc2fbd669c0cc2e05aef5",
         "type": "github"
       },
       "original": {
@@ -255,11 +255,11 @@
     },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1710838473,
-        "narHash": "sha256-RLvwdQSENKOaLdKhNie8XqHmTXzNm00/M/THj6zplQo=",
+        "lastModified": 1713564160,
+        "narHash": "sha256-YguPZpiejgzLEcO36/SZULjJQ55iWcjAmf3lYiyV1Fo=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "fa9f817df522ac294016af3d40ccff82f5fd3a63",
+        "rev": "bc194f70731cc5d2b046a6c1b3b15f170f05999c",
         "type": "github"
       },
       "original": {
@@ -288,11 +288,11 @@
     "nixpkgs-lib": {
       "locked": {
         "dir": "lib",
-        "lastModified": 1709237383,
-        "narHash": "sha256-cy6ArO4k5qTx+l5o+0mL9f5fa86tYUX3ozE1S+Txlds=",
+        "lastModified": 1711703276,
+        "narHash": "sha256-iMUFArF0WCatKK6RzfUJknjem0H9m4KgorO/p3Dopkk=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "1536926ef5621b09bba54035ae2bb6d806d72ac8",
+        "rev": "d8fe5e6c92d0d190646fb9f1056741a229980089",
         "type": "github"
       },
       "original": {
@@ -405,11 +405,11 @@
     },
     "unstable": {
       "locked": {
-        "lastModified": 1710806803,
-        "narHash": "sha256-qrxvLS888pNJFwJdK+hf1wpRCSQcqA6W5+Ox202NDa0=",
+        "lastModified": 1713537308,
+        "narHash": "sha256-XtTSSIB2DA6tOv+l0FhvfDMiyCmhoRbNB+0SeInZkbk=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "b06025f1533a1e07b6db3e75151caa155d1c7eb3",
+        "rev": "5c24cf2f0a12ad855f444c30b2421d044120c66f",
         "type": "github"
       },
       "original": {

flake.nix

@@ -39,7 +39,7 @@
 
       imports = [
         inputs.nixos-flake.flakeModule
-        ./public-keys
+        ./logins
         ./lib
         ./overlays
         ./modules
@@ -63,6 +63,7 @@
             deploy-rs
             nixpkgs-fmt
             agenix
+            age-plugin-yubikey
             cachix
             editorconfig-checker
             nodePackages.prettier
@@ -88,14 +89,12 @@
 
           deploy.nodes = self.lib.deploy.mkDeployNodes self.nixosConfigurations {
             nachtigall = {
-              # hostname is set in hosts/nachtigall/networking.nix
+              hostname = "10.7.6.1";
               sshUser = username;
             };
             flora-6 = {
-              hostname = "flora-6.pub.solar";
+              hostname = "10.7.6.2";
               sshUser = username;
-              # Example
-              #sshOpts = [ "-p" "19999" ];
             };
           };
         };

hosts/flora-6/apps/drone.nix

@@ -78,6 +78,7 @@
         extraOptions = [
           "--network=drone-net"
           "--pull=always"
+          "--add-host=nachtigall.pub.solar:10.7.6.1"
         ];
         environment = {
           DRONE_GITEA_SERVER = "https://git.pub.solar";
@@ -101,6 +102,7 @@
         extraOptions = [
           "--network=drone-net"
           "--pull=always"
+          "--add-host=nachtigall.pub.solar:10.7.6.1"
         ];
         environment = {
           DRONE_RPC_HOST = "ci.pub.solar";

hosts/flora-6/configuration.nix

@@ -35,6 +35,7 @@ in
     #systemd.services."systemd-networkd".environment.SYSTEMD_LOG_LEVEL = "debug";
     systemd.network.wait-online.ignoredInterfaces = [
       "docker0"
+      "wg-ssh"
     ];
 
     # List services that you want to enable:

hosts/flora-6/default.nix

@@ -7,6 +7,7 @@
       ./hardware-configuration.nix
       ./configuration.nix
       ./triton-vmtools.nix
+      ./wireguard.nix
 
       ./apps/caddy.nix
 

hosts/flora-6/wireguard.nix

@@ -0,0 +1,40 @@
+{
+  config,
+  pkgs,
+  flake,
+  ... }:
+{
+  networking.firewall.allowedUDPPorts = [ 51820 ];
+
+  age.secrets.wg-private-key.file = "${flake.self}/secrets/flora6-wg-private-key.age";
+
+  networking.wireguard.interfaces = {
+    wg-ssh = {
+      listenPort = 51820;
+      mtu = 1300;
+      ips = [
+        "10.7.6.2/32"
+        "fd00:fae:fae:fae:fae:2::/96"
+      ];
+      privateKeyFile = config.age.secrets.wg-private-key.path;
+      peers = flake.self.logins.admins.wireguardDevices ++ [
+        { # nachtigall.pub.solar
+          endpoint = "138.201.80.102:51820";
+          publicKey = "qzNywKY9RvqTnDO8eLik75/SHveaSk9OObilDzv+xkk=";
+          allowedIPs = [ "10.7.6.1/32" "fd00:fae:fae:fae:fae:1::/96" ];
+        }
+      ];
+    };
+  };
+
+  services.openssh.listenAddresses = [ 
+    {
+      addr = "10.7.6.2";
+      port = 22;
+    }
+    {
+      addr = "[fd00:fae:fae:fae:fae:2::]";
+      port = 22;
+    }
+  ];
+}

hosts/nachtigall/apps/forgejo.nix

@@ -16,6 +16,19 @@
     owner = "gitea";
   };
 
+  age.secrets.forgejo-ssh-private-key = {
+    file = "${flake.self}/secrets/forgejo-ssh-private-key.age";
+    mode = "600";
+    owner = "gitea";
+    path = "/etc/forgejo/ssh/id_forgejo";
+  };
+
+  environment.etc."forgejo/ssh/id_forgejo.pub" = {
+    text = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCkPjvF2tZ2lZtkXed6lBvaPUpsNrI5kHlCNEf4LyFtgFXHoUL8UD3Bz9Fn1S+SDkdBMw/SumjvUf7TEGqQqzmFbG7+nWdWg2L00VdN8Kp8W+kKPBByJrzjDUIGhIMt7obaZnlSAVO5Cdqc1Q6bA9POLjSHIBxSD3QUs2pjUCkciNcEtL93easuXnlMwoYa217n5sA8n+BZmOJAcmA/UxYvKsqYlpJxa44m8JgMTy+5L08i/zkx9/FwniOcKcLedxmjZfV8raitDy34LslT2nBNG4I+em7qhKhSScn/cfyPvARiK71pk/rTx9mxBEjcGAkp3+hiA3Nyms0h/qTUh8yGyhbOn8hiro34HEKswXDN1HRfseyyZ4TqOoIC07F53x4OliYA0B+QbvwOemTX2XAWHfU4xEYrIhR46o3Eu5ooOM9HZLLYzIzKjsj/rpuKalFZ+9IeT/PJ/DrbgOEBlJGTu4XucEYXSiIvWB7G9WXij7TXKYbsRAFho9jw+9UZWklFAh9dcUKlX9YxafxOrw9DhJK620hblHLY9wPPFCbZVXDGfqdtn+ncRReMAw6N3VYqxMgnxd+OC52SMsSUi9VaL26i2UvEBwNYuim8GDnVabu/ciQLHMgifBONuF9sKD58ee5nnKgtYLDy9zU86aHBU78Ijew+WhYitO7qejMHMQ==";
+    mode = "600";
+    user = "gitea";
+  };
+
   services.nginx.virtualHosts."git.pub.solar" = {
     enableACME = true;
     forceSSL = true;
@@ -41,11 +54,17 @@
 
   users.groups.gitea = {};
 
+  # Expose SSH port only for forgejo SSH
+  networking.firewall.interfaces.enp35s0.allowedTCPPorts = [ 2223 ];
+  networking.firewall.extraCommands = ''
+    iptables -t nat -i enp35s0 -I PREROUTING -p tcp --dport 22 -j REDIRECT --to-ports 2223
+    ip6tables -t nat -i enp35s0 -I PREROUTING -p tcp --dport 22 -j REDIRECT --to-ports 2223
+  '';
+
   services.forgejo = {
     enable = true;
     user = "gitea";
     group = "gitea";
-    package = pkgs.forgejo;
     database = {
       type = "postgres";
       passwordFile = config.age.secrets.forgejo-database-password.path;
@@ -63,6 +82,9 @@
         DOMAIN = "git.pub.solar";
         HTTP_ADDR = "127.0.0.1";
         HTTP_PORT = 3000;
+        START_SSH_SERVER = true;
+        SSH_LISTEN_PORT = 2223;
+        SSH_SERVER_HOST_KEYS = "${config.age.secrets."forgejo-ssh-private-key".path}";
       };
 
       log.LEVEL = "Warn";
@@ -111,6 +133,19 @@
         # the value of DEFAULT_ACTIONS_URL is prepended to it.
         DEFAULT_ACTIONS_URL = "https://code.forgejo.org";
       };
+
+      # https://forgejo.org/docs/next/admin/recommendations/#securitylogin_remember_days
+      security = {
+        LOGIN_REMEMBER_DAYS = 365;
+      };
+
+      # https://forgejo.org/docs/next/admin/config-cheat-sheet/#indexer-indexer
+      indexer = {
+        REPO_INDEXER_ENABLED = true;
+        REPO_INDEXER_PATH = "indexers/repos.bleve";
+        MAX_FILE_SIZE = 1048576;
+        REPO_INDEXER_EXCLUDE = "resources/bin/**";
+      };
     };
   };
 
@@ -155,6 +190,11 @@
     backupCleanupCommand = ''
       rm /tmp/forgejo-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 
   services.restic.backups.forgejo-storagebox = {
@@ -174,5 +214,10 @@
     backupCleanupCommand = ''
       rm /tmp/forgejo-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 }

hosts/nachtigall/apps/keycloak.nix

@@ -64,6 +64,11 @@
     backupCleanupCommand = ''
       rm /tmp/keycloak-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 
   services.restic.backups.keycloak-storagebox = {
@@ -82,5 +87,10 @@
     backupCleanupCommand = ''
       rm /tmp/keycloak-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 }

hosts/nachtigall/apps/mailman.nix

@@ -94,6 +94,11 @@
     initialize = true;
     passwordFile = config.age.secrets."restic-repo-droppie".path;
     repository = "sftp:yule@droppie.b12f.io:/media/internal/pub.solar";
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 
   services.restic.backups.mailman-storagebox = {
@@ -109,5 +114,10 @@
     initialize = true;
     passwordFile = config.age.secrets."restic-repo-storagebox".path;
     repository = "sftp:u377325@u377325.your-storagebox.de:/backups";
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 }

hosts/nachtigall/apps/mastodon.nix

@@ -61,6 +61,9 @@
       passwordFile = "/run/agenix/mastodon-smtp-password";
       fromAddress = "mastodon-notifications@pub.solar";
     };
+    mediaAutoRemove = {
+      olderThanDays = 7;
+    };
     extraEnvFiles = [
       "/run/agenix/mastodon-extra-env-secrets"
     ];
@@ -111,6 +114,11 @@
     backupCleanupCommand = ''
       rm /tmp/mastodon-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 
   services.restic.backups.mastodon-storagebox = {
@@ -129,5 +137,10 @@
     backupCleanupCommand = ''
       rm /tmp/mastodon-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 }

hosts/nachtigall/apps/matrix/synapse.nix

@@ -312,5 +312,10 @@ in
     backupCleanupCommand = ''
       rm /tmp/matrix-synapse-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 }

hosts/nachtigall/apps/mediawiki.nix

@@ -208,7 +208,7 @@ in
       backend = "docker";
 
       containers."mediawiki" = {
-        image = "git.pub.solar/pub-solar/mediawiki-oidc-docker:1.41.0";
+        image = "git.pub.solar/pub-solar/mediawiki-oidc-docker:1.41.1";
         user = "1000:${builtins.toString gid}";
         autoStart = true;
 

hosts/nachtigall/apps/nextcloud.nix

@@ -97,6 +97,7 @@
       integrity.check.disabled = false;
       updater.release.channel = "stable";
       loglevel = 0;
+      maintenance_window_start = "1";
       # maintenance = false;
       app_install_overwrite = [
         "pdfdraw"
@@ -149,6 +150,11 @@
     backupCleanupCommand = ''
       rm /tmp/nextcloud-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 
   services.restic.backups.nextcloud-storagebox = {
@@ -168,5 +174,10 @@
     backupCleanupCommand = ''
       rm /tmp/nextcloud-backup.sql
     '';
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 3"
+    ];
   };
 }

hosts/nachtigall/apps/nginx.nix

@@ -24,6 +24,13 @@ in
       # https://my.f5.com/manage/s/article/K51798430
       proxy_headers_hash_bucket_size 128;
     '';
+    appendConfig = ''
+      # Number of CPU cores
+      worker_processes 8;
+    '';
+    eventsConfig = ''
+      worker_connections 1024;
+    '';
   };
 
   security.acme = {

hosts/nachtigall/apps/obs-portal.nix

@@ -0,0 +1,140 @@
+{ config
+, lib
+, pkgs
+, self
+, flake
+, ...
+}: let 
+  configPy = pkgs.writeText "obs-portal-config.py" ''
+DEBUG = False
+VERBOSE = DEBUG
+AUTO_RESTART = DEBUG
+LEAN_MODE = False
+FRONTEND_URL = None
+FRONTEND_HTTPS = True
+FRONTEND_DIR = "../frontend/build/"
+FRONTEND_CONFIG = {
+    "imprintUrl": "https://pub.solar/about",
+    "privacyPolicyUrl": "https://pub.solar/privacy",
+    "mapHome": {"zoom": 12, "latitude": 50.93, "longitude": 6.97},
+    "banner": {
+        "text": "This is an installation serving the Cologne/Bonn region run for Team OBSKöln by pub.solar n.e.V.",
+        "style": "info"
+    },
+}
+TILES_FILE = None
+ADDITIONAL_CORS_ORIGINS = None
+  '';
+
+  env = {
+    OBS_KEYCLOAK_URI = "auth.pub.solar";
+    OBS_PORTAL_URI = "obs-portal.pub.solar";
+
+    OBS_POSTGRES_MAX_OVERFLOW = "20";
+    OBS_POSTGRES_POOL_SIZE = "40";
+
+    OBS_HOST = "0.0.0.0";
+    OBS_PORT = "3000";
+    OBS_KEYCLOAK_URL = "https://auth.pub.solar/realms/pub.solar/";
+    OBS_KEYCLOAK_CLIENT_ID = "openbikesensor-portal";
+    OBS_DEDICATED_WORKER = "True";
+    OBS_DATA_DIR = "/data";
+    OBS_PROXIES_COUNT = "1";
+  };
+in {
+  age.secrets.obs-portal-env = {
+    file = "${flake.self}/secrets/obs-portal-env.age";
+    mode = "600";
+  };
+
+  age.secrets.obs-portal-database-env = {
+    file = "${flake.self}/secrets/obs-portal-database-env.age";
+    mode = "600";
+  };
+
+  systemd.services."docker-network-obs-portal" =
+    let
+      docker = config.virtualisation.oci-containers.backend;
+      dockerBin = "${pkgs.${docker}}/bin/${docker}";
+    in
+    {
+      serviceConfig.Type = "oneshot";
+      before = [ "docker-obs-portal.service" ];
+      script = ''
+        ${dockerBin} network inspect obs-portal-net >/dev/null 2>&1 || ${dockerBin} network create obs-portal-net --subnet 172.20.0.0/24
+      '';
+    };
+
+  services.nginx.virtualHosts."obs-portal.pub.solar" = {
+    enableACME = true;
+    forceSSL = true;
+
+    locations."/" = {
+      proxyWebsockets = true;
+      extraConfig = ''
+        proxy_pass http://127.0.0.1:3001;
+        proxy_set_header Host $host;
+      '';
+    };
+  };
+
+  virtualisation = {
+    oci-containers = {
+      backend = "docker";
+
+      containers."obs-portal" = {
+        image = "git.pub.solar/pub-solar/obs-portal:latest";
+        autoStart = true;
+        ports = [ "localhost:3001:${env.OBS_PORT}" ];
+
+        environment = env;
+        environmentFiles = [ config.age.secrets.obs-portal-env.path ];
+
+        volumes = [
+          "${configPy}:/opt/obs/api/config.py"
+          "/var/lib/obs-portal${env.OBS_DATA_DIR}:${env.OBS_DATA_DIR}"
+          "/var/lib/obs-portal/tiles/:/tiles"
+          "/var/lib/obs-portal/pbf/:/pbf"
+        ];
+
+        extraOptions = [
+          "--network=obs-portal-net"
+        ];
+      };
+
+      containers."obs-portal-worker" = {
+        image = "git.pub.solar/pub-solar/obs-portal:latest";
+        autoStart = true;
+
+        cmd = [ "python" "tools/process_track.py" ];
+
+        environment = env;
+        environmentFiles = [ config.age.secrets.obs-portal-env.path ];
+
+        volumes = [
+          "${configPy}:/opt/obs/api/config.py"
+          "/var/lib/obs-portal${env.OBS_DATA_DIR}:${env.OBS_DATA_DIR}"
+        ];
+
+        extraOptions = [
+          "--network=obs-portal-net"
+        ];
+      };
+
+      containers."obs-portal-db" = {
+        image = "openmaptiles/postgis:7.0";
+        autoStart = true;
+
+        environmentFiles = [ config.age.secrets.obs-portal-database-env.path ];
+
+        volumes = [
+          "/var/lib/postgres-obs-portal/data:/var/lib/postgresql/data"
+        ];
+
+        extraOptions = [
+          "--network=obs-portal-net"
+        ];
+      };
+    };
+  };
+}

hosts/nachtigall/default.nix

@@ -8,6 +8,7 @@
       ./configuration.nix
 
       ./networking.nix
+      ./wireguard.nix
       ./backups.nix
       ./apps/nginx.nix
 
@@ -31,6 +32,7 @@
       ./apps/promtail.nix
       ./apps/searx.nix
       ./apps/tmate.nix
+      ./apps/obs-portal.nix
 
       ./apps/matrix/irc.nix
       ./apps/matrix/mautrix-telegram.nix

hosts/nachtigall/networking.nix

@@ -1,4 +1,8 @@
-{ config, pkgs, ... }:
+{
+  config,
+  pkgs,
+  flake,
+  ... }:
 {
 
   networking.hostName = "nachtigall";

hosts/nachtigall/wireguard.nix

@@ -0,0 +1,40 @@
+{
+  config,
+  pkgs,
+  flake,
+  ... }:
+{
+  networking.firewall.allowedUDPPorts = [ 51820 ];
+
+  age.secrets.wg-private-key.file = "${flake.self}/secrets/nachtigall-wg-private-key.age";
+
+  networking.wireguard.interfaces = {
+    wg-ssh = {
+      listenPort = 51820;
+      mtu = 1300;
+      ips = [
+        "10.7.6.1/32"
+        "fd00:fae:fae:fae:fae:1::/96"
+      ];
+      privateKeyFile = config.age.secrets.wg-private-key.path;
+      peers = flake.self.logins.admins.wireguardDevices ++ [
+        { # flora-6.pub.solar
+          endpoint = "80.71.153.210:51820";
+          publicKey = "jtSR5G2P/nm9s8WrVc26Xc/SQLupRxyXE+5eIeqlsTU=";
+          allowedIPs = [ "10.7.6.2/32" "fd00:fae:fae:fae:fae:2::/96" ];
+        }
+      ];
+    };
+  };
+
+  services.openssh.listenAddresses = [ 
+    {
+      addr = "10.7.6.1";
+      port = 22;
+    }
+    {
+      addr = "[fd00:fae:fae:fae:fae:1::]";
+      port = 22;
+    }
+  ];
+}

logins/admins.nix

@@ -0,0 +1,72 @@
+{
+  axeman = rec {
+    sshPubKeys = {
+      axeman-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMNeQYLFauAbzDyIbKC86NUh9yZfiyBm/BtIdkcpZnSU axeman@tuxnix";
+    };
+
+    secretEncryptionKeys = sshPubKeys;
+
+    wireguardDevices = [
+      {
+        # tuxnix
+        publicKey = "fTvULvdsc92binFaBV+uWwFi33bi8InShcaPnoxUZEA=";
+        allowedIPs = [ "10.7.6.203/32" "fd00:fae:fae:fae:fae:203::/96" ];
+      }
+    ];
+  };
+
+  b12f = rec {
+    sshPubKeys = {
+      b12f-gpg = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDVbUEOgciblRPOCaCkkwfYoKLjmJ6JKxnfg6MY7sN3W1/N4AsC27bvYPkYI66d4M3Ygi6nztaUrIIKBOPZrQtS0vx1jqosmcDwBMttNI7u4LdSDjGMEGB4zJdfR60HFuzpSNaBI/nKMWcAxr8v1KODy/mKTQ7fnMDN15OhvE7sAZe26B6IptUbG1DLuouezd4AW0OwQ3c6hVIuv5eF96OKrwFZ9XpNyYAashy8WTYqJWJRb71DV8oiqT9b3sN0Dy+7nUAPcLvJdwUDGjHQvnklgFUupKtrPhpRWqgJ41l4ebb1DCxmoL2zpdVohUK4eVC9ELdplvXtK+EJIJ1lKcDAYduYcxk//3+EdUDH0IkfXvz0Tomryu2BeyxURdMPzQh+ctHUWNI49tByx/mWrEqSu+XdgvtcumVg+jNUZKL9eA++xxuOan7H/OyshptLugZHd2e9JNM34NEOUEptq7LtHD5pEdXRV1ZT1IOsuSoDtdX14GeP2GSl21eKLnvSu9g8nGULIsx9hI3CrrlvvL9JU+Aymb4iEvqLhDeUNE643uYQad6P2SuK0kLQ/9Ny0z3y6bgglGn2uDUiAOPd8c+gFRRkMWvAWjWQi3iIR9TYBS4Z+CeYmUv8X2UCRcQPBn1wt69rvE9RcfHqRLZTUE5SpstQ0rXLinXmRA/WQV5Bdw== yubi-gpg";
+    };
+
+    secretEncryptionKeys = {
+      bbcom = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmXpOU6vzQiVSSYCoxHYv7wDxC63Qg3dxlAMR6AOzwIABCU5PFFNcO0NWYms/YR7MOViorl+19LCLRABar9JgHU1n+uqxKV6eGph3OPeMp5sN8LAh7C9N+TZj8iJzBxQ3ch+Z/LdmLRwYNJ7KSUI+gwGK6xRS3+z1022Y4P0G0sx7IeCBl4lealQEIIF10ZOfjUdBcLQar7XTc5AxyGKnHCerXHRtccCoadLQujk0AvPXbv3Ma4JwX9X++AnCWRWakqS5UInu2tGuZ/6Hrjd2a9AKWjTaBVDcbYqCvY4XVuMj2/A2bCceFBaoi41apybSk26FSFTU4qiEUNQ6lxeOwG4+1NCXyHe2bGI4VyoxinDYa8vLLzXIRfTRA0qoGfCweXNeWPf0jMqASkUKaSOH5Ot7O5ps34r0j9pWzavDid8QeKJPyhxKuF1a5G4iBEZ0O9vuti60dPSjJPci9oTxbune2/jb7Sa0yO06DtLFJ2ncr5f70s/BDxKk4XIwQLy+KsvzlQEGdY8yA6xv28bOGxL3sQ0HE2pDTsvIbAisVOKzdJeolStL9MM5W8Hg0r/KkGj2bg0TfoRp1xHV9hjKkvJrsQ6okaPvNFeZq0HXzPhWMOVQ+/46z80uaQ1ByRLr3FTwuWJ7F/73ndfxiq6bDE4z2Ji0vOjeWJm6HCxTdGw== hello@benjaminbaedorf.com";
+      yubi485 = "age1yubikey1qgxuu2x3uzw7k5pg5sp2dv43edhwdz3xuhj7kjqrnw0p8t0l67c5yz9nm6q";
+      yubi464 = "age1yubikey1qd7szmr9ux2znl4x4hzykkwaru60nr4ufu6kdd88sm7657gjz4x5w0jy4y7";
+    } // sshPubKeys;
+
+    wireguardDevices = [
+      { # stroopwafel
+        publicKey = "NNb7T8Jmn+V2dTZ8T6Fcq7hGomHGDckKoV3kK2oAhSE=";
+        allowedIPs = [ "10.7.6.200/32" "fd00:fae:fae:fae:fae:200::/96" ];
+      }
+    ];
+  };
+
+  hensoko = rec {
+    sshPubKeys = {
+      hensoko-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEbaQdxp7Flz6ttELe63rn+Nt9g43qJOLih6VCMP4gPb";
+      hensoko-2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAqkqMYgncrnczcW/0PY+Z+FmNXXpgw6D9JWTTwiainy";
+    };
+
+    secretEncryptionKeys = sshPubKeys;
+    wireguardDevices = [
+      { # judy
+        publicKey = "I+gN7v1VXkAGoSir6L8aebtLbguvy5nAx1QVDTzdckk=";
+        allowedIPs = [ "10.7.6.202/32" "fd00:fae:fae:fae:fae:202::/96" ];
+      }
+    ];
+  };
+
+  teutat3s = {
+    sshPubKeys = {
+      teutat3s-1 = "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFro/k4Mgqyh8yV/7Zwjc0dv60ZM7bROBU9JNd99P/4co6fxPt1pJiU/pEz2Dax/HODxgcO+jFZfvPEuLMCeAl0= YubiKey #10593996 PIV Slot 9a";
+    };
+
+    secretEncryptionKeys = {
+      teutat3s-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHcU6KPy4b1MQXd6EJhcYwbJu7E+0IrBZF/IP6T7gbMf teutat3s@dumpyourvms";
+    };
+
+    wireguardDevices = [
+      { # dumpyourvms
+        publicKey = "3UrVLQrwXnPAVXPiTAd7eM3fZYxnFSYgKAGpNMUwnUk=";
+        allowedIPs = [ "10.7.6.201/32" "fd00:fae:fae:fae:fae:201::/96" ];
+      }
+      { # ryzensun
+        publicKey = "oVF2/s7eIxyVjtG0MhKPx5SZ1JllZg+ZFVF2eVYtPGo=";
+        allowedIPs = [ "10.7.6.204/32" "fd00:fae:fae:fae:fae:204::/96" ];
+      }
+    ];
+  };
+}

logins/default.nix

@@ -0,0 +1,14 @@
+{ lib, ... }: let
+  admins = import ./admins.nix;
+  robots = import ./robots.nix;
+in {
+  flake = {
+    logins = {
+      admins = lib.lists.foldl (logins: adminConfig: {
+        sshPubKeys = logins.sshPubKeys ++ (lib.attrsets.attrValues adminConfig.sshPubKeys);
+        wireguardDevices = logins.wireguardDevices ++ (if adminConfig ? "wireguardDevices" then adminConfig.wireguardDevices else []);
+      }) { sshPubKeys = []; wireguardDevices = []; } (lib.attrsets.attrValues admins);
+      robots.sshPubKeys = lib.attrsets.attrValues robots;
+    };
+  };
+}

modules/networking.nix

@@ -1,6 +1,15 @@
-{ pkgs, ... }: {
+{ pkgs, lib, ... }: {
+  # Don't expose SSH via public interfaces
+  networking.firewall.interfaces.wg-ssh.allowedTCPPorts = [ 22 ];
+
+  networking.hosts = {
+      "10.7.6.1" = ["nachtigall.pub.solar"];
+      "10.7.6.2" = ["flora-6.pub.solar"];
+  };
+
   services.openssh = {
     enable = true;
+    openFirewall = lib.mkDefault false;
     settings = {
       PermitRootLogin = "prohibit-password";
       PasswordAuthentication = false;
@@ -27,14 +36,11 @@
 
   services.resolved = {
     enable = true;
-    # DNSSEC=false because of random SERVFAIL responses with Greenbaum DNS
-    # when using allow-downgrade, see https://github.com/systemd/systemd/issues/10579
     extraConfig = ''
       DNS=193.110.81.0#dns0.eu 185.253.5.0#dns0.eu 2a0f:fc80::#dns0.eu 2a0f:fc81::#dns0.eu 9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
       FallbackDNS=5.1.66.255#dot.ffmuc.net 185.150.99.255#dot.ffmuc.net 2001:678:e68:f000::#dot.ffmuc.net 2001:678:ed0:f000::#dot.ffmuc.net
       Domains=~.
       DNSOverTLS=yes
-      DNSSEC=false
     '';
   };
 }

modules/unlock-zfs-on-boot.nix

@@ -10,7 +10,7 @@
 
       # Please create this manually the first time.
       hostKeys = [ "/etc/secrets/initrd/ssh_host_ed25519_key" ];
-      authorizedKeys = flake.self.publicKeys.admins;
+      authorizedKeys = flake.self.logins.admins.sshPubKeys;
     };
     # this will automatically load the zfs password prompt on login
     # and kill the other prompt so boot can continue

modules/users.nix

@@ -4,12 +4,12 @@
     group = flake.self.username;
     extraGroups = [ "wheel" "docker" ];
     isNormalUser = true;
-    openssh.authorizedKeys.keys = flake.self.publicKeys.admins;
+    openssh.authorizedKeys.keys = flake.self.logins.admins.sshPubKeys;
   };
   users.groups.${flake.self.username} = { };
 
   # TODO: Remove when we stop locking ourselves out.
-  users.users.root.openssh.authorizedKeys.keys = flake.self.publicKeys.admins;
+  users.users.root.openssh.authorizedKeys.keys = flake.self.logins.admins.sshPubKeys;
 
   users.users.hakkonaut = {
     description = "CI and automation user";
@@ -19,7 +19,7 @@
     uid = 998;
     group = "hakkonaut";
     isSystemUser = true;
-    openssh.authorizedKeys.keys = flake.self.publicKeys.robots;
+    openssh.authorizedKeys.keys = flake.self.logins.robots.sshPubKeys;
   };
 
   users.groups.hakkonaut = { };

public-keys/admins.nix

@@ -1,13 +0,0 @@
-{
-  axeman-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMNeQYLFauAbzDyIbKC86NUh9yZfiyBm/BtIdkcpZnSU axeman@tuxnix";
-
-  b12f-yubi-backup = "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEST9eyAY3nzGYNnqDYfWHu+89LZsOjyKHMqCFvtP7vrgB7F7JbbECjdjAXEOfPDSCVwtMMpq8JJXeRMjpsD0rw= @b12f Yubi Backup";
-  b12f-gpg = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDVbUEOgciblRPOCaCkkwfYoKLjmJ6JKxnfg6MY7sN3W1/N4AsC27bvYPkYI66d4M3Ygi6nztaUrIIKBOPZrQtS0vx1jqosmcDwBMttNI7u4LdSDjGMEGB4zJdfR60HFuzpSNaBI/nKMWcAxr8v1KODy/mKTQ7fnMDN15OhvE7sAZe26B6IptUbG1DLuouezd4AW0OwQ3c6hVIuv5eF96OKrwFZ9XpNyYAashy8WTYqJWJRb71DV8oiqT9b3sN0Dy+7nUAPcLvJdwUDGjHQvnklgFUupKtrPhpRWqgJ41l4ebb1DCxmoL2zpdVohUK4eVC9ELdplvXtK+EJIJ1lKcDAYduYcxk//3+EdUDH0IkfXvz0Tomryu2BeyxURdMPzQh+ctHUWNI49tByx/mWrEqSu+XdgvtcumVg+jNUZKL9eA++xxuOan7H/OyshptLugZHd2e9JNM34NEOUEptq7LtHD5pEdXRV1ZT1IOsuSoDtdX14GeP2GSl21eKLnvSu9g8nGULIsx9hI3CrrlvvL9JU+Aymb4iEvqLhDeUNE643uYQad6P2SuK0kLQ/9Ny0z3y6bgglGn2uDUiAOPd8c+gFRRkMWvAWjWQi3iIR9TYBS4Z+CeYmUv8X2UCRcQPBn1wt69rvE9RcfHqRLZTUE5SpstQ0rXLinXmRA/WQV5Bdw== yubi-gpg";
-  b12f-464-fido2 = "sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIHUbowjUtBiOPWi+TCHGToFwIsMDY6s7IRev6buVVdWxAAAACHNzaDpiMTJm yubi@464";
-  b12f-485-fido2 = "sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIDyxaJNw0jXREOzQfa0E2RQE/xLD/VddDldbdSmS8uf9AAAACHNzaDpiMTJm yubi@485";
-
-  hensoko-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEbaQdxp7Flz6ttELe63rn+Nt9g43qJOLih6VCMP4gPb";
-  hensoko-2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAqkqMYgncrnczcW/0PY+Z+FmNXXpgw6D9JWTTwiainy";
-  teutat3s-1 = "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFro/k4Mgqyh8yV/7Zwjc0dv60ZM7bROBU9JNd99P/4co6fxPt1pJiU/pEz2Dax/HODxgcO+jFZfvPEuLMCeAl0= YubiKey #10593996 PIV Slot 9a";
-
-}

secrets/age-yubikey-464-identity.txt

@@ -0,0 +1 @@
+AGE-PLUGIN-YUBIKEY-1HZCCGQVZH5WV7DCL6V837

secrets/age-yubikey-485-identity.txt

@@ -0,0 +1 @@
+AGE-PLUGIN-YUBIKEY-1EKCCGQVZE64TLZCKYUCW7

secrets/flora6-wg-private-key.age

@@ -0,0 +1,42 @@
+age-encryption.org/v1
+-> ssh-ed25519 Y0ZZaw FvsdIE/inJoLVSosWXATnFbAAVjVuf7jlEC3nSUF6Ug
+gX84OKgWdfkGBN+NFy11BxIb4WX1z9UkPA4u2Q1uV+g
+-> ssh-ed25519 uYcDNw z5Veza0uVwqCqGCGYzGmXPcyaV9HztEN39cWFbSG7yg
+UWZQcDP1vMsYoWwMQlr4YmzWYw2EKm/s5zJVHNf2M0U
+-> ssh-rsa f5THog
+v1kqiU+cx65mvTNeuAhK65eBEk1vmkABRYgcmFIrdr4eY3pru+FaQTfMhTI9HjcO
+OTU0YPxxSadbUCaN6Z3QnTv5qowwOQlEsWK+RMsOZgnyRQHa2SIrhfHz7v+n8BTF
+8BYB4UBJpD3aLqM7VED6dYls178HUbiq34ohrG2vY5PHE72xTU60amv9NcJhSJPR
+twZPiSp3I14MlJU4bboS1YBaEmgxvbXru0DwuoQLw3OUrH7xOggVoSJxm8lVyjR2
+oFYS5wdnrhAIEsJ0lTsO5fvq9Dmie7qoL60rbBbue9lPk1nD1NlUe3akd4IIo36R
+kDbthUYluVSJON3o/wenSvJDOw3N3t8bu2+/XfWAd2NL9SPBijMQJtqjK8EAtmz9
+OjBMjJGQzVdBxRP9U3CWYIwaqYQfWhXXY4AXTwIMsfmeV8ZHZsId3Y156p0NaKg6
+NGb7eX/AWmcdNTp8ZCqlb4QexICrVd7XDkNbPHkYPUOdUhaMyS+T7YU8Qs3YWroP
+Bw63QMWbvo1l4HO/3HeIKlzIXTjLEi6PjTiWb7vM4GuoCwjdDg5djMEj4nsvDyea
+B9EBTEcoP2oj47wgsX0nfV5bKAQ4y8AN4ZNWb00vjN9ybBbLK3q//1DrEWmddieF
+t6FyZXvZH0Gf6y5OO56yRp/vmxvKFcvxqUA3P8bPAnw
+-> ssh-rsa kFDS0A
+c+0wRUbjzdJiBhdKAVlE8yxt1O3t4oQ438F5HjMPohEXSFLiNFi4Y0JQsw6qn3GP
+hySsyIoj9G+cI9FDPjTFPmE7O1SHrd2LqBZGukyswDXX8CpwmZ7vfqfK2lCgKfos
+SSPiGaYk+HlQF2QfX/xdgQ2PbFXHnDy8LZ9AfZP04PrnK9wqdiEXwmkWZ/Lu1P+V
+Wb/28BYxcfkseAprFr/KSJLoNuD9UphRhQwRklmjADnf0lep3vHccxz1Oo5flu5M
+AD47r+0bLGM+w3epCF1GyR4L2lEBaD8pkVOt3/zIdjn8nFZVNJwjshToazvnVEd3
+Vd9Uas58AyxcT7Dk/QaVO7c5KJDdfSuxnT1zElkM2ZQM4lEueTJYDBJGyfubb30y
+Z7re/MsLOh0jNJbb0r1KOkzwpcdm9iyvi26eaGsX7Q1Gb2pzOYFxD1vSUUC6A6Hp
+W5X6fKsiBPreYLf5MV6p9r2YJPdX4SJiq4XztQi1PL+ndq1h8wskxk3Pyvk9fhle
+iC5owZ8/FikfC/1oEa2KayeLyYB001BUuktevzfH2GmbqLkR9wBGw5vUJzOO4vOW
+o8SVCSUxSrG8S+HQksOSXFWywkdBDhqc8eyRUtb+6iqqMA2Q4GDqktSCB1KeBYD6
+OalH6bo4H1ddV8LPMOKcFtjmTPuum43C7bNge2rxhgg
+-> piv-p256 vRzPNw A/utfOjPG1zs1Lf2FOWDHhJIJW1PIHmKFqFvBZZycHPn
+EfGFh9R0PDgskQg00z6thQ1YozT5ZiBhzNN9iTXWDe4
+-> piv-p256 zqq/iw A0RjdOkfYmTlYCwM3aFLdXfBimXMGzVh21A5QxZ217xW
+7J9cRYpr1uhQPE0VjvLAwyS7jNSK0+qjA9xUMeRwYos
+-> ssh-ed25519 YFSOsg w8ljrS1oRdB9RT8Odi5UOPjEtFL3WBlQUAH9Y7gp3WM
+xcrbEm66K6mNrJ9+877YEgWUdxW85YyS1z8CGMyYxeE
+-> ssh-ed25519 iHV63A O0bMGpauAYAuiAtbITj+lQOS0LuFl/BDVxIUTly8tQM
+0Kiu4sNN0joX5D4eB42oQ/iRSntsJI5JNKOmkQeyLGE
+-> ssh-ed25519 BVsyTA k/0Rtr9qbFH7V6DyCRtyqdAHU1b7D7DNGV8pPPJmrnk
+dJ29gcfSxaVQ46XbW021PxPotZ8ZG2zjostJme9GUZQ
+--- 1V0sJP5JIa9GZ0F0hf1GAFX3LNkPSNsxNhqM9cH7Rgc
+|¿‘#ø©mÌæR„Ö5wä­ÎQòÅÐf1Ü
ÑÁZ·MUüèOÃfãÜ:GÓ^<5E>ì•!<21>
+gÐG29wíƒÙ_B‘­ìdêÿ

secrets/nachtigall-wg-private-key.age

@@ -0,0 +1,41 @@
+age-encryption.org/v1
+-> ssh-ed25519 iDKjwg 1m2Nkhw2R1InZFZrOkzQCoQy4s/kduXyM44yWifllXc
+cxz6EWfaIJUjEkXEExFGKPrrl4iXnchkFfMiCpDgnZ8
+-> ssh-ed25519 uYcDNw nVtsI77gUtZKmvu6o/jkvh/Ab8KDgRuL7V6MDuFtBnk
+P7xVJA1a1ioe2tROajY1uvS1kLGrJW+YrXVf2Z2K2a4
+-> ssh-rsa f5THog
+rr17rPe7lJ2Zc0nsHhEch7mG7D27lnaMbAJ2Zsn4oHDXFn4cnSw4L/Zf+aZVIpNY
+ew2u24yBE5ButBh8t0wm2Di2SBir4cAQob7160Py5ZpqOHBGxACgxhfZm7f/FzLZ
+Ue0CUKebJI8KAqkjyayLLzESMECT5buhoJ4+K8U/B6O8NgGPrjS1Xjx1zCAs8tsG
+kQz2KsBFnIEH20qmj2ezmijJdkUJbyX2389jCIzZ95wOG0RcUH1+s0aMcuvvLptS
+05nSlmOlnwv7M8Jkwg+BC6l6xpoG3zpQDReEBTT3DYMRL3sNPV9eIHcPrWIXlANk
+7vqLPxNlu/gHhQSijcPICH0YiDZ3MIJdXtqVHxCFWmXlPAzfkSMwg2k3WT8fMSJ9
+ajEM0i6AIjaNAeY6cY87kGmfSjwRTSEbDSkC0B5VV1h2CZJDot7+9eZQ1HcwnP3j
+iLTijtB+dMAzpnQ8kA9bGnuOurTB3Jy+JxwejO21J1/rxBA+P0nATufnk5olhTKS
+vqkor0rxkV379SMpHLpbg4IbwdIjp+77GDJkofcAxZI8tmU2IF19dC1UsDfz15N/
+b984i7PpJ115U2oSbwBZ8WThx1i8I47/mabTU32IXvhfdsp9QmBoBIqUqdgHsU43
+LSBHRHiMy+3BfNA0M52oWEThtScOeqzwo3oSBCTM3xI
+-> ssh-rsa kFDS0A
+fgw9rO7pT7MLo1nNvZ05Ry+Gyjb27Trc5kZ7KYYya1BpCKjLnYwOaaoLtoHkGnuz
+bPJ4ouyMsWUiPpT/SZ5/uNHlSDS9dNF0RTzCAqSi31CwY5KFTfStzsOKeUvxCcGp
+Z9uyOEr1sOl1+gORWphrHmllSrXFAHHgOorLrtACkrQMxn678Wko13CFvDhtkl+l
+sqi+l+B5ffeJsaHmCLmrROGzWrCnT/1zwJV5KMF0HjBSOi+Fl+HxA9s6UCEHxTy/
+H8GvOooDGczgjg06yI2Puzo+DvhE/XOeFOoM/cLdGPnq/R8Mo4r4BDeBnQqbbBCI
+4LV0Ybz0jVpAHHCu5kAxIzc68d1mwmxYPW4pxMVDGaZKGoBnA9jkHA0DD0TKe62D
+ZBWtKAZb3gD4yDZfcbZABuXFszmFzKRmoE8YLmZDw0GwLu/It+ZtL9cxUZ+YmknP
+ZhBcy1NTlPhXlJdZBWImK8KKluf03BjBIAFm+ZGT1FiCnZft5SZFDf7PGq+PvRwT
+wk6UMeBiVbJvpVtjthHbur5FxXG+ly9wa9Y5bP3K2VnJkVcVt6NhkJ6Hg+g2FIZ4
+gzq+5azkX+7nSNr0dSR1Phk4j+6aahRc2Gb7SiMqo6nwKuWBL6SQRDuKwP1PaPvm
+aGfsduWhKZQM5ZeXBYkdgQqLgx4oAgbI2SujRaJlykE
+-> piv-p256 vRzPNw A77uRo1hsdtaU8Fze62NI3AocU7srSmd5A7y1PbUVEyQ
+LgD5sj6ZGGYiDausGO5lxERV71MFkZltzP3W4JIK59M
+-> piv-p256 zqq/iw A7rWVvgXoLOrF3w8wyR27/fGAPxeknuBMVF1yeNceSkN
+qAe7DwmCiFz72fy0Ica3SWZYNyvlsE1M/Odma5FKlyI
+-> ssh-ed25519 YFSOsg Hld4L4nxmssu+4vwIEE4Q13Xapfn38R42+MdT3c5Jyg
+gW3YzRgpc8SKyTp6o4BqmqFurr+lak+hKvYLFGdm2s8
+-> ssh-ed25519 iHV63A ODXmcURhm3oMgB5t4kigz1LoXMl0IqG7zUUog0FXRDw
+pa37B1B4FFTrh4UHDh2O4VBSQyxlaozHDNR8PCQ+gis
+-> ssh-ed25519 BVsyTA 1dkpnnRlhnqueC91EW7xn/q4MUUvleN23KyiTJM1ZlI
+QvpM4QaFx4ey3EZ8TNnbJjdeIgR5Nfbugw3X2Xv27wY
+--- dHSohj4s4bp6X8I2em011HuWwNNIDis6h4e/44CnTIU
+€Ð·^Pvî
^4YYpµä'äå}Xób½q5°½âW¦ ˜nv‹îß°B=í÷³¿ƒÐ÷*Å%Ñþ‹<C3BE>Ù¡nãÕi˜ÖÔT²]

secrets/obs-portal-database-env.age

@@ -0,0 +1,27 @@
+age-encryption.org/v1
+-> ssh-ed25519 iDKjwg hAoEiOaK1U0HImALePEYHiE6xebOOqtVujaBWgNBZF8
+ecf/ykqYPihRJxI/Y7Oh6QhWSyncwevlzEZoRqm3aGM
+-> ssh-ed25519 uYcDNw NcIttsTn6wPCmoOYGtZ66IYhthjLDI3sYFe4pbW6cB4
+9hv4dEYoXXWSZ2pG1hy68vmTf++v+g3q7wVhT6cAog0
+-> ssh-rsa kFDS0A
+KoW3J2Tw90chM6Oy17umOQN0WFI4je7CBk3IgdImsd4Mz5q17/nXlhVlFFhx4ZEk
+Or9LaqytVk1NA6J4+suMRlx4Pd6oberXu1KBkFQMr1B3LKhNOaOZ+W1mrbQLGG9U
+YUTyOpkHxVkw0IOsvxB/0reMCHtjKHo661zFjim1YFmEk0WRt4hU1XqsMNiE4wbc
+GF0t9EWMN2pU2p7DpX/DzVTqu8yk8SQhCZc9kfzWcuawwf0rcjwUJ/Rk1MH5tMpK
+odRXXl1slPPwQinE+KJqeyrfuRDHqwqmxnOfOWG6KQwWkVSE1btiHEvfuuLOjSjl
+3wO+veRC9hW5sSCPANoFbuSQ1dprmoyaZnOyeRTbgw91ks/ogLBezF/KSkaMQeHx
+XRnfcceBmeeqHl9L3Z+3EmBjwIqu2Og0pvhDU8G/ZeA0cHS/22QYGzeD/gOqaEW7
+d1VyA6LZd8PxIjoBamdipIpY0TqZ8+cA/yaUKNnYXXRSlKQ5ggPxh7ZXfvRbGg+m
+WbNiHxBPcTK7/Bpzes4LJVcx0Ar4XeDxVQe1MITLpFWh+FDEQZEA3630JngZ153J
+vBvw+VFedPSr6Ov+/33/J3LKC0XRatGnc++AWfo4rWPLCE6qovEDyY+wmct8gv0j
+rMEK7OaNfyy+Z21mjrkwcEUbyoGt9ksEplaRblE0Lsk
+-> ssh-ed25519 YFSOsg LmLRtBYMSzjid3VkUgAQvDOS9r0imWSKE7fm0t/x41Y
+0mae0vsNmaS5aVOKezXit7KV44JKLpU+GWpuA++dCVo
+-> ssh-ed25519 iHV63A Tc2z2JciftAikoj4Hv9IBgkcYWAcyGuPJTNA3Yw2K1w
+cO5o/pbaZAtTvXUskOah9vWP/Tuvyi3QDM7g4AQ+b8s
+-> ssh-ed25519 BVsyTA mk6n6ytaI4V9JVoUZFtwfFOgaLYc6gvVOcSZXQj/FVI
+etqbUCqe0eY81qaVco7pMJjhfM+sA/bXLMW0bEsCLxI
+--- CmNq6ZPxFoFTsySVfr7BTHV0tm9cbRYGG6IR7DNgbEY
+!è烈í}
+ùSê<>ŸSl®Ds;!ÁjršZçR"—ë#ž­¿»ÙÅ~!›Ÿ¤6AùwEn ? kËAcx~—ŽÜGVæ&M¯ý¾ä,
+a›U

secrets/secrets.nix

@@ -1,21 +1,10 @@
 let
-  # set ssh public keys here for your system and user
-  axeman-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMNeQYLFauAbzDyIbKC86NUh9yZfiyBm/BtIdkcpZnSU axeman@tuxnix";
-  b12f-bbcom = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmXpOU6vzQiVSSYCoxHYv7wDxC63Qg3dxlAMR6AOzwIABCU5PFFNcO0NWYms/YR7MOViorl+19LCLRABar9JgHU1n+uqxKV6eGph3OPeMp5sN8LAh7C9N+TZj8iJzBxQ3ch+Z/LdmLRwYNJ7KSUI+gwGK6xRS3+z1022Y4P0G0sx7IeCBl4lealQEIIF10ZOfjUdBcLQar7XTc5AxyGKnHCerXHRtccCoadLQujk0AvPXbv3Ma4JwX9X++AnCWRWakqS5UInu2tGuZ/6Hrjd2a9AKWjTaBVDcbYqCvY4XVuMj2/A2bCceFBaoi41apybSk26FSFTU4qiEUNQ6lxeOwG4+1NCXyHe2bGI4VyoxinDYa8vLLzXIRfTRA0qoGfCweXNeWPf0jMqASkUKaSOH5Ot7O5ps34r0j9pWzavDid8QeKJPyhxKuF1a5G4iBEZ0O9vuti60dPSjJPci9oTxbune2/jb7Sa0yO06DtLFJ2ncr5f70s/BDxKk4XIwQLy+KsvzlQEGdY8yA6xv28bOGxL3sQ0HE2pDTsvIbAisVOKzdJeolStL9MM5W8Hg0r/KkGj2bg0TfoRp1xHV9hjKkvJrsQ6okaPvNFeZq0HXzPhWMOVQ+/46z80uaQ1ByRLr3FTwuWJ7F/73ndfxiq6bDE4z2Ji0vOjeWJm6HCxTdGw== hello@benjaminbaedorf.com";
-  hensoko-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEbaQdxp7Flz6ttELe63rn+Nt9g43qJOLih6VCMP4gPb";
-  hensoko-2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAqkqMYgncrnczcW/0PY+Z+FmNXXpgw6D9JWTTwiainy";
-  teutat3s-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHcU6KPy4b1MQXd6EJhcYwbJu7E+0IrBZF/IP6T7gbMf teutat3s@dumpyourvms";
+  admins = import ../logins/admins.nix;
 
   nachtigall-host = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP7G0ufi+MNvaAZLDgpieHrABPGN7e/kD5kMFwSk4ABj root@nachtigall";
   flora-6-host = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGP1InpTBN4AlF/4V8HHumAMLJzeO8DpzjUv9Co/+J09 root@flora-6";
 
-  baseKeys = [
-    axeman-1
-    b12f-bbcom
-    hensoko-1
-    hensoko-2
-    teutat3s-1
-  ];
+  adminKeys = builtins.foldl' (keys: login: keys ++ (builtins.attrValues login.secretEncryptionKeys)) [] (builtins.attrValues admins);
 
   nachtigallKeys = [
     nachtigall-host
@@ -27,48 +16,55 @@ let
 in
 {
   # ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBB5XaH02a6+TchnyQED2VwaltPgeFCbildbE2h6nF5e root@nachtigall
-  "nachtigall-root-ssh-key.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "nachtigall-root-ssh-key.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "mastodon-secret-key-base.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "mastodon-otp-secret.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "mastodon-vapid-private-key.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "mastodon-vapid-public-key.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "mastodon-smtp-password.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "mastodon-extra-env-secrets.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "nachtigall-wg-private-key.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "flora6-wg-private-key.age".publicKeys = flora6Keys ++ adminKeys;
 
-  "keycloak-database-password.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "mastodon-secret-key-base.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "mastodon-otp-secret.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "mastodon-vapid-private-key.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "mastodon-vapid-public-key.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "mastodon-smtp-password.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "mastodon-extra-env-secrets.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "forgejo-actions-runner-token.age".publicKeys = flora6Keys ++ baseKeys;
-  "forgejo-database-password.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "forgejo-mailer-password.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "keycloak-database-password.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "matrix-mautrix-telegram-env-file.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "matrix-synapse-signing-key.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "matrix-synapse-secret-config.yaml.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "matrix-synapse-sliding-sync-secret.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "forgejo-actions-runner-token.age".publicKeys = flora6Keys ++ adminKeys;
+  "forgejo-database-password.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "forgejo-mailer-password.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "forgejo-ssh-private-key.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "nextcloud-secrets.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "nextcloud-admin-pass.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "matrix-mautrix-telegram-env-file.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "matrix-synapse-signing-key.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "matrix-synapse-secret-config.yaml.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "matrix-synapse-sliding-sync-secret.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "searx-environment.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "nextcloud-secrets.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "nextcloud-admin-pass.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "restic-repo-droppie.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "restic-repo-storagebox.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "searx-environment.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "drone-db-secrets.age".publicKeys = flora6Keys ++ baseKeys;
-  "drone-secrets.age".publicKeys = flora6Keys ++ baseKeys;
+  "restic-repo-droppie.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "restic-repo-storagebox.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "mediawiki-database-password.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "mediawiki-admin-password.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "mediawiki-oidc-client-secret.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "mediawiki-secret-key.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "drone-db-secrets.age".publicKeys = flora6Keys ++ adminKeys;
+  "drone-secrets.age".publicKeys = flora6Keys ++ adminKeys;
 
-  "coturn-static-auth-secret.age".publicKeys = nachtigallKeys ++ baseKeys;
+  "mediawiki-database-password.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "mediawiki-admin-password.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "mediawiki-oidc-client-secret.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "mediawiki-secret-key.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "grafana-admin-password.age".publicKeys = flora6Keys ++ baseKeys;
-  "grafana-keycloak-client-secret.age".publicKeys = flora6Keys ++ baseKeys;
-  "grafana-smtp-password.age".publicKeys = flora6Keys ++ baseKeys;
+  "coturn-static-auth-secret.age".publicKeys = nachtigallKeys ++ adminKeys;
 
-  "nachtigall-metrics-nginx-basic-auth.age".publicKeys = nachtigallKeys ++ baseKeys;
-  "nachtigall-metrics-prometheus-basic-auth-password.age".publicKeys = flora6Keys ++ nachtigallKeys ++ baseKeys;
+  "grafana-admin-password.age".publicKeys = flora6Keys ++ adminKeys;
+  "grafana-keycloak-client-secret.age".publicKeys = flora6Keys ++ adminKeys;
+  "grafana-smtp-password.age".publicKeys = flora6Keys ++ adminKeys;
+
+  "nachtigall-metrics-nginx-basic-auth.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "nachtigall-metrics-prometheus-basic-auth-password.age".publicKeys = flora6Keys ++ nachtigallKeys ++ adminKeys;
+
+  "obs-portal-env.age".publicKeys = nachtigallKeys ++ adminKeys;
+  "obs-portal-database-env.age".publicKeys = nachtigallKeys ++ adminKeys;
 }