diff --git a/.github/workflows/matrix.yml b/.github/workflows/matrix.yml index 281b1496..2f9e7953 100644 --- a/.github/workflows/matrix.yml +++ b/.github/workflows/matrix.yml @@ -13,7 +13,7 @@ jobs: - name: Check out uses: actions/checkout@v3 - name: Run yamllint - uses: frenck/action-yamllint@v1.3.1 + uses: frenck/action-yamllint@v1.4.0 ansible-lint: name: ansible-lint runs-on: ubuntu-latest @@ -21,6 +21,6 @@ jobs: - name: Check out uses: actions/checkout@v3 - name: Run ansible-lint - uses: ansible-community/ansible-lint-action@v6.10.2 + uses: ansible-community/ansible-lint-action@v6.11.0 with: path: roles/custom diff --git a/CHANGELOG.md b/CHANGELOG.md index 3be19f00..69e932a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,112 @@ +# 2023-02-10 + +## Draupnir moderation tool (bot) support + +Thanks to [FSG-Cat](https://github.com/FSG-Cat), the playbook can now install and configure the [Draupnir](https://github.com/Gnuxie/Draupnir) moderation tool (bot). Draupnir is a fork of [Mjolnir](docs/configuring-playbook-bot-mjolnir.md) (which the playbook has supported for a long time) maintained by Mjolnir's former lead developer. + +Additional details are available in [Setting up Draupnir](docs/configuring-playbook-bot-draupnir.md). + + +# 2023-02-05 + +## The matrix-prometheus-postgres-exporter role lives independently now + +**TLDR**: the `matrix-prometheus-postgres-exporter` role is now included from another repository. Some variables have been renamed. All functionality remains intact. + +The `matrix-prometheus-postgres-exporter` role (which configures [Prometheus Postgres Exporter](https://github.com/prometheus-community/postgres_exporter)) has been extracted from the playbook and now lives in its own repository at https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter. + +It's still part of the playbook, but is now installed via `ansible-galaxy` (by running `just roles` / `make roles`). Some variables have been renamed (`matrix_prometheus_postgres_exporter_` -> `prometheus_postgres_exporter_`, etc.). The playbook will report all variables that you need to rename to get upgraded. All functionality remains intact. + +The `matrix-prometheus-services-proxy-connect` role has bee adjusted to help integrate the new `prometheus_postgres_exporter` role with our own services (`matrix-nginx-proxy`) + +Other roles which aren't strictly related to Matrix are likely to follow this fate of moving to their own repositories. Extracting them out allows other Ansible playbooks to make use of these roles easily. + + +# 2023-01-26 + +## Coturn can now use host-networking + +Large Coturn deployments (with a huge range of ports specified via `matrix_coturn_turn_udp_min_port` and `matrix_coturn_turn_udp_max_port`) experience a huge slowdown with how Docker publishes all these ports (setting up firewall forwarding rules), which leads to a very slow Coturn service startup and shutdown. + +Such deployments don't need to run Coturn within a private container network anymore. Coturn can now run with host-networking by using configuration like this: + +```yaml +matrix_coturn_docker_network: host +``` + +With such a configuration, **Docker no longer needs to configure thousands of firewall forwarding rules** each time Coturn starts and stops. +This, however, means that **you will need to ensure these ports are open** in your firewall yourself. + +Thanks to us [tightening Coturn security](#backward-compatibility-tightening-coturn-security-can-lead-to-connectivity-issues), running Coturn with host-networking should be safe and not expose neither other services running on the host, nor other services running on the local network. + + +## (Backward Compatibility) Tightening Coturn security can lead to connectivity issues + +**TLDR**: users who run and access their Matrix server on a private network (likely a small minority of users) may experience connectivity issues with our new default Coturn blocklists. They may need to override `matrix_coturn_denied_peer_ips` and remove some IP ranges from it. + +Inspired by [this security article](https://www.rtcsec.com/article/cve-2020-26262-bypass-of-coturns-access-control-protection/), we've decided to make use of Coturn's `denied-peer-ip` functionality to prevent relaying network traffic to certain private IP subnets. This ensures that your Coturn server won't accidentally try to forward traffic to certain services running on your local networks. We run Coturn in a container and in a private container network by default, which should prevent such access anyway, but having additional block layers in place is better. + +If you access your Matrix server from a local network and need Coturn to relay to private IP addresses, you may observe that relaying is now blocked due to our new default `denied-peer-ip` lists (specified in `matrix_coturn_denied_peer_ips`). If you experience such connectivity problems, consider overriding this setting in your `vars.yml` file and removing certain networks from it. + +We've also added `no-multicast-peers` to the default Coturn configuration, but we don't expect this to cause trouble for most people. + + +# 2023-01-21 + +## The matrix-prometheus-node-exporter role lives independently now + +**TLDR**: the `matrix-prometheus-node-exporter` role is now included from another repository. Some variables have been renamed. All functionality remains intact. + +The `matrix-prometheus-node-exporter` role (which configures [Prometheus node exporter](https://github.com/prometheus/node_exporter)) has been extracted from the playbook and now lives in its own repository at https://gitlab.com/etke.cc/roles/prometheus_node_exporter. + +It's still part of the playbook, but is now installed via `ansible-galaxy` (by running `just roles` / `make roles`). Some variables have been renamed (`matrix_prometheus_node_exporter_` -> `prometheus_node_exporter_`, etc.). The playbook will report all variables that you need to rename to get upgraded. All functionality remains intact. + +A new `matrix-prometheus-services-proxy-connect` role was added to the playbook to help integrate the new `prometheus_node_exporter` role with our own services (`matrix-nginx-proxy`) + +Other roles which aren't strictly related to Matrix are likely to follow this fate of moving to their own repositories. Extracting them out allows other Ansible playbooks to make use of these roles easily. + + +# 2023-01-13 + +## Support for running commands via just + +We've previously used [make](https://www.gnu.org/software/make/) for easily running some playbook commands (e.g. `make roles` which triggers `ansible-galaxy`, see [Makefile](Makefile)). +Our `Makefile` is still around and you can still run these commands. + +In addition, we've added support for running commands via [just](https://github.com/casey/just) - a more modern command-runner alternative to `make`. Instead of `make roles`, you can now run `just roles` to accomplish the same. + +Our [justfile](justfile) already defines some additional helpful **shortcut** commands that weren't part of our `Makefile`. Here are some examples: + +- `just install-all` to trigger the much longer `ansible-playbook -i inventory/hosts setup.yml --tags=install-all,ensure-matrix-users-created,start` command +- `just install-all --ask-vault-pass` - commands also support additional arguments (`--ask-vault-pass` will be appended to the above installation command) +- `just run-tags install-mautrix-slack,start` - to run specific playbook tags +- `just start-all` - (re-)starts all services +- `just stop-group postgres` - to stop only the Postgres service +- `just register-user john secret-password yes` - registers a `john` user with the `secret-password` password and admin access (admin = `yes`) + +Additional helpful commands and shortcuts may be defined in the future. + +This is all completely optional. If you find it difficult to [install `just`](https://github.com/casey/just#installation) or don't find any of this convenient, feel free to run all commands manually. + + +# 2023-01-11 + +## mautrix-slack support + +Thanks to [Cody Neiman](https://github.com/xangelix)'s efforts, the playbook now supports bridging to [Slack](https://slack.com/) via the [mautrix-slack](https://mau.dev/mautrix/slack) bridge. See our [Setting up Mautrix Slack bridging](docs/configuring-playbook-bridge-mautrix-slack.md) documentation page for getting started. + +**Note**: this is a new Slack bridge. The playbook still retains Slack bridging via [matrix-appservice-slack](docs/configuring-playbook-bridge-appservice-slack.md) and [mx-puppet-slack](docs/configuring-playbook-bridge-mx-puppet-slack.md). You're free to use the bridge that serves you better, or even all three of them (for different users and use-cases). + + +# 2023-01-10 + +## ChatGPT support + +Thanks to [@bertybuttface](https://github.com/bertybuttface), the playbook can now help you set up [matrix-chatgpt-bot](https://github.com/matrixgpt/matrix-chatgpt-bot) - a bot through which you can talk to the [ChatGPT](https://openai.com/blog/chatgpt/) model. + +See our [Setting up matrix-bot-chatgpt](docs/configuring-playbook-bot-chatgpt.md) documentation to get started. + + # 2022-11-30 ## matrix-postgres-backup has been replaced by the com.devture.ansible.role.postgres_backup external role @@ -353,7 +462,7 @@ matrix_homeserver_implementation: conduit Thanks to [MdotAmaan](https://github.com/MdotAmaan)'s efforts, the playbook now supports bridging to [Discord](https://discordapp.com/) via the [mautrix-discord](https://mau.dev/mautrix/discord) bridge. See our [Setting up Mautrix Discord bridging](docs/configuring-playbook-bridge-mautrix-discord.md) documentation page for getting started. -**Note**: this is a new Discord bridge. The playbook still retains Discord bridging via [matrix-appservice-discord](docs/configuring-playbook-bridge-appservice-discord.md) and [mx-puppet-discord](docs/configuring-playbook-bridge-mx-puppet-discord.md). You're free too use the bridge that serves you better, or even all three of them (for different users and use-cases). +**Note**: this is a new Discord bridge. The playbook still retains Discord bridging via [matrix-appservice-discord](docs/configuring-playbook-bridge-appservice-discord.md) and [mx-puppet-discord](docs/configuring-playbook-bridge-mx-puppet-discord.md). You're free to use the bridge that serves you better, or even all three of them (for different users and use-cases). # 2022-07-27 @@ -434,7 +543,7 @@ See our [Setting up the ntfy push notifications server](docs/configuring-playboo **If you're using node-exporter** (`matrix_prometheus_node_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_node_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/node-exporter`. -**If you're using [postgres-exporter](docs/configuring-playbook-prometheus-postgres.md)** (`matrix_prometheus_postgres_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_postgres_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/postgres-exporter`. +**If you're using [postgres-exporter](docs/configuring-playbook-prometheus-postgres.md)** (`prometheus_postgres_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/postgres-exporter`. **If you're using Synapse** and would like to collect its metrics from an external Prometheus server, you may find that: diff --git a/README.md b/README.md index 177c2a0d..94105841 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,8 @@ Bridges can be used to connect your matrix installation with third-party communi | Name | Default? | Description | Documentation | | ---- | -------- | ----------- | ------------- | -[mautrix-discord](https://github.com/mautrix/discord) | x | Bridge for bridging your Matrix server to [Discord](https://discord.com/) | [Link](docs/configuring-playbook-bridge-mautrix-discord.md) | +| [mautrix-discord](https://github.com/mautrix/discord) | x | Bridge for bridging your Matrix server to [Discord](https://discord.com/) | [Link](docs/configuring-playbook-bridge-mautrix-discord.md) | +| [mautrix-slack](https://github.com/mautrix/slack) | x | Bridge for bridging your Matrix server to [Slack](https://slack.com/) | [Link](docs/configuring-playbook-bridge-mautrix-slack.md) | | [mautrix-telegram](https://github.com/mautrix/telegram) | x | Bridge for bridging your Matrix server to [Telegram](https://telegram.org/) | [Link](docs/configuring-playbook-bridge-mautrix-telegram.md) | | [mautrix-whatsapp](https://github.com/mautrix/whatsapp) | x | Bridge for bridging your Matrix server to [WhatsApp](https://www.whatsapp.com/) | [Link](docs/configuring-playbook-bridge-mautrix-whatsapp.md) | | [mautrix-facebook](https://github.com/mautrix/facebook) | x | Bridge for bridging your Matrix server to [Facebook](https://facebook.com/) | [Link](docs/configuring-playbook-bridge-mautrix-facebook.md) | @@ -130,6 +131,7 @@ Bots provide various additional functionality to your installation. | [Go-NEB](https://github.com/matrix-org/go-neb) | x | A multi functional bot written in Go | [Link](docs/configuring-playbook-bot-go-neb.md) | | [Mjolnir](https://github.com/matrix-org/mjolnir) | x | A moderation tool for Matrix | [Link](docs/configuring-playbook-bot-mjolnir.md) | | [Buscarron](https://gitlab.com/etke.cc/buscarron) | x | Web forms (HTTP POST) to matrix | [Link](docs/configuring-playbook-bot-buscarron.md) | +| [matrix-chatgpt-bot](https://github.com/matrixgpt/matrix-chatgpt-bot) | x | ChatGPT from matrix | [Link](docs/configuring-playbook-bot-chatgpt.md) | ### Administration diff --git a/docs/configuring-playbook-bot-chatgpt.md b/docs/configuring-playbook-bot-chatgpt.md new file mode 100644 index 00000000..7a006816 --- /dev/null +++ b/docs/configuring-playbook-bot-chatgpt.md @@ -0,0 +1,62 @@ +# Setting up ChatGPT (optional) + +The playbook can install and configure [matrix-chatgpt-bot](https://github.com/matrixgpt/matrix-chatgpt-bot) for you. + +Talk to [ChatGPT](https://openai.com/blog/chatgpt/) via your favourite Matrix client! + + +## 1. Register the bot account + +The playbook does not automatically create users for you. The bot requires an access token to be able to connect to your homeserver. + +You **need to register the bot user manually** before setting up the bot. + +Choose a strong password for the bot. You can generate a good password with a command like this: `pwgen -s 64 1`. + +You can use the playbook to [register a new user](registering-users.md): + +``` +ansible-playbook -i inventory/hosts setup.yml --extra-vars='username=bot.chatgpt password=PASSWORD_FOR_THE_BOT admin=no' --tags=register-user +``` + + +## 2. Get an access token + +Refer to the documentation on [how to obtain an access token](obtaining-access-tokens.md). + + +## 3. Adjusting the playbook configuration + +Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file (adapt to your needs): + +```yaml +matrix_bot_chatgpt_enabled: true + +# Obtain a new API key from https://platform.openai.com/account/api-keys +matrix_bot_chatgpt_openai_api_key: '' + +# This is the default username +# matrix_bot_chatgpt_matrix_bot_username_localpart: 'bot.chatgpt' + +# Matrix access token (from bot user above) +# see: https://webapps.stackexchange.com/questions/131056/how-to-get-an-access-token-for-element-riot-matrix +matrix_bot_chatgpt_matrix_access_token: '' +``` + +You will need to get tokens for ChatGPT. + + +## 4. Installing + +After configuring the playbook, run the [installation](installing.md) command again: + +```sh +ansible-playbook -i inventory/hosts setup.yml --tags=install-all,start +``` + + +## Usage + +To use the bot, invite the `@bot.chatgpt:DOMAIN` to the room you specified in a config, after that start speaking to it, use the prefix if you configured one or mention the bot. + +You can also refer to the upstream [documentation](https://github.com/matrixgpt/matrix-chatgpt-bot). diff --git a/docs/configuring-playbook-bot-draupnir.md b/docs/configuring-playbook-bot-draupnir.md new file mode 100644 index 00000000..6704ad65 --- /dev/null +++ b/docs/configuring-playbook-bot-draupnir.md @@ -0,0 +1,114 @@ +# Setting up draupnir (optional) + +The playbook can install and configure the [draupnir](https://github.com/Gnuxie/Draupnir) moderation bot for you. + +See the project's [documentation](https://github.com/Gnuxie/Draupnir) to learn what it does and why it might be useful to you. + +If your migrating from Mjolnir skip to step 5b. + +## 1. Register the bot account + +The playbook does not automatically create users for you. The bot requires an access token to be able to connect to your homeserver. + +You **need to register the bot user manually** before setting up the bot. + +Choose a strong password for the bot. You can generate a good password with a command like this: `pwgen -s 64 1`. + +You can use the playbook to [register a new user](registering-users.md): + +``` +ansible-playbook -i inventory/hosts setup.yml --extra-vars='username=bot.draupnir password=PASSWORD_FOR_THE_BOT admin=no' --tags=register-user +``` + +If you would like draupnir to be able to deactivate users, move aliases, shutdown rooms, etc then it must be a server admin so you need to change `admin=no` to `admin=yes` in the command above. + + +## 2. Get an access token + +Refer to the documentation on [how to obtain an access token](obtaining-access-tokens.md). + + +## 3. Make sure the account is free from rate limiting + +You will need to prevent Synapse from rate limiting the bot's account. This is not an optional step. If you do not do this step draupnir will crash. This can be done using Synapse's [admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#override-ratelimiting-for-users). This can also be manually done by editing the Synapse database. Manually editing the Synapse database is rarely a good idea. Please ask for help if you are uncomfortable with these steps. + +1. Copy the statement below into a text editor. + + ``` + INSERT INTO ratelimit_override VALUES ('@bot.draupnir:DOMAIN', 0, 0); + ``` + +1. Change the username (`@bot.draupnir:DOMAIN`) to the username you used when you registered the bot's account. You must change `DOMAIN` to your server's domain. + +1. Get a database terminal by following these steps: [maintenance-postgres.md#getting-a-database-terminal](maintenance-postgres.md#getting-a-database-terminal) + +1. Connect to Synapse's database by typing `\connect synapse` into the database terminal + +1. Paste in the `INSERT INTO` command that you edited and press enter. + +You can run `SELECT * FROM ratelimit_override;` to see if it worked. If the output looks like this: + +``` + user_id | messages_per_second | burst_count +-----------------------+---------------------+------------- + @bot.draupnir:raim.ist | 0 | 0` +``` +then you did it correctly. + + +## 4. Create a management room + +Using your own account, create a new invite only room that you will use to manage the bot. This is the room where you will see the status of the bot and where you will send commands to the bot, such as the command to ban a user from another room. Anyone in this room can control the bot so it is important that you only invite trusted users to this room. The room must be unencrypted since the playbook does not support installing Pantalaimon yet. + +Once you have created the room you need to copy the room ID so you can tell the bot to use that room. In Element you can do this by going to the room's settings, clicking Advanced, and then coping the internal room ID. The room ID will look something like `!QvgVuKq0ha8glOLGMG:DOMAIN`. + +Finally invite the `@bot.draupnir:DOMAIN` account you created earlier into the room. + + +## 5a. Adjusting the playbook configuration + +Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file (adapt to your needs): + +You must replace `ACCESS_TOKEN_FROM_STEP_2_GOES_HERE` and `ROOM_ID_FROM_STEP_4_GOES_HERE` with the your own values. + +```yaml +matrix_bot_draupnir_enabled: true + +matrix_bot_draupnir_access_token: "ACCESS_TOKEN_FROM_STEP_2_GOES_HERE" + +matrix_bot_draupnir_management_room: "ROOM_ID_FROM_STEP_4_GOES_HERE" +``` + +## 5b. Migrating from Mjolnir (Only required if migrating.) + +Replace your matrix_bot_mjolnir config with matrix_bot_draupnir config. Also disable mjolnir if you're doing migration. +That is all you need to do due to that Draupnir can complete migration on its own. + +## 6. Installing + +After configuring the playbook, run the [installation](installing.md) command: + +``` +ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start +``` + + +## Usage + +You can refer to the upstream [documentation](https://github.com/Gnuxie/Draupnir) for additional ways to use and configure draupnir. Check out their [quickstart guide](https://github.com/matrix-org/draupnir/blob/main/docs/moderators.md#quick-usage) for some basic commands you can give to the bot. + +You can configure additional options by adding the `matrix_bot_draupnir_configuration_extension_yaml` variable to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file. + +For example to change draupnir's `recordIgnoredInvites` option to `true` you would add the following to your `vars.yml` file. + +```yaml +matrix_bot_draupnir_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_bot_draupnir_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_bot_draupnir_configuration_yaml`. + recordIgnoredInvites: true +``` diff --git a/docs/configuring-playbook-bridge-appservice-slack.md b/docs/configuring-playbook-bridge-appservice-slack.md index fc2bf166..3e6f798e 100644 --- a/docs/configuring-playbook-bridge-appservice-slack.md +++ b/docs/configuring-playbook-bridge-appservice-slack.md @@ -1,6 +1,6 @@ # Setting up Appservice Slack (optional) -**Note**: bridging to [Slack](https://slack.com) can also happen via the [mx-puppet-slack](configuring-playbook-bridge-mx-puppet-slack.md) bridge supported by the playbook. +**Note**: bridging to [Slack](https://slack.com) can also happen via the [mx-puppet-slack](configuring-playbook-bridge-mx-puppet-slack.md) and [mautrix-slack](configuring-playbook-bridge-mautrix-slack.md) bridges supported by the playbook. The playbook can install and configure [matrix-appservice-slack](https://github.com/matrix-org/matrix-appservice-slack) for you. diff --git a/docs/configuring-playbook-bridge-mautrix-slack.md b/docs/configuring-playbook-bridge-mautrix-slack.md new file mode 100644 index 00000000..d0a60aa7 --- /dev/null +++ b/docs/configuring-playbook-bridge-mautrix-slack.md @@ -0,0 +1,77 @@ +# Setting up Mautrix Slack (optional) + +**Note**: bridging to [Slack](https://slack.com/) can also happen via the [mx-puppet-slack](configuring-playbook-bridge-mx-puppet-slack.md) and [matrix-appservice-slack](configuring-playbook-bridge-appservice-slack.md) bridges supported by the playbook. +- For using as a Bot we recommend the [Appservice Slack](configuring-playbook-bridge-appservice-slack.md), because it supports plumbing. +- For personal use with a slack account we recommend the `mautrix-slack` bridge (the one being discussed here), because it is the most fully-featured and stable of the 3 Slack bridges supported by the playbook. + +The playbook can install and configure [mautrix-slack](https://github.com/mautrix/slack) for you. + +See the project's [documentation](https://docs.mau.fi/bridges/go/slack/index.html) to learn what it does and why it might be useful to you. + +Note that as of Oct 2022, support for multiple Matrix users using the bot is incomplete. Different users do not yet share the bridged channels. Everyone gets their own copy. + +See the [features and roadmap](https://github.com/mautrix/slack/blob/main/ROADMAP.md) for more information. + + +## Prerequisites + +For using this bridge, you would need to authenticate by **providing your username and password** (legacy) or by using a **token login**. See more information in the [docs](https://docs.mau.fi/bridges/go/slack/authentication.html). + +Note that neither of these methods are officially supported by Slack. [matrix-appservice-slack](configuring-playbook-bridge-appservice-slack.md) uses a Slack bot account which is the only officially supported method for bridging a Slack channel. + + +## Installing + +To enable the bridge, add this to your `vars.yml` file: + +```yaml +matrix_mautrix_slack_enabled: true +``` + +You may optionally wish to add some [Additional configuration](#additional-configuration), or to [prepare for double-puppeting](#set-up-double-puppeting) before the initial installation. + +After adjusting your `vars.yml` file, re-run the playbook and restart all services: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start` + +To make use of the bridge, see [Usage](#usage) below. + + +### Additional configuration + +There are some additional options you may wish to configure with the bridge. + +Take a look at: + +- `roles/custom/matrix-bridge-mautrix-slack/defaults/main.yml` for some variables that you can customize via your `vars.yml` file +- `roles/custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2` for the bridge's default configuration. You can override settings (even those that don't have dedicated playbook variables) using the `matrix_mautrix_slack_configuration_extension_yaml` variable + + +### Set up Double Puppeting + +If you'd like to use [Double Puppeting](https://docs.mau.fi/bridges/general/double-puppeting.html) (hint: you most likely do), you have 2 ways of going about it. + +#### Method 1: automatically, by enabling Shared Secret Auth + +The bridge will automatically perform Double Puppeting if you enable [Shared Secret Auth](configuring-playbook-shared-secret-auth.md) for this playbook. + +This is the recommended way of setting up Double Puppeting, as it's easier to accomplish, works for all your users automatically, and has less of a chance of breaking in the future. + +#### Method 2: manually, by asking each user to provide a working access token + +**Note**: This method for enabling Double Puppeting can be configured only after you've already set up bridging (see [Usage](#usage)). + +When using this method, **each user** that wishes to enable Double Puppeting needs to follow the following steps: + +- retrieve a Matrix access token for yourself. Refer to the documentation on [how to do that](obtaining-access-tokens.md). + +- send the access token to the bot. Example: `login-matrix MATRIX_ACCESS_TOKEN_HERE` + +- make sure you don't log out the `Mautrix-Slack` device some time in the future, as that would break the Double Puppeting feature + + +## Usage + +1. Start a chat with `@slackbot:YOUR_DOMAIN` (where `YOUR_DOMAIN` is your base domain, not the `matrix.` domain). +2. If you would like to login to Slack using a token, send the `login-token` command, otherwise, send the `login-password` command. Read [here](https://docs.mau.fi/bridges/go/slack/authentication.html) on how to retrieve your token and cookie token. +3. The bot should respond with "Successfully logged into for team " +4. Now that you're logged in, you can send a `help` command to the bot again, to see additional commands you have access to. +5. Slack channels should automatically begin bridging if you authenticated using a token. Otherwise, you must wait to receive a message in the channel if you used password authentication. diff --git a/docs/configuring-playbook-bridge-mautrix-whatsapp.md b/docs/configuring-playbook-bridge-mautrix-whatsapp.md index 8ae6e5a0..b08556fe 100644 --- a/docs/configuring-playbook-bridge-mautrix-whatsapp.md +++ b/docs/configuring-playbook-bridge-mautrix-whatsapp.md @@ -11,6 +11,19 @@ matrix_mautrix_whatsapp_enabled: true ``` Whatsapp multidevice beta is required, now it is enough if Whatsapp is connected to the Internet every 2 weeks. +The relay bot functionality is off by default. If you would like to enable the relay bot, add the following to your `vars.yml` file: +```yaml +matrix_mautrix_whatsapp_bridge_relay_enabled: true +``` + +By default, only admins are allowed to set themselves as relay users. To allow anyone on your homeserver to set themselves as relay users add this to your `vars.yml` file: +```yaml +matrix_mautrix_whatsapp_bridge_relay_admin_only: false +``` + +If you want to activate the relay bot in a room, use `!whatsapp set-relay`. +Use `!whatsapp unset-relay` to deactivate. + ## Enable backfilling history This requires a server with MSC2716 support, which is currently an experimental feature in synapse. Note that as of Synapse 1.46, there are still some bugs with the implementation, especially if using event persistence workers. diff --git a/docs/configuring-playbook-bridge-mx-puppet-slack.md b/docs/configuring-playbook-bridge-mx-puppet-slack.md index 8db159d7..c195b8b5 100644 --- a/docs/configuring-playbook-bridge-mx-puppet-slack.md +++ b/docs/configuring-playbook-bridge-mx-puppet-slack.md @@ -1,8 +1,7 @@ # Setting up MX Puppet Slack (optional) **Note**: bridging to [Slack](https://slack.com) can also happen via the -[matrix-appservice-slack](configuring-playbook-bridge-appservice-slack.md) -bridge supported by the playbook. +[matrix-appservice-slack](configuring-playbook-bridge-appservice-slack.md) and [mautrix-slack](configuring-playbook-bridge-mautrix-slack.md) bridges supported by the playbook. The playbook can install and configure [Beeper](https://www.beeper.com/)-maintained fork of [mx-puppet-slack](https://gitlab.com/beeper/mx-puppet-monorepo) for you. diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md index 0b785c35..956b0573 100644 --- a/docs/configuring-playbook-prometheus-grafana.md +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -10,10 +10,10 @@ Remember to add `stats.` to DNS as described in [Configuring DNS](c matrix_prometheus_enabled: true # You can remove this, if unnecessary. -matrix_prometheus_node_exporter_enabled: true +prometheus_node_exporter_enabled: true # You can remove this, if unnecessary. -matrix_prometheus_postgres_exporter_enabled: true +prometheus_postgres_exporter_enabled: true # You can remove this, if unnecessary. matrix_prometheus_nginxlog_exporter_enabled: true @@ -40,8 +40,8 @@ The retention policy of Prometheus metrics is [15 days by default](https://prome Name | Description -----|---------- `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. -`matrix_prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures -`matrix_prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. +`prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures +`prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. `matrix_prometheus_nginxlog_exporter_enabled`|[NGINX Log Exporter](configuring-playbook-prometheus-nginxlog.md) is an addon of sorts to expose NGINX logs to Prometheus. `matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.` subdomain) the dashboards with the graphs that we're interested in `matrix_grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option. @@ -52,7 +52,7 @@ Name | Description Metrics and resulting graphs can contain a lot of information. This includes system specs but also usage patterns. This applies especially to small personal/family scale homeservers. Someone might be able to figure out when you wake up and go to sleep by looking at the graphs over time. Think about this before enabling anonymous access. And you should really not forget to change your Grafana password. -Most of our docker containers run with limited system access, but the `prometheus-node-exporter` has access to the host network stack and (readonly) root filesystem. This is required to report on them. If you don't like that, you can set `matrix_prometheus_node_exporter_enabled: false` (which is actually the default). You will still get Synapse metrics with this container disabled. Both of the dashboards will always be enabled, so you can still look at historical data after disabling either source. +Most of our docker containers run with limited system access, but the `prometheus-node-exporter` has access to the host network stack and (readonly) root filesystem. This is required to report on them. If you don't like that, you can set `prometheus_node_exporter_enabled: false` (which is actually the default). You will still get Synapse metrics with this container disabled. Both of the dashboards will always be enabled, so you can still look at historical data after disabling either source. ## Collecting metrics to an external Prometheus server @@ -74,11 +74,11 @@ Name | Description `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`|Set this to the Basic Authentication credentials (raw `htpasswd` file content) used to protect `/metrics/*`. This htpasswd-file needs to be generated with the `htpasswd` tool and can include multiple username/password pairs. If you only need one credential, use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` instead. `matrix_synapse_metrics_enabled`|Set this to `true` to make Synapse expose metrics (locally, on the container network) `matrix_synapse_metrics_proxying_enabled`|Set this to `true` to expose Synapse's metrics on `https://matrix.DOMAIN/metrics/synapse/main-process` and `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`). Read [below](#collecting-synapse-worker-metrics-to-an-external-prometheus-server) if you're running a Synapse worker setup (`matrix_synapse_workers_enabled: true`). -`matrix_prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter (locally, on the container network) -`matrix_prometheus_node_exporter_metrics_proxying_enabled`|Set this to `true` to expose the node (general system stats) metrics on `https://matrix.DOMAIN/metrics/node-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) -`matrix_prometheus_postgres_exporter_enabled`|Set this to `true` to enable the [Postgres exporter](configuring-playbook-prometheus-postgres.md) (locally, on the container network) +`prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter (locally, on the container network) +`matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled`|Set this to `true` to expose the node (general system stats) metrics on `https://matrix.DOMAIN/metrics/node-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`prometheus_postgres_exporter_enabled`|Set this to `true` to enable the [Postgres exporter](configuring-playbook-prometheus-postgres.md) (locally, on the container network) `matrix_prometheus_nginxlog_exporter_enabled`|Set this to `true` to enable the [NGINX Log exporter](configuring-playbook-prometheus-nginxlog.md) (locally, on the container network) -`matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|Set this to `true` to expose the [Postgres exporter](configuring-playbook-prometheus-postgres.md) metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled`|Set this to `true` to expose the [Postgres exporter](configuring-playbook-prometheus-postgres.md) metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) `matrix_bridge_hookshot_metrics_enabled`|Set this to `true` to make [Hookshot](configuring-playbook-bridge-hookshot.md) expose metrics (locally, on the container network) `matrix_bridge_hookshot_metrics_proxying_enabled`|Set this to `true` to expose the [Hookshot](configuring-playbook-bridge-hookshot.md) metrics on `https://matrix.DOMAIN/metrics/hookshot` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) `matrix_SERVICE_metrics_proxying_enabled`|Various other services/roles may provide similar `_metrics_enabled` and `_metrics_proxying_enabled` variables for exposing their metrics. Refer to each role for details. Only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` diff --git a/docs/configuring-playbook-prometheus-postgres.md b/docs/configuring-playbook-prometheus-postgres.md index 6fd13a9e..c3c8e0bf 100644 --- a/docs/configuring-playbook-prometheus-postgres.md +++ b/docs/configuring-playbook-prometheus-postgres.md @@ -6,17 +6,17 @@ You can enable this with the following settings in your configuration file (`inv ```yaml -matrix_prometheus_postgres_exporter_enabled: true +prometheus_postgres_exporter_enabled: true ``` ## What does it do? Name | Description -----|---------- -`matrix_prometheus_postgres_exporter_enabled`|Enable the postgres prometheus exporter. This sets up the docker container, connects it to the database and adds a 'job' to the prometheus config which tells prometheus about this new exporter. The default is 'false' -`matrix_prometheus_postgres_exporter_database_username`| The 'username' for the user that the exporter uses to connect to the database. The default is 'matrix_prometheus_postgres_exporter' -`matrix_prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database. By default, this is auto-generated by the playbook -`matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|If set to `true`, exposes the Postgres exporter metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` for usage with an [external Prometheus server](configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`prometheus_postgres_exporter_enabled`|Enable the postgres prometheus exporter. This sets up the docker container, connects it to the database and adds a 'job' to the prometheus config which tells prometheus about this new exporter. The default is 'false' +`prometheus_postgres_exporter_database_username`| The 'username' for the user that the exporter uses to connect to the database. The default is 'matrix_prometheus_postgres_exporter' +`prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database. By default, this is auto-generated by the playbook +`matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled`|If set to `true`, exposes the Postgres exporter metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` for usage with an [external Prometheus server](configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) ## More information diff --git a/docs/configuring-playbook-synapse-s3-storage-provider.md b/docs/configuring-playbook-synapse-s3-storage-provider.md index 6022eaad..d5d5ca5e 100644 --- a/docs/configuring-playbook-synapse-s3-storage-provider.md +++ b/docs/configuring-playbook-synapse-s3-storage-provider.md @@ -3,8 +3,6 @@ If you'd like to store Synapse's content repository (`media_store`) files on Amazon S3 (or other S3-compatible service), you can use the [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider) media provider module for Synapse. -**`synapse-s3-storage-provider` support is very new and still relatively untested. Using it may cause data loss.** - An alternative (which has worse performance) is to use [Goofys to mount the S3 store to the local filesystem](configuring-playbook-s3-goofys.md). @@ -68,11 +66,11 @@ This launches a Synapse container, which has access to the local media store, Po Then use the following commands (`$` values come from environment variables - they're **not placeholders** that you need to substitute): -- `s3_media_upload update-db $UPDATE_DB_DURATION` - create a local SQLite database (`cache.db`) with a list of media repository files (from the `synapse` Postgres database) eligible for operating on +1. `s3_media_upload update-db $UPDATE_DB_DURATION` - create a local SQLite database (`cache.db`) with a list of media repository files (from the `synapse` Postgres database) eligible for operating on - `$UPDATE_DB_DURATION` is influenced by the `matrix_synapse_ext_synapse_s3_storage_provider_update_db_day_count` variable (defaults to `0`) - `$UPDATE_DB_DURATION` defaults to `0d` (0 days), which means **include files which haven't been accessed for more than 0 days** (that is, **all files will be included**). -- `s3_media_upload check-deleted $MEDIA_PATH` - check whether files in the local cache still exist in the local media repository directory -- `s3_media_upload upload $MEDIA_PATH $BUCKET --delete --storage-class $STORAGE_CLASS --endpoint-url $ENDPOINT` - uploads locally-stored files to S3 and deletes them from the local media repository directory +2. `s3_media_upload check-deleted $MEDIA_PATH` - check whether files in the local cache still exist in the local media repository directory +3. `s3_media_upload upload $MEDIA_PATH $BUCKET --delete --storage-class $STORAGE_CLASS --endpoint-url $ENDPOINT` - uploads locally-stored files to S3 and deletes them from the local media repository directory The `s3_media_upload upload` command may take a lot of time to complete. @@ -93,13 +91,21 @@ To migrate your existing local data to S3, we recommend to: #### Copying data to Amazon S3 -Generally, you need to use the `aws s3` tool. +To copy to AWS S3, start a container on the Matrix server like this: -This documentation section could use an improvement. Ideally, we'd come up with a guide like the one used in [Copying data to Backblaze B2](#copying-data-to-backblaze-b2) - running `aws s3` in a container, etc. +```sh +docker run -it --rm \ +-w /work \ +--env-file=/matrix/synapse/ext/s3-storage-provider/env \ +--mount type=bind,src=/matrix/synapse/storage/media-store,dst=/work,ro \ +--entrypoint=/bin/sh \ +docker.io/amazon/aws-cli:2.9.16 \ +-c 'aws s3 sync /work/. s3://$BUCKET/' +``` #### Copying data to Backblaze B2 -To copy to Backblaze B2, start a container like this: +To copy to Backblaze B2, start a container on the Matrix server like this: ```sh docker run -it --rm \ @@ -109,7 +115,7 @@ docker run -it --rm \ --env='B2_BUCKET_NAME=YOUR_BUCKET_NAME_GOES_HERE' \ --mount type=bind,src=/matrix/synapse/storage/media-store,dst=/work,ro \ --entrypoint=/bin/sh \ -tianon/backblaze-b2:3.6.0 \ +docker.io/tianon/backblaze-b2:3.6.0 \ -c 'b2 authorize-account $B2_KEY_ID $B2_KEY_SECRET && b2 sync /work b2://$B2_BUCKET_NAME --skipNewer' ``` diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index fa00c858..9ab50763 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -106,6 +106,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up Mautrix Telegram bridging](configuring-playbook-bridge-mautrix-telegram.md) (optional) +- [Setting up Mautrix Slack bridging](configuring-playbook-bridge-mautrix-slack.md) (optional) + - [Setting up Mautrix Whatsapp bridging](configuring-playbook-bridge-mautrix-whatsapp.md) (optional) - [Setting up Mautrix Facebook bridging](configuring-playbook-bridge-mautrix-facebook.md) (optional) @@ -161,6 +163,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins ### Bots +- [Setting up matrix-bot-chatgpt](configuring-playbook-bot-chatgpt.md) - a bot through which you can talk to the [ChatGPT](https://openai.com/blog/chatgpt/) model(optional) + - [Setting up matrix-reminder-bot](configuring-playbook-bot-matrix-reminder-bot.md) - a bot to remind you about stuff (optional) - [Setting up matrix-registration-bot](configuring-playbook-bot-matrix-registration-bot.md) - a bot to create and manage registration tokens to invite users (optional) @@ -173,6 +177,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up Mjolnir](configuring-playbook-bot-mjolnir.md) - a moderation tool/bot (optional) +- [Setting up Draupnir](configuring-playbook-bot-draupnir.md) - a moderation tool/bot (optional), forked from Mjolnir and maintained by its former leader developer + - [Setting up Buscarron](configuring-playbook-bot-buscarron.md) - a bot you can use to send any form (HTTP POST, HTML) to a (encrypted) Matrix room (optional) diff --git a/docs/installing.md b/docs/installing.md index 3bbc9a44..fccc1caf 100644 --- a/docs/installing.md +++ b/docs/installing.md @@ -2,7 +2,7 @@ If you've [configured your DNS](configuring-dns.md) and have [configured the playbook](configuring-playbook.md), you can start the installation procedure. -**Before installing** and each time you update the playbook in the future, you will need to update the Ansible roles in this playbook by running `make roles`. `make roles` is a shortcut (a `roles` target defined in [`Makefile`](Makefile) and executed by the [`make`](https://www.gnu.org/software/make/) utility) which ultimately runs [ansible-galaxy](https://docs.ansible.com/ansible/latest/cli/ansible-galaxy.html) to download Ansible roles. If you don't have `make`, you can also manually run the `roles` commands seen in the `Makefile`. +**Before installing** and each time you update the playbook in the future, you will need to update the Ansible roles in this playbook by running `just roles`. `just roles` is a shortcut (a `roles` target defined in [`justfile`](../justfile) and executed by the [`just`](https://github.com/casey/just) utility) which ultimately runs [ansible-galaxy](https://docs.ansible.com/ansible/latest/cli/ansible-galaxy.html) to download Ansible roles. If you don't have `just`, you can also manually run the `roles` commands seen in the `justfile`. ## Playbook tags introduction @@ -57,7 +57,7 @@ Proceed to [Maintaining your setup in the future](#2-maintaining-your-setup-in-t If you will be importing data into your newly created Matrix server, install it, but **do not** start its services just yet. Starting its services or messing with its database now will affect your data import later on. -To do the installation **without** starting services, run only the `setup-all` tag: +To do the installation **without** starting services, run only the `install-all` tag: ```sh ansible-playbook -i inventory/hosts setup.yml --tags=install-all @@ -88,6 +88,8 @@ Feel free to **re-run the setup command any time** you think something is off wi Note that if you remove components from `vars.yml`, or if we switch some component from being installed by default to not being installed by default anymore, you'd need to run the setup command with `--tags=setup-all` instead of `--tags=install-all`. See [Playbook tags introduction](#playbook-tags-introduction) +A way to invoke these `ansible-playbook` commands with less typing in the future is to use [just](https://github.com/casey/just) to run them: `just install-all` or `just setup-all`. See [our `justfile`](../justfile) for more information. + ## 3. Finalize the installation diff --git a/docs/maintenance-upgrading-services.md b/docs/maintenance-upgrading-services.md index d1c707fd..b65eadd7 100644 --- a/docs/maintenance-upgrading-services.md +++ b/docs/maintenance-upgrading-services.md @@ -10,8 +10,8 @@ To upgrade services: - take a look at [the changelog](../CHANGELOG.md) to see if there have been any backward-incompatible changes that you need to take care of -- download the upstream Ansible roles used by the playbook by running `make roles` +- download the upstream Ansible roles used by the playbook by running `just roles` -- re-run the [playbook setup](installing.md) and restart all serivces: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,ensure-matrix-users-created,start` +- re-run the [playbook setup](installing.md) and restart all services: `just setup-all` **Note**: major version upgrades to the internal PostgreSQL database are not done automatically. To upgrade it, refer to the [upgrading PostgreSQL guide](maintenance-postgres.md#upgrading-postgresql). diff --git a/docs/prerequisites.md b/docs/prerequisites.md index c0a90640..83ae42da 100644 --- a/docs/prerequisites.md +++ b/docs/prerequisites.md @@ -22,7 +22,7 @@ If your distro runs within an [LXC container](https://linuxcontainers.org/), you - [`git`](https://git-scm.com/) is the recommended way to download the playbook to your computer. `git` may also be required on the server if you will be [self-building](self-building.md) components. -- [`make`](https://www.gnu.org/software/make/) for running `make roles`, etc. (see [`Makefile`](../Makefile)), although you can also run these commands manually (without `make`) +- [`just`](https://github.com/casey/just) for running `just roles`, etc. (see [`justfile`](../justfile)), although you can also run these commands manually - An HTTPS-capable web server at the base domain name (``) which is capable of serving static files. Unless you decide to [Serve the base domain from the Matrix server](configuring-playbook-base-domain-serving.md) or alternatively, to use DNS SRV records for [Server Delegation](howto-server-delegation.md). diff --git a/docs/registering-users.md b/docs/registering-users.md index 321346b6..51fdef34 100644 --- a/docs/registering-users.md +++ b/docs/registering-users.md @@ -16,13 +16,21 @@ Table of contents: You can do it via this Ansible playbook (make sure to edit the `` and `` part below): +```sh +just register-user + +# Example: `just register-user john secret-password yes` ``` + +**or** by invoking `ansible-playbook` manually: + +```sh ansible-playbook -i inventory/hosts setup.yml --extra-vars='username= password= admin=' --tags=register-user ``` **or** using the command-line after **SSH**-ing to your server (requires that [all services have been started](#starting-the-services)): -``` +```sh /matrix/synapse/bin/register-user ``` diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 988af72e..fe83d466 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -53,6 +53,8 @@ matrix_homeserver_container_extra_arguments_auto: | + (['--mount type=bind,src=' + matrix_mautrix_discord_config_path + '/registration.yaml,dst=/matrix-mautrix-discord-registration.yaml,ro'] if matrix_mautrix_discord_enabled else []) + + (['--mount type=bind,src=' + matrix_mautrix_slack_config_path + '/registration.yaml,dst=/matrix-mautrix-slack-registration.yaml,ro'] if matrix_mautrix_slack_enabled else []) + + (['--mount type=bind,src=' + matrix_mautrix_facebook_config_path + '/registration.yaml,dst=/matrix-mautrix-facebook-registration.yaml,ro'] if matrix_mautrix_facebook_enabled else []) + (['--mount type=bind,src=' + matrix_mautrix_googlechat_config_path + '/registration.yaml,dst=/matrix-mautrix-googlechat-registration.yaml,ro'] if matrix_mautrix_googlechat_enabled else []) @@ -108,6 +110,8 @@ matrix_homeserver_app_service_config_files_auto: | + (['/matrix-mautrix-discord-registration.yaml'] if matrix_mautrix_discord_enabled else []) + + (['/matrix-mautrix-slack-registration.yaml'] if matrix_mautrix_slack_enabled else []) + + (['/matrix-mautrix-facebook-registration.yaml'] if matrix_mautrix_facebook_enabled else []) + (['/matrix-mautrix-googlechat-registration.yaml'] if matrix_mautrix_googlechat_enabled else []) @@ -185,8 +189,12 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-bot-mjolnir.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'mjolnir']}] if matrix_bot_mjolnir_enabled else []) + + ([{'name': 'matrix-bot-draupnir.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'draupnir']}] if matrix_bot_draupnir_enabled else []) + + ([{'name': 'matrix-bot-postmoogle.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'postmoogle']}] if matrix_bot_postmoogle_enabled else []) + + ([{'name': 'matrix-bot-chatgpt.service', 'priority': 2200, 'groups': ['matrix', 'bots', 'chatgpt']}] if matrix_bot_chatgpt_enabled else []) + + ([{'name': 'matrix-appservice-discord.service', 'priority': 2000, 'groups': ['matrix', 'bridges', 'appservice-discord']}] if matrix_appservice_discord_enabled else []) + ([{'name': 'matrix-appservice-irc.service', 'priority': 2000, 'groups': ['matrix', 'bridges', 'appservice-irc']}] if matrix_appservice_irc_enabled else []) @@ -209,6 +217,8 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-mautrix-discord.service', 'priority': 2000, 'groups': ['matrix', 'bridges', 'mautrix-discord']}] if matrix_mautrix_discord_enabled else []) + + ([{'name': 'matrix-mautrix-slack.service', 'priority': 2000, 'groups': ['matrix', 'bridges', 'mautrix-slack']}] if matrix_mautrix_slack_enabled else []) + + ([{'name': 'matrix-mautrix-facebook.service', 'priority': 2000, 'groups': ['matrix', 'bridges', 'mautrix-facebook']}] if matrix_mautrix_facebook_enabled else []) + ([{'name': 'matrix-mautrix-googlechat.service', 'priority': 2000, 'groups': ['matrix', 'bridges', 'mautrix-googlechat']}] if matrix_mautrix_googlechat_enabled else []) @@ -283,7 +293,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-nginx-proxy.service', 'priority': 3000, 'groups': ['matrix', 'nginx', 'reverse-proxies']}] if matrix_nginx_proxy_enabled else []) + - (matrix_ssl_renewal_systemd_units_list | selectattr('applicable') | selectattr('enableable')) + (matrix_ssl_renewal_systemd_units_list | selectattr('applicable') | selectattr('enableable') | list ) + ([{'name': 'matrix-ntfy.service', 'priority': 800, 'groups': ['matrix', 'ntfy']}] if matrix_ntfy_enabled else []) + @@ -293,11 +303,11 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-prometheus.service', 'priority': 4000, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-core']}] if matrix_prometheus_enabled else []) + - ([{'name': 'matrix-prometheus-node-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_node_exporter_enabled else []) + ([{'name': (prometheus_node_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if prometheus_node_exporter_enabled else []) + - ([{'name': 'matrix-prometheus-postgres-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_postgres_exporter_enabled else []) + ([{'name': (prometheus_postgres_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if prometheus_postgres_exporter_enabled else []) + - ([{'name': 'matrix-prometheus-nginxlog-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_nginxlog_exporter_enabled else []) + ([{'name': 'matrix-prometheus-nginxlog-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-exporters']}] if matrix_prometheus_nginxlog_exporter_enabled else []) + ([{'name': 'matrix-redis', 'priority': 750, 'groups': ['matrix', 'redis']}] if matrix_redis_enabled else []) + @@ -727,6 +737,46 @@ matrix_mautrix_discord_database_password: "{{ '%s' | format(matrix_homeserver_ge ###################################################################### +###################################################################### +# +# matrix-bridge-mautrix-slack +# +###################################################################### + +# We don't enable bridges by default. +matrix_mautrix_slack_enabled: false + +matrix_mautrix_slack_container_image_self_build: "{{ matrix_architecture not in ['arm64', 'amd64'] }}" + +matrix_mautrix_slack_systemd_required_services_list: | + {{ + ['docker.service'] + + + ['matrix-' + matrix_homeserver_implementation + '.service'] + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + + + (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else []) + }} + +matrix_mautrix_slack_appservice_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'mauslack.as.tok', rounds=655555) | to_uuid }}" + +matrix_mautrix_slack_homeserver_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'mauslack.hs.tok', rounds=655555) | to_uuid }}" + +matrix_mautrix_slack_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" + +# Postgres is the default, except if not using internal Postgres server +matrix_mautrix_slack_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" +matrix_mautrix_slack_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" +matrix_mautrix_slack_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'mauslack.db', rounds=655555) | to_uuid }}" + +###################################################################### +# +# /matrix-bridge-mautrix-slack +# +###################################################################### + + ###################################################################### # # matrix-bridge-mautrix-facebook @@ -761,6 +811,12 @@ matrix_mautrix_facebook_login_shared_secret: "{{ matrix_synapse_ext_password_pro matrix_mautrix_facebook_bridge_presence: "{{ matrix_synapse_presence_enabled if matrix_synapse_enabled else true }}" +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_mautrix_facebook_metrics_enabled` +# - `matrix_mautrix_facebook_proxying_metrics_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_mautrix_facebook_metrics_enabled: "{{ matrix_prometheus_enabled }}" + # We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain # and point them to a migration path. matrix_mautrix_facebook_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -888,6 +944,12 @@ matrix_mautrix_instagram_login_shared_secret: "{{ matrix_synapse_ext_password_pr matrix_mautrix_instagram_bridge_presence: "{{ matrix_synapse_presence_enabled if matrix_synapse_enabled else true }}" +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_mautrix_instagram_metrics_enabled` +# - `matrix_mautrix_instagram_proxying_metrics_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_mautrix_instagram_metrics_enabled: "{{ matrix_prometheus_enabled }}" + # We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain # and point them to a migration path. matrix_mautrix_instagram_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" @@ -933,6 +995,12 @@ matrix_mautrix_signal_appservice_token: "{{ '%s' | format(matrix_homeserver_gene matrix_mautrix_signal_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_mautrix_signal_metrics_enabled` +# - `matrix_mautrix_signal_proxying_metrics_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_mautrix_signal_metrics_enabled: "{{ matrix_prometheus_enabled }}" + matrix_mautrix_signal_database_engine: 'postgres' matrix_mautrix_signal_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" matrix_mautrix_signal_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'mau.signal.db', rounds=655555) | to_uuid }}" @@ -1057,6 +1125,12 @@ matrix_mautrix_whatsapp_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge matrix_mautrix_whatsapp_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_mautrix_twitter_metrics_enabled` +# - `matrix_mautrix_twitter_proxying_metrics_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_mautrix_twitter_metrics_enabled: "{{ matrix_prometheus_enabled }}" + # Postgres is the default, except if not using internal Postgres server matrix_mautrix_whatsapp_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}" matrix_mautrix_whatsapp_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" @@ -1616,6 +1690,24 @@ matrix_bot_postmoogle_database_password: "{{ '%s' | format(matrix_homeserver_gen ###################################################################### +###################################################################### +# +# matrix-bot-chatgpt +# +###################################################################### + +# We don't enable bots by default. +matrix_bot_chatgpt_enabled: false + +matrix_bot_chatgpt_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm64'] }}" + +###################################################################### +# +# /matrix-bot-chatgpt +# +###################################################################### + + ###################################################################### # # matrix-bot-go-neb @@ -1671,6 +1763,34 @@ matrix_bot_mjolnir_systemd_required_services_list: | # ###################################################################### +###################################################################### +# +# matrix-bot-draupnir +# +###################################################################### + +# We don't enable bots by default. +matrix_bot_draupnir_enabled: false + +matrix_bot_draupnir_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" + +matrix_bot_draupnir_systemd_required_services_list: | + {{ + ['docker.service'] + + + ['matrix-' + matrix_homeserver_implementation + '.service'] + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + + + (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else []) + }} + +###################################################################### +# +# /matrix-bot-draupnir +# +###################################################################### + ###################################################################### # # matrix-backup-borg @@ -2227,7 +2347,7 @@ matrix_ssl_domains_to_obtain_certificates_for: | + ([matrix_server_fqn_ntfy] if matrix_ntfy_enabled else []) + - ([matrix_bot_postmoogle_domain] if matrix_bot_postmoogle_enabled else []) + (matrix_bot_postmoogle_domains if matrix_bot_postmoogle_enabled else []) + ([matrix_domain] if matrix_nginx_proxy_base_domain_serving_enabled else []) + @@ -2455,6 +2575,12 @@ devture_postgres_managed_databases_auto: | 'password': matrix_mautrix_discord_database_password, }] if (matrix_mautrix_discord_enabled and matrix_mautrix_discord_database_engine == 'postgres' and matrix_mautrix_discord_database_hostname == devture_postgres_connection_hostname) else []) + + ([{ + 'name': matrix_mautrix_slack_database_name, + 'username': matrix_mautrix_slack_database_username, + 'password': matrix_mautrix_slack_database_password, + }] if (matrix_mautrix_slack_enabled and matrix_mautrix_slack_database_engine == 'postgres' and matrix_mautrix_slack_database_hostname == devture_postgres_connection_hostname) else []) + + ([{ 'name': matrix_mx_puppet_slack_database_name, 'username': matrix_mx_puppet_slack_database_username, @@ -2504,10 +2630,10 @@ devture_postgres_managed_databases_auto: | }] if (matrix_etherpad_enabled and matrix_etherpad_database_engine == 'postgres' and matrix_etherpad_database_hostname == devture_postgres_connection_hostname) else []) + ([{ - 'name': matrix_prometheus_postgres_exporter_database_name, - 'username': matrix_prometheus_postgres_exporter_database_username, - 'password': matrix_prometheus_postgres_exporter_database_password, - }] if (matrix_prometheus_postgres_exporter_enabled and matrix_prometheus_postgres_exporter_database_hostname == devture_postgres_connection_hostname) else []) + 'name': prometheus_postgres_exporter_database_name, + 'username': prometheus_postgres_exporter_database_username, + 'password': prometheus_postgres_exporter_database_password, + }] if (prometheus_postgres_exporter_enabled and prometheus_postgres_exporter_database_hostname == devture_postgres_connection_hostname) else []) }} @@ -2866,15 +2992,67 @@ matrix_synapse_admin_container_image_self_build: "{{ matrix_architecture not in ###################################################################### # -# matrix-prometheus-node-exporter +# etke/prometheus_node_exporter # ###################################################################### -matrix_prometheus_node_exporter_enabled: false +prometheus_node_exporter_enabled: false + +prometheus_node_exporter_identifier: matrix-prometheus-node-exporter + +prometheus_node_exporter_base_path: "{{ matrix_base_data_path }}/prometheus-node-exporter" + +prometheus_node_exporter_uid: "{{ matrix_user_uid }}" +prometheus_node_exporter_gid: "{{ matrix_user_gid }}" + +prometheus_node_exporter_server_fqn: "{{ matrix_server_fqn_matrix }}" + +prometheus_node_exporter_container_network: "{{ matrix_docker_network }}" + +prometheus_node_exporter_container_labels_traefik_enabled: false ###################################################################### # -# /matrix-prometheus-node-exporter +# /etke/prometheus_node_exporter +# +###################################################################### + +###################################################################### +# +# etke/prometheus_postgres_exporter +# +###################################################################### + +prometheus_postgres_exporter_enabled: false + +prometheus_postgres_exporter_identifier: matrix-prometheus-postgres-exporter + +prometheus_postgres_exporter_base_path: "{{ matrix_base_data_path }}/prometheus-postgres-exporter" + +prometheus_postgres_exporter_uid: "{{ matrix_user_uid }}" +prometheus_postgres_exporter_gid: "{{ matrix_user_gid }}" + +prometheus_postgres_exporter_server_fqn: "{{ matrix_server_fqn_matrix }}" + +prometheus_postgres_exporter_container_network: "{{ matrix_docker_network }}" + +prometheus_postgres_exporter_container_labels_traefik_enabled: false + +prometheus_postgres_exporter_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" +prometheus_postgres_exporter_database_username: matrix_prometheus_postgres_exporter +prometheus_postgres_exporter_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'prometheus.pg.db', rounds=655555) | to_uuid }}" +prometheus_postgres_exporter_database_name: matrix_prometheus_postgres_exporter + +prometheus_postgres_exporter_systemd_required_services_list: | + {{ + ['docker.service'] + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + }} + +###################################################################### +# +# /etke/prometheus_postgres_exporter # ###################################################################### @@ -2910,11 +3088,11 @@ matrix_prometheus_scraper_synapse_targets: ['matrix-synapse:{{ matrix_synapse_me matrix_prometheus_scraper_synapse_workers_enabled_list: "{{ matrix_synapse_workers_enabled_list }}" matrix_prometheus_scraper_synapse_rules_synapse_tag: "{{ matrix_synapse_docker_image_tag }}" -matrix_prometheus_scraper_node_enabled: "{{ matrix_prometheus_node_exporter_enabled }}" -matrix_prometheus_scraper_node_targets: "{{ ['matrix-prometheus-node-exporter:9100'] if matrix_prometheus_node_exporter_enabled else [] }}" +matrix_prometheus_scraper_node_enabled: "{{ prometheus_node_exporter_enabled }}" +matrix_prometheus_scraper_node_targets: "{{ [prometheus_node_exporter_identifier + ':9100'] if prometheus_node_exporter_enabled else [] }}" -matrix_prometheus_scraper_postgres_enabled: "{{ matrix_prometheus_postgres_exporter_enabled }}" -matrix_prometheus_scraper_postgres_targets: "{{ ['matrix-prometheus-postgres-exporter:'+ matrix_prometheus_postgres_exporter_port | string] if matrix_prometheus_scraper_postgres_enabled else [] }}" +matrix_prometheus_scraper_postgres_enabled: "{{ prometheus_postgres_exporter_enabled }}" +matrix_prometheus_scraper_postgres_targets: "{{ [prometheus_postgres_exporter_identifier + ':' + prometheus_postgres_exporter_port | string] if matrix_prometheus_scraper_postgres_enabled else [] }}" matrix_prometheus_scraper_hookshot_enabled: "{{ matrix_hookshot_metrics_enabled|default(false) }}" matrix_prometheus_scraper_hookshot_targets: "{{ [matrix_hookshot_container_url | string +':'+ matrix_hookshot_metrics_port | string] if matrix_hookshot_metrics_enabled else [] }}" @@ -2929,29 +3107,6 @@ matrix_prometheus_scraper_nginxlog_server_port: "{{ (matrix_prometheus_nginxlog_ # ###################################################################### -###################################################################### -# -# matrix-prometheus-postgres-exporter -# -###################################################################### - -matrix_prometheus_postgres_exporter_enabled: false -matrix_prometheus_postgres_exporter_database_hostname: "{{ devture_postgres_connection_hostname if devture_postgres_enabled else '' }}" -matrix_prometheus_postgres_exporter_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'prometheus.pg.db', rounds=655555) | to_uuid }}" - -matrix_prometheus_postgres_exporter_systemd_required_services_list: | - {{ - ['docker.service'] - + - ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) - }} - -###################################################################### -# -# /matrix-prometheus-postgres-exporter -# -###################################################################### - ###################################################################### # # matrix-grafana @@ -2969,9 +3124,9 @@ matrix_grafana_dashboard_download_urls: | {{ (matrix_synapse_grafana_dashboard_urls if matrix_homeserver_implementation == 'synapse' and matrix_synapse_metrics_enabled else []) + - (matrix_prometheus_node_exporter_dashboard_urls if matrix_prometheus_node_exporter_enabled else []) + (prometheus_node_exporter_dashboard_urls if prometheus_node_exporter_enabled else []) + - (matrix_prometheus_postgres_exporter_dashboard_urls if matrix_prometheus_postgres_exporter_enabled else []) + (prometheus_postgres_exporter_dashboard_urls if prometheus_postgres_exporter_enabled else []) + (matrix_prometheus_nginxlog_exporter_dashboard_urls if matrix_prometheus_nginxlog_exporter_enabled else []) }} @@ -2985,13 +3140,6 @@ matrix_grafana_default_home_dashboard_path: |- }[matrix_homeserver_implementation] }} -matrix_grafana_systemd_wanted_services_list: | - {{ - [] - + - (['matrix-prometheus-postgres-exporter.service'] if matrix_prometheus_postgres_exporter_enabled else []) - }} - ###################################################################### # # /matrix-grafana diff --git a/justfile b/justfile new file mode 100644 index 00000000..4b6dfbfb --- /dev/null +++ b/justfile @@ -0,0 +1,44 @@ +# Shows help +default: + @just --list --justfile {{ justfile() }} + +# Pulls external Ansible roles +roles: + rm -rf roles/galaxy + ansible-galaxy install -r requirements.yml -p roles/galaxy/ --force + +# Runs ansible-lint against all roles in the playbook +lint: + ansible-lint + +# Runs the playbook with --tags=install-all,ensure-matrix-users-created,start and optional arguments +install-all *extra_args: (run-tags "install-all,ensure-matrix-users-created,start" extra_args) + +# Runs the playbook with --tags=setup-all,ensure-matrix-users-created,start and optional arguments +setup-all *extra_args: (run-tags "setup-all,ensure-matrix-users-created,start" extra_args) + +# Runs the playbook with the given list of arguments +run +extra_args: + time ansible-playbook -i inventory/hosts setup.yml {{ extra_args }} + +# Runs the playbook with the given list of comma-separated tags and optional arguments +run-tags tags *extra_args: + just --justfile {{ justfile() }} run --tags={{ tags }} {{ extra_args }} + +# Runs the playbook in user-registration mode +register-user username password admin_yes_or_no *extra_args: + time ansible-playbook -i inventory/hosts setup.yml --tags=register-user --extra-vars="username={{ username }} password={{ password }} admin={{ admin_yes_or_no }}" {{ extra_args }} + +# Starts all services +start-all *extra_args: (run-tags "start-all" extra_args) + +# Starts a specific service group +start-group group *extra_args: + @just --justfile {{ justfile() }} run-tags start-group --extra-vars="group={{ group }}" {{ extra_args }} + +# Stops all services +stop-all *extra_args: (run-tags "stop-all" extra_args) + +# Stops a specific service group +stop-group group *extra_args: + @just --justfile {{ justfile() }} run-tags stop-group --extra-vars="group={{ group }}" {{ extra_args }} diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index a74b982d..54324293 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -4,7 +4,7 @@ become: true roles: - # Most of the roles below are not distributed with the playbook, but downloaded separately using `ansible-galaxy` via the `make roles` command (see `Makefile`). + # Most of the roles below are not distributed with the playbook, but downloaded separately using `ansible-galaxy` via the `just roles` command (see `justfile`). - role: galaxy/com.devture.ansible.role.playbook_help - role: galaxy/com.devture.ansible.role.systemd_docker_base @@ -61,6 +61,7 @@ - custom/matrix-bridge-mautrix-telegram - custom/matrix-bridge-mautrix-whatsapp - custom/matrix-bridge-mautrix-discord + - custom/matrix-bridge-mautrix-slack - custom/matrix-bridge-mx-puppet-discord - custom/matrix-bridge-mx-puppet-groupme - custom/matrix-bridge-mx-puppet-steam @@ -78,17 +79,20 @@ - custom/matrix-bot-postmoogle - custom/matrix-bot-go-neb - custom/matrix-bot-mjolnir + - custom/matrix-bot-draupnir + - custom/matrix-bot-chatgpt - custom/matrix-cactus-comments - custom/matrix-synapse - custom/matrix-synapse-reverse-proxy-companion - custom/matrix-dendrite - custom/matrix-conduit - custom/matrix-synapse-admin - - custom/matrix-prometheus-node-exporter - - custom/matrix-prometheus-postgres-exporter + - galaxy/prometheus_node_exporter + - galaxy/prometheus_postgres_exporter - custom/matrix-prometheus-nginxlog-exporter - custom/matrix-prometheus - custom/matrix-grafana + - custom/matrix-prometheus-services-proxy-connect - custom/matrix-registration - custom/matrix-client-element - custom/matrix-client-hydrogen diff --git a/requirements.yml b/requirements.yml index 93c80215..4797423d 100644 --- a/requirements.yml +++ b/requirements.yml @@ -13,13 +13,13 @@ version: 327d2e17f5189ac2480d6012f58cf64a2b46efba - src: git+https://github.com/devture/com.devture.ansible.role.timesync.git - version: 660f384f176a9ea3b5cc702bde39e7dc10bf6186 + version: 3d5bb2976815958cdce3f368fa34fb51554f899b - src: git+https://github.com/devture/com.devture.ansible.role.playbook_state_preserver.git version: ff2fd42e1c1a9e28e3312bbd725395f9c2fc7f16 - src: git+https://github.com/devture/com.devture.ansible.role.postgres.git - version: e75973e3a4edc12dfc3e880e43b12ebecbf82c61 + version: a1bb78d194434b38005f3a9e623bfa4b2c06c7bc - src: git+https://github.com/devture/com.devture.ansible.role.postgres_backup.git version: 77b1f9ae1aafa31c9078178c1036bf744c99d08b @@ -29,3 +29,9 @@ - src: git+https://github.com/devture/com.devture.ansible.role.playbook_runtime_messages.git version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 + +- src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git + version: v1.5.0-2 + +- src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git + version: v0.11.1-2 diff --git a/roles/custom/matrix-backup-borg/defaults/main.yml b/roles/custom/matrix-backup-borg/defaults/main.yml index 472abe97..ad5a0281 100644 --- a/roles/custom/matrix-backup-borg/defaults/main.yml +++ b/roles/custom/matrix-backup-borg/defaults/main.yml @@ -11,8 +11,11 @@ matrix_backup_borg_docker_repo: "https://gitlab.com/etke.cc/borgmatic" matrix_backup_borg_docker_repo_version: main matrix_backup_borg_docker_src_files_path: "{{ matrix_backup_borg_base_path }}/docker-src" -# version determined automatically, based on postgres server version (if enabled), otherwise latest is used +# image and postgres version determined automatically, based on detected postgres server version (if enabled), otherwise latest is used matrix_backup_borg_version: "" +matrix_backup_borg_postgres_version: "" +matrix_backup_borg_borg_version: 1.2.3 +matrix_backup_borg_borgmatic_version: 1.7.6 matrix_backup_borg_docker_image: "{{ matrix_backup_borg_docker_image_name_prefix }}etke.cc/borgmatic:{{ matrix_backup_borg_version }}" matrix_backup_borg_docker_image_name_prefix: "{{ 'localhost/' if matrix_backup_borg_container_image_self_build else 'registry.gitlab.com/' }}" matrix_backup_borg_docker_image_force_pull: "{{ matrix_backup_borg_docker_image.endswith(':latest') or matrix_backup_borg_version | default('') == '' }}" diff --git a/roles/custom/matrix-backup-borg/tasks/setup_install.yml b/roles/custom/matrix-backup-borg/tasks/setup_install.yml index 647aabf4..41d95371 100644 --- a/roles/custom/matrix-backup-borg/tasks/setup_install.yml +++ b/roles/custom/matrix-backup-borg/tasks/setup_install.yml @@ -2,25 +2,25 @@ - when: matrix_backup_borg_postgresql_enabled | bool and matrix_backup_borg_version == '' block: - - name: Fail with matrix_backup_borg_version advice if Postgres not enabled - ansible.builtin.fail: - msg: >- - You are not running a built-in Postgres server (`devture_postgres_enabled: false`), so auto-detecting its version and setting `matrix_backup_borg_version` automatically based on that cannot happen. - Consider setting `matrix_backup_borg_version` to your Postgres version manually. - when: not devture_postgres_enabled + - name: Fail with matrix_backup_borg_version advice if Postgres not enabled + ansible.builtin.fail: + msg: >- + You are not running a built-in Postgres server (`devture_postgres_enabled: false`), so auto-detecting its version and setting `matrix_backup_borg_version` automatically based on that cannot happen. + Consider setting `matrix_backup_borg_version` to your Postgres version manually. + when: not devture_postgres_enabled - - ansible.builtin.include_role: - name: galaxy/com.devture.ansible.role.postgres - tasks_from: detect_existing_postgres_version + - ansible.builtin.include_role: + name: galaxy/com.devture.ansible.role.postgres + tasks_from: detect_existing_postgres_version - - name: Fail if detected Postgres version is unsupported - ansible.builtin.fail: - msg: "You cannot use borg backup with such an old version ({{ devture_postgres_detected_version }}) of Postgres. Consider upgrading - link to docs for upgrading Postgres: docs/maintenance-postgres.md#upgrading-postgresql" - when: "devture_postgres_detected_version not in matrix_backup_borg_supported_postgres_versions" + - name: Fail if detected Postgres version is unsupported + ansible.builtin.fail: + msg: "You cannot use borg backup with such an old version ({{ devture_postgres_detected_version }}) of Postgres. Consider upgrading - link to docs for upgrading Postgres: docs/maintenance-postgres.md#upgrading-postgresql" + when: "devture_postgres_detected_version not in matrix_backup_borg_supported_postgres_versions" - - name: Set the correct borg backup version to use - ansible.builtin.set_fact: - matrix_backup_borg_version: "{{ devture_postgres_detected_version }}" + - name: Set the correct borg backup version to use + ansible.builtin.set_fact: + matrix_backup_borg_version: "{{ devture_postgres_detected_version }}-{{ matrix_backup_borg_borg_version }}-{{ matrix_backup_borg_borgmatic_version }}" - name: Ensure borg paths exist ansible.builtin.file: diff --git a/roles/custom/matrix-bot-buscarron/defaults/main.yml b/roles/custom/matrix-bot-buscarron/defaults/main.yml index 290248e0..f887e262 100644 --- a/roles/custom/matrix-bot-buscarron/defaults/main.yml +++ b/roles/custom/matrix-bot-buscarron/defaults/main.yml @@ -9,9 +9,9 @@ matrix_bot_buscarron_docker_repo: "https://gitlab.com/etke.cc/buscarron.git" matrix_bot_buscarron_docker_repo_version: "{{ matrix_bot_buscarron_version }}" matrix_bot_buscarron_docker_src_files_path: "{{ matrix_base_data_path }}/buscarron/docker-src" -matrix_bot_buscarron_version: v1.3.0 -matrix_bot_buscarron_docker_image: "{{ matrix_bot_buscarron_docker_image_name_prefix }}buscarron:{{ matrix_bot_buscarron_version }}" -matrix_bot_buscarron_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_buscarron_container_image_self_build else 'registry.gitlab.com/etke.cc/' }}" +matrix_bot_buscarron_version: v1.3.1 +matrix_bot_buscarron_docker_image: "{{ matrix_bot_buscarron_docker_image_name_prefix }}etke.cc/buscarron:{{ matrix_bot_buscarron_version }}" +matrix_bot_buscarron_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_buscarron_container_image_self_build else 'registry.gitlab.com/' }}" matrix_bot_buscarron_docker_image_force_pull: "{{ matrix_bot_buscarron_docker_image.endswith(':latest') }}" matrix_bot_buscarron_base_path: "{{ matrix_base_data_path }}/buscarron" diff --git a/roles/custom/matrix-bot-chatgpt/defaults/main.yml b/roles/custom/matrix-bot-chatgpt/defaults/main.yml new file mode 100644 index 00000000..489b22c4 --- /dev/null +++ b/roles/custom/matrix-bot-chatgpt/defaults/main.yml @@ -0,0 +1,83 @@ +--- +# chatgpt is a bot for chatting to openAI chatgpt matrix bot +# Project source code URL: https://github.com/matrixgpt/matrix-chatgpt-bot + +matrix_bot_chatgpt_enabled: true + +matrix_bot_chatgpt_container_image_self_build: false +matrix_bot_chatgpt_docker_repo: "https://github.com/matrixgpt/matrix-chatgpt-bot" +matrix_bot_chatgpt_docker_repo_version: "{{ 'latest' if matrix_bot_chatgpt_version == 'latest' else matrix_bot_chatgpt_version }}" +matrix_bot_chatgpt_docker_src_files_path: "{{ matrix_base_data_path }}/chatgpt/docker-src" + +matrix_bot_chatgpt_version: 2.2.1 +matrix_bot_chatgpt_docker_image: "{{ matrix_bot_chatgpt_docker_image_name_prefix }}matrixgpt/matrix-chatgpt-bot:{{ matrix_bot_chatgpt_version }}" +matrix_bot_chatgpt_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_chatgpt_container_image_self_build else 'ghcr.io/' }}" +matrix_bot_chatgpt_docker_image_force_pull: "{{ matrix_bot_chatgpt_docker_image.endswith(':latest') }}" + +matrix_bot_chatgpt_base_path: "{{ matrix_base_data_path }}/chatgpt" +matrix_bot_chatgpt_config_path: "{{ matrix_bot_chatgpt_base_path }}/config" +matrix_bot_chatgpt_data_path: "{{ matrix_bot_chatgpt_base_path }}/data" + +# A list of extra arguments to pass to the container +matrix_bot_chatgpt_container_extra_arguments: [] + +# List of systemd services that matrix-bot-chatgpt.service depends on +matrix_bot_chatgpt_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-bot-chatgpt.service wants +matrix_bot_chatgpt_systemd_wanted_services_list: [] + +# ChatGPT Settings +matrix_bot_chatgpt_openai_api_key: '' +matrix_bot_chatgpt_context: 'thread' # CHATGPT_CONTEXT="thread" + +matrix_bot_chatgpt_keyv_backend: 'file' +matrix_bot_chatgpt_keyv_url: '' +matrix_bot_chatgpt_keyv_bot_encryption: false +matrix_bot_chatgpt_keyv_bot_storage: true + +# Matrix Static Settings (required, see notes) +# Defaults to "https://matrix.org" +matrix_bot_chatgpt_matrix_homeserver_url: "{{ matrix_homeserver_container_url }}" # MATRIX_HOMESERVER_URL= +# With the @ and :DOMAIN, ie @SOMETHING:DOMAIN, needs to be set, created manually beforehand. +matrix_bot_chatgpt_matrix_bot_username_localpart: 'bot.chatgpt' +matrix_bot_chatgpt_matrix_bot_username: "@{{ matrix_bot_chatgpt_matrix_bot_username_localpart }}:{{ matrix_domain }}" # MATRIX_BOT_USERNAME= +# Set `MATRIX_BOT_PASSWORD` the bot will print an `MATRIX_ACCESS_TOKEN` to the terminal +# or https://webapps.stackexchange.com/questions/131056/how-to-get-an-access-token-for-element-riot-matrix +matrix_bot_chatgpt_matrix_access_token: '' # MATRIX_ACCESS_TOKEN= +# Once `MATRIX_BOT_ACCESS_TOKEN` is set this is no longer used. +matrix_bot_chatgpt_matrix_bot_password: '' # MATRIX_BOT_PASSWORD= + +# Matrix Configurable Settings Defaults (optional) +matrix_bot_chatgpt_matrix_default_prefix: "!chatgpt " # MATRIX_DEFAULT_PREFIX= Leave prefix blank to reply to all messages, trailing space matters +matrix_bot_chatgpt_matrix_default_prefix_reply: false # MATRIX_DEFAULT_PREFIX_REPLY= + +# Matrix Access Control (optional) +matrix_bot_chatgpt_matrix_blacklist: '' +matrix_bot_chatgpt_matrix_whitelist: ':{{ matrix_domain }}' + +# Matrix Feature Flags (optional) +matrix_bot_chatgpt_matrix_autojoin: true # MATRIX_AUTOJOIN=true +matrix_bot_chatgpt_matrix_encryption: true # MATRIX_ENCRYPTION=true +matrix_bot_chatgpt_matrix_threads: true # MATRIX_THREADS=true +matrix_bot_chatgpt_matrix_rich_text: true # MATRIX_RICH_TEXT=true + + +# A list of admins +# Example set of rules: +# matrix_bot_chatgpt_admins: +# - @someone:example.com +# - @another:example.com +# - @bot.*:example.com +# - @*:another.com +# matrix_bot_chatgpt_admins: "{{ [matrix_admin] if matrix_admin else [] }}" + +# Log level +# matrix_bot_chatgpt_loglevel: 'INFO' + +# Additional environment variables to pass to the chatgpt container +# +# Example: +# matrix_bot_chatgpt_environment_variables_extension: | +# chatgpt_TEXT_DONE=Done +matrix_bot_chatgpt_environment_variables_extension: '' diff --git a/roles/custom/matrix-bot-chatgpt/tasks/main.yml b/roles/custom/matrix-bot-chatgpt/tasks/main.yml new file mode 100644 index 00000000..c5487fb8 --- /dev/null +++ b/roles/custom/matrix-bot-chatgpt/tasks/main.yml @@ -0,0 +1,20 @@ +--- + +- block: + - when: matrix_bot_chatgpt_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - when: matrix_bot_chatgpt_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + tags: + - setup-all + - setup-bot-chatgpt + - install-all + - install-bot-chatgpt + +- block: + - when: not matrix_bot_chatgpt_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + tags: + - setup-all + - setup-bot-chatgpt diff --git a/roles/custom/matrix-bot-chatgpt/tasks/setup_install.yml b/roles/custom/matrix-bot-chatgpt/tasks/setup_install.yml new file mode 100644 index 00000000..5ae5ea95 --- /dev/null +++ b/roles/custom/matrix-bot-chatgpt/tasks/setup_install.yml @@ -0,0 +1,63 @@ +--- +- name: Ensure chatgpt paths exist + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - {path: "{{ matrix_bot_chatgpt_config_path }}", when: true} + - {path: "{{ matrix_bot_chatgpt_data_path }}", when: true} + - {path: "{{ matrix_bot_chatgpt_docker_src_files_path }}", when: matrix_bot_chatgpt_container_image_self_build} + when: "item.when | bool" + +- name: Ensure chatgpt environment variables file created + ansible.builtin.template: + src: "{{ role_path }}/templates/env.j2" + dest: "{{ matrix_bot_chatgpt_config_path }}/env" + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + mode: 0640 + +- name: Ensure chatgpt image is pulled + community.docker.docker_image: + name: "{{ matrix_bot_chatgpt_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_bot_chatgpt_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_chatgpt_docker_image_force_pull }}" + when: "not matrix_bot_chatgpt_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure chatgpt repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_bot_chatgpt_docker_repo }}" + version: "{{ matrix_bot_chatgpt_docker_repo_version }}" + dest: "{{ matrix_bot_chatgpt_docker_src_files_path }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_bot_chatgpt_git_pull_results + when: "matrix_bot_chatgpt_container_image_self_build | bool" + +- name: Ensure chatgpt image is built + community.docker.docker_image: + name: "{{ matrix_bot_chatgpt_docker_image }}" + source: build + force_source: "{{ matrix_bot_chatgpt_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_bot_chatgpt_docker_src_files_path }}" + pull: true + when: "matrix_bot_chatgpt_container_image_self_build | bool" + +- name: Ensure matrix-bot-chatgpt.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-bot-chatgpt.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-chatgpt.service" + mode: 0644 + register: matrix_bot_chatgpt_systemd_service_result diff --git a/roles/custom/matrix-bot-chatgpt/tasks/setup_uninstall.yml b/roles/custom/matrix-bot-chatgpt/tasks/setup_uninstall.yml new file mode 100644 index 00000000..4a50d733 --- /dev/null +++ b/roles/custom/matrix-bot-chatgpt/tasks/setup_uninstall.yml @@ -0,0 +1,25 @@ +--- + +- name: Check existence of matrix-chatgpt service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-chatgpt.service" + register: matrix_bot_chatgpt_service_stat + +- when: matrix_bot_chatgpt_service_stat.stat.exists | bool + block: + - name: Ensure matrix-chatgpt is stopped + ansible.builtin.service: + name: matrix-bot-chatgpt + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-bot-chatgpt.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-chatgpt.service" + state: absent + + - name: Ensure Matrix chatgpt paths don't exist + ansible.builtin.file: + path: "{{ matrix_bot_chatgpt_base_path }}" + state: absent diff --git a/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml new file mode 100644 index 00000000..ecc2b8d2 --- /dev/null +++ b/roles/custom/matrix-bot-chatgpt/tasks/validate_config.yml @@ -0,0 +1,22 @@ +--- + +- name: Fail if required Chatgpt settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item.name }}`). + when: "item.when | bool and vars[item.name] == ''" + with_items: + - {'name': 'matrix_bot_chatgpt_openai_api_key', when: true} + - {'name': 'matrix_bot_chatgpt_matrix_bot_username', when: true} + + +- name: Fail if OpenAI configuration not up-to-date. + ansible.builtin.fail: + msg: >- + Your configuration contains a varible that is no longer used. + Please change your configuration to remove the variable (`{{ item.name }}`). + when: "item.name in vars" + with_items: + - {'name': 'matrix_bot_chatgpt_openai_email'} + - {'name': 'matrix_bot_chatgpt_openai_password'} + - {'name': 'matrix_bot_chatgpt_openai_login_type'} diff --git a/roles/custom/matrix-bot-chatgpt/templates/env.j2 b/roles/custom/matrix-bot-chatgpt/templates/env.j2 new file mode 100644 index 00000000..40b2f9ff --- /dev/null +++ b/roles/custom/matrix-bot-chatgpt/templates/env.j2 @@ -0,0 +1,29 @@ +MATRIX_HOMESERVER_URL={{ matrix_bot_chatgpt_matrix_homeserver_url }} +MATRIX_ACCESS_TOKEN={{ matrix_bot_chatgpt_matrix_access_token }} + +OPENAI_API_KEY={{ matrix_bot_chatgpt_openai_api_key }} +CHATGPT_CONTEXT={{ matrix_bot_chatgpt_context }} + +KEYV_BACKEND={{ matrix_bot_chatgpt_keyv_backend }} +KEYV_URL={{ matrix_bot_chatgpt_keyv_url }} +KEYV_BOT_ENCRYPTION={{ matrix_bot_chatgpt_keyv_bot_encryption|lower }} +KEYV_BOT_STORAGE={{ matrix_bot_chatgpt_keyv_bot_storage|lower }} + +# With the @ and :DOMAIN, ie @SOMETHING:DOMAIN +MATRIX_BOT_USERNAME={{ matrix_bot_chatgpt_matrix_bot_username }} +MATRIX_BOT_PASSWORD={{ matrix_bot_chatgpt_matrix_bot_password }} + +MATRIX_DEFAULT_PREFIX={{ matrix_bot_chatgpt_matrix_default_prefix }} +MATRIX_DEFAULT_PREFIX_REPLY={{ matrix_bot_chatgpt_matrix_default_prefix_reply|lower }} + +MATRIX_BLACKLIST={{ matrix_bot_chatgpt_matrix_blacklist }} +MATRIX_WHITELIST={{ matrix_bot_chatgpt_matrix_whitelist }} + +MATRIX_AUTOJOIN={{ matrix_bot_chatgpt_matrix_autojoin|lower }} +MATRIX_ENCRYPTION={{ matrix_bot_chatgpt_matrix_encryption|lower }} +MATRIX_THREADS={{ matrix_bot_chatgpt_matrix_threads|lower }} +MATRIX_RICH_TEXT={{ matrix_bot_chatgpt_matrix_rich_text|lower }} + +DATA_PATH=/data/ + +{{ matrix_bot_chatgpt_environment_variables_extension }} diff --git a/roles/custom/matrix-prometheus-postgres-exporter/templates/systemd/matrix-prometheus-postgres-exporter.service.j2 b/roles/custom/matrix-bot-chatgpt/templates/systemd/matrix-bot-chatgpt.service.j2 similarity index 54% rename from roles/custom/matrix-prometheus-postgres-exporter/templates/systemd/matrix-prometheus-postgres-exporter.service.j2 rename to roles/custom/matrix-bot-chatgpt/templates/systemd/matrix-bot-chatgpt.service.j2 index 5c3fbac6..8e4a5020 100644 --- a/roles/custom/matrix-prometheus-postgres-exporter/templates/systemd/matrix-prometheus-postgres-exporter.service.j2 +++ b/roles/custom/matrix-bot-chatgpt/templates/systemd/matrix-bot-chatgpt.service.j2 @@ -1,11 +1,11 @@ #jinja2: lstrip_blocks: "True" [Unit] -Description=matrix-prometheus-postgres-exporter -{% for service in matrix_prometheus_postgres_exporter_systemd_required_services_list %} +Description=Matrix chatgpt bot +{% for service in matrix_bot_chatgpt_systemd_required_services_list %} Requires={{ service }} After={{ service }} {% endfor %} -{% for service in matrix_prometheus_postgres_exporter_systemd_wanted_services_list %} +{% for service in matrix_bot_chatgpt_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} DefaultDependencies=no @@ -13,30 +13,29 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus-postgres-exporter 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus-postgres-exporter 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-chatgpt 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-chatgpt 2>/dev/null || true' - -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-prometheus-postgres-exporter \ +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-bot-chatgpt \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - {% for arg in matrix_prometheus_postgres_exporter_container_extra_arguments %} + --tmpfs /tmp \ + --network={{ matrix_docker_network }} \ + --env-file={{ matrix_bot_chatgpt_config_path }}/env \ + --mount type=bind,src={{ matrix_bot_chatgpt_data_path }},dst=/data \ + --env HOME=/data/home \ + {% for arg in matrix_bot_chatgpt_container_extra_arguments %} {{ arg }} \ {% endfor %} - --network={{ matrix_docker_network }} \ - {% if matrix_prometheus_postgres_exporter_container_http_host_bind_port %} - -p {{ matrix_prometheus_postgres_exporter_container_http_host_bind_port }}:{{matrix_prometheus_postgres_exporter_port}} \ - {% endif %} - --pid=host \ - {{ matrix_prometheus_postgres_exporter_docker_image }} + {{ matrix_bot_chatgpt_docker_image }} -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus-postgres-exporter 2>/dev/null || true' -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus-postgres-exporter 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-chatgpt 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-chatgpt 2>/dev/null || true' Restart=always RestartSec=30 -SyslogIdentifier=matrix-prometheus-postgres-exporter +SyslogIdentifier=matrix-bot-chatgpt [Install] WantedBy=multi-user.target diff --git a/roles/custom/matrix-bot-draupnir/defaults/main.yml b/roles/custom/matrix-bot-draupnir/defaults/main.yml new file mode 100644 index 00000000..f0aa4da8 --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/defaults/main.yml @@ -0,0 +1,59 @@ +--- +# A moderation tool for Matrix +# Project source code URL: https://github.com/Gnuxie/Draupnir + +matrix_bot_draupnir_enabled: true + +matrix_bot_draupnir_version: "v1.80.0-beta.0" + +matrix_bot_draupnir_container_image_self_build: false +matrix_bot_draupnir_container_image_self_build_repo: "https://github.com/Gnuxie/Draupnir.git" + +matrix_bot_draupnir_docker_image: "{{ matrix_bot_draupnir_docker_image_name_prefix }}gnuxie/draupnir:{{ matrix_bot_draupnir_version }}" +matrix_bot_draupnir_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_draupnir_container_image_self_build else matrix_container_global_registry_prefix }}" +matrix_bot_draupnir_docker_image_force_pull: "{{ matrix_bot_draupnir_docker_image.endswith(':latest') }}" + +matrix_bot_draupnir_base_path: "{{ matrix_base_data_path }}/draupnir" +matrix_bot_draupnir_config_path: "{{ matrix_bot_draupnir_base_path }}/config" +matrix_bot_draupnir_data_path: "{{ matrix_bot_draupnir_base_path }}/data" +matrix_bot_draupnir_docker_src_files_path: "{{ matrix_bot_draupnir_base_path }}/docker-src" + +# A list of extra arguments to pass to the container +matrix_bot_draupnir_container_extra_arguments: [] + +# List of systemd services that matrix-bot-draupnir.service depends on +matrix_bot_draupnir_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-bot-draupnir.service wants +matrix_bot_draupnir_systemd_wanted_services_list: [] + +# The access token for the bot user +matrix_bot_draupnir_access_token: "" + +# The room ID where people can use the bot. The bot has no access controls, so +# anyone in this room can use the bot - secure your room! +# This should be a room alias or room ID - not a matrix.to URL. +# Note: draupnir is fairly verbose - expect a lot of messages from it. +matrix_bot_draupnir_management_room: "" + +# Default configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. +# +# For a more advanced customization, you can extend the default (see `matrix_bot_draupnir_configuration_extension_yaml`) +# or completely replace this variable with your own template. +matrix_bot_draupnir_configuration_yaml: "{{ lookup('template', 'templates/production.yaml.j2') }}" + +matrix_bot_draupnir_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_bot_draupnir_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_bot_draupnir_configuration_yaml`. + +matrix_bot_draupnir_configuration_extension: "{{ matrix_bot_draupnir_configuration_extension_yaml | from_yaml if matrix_bot_draupnir_configuration_extension_yaml | from_yaml is mapping else {} }}" + +# Holds the final configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_bot_draupnir_configuration_yaml`. +matrix_bot_draupnir_configuration: "{{ matrix_bot_draupnir_configuration_yaml | from_yaml | combine(matrix_bot_draupnir_configuration_extension, recursive=True) }}" diff --git a/roles/custom/matrix-bot-draupnir/tasks/main.yml b/roles/custom/matrix-bot-draupnir/tasks/main.yml new file mode 100644 index 00000000..686fe298 --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/tasks/main.yml @@ -0,0 +1,20 @@ +--- + +- block: + - when: matrix_bot_draupnir_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - when: matrix_bot_draupnir_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + tags: + - setup-all + - setup-bot-draupnir + - install-all + - install-bot-draupnir + +- block: + - when: not matrix_bot_draupnir_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + tags: + - setup-all + - setup-bot-draupnir diff --git a/roles/custom/matrix-bot-draupnir/tasks/setup_install.yml b/roles/custom/matrix-bot-draupnir/tasks/setup_install.yml new file mode 100644 index 00000000..4808f71f --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/tasks/setup_install.yml @@ -0,0 +1,74 @@ +--- + +- ansible.builtin.set_fact: + matrix_bot_draupnir_requires_restart: false + +- name: Ensure matrix-bot-draupnir paths exist + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - {path: "{{ matrix_bot_draupnir_base_path }}", when: true} + - {path: "{{ matrix_bot_draupnir_config_path }}", when: true} + - {path: "{{ matrix_bot_draupnir_data_path }}", when: true} + - {path: "{{ matrix_bot_draupnir_docker_src_files_path }}", when: "{{ matrix_bot_draupnir_container_image_self_build }}"} + when: "item.when | bool" + +- name: Ensure draupnir Docker image is pulled + community.docker.docker_image: + name: "{{ matrix_bot_draupnir_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_bot_draupnir_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_draupnir_docker_image_force_pull }}" + when: "not matrix_bot_draupnir_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure draupnir repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_bot_draupnir_container_image_self_build_repo }}" + dest: "{{ matrix_bot_draupnir_docker_src_files_path }}" + version: "{{ matrix_bot_draupnir_docker_image.split(':')[1] }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_bot_draupnir_git_pull_results + when: "matrix_bot_draupnir_container_image_self_build | bool" + +- name: Ensure draupnir Docker image is built + community.docker.docker_image: + name: "{{ matrix_bot_draupnir_docker_image }}" + source: build + force_source: "{{ matrix_bot_draupnir_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_bot_draupnir_docker_src_files_path }}" + pull: true + when: "matrix_bot_draupnir_container_image_self_build | bool" + +- name: Ensure matrix-bot-draupnir config installed + ansible.builtin.copy: + content: "{{ matrix_bot_draupnir_configuration | to_nice_yaml(indent=2, width=999999) }}" + dest: "{{ matrix_bot_draupnir_config_path }}/production.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-bot-draupnir.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-bot-draupnir.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-draupnir.service" + mode: 0644 + register: matrix_bot_draupnir_systemd_service_result + +- name: Ensure matrix-bot-draupnir.service restarted, if necessary + ansible.builtin.service: + name: "matrix-bot-draupnir.service" + state: restarted + daemon_reload: true + when: "matrix_bot_draupnir_requires_restart | bool" diff --git a/roles/custom/matrix-bot-draupnir/tasks/setup_uninstall.yml b/roles/custom/matrix-bot-draupnir/tasks/setup_uninstall.yml new file mode 100644 index 00000000..10583a0b --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/tasks/setup_uninstall.yml @@ -0,0 +1,25 @@ +--- + +- name: Check existence of matrix-bot-draupnir service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-draupnir.service" + register: matrix_bot_draupnir_service_stat + +- when: matrix_bot_draupnir_service_stat.stat.exists | bool + block: + - name: Ensure matrix-bot-draupnir is stopped + ansible.builtin.service: + name: matrix-bot-draupnir + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-bot-draupnir.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-bot-draupnir.service" + state: absent + + - name: Ensure matrix-bot-draupnir paths don't exist + ansible.builtin.file: + path: "{{ matrix_bot_draupnir_base_path }}" + state: absent diff --git a/roles/custom/matrix-bot-draupnir/tasks/validate_config.yml b/roles/custom/matrix-bot-draupnir/tasks/validate_config.yml new file mode 100644 index 00000000..b3828189 --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/tasks/validate_config.yml @@ -0,0 +1,9 @@ +--- + +- name: Fail if required variables are undefined + ansible.builtin.fail: + msg: "The `{{ item }}` variable must be defined and have a non-null value." + with_items: + - "matrix_bot_draupnir_access_token" + - "matrix_bot_draupnir_management_room" + when: "vars[item] == '' or vars[item] is none" diff --git a/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 new file mode 100644 index 00000000..dcc61b1b --- /dev/null +++ b/roles/custom/matrix-bot-draupnir/templates/production.yaml.j2 @@ -0,0 +1,247 @@ +# Endpoint URL that draupnir uses to interact with the matrix homeserver (client-server API), +# set this to the pantalaimon URL if you're using that. +homeserverUrl: "{{ matrix_homeserver_url }}" + +# Endpoint URL that draupnir could use to fetch events related to reports (client-server API and /_synapse/), +# only set this to the public-internet homeserver client API URL, do NOT set this to the pantalaimon URL. +rawHomeserverUrl: "{{ matrix_homeserver_url }}" + +# Matrix Access Token to use, draupnir will only use this if pantalaimon.use is false. +accessToken: "{{ matrix_bot_draupnir_access_token }}" + +# Options related to Pantalaimon (https://github.com/matrix-org/pantalaimon) +#pantalaimon: +# # Whether or not draupnir will use pantalaimon to access the matrix homeserver, +# # set to `true` if you're using pantalaimon. +# # +# # Be sure to point homeserverUrl to the pantalaimon instance. +# # +# # draupnir will log in using the given username and password once, +# # then store the resulting access token in a file under dataPath. +# use: false +# +# # The username to login with. +# username: draupnir +# +# # The password draupnir will login with. +# # +# # After successfully logging in once, this will be ignored, so this value can be blanked after first startup. +# password: your_password + +# The path draupnir will store its state/data in, leave default ("/data/storage") when using containers. +dataPath: "/data" + +# If true (the default), draupnir will only accept invites from users present in managementRoom. +autojoinOnlyIfManager: true + +# If `autojoinOnlyIfManager` is false, only the members in this space can invite +# the bot to new rooms. +#acceptInvitesFromSpace: "!example:example.org" + +# Whether draupnir should report ignored invites to the management room (if autojoinOnlyIfManager is true). +recordIgnoredInvites: false + +# The room ID (or room alias) of the management room, anyone in this room can issue commands to draupnir. +# +# draupnir has no more granular access controls other than this, be sure you trust everyone in this room - secure it! +# +# This should be a room alias or room ID - not a matrix.to URL. +# +# Note: By default, draupnir is fairly verbose - expect a lot of messages in this room. +# (see verboseLogging to adjust this a bit.) +managementRoom: "{{ matrix_bot_draupnir_management_room }}" + +# Whether draupnir should log a lot more messages in the room, +# mainly involves "all-OK" messages, and debugging messages for when draupnir checks bans in a room. +verboseLogging: false + +# The log level of terminal (or container) output, +# can be one of DEBUG, INFO, WARN and ERROR, in increasing order of importance and severity. +# +# This should be at INFO or DEBUG in order to get support for draupnir problems. +logLevel: "INFO" + +# Whether or not draupnir should synchronize policy lists immediately after startup. +# Equivalent to running '!draupnir sync'. +syncOnStartup: true + +# Whether or not draupnir should check moderation permissions in all protected rooms on startup. +# Equivalent to running `!draupnir verify`. +verifyPermissionsOnStartup: true + +# Whether or not draupnir should actually apply bans and policy lists, +# turn on to trial some untrusted configuration or lists. +noop: false + +# Whether draupnir should check member lists quicker (by using a different endpoint), +# keep in mind that enabling this will miss invited (but not joined) users. +# +# Turn on if your bot is in (very) large rooms, or in large amounts of rooms. +fasterMembershipChecks: false + +# A case-insensitive list of ban reasons to have the bot also automatically redact the user's messages for. +# +# If the bot sees you ban a user with a reason that is an (exact case-insensitive) match to this list, +# it will also remove the user's messages automatically. +# +# Typically this is useful to avoid having to give two commands to the bot. +# Advanced: Use asterisks to have the reason match using "globs" +# (f.e. "spam*testing" would match "spam for testing" as well as "spamtesting"). +# +# See here for more info: https://www.digitalocean.com/community/tools/glob +# Note: Keep in mind that glob is NOT regex! +automaticallyRedactForReasons: + - "spam" + - "advertising" + +# A list of rooms to protect. draupnir will add this to the list it knows from its account data. +# +# It won't, however, add it to the account data. +# Manually add the room via '!draupnir rooms add' to have it stay protected regardless if this config value changes. +# +# Note: These must be matrix.to URLs +#protectedRooms: +# - "https://matrix.to/#/#yourroom:example.org" + +# Whether or not to add all joined rooms to the "protected rooms" list +# (excluding the management room and watched policy list rooms, see below). +# +# Note that this effectively makes the protectedRooms and associated commands useless +# for regular rooms. +# +# Note: the management room is *excluded* from this condition. +# Explicitly add it as a protected room to protect it. +# +# Note: Ban list rooms the bot is watching but didn't create will not be protected. +# Explicitly add these rooms as a protected room list if you want them protected. +protectAllJoinedRooms: false + +# Increase this delay to have Mjölnir wait longer between two consecutive backgrounded +# operations. The total duration of operations will be longer, but the homeserver won't +# be affected as much. Conversely, decrease this delay to have Mjölnir chain operations +# faster. The total duration of operations will generally be shorter, but the performance +# of the homeserver may be more impacted. +backgroundDelayMS: 500 + +# Server administration commands, these commands will only work if draupnir is +# a global server administrator, and the bot's server is a Synapse instance. +#admin: +# # Whether or not draupnir can temporarily take control of any eligible account from the local homeserver who's in the room +# # (with enough permissions) to "make" a user an admin. +# # +# # This only works if a local user with enough admin permissions is present in the room. +# enableMakeRoomAdminCommand: false + +# Misc options for command handling and commands +commands: + # Whether or not the `!draupnir` prefix is necessary to submit commands. + # + # If `true`, will allow commands like `!ban`, `!help`, etc. + # + # Note: draupnir can also be pinged by display name instead of having to use + # the !draupnir prefix. For example, "my_moderator_bot: ban @spammer:example.org" + # will address only my_moderator_bot. + allowNoPrefix: false + + # Any additional bot prefixes that draupnir will listen to. i.e. adding `mod` will allow `!mod help`. + additionalPrefixes: + - "draupnir_bot" + - "draupnir" + + # Whether or not commands with a wildcard (*) will require an additional `--force` argument + # in the command to be able to be submitted. + confirmWildcardBan: true + +# Configuration specific to certain toggle-able protections +#protections: +# # Configuration for the wordlist plugin, which can ban users based if they say certain +# # blocked words shortly after joining. +# wordlist: +# # A list of case-insensitive keywords that the WordList protection will watch for from new users. +# # +# # WordList will ban users who use these words when first joining a room, so take caution when selecting them. +# # +# # For advanced usage, regex can also be used, see the following links for more information; +# # - https://www.digitalocean.com/community/tutorials/an-introduction-to-regular-expressions +# # - https://regexr.com/ +# # - https://regexone.com/ +# words: +# - "LoReM" +# - "IpSuM" +# - "DoLoR" +# - "aMeT" +# +# # For how long (in minutes) the user is "new" to the WordList plugin. +# # +# # After this time, the user will no longer be banned for using a word in the above wordlist. +# # +# # Set to zero to disable the timeout and make users *always* appear "new". +# # (users will always be banned if they say a bad word) +# minutesBeforeTrusting: 20 + +# Options for advanced monitoring of the health of the bot. +health: + # healthz options. These options are best for use in container environments + # like Kubernetes to detect how healthy the service is. The bot will report + # that it is unhealthy until it is able to process user requests. Typically + # this means that it'll flag itself as unhealthy for a number of minutes + # before saying "Now monitoring rooms" and flagging itself healthy. + # + # Health is flagged through HTTP status codes, defined below. + healthz: + # Whether the healthz integration should be enabled (default false) + enabled: false + + # The port to expose the webserver on. Defaults to 8080. + port: 8080 + + # The address to listen for requests on. Defaults to all addresses. + address: "0.0.0.0" + + # The path to expose the monitoring endpoint at. Defaults to `/healthz` + endpoint: "/healthz" + + # The HTTP status code which reports that the bot is healthy/ready to + # process requests. Typically this should not be changed. Defaults to + # 200. + healthyStatus: 200 + + # The HTTP status code which reports that the bot is not healthy/ready. + # Defaults to 418. + unhealthyStatus: 418 + +# Options for exposing web APIs. +#web: +# # Whether to enable web APIs. +# enabled: false +# +# # The port to expose the webserver on. Defaults to 8080. +# port: 8080 +# +# # The address to listen for requests on. Defaults to only the current +# # computer. +# address: localhost +# +# # Alternative setting to open to the entire web. Be careful, +# # as this will increase your security perimeter: +# # +# # address: "0.0.0.0" +# +# # A web API designed to intercept Matrix API +# # POST /_matrix/client/r0/rooms/{roomId}/report/{eventId} +# # and display readable abuse reports in the moderation room. +# # +# # If you wish to take advantage of this feature, you will need +# # to configure a reverse proxy, see e.g. test/nginx.conf +# abuseReporting: +# # Whether to enable this feature. +# enabled: false + +# Whether or not to actively poll synapse for abuse reports, to be used +# instead of intercepting client calls to synapse's abuse endpoint, when that +# isn't possible/practical. +pollReports: false + +# Whether or not new reports, received either by webapi or polling, +# should be printed to our managementRoom. +displayReports: false diff --git a/roles/custom/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 b/roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-draupnir.service.j2 similarity index 52% rename from roles/custom/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 rename to roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-draupnir.service.j2 index 82aca453..6995bcc3 100644 --- a/roles/custom/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 +++ b/roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-draupnir.service.j2 @@ -1,11 +1,11 @@ #jinja2: lstrip_blocks: "True" [Unit] -Description=matrix-prometheus-node-exporter -{% for service in matrix_prometheus_node_exporter_systemd_required_services_list %} +Description=Matrix Draupnir bot +{% for service in matrix_bot_draupnir_systemd_required_services_list %} Requires={{ service }} After={{ service }} {% endfor %} -{% for service in matrix_prometheus_node_exporter_systemd_wanted_services_list %} +{% for service in matrix_bot_draupnir_systemd_required_services_list %} Wants={{ service }} {% endfor %} DefaultDependencies=no @@ -13,32 +13,30 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus-node-exporter 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus-node-exporter 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-draupnir 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-draupnir 2>/dev/null || true' +# Intentional delay, so that the homeserver (we likely depend on) can manage to start. +ExecStartPre={{ matrix_host_command_sleep }} 5 -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-prometheus-node-exporter \ +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-bot-draupnir \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ - {% for arg in matrix_prometheus_node_exporter_container_extra_arguments %} + --network={{ matrix_docker_network }} \ + --mount type=bind,src={{ matrix_bot_draupnir_config_path }},dst=/data/config,ro \ + --mount type=bind,src={{ matrix_bot_draupnir_data_path }},dst=/data \ + {% for arg in matrix_bot_draupnir_container_extra_arguments %} {{ arg }} \ {% endfor %} - --network={{ matrix_docker_network }} \ - {% if matrix_prometheus_node_exporter_container_http_host_bind_port %} - -p {{ matrix_prometheus_node_exporter_container_http_host_bind_port }}:9100 \ - {% endif %} - --pid=host \ - --mount type=bind,src=/,dst=/host,ro,bind-propagation=rslave \ - {{ matrix_prometheus_node_exporter_docker_image }} \ - --path.rootfs=/host {{ matrix_prometheus_node_exporter_process_extra_arguments|join(' ') }} + {{ matrix_bot_draupnir_docker_image }} -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus-node-exporter 2>/dev/null || true' -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus-node-exporter 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-bot-draupnir 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-bot-draupnir 2>/dev/null || true' Restart=always RestartSec=30 -SyslogIdentifier=matrix-prometheus-node-exporter +SyslogIdentifier=matrix-bot-draupnir [Install] WantedBy=multi-user.target diff --git a/roles/custom/matrix-bot-honoroit/defaults/main.yml b/roles/custom/matrix-bot-honoroit/defaults/main.yml index 186ef3f4..e880df4f 100644 --- a/roles/custom/matrix-bot-honoroit/defaults/main.yml +++ b/roles/custom/matrix-bot-honoroit/defaults/main.yml @@ -10,8 +10,8 @@ matrix_bot_honoroit_docker_repo_version: "{{ matrix_bot_honoroit_version }}" matrix_bot_honoroit_docker_src_files_path: "{{ matrix_base_data_path }}/honoroit/docker-src" matrix_bot_honoroit_version: v0.9.16 -matrix_bot_honoroit_docker_image: "{{ matrix_bot_honoroit_docker_image_name_prefix }}honoroit:{{ matrix_bot_honoroit_version }}" -matrix_bot_honoroit_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_honoroit_container_image_self_build else 'registry.gitlab.com/etke.cc/' }}" +matrix_bot_honoroit_docker_image: "{{ matrix_bot_honoroit_docker_image_name_prefix }}etke.cc/honoroit:{{ matrix_bot_honoroit_version }}" +matrix_bot_honoroit_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_honoroit_container_image_self_build else 'registry.gitlab.com/' }}" matrix_bot_honoroit_docker_image_force_pull: "{{ matrix_bot_honoroit_docker_image.endswith(':latest') }}" matrix_bot_honoroit_base_path: "{{ matrix_base_data_path }}/honoroit" diff --git a/roles/custom/matrix-bot-maubot/defaults/main.yml b/roles/custom/matrix-bot-maubot/defaults/main.yml index 71353914..caee9b13 100644 --- a/roles/custom/matrix-bot-maubot/defaults/main.yml +++ b/roles/custom/matrix-bot-maubot/defaults/main.yml @@ -10,8 +10,9 @@ matrix_bot_maubot_docker_src_files_path: "{{ matrix_bot_maubot_base_path }}/dock matrix_bot_maubot_docker_repo_version: "{{ 'master' if matrix_bot_maubot_version == 'latest' else matrix_bot_maubot_version }}" -matrix_bot_maubot_version: v0.3.1 -matrix_bot_maubot_docker_image: "dock.mau.dev/maubot/maubot:{{ matrix_bot_maubot_version }}" +matrix_bot_maubot_version: v0.4.0 +matrix_bot_maubot_docker_image: "{{ matrix_bot_maubot_docker_image_name_prefix }}maubot/maubot:{{ matrix_bot_maubot_version }}" +matrix_bot_maubot_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_maubot_container_image_self_build else 'dock.mau.dev/' }}" matrix_bot_maubot_docker_image_force_pull: "{{ matrix_bot_maubot_docker_image.endswith(':latest') }}" matrix_bot_maubot_base_path: "{{ matrix_base_data_path }}/maubot" diff --git a/roles/custom/matrix-bot-mjolnir/defaults/main.yml b/roles/custom/matrix-bot-mjolnir/defaults/main.yml index b17f2f4b..ecbbdb88 100644 --- a/roles/custom/matrix-bot-mjolnir/defaults/main.yml +++ b/roles/custom/matrix-bot-mjolnir/defaults/main.yml @@ -4,7 +4,7 @@ matrix_bot_mjolnir_enabled: true -matrix_bot_mjolnir_version: "v1.6.1" +matrix_bot_mjolnir_version: "v1.6.4" matrix_bot_mjolnir_container_image_self_build: false matrix_bot_mjolnir_container_image_self_build_repo: "https://github.com/matrix-org/mjolnir.git" diff --git a/roles/custom/matrix-bot-postmoogle/defaults/main.yml b/roles/custom/matrix-bot-postmoogle/defaults/main.yml index 14795d1f..9f3dad24 100644 --- a/roles/custom/matrix-bot-postmoogle/defaults/main.yml +++ b/roles/custom/matrix-bot-postmoogle/defaults/main.yml @@ -9,9 +9,9 @@ matrix_bot_postmoogle_docker_repo: "https://gitlab.com/etke.cc/postmoogle.git" matrix_bot_postmoogle_docker_repo_version: "{{ 'main' if matrix_bot_postmoogle_version == 'latest' else matrix_bot_postmoogle_version }}" matrix_bot_postmoogle_docker_src_files_path: "{{ matrix_base_data_path }}/postmoogle/docker-src" -matrix_bot_postmoogle_version: v0.9.10 -matrix_bot_postmoogle_docker_image: "{{ matrix_bot_postmoogle_docker_image_name_prefix }}postmoogle:{{ matrix_bot_postmoogle_version }}" -matrix_bot_postmoogle_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_postmoogle_container_image_self_build else 'registry.gitlab.com/etke.cc/' }}" +matrix_bot_postmoogle_version: v0.9.12 +matrix_bot_postmoogle_docker_image: "{{ matrix_bot_postmoogle_docker_image_name_prefix }}etke.cc/postmoogle:{{ matrix_bot_postmoogle_version }}" +matrix_bot_postmoogle_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_postmoogle_container_image_self_build else 'registry.gitlab.com/' }}" matrix_bot_postmoogle_docker_image_force_pull: "{{ matrix_bot_postmoogle_docker_image.endswith(':latest') }}" matrix_bot_postmoogle_base_path: "{{ matrix_base_data_path }}/postmoogle" diff --git a/roles/custom/matrix-bot-postmoogle/templates/systemd/matrix-bot-postmoogle.service.j2 b/roles/custom/matrix-bot-postmoogle/templates/systemd/matrix-bot-postmoogle.service.j2 index f2610600..ab1177f6 100644 --- a/roles/custom/matrix-bot-postmoogle/templates/systemd/matrix-bot-postmoogle.service.j2 +++ b/roles/custom/matrix-bot-postmoogle/templates/systemd/matrix-bot-postmoogle.service.j2 @@ -1,6 +1,6 @@ #jinja2: lstrip_blocks: "True" [Unit] -Description=Matrix helpdesk bot +Description=Matrix Postmoogle bot {% for service in matrix_bot_postmoogle_systemd_required_services_list %} Requires={{ service }} After={{ service }} diff --git a/roles/custom/matrix-bridge-heisenbridge/defaults/main.yml b/roles/custom/matrix-bridge-heisenbridge/defaults/main.yml index 48101db1..ba5471cc 100644 --- a/roles/custom/matrix-bridge-heisenbridge/defaults/main.yml +++ b/roles/custom/matrix-bridge-heisenbridge/defaults/main.yml @@ -4,7 +4,7 @@ matrix_heisenbridge_enabled: true -matrix_heisenbridge_version: 1.14.0 +matrix_heisenbridge_version: 1.14.2 matrix_heisenbridge_docker_image: "{{ matrix_container_global_registry_prefix }}hif1/heisenbridge:{{ matrix_heisenbridge_version }}" matrix_heisenbridge_docker_image_force_pull: "{{ matrix_heisenbridge_docker_image.endswith(':latest') }}" diff --git a/roles/custom/matrix-bridge-hookshot/defaults/main.yml b/roles/custom/matrix-bridge-hookshot/defaults/main.yml index 14c0e6df..aa4ae45c 100644 --- a/roles/custom/matrix-bridge-hookshot/defaults/main.yml +++ b/roles/custom/matrix-bridge-hookshot/defaults/main.yml @@ -10,7 +10,7 @@ matrix_hookshot_container_image_self_build: false matrix_hookshot_container_image_self_build_repo: "https://github.com/matrix-org/matrix-hookshot.git" matrix_hookshot_container_image_self_build_branch: "{{ 'main' if matrix_hookshot_version == 'latest' else matrix_hookshot_version }}" -matrix_hookshot_version: 2.5.0 +matrix_hookshot_version: 2.7.0 matrix_hookshot_docker_image: "{{ matrix_hookshot_docker_image_name_prefix }}halfshot/matrix-hookshot:{{ matrix_hookshot_version }}" matrix_hookshot_docker_image_name_prefix: "{{ 'localhost/' if matrix_hookshot_container_image_self_build else matrix_container_global_registry_prefix }}" diff --git a/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml index a66d5c6a..0211f8f3 100644 --- a/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-discord/defaults/main.yml @@ -8,7 +8,7 @@ matrix_mautrix_discord_container_image_self_build: false matrix_mautrix_discord_container_image_self_build_repo: "https://mau.dev/mautrix/discord.git" matrix_mautrix_discord_container_image_self_build_branch: "{{ 'main' if matrix_mautrix_discord_version == 'latest' else matrix_mautrix_discord_version }}" -matrix_mautrix_discord_version: latest +matrix_mautrix_discord_version: v0.1.0 # See: https://mau.dev/mautrix/discord/container_registry matrix_mautrix_discord_docker_image: "{{ matrix_mautrix_discord_docker_image_name_prefix }}mautrix/discord:{{ matrix_mautrix_discord_version }}" matrix_mautrix_discord_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_discord_container_image_self_build else 'dock.mau.dev/' }}" diff --git a/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml index 98972014..8c19139a 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-facebook/defaults/main.yml @@ -46,6 +46,16 @@ matrix_mautrix_facebook_homeserver_token: '' # If false, created portal rooms will never be federated. matrix_mautrix_facebook_federate_rooms: true +# Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_facebook_metrics_proxying_enabled`. +matrix_mautrix_facebook_metrics_enabled: false + +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-facebook`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_facebook_metrics_proxying_enabled: false + matrix_mautrix_facebook_bridge_permissions: | {{ {matrix_mautrix_facebook_homeserver_domain: 'user'} diff --git a/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml index b9f95cb7..17e2b138 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-bridge-mautrix-facebook/tasks/inject_into_nginx_proxy.yml @@ -42,3 +42,28 @@ URL endpoint to the matrix-mautrix-facebook container. You can expose the container's port using the `matrix_mautrix_facebook_container_http_host_bind_port` variable. when: "not matrix_nginx_proxy_enabled | default(False) | bool" + +- when: matrix_mautrix_facebook_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-facebook metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-facebook) + ansible.builtin.set_fact: + matrix_mautrix_facebook_nginx_metrics_configuration_block: | + location /metrics/mautrix-facebook { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-facebook:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + + - name: Register mautrix-facebook metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-facebook) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_facebook_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 index b8e25fee..184ea780 100644 --- a/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 @@ -62,7 +62,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: false + enabled: {{ matrix_mautrix_facebook_metrics_enabled | to_json }} listen_port: 8000 # Bridge config diff --git a/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml index 0e726aa8..80d59162 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-instagram/defaults/main.yml @@ -47,6 +47,16 @@ matrix_mautrix_instagram_homeserver_token: '' # If false, created portal rooms will never be federated. matrix_mautrix_instagram_federate_rooms: true +# Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_instagram_metrics_proxying_enabled`. +matrix_mautrix_instagram_metrics_enabled: false + +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-instagram`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_instagram_metrics_proxying_enabled: false + # Database-related configuration fields. # # To use Postgres: diff --git a/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 00000000..99a36c14 --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-instagram/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,35 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append mautrix-whatapp-metrics's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-instagram role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- when: matrix_mautrix_instagram_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-instagram metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-instagram) + ansible.builtin.set_fact: + matrix_mautrix_instagram_nginx_metrics_configuration_block: | + location /metrics/mautrix-instagram { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-instagram:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + + - name: Register mautrix-instagram metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-instagram) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_instagram_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml index 2bd0417e..0b3a722d 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-instagram/tasks/main.yml @@ -1,5 +1,14 @@ --- +- block: + - when: matrix_mautrix_instagram_enabled | bool and matrix_mautrix_instagram_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + - block: - when: matrix_mautrix_instagram_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" diff --git a/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 index 039b9bfe..93bbfe39 100644 --- a/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 @@ -57,7 +57,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: false + enabled: {{ matrix_mautrix_instagram_metrics_enabled | to_json }} listen_port: 8000 instagram: diff --git a/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml index 3bdb02c8..8cd285ca 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-signal/defaults/main.yml @@ -12,7 +12,8 @@ matrix_mautrix_signal_docker_src_files_path: "{{ matrix_base_data_path }}/mautri matrix_mautrix_signal_version: v0.4.2 matrix_mautrix_signal_daemon_version: 0.23.1 # See: https://mau.dev/mautrix/signal/container_registry -matrix_mautrix_signal_docker_image: "dock.mau.dev/mautrix/signal:{{ matrix_mautrix_signal_version }}" +matrix_mautrix_signal_docker_image: "{{ matrix_mautrix_signal_docker_image_name_prefix }}mautrix/signal:{{ matrix_mautrix_signal_version }}" +matrix_mautrix_signal_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_signal_container_image_self_build else 'dock.mau.dev/' }}" matrix_mautrix_signal_docker_image_force_pull: "{{ matrix_mautrix_signal_docker_image.endswith(':latest') }}" matrix_mautrix_signal_daemon_container_image_self_build: false @@ -20,7 +21,8 @@ matrix_mautrix_signal_daemon_docker_repo: "https://gitlab.com/signald/signald" matrix_mautrix_signal_daemon_docker_repo_version: "{{ 'master' if matrix_mautrix_signal_daemon_version == 'latest' else matrix_mautrix_signal_daemon_version }}" matrix_mautrix_signal_daemon_docker_src_files_path: "{{ matrix_base_data_path }}/mautrix-signald/docker-src" -matrix_mautrix_signal_daemon_docker_image: "docker.io/signald/signald:{{ matrix_mautrix_signal_daemon_docker_image_tag }}" +matrix_mautrix_signal_daemon_docker_image: "{{ matrix_mautrix_signal_daemon_docker_image_name_prefix }}signald/signald:{{ matrix_mautrix_signal_daemon_docker_image_tag }}" +matrix_mautrix_signal_daemon_docker_image_name_prefix: "docker.io/" matrix_mautrix_signal_daemon_docker_image_force_pull: "{{ matrix_mautrix_signal_daemon_docker_image_tag.endswith(':latest') }}" matrix_mautrix_signal_daemon_docker_image_tag: "{{ matrix_mautrix_signal_daemon_version }}" @@ -68,6 +70,16 @@ matrix_mautrix_signal_logging_level: WARNING # If false, created portal rooms will never be federated. matrix_mautrix_signal_federate_rooms: true +# Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_signal_metrics_proxying_enabled`. +matrix_mautrix_signal_metrics_enabled: false + +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-signal`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_signal_metrics_proxying_enabled: false + # Database-related configuration fields # # This bridge only supports postgres. diff --git a/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 00000000..fe85254b --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-signal/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,35 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append mautrix-whatapp-metrics's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-signal role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- when: matrix_mautrix_signal_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-signal metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-signal) + ansible.builtin.set_fact: + matrix_mautrix_signal_nginx_metrics_configuration_block: | + location /metrics/mautrix-signal { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-signal:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + + - name: Register mautrix-signal metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-signal) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_signal_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml index 0c0a1239..9473344b 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-signal/tasks/main.yml @@ -1,5 +1,14 @@ --- +- block: + - when: matrix_mautrix_signal_enabled | bool and matrix_mautrix_signal_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + - block: - when: matrix_mautrix_signal_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" diff --git a/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 index 796a6e41..828ff940 100644 --- a/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-signal/templates/config.yaml.j2 @@ -67,7 +67,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: false + enabled: {{ matrix_mautrix_signal_metrics_enabled | to_json }} listen_port: 8000 # Manhole config. diff --git a/roles/custom/matrix-bridge-mautrix-slack/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-slack/defaults/main.yml new file mode 100644 index 00000000..4d8e3f2f --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-slack/defaults/main.yml @@ -0,0 +1,138 @@ +--- +# mautrix-slack is a Matrix <-> Slack bridge +# Project source code URL: https://github.com/mautrix/slack + +matrix_mautrix_slack_enabled: true + +matrix_mautrix_slack_container_image_self_build: false +matrix_mautrix_slack_container_image_self_build_repo: "https://mau.dev/mautrix/slack.git" +matrix_mautrix_slack_container_image_self_build_branch: "{{ 'main' if matrix_mautrix_slack_version == 'latest' else matrix_mautrix_slack_version }}" + +matrix_mautrix_slack_version: latest +# See: https://mau.dev/mautrix/slack/container_registry +matrix_mautrix_slack_docker_image: "{{ matrix_mautrix_slack_docker_image_name_prefix }}mautrix/slack:{{ matrix_mautrix_slack_version }}" +matrix_mautrix_slack_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_slack_container_image_self_build else 'dock.mau.dev/' }}" +matrix_mautrix_slack_docker_image_force_pull: "{{ matrix_mautrix_slack_docker_image.endswith(':latest') }}" + +matrix_mautrix_slack_base_path: "{{ matrix_base_data_path }}/mautrix-slack" +matrix_mautrix_slack_config_path: "{{ matrix_mautrix_slack_base_path }}/config" +matrix_mautrix_slack_data_path: "{{ matrix_mautrix_slack_base_path }}/data" +matrix_mautrix_slack_docker_src_files_path: "{{ matrix_mautrix_slack_base_path }}/docker-src" + +matrix_mautrix_slack_homeserver_address: "{{ matrix_homeserver_container_url }}" +matrix_mautrix_slack_homeserver_domain: "{{ matrix_domain }}" +matrix_mautrix_slack_appservice_address: "http://matrix-mautrix-slack:8080" + +matrix_mautrix_slack_command_prefix: "!slack" + +matrix_mautrix_slack_bridge_permissions: | + {{ + {matrix_mautrix_slack_homeserver_domain: 'user'} + | combine({matrix_admin: 'admin'} if matrix_admin else {}) + }} + +# A list of extra arguments to pass to the container +matrix_mautrix_slack_container_extra_arguments: [] + +# List of systemd services that matrix-mautrix-slack.service depends on. +matrix_mautrix_slack_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-mautrix-slack.service wants +matrix_mautrix_slack_systemd_wanted_services_list: [] + +matrix_mautrix_slack_appservice_token: '' +matrix_mautrix_slack_homeserver_token: '' + +matrix_mautrix_slack_appservice_bot_username: slackbot + +# Minimum severity of journal log messages. +# Options: debug, info, warn, error, fatal +matrix_mautrix_slack_logging_level: 'warn' + +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_mautrix_slack_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_mautrix_slack_database_*` variables +matrix_mautrix_slack_database_engine: 'sqlite' + +matrix_mautrix_slack_sqlite_database_path_local: "{{ matrix_mautrix_slack_data_path }}/mautrix-slack.db" +matrix_mautrix_slack_sqlite_database_path_in_container: "/data/mautrix-slack.db" + +matrix_mautrix_slack_database_username: 'matrix_mautrix_slack' +matrix_mautrix_slack_database_password: 'some-password' +matrix_mautrix_slack_database_hostname: '' +matrix_mautrix_slack_database_port: 5432 +matrix_mautrix_slack_database_name: 'matrix_mautrix_slack' + +matrix_mautrix_slack_database_connection_string: 'postgresql://{{ matrix_mautrix_slack_database_username }}:{{ matrix_mautrix_slack_database_password }}@{{ matrix_mautrix_slack_database_hostname }}:{{ matrix_mautrix_slack_database_port }}/{{ matrix_mautrix_slack_database_name }}?sslmode=disable' + +matrix_mautrix_slack_appservice_database_type: "{{ + { + 'sqlite': 'sqlite3', + 'postgres':'postgres', + }[matrix_mautrix_slack_database_engine] +}}" + +matrix_mautrix_slack_appservice_database_uri: "{{ + { + 'sqlite': matrix_mautrix_slack_sqlite_database_path_in_container, + 'postgres': matrix_mautrix_slack_database_connection_string, + }[matrix_mautrix_slack_database_engine] +}}" + +# Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). +matrix_mautrix_slack_login_shared_secret: '' +matrix_mautrix_slack_bridge_login_shared_secret_map: + "{{ {matrix_mautrix_slack_homeserver_domain: matrix_mautrix_slack_login_shared_secret} if matrix_mautrix_slack_login_shared_secret else {} }}" + +# Servers to always allow double puppeting from +matrix_mautrix_slack_bridge_double_puppet_server_map: + "{{ matrix_mautrix_slack_homeserver_domain : matrix_mautrix_slack_homeserver_address }}" + +# Default mautrix-slack configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. +# +# For a more advanced customization, you can extend the default (see `matrix_mautrix_slack_configuration_extension_yaml`) +# or completely replace this variable with your own template. +matrix_mautrix_slack_configuration_yaml: "{{ lookup('template', 'templates/config.yaml.j2') }}" + +matrix_mautrix_slack_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_mautrix_slack_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_mautrix_slack_configuration_yaml`. + +matrix_mautrix_slack_configuration_extension: "{{ matrix_mautrix_slack_configuration_extension_yaml | from_yaml if matrix_mautrix_slack_configuration_extension_yaml | from_yaml is mapping else {} }}" + +# Holds the final configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_mautrix_slack_configuration_yaml`. +matrix_mautrix_slack_configuration: "{{ matrix_mautrix_slack_configuration_yaml | from_yaml | combine(matrix_mautrix_slack_configuration_extension, recursive=True) }}" + +matrix_mautrix_slack_registration_yaml: | + id: slack + url: {{ matrix_mautrix_slack_appservice_address }} + as_token: "{{ matrix_mautrix_slack_appservice_token }}" + hs_token: "{{ matrix_mautrix_slack_homeserver_token }}" + # See https://github.com/mautrix/signal/issues/43 + sender_localpart: _bot_{{ matrix_mautrix_slack_appservice_bot_username }} + rate_limited: false + namespaces: + users: + - regex: '^@slack_[a-z-A-Z0-9\-]+:{{ matrix_mautrix_slack_homeserver_domain | regex_escape }}$' + exclusive: true + - exclusive: true + regex: '^@{{ matrix_mautrix_slack_appservice_bot_username | regex_escape }}:{{ matrix_mautrix_slack_homeserver_domain | regex_escape }}$' + de.sorunome.msc2409.push_ephemeral: true + +matrix_mautrix_slack_registration: "{{ matrix_mautrix_slack_registration_yaml | from_yaml }}" + +# Enable End-to-bridge encryption +matrix_mautrix_slack_bridge_encryption_allow: false +matrix_mautrix_slack_bridge_encryption_default: "{{ matrix_mautrix_slack_bridge_encryption_allow }}" +matrix_mautrix_slack_bridge_encryption_key_sharing_allow: "{{ matrix_mautrix_slack_bridge_encryption_allow }}" diff --git a/roles/custom/matrix-bridge-mautrix-slack/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-slack/tasks/main.yml new file mode 100644 index 00000000..bff90549 --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-slack/tasks/main.yml @@ -0,0 +1,20 @@ +--- + +- block: + - when: matrix_mautrix_slack_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - when: matrix_mautrix_slack_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + tags: + - setup-all + - setup-mautrix-slack + - install-all + - install-mautrix-slack + +- block: + - when: not matrix_mautrix_slack_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + tags: + - setup-all + - setup-mautrix-slack diff --git a/roles/custom/matrix-bridge-mautrix-slack/tasks/setup_install.yml b/roles/custom/matrix-bridge-mautrix-slack/tasks/setup_install.yml new file mode 100644 index 00000000..f266f026 --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-slack/tasks/setup_install.yml @@ -0,0 +1,108 @@ +--- + +- ansible.builtin.set_fact: + matrix_mautrix_slack_requires_restart: false + +- when: "matrix_mautrix_slack_database_engine == 'postgres'" + block: + - name: Check if an SQLite database already exists + ansible.builtin.stat: + path: "{{ matrix_mautrix_slack_sqlite_database_path_local }}" + register: matrix_mautrix_slack_sqlite_database_path_local_stat_result + + - when: "matrix_mautrix_slack_sqlite_database_path_local_stat_result.stat.exists | bool" + block: + - ansible.builtin.include_role: + name: galaxy/com.devture.ansible.role.postgres + tasks_from: migrate_db_to_postgres + vars: + devture_postgres_db_migration_request: + src: "{{ matrix_mautrix_slack_sqlite_database_path_local }}" + dst: "{{ matrix_mautrix_slack_database_connection_string }}" + caller: "{{ role_path | basename }}" + engine_variable_name: 'matrix_mautrix_slack_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mautrix-slack.service'] + pgloader_options: ['--with "quote identifiers"'] + + - ansible.builtin.set_fact: + matrix_mautrix_slack_requires_restart: true + +- name: Ensure Mautrix Slack paths exists + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - {path: "{{ matrix_mautrix_slack_base_path }}", when: true} + - {path: "{{ matrix_mautrix_slack_config_path }}", when: true} + - {path: "{{ matrix_mautrix_slack_data_path }}", when: true} + - {path: "{{ matrix_mautrix_slack_docker_src_files_path }}", when: "{{ matrix_mautrix_slack_container_image_self_build }}"} + when: item.when | bool + +- name: Ensure Mautrix Slack image is pulled + community.docker.docker_image: + name: "{{ matrix_mautrix_slack_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_mautrix_slack_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mautrix_slack_docker_image_force_pull }}" + when: not matrix_mautrix_slack_container_image_self_build + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure Mautrix slack repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_mautrix_slack_container_image_self_build_repo }}" + dest: "{{ matrix_mautrix_slack_docker_src_files_path }}" + version: "{{ matrix_mautrix_slack_container_image_self_build_branch }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_mautrix_slack_git_pull_results + when: "matrix_mautrix_slack_container_image_self_build | bool" + +- name: Ensure Mautrix slack Docker image is built + community.docker.docker_image: + name: "{{ matrix_mautrix_slack_docker_image }}" + source: build + force_source: "{{ matrix_mautrix_slack_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mautrix_slack_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_mautrix_slack_docker_src_files_path }}" + pull: true + when: "matrix_mautrix_slack_container_image_self_build | bool" + +- name: Ensure mautrix-slack config.yaml installed + ansible.builtin.copy: + content: "{{ matrix_mautrix_slack_configuration | to_nice_yaml(indent=2, width=999999) }}" + dest: "{{ matrix_mautrix_slack_config_path }}/config.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure mautrix-slack registration.yaml installed + ansible.builtin.copy: + content: "{{ matrix_mautrix_slack_registration | to_nice_yaml(indent=2, width=999999) }}" + dest: "{{ matrix_mautrix_slack_config_path }}/registration.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-mautrix-slack.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-mautrix-slack.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-mautrix-slack.service" + mode: 0644 + register: matrix_mautrix_slack_systemd_service_result + +- name: Ensure matrix-mautrix-slack.service restarted, if necessary + ansible.builtin.service: + name: "matrix-mautrix-slack.service" + state: restarted + daemon_reload: true + when: "matrix_mautrix_slack_requires_restart | bool" diff --git a/roles/custom/matrix-bridge-mautrix-slack/tasks/setup_uninstall.yml b/roles/custom/matrix-bridge-mautrix-slack/tasks/setup_uninstall.yml new file mode 100644 index 00000000..19350b44 --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-slack/tasks/setup_uninstall.yml @@ -0,0 +1,20 @@ +--- + +- name: Check existence of matrix-mautrix-slack service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-mautrix-slack.service" + register: matrix_mautrix_slack_service_stat + +- when: matrix_mautrix_slack_service_stat.stat.exists | bool + block: + - name: Ensure matrix-mautrix-slack is stopped + ansible.builtin.service: + name: matrix-mautrix-slack + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-mautrix-slack.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-mautrix-slack.service" + state: absent diff --git a/roles/custom/matrix-bridge-mautrix-slack/tasks/validate_config.yml b/roles/custom/matrix-bridge-mautrix-slack/tasks/validate_config.yml new file mode 100644 index 00000000..3c764a4e --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-slack/tasks/validate_config.yml @@ -0,0 +1,20 @@ +--- + +- name: Fail if required mautrix-slack settings not defined + ansible.builtin.fail: + msg: >- + You need to define a required configuration setting (`{{ item.name }}`). + when: "item.when | bool and vars[item.name] == ''" + with_items: + - {'name': 'matrix_mautrix_slack_appservice_token', when: true} + - {'name': 'matrix_mautrix_slack_homeserver_token', when: true} + - {'name': 'matrix_mautrix_slack_database_hostname', when: "{{ matrix_mautrix_slack_database_engine == 'postgres' }}"} + +- name: Fail if appservice-slack and mautrix-slack have conflicting bot usernames + when: matrix_appservice_slack_enabled | default(False) | bool and matrix_mautrix_slack_appservice_bot_username == matrix_appservice_slack_bot_name | default ('') + ansible.builtin.fail: + msg: | + The appservice-slack and mautrix-slack components are both enabled and use the same bot username ({{ matrix_mautrix_slack_appservice_bot_username }}), as per their default configuration, which causes a conflcit. + To resolve the conflict, make one of these components use a different username. + Consider either changing `matrix_mautrix_slack_appservice_bot_username` (the bot username for the mautrix-slack component) or `matrix_appservice_slack_bot_name` (the bot username for the appservice-slack component). + We recommend that you change the username for the newly-added (and yet unused) component. diff --git a/roles/custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2 new file mode 100644 index 00000000..5c60c66f --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-slack/templates/config.yaml.j2 @@ -0,0 +1,240 @@ +#jinja2: lstrip_blocks: "True" +# Homeserver details. +homeserver: + # The address that this appservice can use to connect to the homeserver. + address: {{ matrix_mautrix_slack_homeserver_address | to_json }} + # The domain of the homeserver (for MXIDs, etc). + domain: {{ matrix_mautrix_slack_homeserver_domain | to_json }} + + # What software is the homeserver running? + # Standard Matrix homeservers like Synapse, Dendrite and Conduit should just use "standard" here. + software: standard + # The URL to push real-time bridge status to. + # If set, the bridge will make POST requests to this URL whenever a user's slack connection state changes. + # The bridge will use the appservice as_token to authorize requests. + status_endpoint: null + # Endpoint for reporting per-message status. + message_send_checkpoint_endpoint: null + # Does the homeserver support https://github.com/matrix-org/matrix-spec-proposals/pull/2246? + async_media: false + +# Application service host/registration related details. +# Changing these values requires regeneration of the registration. +appservice: + # The address that the homeserver can use to connect to this appservice. + address: {{ matrix_mautrix_slack_appservice_address | to_json }} + + # The hostname and port where this appservice should listen. + hostname: 0.0.0.0 + port: 8080 + + # Database config. + database: + # The database type. "sqlite3-fk-wal" and "postgres" are supported. + type: {{ matrix_mautrix_slack_appservice_database_type|to_json }} + # The database URI. + # SQLite: A raw file path is supported, but `file:?_txlock=immediate` is recommended. + # https://github.com/mattn/go-sqlite3#connection-string + # Postgres: Connection string. For example, postgres://user:password@host/database?sslmode=disable + # To connect via Unix socket, use something like postgres:///dbname?host=/var/run/postgresql + uri: {{ matrix_mautrix_slack_appservice_database_uri|to_json }} + # Maximum number of connections. Mostly relevant for Postgres. + max_open_conns: 20 + max_idle_conns: 2 + # Maximum connection idle time and lifetime before they're closed. Disabled if null. + # Parsed with https://pkg.go.dev/time#ParseDuration + max_conn_idle_time: null + max_conn_lifetime: null + + # The unique ID of this appservice. + id: slack + # Appservice bot details. + bot: + # Username of the appservice bot. + username: {{ matrix_mautrix_slack_appservice_bot_username|to_json }} + # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty + # to leave display name/avatar as-is. + displayname: Slack bridge bot + avatar: mxc://maunium.net/pVtzLmChZejGxLqmXtQjFxem + # Whether or not to receive ephemeral events via appservice transactions. + # Requires MSC2409 support (i.e. Synapse 1.22+). + # You should disable bridge -> sync_with_custom_puppets when this is enabled. + ephemeral_events: true + + # Should incoming events be handled asynchronously? + # This may be necessary for large public instances with lots of messages going through. + # However, messages will not be guaranteed to be bridged in the same order they were sent in. + async_transactions: false + + # Authentication tokens for AS <-> HS communication. Autogenerated; do not modify. + as_token: {{ matrix_mautrix_slack_appservice_token | to_json }} + hs_token: {{ matrix_mautrix_slack_homeserver_token | to_json }} + +# Bridge config +bridge: + # Localpart template of MXIDs for Slack users. + # {{ '{{.}}' }} is replaced with the internal ID of the Slack user. + username_template: "{{ 'slack_{{.}}' }}" + # Displayname template for Slack users. + # TODO: document variables + displayname_template: "{{ '{{.RealName}} (S)' }}" + bot_displayname_template: "{{ '{{.Name}} (bot)' }}" + channel_name_template: "{{ '#{{.Name}} ({{.TeamName}})' }}" + + portal_message_buffer: 128 + + # Should the bridge send a read receipt from the bridge bot when a message has been sent to Slack? + delivery_receipts: true + # Whether the bridge should send the message status as a custom com.beeper.message_send_status event. + message_status_events: false + # Whether the bridge should send error notices via m.notice events when a message fails to bridge. + message_error_notices: true + + # Should the bridge sync with double puppeting to receive EDUs that aren't normally sent to appservices. + sync_with_custom_puppets: false + # Should the bridge update the m.direct account data event when double puppeting is enabled. + # Note that updating the m.direct event is not atomic (except with mautrix-asmux) + # and is therefore prone to race conditions. + sync_direct_chat_list: false + + # Servers to always allow double puppeting from + double_puppet_server_map: + "{{ matrix_mautrix_slack_homeserver_domain }}": {{ matrix_mautrix_slack_homeserver_address }} + # Allow using double puppeting from any server with a valid client .well-known file. + double_puppet_allow_discovery: false + # Shared secrets for https://github.com/devture/matrix-synapse-shared-secret-auth + # + # If set, double puppeting will be enabled automatically for local users + # instead of users having to find an access token and run `login-matrix` + # manually. + login_shared_secret_map: {{ matrix_mautrix_slack_bridge_login_shared_secret_map|to_json }} + + message_handling_timeout: + # Send an error message after this timeout, but keep waiting for the response until the deadline. + # This is counted from the origin_server_ts, so the warning time is consistent regardless of the source of delay. + # If the message is older than this when it reaches the bridge, the message won't be handled at all. + error_after: 10s + # Drop messages after this timeout. They may still go through if the message got sent to the servers. + # This is counted from the time the bridge starts handling the message. + deadline: 60s + + # The prefix for commands. Only required in non-management rooms. + command_prefix: "{{ matrix_mautrix_slack_command_prefix }}" + # Messages sent upon joining a management room. + # Markdown is supported. The defaults are listed below. + management_room_text: + # Sent when joining a room. + welcome: "Hello, I'm a Slack bridge bot." + # Sent when joining a management room and the user is already logged in. + welcome_connected: "Use `help` for help." + # Sent when joining a management room and the user is not logged in. + welcome_unconnected: "Use `help` for help, or `login-token` or `login-password` to log in." + # Optional extra text sent when joining a management room. + additional_help: "" + + backfill: + # Allow backfilling at all? Requires MSC2716 support on homeserver. + enable: false + + # If a backfilled chat is older than this number of hours, mark it as read even if it's unread on Slack. + # Set to -1 to let any chat be unread. + unread_hours_threshold: 720 + + # Number of messages to immediately backfill when creating a portal. + immediate_messages: 10 + + # Settings for incremental backfill of history. + incremental: + # Maximum number of messages to backfill per batch. + messages_per_batch: 100 + # The number of seconds to wait after backfilling the batch of messages. + post_batch_delay: 20 + # The maximum number of messages to backfill per portal, split by the chat type. + # If set to -1, all messages in the chat will eventually be backfilled. + max_messages: + # Channels + channel: -1 + # Group direct messages + group_dm: -1 + # 1:1 direct messages + dm: -1 + + # End-to-bridge encryption support options. + # + # See https://docs.mau.fi/bridges/general/end-to-bridge-encryption.html for more info. + encryption: + # Allow encryption, work in group chat rooms with e2ee enabled + allow: {{ matrix_mautrix_slack_bridge_encryption_allow|to_json }} + # Default to encryption, force-enable encryption in all portals the bridge creates + # This will cause the bridge bot to be in private chats for the encryption to work properly. + default: {{ matrix_mautrix_slack_bridge_encryption_default|to_json }} + # Whether to use MSC2409/MSC3202 instead of /sync long polling for receiving encryption-related data. + appservice: false + # Require encryption, drop any unencrypted messages. + require: false + # Enable key sharing? If enabled, key requests for rooms where users are in will be fulfilled. + # You must use a client that supports requesting keys from other users to use this feature. + allow_key_sharing: {{ matrix_mautrix_slack_bridge_encryption_key_sharing_allow|to_json }} + # What level of device verification should be required from users? + # + # Valid levels: + # unverified - Send keys to all device in the room. + # cross-signed-untrusted - Require valid cross-signing, but trust all cross-signing keys. + # cross-signed-tofu - Require valid cross-signing, trust cross-signing keys on first use (and reject changes). + # cross-signed-verified - Require valid cross-signing, plus a valid user signature from the bridge bot. + # Note that creating user signatures from the bridge bot is not currently possible. + # verified - Require manual per-device verification + # (currently only possible by modifying the `trust` column in the `crypto_device` database table). + verification_levels: + # Minimum level for which the bridge should send keys to when bridging messages from WhatsApp to Matrix. + receive: unverified + # Minimum level that the bridge should accept for incoming Matrix messages. + send: unverified + # Minimum level that the bridge should require for accepting key requests. + share: cross-signed-tofu + # Options for Megolm room key rotation. These options allow you to + # configure the m.room.encryption event content. See: + # https://spec.matrix.org/v1.3/client-server-api/#mroomencryption for + # more information about that event. + rotation: + # Enable custom Megolm room key rotation settings. Note that these + # settings will only apply to rooms created after this option is + # set. + enable_custom: false + # The maximum number of milliseconds a session should be used + # before changing it. The Matrix spec recommends 604800000 (a week) + # as the default. + milliseconds: 604800000 + # The maximum number of messages that should be sent with a given a + # session before changing it. The Matrix spec recommends 100 as the + # default. + messages: 100 + + # Settings for provisioning API + provisioning: + # Prefix for the provisioning API paths. + prefix: /_matrix/provision + # Shared secret for authentication. If set to "generate", a random secret will be generated, + # or if set to "disable", the provisioning API will be disabled. + shared_secret: generate + + # Permissions for using the bridge. + # Permitted values: + # relay - Talk through the relaybot (if enabled), no access otherwise + # user - Access to use the bridge to chat with a Slack account. + # admin - User level and some additional administration tools + # Permitted keys: + # * - All Matrix users + # domain - All users on that homeserver + # mxid - Specific user + permissions: {{ matrix_mautrix_slack_bridge_permissions|to_json }} + +logging: + directory: ./logs + file_name_format: '' + file_date_format: "2006-01-02" + file_mode: 384 + timestamp_format: Jan _2, 2006 15:04:05 + print_level: {{ matrix_mautrix_slack_logging_level | to_json }} + print_json: false + file_json: false diff --git a/roles/custom/matrix-bridge-mautrix-slack/templates/systemd/matrix-mautrix-slack.service.j2 b/roles/custom/matrix-bridge-mautrix-slack/templates/systemd/matrix-mautrix-slack.service.j2 new file mode 100644 index 00000000..af5f3c03 --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-slack/templates/systemd/matrix-mautrix-slack.service.j2 @@ -0,0 +1,43 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Matrix Mautrix Slack bridge +{% for service in matrix_mautrix_slack_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_mautrix_slack_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-mautrix-slack 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-mautrix-slack 2>/dev/null || true' + +# Intentional delay, so that the homeserver (we likely depend on) can manage to start. +ExecStartPre={{ matrix_host_command_sleep }} 5 + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-mautrix-slack \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --network={{ matrix_docker_network }} \ + --mount type=bind,src={{ matrix_mautrix_slack_config_path }},dst=/config,ro \ + --mount type=bind,src={{ matrix_mautrix_slack_data_path }},dst=/data \ + --workdir=/data \ + {% for arg in matrix_mautrix_slack_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_mautrix_slack_docker_image }} \ + /usr/bin/mautrix-slack -c /config/config.yaml -r /config/registration.yaml --no-update + +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-mautrix-slack 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-mautrix-slack 2>/dev/null || true' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-mautrix-slack + +[Install] +WantedBy=multi-user.target diff --git a/roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml index 5d2c0c88..1221da84 100644 --- a/roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-telegram/defaults/main.yml @@ -9,7 +9,8 @@ matrix_telegram_lottieconverter_container_image_self_build_mask_arch: false matrix_telegram_lottieconverter_docker_repo: "https://mau.dev/tulir/lottieconverter.git" matrix_telegram_lottieconverter_docker_repo_version: "master" matrix_telegram_lottieconverter_docker_src_files_path: "{{ matrix_base_data_path }}/lotticonverter/docker-src" -matrix_telegram_lottieconverter_docker_image: "dock.mau.dev/tulir/lottieconverter:alpine-3.16" # needs to be adjusted according to the FROM clause of Dockerfile of mautrix-telegram +matrix_telegram_lottieconverter_docker_image: "{{ matrix_telegram_lottieconverter_docker_image_name_prefix }}tulir/lottieconverter:alpine-3.16" # needs to be adjusted according to the FROM clause of Dockerfile of mautrix-telegram +matrix_telegram_lottieconverter_docker_image_name_prefix: "{{ 'localhost/' if matrix_telegram_lottieconverter_container_image_self_build else 'dock.mau.dev/' }}" matrix_mautrix_telegram_container_image_self_build: false matrix_mautrix_telegram_docker_repo: "https://mau.dev/mautrix/telegram.git" @@ -18,7 +19,8 @@ matrix_mautrix_telegram_docker_src_files_path: "{{ matrix_base_data_path }}/maut matrix_mautrix_telegram_version: v0.12.2 # See: https://mau.dev/mautrix/telegram/container_registry -matrix_mautrix_telegram_docker_image: "dock.mau.dev/mautrix/telegram:{{ matrix_mautrix_telegram_version }}" +matrix_mautrix_telegram_docker_image: "{{ matrix_mautrix_telegram_docker_image_name_prefix }}mautrix/telegram:{{ matrix_mautrix_telegram_version }}" +matrix_mautrix_telegram_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_telegram_container_image_self_build else 'dock.mau.dev/' }}" matrix_mautrix_telegram_docker_image_force_pull: "{{ matrix_mautrix_telegram_docker_image.endswith(':latest') }}" matrix_mautrix_telegram_base_path: "{{ matrix_base_data_path }}/mautrix-telegram" diff --git a/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml index 22475596..8f38bb00 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-twitter/defaults/main.yml @@ -78,6 +78,16 @@ matrix_mautrix_twitter_appservice_bot_username: twitterbot # Specifies the default log level for all bridge loggers. matrix_mautrix_twitter_logging_level: WARNING +# Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_twitter_metrics_proxying_enabled`. +matrix_mautrix_twitter_metrics_enabled: false + +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-twitter`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_twitter_metrics_proxying_enabled: false + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 00000000..2efad7cd --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-twitter/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,35 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append mautrix-whatapp-metrics's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-twitter role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- when: matrix_mautrix_twitter_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-twitter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-twitter) + ansible.builtin.set_fact: + matrix_mautrix_twitter_nginx_metrics_configuration_block: | + location /metrics/mautrix-twitter { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-twitter:8000"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + + - name: Register mautrix-twitter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-twitter) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_twitter_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml index f8dbc28b..9eb444a7 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-twitter/tasks/main.yml @@ -1,5 +1,14 @@ --- +- block: + - when: matrix_mautrix_twitter_enabled | bool and matrix_mautrix_twitter_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + - block: - when: matrix_mautrix_twitter_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" diff --git a/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 index da823d1e..6e12a755 100644 --- a/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 @@ -65,7 +65,7 @@ appservice: # Prometheus telemetry config. Requires prometheus-client to be installed. metrics: - enabled: false + enabled: {{ matrix_mautrix_twitter_metrics_enabled | to_json }} listen_port: 8000 # Bridge config diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml index 1307d09e..dd628162 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -8,7 +8,7 @@ matrix_mautrix_whatsapp_container_image_self_build: false matrix_mautrix_whatsapp_container_image_self_build_repo: "https://mau.dev/mautrix/whatsapp.git" matrix_mautrix_whatsapp_container_image_self_build_branch: "{{ 'master' if matrix_mautrix_whatsapp_version == 'latest' else matrix_mautrix_whatsapp_version }}" -matrix_mautrix_whatsapp_version: v0.8.0 +matrix_mautrix_whatsapp_version: v0.8.1 # See: https://mau.dev/mautrix/whatsapp/container_registry matrix_mautrix_whatsapp_docker_image: "{{ matrix_mautrix_whatsapp_docker_image_name_prefix }}mautrix/whatsapp:{{ matrix_mautrix_whatsapp_version }}" matrix_mautrix_whatsapp_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_whatsapp_container_image_self_build else 'dock.mau.dev/' }}" @@ -47,6 +47,16 @@ matrix_mautrix_whatsapp_logging_level: 'warn' # If false, created portal rooms will never be federated. matrix_mautrix_whatsapp_federate_rooms: true +# Whether or not metrics endpoint should be enabled. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_mautrix_whatsapp_metrics_proxying_enabled`. +matrix_mautrix_whatsapp_metrics_enabled: false + +# Controls whether metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/mautrix-whatsapp`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_mautrix_whatsapp_metrics_proxying_enabled: false + # Database-related configuration fields. # # To use SQLite, stick to these defaults. @@ -102,6 +112,12 @@ matrix_mautrix_whatsapp_bridge_permissions: | | combine({matrix_admin: 'admin'} if matrix_admin else {}) }} +# Enable bridge relay functionality +matrix_mautrix_whatsapp_bridge_relay_enabled: false + +# Only allow admins on this home server to set themselves as a relay user +matrix_mautrix_whatsapp_bridge_relay_admin_only: true + # Default mautrix-whatsapp configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 00000000..03113dcc --- /dev/null +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,35 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append mautrix-whatapp-metrics's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-bridge-mautrix-whatsapp role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- when: matrix_mautrix_whatsapp_metrics_proxying_enabled | bool + block: + - name: Generate mautrix-whatsapp metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-whatsapp) + ansible.builtin.set_fact: + matrix_mautrix_whatsapp_nginx_metrics_configuration_block: | + location /metrics/mautrix-whatsapp { + {% if matrix_nginx_proxy_enabled | default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-mautrix-whatsapp:8001"; + proxy_pass http://$backend/metrics; + {% else %} + return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; + {% endif %} + } + + - name: Register mautrix-whatsapp metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/mautrix-whatsapp) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_mautrix_whatsapp_nginx_metrics_configuration_block] + }} diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml index fdb65431..b49a1300 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/tasks/main.yml @@ -1,5 +1,14 @@ --- +- block: + - when: matrix_mautrix_whatsapp_enabled | bool and matrix_mautrix_whatsapp_metrics_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + - block: - when: matrix_mautrix_whatsapp_enabled | bool ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" diff --git a/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 b/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 index 4b5af2e9..ad381f6a 100644 --- a/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 +++ b/roles/custom/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 @@ -67,9 +67,9 @@ segment_key: null # Prometheus config. metrics: # Enable prometheus metrics? - enabled: false + enabled: {{ matrix_mautrix_whatsapp_metrics_enabled | to_json }} # IP and port where the metrics listener should be. The path is always /metrics - listen: 127.0.0.1:8001 + listen: 0.0.0.0:8001 # Config for things that are directly sent to WhatsApp. whatsapp: @@ -396,9 +396,9 @@ bridge: relay: # Whether relay mode should be allowed. If allowed, `!wa set-relay` can be used to turn any # authenticated user into a relaybot for that chat. - enabled: false + enabled: {{ matrix_mautrix_whatsapp_bridge_relay_enabled | to_json }} # Should only admins be allowed to set themselves as relay users? - admin_only: true + admin_only: {{ matrix_mautrix_whatsapp_bridge_relay_admin_only | to_json }} # The formats to use when sending messages to WhatsApp via the relaybot. message_formats: m.text: "{{ '{{ .Sender.Displayname }}' }}: {{ '{{ .Message }}' }}" diff --git a/roles/custom/matrix-cactus-comments/defaults/main.yml b/roles/custom/matrix-cactus-comments/defaults/main.yml index 224a5348..80f8c15e 100644 --- a/roles/custom/matrix-cactus-comments/defaults/main.yml +++ b/roles/custom/matrix-cactus-comments/defaults/main.yml @@ -31,10 +31,12 @@ matrix_cactus_comments_version: 0.9.0 matrix_cactus_comments_docker_image: "{{ matrix_container_global_registry_prefix }}cactuscomments/cactus-appservice:{{ matrix_cactus_comments_version }}" matrix_cactus_comments_docker_image_force_pull: "{{ matrix_cactus_comments_docker_image.endswith(':latest') }}" -# matrix_cactus_comments_client_version specifies the version of the cactus-client release to use. +# The released & compiled client is uploaded to a URL that changes per commit. Therefore it is necessary to check the +# available versions and adjust the url manually. # For available versions, see: https://gitlab.com/cactus-comments/cactus-client/-/releases # Also see: `matrix_cactus_comments_client_local_dir` -matrix_cactus_comments_client_version: "0.13.0" +matrix_cactus_comments_webclient_js_url: "https://gateway.pinata.cloud/ipfs/QmSiWN27KZZ1XE32jKwifBnS3nWTUcFGNArKzur2nmDgoL/v0.13.0/cactus.js" +matrix_cactus_comments_webclient_css_url: "https://gateway.pinata.cloud/ipfs/QmSiWN27KZZ1XE32jKwifBnS3nWTUcFGNArKzur2nmDgoL/v0.13.0/style.css" # matrix_cactus_comments_client_local_dir specifies a local directory (on the Ansible controller, not on the remote server) with cactus-client files to use. # This is an alternative to `matrix_cactus_comments_client_version`, to be used when you'd like to diff --git a/roles/custom/matrix-cactus-comments/tasks/setup_install.yml b/roles/custom/matrix-cactus-comments/tasks/setup_install.yml index 8a8bf326..03b97d0a 100644 --- a/roles/custom/matrix-cactus-comments/tasks/setup_install.yml +++ b/roles/custom/matrix-cactus-comments/tasks/setup_install.yml @@ -67,37 +67,16 @@ - when: matrix_cactus_comments_client_local_dir | length == 0 block: - - name: Download client binary to local folder + - name: Download web client js ansible.builtin.get_url: - url: "https://gitlab.com/cactus-comments/cactus-client/-/archive/v{{ matrix_cactus_comments_client_version }}/cactus-client-v{{ matrix_cactus_comments_client_version }}.tar.gz" - dest: "/tmp/cactus-comments-{{ matrix_cactus_comments_client_version }}.tar.gz" - mode: '0644' - register: _download_client - until: _download_client is succeeded - retries: 5 - delay: 2 - check_mode: false - - - name: Unpack client - ansible.builtin.unarchive: - src: "/tmp/cactus-comments-{{ matrix_cactus_comments_client_version }}.tar.gz" - dest: "/tmp/" - remote_src: true - mode: 0600 - check_mode: false - - - name: Propagate client javascript file - ansible.builtin.copy: - src: "/tmp/cactus-client-v{{ matrix_cactus_comments_client_version }}/src/cactus.js" - remote_src: true + url: "{{ matrix_cactus_comments_webclient_js_url }}" dest: "{{ matrix_cactus_comments_client_path }}/cactus.js" mode: "{{ matrix_cactus_comments_client_file_permissions }}" owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" - - name: Propagate client style file - ansible.builtin.copy: - src: "/tmp/cactus-client-v{{ matrix_cactus_comments_client_version }}/src/style.css" - remote_src: true + - name: Download web client css + ansible.builtin.get_url: + url: "{{ matrix_cactus_comments_webclient_css_url }}" dest: "{{ matrix_cactus_comments_client_path }}/style.css" mode: "{{ matrix_cactus_comments_client_file_permissions }}" owner: "{{ matrix_user_username }}" diff --git a/roles/custom/matrix-client-cinny/defaults/main.yml b/roles/custom/matrix-client-cinny/defaults/main.yml index de974eb1..5b15b42c 100644 --- a/roles/custom/matrix-client-cinny/defaults/main.yml +++ b/roles/custom/matrix-client-cinny/defaults/main.yml @@ -6,7 +6,7 @@ matrix_client_cinny_enabled: true matrix_client_cinny_container_image_self_build: false matrix_client_cinny_container_image_self_build_repo: "https://github.com/ajbura/cinny.git" -matrix_client_cinny_version: v2.2.2 +matrix_client_cinny_version: v2.2.4 matrix_client_cinny_docker_image: "{{ matrix_client_cinny_docker_image_name_prefix }}ajbura/cinny:{{ matrix_client_cinny_version }}" matrix_client_cinny_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_cinny_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_cinny_docker_image_force_pull: "{{ matrix_client_cinny_docker_image.endswith(':latest') }}" diff --git a/roles/custom/matrix-client-element/defaults/main.yml b/roles/custom/matrix-client-element/defaults/main.yml index d359108c..85d8d6cf 100644 --- a/roles/custom/matrix-client-element/defaults/main.yml +++ b/roles/custom/matrix-client-element/defaults/main.yml @@ -10,7 +10,7 @@ matrix_client_element_container_image_self_build_repo: "https://github.com/vecto # - https://github.com/vector-im/element-web/issues/19544 matrix_client_element_container_image_self_build_low_memory_system_patch_enabled: "{{ ansible_memtotal_mb < 4096 }}" -matrix_client_element_version: v1.11.17 +matrix_client_element_version: v1.11.22 matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:{{ matrix_client_element_version }}" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" diff --git a/roles/custom/matrix-client-element/tasks/prepare_themes.yml b/roles/custom/matrix-client-element/tasks/prepare_themes.yml index 3f5c9783..70646c94 100644 --- a/roles/custom/matrix-client-element/tasks/prepare_themes.yml +++ b/roles/custom/matrix-client-element/tasks/prepare_themes.yml @@ -9,29 +9,29 @@ delegate_to: 127.0.0.1 become: false block: - - name: Ensure Element themes repository is pulled - ansible.builtin.git: - repo: "{{ matrix_client_element_themes_repository_url }}" - version: "{{ matrix_client_element_themes_repository_version }}" - dest: "{{ role_path }}/files/scratchpad/themes" + - name: Ensure Element themes repository is pulled + ansible.builtin.git: + repo: "{{ matrix_client_element_themes_repository_url }}" + version: "{{ matrix_client_element_themes_repository_version }}" + dest: "{{ role_path }}/files/scratchpad/themes" - - name: Find all Element theme files - ansible.builtin.find: - paths: "{{ role_path }}/files/scratchpad/themes" - patterns: "*.json" - recurse: true - register: matrix_client_element_theme_file_list + - name: Find all Element theme files + ansible.builtin.find: + paths: "{{ role_path }}/files/scratchpad/themes" + patterns: "*.json" + recurse: true + register: matrix_client_element_theme_file_list - - name: Read Element theme - ansible.builtin.slurp: - path: "{{ item.path }}" - register: "matrix_client_element_theme_file_contents" - with_items: "{{ matrix_client_element_theme_file_list.files }}" + - name: Read Element theme + ansible.builtin.slurp: + path: "{{ item.path }}" + register: "matrix_client_element_theme_file_contents" + with_items: "{{ matrix_client_element_theme_file_list.files }}" - - name: Load Element theme - ansible.builtin.set_fact: - matrix_client_element_settingDefaults_custom_themes: "{{ matrix_client_element_settingDefaults_custom_themes + [item['content'] | b64decode | from_json] }}" # noqa var-naming - with_items: "{{ matrix_client_element_theme_file_contents.results }}" + - name: Load Element theme + ansible.builtin.set_fact: + matrix_client_element_settingDefaults_custom_themes: "{{ matrix_client_element_settingDefaults_custom_themes + [item['content'] | b64decode | from_json] }}" # noqa var-naming + with_items: "{{ matrix_client_element_theme_file_contents.results }}" # # Tasks related to getting rid of Element themes (if it was previously enabled) diff --git a/roles/custom/matrix-coturn/defaults/main.yml b/roles/custom/matrix-coturn/defaults/main.yml index 3a125de4..78efc73c 100644 --- a/roles/custom/matrix-coturn/defaults/main.yml +++ b/roles/custom/matrix-coturn/defaults/main.yml @@ -8,7 +8,7 @@ matrix_coturn_container_image_self_build_repo: "https://github.com/coturn/coturn matrix_coturn_container_image_self_build_repo_version: "docker/{{ matrix_coturn_version }}" matrix_coturn_container_image_self_build_repo_dockerfile_path: "docker/coturn/alpine/Dockerfile" -matrix_coturn_version: 4.6.1-r0 +matrix_coturn_version: 4.6.1-r1 matrix_coturn_docker_image: "{{ matrix_coturn_docker_image_name_prefix }}coturn/coturn:{{ matrix_coturn_version }}-alpine" matrix_coturn_docker_image_name_prefix: "{{ 'localhost/' if matrix_coturn_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_coturn_docker_image_force_pull: "{{ matrix_coturn_docker_image.endswith(':latest') }}" @@ -20,6 +20,13 @@ matrix_coturn_docker_image_force_pull: "{{ matrix_coturn_docker_image.endswith(' # # Setting up deny/allow rules with `matrix_coturn_allowed_peer_ips`/`matrix_coturn_denied_peer_ips` is also # possible for achieving such isolation, but is more complicated due to the dynamic nature of Docker networking. +# +# Setting `matrix_coturn_docker_network` to 'host' will run the container with host networking, +# which will drastically improve performance when thousands of ports are opened due to Docker not having to set up forwarding rules for each port. +# Running with host networking can be dangerous, as it potentially exposes your local network and its services to Coturn peers. +# Regardless of the networking mode, we apply a deny list which via `matrix_coturn_denied_peer_ips`, +# which hopefully prevents access to such private network ranges. +# When running in host-networking mode, you need to adjust the firewall yourself, so that ports are opened. matrix_coturn_docker_network: "matrix-coturn" matrix_coturn_base_path: "{{ matrix_base_data_path }}/coturn" @@ -41,20 +48,20 @@ matrix_coturn_container_extra_arguments: [] # Controls whether the Coturn container exposes its plain STUN port (tcp/3478 and udp/3478 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:3478"), or empty string to not expose. -matrix_coturn_container_stun_plain_host_bind_port: '3478' +matrix_coturn_container_stun_plain_host_bind_port: "{{ '3478' if matrix_coturn_docker_network != 'host' else '' }}" # Controls whether the Coturn container exposes its TLS STUN port (tcp/5349 and udp/5349 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:5349"), or empty string to not expose. -matrix_coturn_container_stun_tls_host_bind_port: '5349' +matrix_coturn_container_stun_tls_host_bind_port: "{{ '5349' if matrix_coturn_docker_network != 'host' else '' }}" # Controls whether the Coturn container exposes its TURN UDP port range and which interface to do it on. # # Takes an interface "" (e.g. "127.0.0.1"), or empty string to listen on all interfaces. -# Takes a null/none value (`~`) to prevent listening. +# Takes a null/none value (`~`) or 'none' (as a string) to prevent listening. # # The UDP port-range itself is specified using `matrix_coturn_turn_udp_min_port` and `matrix_coturn_turn_udp_max_port`. -matrix_coturn_container_turn_range_listen_interface: '' +matrix_coturn_container_turn_range_listen_interface: "{{ '' if matrix_coturn_docker_network != 'host' else 'none' }}" # UDP port-range to use for TURN matrix_coturn_turn_udp_min_port: 49152 @@ -69,10 +76,54 @@ matrix_coturn_turn_external_ip_address: '' matrix_coturn_turn_external_ip_addresses: ["{{ matrix_coturn_turn_external_ip_address }}"] matrix_coturn_allowed_peer_ips: [] -matrix_coturn_denied_peer_ips: [] + +# We block loopback interfaces and private networks by default to prevent private resources from being accessible. +# This is especially important when Coturn does not run within a container network (e.g. `matrix_coturn_docker_network: host`). +# +# Learn more: https://www.rtcsec.com/article/cve-2020-26262-bypass-of-coturns-access-control-protection/ +# +# If you're running Coturn for local network peers, you may wish to override these rules. +matrix_coturn_denied_peer_ips: + - 0.0.0.0-0.255.255.255 + - 10.0.0.0-10.255.255.255 + - 100.64.0.0-100.127.255.255 + - 127.0.0.0-127.255.255.255 + - 169.254.0.0-169.254.255.255 + - 172.16.0.0-172.31.255.255 + - 192.0.0.0-192.0.0.255 + - 192.0.2.0-192.0.2.255 + - 192.88.99.0-192.88.99.255 + - 192.168.0.0-192.168.255.255 + - 198.18.0.0-198.19.255.255 + - 198.51.100.0-198.51.100.255 + - 203.0.113.0-203.0.113.255 + - 240.0.0.0-255.255.255.255 + - ::1 + - 64:ff9b::-64:ff9b::ffff:ffff + - ::ffff:0.0.0.0-::ffff:255.255.255.255 + - 100::-100::ffff:ffff:ffff:ffff + - 2001::-2001:1ff:ffff:ffff:ffff:ffff:ffff:ffff + - 2002::-2002:ffff:ffff:ffff:ffff:ffff:ffff:ffff + - fc00::-fdff:ffff:ffff:ffff:ffff:ffff:ffff:ffff + - fe80::-febf:ffff:ffff:ffff:ffff:ffff:ffff:ffff + matrix_coturn_user_quota: null matrix_coturn_total_quota: null +# Controls whether `no-tcp-relay` is added to the configuration +matrix_coturn_no_tcp_relay_enabled: true + +# Controls whether `no-multicast-peers` is added to the configuration +matrix_coturn_no_multicast_peers_enabled: true + +# Additional configuration to be passed to turnserver.conf +# Example: +# matrix_coturn_additional_configuration: | +# simple-log +# aux-server=1.2.3.4 +# relay-ip=4.3.2.1 +matrix_coturn_additional_configuration: '' + # To enable TLS, you need to provide paths to certificates. # Paths defined in `matrix_coturn_tls_cert_path` and `matrix_coturn_tls_key_path` are in-container paths. # Files on the host can be mounted into the container using `matrix_coturn_container_additional_volumes`. diff --git a/roles/custom/matrix-coturn/tasks/setup_install.yml b/roles/custom/matrix-coturn/tasks/setup_install.yml index ab94db27..44325f46 100644 --- a/roles/custom/matrix-coturn/tasks/setup_install.yml +++ b/roles/custom/matrix-coturn/tasks/setup_install.yml @@ -62,7 +62,8 @@ owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" -- name: Ensure Coturn network is created in Docker +- when: matrix_coturn_docker_network not in ['', 'host'] + name: Ensure Coturn network is created in Docker community.docker.docker_network: name: "{{ matrix_coturn_docker_network }}" driver: bridge diff --git a/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 b/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 index 3a0782bf..8c0272cf 100644 --- a/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 +++ b/roles/custom/matrix-coturn/templates/systemd/matrix-coturn.service.j2 @@ -30,7 +30,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name -p {{ matrix_coturn_container_stun_tls_host_bind_port }}:5349 \ -p {{ matrix_coturn_container_stun_tls_host_bind_port }}:5349/udp \ {% endif %} - {% if matrix_coturn_container_turn_range_listen_interface is not none %} + {% if matrix_coturn_container_turn_range_listen_interface is not in [none, 'none'] %} -p {{ matrix_coturn_container_turn_range_listen_interface }}{{ ':' if matrix_coturn_container_turn_range_listen_interface else '' }}{{ matrix_coturn_turn_udp_min_port }}-{{ matrix_coturn_turn_udp_max_port }}:{{ matrix_coturn_turn_udp_min_port }}-{{ matrix_coturn_turn_udp_max_port }}/udp \ {% endif %} --mount type=bind,src={{ matrix_coturn_config_path }},dst=/turnserver.conf,ro \ diff --git a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 index 1bdf310a..dfc3a87d 100644 --- a/roles/custom/matrix-coturn/templates/turnserver.conf.j2 +++ b/roles/custom/matrix-coturn/templates/turnserver.conf.j2 @@ -30,7 +30,10 @@ no-dtls {% endif %} prod + +{% if matrix_coturn_no_tcp_relay_enabled %} no-tcp-relay +{% endif %} {% if matrix_coturn_user_quota != None %} user-quota={{ matrix_coturn_user_quota }} @@ -39,9 +42,15 @@ user-quota={{ matrix_coturn_user_quota }} total-quota={{ matrix_coturn_total_quota }} {% endif %} +{% if matrix_coturn_no_multicast_peers_enabled %} +no-multicast-peers +{% endif %} + {% for ip_range in matrix_coturn_denied_peer_ips %} denied-peer-ip={{ ip_range }} {% endfor %} {% for ip_range in matrix_coturn_allowed_peer_ips %} allowed-peer-ip={{ ip_range }} {% endfor %} + +{{ matrix_coturn_additional_configuration }} diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 32f262ad..6a495f8f 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -6,7 +6,7 @@ matrix_dendrite_enabled: true matrix_dendrite_docker_image: "{{ matrix_dendrite_docker_image_name_prefix }}matrixdotorg/dendrite-monolith:{{ matrix_dendrite_docker_image_tag }}" matrix_dendrite_docker_image_name_prefix: "docker.io/" -matrix_dendrite_docker_image_tag: "v0.10.8" +matrix_dendrite_docker_image_tag: "v0.11.0" matrix_dendrite_docker_image_force_pull: "{{ matrix_dendrite_docker_image.endswith(':latest') }}" matrix_dendrite_base_path: "{{ matrix_base_data_path }}/dendrite" diff --git a/roles/custom/matrix-grafana/defaults/main.yml b/roles/custom/matrix-grafana/defaults/main.yml index 44821b57..5b730480 100644 --- a/roles/custom/matrix-grafana/defaults/main.yml +++ b/roles/custom/matrix-grafana/defaults/main.yml @@ -5,7 +5,7 @@ matrix_grafana_enabled: true -matrix_grafana_version: 9.3.2 +matrix_grafana_version: 9.3.6 matrix_grafana_docker_image: "{{ matrix_container_global_registry_prefix }}grafana/grafana:{{ matrix_grafana_version }}" matrix_grafana_docker_image_force_pull: "{{ matrix_grafana_docker_image.endswith(':latest') }}" diff --git a/roles/custom/matrix-jitsi/defaults/main.yml b/roles/custom/matrix-jitsi/defaults/main.yml index 66d06e3b..cc24d356 100644 --- a/roles/custom/matrix-jitsi/defaults/main.yml +++ b/roles/custom/matrix-jitsi/defaults/main.yml @@ -91,7 +91,7 @@ matrix_jitsi_jibri_recorder_password: '' matrix_jitsi_enable_lobby: false -matrix_jitsi_version: stable-8138-1 +matrix_jitsi_version: stable-8252 matrix_jitsi_container_image_tag: "{{ matrix_jitsi_version }}" # for backward-compatibility matrix_jitsi_web_docker_image: "{{ matrix_container_global_registry_prefix }}jitsi/web:{{ matrix_jitsi_container_image_tag }}" @@ -313,3 +313,8 @@ matrix_jitsi_jvb_container_colibri_ws_host_bind_port: '' # # The setting requires an integer to be set for usage and allows a user to specify the max number of particpants on a conference. matrix_prosody_jitsi_max_participants: '' + +# Allows abailty to set XMPP AUTH user if using JIGASI +matrix_jitsi_jigasi_xmpp_user: '' +# Allows abailty to set XMPP AUTH pasword if using JIGASI +matrix_jitsi_jigasi_xmpp_password: '' diff --git a/roles/custom/matrix-jitsi/templates/prosody/env.j2 b/roles/custom/matrix-jitsi/templates/prosody/env.j2 index 541727b8..e4582f6b 100644 --- a/roles/custom/matrix-jitsi/templates/prosody/env.j2 +++ b/roles/custom/matrix-jitsi/templates/prosody/env.j2 @@ -15,8 +15,8 @@ JIBRI_XMPP_PASSWORD={{ matrix_jitsi_jibri_xmpp_password }} JICOFO_AUTH_USER={{ matrix_jitsi_jicofo_auth_user }} JICOFO_AUTH_PASSWORD={{ matrix_jitsi_jicofo_auth_password }} JICOFO_COMPONENT_SECRET -JIGASI_XMPP_USER= -JIGASI_XMPP_PASSWORD= +JIGASI_XMPP_USER={{ matrix_jitsi_jigasi_xmpp_user }} +JIGASI_XMPP_PASSWORD={{ matrix_jitsi_jigasi_xmpp_password }} JVB_AUTH_USER={{ matrix_jitsi_jvb_auth_user }} JVB_AUTH_PASSWORD={{ matrix_jitsi_jvb_auth_password }} JWT_APP_ID diff --git a/roles/custom/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 b/roles/custom/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 index 19bd5720..55505ab6 100644 --- a/roles/custom/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 +++ b/roles/custom/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 @@ -18,7 +18,9 @@ ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_s # ma1sd writes an SQLite shared library (libsqlitejdbc.so) to /tmp and executes it from there, # so /tmp needs to be mounted with an exec option. -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-ma1sd \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-ma1sd \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ @@ -39,9 +41,11 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_ma1sd_docker_image }} {% for network in matrix_ma1sd_container_additional_networks %} -ExecStartPost={{ devture_systemd_docker_base_host_command_sh }} -c 'container_name=matrix-ma1sd; network_name={{ network }}; attempt=0; while [ $attempt -le 29 ]; do attempt=$(( $attempt + 1 )); if [ "`{{ devture_systemd_docker_base_host_command_docker }} inspect -f {{ '{{.State.Running}}' }} $container_name 2> /dev/null`" = "true" ]; then break; fi; sleep 1; done; {{ devture_systemd_docker_base_host_command_docker }} network connect $network_name $container_name' +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-ma1sd {% endfor %} +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-ma1sd + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-ma1sd 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-ma1sd 2>/dev/null || true' Restart=always diff --git a/roles/custom/matrix-nginx-proxy/defaults/main.yml b/roles/custom/matrix-nginx-proxy/defaults/main.yml index 5d5329b0..46997b3c 100644 --- a/roles/custom/matrix-nginx-proxy/defaults/main.yml +++ b/roles/custom/matrix-nginx-proxy/defaults/main.yml @@ -21,6 +21,10 @@ matrix_nginx_proxy_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-nginx-proxy.service wants matrix_nginx_proxy_systemd_wanted_services_list: [] +# The base container network. +# Also see: matrix_nginx_proxy_container_additional_networks +matrix_nginx_proxy_container_network: "{{ matrix_docker_network }}" + # A list of additional container networks that matrix-nginx-proxy would be connected to. # The playbook does not create these networks, so make sure they already exist. # diff --git a/roles/custom/matrix-nginx-proxy/templates/bin/lets-encrypt-certificates-renew.j2 b/roles/custom/matrix-nginx-proxy/templates/bin/lets-encrypt-certificates-renew.j2 index 89113629..5f235ea2 100644 --- a/roles/custom/matrix-nginx-proxy/templates/bin/lets-encrypt-certificates-renew.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/bin/lets-encrypt-certificates-renew.j2 @@ -12,7 +12,7 @@ docker run \ --name=matrix-certbot \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ - --network="{{ matrix_docker_network }}" \ + --network="{{ matrix_nginx_proxy_container_network }}" \ -p 127.0.0.1:{{ matrix_ssl_lets_encrypt_certbot_standalone_http_port }}:8080 \ --mount type=bind,src={{ matrix_ssl_config_dir_path }},dst=/etc/letsencrypt \ --mount type=bind,src={{ matrix_ssl_log_dir_path }},dst=/var/log/letsencrypt \ diff --git a/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 b/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 index 0a7e9052..301eb9d5 100755 --- a/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 @@ -16,13 +16,15 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-nginx-proxy 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-nginx-proxy 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-nginx-proxy \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-nginx-proxy \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ --read-only \ --tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_nginx_proxy_tmp_directory_size_mb }}m \ - --network={{ matrix_docker_network }} \ + --network={{ matrix_nginx_proxy_container_network }} \ {% if matrix_nginx_proxy_container_http_host_bind_port %} -p {{ matrix_nginx_proxy_container_http_host_bind_port }}:8080 \ {% endif %} @@ -48,9 +50,11 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_nginx_proxy_docker_image }} {% for network in matrix_nginx_proxy_container_additional_networks %} -ExecStartPost={{ devture_systemd_docker_base_host_command_sh }} -c 'container_name=matrix-nginx-proxy; network_name={{ network }}; attempt=0; while [ $attempt -le 29 ]; do attempt=$(( $attempt + 1 )); if [ "`{{ devture_systemd_docker_base_host_command_docker }} inspect -f {{ '{{.State.Running}}' }} $container_name 2> /dev/null`" = "true" ]; then break; fi; sleep 1; done; {{ devture_systemd_docker_base_host_command_docker }} network connect $network_name $container_name' +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-nginx-proxy {% endfor %} +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-nginx-proxy + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-nginx-proxy 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-nginx-proxy 2>/dev/null || true' ExecReload={{ devture_systemd_docker_base_host_command_docker }} exec matrix-nginx-proxy /usr/sbin/nginx -s reload diff --git a/roles/custom/matrix-prometheus-node-exporter/defaults/main.yml b/roles/custom/matrix-prometheus-node-exporter/defaults/main.yml deleted file mode 100644 index 070e8616..00000000 --- a/roles/custom/matrix-prometheus-node-exporter/defaults/main.yml +++ /dev/null @@ -1,67 +0,0 @@ ---- -# matrix-prometheus-node-exporter is an Prometheus exporter for machine metrics -# See: https://prometheus.io/docs/guides/node-exporter/ -# Project source code URL: https://github.com/prometheus/node_exporter - -matrix_prometheus_node_exporter_enabled: false - -matrix_prometheus_node_exporter_version: v1.5.0 -matrix_prometheus_node_exporter_docker_image: "{{ matrix_container_global_registry_prefix }}prom/node-exporter:{{ matrix_prometheus_node_exporter_version }}" -matrix_prometheus_node_exporter_docker_image_force_pull: "{{ matrix_prometheus_node_exporter_docker_image.endswith(':latest') }}" - -# A list of extra arguments to pass to the container -matrix_prometheus_node_exporter_container_extra_arguments: [] - -# A list of extra arguments to pass to the node_exporter process -# -# Example: -# matrix_prometheus_node_exporter_process_extra_arguments: -# - "--collector.systemd" -# - "--collector.logind" -# -# Note: the above is just an example. Various collectors may require various tweaks to be able to run. -# Running the systemd collector requires the following `matrix_prometheus_node_exporter_container_extra_arguments`: -# - the socket to be mounted as well (`--mount type=bind,src=/var/run/dbus/system_bus_socket,dst=/var/run/dbus/system_bus_socket,ro,bind-propagation=rslave`) -# - (on AppArmor-based distros) disabling AppArmor protection (`--security-opt apparmor=unconfined`) -matrix_prometheus_node_exporter_process_extra_arguments: [] - -# List of systemd services that matrix-prometheus.service depends on -matrix_prometheus_node_exporter_systemd_required_services_list: ['docker.service'] - -# List of systemd services that matrix-prometheus.service wants -matrix_prometheus_node_exporter_systemd_wanted_services_list: [] - -# Controls whether node-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/node-exporter`. -# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. -# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. -matrix_prometheus_node_exporter_metrics_proxying_enabled: false - -# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9100 in the container). -# -# Takes an ":" value (e.g. "127.0.0.1:9100"), just a port number or empty string to not expose. -# -# You likely don't need to do this. See `matrix_prometheus_node_exporter_metrics_proxying_enabled`. -# -# Official recommendations are to run this container with `--net=host`, -# but we don't do that, since it: -# - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008) -# - or listens on a loopback interface only (--net=host and 127.0.0.1:9100), which is not reachable from another container (like `matrix-prometheus`) -# -# Using `--net=host` and binding to Docker's `matrix` bridge network may be a solution to both, -# but that's trickier to accomplish and won't necessarily work (hasn't been tested). -# -# Not using `--net=host` means that our network statistic reports are likely broken (inaccurate), -# because node-exporter can't see all interfaces, etc. -# For now, we'll live with that, until someone develops a better solution. -matrix_prometheus_node_exporter_container_http_host_bind_port: '' - -# If you are supplying your own NGINX proxy but want to use the provided exporters you will have to supply an ":" value for the containers to bind to on your host. -# If matrix_prometheus_node_exporter_container_http_host_bind_port is set to just a port number, this will default to "127.0.0.1:" -# If matrix_prometheus_node_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used -# Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) -matrix_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + matrix_prometheus_node_exporter_container_http_host_bind_port_number_raw if not ':' in matrix_prometheus_node_exporter_container_http_host_bind_port else (matrix_prometheus_node_exporter_container_http_host_bind_port if matrix_prometheus_node_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" - -# matrix_prometheus_node_exporter_dashboard_urls contains a list of URLs with Grafana dashboard definitions. -# If the Grafana role is enabled, these dashboards will be downloaded. -matrix_prometheus_node_exporter_dashboard_urls: - - https://raw.githubusercontent.com/rfrail3/grafana-dashboards/master/prometheus/node-exporter-full.json diff --git a/roles/custom/matrix-prometheus-node-exporter/tasks/main.yml b/roles/custom/matrix-prometheus-node-exporter/tasks/main.yml deleted file mode 100644 index 6a0cd830..00000000 --- a/roles/custom/matrix-prometheus-node-exporter/tasks/main.yml +++ /dev/null @@ -1,26 +0,0 @@ ---- - -- block: - - when: matrix_prometheus_node_exporter_enabled | bool and matrix_prometheus_node_exporter_metrics_proxying_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: - - setup-all - - setup-nginx-proxy - - install-all - - install-nginx-proxy - -- block: - - when: matrix_prometheus_node_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-prometheus-node-exporter - - install-all - - install-prometheus-node-exporter - -- block: - - when: not matrix_prometheus_node_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-prometheus-node-exporter diff --git a/roles/custom/matrix-prometheus-node-exporter/tasks/setup_install.yml b/roles/custom/matrix-prometheus-node-exporter/tasks/setup_install.yml deleted file mode 100644 index 3c0af96b..00000000 --- a/roles/custom/matrix-prometheus-node-exporter/tasks/setup_install.yml +++ /dev/null @@ -1,19 +0,0 @@ ---- - -- name: Ensure matrix-prometheus-node-exporter image is pulled - community.docker.docker_image: - name: "{{ matrix_prometheus_node_exporter_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_prometheus_node_exporter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_node_exporter_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure matrix-prometheus-node-exporter.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-prometheus-node-exporter.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-node-exporter.service" - mode: 0644 - register: matrix_prometheus_node_exporter_systemd_service_result diff --git a/roles/custom/matrix-prometheus-node-exporter/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus-node-exporter/tasks/setup_uninstall.yml deleted file mode 100644 index b5ef4f0f..00000000 --- a/roles/custom/matrix-prometheus-node-exporter/tasks/setup_uninstall.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- name: Check existence of matrix-prometheus-node-exporter service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-node-exporter.service" - register: matrix_prometheus_node_exporter_service_stat - -- when: matrix_prometheus_node_exporter_service_stat.stat.exists | bool - block: - - name: Ensure matrix-prometheus-node-exporter is stopped - ansible.builtin.service: - name: matrix-prometheus-node-exporter - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-prometheus-node-exporter.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-node-exporter.service" - state: absent diff --git a/roles/custom/matrix-prometheus-node-exporter/vars/main.yml b/roles/custom/matrix-prometheus-node-exporter/vars/main.yml deleted file mode 100644 index 952dc205..00000000 --- a/roles/custom/matrix-prometheus-node-exporter/vars/main.yml +++ /dev/null @@ -1,5 +0,0 @@ ---- - -# `matrix_prometheus_node_exporter_container_http_host_bind_port_number_raw` contains the raw port number extracted from `matrix_prometheus_node_exporter_container_http_host_bind_port`, -# which can contain values like this: ('1234', '127.0.0.1:1234', '0.0.0.0:1234') -matrix_prometheus_node_exporter_container_http_host_bind_port_number_raw: "{{ '' if matrix_prometheus_node_exporter_container_http_host_bind_port == '' else (matrix_prometheus_node_exporter_container_http_host_bind_port.split(':')[1] if ':' in matrix_prometheus_node_exporter_container_http_host_bind_port else matrix_prometheus_node_exporter_container_http_host_bind_port) }}" diff --git a/roles/custom/matrix-prometheus-postgres-exporter/defaults/main.yml b/roles/custom/matrix-prometheus-postgres-exporter/defaults/main.yml deleted file mode 100644 index 5fc8e919..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/defaults/main.yml +++ /dev/null @@ -1,64 +0,0 @@ ---- -# matrix-prometheus-postgres-exporter is an Prometheus exporter for postgres metrics -# Project source code URL: https://github.com/prometheus-community/postgres_exporter - -matrix_prometheus_postgres_exporter_enabled: false - -matrix_prometheus_postgres_exporter_version: v0.11.1 -matrix_prometheus_postgres_exporter_port: 9187 - -matrix_prometheus_postgres_exporter_docker_image: "quay.io/prometheuscommunity/postgres-exporter:{{ matrix_prometheus_postgres_exporter_version }}" -matrix_prometheus_postgres_exporter_docker_image_force_pull: "{{ matrix_prometheus_postgres_exporter_docker_image.endswith(':latest') }}" - -# A list of extra arguments to pass to the container -matrix_prometheus_postgres_exporter_container_extra_arguments: ["-e PG_EXPORTER_AUTO_DISCOVER_DATABASES=true", - "-e PG_EXPORTER_WEB_LISTEN_ADDRESS=\":{{ matrix_prometheus_postgres_exporter_port }}\"", - "-e DATA_SOURCE_NAME=\"postgresql://{{ matrix_prometheus_postgres_exporter_database_username }}:{{ matrix_prometheus_postgres_exporter_database_password }}@{{ matrix_prometheus_postgres_exporter_database_hostname }}:5432/{{ matrix_prometheus_postgres_exporter_database_name }}?sslmode=disable\""] - -# List of systemd services that matrix-prometheus-postgres-exporter.service depends on -matrix_prometheus_postgres_exporter_systemd_required_services_list: ['docker.service'] - -# List of systemd services that matrix-prometheus-postgres-exporter.service wants -matrix_prometheus_postgres_exporter_systemd_wanted_services_list: [] - -# details for connecting to the database -matrix_prometheus_postgres_exporter_database_username: 'matrix_prometheus_postgres_exporter' -matrix_prometheus_postgres_exporter_database_password: 'some-password' -matrix_prometheus_postgres_exporter_database_hostname: '' -matrix_prometheus_postgres_exporter_database_port: 5432 -matrix_prometheus_postgres_exporter_database_name: 'matrix_prometheus_postgres_exporter' - -# Controls whether postgres-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/postgres-exporter`. -# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. -# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. -matrix_prometheus_postgres_exporter_metrics_proxying_enabled: false - -# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9187 in the container). -# -# Takes an ":" value (e.g. "127.0.0.1:9187"), just a port number or an empty string to not expose. -# -# You likely don't need to do this. See `matrix_prometheus_postgres_exporter_metrics_proxying_enabled`. -# -# Official recommendations are to run this container with `--net=host`, -# but we don't do that, since it: -# - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008) -# - or listens on a loopback interface only (--net=host and 127.0.0.1:9187), which is not reachable from another container (like `matrix-prometheus`) -# -# Using `--net=host` and binding to Docker's `matrix` bridge network may be a solution to both, -# but that's trickier to accomplish and won't necessarily work (hasn't been tested). -# -# Not using `--net=host` means that our network statistic reports are likely broken (inaccurate), -# because node-exporter can't see all interfaces, etc. -# For now, we'll live with that, until someone develops a better solution. -matrix_prometheus_postgres_exporter_container_http_host_bind_port: '' - -# If you are supplying your own NGINX proxy but want to use the provided exporters you will have to supply an ":" value for the containers to bind to on your host. -# If matrix_prometheus_postgres_exporter_container_http_host_bind_port is set to just a port number, this will default to "127.0.0.1:" -# If matrix_prometheus_postgres_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used -# Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) -matrix_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + matrix_prometheus_postgres_exporter_container_http_host_bind_port_number_raw if not ':' in matrix_prometheus_postgres_exporter_container_http_host_bind_port else (matrix_prometheus_postgres_exporter_container_http_host_bind_port if matrix_prometheus_postgres_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" - -# matrix_prometheus_postgres_exporter_dashboard_urls contains a list of URLs with Grafana dashboard definitions. -# If the Grafana role is enabled, these dashboards will be downloaded. -matrix_prometheus_postgres_exporter_dashboard_urls: - - "https://grafana.com/api/dashboards/9628/revisions/7/download" diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/main.yml b/roles/custom/matrix-prometheus-postgres-exporter/tasks/main.yml deleted file mode 100644 index 7ff6d15a..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/main.yml +++ /dev/null @@ -1,29 +0,0 @@ ---- - -- block: - - when: matrix_prometheus_postgres_exporter_enabled | bool and matrix_prometheus_postgres_exporter_metrics_proxying_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: - - setup-all - - setup-nginx-proxy - - install-all - - install-nginx-proxy - -- block: - - when: matrix_prometheus_postgres_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" - - - when: matrix_prometheus_postgres_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-prometheus-postgres-exporter - - install-all - - install-prometheus-postgres-exporter - -- block: - - when: not matrix_prometheus_postgres_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-prometheus-postgres-exporter diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_install.yml b/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_install.yml deleted file mode 100644 index ee0f9bef..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_install.yml +++ /dev/null @@ -1,19 +0,0 @@ ---- - -- name: Ensure matrix-prometheus-postgres-exporter image is pulled - community.docker.docker_image: - name: "{{ matrix_prometheus_postgres_exporter_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_prometheus_postgres_exporter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_postgres_exporter_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure matrix-prometheus-postgres-exporter.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-prometheus-postgres-exporter.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-postgres-exporter.service" - mode: 0644 - register: matrix_prometheus_postgres_exporter_systemd_service_result diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_uninstall.yml deleted file mode 100644 index bdf72042..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/setup_uninstall.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- name: Check existence of matrix-prometheus-postgres-exporter service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-postgres-exporter.service" - register: matrix_prometheus_postgres_exporter_service_stat - -- when: matrix_prometheus_postgres_exporter_service_stat.stat.exists | bool - block: - - name: Ensure matrix-prometheus-postgres-exporter is stopped - ansible.builtin.service: - name: matrix-prometheus-postgres-exporter - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-prometheus-postgres-exporter.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-postgres-exporter.service" - state: absent diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/validate_config.yml b/roles/custom/matrix-prometheus-postgres-exporter/tasks/validate_config.yml deleted file mode 100644 index 6acf784f..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/validate_config.yml +++ /dev/null @@ -1,9 +0,0 @@ ---- - -- name: Fail if required prometheus-postgres-exporter settings not defined - ansible.builtin.fail: - msg: > - You need to define a required configuration setting (`{{ item }}`). - when: "vars[item] == ''" - with_items: - - matrix_prometheus_postgres_exporter_database_hostname diff --git a/roles/custom/matrix-prometheus-postgres-exporter/vars/main.yml b/roles/custom/matrix-prometheus-postgres-exporter/vars/main.yml deleted file mode 100644 index aed3b216..00000000 --- a/roles/custom/matrix-prometheus-postgres-exporter/vars/main.yml +++ /dev/null @@ -1,5 +0,0 @@ ---- - -# `matrix_prometheus_postgres_exporter_container_http_host_bind_port_number_raw` contains the raw port number extracted from `matrix_prometheus_postgres_exporter_container_http_host_bind_port`, -# which can contain values like this: ('1234', '127.0.0.1:1234', '0.0.0.0:1234') -matrix_prometheus_postgres_exporter_container_http_host_bind_port_number_raw: "{{ '' if matrix_prometheus_postgres_exporter_container_http_host_bind_port == '' else (matrix_prometheus_postgres_exporter_container_http_host_bind_port.split(':')[1] if ':' in matrix_prometheus_postgres_exporter_container_http_host_bind_port else matrix_prometheus_postgres_exporter_container_http_host_bind_port) }}" diff --git a/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml b/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml new file mode 100644 index 00000000..30ef2c13 --- /dev/null +++ b/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml @@ -0,0 +1,27 @@ +--- +# matrix-prometheus-services-proxy-connect is a role which helps integrate +# the various Prometheus roles, which may live outside this Matrix playbook, into it. + + +# Controls whether node-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/node-exporter`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled: false + +# If you are supplying your own NGINX proxy but want to use the provided exporters you will have to supply an ":" value for the containers to bind to on your host. +# If prometheus_node_exporter_container_http_host_bind_port is set to just a port number, this will default to "127.0.0.1:" +# If prometheus_node_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used +# Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) +matrix_prometheus_services_proxy_connect_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + prometheus_node_exporter_container_http_host_bind_port_number_raw if not ':' in prometheus_node_exporter_container_http_host_bind_port else (prometheus_node_exporter_container_http_host_bind_port if prometheus_node_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" + + +# Controls whether postgres-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/postgres-exporter`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled: false + +# If you are supplying your own NGINX proxy but want to use the provided exporters you will have to supply an ":" value for the containers to bind to on your host. +# If prometheus_node_exporter_container_http_host_bind_port is set to just a port number, this will default to "127.0.0.1:" +# If prometheus_node_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used +# Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) +matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + prometheus_postgres_exporter_container_http_host_bind_port_number_raw if not ':' in prometheus_postgres_exporter_container_http_host_bind_port else (prometheus_postgres_exporter_container_http_host_bind_port if prometheus_postgres_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" diff --git a/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml new file mode 100644 index 00000000..e5654688 --- /dev/null +++ b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml @@ -0,0 +1,28 @@ +--- + +- name: Fail if matrix-nginx-proxy role executed before matrix-prometheus-services-proxy-connect + ansible.builtin.fail: + msg: >- + Trying to append node-exporter's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-prometheus-services-proxy-connect role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- block: + - when: prometheus_node_exporter_enabled | bool and matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy + +- block: + - when: prometheus_node_exporter_enabled | bool and matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy diff --git a/roles/custom/matrix-prometheus-node-exporter/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml similarity index 51% rename from roles/custom/matrix-prometheus-node-exporter/tasks/inject_into_nginx_proxy.yml rename to roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml index 24e05ed8..4c86e8aa 100644 --- a/roles/custom/matrix-prometheus-node-exporter/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml @@ -1,25 +1,16 @@ --- -- name: Fail if matrix-nginx-proxy role already executed - ansible.builtin.fail: - msg: >- - Trying to append node-exporter's reverse-proxying configuration to matrix-nginx-proxy, - but it's pointless since the matrix-nginx-proxy role had already executed. - To fix this, please change the order of roles in your playbook, - so that the matrix-nginx-proxy role would run after the matrix-prometheus-node-exporter role. - when: matrix_nginx_proxy_role_executed | default(False) | bool - - name: Generate node-exporter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/node-exporter) ansible.builtin.set_fact: - matrix_prometheus_node_exporter_nginx_metrics_configuration_block: | + matrix_prometheus_services_proxy_connect_node_exporter_nginx_metrics_configuration_block: | location /metrics/node-exporter { {% if matrix_nginx_proxy_enabled | default(False) %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; - set $backend "matrix-prometheus-node-exporter:9100"; + set $backend "{{ prometheus_node_exporter_identifier }}:9100"; proxy_pass http://$backend/metrics; - {% elif matrix_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host != '' %} - proxy_pass http://{{ matrix_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host }}/metrics; + {% elif matrix_prometheus_services_proxy_connect_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host != '' %} + proxy_pass http://{{ matrix_prometheus_services_proxy_connect_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host }}/metrics; {% else %} return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; {% endif %} @@ -31,5 +22,5 @@ {{ matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + - [matrix_prometheus_node_exporter_nginx_metrics_configuration_block] + [matrix_prometheus_services_proxy_connect_node_exporter_nginx_metrics_configuration_block] }} diff --git a/roles/custom/matrix-prometheus-postgres-exporter/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml similarity index 51% rename from roles/custom/matrix-prometheus-postgres-exporter/tasks/inject_into_nginx_proxy.yml rename to roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml index 121efc17..34796081 100644 --- a/roles/custom/matrix-prometheus-postgres-exporter/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-postgres-exporter/inject_into_nginx_proxy.yml @@ -1,25 +1,16 @@ --- -- name: Fail if matrix-nginx-proxy role already executed - ansible.builtin.fail: - msg: >- - Trying to append postgres-exporter's reverse-proxying configuration to matrix-nginx-proxy, - but it's pointless since the matrix-nginx-proxy role had already executed. - To fix this, please change the order of roles in your playbook, - so that the matrix-nginx-proxy role would run after the matrix-prometheus-postgres-exporter role. - when: matrix_nginx_proxy_role_executed | default(False) | bool - - name: Generate postgres-exporter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/postgres-exporter) ansible.builtin.set_fact: - matrix_prometheus_postgres_exporter_nginx_metrics_configuration_block: | + matrix_prometheus_services_proxy_connect_postgres_exporter_nginx_metrics_configuration_block: | location /metrics/postgres-exporter { {% if matrix_nginx_proxy_enabled | default(False) %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; - set $backend "matrix-prometheus-postgres-exporter:9187"; + set $backend "{{ prometheus_postgres_exporter_identifier }}:{{ prometheus_postgres_exporter_port }}"; proxy_pass http://$backend/metrics; - {% elif matrix_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host != '' %} - proxy_pass http://{{ matrix_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host }}/metrics; + {% elif matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host != '' %} + proxy_pass http://{{ matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host }}/metrics; {% else %} return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; {% endif %} @@ -31,5 +22,5 @@ {{ matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + - [matrix_prometheus_postgres_exporter_nginx_metrics_configuration_block] + [matrix_prometheus_services_proxy_connect_postgres_exporter_nginx_metrics_configuration_block] }} diff --git a/roles/custom/matrix-prometheus/defaults/main.yml b/roles/custom/matrix-prometheus/defaults/main.yml index 5216ccd7..956faf46 100644 --- a/roles/custom/matrix-prometheus/defaults/main.yml +++ b/roles/custom/matrix-prometheus/defaults/main.yml @@ -5,7 +5,7 @@ matrix_prometheus_enabled: false -matrix_prometheus_version: v2.41.0 +matrix_prometheus_version: v2.42.0 matrix_prometheus_docker_image: "{{ matrix_container_global_registry_prefix }}prom/prometheus:{{ matrix_prometheus_version }}" matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}" diff --git a/roles/custom/matrix-synapse-admin/defaults/main.yml b/roles/custom/matrix-synapse-admin/defaults/main.yml index 9f4510b6..0f5c7af6 100644 --- a/roles/custom/matrix-synapse-admin/defaults/main.yml +++ b/roles/custom/matrix-synapse-admin/defaults/main.yml @@ -9,7 +9,7 @@ matrix_synapse_admin_container_image_self_build_repo: "https://github.com/Awesom matrix_synapse_admin_docker_src_files_path: "{{ matrix_base_data_path }}/synapse-admin/docker-src" -matrix_synapse_admin_version: 0.8.5 +matrix_synapse_admin_version: 0.8.7 matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:{{ matrix_synapse_admin_version }}" matrix_synapse_admin_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_admin_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_synapse_admin_docker_image_force_pull: "{{ matrix_synapse_admin_docker_image.endswith(':latest') }}" diff --git a/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/setup_uninstall.yml b/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/setup_uninstall.yml index bb1b534c..7b820b35 100644 --- a/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/setup_uninstall.yml +++ b/roles/custom/matrix-synapse-reverse-proxy-companion/tasks/setup_uninstall.yml @@ -9,7 +9,7 @@ block: - name: Ensure matrix-synapse-reverse-proxy-companion.service is stopped ansible.builtin.service: - name: matrix_synapse_reverse_proxy_companion_service_stat + name: matrix-synapse-reverse-proxy-companion state: stopped enabled: false daemon_reload: true diff --git a/roles/custom/matrix-synapse-reverse-proxy-companion/templates/systemd/matrix-synapse-reverse-proxy-companion.service.j2 b/roles/custom/matrix-synapse-reverse-proxy-companion/templates/systemd/matrix-synapse-reverse-proxy-companion.service.j2 index 2b548ef8..125d7f49 100755 --- a/roles/custom/matrix-synapse-reverse-proxy-companion/templates/systemd/matrix-synapse-reverse-proxy-companion.service.j2 +++ b/roles/custom/matrix-synapse-reverse-proxy-companion/templates/systemd/matrix-synapse-reverse-proxy-companion.service.j2 @@ -16,7 +16,7 @@ Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-reverse-proxy-companion 2>/dev/null || true' ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-reverse-proxy-companion 2>/dev/null || true' -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run \ +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ --rm \ --name=matrix-synapse-reverse-proxy-companion \ --log-driver=none \ @@ -39,9 +39,11 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run \ {{ matrix_synapse_reverse_proxy_companion_container_image }} {% for network in matrix_synapse_reverse_proxy_companion_container_additional_networks %} -ExecStartPost={{ devture_systemd_docker_base_host_command_sh }} -c 'container_name=matrix-synapse-reverse-proxy-companion; network_name={{ network }}; attempt=0; while [ $attempt -le 29 ]; do attempt=$(( $attempt + 1 )); if [ "`{{ devture_systemd_docker_base_host_command_docker }} inspect -f {{ '{{.State.Running}}' }} $container_name 2> /dev/null`" = "true" ]; then break; fi; sleep 1; done; {{ devture_systemd_docker_base_host_command_docker }} network connect $network_name $container_name' +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-synapse-reverse-proxy-companion {% endfor %} +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-synapse-reverse-proxy-companion + ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-reverse-proxy-companion 2>/dev/null || true' ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-reverse-proxy-companion 2>/dev/null || true' ExecReload={{ devture_systemd_docker_base_host_command_docker }} exec matrix-synapse-reverse-proxy-companion /usr/sbin/nginx -s reload diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index 07eb803e..18165dd9 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -36,7 +36,7 @@ matrix_synapse_container_image_customizations_dockerfile_body_custom: '' matrix_synapse_docker_image: "{{ matrix_synapse_docker_image_name_prefix }}matrixdotorg/synapse:{{ matrix_synapse_docker_image_tag }}" matrix_synapse_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_container_image_self_build else matrix_container_global_registry_prefix }}" -matrix_synapse_version: v1.74.0 +matrix_synapse_version: v1.76.0 matrix_synapse_docker_image_tag: "{{ matrix_synapse_version }}" matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith(':latest') }}" @@ -123,7 +123,7 @@ matrix_synapse_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-synapse.service wants matrix_synapse_systemd_wanted_services_list: [] -matrix_synapse_in_container_python_packages_path: "/usr/local/lib/python3.9/site-packages" +matrix_synapse_in_container_python_packages_path: "/usr/local/lib/python3.11/site-packages" # Specifies which template files to use when configuring Synapse. # If you'd like to have your own different configuration, feel free to copy and paste @@ -725,6 +725,7 @@ matrix_synapse_ext_password_provider_shared_secret_config_yaml: | matrix_synapse_ext_password_provider_ldap_enabled: false matrix_synapse_ext_password_provider_ldap_uri: "ldap://ldap.mydomain.tld:389" matrix_synapse_ext_password_provider_ldap_start_tls: true +matrix_synapse_ext_password_provider_ldap_mode: "search" matrix_synapse_ext_password_provider_ldap_base: "" matrix_synapse_ext_password_provider_ldap_attributes_uid: "uid" matrix_synapse_ext_password_provider_ldap_attributes_mail: "mail" @@ -791,7 +792,7 @@ matrix_synapse_ext_encryption_config_yaml: | # Installing it requires building a customized Docker image for Synapse (see `matrix_synapse_container_image_customizations_enabled`). # Enabling this will enable customizations and inject the appropriate Dockerfile clauses for installing synapse-s3-storage-provider. matrix_synapse_ext_synapse_s3_storage_provider_enabled: false -matrix_synapse_ext_synapse_s3_storage_provider_version: 1.1.2 +matrix_synapse_ext_synapse_s3_storage_provider_version: 1.2.0 # Controls whether media from this (local) server is stored in s3-storage-provider matrix_synapse_ext_synapse_s3_storage_provider_store_local: true # Controls whether media from remote servers is stored in s3-storage-provider @@ -806,6 +807,9 @@ matrix_synapse_ext_synapse_s3_storage_provider_config_region_name: '' matrix_synapse_ext_synapse_s3_storage_provider_config_endpoint_url: '' matrix_synapse_ext_synapse_s3_storage_provider_config_access_key_id: '' matrix_synapse_ext_synapse_s3_storage_provider_config_secret_access_key: '' +matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_enabled: false +matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_key: '' +matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_algo: 'AES256' matrix_synapse_ext_synapse_s3_storage_provider_config_storage_class: STANDARD matrix_synapse_ext_synapse_s3_storage_provider_config_threadpool_size: 40 # matrix_synapse_ext_synapse_s3_storage_provider_update_db_day_count is a day value (number) for the `s3_media_upload update-db` command. @@ -815,7 +819,8 @@ matrix_synapse_ext_synapse_s3_storage_provider_update_db_day_count: 0 matrix_s3_media_store_enabled: false matrix_s3_media_store_custom_endpoint_enabled: false -matrix_s3_goofys_docker_image: "ewoutp/goofys:latest" +matrix_s3_goofys_docker_image: "{{ matrix_s3_goofys_docker_image_name_prefix }}ewoutp/goofys:latest" +matrix_s3_goofys_docker_image_name_prefix: "docker.io/" matrix_s3_goofys_docker_image_force_pull: "{{ matrix_s3_goofys_docker_image.endswith(':latest') }}" matrix_s3_media_store_custom_endpoint: "your-custom-endpoint" matrix_s3_media_store_bucket_name: "your-bucket-name" @@ -844,7 +849,7 @@ matrix_synapse_room_list_publication_rules: room_id: "*" action: allow -matrix_synapse_default_room_version: "9" +matrix_synapse_default_room_version: "10" # Controls the Synapse `spam_checker` setting. # @@ -891,7 +896,8 @@ matrix_synapse_redaction_retention_period: 7d matrix_synapse_user_ips_max_age: 28d -matrix_synapse_rust_synapse_compress_state_docker_image: "registry.gitlab.com/mb-saces/rust-synapse-compress-state:latest" +matrix_synapse_rust_synapse_compress_state_docker_image: "{{ matrix_synapse_rust_synapse_compress_state_docker_image_name_prefix }}mb-saces/rust-synapse-compress-state:latest" +matrix_synapse_rust_synapse_compress_state_docker_image_name_prefix: "registry.gitlab.com/" matrix_synapse_rust_synapse_compress_state_docker_image_force_pull: "{{ matrix_synapse_rust_synapse_compress_state_docker_image.endswith(':latest') }}" matrix_synapse_rust_synapse_compress_state_base_path: "{{ matrix_base_data_path }}/rust-synapse-compress-state" diff --git a/roles/custom/matrix-synapse/tasks/synapse/workers/init.yml b/roles/custom/matrix-synapse/tasks/synapse/workers/init.yml index d88884d0..359fe031 100644 --- a/roles/custom/matrix-synapse/tasks/synapse/workers/init.yml +++ b/roles/custom/matrix-synapse/tasks/synapse/workers/init.yml @@ -46,7 +46,7 @@ id: "federation-sender-{{ item }}" name: "matrix-synapse-worker-federation-sender-{{ item }}" type: 'federation_sender' - app: 'federation_sender' + app: 'generic_worker' webserving: false port: 0 metrics_port: "{{ matrix_synapse_workers_federation_sender_workers_metrics_range_start + item }}" @@ -58,14 +58,13 @@ matrix_synapse_federation_sender_instances: "{{ matrix_synapse_federation_sender_instances + [item.ansible_facts.worker.name] }}" with_items: "{{ matrix_synapse_workers_list_results_federation_sender_workers.results }}" -# This type of worker can only have a count of 1, at most - name: Build pusher workers ansible.builtin.set_fact: worker: id: "pusher-{{ item }}" name: "matrix-synapse-worker-pusher-{{ item }}" type: 'pusher' - app: 'pusher' + app: 'generic_worker' webserving: false port: 0 metrics_port: "{{ matrix_synapse_workers_pusher_workers_metrics_range_start + item }}" diff --git a/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 b/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 index a1174bce..a968965b 100644 --- a/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 +++ b/roles/custom/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 @@ -17,7 +17,7 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name --mount type=bind,src=/etc/passwd,dst=/etc/passwd,ro \ --mount type=bind,src=/etc/group,dst=/etc/group,ro \ --mount type=bind,src={{ matrix_s3_media_store_path }},dst=/s3,bind-propagation=shared \ - --security-opt apparmor:unconfined \ + --security-opt apparmor=unconfined \ --cap-add mknod \ --cap-add sys_admin \ --device=/dev/fuse \ diff --git a/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/bin/migrate.j2 b/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/bin/migrate.j2 index 6ce085f2..0f47b615 100644 --- a/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/bin/migrate.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/bin/migrate.j2 @@ -10,4 +10,4 @@ --network={{ matrix_docker_network }} \ --entrypoint=/bin/bash \ {{ matrix_synapse_docker_image_final }} \ - -c 's3_media_upload update-db $UPDATE_DB_DURATION && s3_media_upload --no-progress check-deleted $MEDIA_PATH && s3_media_upload --no-progress upload $MEDIA_PATH $BUCKET --delete --storage-class $STORAGE_CLASS --endpoint-url $ENDPOINT' + -c 's3_media_upload update-db $UPDATE_DB_DURATION && s3_media_upload --no-progress check-deleted $MEDIA_PATH && s3_media_upload --no-progress upload $MEDIA_PATH $BUCKET --delete --storage-class $STORAGE_CLASS --endpoint-url $ENDPOINT {% if matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_enabled %}--sse-customer-algo $SSE_CUSTOMER_ALGO --sse-customer-key $SSE_CUSTOMER_KEY{% endif %}' diff --git a/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/env.j2 b/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/env.j2 index 6dfcbe41..58d26255 100644 --- a/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/env.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/env.j2 @@ -4,6 +4,12 @@ AWS_DEFAULT_REGION={{ matrix_synapse_ext_synapse_s3_storage_provider_config_regi ENDPOINT={{ matrix_synapse_ext_synapse_s3_storage_provider_config_endpoint_url }} BUCKET={{ matrix_synapse_ext_synapse_s3_storage_provider_config_bucket }} + +{% if matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_enabled %} +SSE_CUSTOMER_KEY={{ matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_key }} +SSE_CUSTOMER_ALGO={{ matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_algo }} +{% endif %} + STORAGE_CLASS={{ matrix_synapse_ext_synapse_s3_storage_provider_config_storage_class }} MEDIA_PATH=/matrix-media-store-parent/{{ matrix_synapse_media_store_directory_name }} diff --git a/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/media_storage_provider.yaml.j2 b/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/media_storage_provider.yaml.j2 index 97b0f5f2..ac2b58db 100644 --- a/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/media_storage_provider.yaml.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/ext/s3-storage-provider/media_storage_provider.yaml.j2 @@ -9,6 +9,11 @@ config: access_key_id: {{ matrix_synapse_ext_synapse_s3_storage_provider_config_access_key_id | to_json }} secret_access_key: {{ matrix_synapse_ext_synapse_s3_storage_provider_config_secret_access_key | to_json }} +{% if matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_enabled %} + sse_customer_key: {{ matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_key | to_json }} + sse_customer_algo: {{ matrix_synapse_ext_synapse_s3_storage_provider_config_sse_customer_algo | to_json }} +{% endif %} + storage_class: {{ matrix_synapse_ext_synapse_s3_storage_provider_config_storage_class | to_json }} threadpool_size: {{ matrix_synapse_ext_synapse_s3_storage_provider_config_threadpool_size | to_json }} diff --git a/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 index d2dd1c74..1885aeef 100644 --- a/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 +++ b/roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2 @@ -2511,6 +2511,7 @@ password_providers: - module: "ldap_auth_provider.LdapAuthProvider" config: enabled: true + mode: {{ matrix_synapse_ext_password_provider_ldap_mode | string | to_json }} uri: {{ matrix_synapse_ext_password_provider_ldap_uri | string|to_json }} start_tls: {{ matrix_synapse_ext_password_provider_ldap_start_tls|to_json }} base: {{ matrix_synapse_ext_password_provider_ldap_base | string|to_json }} diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml new file mode 100644 index 00000000..496c4c75 --- /dev/null +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -0,0 +1,7 @@ +--- + +# Controls if (`matrix_prometheus_node_exporter` -> `prometheus_node_exporter`) validation will run. +matrix_playbook_migration_matrix_prometheus_node_exporter_migration_validation_enabled: true + +# Controls if (`matrix_prometheus_postgres_exporter` -> `prometheus_postgres_exporter`) validation will run. +matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index f4ce2bdc..b04c9e0e 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -34,6 +34,8 @@ - {'old': 'matrix_systemd_services_list', 'new': 'devture_systemd_service_manager_services_list_additional'} - {'old': 'matrix_common_after_systemd_service_start_wait_for_timeout_seconds', 'new': 'devture_systemd_service_manager_up_verification_delay_seconds'} - {'old': 'matrix_systemd_services_autostart_enabled', 'new': 'devture_systemd_service_manager_services_autostart_enabled'} + - {'old': 'matrix_prometheus_node_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled'} + - {'old': 'matrix_prometheus_postgres_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_postgres_exporter_metrics_proxying_enabled'} - name: (Deprecation) Catch and report matrix_postgres variables ansible.builtin.fail: @@ -48,3 +50,33 @@ The following variables in your configuration need to be renamed: {{ vars | dict2items | selectattr('key', 'match', 'matrix_postgres_.*') | map (attribute='key') | join(', ') }} when: "vars | dict2items | selectattr('key', 'match', 'matrix_postgres_.*') | list | items2dict" + +- when: matrix_playbook_migration_matrix_prometheus_node_exporter_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_matrix_prometheus_node_exporter_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_prometheus_node_exporter_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_prometheus_node_exporter variables + ansible.builtin.fail: + msg: >- + The matrix-prometheus-node-exporter role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/prometheus_node_exporter. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_prometheus_node_exporter_` -> `prometheus_node_exporter_`). + We found usage of the following variables: {{ matrix_playbook_migration_matrix_prometheus_node_exporter_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_matrix_prometheus_node_exporter_migration_vars | length > 0" + +- when: matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_prometheus_postgres_exporter_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_prometheus_postgres_exporter variables + ansible.builtin.fail: + msg: >- + The matrix-prometheus-postgres-exporter role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_prometheus_postgres_exporter_` -> `prometheus_postgres_exporter_`). + We found usage of the following variables: {{ matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_matrix_prometheus_postgres_exporter_migration_vars | length > 0"