diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f9fe2aa..61270ead 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,43 @@ +# 2022-06-23 + +## (Potential Backward Compatibility Break) Changes around metrics collection + +**TLDR**: we've made extensive **changes to metrics exposure/collection, which concern people using an external Prometheus server**. If you don't know what that is, you don't need to read below. + +**Why do major changes to metrics**? Because various services were exposing metrics in different, hacky, ways. Synapse was exposing metrics at `/_synapse/metrics` and `/_synapse-worker-.../metrics` on the `matrix.DOMAIN`. The Hookshot role was **repurposing** the Granana web UI domain (`stats.DOMAIN`) for exposing its metrics on `stats.DOMAIN/hookshot/metrics`, while protecting these routes using Basic Authentication **normally used for Synapse** (`/_synapse/metrics`). Node-exporter and Postgres-exporter roles were advising for more `stats.DOMAIN` usage in manual ways. Each role was doing things differently and mixing variables from other roles. Each metrics endpoint was ending up in a different place, protected by who knows what Basic Authentication credentials (if protected at all). + +**The solution**: a completely revamped way to expose metrics to an external Prometheus server. We are **introducing new `https://matrix.DOMAIN/metrics/*` endpoints**, where various services *can* expose their metrics, for collection by external Prometheus servers. To enable the `/metrics/*` endpoints, use `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. There's also a way to protect access using [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication). See the `matrix-nginx-proxy` role or our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation for additional variables around `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. + +**If you are using the [Hookshot bridge](docs/configuring-playbook-bridge-hookshot.md)**, you may find that: +1. **Metrics may not be enabled by default anymore**: + - If Prometheus is enabled (`matrix_prometheus_enabled: true`), then Hookshot metrics will be enabled automatically (`matrix_hookshot_metrics_enabled: true`). These metrics will be collected from the local (in-container) Prometheus over the container network. + - **If Prometheus is not enabled** (you are either not using Prometheus or are using an external one), **Hookshot metrics will not be enabled by default anymore**. Feel free to enable them by setting `matrix_hookshot_metrics_enabled: true`. Also, see below. +2. When metrics are meant to be **consumed by an external Prometheus server**, `matrix_hookshot_metrics_proxying_enabled` needs to be set to `true`, so that metrics would be exposed (proxied) "publicly" on `https://matrix.DOMAIN/metrics/hookshot`. To make use of this, you'll also need to enable the new `https://matrix.DOMAIN/metrics/*` endpoints mentioned above, using `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. Learn more in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. +3. **We've changed the URL we're exposing Hookshot metrics at** for external Prometheus servers. Until now, you were advised to consume Hookshot metrics from `https://stats.DOMAIN/hookshot/metrics` (working in conjunction with `matrix_nginx_proxy_proxy_synapse_metrics`). From now on, **this no longer works**. As described above, you need to start consuming metrics from `https://matrix.DOMAIN/metrics/hookshot`. + +**If you're using node-exporter** (`matrix_prometheus_node_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_node_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/node-exporter`. + +**If you're using [postgres-exporter](docs/configuring-playbook-prometheus-postgres.md)** (`matrix_prometheus_postgres_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_postgres_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/postgres-exporter`. + +**If you're using Synapse** and would like to collect its metrics from an external Prometheus server, you may find that: + +1. Exposing metrics is now done using `matrix_synapse_metrics_proxying_enabled`, not `matrix_nginx_proxy_proxy_synapse_metrics: true`. You may still need to enable metrics using `matrix_synapse_metrics_enabled: true` before exposing them. +2. Protecting metrics endpoints using [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) is now done in another way. See our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation +3. If Synapse metrics are exposed, they will be made available at `https://matrix.DOMAIN/metrics/synapse/main-process` or `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID` (when workers are enabled), not at `https://matrix.DOMAIN/_synapse/metrics` and `https://matrix.DOMAIN/_synapse-worker-.../metrics` +4. The playbook still generates an `external_prometheus.yml.example` sample file for scraping Synapse from Prometheus as described in [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server), but it's now saved under `/matrix/synapse` (not `/matrix`). + +**If you where already using a external Prometheus server** before this change, and you gave a hashed version of the password as a variable, the playbook will now take care of hashing the password for you. Thus, you need to provide the non-hashed version now. + +# 2022-06-13 + +## go-skype-bridge bridging support + +Thanks to [CyberShadow](https://github.com/CyberShadow), the playbook can now install the [go-skype-bridge](https://github.com/kelaresg/go-skype-bridge) bridge for bridging Matrix to [Skype](https://www.skype.com/). + +See our [Setting up Go Skype Bridge](docs/configuring-playbook-bridge-go-skype-bridge.md) documentation to get started. + +The playbook has supported [mx-puppet-skype](https://github.com/Sorunome/mx-puppet-skype) bridging (see [Setting up MX Puppet Skype bridging](docs/configuring-playbook-bridge-mx-puppet-skype.md)) since [2020-04-09](#2020-04-09), but `mx-puppet-skype` is reportedly broken. + # 2022-06-09 ## Running Ansible in a container can now happen on the Matrix server itself diff --git a/docs/configuring-playbook-bridge-go-skype-bridge.md b/docs/configuring-playbook-bridge-go-skype-bridge.md new file mode 100644 index 00000000..a7a4fbc8 --- /dev/null +++ b/docs/configuring-playbook-bridge-go-skype-bridge.md @@ -0,0 +1,23 @@ +# Setting up Go Skype Bridge (optional) + +The playbook can install and configure +[go-skype-bridge](https://github.com/kelaresg/go-skype-bridge) for you. + +See the project page to learn what it does and why it might be useful to you. + +To enable the [Skype](https://www.skype.com/) bridge just use the following +playbook configuration: + + +```yaml +matrix_go_skype_bridge_enabled: true +``` + + +## Usage + +Once the bot is enabled, you need to start a chat with `Skype bridge bot` +with the handle `@skypebridgebot:YOUR_DOMAIN` (where `YOUR_DOMAIN` is your base +domain, not the `matrix.` domain). + +Send `help` to the bot to see the commands available. diff --git a/docs/configuring-playbook-bridge-hookshot.md b/docs/configuring-playbook-bridge-hookshot.md index f47f24c0..3e8a54a2 100644 --- a/docs/configuring-playbook-bridge-hookshot.md +++ b/docs/configuring-playbook-bridge-hookshot.md @@ -14,7 +14,7 @@ Refer to the [official instructions](https://matrix-org.github.io/matrix-hooksho 1. For each of the services (GitHub, GitLab, Jira, Figma, generic webhooks) fill in the respective variables `matrix_hookshot_service_*` listed in [main.yml](/roles/matrix-bridge-hookshot/defaults/main.yml) as required. 2. Take special note of the `matrix_hookshot_*_enabled` variables. Services that need no further configuration are enabled by default (GitLab, Generic), while you must first add the required configuration and enable the others (GitHub, Jira, Figma). -3. If you're setting up the GitHub bridge, you'll need to generate and download a private key file after you created your GitHub app. Copy the contents of that file to the variable `matrix_hookshot_github_private_key` so the playbook can install it for you, or use one of the [other methods](#manage-github-private-key-with-matrix-aux-role) explained below. +3. If you're setting up the GitHub bridge, you'll need to generate and download a private key file after you created your GitHub app. Copy the contents of that file to the variable `matrix_hookshot_github_private_key` so the playbook can install it for you, or use one of the [other methods](#manage-github-private-key-with-matrix-aux-role) explained below. 4. If you've already installed Matrix services using the playbook before, you'll need to re-run it (`--tags=setup-all,start`). If not, proceed with [configuring other playbook services](configuring-playbook.md) and then with [Installing](installing.md). Get back to this guide once ready. Hookshot can be set up individually using the tag `setup-hookshot`. 5. Refer to [Hookshot's official instructions](https://matrix-org.github.io/matrix-hookshot/latest/usage.html) to start using the bridge. **Important:** Note that the different listeners are bound to certain paths which might differ from those assumed by the hookshot documentation, see [URLs for bridges setup](urls-for-bridges-setup) below. @@ -32,8 +32,8 @@ Unless indicated otherwise, the following endpoints are reachable on your `matri | figma endpoint | `/hookshot/webhooks/figma/webhook` | `matrix_hookshot_figma_endpoint` | Figma | | provisioning | `/hookshot/v1/` | `matrix_hookshot_provisioning_endpoint` | Dimension [provisioning](#provisioning-api) | | appservice | `/hookshot/_matrix/app/` | `matrix_hookshot_appservice_endpoint` | Matrix server | -| widgets | `/hookshot/widgetapi/` | `/matrix_hookshot_widgets_endpoint` | Widgets | -| metrics | `/hookshot/metrics/` (on `stats.` subdomain) | `matrix_hookshot_metrics_endpoint` | Prometheus | +| widgets | `/hookshot/widgetapi/` | `matrix_hookshot_widgets_endpoint` | Widgets | +| metrics | `/metrics/hookshot` | `matrix_hookshot_metrics_enabled` and `matrix_hookshot_metrics_proxying_enabled`. Requires `/metrics/*` endpoints to also be enabled via `matrix_nginx_proxy_proxy_matrix_metrics_enabled` (see the `matrix-nginx-proxy` role). Read more in the [Metrics section](#metrics) below. | Prometheus | See also `matrix_hookshot_matrix_nginx_proxy_configuration` in [init.yml](/roles/matrix-bridge-hookshot/tasks/init.yml). @@ -63,7 +63,14 @@ The provisioning API will be enabled automatically if you set `matrix_dimension_ ### Metrics -If metrics are enabled, they will be automatically available in the builtin Prometheus and Grafana, but you need to set up your own Dashboard for now. If additionally metrics proxying for use with external Prometheus is enabled (`matrix_nginx_proxy_proxy_synapse_metrics`), hookshot metrics will also be available (at `matrix_hookshot_metrics_endpoint`, default `/hookshot/metrics`, on the stats subdomain) and with the same password. See also [the Prometheus and Grafana docs](../configuring-playbook-prometheus-grafana.md). +Metrics are **only enabled by default** if the builtin [Prometheus](configuring-playbook-prometheus-grafana.md) is enabled (by default, Prometheus isn't enabled). If so, metrics will automatically be collected by Prometheus and made available in Grafana. You will, however, need to set up your own Dashboard for displaying them. + +To explicitly enable metrics, use `matrix_hookshot_metrics_enabled: true`. This only exposes metrics over the container network, however. + +**To collect metrics from an external Prometheus server**, besides enabling metrics as described above, you will also need to: + +- enable the `https://matrix.DOMAIN/metrics/*` endpoints on `matrix.DOMAIN` using `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` (see the `matrix-nginx-role` or [the Prometheus and Grafana docs](configuring-playbook-prometheus-grafana.md) for enabling this feature) +- expose the Hookshot metrics under `https://matrix.DOMAIN/metrics/hookshot` by setting `matrix_hookshot_metrics_proxying_enabled: true` ### Collision with matrix-appservice-webhooks diff --git a/docs/configuring-playbook-bridge-mautrix-facebook.md b/docs/configuring-playbook-bridge-mautrix-facebook.md index 1845682f..bb8d1430 100644 --- a/docs/configuring-playbook-bridge-mautrix-facebook.md +++ b/docs/configuring-playbook-bridge-mautrix-facebook.md @@ -91,3 +91,5 @@ Once connected, you should be able to verify that you're browsing the web throug Then proceed to log in to [Facebook/Messenger](https://www.facebook.com/). Once logged in, proceed to [set up bridging](#usage). + +If that doesn't work, enable 2FA [Facebook help page on enabling 2FA](https://www.facebook.com/help/148233965247823) and try to login again with a new password, and entering the 2FA code when prompted, it may take more then one try, in between attempts, check facebook.com to see if they are requiring another password change diff --git a/docs/configuring-playbook-bridge-mx-puppet-skype.md b/docs/configuring-playbook-bridge-mx-puppet-skype.md index cca5e305..ff4e636e 100644 --- a/docs/configuring-playbook-bridge-mx-puppet-skype.md +++ b/docs/configuring-playbook-bridge-mx-puppet-skype.md @@ -1,5 +1,7 @@ # Setting up MX Puppet Skype (optional) +**Note**: bridging to [Skype](https://www.skype.com/) can also happen via the [go-skype-bridge](configuring-playbook-bridge-go-skype-bridge.md) bridge supported by the playbook. In fact, bridging via `mx-puppet-skype` has often been reported as broken, so we recommend that you go directly for `go-skype-bridge`, instead of this. + The playbook can install and configure [mx-puppet-skype](https://github.com/Sorunome/mx-puppet-skype) for you. diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md index f178def2..b2878c12 100644 --- a/docs/configuring-playbook-prometheus-grafana.md +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -9,8 +9,12 @@ Remember to add `stats.` to DNS as described in [Configuring DNS](c ```yaml matrix_prometheus_enabled: true +# You can remove this, if unnecessary. matrix_prometheus_node_exporter_enabled: true +# You can remove this, if unnecessary. +matrix_prometheus_postgres_exporter_enabled: true + matrix_grafana_enabled: true matrix_grafana_anonymous_access: false @@ -34,6 +38,7 @@ Name | Description -----|---------- `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. `matrix_prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures +`matrix_prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. `matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.` subdomain) the dashboards with the graphs that we're interested in `matrix_grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option. `matrix_grafana_default_admin_user`
`matrix_grafana_default_admin_password`|By default Grafana creates a user with `admin` as the username and password. If you feel this is insecure and you want to change it beforehand, you can do that here @@ -48,28 +53,55 @@ Most of our docker containers run with limited system access, but the `prometheu ## Collecting metrics to an external Prometheus server -If you wish, you could expose homeserver metrics without enabling (installing) Prometheus and Grafana via the playbook. This may be useful for hooking Matrix services to an external Prometheus/Grafana installation. +**If the integrated Prometheus server is enabled** (`matrix_prometheus_enabled: true`), metrics are collected by it from each service via communication that happens over the container network. Each service does not need to expose its metrics "publicly". -To do this, you may be interested in the following variables: +When you'd like **to collect metrics from an external Prometheus server**, you need to expose service metrics outside of the container network. + +The playbook provides a single endpoint (`https://matrix.DOMAIN/metrics/*`), under which various services may expose their metrics (e.g. `/metrics/node-exporter`, `/metrics/postgres-exporter`, `/metrics/hookshot`, etc). To enable this `/metrics/*` feature, use `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. To protect access using [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication), see `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled` below. + +The following variables may be of interest: Name | Description -----|---------- +`matrix_nginx_proxy_proxy_matrix_metrics_enabled`|Set this to `true` to enable metrics exposure for various services on `https://matrix.DOMAIN/metrics/*`. Refer to the individual `matrix_SERVICE_metrics_proxying_enabled` variables below for exposing metrics for each individual service. +`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled`|Set this to `true` to protect all `https://matrix.DOMAIN/metrics/*` endpoints with [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) (see the other variables below for supplying the actual credentials). When enabled, all endpoints beneath `/metrics` will be protected with the same credentials +`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username`|Set this to the Basic Authentication username you'd like to protect `/metrics/*` with. You also need to set `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password`. If one username/password pair is not enough, you can leave the `username` and `password` variables unset and use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content` instead +`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password`|Set this to the Basic Authentication password you'd like to protect `/metrics/*` with +`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`|Set this to the Basic Authentication credentials (raw `htpasswd` file content) used to protect `/metrics/*`. This htpasswd-file needs to be generated with the `htpasswd` tool and can include multiple username/password pairs. If you only need one credential, use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` instead. `matrix_synapse_metrics_enabled`|Set this to `true` to make Synapse expose metrics (locally, on the container network) -`matrix_nginx_proxy_proxy_synapse_metrics`|Set this to `true` to make matrix-nginx-proxy expose the Synapse metrics at `https://matrix.DOMAIN/_synapse/metrics` -`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled`|Set this to `true` to password-protect (using HTTP Basic Auth) `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus`, the password is defined in `matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`) -`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`|Set this to a password to use for HTTP Basic Auth for protecting `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus` - it's not configurable). Do not write the password in plain text. See `man 1 htpasswd` or use `htpasswd -c mypass.htpasswd prometheus` to generate the expected hash for nginx. -`matrix_server_fqn_grafana`|Use this variable to override the domain at which the Grafana web user-interface is at (defaults to `stats.DOMAIN`) +`matrix_synapse_metrics_proxying_enabled`|Set this to `true` to expose Synapse's metrics on `https://matrix.DOMAIN/metrics/synapse/main-process` and `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`). Read [below](#collecting-synapse-worker-metrics-to-an-external-prometheus-server) if you're running a Synapse worker setup (`matrix_synapse_workers_enabled: true`). +`matrix_prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter (locally, on the container network) +`matrix_prometheus_node_exporter_metrics_proxying_enabled`|Set this to `true` to expose the node (general system stats) metrics on `https://matrix.DOMAIN/metrics/node-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`matrix_prometheus_postgres_exporter_enabled`|Set this to `true` to enable the [Postgres exporter](configuring-playbook-prometheus-postgres.md) (locally, on the container network) +`matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|Set this to `true` to expose the [Postgres exporter](configuring-playbook-prometheus-postgres.md) metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`matrix_bridge_hookshot_metrics_enabled`|Set this to `true` to make [Hookshot](configuring-playbook-bridge-hookshot.md) expose metrics (locally, on the container network) +`matrix_bridge_hookshot_metrics_proxying_enabled`|Set this to `true` to expose the [Hookshot](configuring-playbook-bridge-hookshot.md) metrics on `https://matrix.DOMAIN/metrics/hookshot` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`matrix_SERVICE_metrics_proxying_enabled`|Various other services/roles may provide similar `_metrics_enabled` and `_metrics_proxying_enabled` variables for exposing their metrics. Refer to each role for details. Only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` +`matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks`|Add nginx `location` blocks to this list if you'd like to expose additional exporters manually (see below) -### Collecting worker metrics to an external Prometheus server +Example for how to make use of `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks` for exposing additional metrics locations: +```nginx +matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks: + - 'location /metrics/another-service { + resolver 127.0.0.11 valid=5s; + proxy_pass http://matrix-another-service:9100/metrics; + }' +``` -If you are using workers (`matrix_synapse_workers_enabled`) and have enabled `matrix_nginx_proxy_proxy_synapse_metrics` as described above, the playbook will also automatically proxy the all worker threads's metrics to `https://matrix.DOMAIN/_synapse-worker-TYPE-ID/metrics`, where `TYPE` corresponds to the type and `ID` to the instanceId of a worker as exemplified in `matrix_synapse_workers_enabled_list`. +Using `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks` only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` (see above). -The playbook also generates an exemplary prometheus.yml config file (`matrix_base_data_path/external_prometheus.yml.template`) with all the correct paths which you can copy to your Prometheus server and adapt to your needs, especially edit the specified `password_file` path and contents and path to your `synapse-v2.rules`. +Note : The playbook will hash the basic_auth password for you on setup. Thus, you need to give the plain-text version of the password as a variable. + +### Collecting Synapse worker metrics to an external Prometheus server + +If you are using workers (`matrix_synapse_workers_enabled: true`) and have enabled `matrix_synapse_metrics_proxying_enabled` as described above, the playbook will also automatically expose all Synapse worker threads' metrics to `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID`, where `TYPE` corresponds to the type and `ID` to the instanceId of a worker as exemplified in `matrix_synapse_workers_enabled_list`. + +The playbook also generates an exemplary config file (`/matrix/synapse/external_prometheus.yml.template`) with all the correct paths which you can copy to your Prometheus server and adapt to your needs. Make sure to edit the specified `password_file` path and contents and path to your `synapse-v2.rules`. It will look a bit like this: ```yaml scrape_configs: - job_name: 'synapse' - metrics_path: /_synapse/metrics + metrics_path: /metrics/synapse/main-process scheme: https basic_auth: username: prometheus @@ -80,7 +112,7 @@ scrape_configs: job: "master" index: 1 - job_name: 'synapse-generic_worker-1' - metrics_path: /_synapse-worker-generic_worker-18111/metrics + metrics_path: /metrics/synapse/worker/generic_worker-18111 scheme: https basic_auth: username: prometheus @@ -92,38 +124,6 @@ scrape_configs: index: 18111 ``` -### Collecting system and Postgres metrics to an external Prometheus server (advanced) - -When you normally enable the Prometheus and Grafana via the playbook, it will also show general system (via node-exporter) and Postgres (via postgres-exporter) stats. If you are instead collecting your metrics to an external Prometheus server, you can follow this advanced configuration example to also export these stats. - -It would be possible to use `matrix_prometheus_node_exporter_container_http_host_bind_port` etc., but that is not always the best choice, for example because your server is on a public network. - -Use the following variables in addition to the ones mentioned above: - -Name | Description ------|---------- -`matrix_nginx_proxy_proxy_grafana_enabled`|Set this to `true` to make the stats subdomain (`matrix_server_fqn_grafana`) available via the Nginx proxy -`matrix_ssl_additional_domains_to_obtain_certificates_for`|Add `"{{ matrix_server_fqn_grafana }}"` to this list to have letsencrypt fetch a certificate for the stats subdomain -`matrix_prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter -`matrix_prometheus_postgres_exporter_enabled`|Set this to `true` to enable the Postgres exporter -`matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks`|Add locations to this list depending on which of the above exporters you enabled (see below) - -```nginx -matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks: - - 'location /node-exporter/ { - resolver 127.0.0.11 valid=5s; - proxy_pass http://matrix-prometheus-node-exporter:9100/; - auth_basic "protected"; - auth_basic_user_file /nginx-data/matrix-synapse-metrics-htpasswd; - }' - - 'location /postgres-exporter/ { - resolver 127.0.0.11 valid=5s; - proxy_pass http://matrix-prometheus-postgres-exporter:9187/; - auth_basic "protected"; - auth_basic_user_file /nginx-data/matrix-synapse-metrics-htpasswd; - }' -``` -You can customize the `location`s to your liking, just point your Prometheus to there later (e.g. `stats.DOMAIN/node-exporter/metrics`). Nginx is very picky about the `proxy_pass`syntax: take care to follow the example closely and note the trailing slash as well as absent use of variables. postgres-exporter uses the nonstandard port 9187. ## More information @@ -131,4 +131,3 @@ You can customize the `location`s to your liking, just point your Prometheus to - [The Prometheus scraping rules](https://github.com/matrix-org/synapse/tree/master/contrib/prometheus) (we use v2) - [The Synapse Grafana dashboard](https://github.com/matrix-org/synapse/tree/master/contrib/grafana) - [The Node Exporter dashboard](https://github.com/rfrail3/grafana-dashboards) (for generic non-synapse performance graphs) - diff --git a/docs/configuring-playbook-prometheus-postgres.md b/docs/configuring-playbook-prometheus-postgres.md index 34407aae..6fd13a9e 100644 --- a/docs/configuring-playbook-prometheus-postgres.md +++ b/docs/configuring-playbook-prometheus-postgres.md @@ -7,11 +7,6 @@ You can enable this with the following settings in your configuration file (`inv ```yaml matrix_prometheus_postgres_exporter_enabled: true - -# the role creates a postgres user as credential. You can configure these if required: -matrix_prometheus_postgres_exporter_database_username: 'matrix_prometheus_postgres_exporter' -matrix_prometheus_postgres_exporter_database_password: 'some-password' - ``` ## What does it do? @@ -20,7 +15,8 @@ Name | Description -----|---------- `matrix_prometheus_postgres_exporter_enabled`|Enable the postgres prometheus exporter. This sets up the docker container, connects it to the database and adds a 'job' to the prometheus config which tells prometheus about this new exporter. The default is 'false' `matrix_prometheus_postgres_exporter_database_username`| The 'username' for the user that the exporter uses to connect to the database. The default is 'matrix_prometheus_postgres_exporter' -`matrix_prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database. +`matrix_prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database. By default, this is auto-generated by the playbook +`matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|If set to `true`, exposes the Postgres exporter metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` for usage with an [external Prometheus server](configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) ## More information diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index 16a7aeeb..3bfb01bd 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -120,7 +120,7 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up matrix-hookshot](configuring-playbook-bridge-hookshot.md) - a bridge between Matrix and multiple project management services, such as [GitHub](https://github.com), [GitLab](https://about.gitlab.com) and [JIRA](https://www.atlassian.com/software/jira). (optional) -- [Setting up MX Puppet Skype bridging](configuring-playbook-bridge-mx-puppet-skype.md) (optional) +- [Setting up MX Puppet Skype bridging](configuring-playbook-bridge-mx-puppet-skype.md) (optional) - often reported as broken; see **Go Skype Bridge** (below) as an alternative - [Setting up MX Puppet Slack bridging](configuring-playbook-bridge-mx-puppet-slack.md) (optional) @@ -134,6 +134,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up MX Puppet Steam bridging](configuring-playbook-bridge-mx-puppet-steam.md) (optional) +- [Setting up Go Skype Bridge bridging](configuring-playbook-bridge-go-skype-bridge.md) (optional) + - [Setting up Email2Matrix](configuring-playbook-email2matrix.md) (optional) - [Setting up Matrix SMS bridging](configuring-playbook-bridge-matrix-bridge-sms.md) (optional) diff --git a/examples/caddy2/Caddyfile b/examples/caddy2/Caddyfile index 7d8c193d..162e539e 100644 --- a/examples/caddy2/Caddyfile +++ b/examples/caddy2/Caddyfile @@ -214,20 +214,21 @@ element.DOMAIN.tld { # } # } #} -DOMAIN.com { - @wellknown { - path /.well-known/matrix/* - } - - handle @wellknown { - reverse_proxy https://matrix.DOMAIN.com { - header_up Host {http.reverse_proxy.upstream.hostport} - } - } - # Configration for the base domain goes here - # handle { - # header -Server - # encode zstd gzip - # reverse_proxy localhost:4020 - # } -} +#DOMAIN.com { +# Uncomment this if you are following "(Option 3): Setting up reverse-proxying of the well-known files from the base domain's server to the Matrix server" of https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-well-known.md#option-3-setting-up-reverse-proxying-of-the-well-known-files-from-the-base-domains-server-to-the-matrix-server +# @wellknown { +# path /.well-known/matrix/* +# } +# +# handle @wellknown { +# reverse_proxy https://matrix.DOMAIN.com { +# header_up Host {http.reverse_proxy.upstream.hostport} +# } +# } +# # Configration for the base domain goes here +# # handle { +# # header -Server +# # encode zstd gzip +# # reverse_proxy localhost:4020 +# # } +#} diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 8788ba10..394e26dc 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -252,6 +252,44 @@ matrix_beeper_linkedin_database_password: "{{ '%s' | format(matrix_homeserver_ge # ###################################################################### +###################################################################### +# +# matrix-bridge-go-skype-bridge +# +###################################################################### + +# We don't enable bridges by default. +matrix_go_skype_bridge_enabled: false + +matrix_go_skype_bridge_container_image_self_build: true + +matrix_go_skype_bridge_systemd_required_services_list: | + {{ + ['docker.service'] + + + ['matrix-' + matrix_homeserver_implementation + '.service'] + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) + + + (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else []) + }} + +matrix_go_skype_bridge_appservice_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'skype.as.token') | to_uuid }}" + +matrix_go_skype_bridge_homeserver_token: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'skype.hs.token') | to_uuid }}" + +matrix_go_skype_bridge_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" + +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_go_skype_bridge_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_go_skype_bridge_database_password: "{{ '%s' | format(matrix_homeserver_generic_secret_key) | password_hash('sha512', 'goskype.db') | to_uuid }}" + +###################################################################### +# +# /matrix-bridge-go-skype-bridge +# +###################################################################### + ###################################################################### # # matrix-bridge-mautrix-facebook @@ -673,8 +711,13 @@ matrix_hookshot_container_http_host_bind_ports: "{{ [] if matrix_nginx_proxy_ena matrix_hookshot_provisioning_enabled: "{{ matrix_hookshot_provisioning_secret and matrix_dimension_enabled }}" -matrix_hookshot_proxy_metrics: "{{ matrix_nginx_proxy_proxy_synapse_metrics }}" -matrix_hookshot_proxy_metrics_basic_auth_enabled: "{{ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled }}" +# We only enable metrics (locally, in the container network) for the bridge if Prometheus is enabled. +# +# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely: +# - `matrix_hookshot_metrics_enabled` +# - `matrix_hookshot_metrics_proxying_enabled` +# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled` +matrix_hookshot_metrics_enabled: "{{ matrix_prometheus_enabled }}" matrix_hookshot_urlprefix_port_enabled: "{{ matrix_nginx_proxy_container_https_host_bind_port == 443 if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_https_host_bind_port == 80 }}" matrix_hookshot_urlprefix_port: ":{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}" @@ -1222,7 +1265,7 @@ matrix_corporal_matrix_registration_shared_secret: "{{ matrix_synapse_registrati matrix_coturn_enabled: true -matrix_coturn_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" +matrix_coturn_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm32', 'arm64'] }}" matrix_coturn_turn_external_ip_address: "{{ ansible_host }}" @@ -1548,13 +1591,6 @@ matrix_nginx_proxy_proxy_matrix_federation_port: "{{ matrix_federation_public_po matrix_nginx_proxy_container_federation_host_bind_port: "{{ matrix_federation_public_port }}" -# This used to be hooked to `matrix_synapse_metrics_enabled`, but we don't do it anymore. -# The fact that someone wishes to enable Synapse metrics does not necessarily mean they want to make them public. -# A local Prometheus can consume them over the container network. -matrix_nginx_proxy_proxy_synapse_metrics: false -matrix_nginx_proxy_proxy_synapse_metrics_addr_with_container: "matrix-synapse:{{ matrix_synapse_metrics_port }}" -matrix_nginx_proxy_proxy_synapse_metrics_addr_sans_container: "127.0.0.1:{{ matrix_synapse_metrics_port }}" - matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled: "{{ matrix_ma1sd_enabled }}" matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_with_container: "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container }}" matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container: "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container }}" @@ -1576,8 +1612,6 @@ matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}" matrix_nginx_proxy_synapse_frontend_proxy_locations: "{{ matrix_synapse_workers_frontend_proxy_endpoints|default([]) }}" -matrix_nginx_proxy_proxy_synapse_workers_enabled_list: "{{ matrix_synapse_workers_enabled_list }}" - matrix_nginx_proxy_systemd_wanted_services_list: | {{ ['matrix-' + matrix_homeserver_implementation + '.service'] @@ -1770,6 +1804,12 @@ matrix_postgres_additional_databases: | 'password': matrix_beeper_linkedin_database_password, }] if (matrix_beeper_linkedin_enabled and matrix_beeper_linkedin_database_engine == 'postgres' and matrix_beeper_linkedin_database_hostname == 'matrix-postgres') else []) + + ([{ + 'name': matrix_go_skype_bridge_database_name, + 'username': matrix_go_skype_bridge_database_username, + 'password': matrix_go_skype_bridge_database_password, + }] if (matrix_go_skype_bridge_enabled and matrix_go_skype_bridge_database_engine == 'postgres' and matrix_go_skype_bridge_database_hostname == 'matrix-postgres') else []) + + ([{ 'name': matrix_mautrix_facebook_database_name, 'username': matrix_mautrix_facebook_database_username, @@ -2152,7 +2192,7 @@ matrix_synapse_admin_enabled: false # Synapse Admin's HTTP port to the local host. matrix_synapse_admin_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8766' }}" -matrix_synapse_admin_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" +matrix_synapse_admin_container_image_self_build: "{{ matrix_architecture not in ['arm64', 'amd64'] }}" ###################################################################### # diff --git a/roles/matrix-base/defaults/main.yml b/roles/matrix-base/defaults/main.yml index 64556378..6b717f80 100644 --- a/roles/matrix-base/defaults/main.yml +++ b/roles/matrix-base/defaults/main.yml @@ -8,6 +8,10 @@ # Example value: example.com matrix_domain: ~ +# The optional matrix admin MXID, used in bridges' configs to set bridge admin user +# Example value: "@someone:{{ matrix_domain }}" +matrix_admin: '' + # This will contain the homeserver implementation that is in use. # Valid values: synapse, dendrite # diff --git a/roles/matrix-bot-buscarron/defaults/main.yml b/roles/matrix-bot-buscarron/defaults/main.yml index c2b44fa1..56686f42 100644 --- a/roles/matrix-bot-buscarron/defaults/main.yml +++ b/roles/matrix-bot-buscarron/defaults/main.yml @@ -8,7 +8,7 @@ matrix_bot_buscarron_container_image_self_build: false matrix_bot_buscarron_docker_repo: "https://gitlab.com/etke.cc/buscarron.git" matrix_bot_buscarron_docker_src_files_path: "{{ matrix_base_data_path }}/buscarron/docker-src" -matrix_bot_buscarron_version: v1.1.0 +matrix_bot_buscarron_version: v1.2.0 matrix_bot_buscarron_docker_image: "{{ matrix_bot_buscarron_docker_image_name_prefix }}buscarron:{{ matrix_bot_buscarron_version }}" matrix_bot_buscarron_docker_image_name_prefix: "{{ 'localhost/' if matrix_bot_buscarron_container_image_self_build else 'registry.gitlab.com/etke.cc/' }}" matrix_bot_buscarron_docker_image_force_pull: "{{ matrix_bot_buscarron_docker_image.endswith(':latest') }}" @@ -76,6 +76,9 @@ matrix_bot_buscarron_homeserver: "{{ matrix_homeserver_container_url }}" # forms configuration matrix_bot_buscarron_forms: [] +# Disable encryption +matrix_bot_buscarron_noencryption: + # Sentry DSN matrix_bot_buscarron_sentry: @@ -88,6 +91,9 @@ matrix_bot_buscarron_spam_hosts: [] # spam email addresses matrix_bot_buscarron_spam_emails: [] +# spam email localparts +matrix_bot_buscarron_spam_localparts: [] + # Ban duration in hours matrix_bot_buscarron_ban_duration: 24 diff --git a/roles/matrix-bot-buscarron/templates/env.j2 b/roles/matrix-bot-buscarron/templates/env.j2 index 876072e1..42da0d34 100644 --- a/roles/matrix-bot-buscarron/templates/env.j2 +++ b/roles/matrix-bot-buscarron/templates/env.j2 @@ -5,6 +5,7 @@ BUSCARRON_DB_DSN={{ matrix_bot_buscarron_database_connection_string }} BUSCARRON_DB_DIALECT={{ matrix_bot_buscarron_database_dialect }} BUSCARRON_SPAM_HOSTS={{ matrix_bot_buscarron_spam_hosts|join(" ") }} BUSCARRON_SPAM_EMAILS={{ matrix_bot_buscarron_spam_emails|join(" ") }} +BUSCARRON_SPAM_LOCALPARTS={{ matrix_bot_buscarron_spam_localparts|join(" ") }} BUSCARRON_SENTRY={{ matrix_bot_buscarron_sentry }} BUSCARRON_LOGLEVEL={{ matrix_bot_buscarron_loglevel }} BUSCARRON_BAN_DURATION={{ matrix_bot_buscarron_ban_duration }} @@ -12,10 +13,12 @@ BUSCARRON_BAN_SIZE={{ matrix_bot_buscarron_ban_size }} BUSCARRON_PM_TOKEN={{ matrix_bot_buscarron_pm_token }} BUSCARRON_PM_FROM={{ matrix_bot_buscarron_pm_from }} BUSCARRON_PM_REPLYTO={{ matrix_bot_buscarron_pm_replyto }} +BUSCARRON_NOENCRYPTION={{ matrix_bot_buscarron_noencryption }} {% set forms = [] %} {% for form in matrix_bot_buscarron_forms -%}{{- forms.append(form.name) -}} BUSCARRON_{{ form.name|upper }}_ROOM={{ form.room|default('') }} BUSCARRON_{{ form.name|upper }}_REDIRECT={{ form.redirect|default('') }} +BUSCARRON_{{ form.name|upper }}_HASDOMAIN={{ form.hasdomain|default('') }} BUSCARRON_{{ form.name|upper }}_RATELIMIT={{ form.ratelimit|default('') }} BUSCARRON_{{ form.name|upper }}_EXTENSIONS={{ form.extensions|default('')|join(' ') }} BUSCARRON_{{ form.name|upper }}_CONFIRMATION_SUBJECT={{ form.confirmation_subject|default('') }} diff --git a/roles/matrix-bridge-beeper-linkedin/defaults/main.yml b/roles/matrix-bridge-beeper-linkedin/defaults/main.yml index e622522d..5b84643c 100644 --- a/roles/matrix-bridge-beeper-linkedin/defaults/main.yml +++ b/roles/matrix-bridge-beeper-linkedin/defaults/main.yml @@ -25,6 +25,8 @@ matrix_beeper_linkedin_homeserver_address: "{{ matrix_homeserver_container_url } matrix_beeper_linkedin_homeserver_domain: "{{ matrix_domain }}" matrix_beeper_linkedin_appservice_address: "http://matrix-beeper-linkedin:29319" +matrix_beeper_linkedin_bridge_presence: true + # A list of extra arguments to pass to the container matrix_beeper_linkedin_container_extra_arguments: [] diff --git a/roles/matrix-bridge-beeper-linkedin/templates/config.yaml.j2 b/roles/matrix-bridge-beeper-linkedin/templates/config.yaml.j2 index 4fb6b055..6b33ffea 100644 --- a/roles/matrix-bridge-beeper-linkedin/templates/config.yaml.j2 +++ b/roles/matrix-bridge-beeper-linkedin/templates/config.yaml.j2 @@ -238,6 +238,9 @@ bridge: # mxid - Specific user permissions: "{{ matrix_beeper_linkedin_homeserver_domain }}": user + {% if matrix_admin %} + "{{ matrix_admin }}": admin + {% endif %} @@ -256,12 +259,12 @@ logging: formatter: colored loggers: mau: - level: DEBUG + level: WARNING paho: - level: INFO + level: WARNING aiohttp: - level: INFO + level: WARNING root: - level: DEBUG + level: WARNING handlers: [ console] diff --git a/roles/matrix-bridge-go-skype-bridge/defaults/main.yml b/roles/matrix-bridge-go-skype-bridge/defaults/main.yml new file mode 100644 index 00000000..95213a00 --- /dev/null +++ b/roles/matrix-bridge-go-skype-bridge/defaults/main.yml @@ -0,0 +1,132 @@ +--- +# Go Skype Bridge is a Matrix <-> Skype bridge +# See: https://github.com/kelaresg/go-skype-bridge + +matrix_go_skype_bridge_enabled: true + +matrix_go_skype_bridge_container_image_self_build: true +matrix_go_skype_bridge_container_image_self_build_repo: "https://github.com/kelaresg/go-skype-bridge.git" +matrix_go_skype_bridge_container_image_self_build_branch: "{{ 'master' if matrix_go_skype_bridge_version == 'latest' else matrix_go_skype_bridge_version }}" + +matrix_go_skype_bridge_version: latest +matrix_go_skype_bridge_docker_image: "{{ matrix_go_skype_bridge_docker_image_name_prefix }}kelaresg/go-skype-bridge:{{ matrix_go_skype_bridge_version }}" +matrix_go_skype_bridge_docker_image_name_prefix: "localhost/" +matrix_go_skype_bridge_docker_image_force_pull: "{{ matrix_go_skype_bridge_docker_image.endswith(':latest') }}" + +matrix_go_skype_bridge_base_path: "{{ matrix_base_data_path }}/go-skype-bridge" +matrix_go_skype_bridge_config_path: "{{ matrix_go_skype_bridge_base_path }}/config" +matrix_go_skype_bridge_data_path: "{{ matrix_go_skype_bridge_base_path }}/data" +matrix_go_skype_bridge_docker_src_files_path: "{{ matrix_go_skype_bridge_base_path }}/docker-src" + +matrix_go_skype_bridge_homeserver_address: "{{ matrix_homeserver_container_url }}" +matrix_go_skype_bridge_homeserver_domain: "{{ matrix_domain }}" +matrix_go_skype_bridge_appservice_address: 'http://matrix-go-skype-bridge:8080' + +# A list of extra arguments to pass to the container +matrix_go_skype_bridge_container_extra_arguments: [] + +# List of systemd services that matrix-go-skype-bridge.service depends on. +matrix_go_skype_bridge_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-go-skype-bridge.service wants +matrix_go_skype_bridge_systemd_wanted_services_list: [] + +matrix_go_skype_bridge_appservice_token: '' +matrix_go_skype_bridge_homeserver_token: '' + +matrix_go_skype_bridge_appservice_bot_username: skypebridgebot + +# Whether or not created rooms should have federation enabled. +# If false, created portal rooms will never be federated. +matrix_go_skype_bridge_federate_rooms: true + +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_go_skype_bridge_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_go_skype_bridge_database_*` variables +matrix_go_skype_bridge_database_engine: 'sqlite' + +matrix_go_skype_bridge_sqlite_database_path_local: "{{ matrix_go_skype_bridge_data_path }}/go-skype-bridge.db" +matrix_go_skype_bridge_sqlite_database_path_in_container: "/data/go-skype-bridge.db" + +matrix_go_skype_bridge_database_username: 'matrix_go_skype_bridge' +matrix_go_skype_bridge_database_password: 'some-password' +matrix_go_skype_bridge_database_hostname: 'matrix-postgres' +matrix_go_skype_bridge_database_port: 5432 +matrix_go_skype_bridge_database_name: 'matrix_go_skype_bridge' + +matrix_go_skype_bridge_database_connection_string: 'postgresql://{{ matrix_go_skype_bridge_database_username }}:{{ matrix_go_skype_bridge_database_password }}@{{ matrix_go_skype_bridge_database_hostname }}:{{ matrix_go_skype_bridge_database_port }}/{{ matrix_go_skype_bridge_database_name }}?sslmode=disable' + +matrix_go_skype_bridge_appservice_database_type: "{{ + { + 'sqlite': 'sqlite3', + 'postgres':'postgres', + }[matrix_go_skype_bridge_database_engine] +}}" + +matrix_go_skype_bridge_appservice_database_uri: "{{ + { + 'sqlite': matrix_go_skype_bridge_sqlite_database_path_in_container, + 'postgres': matrix_go_skype_bridge_database_connection_string, + }[matrix_go_skype_bridge_database_engine] +}}" + +# Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). +matrix_go_skype_bridge_login_shared_secret: '' +matrix_go_skype_bridge_bridge_login_shared_secret_map: + "{{ {matrix_go_skype_bridge_homeserver_domain: matrix_go_skype_bridge_login_shared_secret} if matrix_go_skype_bridge_login_shared_secret else {} }}" + +# Servers to always allow double puppeting from +matrix_go_skype_bridge_bridge_double_puppet_server_map: + "{{ matrix_go_skype_bridge_homeserver_domain : matrix_go_skype_bridge_homeserver_address }}" + +# Default go-skype-bridge configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. +# +# For a more advanced customization, you can extend the default (see `matrix_go_skype_bridge_configuration_extension_yaml`) +# or completely replace this variable with your own template. +matrix_go_skype_bridge_configuration_yaml: "{{ lookup('template', 'templates/config.yaml.j2') }}" + +matrix_go_skype_bridge_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_go_skype_bridge_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_go_skype_bridge_configuration_yaml`. + +matrix_go_skype_bridge_configuration_extension: "{{ matrix_go_skype_bridge_configuration_extension_yaml|from_yaml if matrix_go_skype_bridge_configuration_extension_yaml|from_yaml is mapping else {} }}" + +# Holds the final configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_go_skype_bridge_configuration_yaml`. +matrix_go_skype_bridge_configuration: "{{ matrix_go_skype_bridge_configuration_yaml|from_yaml|combine(matrix_go_skype_bridge_configuration_extension, recursive=True) }}" + +matrix_go_skype_bridge_registration_yaml: | + id: skype + url: {{ matrix_go_skype_bridge_appservice_address }} + as_token: "{{ matrix_go_skype_bridge_appservice_token }}" + hs_token: "{{ matrix_go_skype_bridge_homeserver_token }}" + # See https://github.com/mautrix/signal/issues/43 + sender_localpart: _bot_{{ matrix_go_skype_bridge_appservice_bot_username }} + rate_limited: false + namespaces: + users: + - regex: '^@skype-(.*):{{ matrix_go_skype_bridge_homeserver_domain|regex_escape }}$' + exclusive: true + - exclusive: true + regex: '^@{{ matrix_go_skype_bridge_appservice_bot_username|regex_escape }}:{{ matrix_go_skype_bridge_homeserver_domain|regex_escape }}$' + de.sorunome.msc2409.push_ephemeral: true + +matrix_go_skype_bridge_registration: "{{ matrix_go_skype_bridge_registration_yaml|from_yaml }}" + +# Enable End-to-bridge encryption +matrix_go_skype_bridge_bridge_encryption_allow: false +matrix_go_skype_bridge_bridge_encryption_default: "{{ matrix_go_skype_bridge_bridge_encryption_allow }}" + +# Minimum severity of journal log messages. +# Options: debug, info, warn, error, fatal +matrix_go_skype_bridge_log_level: 'warn' diff --git a/roles/matrix-bridge-go-skype-bridge/tasks/init.yml b/roles/matrix-bridge-go-skype-bridge/tasks/init.yml new file mode 100644 index 00000000..452d835b --- /dev/null +++ b/roles/matrix-bridge-go-skype-bridge/tasks/init.yml @@ -0,0 +1,21 @@ +--- +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-go-skype-bridge.service'] }}" + when: matrix_go_skype_bridge_enabled|bool + +# If the matrix-synapse role is not used, these variables may not exist. +- set_fact: + matrix_synapse_container_extra_arguments: > + {{ + matrix_synapse_container_extra_arguments|default([]) + + + ["--mount type=bind,src={{ matrix_go_skype_bridge_config_path }}/registration.yaml,dst=/matrix-go-skype-bridge-registration.yaml,ro"] + }} + + matrix_synapse_app_service_config_files: > + {{ + matrix_synapse_app_service_config_files|default([]) + + + ["/matrix-go-skype-bridge-registration.yaml"] + }} + when: matrix_go_skype_bridge_enabled|bool diff --git a/roles/matrix-bridge-go-skype-bridge/tasks/main.yml b/roles/matrix-bridge-go-skype-bridge/tasks/main.yml new file mode 100644 index 00000000..456ea53c --- /dev/null +++ b/roles/matrix-bridge-go-skype-bridge/tasks/main.yml @@ -0,0 +1,23 @@ +--- + +- import_tasks: "{{ role_path }}/tasks/init.yml" + tags: + - always + +- import_tasks: "{{ role_path }}/tasks/validate_config.yml" + when: "run_setup|bool and matrix_go_skype_bridge_enabled|bool" + tags: + - setup-all + - setup-go-skype-bridge + +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: "run_setup|bool and matrix_go_skype_bridge_enabled|bool" + tags: + - setup-all + - setup-go-skype-bridge + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: "run_setup|bool and not matrix_go_skype_bridge_enabled|bool" + tags: + - setup-all + - setup-go-skype-bridge diff --git a/roles/matrix-bridge-go-skype-bridge/tasks/setup_install.yml b/roles/matrix-bridge-go-skype-bridge/tasks/setup_install.yml new file mode 100644 index 00000000..c884625b --- /dev/null +++ b/roles/matrix-bridge-go-skype-bridge/tasks/setup_install.yml @@ -0,0 +1,147 @@ +--- + +# If the matrix-synapse role is not used, `matrix_synapse_role_executed` won't exist. +# We don't want to fail in such cases. +- name: Fail if matrix-synapse role already executed + fail: + msg: >- + The matrix-bridge-go-skype-bridge role needs to execute before the matrix-synapse role. + when: "matrix_synapse_role_executed|default(False)" + +- set_fact: + matrix_go_skype_bridge_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_go_skype_bridge_sqlite_database_path_local }}" + register: matrix_go_skype_bridge_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_go_skype_bridge_sqlite_database_path_local }}" + dst: "{{ matrix_go_skype_bridge_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_go_skype_bridge_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-go-skype-bridge.service'] + pgloader_options: ['--with "quote identifiers"'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_go_skype_bridge_requires_restart: true + when: "matrix_go_skype_bridge_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_go_skype_bridge_database_engine == 'postgres'" + + +- name: Ensure Go Skype Bridge paths exists + file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - {path: "{{ matrix_go_skype_bridge_base_path }}", when: true} + - {path: "{{ matrix_go_skype_bridge_config_path }}", when: true} + - {path: "{{ matrix_go_skype_bridge_data_path }}", when: true} + - {path: "{{ matrix_go_skype_bridge_docker_src_files_path }}", when: "{{ matrix_go_skype_bridge_container_image_self_build }}"} + when: item.when|bool + +- name: Ensure Go Skype Bridge image is pulled + docker_image: + name: "{{ matrix_go_skype_bridge_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_go_skype_bridge_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_go_skype_bridge_docker_image_force_pull }}" + when: not matrix_go_skype_bridge_container_image_self_build + register: result + retries: "{{ matrix_container_retries_count }}" + delay: "{{ matrix_container_retries_delay }}" + until: result is not failed + +- name: Ensure Go Skype Bridge repository is present on self-build + git: + repo: "{{ matrix_go_skype_bridge_container_image_self_build_repo }}" + dest: "{{ matrix_go_skype_bridge_docker_src_files_path }}" + version: "{{ matrix_go_skype_bridge_container_image_self_build_branch }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_go_skype_bridge_git_pull_results + when: "matrix_go_skype_bridge_container_image_self_build|bool" + +- name: Ensure Go Skype Bridge Docker image is built + docker_image: + name: "{{ matrix_go_skype_bridge_docker_image }}" + source: build + force_source: "{{ matrix_go_skype_bridge_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_go_skype_bridge_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_go_skype_bridge_docker_src_files_path }}" + pull: true + when: "matrix_go_skype_bridge_container_image_self_build|bool" + +- name: Check if an old database file exists + stat: + path: "{{ matrix_go_skype_bridge_base_path }}/go-skype-bridge.db" + register: matrix_go_skype_bridge_stat_database + +- name: Check if an old matrix state file exists + stat: + path: "{{ matrix_go_skype_bridge_base_path }}/mx-state.json" + register: matrix_go_skype_bridge_stat_mx_state + +- name: (Data relocation) Ensure matrix-go-skype-bridge.service is stopped + service: + name: matrix-go-skype-bridge + state: stopped + enabled: false + daemon_reload: true + failed_when: false + when: "matrix_go_skype_bridge_stat_database.stat.exists" + +- name: (Data relocation) Move go-skype-bridge database file to ./data directory + command: "mv {{ matrix_go_skype_bridge_base_path }}/go-skype-bridge.db {{ matrix_go_skype_bridge_data_path }}/go-skype-bridge.db" + when: "matrix_go_skype_bridge_stat_database.stat.exists" + +- name: (Data relocation) Move go-skype-bridge mx-state file to ./data directory + command: "mv {{ matrix_go_skype_bridge_base_path }}/mx-state.json {{ matrix_go_skype_bridge_data_path }}/mx-state.json" + when: "matrix_go_skype_bridge_stat_mx_state.stat.exists" + +- name: Ensure go-skype-bridge config.yaml installed + copy: + content: "{{ matrix_go_skype_bridge_configuration|to_nice_yaml(indent=2, width=999999) }}" + dest: "{{ matrix_go_skype_bridge_config_path }}/config.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure go-skype-bridge registration.yaml installed + copy: + content: "{{ matrix_go_skype_bridge_registration|to_nice_yaml(indent=2, width=999999) }}" + dest: "{{ matrix_go_skype_bridge_config_path }}/registration.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-go-skype-bridge.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-go-skype-bridge.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-go-skype-bridge.service" + mode: 0644 + register: matrix_go_skype_bridge_systemd_service_result + +- name: Ensure systemd reloaded after matrix-go-skype-bridge.service installation + service: + daemon_reload: true + when: "matrix_go_skype_bridge_systemd_service_result.changed" + +- name: Ensure matrix-go-skype-bridge.service restarted, if necessary + service: + name: "matrix-go-skype-bridge.service" + state: restarted + when: "matrix_go_skype_bridge_requires_restart|bool" diff --git a/roles/matrix-bridge-go-skype-bridge/tasks/setup_uninstall.yml b/roles/matrix-bridge-go-skype-bridge/tasks/setup_uninstall.yml new file mode 100644 index 00000000..19acff85 --- /dev/null +++ b/roles/matrix-bridge-go-skype-bridge/tasks/setup_uninstall.yml @@ -0,0 +1,25 @@ +--- + +- name: Check existence of matrix-go-skype-bridge service + stat: + path: "/etc/systemd/system/matrix-go-skype-bridge.service" + register: matrix_go_skype_bridge_service_stat + +- name: Ensure matrix-go-skype-bridge is stopped + service: + name: matrix-go-skype-bridge + state: stopped + enabled: false + daemon_reload: true + when: "matrix_go_skype_bridge_service_stat.stat.exists" + +- name: Ensure matrix-go-skype-bridge.service doesn't exist + file: + path: "/etc/systemd/system/matrix-go-skype-bridge.service" + state: absent + when: "matrix_go_skype_bridge_service_stat.stat.exists" + +- name: Ensure systemd reloaded after matrix-go-skype-bridge.service removal + service: + daemon_reload: true + when: "matrix_go_skype_bridge_service_stat.stat.exists" diff --git a/roles/matrix-bridge-go-skype-bridge/tasks/validate_config.yml b/roles/matrix-bridge-go-skype-bridge/tasks/validate_config.yml new file mode 100644 index 00000000..7e9f3daf --- /dev/null +++ b/roles/matrix-bridge-go-skype-bridge/tasks/validate_config.yml @@ -0,0 +1,10 @@ +--- + +- name: Fail if required settings not defined + fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - "matrix_go_skype_bridge_appservice_token" + - "matrix_go_skype_bridge_homeserver_token" diff --git a/roles/matrix-bridge-go-skype-bridge/templates/config.yaml.j2 b/roles/matrix-bridge-go-skype-bridge/templates/config.yaml.j2 new file mode 100644 index 00000000..6f7277fe --- /dev/null +++ b/roles/matrix-bridge-go-skype-bridge/templates/config.yaml.j2 @@ -0,0 +1,241 @@ +#jinja2: lstrip_blocks: "True" +# Homeserver details. +homeserver: + # The address that this appservice can use to connect to the homeserver. + address: {{ matrix_go_skype_bridge_homeserver_address }} + # The domain of the homeserver (for MXIDs, etc). + domain: {{ matrix_go_skype_bridge_homeserver_domain }} + # If you don’t know what this is, no need to modify(for parse "mention user/reply message, etc") + server_name: matrix.to + +# Application service host/registration related details. +# Changing these values requires regeneration of the registration. +appservice: + # The address that the homeserver can use to connect to this appservice. + address: {{ matrix_go_skype_bridge_appservice_address }} + + # The hostname and port where this appservice should listen. + hostname: 0.0.0.0 + port: 8080 + + # Database config. + database: + # The database type. "sqlite3" and "postgres" are supported. + type: {{ matrix_go_skype_bridge_appservice_database_type|to_json }} + # The database URI. + # SQLite: File name is enough. https://github.com/mattn/go-sqlite3#connection-string + # Postgres: Connection string. For example, postgres://user:password@host/database?sslmode=disable + uri: {{ matrix_go_skype_bridge_appservice_database_uri|to_json }} + # Maximum number of connections. Mostly relevant for Postgres. + max_open_conns: 20 + max_idle_conns: 2 + + # Settings for provisioning API + provisioning: + # Prefix for the provisioning API paths. + prefix: /_matrix/provision/v1 + # Shared secret for authentication. If set to "disable", the provisioning API will be disabled. + shared_secret: disable + + # The unique ID of this appservice. + id: skype + # Appservice bot details. + bot: + # Username of the appservice bot. + username: skypebridgebot + # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty + # to leave display name/avatar as-is. + displayname: Skype bridge bot + avatar: mxc://matrix.org/kGQUDQyPiwbRXPFkjoBrPyhC + + # Authentication tokens for AS <-> HS communication. Autogenerated; do not modify. + as_token: "{{ matrix_go_skype_bridge_appservice_token }}" + hs_token: "{{ matrix_go_skype_bridge_homeserver_token }}" + +# Bridge config +bridge: + # Localpart template of MXIDs for Skype users. + # {{ '{{.}}' }} is replaced with the phone number of the Skype user. + username_template: {{ 'skype-{{.}}' }} + # Displayname template for Skype users. + # {{ '{{.Notify}}' }} - nickname set by the Skype user + # {{ '{{.Jid}}' }} - phone number (international format) + # The following variables are also available, but will cause problems on multi-user instances: + # {{ '{{.Name}}' }} - display name from contact list + # {{ '{{.Short}}' }} - short display name from contact list + # To use multiple if's, you need to use: {{ '{{else if .Name}}' }}, for example: + # "{{ '{{if .Notify}}' }}{{ '{{.Notify}}' }}{{ '{{else if .Name}}' }}{{ '{{.Name}}' }}{{ '{{else}}' }}{{ '{{.Jid}}' }}{{ '{{end}}' }} (WA)" + displayname_template: "{{ '{{if .DisplayName}}' }}{{ '{{.DisplayName}}' }}{{ '{{else}}' }}{{ '{{.PersonId}}' }}{{ '{{end}}' }} (Skype)" + # Localpart template for per-user room grouping community IDs. + # On startup, the bridge will try to create these communities, add all of the specific user's + # portals to the community, and invite the Matrix user to it. + # (Note that, by default, non-admins might not have your homeserver's permission to create + # communities.) + # {{ '{{.Localpart}}' }} is the MXID localpart and {{ '{{.Server}}' }} is the MXID server part of the user. + community_template: skype-{{ '{{.Localpart}}' }}={{ '{{.Server}}' }} + + # Skype connection timeout in seconds. + connection_timeout: 20 + # If Skype doesn't respond within connection_timeout, should the bridge try to fetch the message + # to see if it was actually bridged? Use this if you have problems with sends timing out but actually + # succeeding. + fetch_message_on_timeout: false + # Whether or not the bridge should send a read receipt from the bridge bot when a message has been + # sent to Skype. If fetch_message_on_timeout is enabled, a successful post-timeout fetch will + # trigger a read receipt too. + delivery_receipts: false + # Number of times to regenerate QR code when logging in. + # The regenerated QR code is sent as an edit and essentially multiplies the login timeout (20 seconds) + login_qr_regen_count: 2 + # Maximum number of times to retry connecting on connection error. + max_connection_attempts: 3 + # Number of seconds to wait between connection attempts. + # Negative numbers are exponential backoff: -connection_retry_delay + 1 + 2^attempts + connection_retry_delay: -1 + # Whether or not the bridge should send a notice to the user's management room when it retries connecting. + # If false, it will only report when it stops retrying. + report_connection_retry: true + # Maximum number of seconds to wait for chats to be sent at startup. + # If this is too low and you have lots of chats, it could cause backfilling to fail. + chat_list_wait: 30 + # Maximum number of seconds to wait to sync portals before force unlocking message processing. + # If this is too low and you have lots of chats, it could cause backfilling to fail. + portal_sync_wait: 600 + + # Whether or not to send call start/end notices to Matrix. + call_notices: + start: true + end: true + + # Number of chats to sync for new users. + # Since some of the obtained conversations are not the conversations that the user needs to see, + # the actual number of conversations displayed on the matrix client will be slightly less than the set value + initial_chat_sync_count: 10 + # Number of old messages to fill when creating new portal rooms. + initial_history_fill_count: 20 + # Whether or not notifications should be turned off while filling initial history. + # Only applicable when using double puppeting. + initial_history_disable_notifications: false + # Maximum number of chats to sync when recovering from downtime. + # Set to -1 to sync all new chats during downtime. + recovery_chat_sync_limit: -1 + # Whether or not to sync history when recovering from downtime. + recovery_history_backfill: true + # Maximum number of seconds since last message in chat to skip + # syncing the chat in any case. This setting will take priority + # over both recovery_chat_sync_limit and initial_chat_sync_count. + # Default is 3 days = 259200 seconds + sync_max_chat_age: 259200 + + # sync contact, Non-martix-standard parameter, defaults to false + sync_contact: false + + # Whether or not to sync with custom puppets to receive EDUs that + # are not normally sent to appservices. + sync_with_custom_puppets: true + + # Servers to always allow double puppeting from + double_puppet_server_map: + "{{ matrix_go_skype_bridge_homeserver_domain }}": {{ matrix_go_skype_bridge_homeserver_address }} + # Allow using double puppeting from any server with a valid client .well-known file. + double_puppet_allow_discovery: false + # Shared secret for https://github.com/devture/matrix-synapse-shared-secret-auth + # + # If set, custom puppets will be enabled automatically for local users + # instead of users having to find an access token and run `login-matrix` + # manually. + login_shared_secret_map: {{ matrix_go_skype_bridge_bridge_login_shared_secret_map|to_json }} + + # Whether or not to invite own Skype user's Matrix puppet into private + # chat portals when backfilling if needed. + # This always uses the default puppet instead of custom puppets due to + # rate limits and timestamp massaging. + invite_own_puppet_for_backfilling: true + # Whether or not to explicitly set the avatar and room name for private + # chat portal rooms. This can be useful if the previous field works fine, + # but causes room avatar/name bugs. + private_chat_portal_meta: true + + # Whether or not thumbnails from Skype should be sent. + # They're disabled by default due to very low resolution. + Skype_thumbnail: false + + # Allow invite permission for user. User can invite any bots to room with Skype + # users (private chat and groups) + allow_user_invite: false + + # The prefix for commands. Only required in non-management rooms. + command_prefix: "!wa" + + # End-to-bridge encryption support options. This requires login_shared_secret to be configured + # in order to get a device for the bridge bot. + # + # Additionally, https://github.com/matrix-org/synapse/pull/5758 is required if using a normal + # application service. + encryption: + # Allow encryption, work in group chat rooms with e2ee enabled + allow: {{ matrix_go_skype_bridge_bridge_encryption_allow|to_json }} + # Default to encryption, force-enable encryption in all portals the bridge creates + # This will cause the bridge bot to be in private chats for the encryption to work properly. + # It is recommended to also set private_chat_portal_meta to true when using this. + default: {{ matrix_go_skype_bridge_bridge_encryption_default|to_json }} + + puppet_id: + # when set to true, the matrixid of the contact (puppet) from the bridge to the matrix will be encrypted into another string + allow: false + # 8 characters + key: '12dsf323' + # Use the username_template prefix. (Warning: At present, username_template cannot be too complicated, otherwise this function may cause unknown errors) + username_template_prefix: 'skype-' + + # Permissions for using the bridge. + # Permitted values: + # relaybot - Talk through the relaybot (if enabled), no access otherwise + # user - Access to use the bridge to chat with a Skype account. + # admin - User level and some additional administration tools + # Permitted keys: + # * - All Matrix users + # domain - All users on that homeserver + # mxid - Specific user + permissions: + "{{ matrix_go_skype_bridge_homeserver_domain }}": user + {% if matrix_admin %} + "{{ matrix_admin }}": admin + {% endif %} + + relaybot: + # Whether or not relaybot support is enabled. + enabled: false + # The management room for the bot. This is where all status notifications are posted and + # in this room, you can use `!wa ` instead of `!wa relaybot `. Omitting + # the command prefix completely like in user management rooms is not possible. + management: '!foo:example.com' + # List of users to invite to all created rooms that include the relaybot. + invites: [] + # The formats to use when sending messages to Skype via the relaybot. + message_formats: + m.text: "{{ '{{ .Sender.Displayname }}' }}: {{ '{{ .Message }}' }}" + m.notice: "{{ '{{ .Sender.Displayname }}' }}:: {{ '{{ .Message }}' }}" + m.emote: "* {{ '{{ .Sender.Displayname }}' }}: {{ '{{ .Message }}' }}" + m.file: "{{ '{{ .Sender.Displayname }}' }}: sent a file" + m.image: "{{ '{{ .Sender.Displayname }}' }}: sent an image" + m.audio: "{{ '{{ .Sender.Displayname }}' }}: sent an audio file" + m.video: "{{ '{{ .Sender.Displayname }}' }}: sent a video" + m.location: "{{ '{{ .Sender.Displayname }}' }}: sent a location" + +# Logging config. +logging: + # The directory for log files. Will be created if not found. + directory: ./logs + # Available variables: .Date for the file date and .Index for different log files on the same day. + # empy/null = journal logging only + file_name_format: + # Date format for file names in the Go time format: https://golang.org/pkg/time/#pkg-constants + file_date_format: "2006-01-02" + # Log file permissions. + file_mode: 0600 + # Timestamp format for log entries in the Go time format. + timestamp_format: "Jan _2, 2006 15:04:05" + # Minimum severity for log messages. + # Options: debug, info, warn, error, fatal + print_level: {{ matrix_go_skype_bridge_log_level }} diff --git a/roles/matrix-bridge-go-skype-bridge/templates/systemd/matrix-go-skype-bridge.service.j2 b/roles/matrix-bridge-go-skype-bridge/templates/systemd/matrix-go-skype-bridge.service.j2 new file mode 100644 index 00000000..fe5ab2d6 --- /dev/null +++ b/roles/matrix-bridge-go-skype-bridge/templates/systemd/matrix-go-skype-bridge.service.j2 @@ -0,0 +1,43 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Matrix Go Skype Bridge bridge +{% for service in matrix_go_skype_bridge_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_go_skype_bridge_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ matrix_systemd_unit_home_path }}" +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-go-skype-bridge 2>/dev/null || true' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-go-skype-bridge 2>/dev/null || true' + +# Intentional delay, so that the homeserver (we likely depend on) can manage to start. +ExecStartPre={{ matrix_host_command_sleep }} 5 + +ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-go-skype-bridge \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --network={{ matrix_docker_network }} \ + -v {{ matrix_go_skype_bridge_config_path }}:/config:z \ + -v {{ matrix_go_skype_bridge_data_path }}:/data:z \ + --workdir=/data \ + {% for arg in matrix_go_skype_bridge_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_go_skype_bridge_docker_image }} \ + /usr/bin/matrix-skype -c /config/config.yaml -r /config/registration.yaml + +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-go-skype-bridge 2>/dev/null || true' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-go-skype-bridge 2>/dev/null || true' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-go-skype-bridge + +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-bridge-hookshot/defaults/main.yml b/roles/matrix-bridge-hookshot/defaults/main.yml index 3a2d5bc9..5d618a6e 100644 --- a/roles/matrix-bridge-hookshot/defaults/main.yml +++ b/roles/matrix-bridge-hookshot/defaults/main.yml @@ -29,13 +29,20 @@ matrix_hookshot_public_endpoint: /hookshot matrix_hookshot_appservice_port: 9993 matrix_hookshot_appservice_endpoint: "{{ matrix_hookshot_public_endpoint }}/_matrix/app" -# Metrics work only in conjunction with matrix_synapse_metrics_enabled etc -matrix_hookshot_metrics_enabled: true +# Controls whether metrics are enabled in the bridge configuration. +# Enabling them is usually enough for a local (in-container) Prometheus to consume them. +# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_hookshot_metrics_proxying_enabled`. +matrix_hookshot_metrics_enabled: false + +# Controls whether Hookshot metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/hookshot`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_hookshot_metrics_proxying_enabled: false + # There is no need to edit ports. # Read the documentation to learn about using hookshot metrics with external Prometheus # If you still want something different, use matrix_hookshot_container_http_host_bind_ports below to expose ports instead. matrix_hookshot_metrics_port: 9001 -matrix_hookshot_metrics_endpoint: "{{ matrix_hookshot_public_endpoint }}/metrics" # There is no need to edit ports. use matrix_hookshot_container_http_host_bind_ports below to expose ports instead. matrix_hookshot_webhook_port: 9000 diff --git a/roles/matrix-bridge-hookshot/tasks/init.yml b/roles/matrix-bridge-hookshot/tasks/init.yml index 14bbcbb3..96d5740a 100644 --- a/roles/matrix-bridge-hookshot/tasks/init.yml +++ b/roles/matrix-bridge-hookshot/tasks/init.yml @@ -103,11 +103,10 @@ [matrix_hookshot_matrix_nginx_proxy_configuration] }} - - name: Generate Matrix hookshot proxying configuration for matrix-nginx-proxy + - name: Generate hookshot metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/hookshot) set_fact: - matrix_hookshot_matrix_nginx_proxy_metrics_configuration: | - {% if matrix_hookshot_metrics_enabled and matrix_hookshot_proxy_metrics %} - location {{ matrix_hookshot_metrics_endpoint }} { + matrix_hookshot_matrix_nginx_proxy_metrics_configuration_matrix_domain: | + location /metrics/hookshot { {% if matrix_nginx_proxy_enabled|default(False) %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; @@ -117,24 +116,18 @@ {# Generic configuration for use outside of our container setup #} proxy_pass http://127.0.0.1:{{ matrix_hookshot_metrics_port }}/metrics; {% endif %} - proxy_set_header Host $host; - {% if matrix_hookshot_proxy_metrics_basic_auth_enabled %} - auth_basic "protected"; - auth_basic_user_file /nginx-data/matrix-synapse-metrics-htpasswd; - {% endif %} } - {% endif %} + when: matrix_hookshot_metrics_enabled|bool and matrix_hookshot_metrics_proxying_enabled|bool - - name: Register hookshot metrics proxying configuration with matrix-nginx-proxy + - name: Register hookshot metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/hookshot) set_fact: - matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks: | + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | {{ - matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks|default([]) + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([]) + - [matrix_hookshot_matrix_nginx_proxy_metrics_configuration] + [matrix_hookshot_matrix_nginx_proxy_metrics_configuration_matrix_domain] }} - tags: - - always + when: matrix_hookshot_metrics_enabled|bool and matrix_hookshot_metrics_proxying_enabled|bool when: matrix_hookshot_enabled|bool - name: Warn about reverse-proxying if matrix-nginx-proxy not used diff --git a/roles/matrix-bridge-hookshot/tasks/validate_config.yml b/roles/matrix-bridge-hookshot/tasks/validate_config.yml index 5da8809e..b9622332 100644 --- a/roles/matrix-bridge-hookshot/tasks/validate_config.yml +++ b/roles/matrix-bridge-hookshot/tasks/validate_config.yml @@ -57,3 +57,16 @@ when: "matrix_hookshot_provisioning_enabled and vars[item] == ''" with_items: - "matrix_hookshot_provisioning_secret" + +- name: (Deprecation) Catch and report old metrics usage + fail: + msg: >- + Your configuration contains a variable (`{{ item }}`), which refers to the old metrics collection system for Hookshot, + which exposed metrics on `https://stats.DOMAIN/hookshot/metrics`. + + We now recommend exposing Hookshot metrics in another way, from another URL. + Refer to the changelog for more details: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#2022-06-22 + with_items: + - matrix_hookshot_proxy_metrics + - matrix_hookshot_metrics_endpoint + when: "item in vars" diff --git a/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 index c3cb1932..2555e985 100644 --- a/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 @@ -203,6 +203,9 @@ bridge: # mxid - Specific user permissions: '{{ matrix_mautrix_facebook_homeserver_domain }}': user + {% if matrix_admin %} + '{{ matrix_admin }}': admin + {% endif %} relay: # Whether relay mode should be allowed. If allowed, `!fb set-relay` can be used to turn any @@ -250,11 +253,11 @@ logging: formatter: colored loggers: mau: - level: DEBUG + level: WARNING paho: - level: INFO + level: WARNING aiohttp: - level: INFO + level: WARNING root: - level: DEBUG + level: WARNING handlers: [console] diff --git a/roles/matrix-bridge-mautrix-googlechat/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-googlechat/templates/config.yaml.j2 index e2af8830..db4394b7 100644 --- a/roles/matrix-bridge-mautrix-googlechat/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-googlechat/templates/config.yaml.j2 @@ -119,6 +119,9 @@ bridge: # mxid - Specific user permissions: '{{ matrix_mautrix_googlechat_homeserver_domain }}': user + {% if matrix_admin %} + '{{ matrix_admin }}': admin + {% endif %} # Python logging configuration. # @@ -138,11 +141,11 @@ logging: formatter: colored loggers: mau: - level: DEBUG + level: WARNING hangups: - level: DEBUG + level: WARNING aiohttp: - level: INFO + level: WARNING root: - level: DEBUG + level: WARNING handlers: [console] diff --git a/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 index 7ff7d539..07f5b2d7 100644 --- a/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 @@ -116,6 +116,9 @@ bridge: # mxid - Specific user permissions: '{{ matrix_mautrix_hangouts_homeserver_domain }}': user + {% if matrix_admin %} + '{{ matrix_admin }}': admin + {% endif %} # Python logging configuration. # @@ -135,11 +138,11 @@ logging: formatter: colored loggers: mau: - level: DEBUG + level: WARNING hangups: - level: DEBUG + level: WARNING aiohttp: - level: INFO + level: WARNING root: - level: DEBUG + level: WARNING handlers: [console] diff --git a/roles/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 index cb74d5c1..994a39a7 100644 --- a/roles/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 @@ -166,7 +166,7 @@ bridge: # been sent to Instagram. delivery_receipts: false # Whether or not delivery errors should be reported as messages in the Matrix room. - delivery_error_reports: false + delivery_error_reports: true # Set this to true to tell the bridge to re-send m.bridge events to all rooms on the next run. # This field will automatically be changed back to false after it, # except if the config file is not writable. @@ -187,6 +187,9 @@ bridge: # mxid - Specific user permissions: "{{ matrix_mautrix_instagram_homeserver_domain }}": user + {% if matrix_admin %} + "{{ matrix_admin }}": admin + {% endif %} # Provisioning API part of the web server for automated portal creation and fetching information. # Used by things like mautrix-manager (https://github.com/tulir/mautrix-manager). provisioning: @@ -216,13 +219,13 @@ logging: formatter: colored loggers: mau: - level: DEBUG + level: WARNING mauigpapi: - level: DEBUG + level: WARNING paho: - level: INFO + level: WARNING aiohttp: - level: INFO + level: WARNING root: - level: DEBUG + level: WARNING handlers: [console] diff --git a/roles/matrix-bridge-mautrix-signal/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-signal/templates/config.yaml.j2 index b831fe9a..53aa550d 100644 --- a/roles/matrix-bridge-mautrix-signal/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-signal/templates/config.yaml.j2 @@ -177,7 +177,7 @@ bridge: # Note that this is not related to Signal delivery receipts. delivery_receipts: false # Whether or not delivery errors should be reported as messages in the Matrix room. (not yet implemented) - delivery_error_reports: false + delivery_error_reports: true # Set this to true to tell the bridge to re-send m.bridge events to all rooms on the next run. # This field will automatically be changed back to false after it, # except if the config file is not writable. @@ -225,6 +225,9 @@ bridge: # mxid - Specific user permissions: {{ matrix_mautrix_signal_bridge_permissions|from_yaml }} + {% if matrix_admin %} + "{{ matrix_admin }}": admin + {% endif %} relay: # Whether or not relay mode should be allowed. If allowed, `!signal set-relay` can be used to turn any @@ -266,9 +269,9 @@ logging: formatter: colored loggers: mau: - level: {{ matrix_mautrix_signal_log_level }} + level: WARNING aiohttp: - level: INFO + level: WARNING root: - level: {{ matrix_mautrix_signal_log_level }} + level: WARNING handlers: [console] diff --git a/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 index 6569ce87..20055ab7 100644 --- a/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 @@ -105,11 +105,11 @@ bridge: # synced when they send messages. The maximum is 10000, after which the Telegram server # will not send any more members. # Defaults to no local limit (-> limited to 10000 by server) - max_initial_member_sync: -1 + max_initial_member_sync: 10 # Whether or not to sync the member list in channels. # If no channel admins have logged into the bridge, the bridge won't be able to sync the member # list regardless of this setting. - sync_channel_members: true + sync_channel_members: false # Whether or not to skip deleted members when syncing members. skip_deleted_members: true # Whether or not to automatically synchronize contacts and chats of Matrix users logged into @@ -204,7 +204,7 @@ bridge: # been sent to Telegram. delivery_receipts: false # Whether or not delivery errors should be reported as messages in the Matrix room. - delivery_error_reports: false + delivery_error_reports: true # Set this to true to tell the bridge to re-send m.bridge events to all rooms on the next run. # This field will automatically be changed back to false after it, # except if the config file is not writable. @@ -291,6 +291,9 @@ bridge: # mxid - Specific user permissions: '{{ matrix_mautrix_telegram_homeserver_domain }}': full + {% if matrix_admin %} + '{{ matrix_admin }}': admin + {% endif %} # Options related to the message relay Telegram bot. relaybot: @@ -401,11 +404,11 @@ logging: formatter: precise loggers: mau: - level: DEBUG + level: WARNING telethon: - level: DEBUG + level: WARNING aiohttp: - level: INFO + level: WARNING root: - level: DEBUG + level: WARNING handlers: [console] diff --git a/roles/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 index f0ae69b2..6b32d47b 100644 --- a/roles/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-twitter/templates/config.yaml.j2 @@ -149,7 +149,7 @@ bridge: # been sent to Twitter. delivery_receipts: false # Whether or not delivery errors should be reported as messages in the Matrix room. - delivery_error_reports: false + delivery_error_reports: true # Whether or not non-fatal polling errors should send notices to the notice room. temporary_disconnect_notices: true # Number of seconds to sleep more than the previous error when a polling error occurs. @@ -175,6 +175,9 @@ bridge: # mxid - Specific user permissions: '{{ matrix_mautrix_twitter_homeserver_domain }}': user + {% if matrix_admin %} + '{{ matrix_admin }}': admin + {% endif %} # Python logging configuration. @@ -195,9 +198,9 @@ logging: formatter: colored loggers: mau: - level: DEBUG + level: WARNING aiohttp: - level: INFO + level: WARNING root: - level: DEBUG + level: WARNING handlers: [console] diff --git a/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml index 5be70051..6e95eeb7 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -8,7 +8,7 @@ matrix_mautrix_whatsapp_container_image_self_build: false matrix_mautrix_whatsapp_container_image_self_build_repo: "https://mau.dev/mautrix/whatsapp.git" matrix_mautrix_whatsapp_container_image_self_build_branch: "{{ 'master' if matrix_mautrix_whatsapp_version == 'latest' else matrix_mautrix_whatsapp_version }}" -matrix_mautrix_whatsapp_version: v0.4.0 +matrix_mautrix_whatsapp_version: v0.5.0 # See: https://mau.dev/mautrix/whatsapp/container_registry matrix_mautrix_whatsapp_docker_image: "{{ matrix_mautrix_whatsapp_docker_image_name_prefix }}mautrix/whatsapp:{{ matrix_mautrix_whatsapp_version }}" matrix_mautrix_whatsapp_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_whatsapp_container_image_self_build else 'dock.mau.dev/' }}" diff --git a/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 index d73718ea..b9862e94 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 @@ -187,6 +187,9 @@ bridge: # mxid - Specific user permissions: "{{ matrix_mautrix_whatsapp_homeserver_domain }}": user + {% if matrix_admin %} + "{{ matrix_admin }}": admin + {% endif %} # Settings for relay mode relay: diff --git a/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 index edb0c280..a1c0cc48 100644 --- a/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 @@ -25,7 +25,7 @@ presence: # Bridge Discord online/offline status enabled: true # How often to send status to the homeserver in milliseconds - interval: 500 + interval: 5000 provisioning: # Regex of Matrix IDs allowed to use the puppet bridge @@ -117,7 +117,7 @@ logging: # Log level of console output # Allowed values starting with most verbose: # silly, debug, verbose, info, warn, error - console: info + console: warn # Date and time formatting lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files diff --git a/roles/matrix-bridge-mx-puppet-groupme/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-groupme/templates/config.yaml.j2 index a9ab7701..357baec7 100644 --- a/roles/matrix-bridge-mx-puppet-groupme/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-groupme/templates/config.yaml.j2 @@ -78,7 +78,7 @@ logging: # Log level of console output # Allowed values starting with most verbose: # silly, debug, verbose, info, warn, error - console: info + console: warn # Date and time formatting lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files diff --git a/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 index 1c4bb1bd..57c5b0f0 100644 --- a/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 @@ -18,7 +18,7 @@ presence: # Bridge Instagram online/offline status enabled: true # How often to send status to the homeserver in milliseconds - interval: 500 + interval: 5000 provisioning: # Regex of Matrix IDs allowed to use the puppet bridge @@ -61,7 +61,7 @@ logging: # Log level of console output # Allowed values starting with most verbose: # silly, debug, verbose, info, warn, error - console: info + console: warn # Date and time formatting lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files diff --git a/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 index 1d6d4828..647f8fc7 100644 --- a/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 @@ -29,7 +29,7 @@ logging: # Log level of console output # Allowed values starting with most verbose: # silly, debug, verbose, info, warn, error - console: info + console: warn # Optionally, you can apply filters to the console logging #console: # level: info @@ -80,7 +80,7 @@ presence: # Bridge online/offline status enabled: true # How often to send status to the homeserver in milliseconds - interval: 500 + interval: 5000 # if the im.vector.user_status state setting should be diabled #disableStatusState: false # A blacklist of remote user IDs for the im.vector.user_status state setting diff --git a/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 index c7497a84..b84fe221 100644 --- a/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 @@ -32,7 +32,7 @@ presence: # Bridge Discord online/offline status enabled: true # How often to send status to the homeserver in milliseconds - interval: 500 + interval: 5000 provisioning: # Regex of Matrix IDs allowed to use the puppet bridge @@ -75,7 +75,7 @@ logging: # Log level of console output # Allowed values starting with most verbose: # silly, debug, verbose, info, warn, error - console: info + console: warn # Date and time formatting lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files diff --git a/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 index fd59471d..0919907d 100644 --- a/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 @@ -78,7 +78,7 @@ logging: # Log level of console output # Allowed values starting with most verbose: # silly, debug, verbose, info, warn, error - console: info + console: warn # Date and time formatting lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files diff --git a/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 index 1d269057..5418ccaf 100644 --- a/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 @@ -28,7 +28,7 @@ presence: # Bridge Twitter online/offline status enabled: true # How often to send status to the homeserver in milliseconds - interval: 500 + interval: 5000 provisioning: # Regex of Matrix IDs allowed to use the puppet bridge @@ -71,7 +71,7 @@ logging: # Log level of console output # Allowed values starting with most verbose: # silly, debug, verbose, info, warn, error - console: info + console: warn # Date and time formatting lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files diff --git a/roles/matrix-client-element/defaults/main.yml b/roles/matrix-client-element/defaults/main.yml index 86b0ee6a..083621f2 100644 --- a/roles/matrix-client-element/defaults/main.yml +++ b/roles/matrix-client-element/defaults/main.yml @@ -9,7 +9,7 @@ matrix_client_element_container_image_self_build_repo: "https://github.com/vecto # - https://github.com/vector-im/element-web/issues/19544 matrix_client_element_container_image_self_build_low_memory_system_patch_enabled: "{{ ansible_memtotal_mb < 4096 }}" -matrix_client_element_version: v1.10.14 +matrix_client_element_version: v1.10.15 matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:{{ matrix_client_element_version }}" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" diff --git a/roles/matrix-corporal/defaults/main.yml b/roles/matrix-corporal/defaults/main.yml index 66896e0e..bd91564a 100644 --- a/roles/matrix-corporal/defaults/main.yml +++ b/roles/matrix-corporal/defaults/main.yml @@ -23,7 +23,7 @@ matrix_corporal_container_extra_arguments: [] # List of systemd services that matrix-corporal.service depends on matrix_corporal_systemd_required_services_list: ['docker.service'] -matrix_corporal_version: 2.2.3 +matrix_corporal_version: 2.3.0 matrix_corporal_docker_image: "{{ matrix_corporal_docker_image_name_prefix }}devture/matrix-corporal:{{ matrix_corporal_docker_image_tag }}" matrix_corporal_docker_image_name_prefix: "{{ 'localhost/' if matrix_corporal_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_corporal_docker_image_tag: "{{ matrix_corporal_version }}" # for backward-compatibility diff --git a/roles/matrix-grafana/defaults/main.yml b/roles/matrix-grafana/defaults/main.yml index 991cb19d..618eaef7 100644 --- a/roles/matrix-grafana/defaults/main.yml +++ b/roles/matrix-grafana/defaults/main.yml @@ -4,7 +4,7 @@ matrix_grafana_enabled: false -matrix_grafana_version: 8.5.3 +matrix_grafana_version: 9.0.1 matrix_grafana_docker_image: "{{ matrix_container_global_registry_prefix }}grafana/grafana:{{ matrix_grafana_version }}" matrix_grafana_docker_image_force_pull: "{{ matrix_grafana_docker_image.endswith(':latest') }}" diff --git a/roles/matrix-mailer/defaults/main.yml b/roles/matrix-mailer/defaults/main.yml index 682126d2..6d3bb2e2 100644 --- a/roles/matrix-mailer/defaults/main.yml +++ b/roles/matrix-mailer/defaults/main.yml @@ -9,7 +9,7 @@ matrix_mailer_container_image_self_build_repository_url: "https://github.com/dev matrix_mailer_container_image_self_build_src_files_path: "{{ matrix_mailer_base_path }}/docker-src" matrix_mailer_container_image_self_build_version: "{{ matrix_mailer_docker_image.split(':')[1] }}" -matrix_mailer_version: 4.95-r0-2 +matrix_mailer_version: 4.95-r0-4 matrix_mailer_docker_image: "{{ matrix_mailer_docker_image_name_prefix }}devture/exim-relay:{{ matrix_mailer_version }}" matrix_mailer_docker_image_name_prefix: "{{ 'localhost/' if matrix_mailer_container_image_self_build else matrix_container_global_registry_prefix }}" matrix_mailer_docker_image_force_pull: "{{ matrix_mailer_docker_image.endswith(':latest') }}" diff --git a/roles/matrix-nginx-proxy/defaults/main.yml b/roles/matrix-nginx-proxy/defaults/main.yml index 64e2e06b..f19eb4ab 100644 --- a/roles/matrix-nginx-proxy/defaults/main.yml +++ b/roles/matrix-nginx-proxy/defaults/main.yml @@ -192,6 +192,58 @@ matrix_nginx_proxy_proxy_grafana_hostname: "{{ matrix_server_fqn_grafana }}" matrix_nginx_proxy_proxy_sygnal_enabled: false matrix_nginx_proxy_proxy_sygnal_hostname: "{{ matrix_server_fqn_sygnal }}" +# Controls whether proxying for (Prometheus) metrics (`/metrics/*`) for the various services should be done (on the matrix domain) +# If the internal Prometheus server (`matrix-prometheus` role) is used, proxying is not necessary, since Prometheus can access each container directly. +# This is only useful when an external Prometheus will be collecting metrics. +# +# To control what kind of metrics are exposed under `/metrics/` (e.g `/metrics/node-exporter`, `/metrics/postgres-exporter`, etc.), +# use `matrix_SERVICE_metrics_proxying_enabled` variables in each respective role. +# Roles inject themselves into the matrix-nginx-proxy configuration. +# +# To protect the metrics endpoints, see `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled` +matrix_nginx_proxy_proxy_matrix_metrics_enabled: false + +# Controls whether Basic Auth is enabled for all `/metrics/*` endpoints. +# +# You can provide the Basic Auth credentials in 2 ways: +# 1. A single username/password pair using `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` +# 2. Using raw content (`htpasswd`-generated file) provided in `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content` +matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled: false + +# `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` specify +# the Basic Auth username/password for protecting `/metrics/*` endpoints. +# Alternatively, use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`. +matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username: "" +matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password: "" + +# `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content` value will be written verbatim to the htpasswd file protecting `/metrics/*` endpoints. +# Use this when a single username/password is not enough and you'd like to get more control over credentials. +# +# Read the manpage at `man 1 htpasswd` to learn more, then encrypt your password, and paste the encrypted value here. +# e.g. `htpasswd -c mypass.htpasswd prometheus` and enter `mysecurepw` when prompted yields `prometheus:$apr1$wZhqsn.U$7LC3kMmjUbjNAZjyMyvYv/` +# The whole thing is needed here. matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content: "prometheus:$apr1$wZhqsn.U$7LC3kMmjUbjNAZjyMyvYv/" +matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content: "" + +# Specifies the path to the htpasswd file holding the htpasswd credentials for protecting `/metrics/*` endpoints +# This is not meant to be modified. +matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_path: "{{ matrix_nginx_proxy_data_path_in_container if matrix_nginx_proxy_enabled else matrix_nginx_proxy_data_path }}/matrix-metrics-htpasswd" + +# Specifies the Apache container image to use +# when `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` are provided. +# This image provides the `htpasswd` tool which we use for generating the htpasswd file protecting `/metrics/*`. +# To avoid using this, use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content` instead of supplying username/password. +# Learn more in: `roles/matrix-nginx-proxy/tasks/nginx-proxy/setup_metrics_auth.yml`. +matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image: "{{ matrix_container_global_registry_prefix }}httpd:{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image_tag }}" +matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image_tag: "2.4.54-alpine3.16" +matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_force_pull: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image_tag.endswith(':latest') }}" + +# A list of strings containing additional configuration blocks to add to the `location /metrics` configuration (matrix-domain.conf). +# Do not modify `matrix_nginx_proxy_proxy_matrix_metrics_additional_location_configuration_blocks` and `matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks`. +# If you'd like to inject your own configuration blocks, use `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks`. +matrix_nginx_proxy_proxy_matrix_metrics_additional_location_configuration_blocks: "{{ matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks + matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks }}" +matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: [] +matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks: [] + # Controls whether proxying for the matrix-corporal API (`/_matrix/corporal`) should be done (on the matrix domain) matrix_nginx_proxy_proxy_matrix_corporal_api_enabled: false matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container: "matrix-corporal:41081" @@ -216,17 +268,6 @@ matrix_nginx_proxy_proxy_matrix_identity_api_enabled: false matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd:{{ matrix_ma1sd_container_port }}" matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:{{ matrix_ma1sd_container_port }}" -# Controls whether proxying for metrics (`/_synapse/metrics`) should be done (on the matrix domain) -matrix_nginx_proxy_proxy_synapse_metrics: false -matrix_nginx_proxy_synapse_workers_enabled_list: [] -matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled: false -# The following value will be written verbatim to the htpasswd file that stores the password for nginx to check against and needs to be encoded appropriately. -# Read the manpage at `man 1 htpasswd` to learn more, then encrypt your password, and paste the encrypted value here. -# e.g. `htpasswd -c mypass.htpasswd prometheus` and enter `mysecurepw` when prompted yields `prometheus:$apr1$wZhqsn.U$7LC3kMmjUbjNAZjyMyvYv/` -# The part after `prometheus:` is needed here. matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key: "$apr1$wZhqsn.U$7LC3kMmjUbjNAZjyMyvYv/" -matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key: "" -matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_path: "{{ matrix_nginx_proxy_data_path_in_container if matrix_nginx_proxy_enabled else matrix_nginx_proxy_data_path }}/matrix-synapse-metrics-htpasswd" - # The addresses where the Matrix Client API is. # Certain extensions (like matrix-corporal) may override this in order to capture all traffic. matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "matrix-nginx-proxy:12080" @@ -259,8 +300,6 @@ matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_prefix_regexes: | (['/_synapse/oidc'] if matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_oidc_api_enabled else []) + (['/_synapse/admin'] if matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_admin_api_enabled else []) - + - (['/_synapse.*/metrics'] if matrix_nginx_proxy_proxy_synapse_metrics else []) }} # Specifies where requests for the root URI (`/`) on the `matrix.` domain should be redirected. @@ -485,7 +524,7 @@ matrix_ssl_lets_encrypt_staging: false # Learn more here: https://eff-certbot.readthedocs.io/en/stable/using.html#changing-the-acme-server matrix_ssl_lets_encrypt_server: '' -matrix_ssl_lets_encrypt_certbot_docker_image: "{{ matrix_container_global_registry_prefix }}certbot/certbot:{{ matrix_ssl_architecture }}-v1.27.0" +matrix_ssl_lets_encrypt_certbot_docker_image: "{{ matrix_container_global_registry_prefix }}certbot/certbot:{{ matrix_ssl_architecture }}-v1.28.0" matrix_ssl_lets_encrypt_certbot_docker_image_force_pull: "{{ matrix_ssl_lets_encrypt_certbot_docker_image.endswith(':latest') }}" matrix_ssl_lets_encrypt_certbot_standalone_http_port: 2402 matrix_ssl_lets_encrypt_support_email: ~ diff --git a/roles/matrix-nginx-proxy/tasks/nginx-proxy/setup_metrics_auth.yml b/roles/matrix-nginx-proxy/tasks/nginx-proxy/setup_metrics_auth.yml new file mode 100644 index 00000000..046746df --- /dev/null +++ b/roles/matrix-nginx-proxy/tasks/nginx-proxy/setup_metrics_auth.yml @@ -0,0 +1,59 @@ +--- + +# When we're dealing with raw htpasswd content, we just store it in the file directly. +- name: Ensure matrix-metrics-htpasswd is present when generated from raw content (protecting /metrics/* URIs) + copy: + content: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content }}" + dest: "{{ matrix_nginx_proxy_data_path }}/matrix-metrics-htpasswd" + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + mode: 0600 + when: not matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username + +# Alternatively, we need to use the `htpasswd` tool to generate the htpasswd file. +# There's an Ansible module that helps with that, but it requires passlib (a Python module) to be installed on the server. +# See: https://docs.ansible.com/ansible/2.3/htpasswd_module.html#requirements-on-host-that-executes-module +# We support various distros, with various versions of Python. Installing additional Python modules can be a hassle. +# As a workaround, we run `htpasswd` from an Apache container image. +- block: + - name: Ensure Apache Docker image is pulled for generating matrix-metrics-htpasswd from username/password (protecting /metrics/* URIs) + docker_image: + name: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_force_pull }}" + register: result + retries: "{{ matrix_container_retries_count }}" + delay: "{{ matrix_container_retries_delay }}" + until: result is not failed + + # We store the password in a file and make the `htpasswd` tool read it from there, + # as opposed to passing it directly on stdin (which will expose it to other processes on the server). + - name: Store metrics password in a temporary file + copy: + content: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password }}" + dest: "/tmp/matrix-nginx-proxy-metrics-password" + mode: 0400 + owner: "{{ matrix_user_uid }}" + group: "{{ matrix_user_gid }}" + + - name: Generate matrix-metrics-htpasswd from username/password (protecting /metrics/* URIs) + command: + cmd: >- + {{ matrix_host_command_docker }} run + --rm + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --cap-drop=ALL + --network=none + --mount type=bind,src={{ matrix_nginx_proxy_data_path }},dst=/data + --mount type=bind,src=/tmp/matrix-nginx-proxy-metrics-password,dst=/password,ro + --entrypoint=/bin/sh + {{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image }} + -c + 'cat /password | htpasswd -i -c /data/matrix-metrics-htpasswd {{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username }} && chmod 600 /data/matrix-metrics-htpasswd' + + - name: Delete temporary metrics password file + file: + path: /tmp/matrix-nginx-proxy-metrics-password + state: absent + when: matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username != '' diff --git a/roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml b/roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml index a559e109..0da9e52c 100644 --- a/roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml +++ b/roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml @@ -31,23 +31,9 @@ mode: 0644 when: matrix_nginx_proxy_enabled|bool -- name: Ensure matrix-synapse-metrics-htpasswd is present (protecting /_synapse/metrics URI) - template: - src: "{{ role_path }}/templates/nginx/matrix-synapse-metrics-htpasswd.j2" - dest: "{{ matrix_nginx_proxy_data_path }}/matrix-synapse-metrics-htpasswd" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - mode: 0400 - when: "matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled|bool and matrix_nginx_proxy_proxy_synapse_metrics|bool" - -- name: Generate sample prometheus.yml for external scraping - template: - src: "{{ role_path }}/templates/prometheus/external_prometheus.yml.example.j2" - dest: "{{ matrix_base_data_path }}/external_prometheus.yml.example" - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - mode: 0644 - when: matrix_nginx_proxy_proxy_synapse_metrics|bool +- name: Setup metrics + include_tasks: "{{ role_path }}/tasks/nginx-proxy/setup_metrics_auth.yml" + when: matrix_nginx_proxy_proxy_matrix_metrics_enabled|bool and matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|bool - name: Ensure Matrix nginx-proxy configured (generic) template: @@ -324,10 +310,15 @@ file: path: "{{ matrix_nginx_proxy_data_path }}/matrix-synapse-metrics-htpasswd" state: absent - when: "not matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled|bool or not matrix_nginx_proxy_proxy_synapse_metrics|bool" -- name: Ensure sample prometheus.yml for external scraping is deleted +# This file is now generated by the matrix-synapse role and saved in the Synapse directory +- name: (Cleanup) Ensure old sample prometheus.yml for external scraping is deleted file: path: "{{ matrix_base_data_path }}/external_prometheus.yml.example" state: absent - when: "not matrix_nginx_proxy_proxy_synapse_metrics|bool" + +- name: Ensure Matrix nginx-proxy htpasswd is deleted (protecting /metrics/* URIs) + file: + path: "{{ matrix_nginx_proxy_data_path }}/matrix-metrics-htpasswd" + state: absent + when: "not matrix_nginx_proxy_proxy_matrix_metrics_enabled|bool or not matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|bool" diff --git a/roles/matrix-nginx-proxy/tasks/validate_config.yml b/roles/matrix-nginx-proxy/tasks/validate_config.yml index 0de93873..c6697e93 100644 --- a/roles/matrix-nginx-proxy/tasks/validate_config.yml +++ b/roles/matrix-nginx-proxy/tasks/validate_config.yml @@ -27,6 +27,14 @@ `matrix_nginx_proxy_ssl_preset` needs to be set to a known value. when: "matrix_nginx_proxy_ssl_preset not in ['modern', 'intermediate', 'old']" +- name: Fail if Basic Auth enabled for metrics, but no credentials supplied + fail: + msg: | + Enabling Basic Auth for metrics (`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled`) requires: + - either a username/password (provided in `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password`) + - or raw htpasswd content (provided in `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`) + when: "matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|bool and (matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content == '' and (matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username == '' or matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password == ''))" + - block: - name: (Deprecation) Catch and report renamed settings fail: @@ -36,6 +44,7 @@ with_items: - {'old': 'host_specific_matrix_ssl_support_email', 'new': 'matrix_ssl_lets_encrypt_support_email'} - {'old': 'host_specific_matrix_ssl_lets_encrypt_support_email', 'new': 'matrix_ssl_lets_encrypt_support_email'} + - {'old': 'matrix_nginx_proxy_proxy_synapse_workers_enabled_list', 'new': ''} when: "item.old in vars" - name: Fail if required variables are undefined @@ -49,3 +58,17 @@ - "matrix_nginx_proxy_proxy_synapse_client_api_addr_sans_container" when: "vars[item] == '' or vars[item] is none" when: "matrix_ssl_retrieval_method == 'lets-encrypt'" + +- name: (Deprecation) Catch and report old metrics usage + fail: + msg: >- + Your configuration contains a variable (`{{ item }}`), which refers to the old metrics collection system for Synapse, + which exposed metrics on `https://matrix.DOMAIN/_synapse/metrics` and `https://matrix.DOMAIN/_synapse-worker-TYPE-ID/metrics`. + + We now recommend exposing Synapse metrics in another way, from another URL. + Refer to the changelog for more details: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#2022-06-22 + with_items: + - matrix_nginx_proxy_proxy_synapse_metrics + - matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled + - matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key + when: "item in vars" diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 index 4abcd40a..2895ba14 100644 --- a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 +++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 @@ -45,6 +45,19 @@ {{ render_nginx_status_location_block(matrix_nginx_proxy_proxy_matrix_nginx_status_allowed_addresses) }} {% endif %} + {% if matrix_nginx_proxy_proxy_matrix_metrics_enabled %} + location /metrics { + {% if matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled %} + auth_basic "protected"; + auth_basic_user_file {{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_path }}; + {% endif %} + + {% for configuration_block in matrix_nginx_proxy_proxy_matrix_metrics_additional_location_configuration_blocks %} + {{- configuration_block }} + {% endfor %} + } + {% endif %} + {% if matrix_nginx_proxy_proxy_matrix_corporal_api_enabled %} location ^~ /_matrix/corporal { {% if matrix_nginx_proxy_enabled %} diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 index 9a1576d4..69f13a1a 100644 --- a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 +++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 @@ -145,45 +145,6 @@ server { {{- configuration_block }} {% endfor %} - {% if matrix_nginx_proxy_proxy_synapse_metrics %} - location /_synapse/metrics { - {% if matrix_nginx_proxy_enabled %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "{{ matrix_nginx_proxy_proxy_synapse_metrics_addr_with_container }}"; - proxy_pass http://$backend; - {% else %} - {# Generic configuration for use outside of our container setup #} - proxy_pass http://{{ matrix_nginx_proxy_proxy_synapse_metrics_addr_sans_container }}; - {% endif %} - - proxy_set_header Host $host; - - {% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %} - auth_basic "protected"; - auth_basic_user_file {{ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_path }}; - {% endif %} - } - {% endif %} - - {% if matrix_nginx_proxy_enabled and matrix_nginx_proxy_proxy_synapse_metrics %} - {% for worker in matrix_nginx_proxy_proxy_synapse_workers_enabled_list %} - {% if worker.metrics_port != 0 %} - location /_synapse-worker-{{ worker.type }}-{{ worker.instanceId }}/metrics { - resolver 127.0.0.11 valid=5s; - set $backend "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}"; - proxy_pass http://$backend/_synapse/metrics; - proxy_set_header Host $host; - - {% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %} - auth_basic "protected"; - auth_basic_user_file {{ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_path }}; - {% endif %} - } - {% endif %} - {% endfor %} - {% endif %} - {# Everything else just goes to the API server ##} location / { {% if matrix_nginx_proxy_enabled %} diff --git a/roles/matrix-nginx-proxy/templates/nginx/matrix-synapse-metrics-htpasswd.j2 b/roles/matrix-nginx-proxy/templates/nginx/matrix-synapse-metrics-htpasswd.j2 deleted file mode 100644 index 1a7247ac..00000000 --- a/roles/matrix-nginx-proxy/templates/nginx/matrix-synapse-metrics-htpasswd.j2 +++ /dev/null @@ -1,3 +0,0 @@ -#jinja2: lstrip_blocks: "True" -# User and password for protecting /_synapse/metrics URI -prometheus:{{ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key }} diff --git a/roles/matrix-postgres/defaults/main.yml b/roles/matrix-postgres/defaults/main.yml index bb820217..b6cbc205 100644 --- a/roles/matrix-postgres/defaults/main.yml +++ b/roles/matrix-postgres/defaults/main.yml @@ -27,7 +27,7 @@ matrix_postgres_docker_image_v10: "{{ matrix_container_global_registry_prefix }} matrix_postgres_docker_image_v11: "{{ matrix_container_global_registry_prefix }}postgres:11.16{{ matrix_postgres_docker_image_suffix }}" matrix_postgres_docker_image_v12: "{{ matrix_container_global_registry_prefix }}postgres:12.11{{ matrix_postgres_docker_image_suffix }}" matrix_postgres_docker_image_v13: "{{ matrix_container_global_registry_prefix }}postgres:13.7{{ matrix_postgres_docker_image_suffix }}" -matrix_postgres_docker_image_v14: "{{ matrix_container_global_registry_prefix }}postgres:14.3{{ matrix_postgres_docker_image_suffix }}" +matrix_postgres_docker_image_v14: "{{ matrix_container_global_registry_prefix }}postgres:14.4{{ matrix_postgres_docker_image_suffix }}" matrix_postgres_docker_image_latest: "{{ matrix_postgres_docker_image_v14 }}" # This variable is assigned at runtime. Overriding its value has no effect. diff --git a/roles/matrix-prometheus-node-exporter/defaults/main.yml b/roles/matrix-prometheus-node-exporter/defaults/main.yml index 5e50a1d7..d061d59c 100644 --- a/roles/matrix-prometheus-node-exporter/defaults/main.yml +++ b/roles/matrix-prometheus-node-exporter/defaults/main.yml @@ -11,16 +11,36 @@ matrix_prometheus_node_exporter_docker_image_force_pull: "{{ matrix_prometheus_n # A list of extra arguments to pass to the container matrix_prometheus_node_exporter_container_extra_arguments: [] +# A list of extra arguments to pass to the node_exporter process +# +# Example: +# matrix_prometheus_node_exporter_process_extra_arguments: +# - "--collector.systemd" +# - "--collector.logind" +# +# Note: the above is just an example. Various collectors may require various tweaks to be able to run. +# Running the systemd collector requires the following `matrix_prometheus_node_exporter_container_extra_arguments`: +# - the socket to be mounted as well (`--mount type=bind,src=/var/run/dbus/system_bus_socket,dst=/var/run/dbus/system_bus_socket,ro,bind-propagation=rslave`) +# - (on AppArmor-based distros) disabling AppArmor protection (`--security-opt apparmor=unconfined`) +matrix_prometheus_node_exporter_process_extra_arguments: [] + # List of systemd services that matrix-prometheus.service depends on matrix_prometheus_node_exporter_systemd_required_services_list: ['docker.service'] # List of systemd services that matrix-prometheus.service wants matrix_prometheus_node_exporter_systemd_wanted_services_list: [] +# Controls whether node-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/node-exporter`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_prometheus_node_exporter_metrics_proxying_enabled: false + # Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9100 in the container). # # Takes an ":" value (e.g. "127.0.0.1:9100"), or empty string to not expose. # +# You likely don't need to do this. See `matrix_prometheus_node_exporter_metrics_proxying_enabled`. +# # Official recommendations are to run this container with `--net=host`, # but we don't do that, since it: # - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008) diff --git a/roles/matrix-prometheus-node-exporter/tasks/init.yml b/roles/matrix-prometheus-node-exporter/tasks/init.yml index db44a7ab..d08340a8 100644 --- a/roles/matrix-prometheus-node-exporter/tasks/init.yml +++ b/roles/matrix-prometheus-node-exporter/tasks/init.yml @@ -3,3 +3,39 @@ - set_fact: matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-prometheus-node-exporter.service'] }}" when: matrix_prometheus_node_exporter_enabled|bool + +- block: + - name: Fail if matrix-nginx-proxy role already executed + fail: + msg: >- + Trying to append node-exporter's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-prometheus-node-exporter role. + when: matrix_nginx_proxy_role_executed|default(False)|bool + + - name: Generate node-exporter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/node-exporter) + set_fact: + matrix_prometheus_node_exporter_nginx_metrics_configuration_block: | + location /metrics/node-exporter { + {% if matrix_nginx_proxy_enabled|default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-prometheus-node-exporter:9100"; + proxy_pass http://$backend/metrics; + {% else %} + {# Generic configuration for use outside of our container setup #} + {# This may be implemented in the future. #} + return 404 "matrix-nginx-proxy is disabled, so metrics are unavailable"; + {% endif %} + } + + - name: Register node-exporter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/node-exporter) + set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([]) + + + [matrix_prometheus_node_exporter_nginx_metrics_configuration_block] + }} + when: matrix_prometheus_node_exporter_enabled|bool and matrix_prometheus_node_exporter_metrics_proxying_enabled|bool diff --git a/roles/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 b/roles/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 index e38b42e3..d0bfa4cc 100644 --- a/roles/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 +++ b/roles/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 @@ -32,7 +32,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-prometheus-nod --pid=host \ --mount type=bind,src=/,dst=/host,ro,bind-propagation=rslave \ {{ matrix_prometheus_node_exporter_docker_image }} \ - --path.rootfs=/host + --path.rootfs=/host {{ matrix_prometheus_node_exporter_process_extra_arguments|join(' ') }} ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-prometheus-node-exporter 2>/dev/null || true' ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-prometheus-node-exporter 2>/dev/null || true' diff --git a/roles/matrix-prometheus-postgres-exporter/defaults/main.yml b/roles/matrix-prometheus-postgres-exporter/defaults/main.yml index c96a6ea8..8c3f435e 100644 --- a/roles/matrix-prometheus-postgres-exporter/defaults/main.yml +++ b/roles/matrix-prometheus-postgres-exporter/defaults/main.yml @@ -28,15 +28,21 @@ matrix_prometheus_postgres_exporter_database_hostname: 'matrix-postgres' matrix_prometheus_postgres_exporter_database_port: 5432 matrix_prometheus_postgres_exporter_database_name: 'matrix_prometheus_postgres_exporter' +# Controls whether postgres-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/postgres-exporter`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_prometheus_postgres_exporter_metrics_proxying_enabled: false -# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9100 in the container). +# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9187 in the container). # -# Takes an ":" value (e.g. "127.0.0.1:9100"), or empty string to not expose. +# Takes an ":" value (e.g. "127.0.0.1:9187"), or empty string to not expose. +# +# You likely don't need to do this. See `matrix_prometheus_postgres_exporter_metrics_proxying_enabled`. # # Official recommendations are to run this container with `--net=host`, # but we don't do that, since it: # - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008) -# - or listens on a loopback interface only (--net=host and 127.0.0.1:9100), which is not reachable from another container (like `matrix-prometheus`) +# - or listens on a loopback interface only (--net=host and 127.0.0.1:9187), which is not reachable from another container (like `matrix-prometheus`) # # Using `--net=host` and binding to Docker's `matrix` bridge network may be a solution to both, # but that's trickier to accomplish and won't necessarily work (hasn't been tested). diff --git a/roles/matrix-prometheus-postgres-exporter/tasks/init.yml b/roles/matrix-prometheus-postgres-exporter/tasks/init.yml index ddea23ab..996cc975 100644 --- a/roles/matrix-prometheus-postgres-exporter/tasks/init.yml +++ b/roles/matrix-prometheus-postgres-exporter/tasks/init.yml @@ -3,3 +3,39 @@ - set_fact: matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-prometheus-postgres-exporter.service'] }}" when: matrix_prometheus_postgres_exporter_enabled|bool + +- block: + - name: Fail if matrix-nginx-proxy role already executed + fail: + msg: >- + Trying to append postgres-exporter's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-prometheus-postgres-exporter role. + when: matrix_nginx_proxy_role_executed|default(False)|bool + + - name: Generate postgres-exporter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/postgres-exporter) + set_fact: + matrix_prometheus_postgres_exporter_nginx_metrics_configuration_block: | + location /metrics/postgres-exporter { + {% if matrix_nginx_proxy_enabled|default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-prometheus-postgres-exporter:9187"; + proxy_pass http://$backend/metrics; + {% else %} + {# Generic configuration for use outside of our container setup #} + {# This may be implemented in the future. #} + return 404 "matrix-nginx-proxy is disabled, so metrics are unavailable"; + {% endif %} + } + + - name: Register postgres-exporter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/postgres-exporter) + set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([]) + + + [matrix_prometheus_postgres_exporter_nginx_metrics_configuration_block] + }} + when: matrix_prometheus_node_exporter_enabled|bool and matrix_prometheus_node_exporter_metrics_proxying_enabled|bool diff --git a/roles/matrix-prometheus/defaults/main.yml b/roles/matrix-prometheus/defaults/main.yml index 843a90e8..28395bd9 100644 --- a/roles/matrix-prometheus/defaults/main.yml +++ b/roles/matrix-prometheus/defaults/main.yml @@ -4,7 +4,7 @@ matrix_prometheus_enabled: false -matrix_prometheus_version: v2.33.3 +matrix_prometheus_version: v2.36.2 matrix_prometheus_docker_image: "{{ matrix_container_global_registry_prefix }}prom/prometheus:{{ matrix_prometheus_version }}" matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}" @@ -26,6 +26,19 @@ matrix_prometheus_systemd_wanted_services_list: [] # Takes an ":" or "" value (e.g. "127.0.0.1:9090"), or empty string to not expose. matrix_prometheus_container_http_host_bind_port: '' +# A list of default arguments to pass to the prometheus process +matrix_prometheus_process_default_arguments: + - "--config.file=/etc/prometheus/prometheus.yml" + - "--storage.tsdb.path=/prometheus" + - "--web.console.libraries=/usr/share/prometheus/console_libraries" + - "--web.console.templates=/usr/share/prometheus/consoles" + +# A list of extra arguments to pass to the prometheus process +matrix_prometheus_process_extra_arguments: [] + +# holds the final list of process arguments +matrix_prometheus_process_arguments: "{{ matrix_prometheus_process_default_arguments + matrix_prometheus_process_extra_arguments }}" + # Tells whether the "synapse" scraper configuration is enabled. matrix_prometheus_scraper_synapse_enabled: false diff --git a/roles/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 b/roles/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 index 8de57201..56e13c13 100644 --- a/roles/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 +++ b/roles/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 @@ -31,7 +31,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-prometheus \ {% for arg in matrix_prometheus_container_extra_arguments %} {{ arg }} \ {% endfor %} - {{ matrix_prometheus_docker_image }} + {{ matrix_prometheus_docker_image }} {{ matrix_prometheus_process_arguments|join(' ') }} ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-prometheus 2>/dev/null || true' ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-prometheus 2>/dev/null || true' diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml index d0debe85..43ee9e57 100644 --- a/roles/matrix-synapse/defaults/main.yml +++ b/roles/matrix-synapse/defaults/main.yml @@ -9,7 +9,7 @@ matrix_synapse_container_image_self_build_repo: "https://github.com/matrix-org/s matrix_synapse_docker_image: "{{ matrix_synapse_docker_image_name_prefix }}matrixdotorg/synapse:{{ matrix_synapse_docker_image_tag }}" matrix_synapse_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_container_image_self_build else matrix_container_global_registry_prefix }}" -matrix_synapse_version: v1.60.0 +matrix_synapse_version: v1.61.1 matrix_synapse_docker_image_tag: "{{ matrix_synapse_version }}" matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith(':latest') }}" @@ -86,6 +86,14 @@ matrix_synapse_form_secret: "{{ matrix_synapse_macaroon_secret_key }}" matrix_synapse_max_upload_size_mb: 50 +# Controls whether local media should be removed under certain conditions, typically for the purpose of saving space. +# should be empty to disable +matrix_synapse_media_retention_local_media_lifetime: +# Controls whether remote media cache (media that is downloaded from other homeservers) +# should be removed under certain conditions, typically for the purpose of saving space. +# should be empty to disable +matrix_synapse_media_retention_remote_media_lifetime: + # The tmpfs at /tmp needs to be large enough to handle multiple concurrent file uploads. matrix_synapse_tmp_directory_size_mb: "{{ matrix_synapse_max_upload_size_mb * 50 }}" @@ -215,9 +223,6 @@ matrix_synapse_recaptcha_private_key: '' # Disabling this option will not delete any tokens previously generated. matrix_synapse_registration_requires_token: false -# Allows non-server-admin users to create groups on this server -matrix_synapse_enable_group_creation: false - # A list of 3PID types which users must supply when registering (possible values: email, msisdn). matrix_synapse_registrations_require_3pid: [] @@ -329,6 +334,14 @@ matrix_url_preview_accept_language: ['en-US', 'en'] matrix_synapse_metrics_enabled: false matrix_synapse_metrics_port: 9100 +# Controls whether Synapse metrics should be proxied (exposed) on: +# - `matrix.DOMAIN/metrics/synapse/main-process` for the main process +# - `matrix.DOMAIN/metrics/synapse/worker/{type}-{id}` for each worker process +# +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_synapse_metrics_proxying_enabled: false + # Enable the Synapse manhole # See https://github.com/matrix-org/synapse/blob/master/docs/manhole.md matrix_synapse_manhole_enabled: false diff --git a/roles/matrix-synapse/tasks/init.yml b/roles/matrix-synapse/tasks/init.yml index 88065049..ffaec05f 100644 --- a/roles/matrix-synapse/tasks/init.yml +++ b/roles/matrix-synapse/tasks/init.yml @@ -25,3 +25,63 @@ - set_fact: matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-goofys.service'] }}" when: matrix_s3_media_store_enabled|bool + +- block: + - name: Fail if matrix-nginx-proxy role already executed + fail: + msg: >- + Trying to append Synapse's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-synapse role. + when: matrix_nginx_proxy_role_executed|default(False)|bool + + - name: Generate synapse metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/synapse/main-process) + set_fact: + matrix_synapse_nginx_metrics_configuration_block: | + location /metrics/synapse/main-process { + {% if matrix_nginx_proxy_enabled|default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-synapse:{{ matrix_synapse_metrics_port }}"; + proxy_pass http://$backend/_synapse/metrics; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://127.0.0.1:{{ matrix_synapse_metrics_port }}/_synapse/metrics; + {% endif %} + } + + - name: Register synapse metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/synapse/main-process) + set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([]) + + + [matrix_synapse_nginx_metrics_configuration_block] + }} + + - name: Generate synapse worker metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/synapse/worker) + set_fact: + matrix_synapse_worker_nginx_metrics_configuration_block: | + {% for worker in matrix_synapse_workers_enabled_list %} + {% if worker.metrics_port != 0 %} + location /metrics/synapse/worker/{{ worker.type }}-{{ worker.instanceId }} { + resolver 127.0.0.11 valid=5s; + set $backend "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}"; + proxy_pass http://$backend/_synapse/metrics; + proxy_set_header Host $host; + } + {% endif %} + {% endfor %} + when: matrix_synapse_workers_enabled_list|length > 0 + + - name: Register synapse worker metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/synapse/worker) + set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([]) + + + [matrix_synapse_worker_nginx_metrics_configuration_block] + }} + when: matrix_synapse_workers_enabled_list|length > 0 + when: matrix_synapse_enabled|bool and matrix_synapse_metrics_proxying_enabled|bool diff --git a/roles/matrix-synapse/tasks/synapse/setup_install.yml b/roles/matrix-synapse/tasks/synapse/setup_install.yml index 2302a6f2..1aaaf7b3 100644 --- a/roles/matrix-synapse/tasks/synapse/setup_install.yml +++ b/roles/matrix-synapse/tasks/synapse/setup_install.yml @@ -119,3 +119,12 @@ src: "{{ role_path }}/templates/synapse/usr-local-bin/matrix-synapse-register-user.j2" dest: "{{ matrix_local_bin_path }}/matrix-synapse-register-user" mode: 0755 + +- name: Generate sample prometheus.yml for external scraping + template: + src: "{{ role_path }}/templates/synapse/prometheus/external_prometheus.yml.example.j2" + dest: "{{ matrix_synapse_base_path }}/external_prometheus.yml.example" + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + mode: 0644 + when: matrix_synapse_metrics_proxying_enabled|bool diff --git a/roles/matrix-synapse/tasks/synapse/setup_uninstall.yml b/roles/matrix-synapse/tasks/synapse/setup_uninstall.yml index 911d1285..1d4fe7ad 100644 --- a/roles/matrix-synapse/tasks/synapse/setup_uninstall.yml +++ b/roles/matrix-synapse/tasks/synapse/setup_uninstall.yml @@ -29,3 +29,9 @@ docker_image: name: "{{ matrix_synapse_docker_image }}" state: absent + +- name: Ensure sample prometheus.yml for external scraping is deleted + file: + path: "{{ matrix_synapse_base_path }}/external_prometheus.yml.example" + state: absent + when: "not matrix_synapse_metrics_proxying_enabled|bool" diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml index bb8a2bcd..ab0ca3a7 100644 --- a/roles/matrix-synapse/tasks/validate_config.yml +++ b/roles/matrix-synapse/tasks/validate_config.yml @@ -60,6 +60,7 @@ - {'old': 'matrix_synapse_trusted_third_party_id_servers', 'new': ''} - {'old': 'matrix_synapse_use_presence', 'new': 'matrix_synapse_presence_enabled'} - {'old': 'matrix_synapse_version_arm64', 'new': ''} + - {'old': 'matrix_synapse_enable_group_creation', 'new': ''} - name: (Deprecation) Catch and report renamed settings in matrix_synapse_configuration_extension_yaml fail: diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 index c4165437..1f7b7dc1 100644 --- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 +++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 @@ -1049,6 +1049,14 @@ media_store_path: "/matrix-media-store-parent/{{ matrix_synapse_media_store_dire # max_upload_size: "{{ matrix_synapse_max_upload_size_mb }}M" +media_retention: +{% if matrix_synapse_media_retention_local_media_lifetime %} + local_media_lifetime: {{ matrix_synapse_media_retention_local_media_lifetime|to_json }} +{% endif %} +{% if matrix_synapse_media_retention_remote_media_lifetime %} + remote_media_lifetime: {{ matrix_synapse_media_retention_remote_media_lifetime|to_json }} +{% endif %} + # Maximum number of pixels that will be thumbnailed # #max_image_pixels: 32M @@ -2601,16 +2609,6 @@ spam_checker: {{ matrix_synapse_spam_checker|to_json }} encryption_enabled_by_default_for_room_type: {{ matrix_synapse_encryption_enabled_by_default_for_room_type|to_json }} -# Uncomment to allow non-server-admin users to create groups on this server -# -enable_group_creation: {{ matrix_synapse_enable_group_creation|to_json }} - -# If enabled, non server admins can only create groups with local parts -# starting with this prefix -# -#group_creation_prefix: "unofficial_" - - # User Directory configuration # diff --git a/roles/matrix-nginx-proxy/templates/prometheus/external_prometheus.yml.example.j2 b/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2 similarity index 51% rename from roles/matrix-nginx-proxy/templates/prometheus/external_prometheus.yml.example.j2 rename to roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2 index cbb2e6f3..b194c3c2 100644 --- a/roles/matrix-nginx-proxy/templates/prometheus/external_prometheus.yml.example.j2 +++ b/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2 @@ -11,29 +11,29 @@ rule_files: scrape_configs: - job_name: 'synapse' - metrics_path: /_synapse/metrics - scheme: {{ 'https' if matrix_nginx_proxy_https_enabled else 'http' }} -{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %} + metrics_path: /metrics/synapse/main-process + scheme: {{ 'https' if matrix_nginx_proxy_https_enabled|default(true) else 'http' }} +{% if matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|default(true) %} basic_auth: username: prometheus password_file: /path/to/your/passwordfile.pwd {% endif %} static_configs: - - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}'] + - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port|default(443) if matrix_nginx_proxy_https_enabled|default(true) else matrix_nginx_proxy_container_http_host_bind_port|default(80) }}'] labels: job: "master" index: "0" -{% for worker in matrix_nginx_proxy_proxy_synapse_workers_enabled_list %} +{% for worker in matrix_synapse_workers_enabled_list %} - job_name: 'synapse-{{ worker.type }}-{{ worker.instanceId }}' - metrics_path: /_synapse-worker-{{ worker.type }}-{{ worker.instanceId }}/metrics - scheme: {{ 'https' if matrix_nginx_proxy_https_enabled else 'http' }} -{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %} + metrics_path: /metrics/synapse/worker/{{ worker.type }}-{{ worker.instanceId }} + scheme: {{ 'https' if matrix_nginx_proxy_https_enabled|default(true) else 'http' }} +{% if matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|default(true) %} basic_auth: username: prometheus password_file: /path/to/your/passwordfile.pwd {% endif %} static_configs: - - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}'] + - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port|default(443) if matrix_nginx_proxy_https_enabled|default(true) else matrix_nginx_proxy_container_http_host_bind_port|default(80) }}'] labels: job: "{{ worker.type }}" index: "{{ worker.instanceId }}" diff --git a/roles/matrix-synapse/templates/synapse/worker.yaml.j2 b/roles/matrix-synapse/templates/synapse/worker.yaml.j2 index 40714f44..239de1f2 100644 --- a/roles/matrix-synapse/templates/synapse/worker.yaml.j2 +++ b/roles/matrix-synapse/templates/synapse/worker.yaml.j2 @@ -26,6 +26,7 @@ worker_listeners: {% if http_resources|length > 0 %} - type: http bind_addresses: ['::'] + x_forwarded: true port: {{ matrix_synapse_worker_details.port }} resources: - names: {{ http_resources|to_json }} diff --git a/roles/matrix-synapse/vars/workers.yml b/roles/matrix-synapse/vars/workers.yml index 91b289c7..e535d2cc 100644 --- a/roles/matrix-synapse/vars/workers.yml +++ b/roles/matrix-synapse/vars/workers.yml @@ -29,9 +29,8 @@ matrix_synapse_workers_generic_worker_endpoints: - ^/_matrix/federation/v1/event_auth/ - ^/_matrix/federation/v1/exchange_third_party_invite/ - ^/_matrix/federation/v1/user/devices/ - - ^/_matrix/federation/v1/get_groups_publicised$ - ^/_matrix/key/v2/query - - ^/_matrix/federation/(v1|unstable/org.matrix.msc2946)/hierarchy/ + - ^/_matrix/federation/v1/hierarchy/ # Inbound federation transaction request - ^/_matrix/federation/v1/send/ @@ -43,15 +42,14 @@ matrix_synapse_workers_generic_worker_endpoints: - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$ - - ^/_matrix/client/(v1|unstable/org.matrix.msc2946)/rooms/.*/hierarchy$ + - ^/_matrix/client/v1/rooms/.*/hierarchy$ + - ^/_matrix/client/unstable/org.matrix.msc2716/rooms/.*/batch_send$ - ^/_matrix/client/unstable/im.nheko.summary/rooms/.*/summary$ - ^/_matrix/client/(r0|v3|unstable)/account/3pid$ + - ^/_matrix/client/(r0|v3|unstable)/account/whoami$ - ^/_matrix/client/(r0|v3|unstable)/devices$ - ^/_matrix/client/versions$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$ - - ^/_matrix/client/(r0|v3|unstable)/joined_groups$ - - ^/_matrix/client/(r0|v3|unstable)/publicised_groups$ - - ^/_matrix/client/(r0|v3|unstable)/publicised_groups/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/search$ @@ -75,31 +73,27 @@ matrix_synapse_workers_generic_worker_endpoints: - ^/_matrix/client/(api/v1|r0|v3|unstable)/join/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/ - # These appear to be conditional and should not be enabled by default. - # We need to fix up our workers-doc-to-yaml.awk parsing script to exclude them. - # For now, they've been commented out manually. - # - # # Device requests - # - ^/_matrix/client/(r0|v3|unstable)/sendToDevice/ - - # # Account data requests - # - ^/_matrix/client/(r0|v3|unstable)/.*/tags - # - ^/_matrix/client/(r0|v3|unstable)/.*/account_data - - # # Receipts requests - # - ^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt - # - ^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers - - # # Presence requests - # - ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/ +# These appear to be conditional and should not be enabled by default. +# We need to fix up our workers-doc-to-yaml.awk parsing script to exclude them. +# For now, they've been commented out manually. +# # Account data requests +# - ^/_matrix/client/(r0|v3|unstable)/.*/tags +# - ^/_matrix/client/(r0|v3|unstable)/.*/account_data +# +# # Receipts requests +# - ^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt +# - ^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers +# +# # Presence requests +# - ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/ + # User directory search requests + - ^/_matrix/client/(r0|v3|unstable)/user_directory/search$ # Additionally, the following REST endpoints can be handled for GET requests: # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually - # ^/_matrix/federation/v1/groups/ # ^/_matrix/client/(api/v1|r0|v3|unstable)/pushrules/ - # ^/_matrix/client/(r0|v3|unstable)/groups/ # Pagination requests can also be handled, but all requests for a given # room must be routed to the same instance. Additionally, care must be taken to @@ -301,6 +295,15 @@ matrix_synapse_workers_generic_worker_endpoints: # This work cannot be load-balanced; please ensure the main process is restarted # after setting this option in the shared configuration! + # User directory updates allow REST endpoints matching the following regular + # expressions to work: + + # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually + # ^/_matrix/client/(r0|v3|unstable)/user_directory/search$ + + # The above endpoints can be routed to any worker, though you may choose to route + # it to the chosen user directory worker. + # This style of configuration supersedes the legacy `synapse.app.user_dir` # worker application type. diff --git a/setup.yml b/setup.yml index ce36d1ce..27aac7a7 100755 --- a/setup.yml +++ b/setup.yml @@ -18,6 +18,7 @@ - matrix-bridge-appservice-webhooks - matrix-bridge-appservice-irc - matrix-bridge-beeper-linkedin + - matrix-bridge-go-skype-bridge - matrix-bridge-mautrix-facebook - matrix-bridge-mautrix-twitter - matrix-bridge-mautrix-hangouts @@ -46,6 +47,7 @@ - matrix-dendrite - matrix-synapse-admin - matrix-prometheus-node-exporter + - matrix-prometheus-postgres-exporter - matrix-prometheus - matrix-grafana - matrix-registration @@ -62,6 +64,5 @@ - matrix-coturn - matrix-aux - matrix-postgres-backup - - matrix-prometheus-postgres-exporter - matrix-backup-borg - matrix-common-after