From a1fb0826183af614c277430f257bfcc729026d53 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 15 Sep 2022 10:32:51 +0300 Subject: [PATCH] Add support for running background tasks on a worker --- CHANGELOG.md | 10 +++++++++- roles/matrix-synapse/defaults/main.yml | 11 +++++++++++ .../tasks/synapse/workers/init.yml | 16 ++++++++++++++++ roles/matrix-synapse/tasks/validate_config.yml | 1 + .../templates/synapse/homeserver.yaml.j2 | 3 +++ roles/matrix-synapse/vars/main.yml | 2 ++ 6 files changed, 42 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 37032735..06cac7a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ People who are interested in running a Synapse worker setup should know that **o - we've added support for [Stream writers](#stream-writers-support) - we've added support for [multiple federation sender workers](#multiple-federation-sender-workers-support) - we've added support for [multiple pusher workers](#multiple-pusher-workers-support) +- we've added support for [running background tasks on a worker](#background-tasks-can-run-on-a-worker) - we've restored support for [`appservice` workers](#appservice-worker-support-is-back) - we've restored support for [`user_dir` workers](#user-directory-worker-support-is-back) - see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes) @@ -40,6 +41,13 @@ From now on, you can have as many as you want to help with your federation traff Until now, we only supported a single `pusher` worker (`matrix_synapse_workers_pusher_workers_count` could either be `0` or `1`). From now on, you can have as many as you want to help with pushing notifications out. +### Background tasks can run on a worker + +From now on, you can put [background task processing on a worker](https://matrix-org.github.io/synapse/latest/workers.html#background-tasks). + +With `matrix_synapse_workers_preset: one-of-each`, you'll get one `background` worker automatically. +You can also control the `background` workers count with `matrix_synapse_workers_background_workers_count`. Only `0` or `1` workers of this type are supported by Synapse. + ### Appservice worker support is back We previously had an `appservice` worker type, which [Synapse deprecated in v1.59.0](https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types). So did we, at the time. @@ -66,7 +74,7 @@ Below we'll discuss **potential backward incompatibilities**. - **Worker names** (container names, systemd services, worker configuration files) **have changed**. Workers are now labeled sequentially (e.g. `matrix-synapse-worker_generic_worker-18111` -> `matrix-synapse-worker-generic-0`). The playbook will handle these changes automatically. -- Due to increased worker types support above, people who use `matrix_synapse_workers_preset: one-of-each` should be aware that with these changes, **the playbook will deploy 8 additional workers** (6 stream writers, 1 `appservice` worker, 1 `user_dir` worker). This **may increase RAM/CPU usage**, etc. If you find your server struggling, consider disabling some workers with the appropriate `matrix_synapse_workers_*_workers_count` variables. +- Due to increased worker types support above, people who use `matrix_synapse_workers_preset: one-of-each` should be aware that with these changes, **the playbook will deploy 9 additional workers** (6 stream writers, 1 `appservice` worker, 1 `user_dir` worker, 1 background task worker). This **may increase RAM/CPU usage**, etc. If you find your server struggling, consider disabling some workers with the appropriate `matrix_synapse_workers_*_workers_count` variables. - **Metric endpoints have also changed** (`/metrics/synapse/worker/generic_worker-18111` -> `/metrics/synapse/worker/generic-worker-0`). If you're [collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server), consider revisiting our [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server) docs and updating your Prometheus configuration. **If you're collecting metrics to the integrated Prometheus server** (not enabled by default), **your Prometheus configuration will be updated automatically**. Old data (from before this change) may stick around though. diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml index 7f6ce40f..672285cc 100644 --- a/roles/matrix-synapse/defaults/main.yml +++ b/roles/matrix-synapse/defaults/main.yml @@ -398,6 +398,7 @@ matrix_synapse_workers_presets: media_repository_workers_count: 0 appservice_workers_count: 0 user_dir_workers_count: 0 + background_workers_count: 0 stream_writer_events_stream_workers_count: 0 stream_writer_typing_stream_workers_count: 0 stream_writer_to_device_stream_workers_count: 0 @@ -411,6 +412,7 @@ matrix_synapse_workers_presets: media_repository_workers_count: 1 appservice_workers_count: 1 user_dir_workers_count: 1 + background_workers_count: 1 stream_writer_events_stream_workers_count: 1 stream_writer_typing_stream_workers_count: 1 stream_writer_to_device_stream_workers_count: 1 @@ -554,6 +556,15 @@ matrix_synapse_workers_user_dir_workers_metrics_range_start: 19661 # `update_user_directory_from_worker` is meant to point to a worker, which is dedicated to updating the user directory and servicing some user directory URL endpoints (`matrix_synapse_workers_user_dir_worker_client_server_endpoints`). matrix_synapse_update_user_directory_from_worker: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list | length > 0) else '' }}" +# matrix_synapse_workers_background_workers_count can only be 0 or 1. More instances are not supported. +# Our implementation uses a generic worker and assigns Synapse to perform background work on this worker using the `run_background_tasks_on` Synapse option. +matrix_synapse_workers_background_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['background_workers_count'] }}" +matrix_synapse_workers_background_workers_metrics_range_start: 19700 + +# matrix_synapse_run_background_tasks_on populates the `run_background_tasks_on` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`). +# `run_background_tasks_on` is meant to point to a worker, which is dedicated to processing background tasks. +matrix_synapse_run_background_tasks_on: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'background') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'background') | list | length > 0) else '' }}" + # Default list of workers to spawn. # # Unless you populate this manually, this list is dynamically generated diff --git a/roles/matrix-synapse/tasks/synapse/workers/init.yml b/roles/matrix-synapse/tasks/synapse/workers/init.yml index c5a53297..0c2b8a79 100644 --- a/roles/matrix-synapse/tasks/synapse/workers/init.yml +++ b/roles/matrix-synapse/tasks/synapse/workers/init.yml @@ -100,6 +100,20 @@ register: "matrix_synapse_workers_list_results_user_dir_workers" loop: "{{ range(0, matrix_synapse_workers_user_dir_workers_count | int) | list }}" +# This type of worker can only have a count of 1, at most +- name: Build background workers + ansible.builtin.set_fact: + worker: + id: "background-{{ item }}" + name: "matrix-synapse-worker-background-{{ item }}" + type: 'background' + app: 'generic_worker' + webserving: false + port: 0 + metrics_port: "{{ matrix_synapse_workers_background_workers_metrics_range_start + item }}" + register: "matrix_synapse_workers_list_results_background_workers" + loop: "{{ range(0, matrix_synapse_workers_background_workers_count | int) | list }}" + - name: Build media_repository workers ansible.builtin.set_fact: worker: @@ -130,6 +144,8 @@ matrix_synapse_workers_list_results_user_dir_workers.results + matrix_synapse_workers_list_results_media_repository_workers.results + + + matrix_synapse_workers_list_results_background_workers.results }} - ansible.builtin.set_fact: diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml index f6b1fe68..79e58154 100644 --- a/roles/matrix-synapse/tasks/validate_config.yml +++ b/roles/matrix-synapse/tasks/validate_config.yml @@ -20,6 +20,7 @@ with_items: - "matrix_synapse_workers_appservice_workers_count" - "matrix_synapse_workers_user_dir_workers_count" + - "matrix_synapse_workers_background_workers_count" - "matrix_synapse_workers_stream_writer_typing_stream_workers_count" - "matrix_synapse_workers_stream_writer_to_device_stream_workers_count" - "matrix_synapse_workers_stream_writer_account_data_stream_workers_count" diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 index ef8684ab..e330617f 100644 --- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 +++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 @@ -2886,6 +2886,9 @@ update_user_directory_from_worker: {{ matrix_synapse_update_user_directory_from_ # data). If not provided this defaults to the main process. # #run_background_tasks_on: worker1 +{% if matrix_synapse_run_background_tasks_on != '' %} +run_background_tasks_on: {{ matrix_synapse_run_background_tasks_on | to_json }} +{% endif %} # A shared secret used by the replication APIs to authenticate HTTP requests # from workers. diff --git a/roles/matrix-synapse/vars/main.yml b/roles/matrix-synapse/vars/main.yml index 2dc00fea..b403b461 100644 --- a/roles/matrix-synapse/vars/main.yml +++ b/roles/matrix-synapse/vars/main.yml @@ -102,6 +102,8 @@ matrix_synapse_known_worker_types: | ['appservice'] + ['user_dir'] + + + ['background'] ) | unique }}