Add support for running background tasks on a worker

This commit is contained in:
Slavi Pantaleev 2022-09-15 10:32:51 +03:00
parent 5f3f460cda
commit a1fb082618
6 changed files with 42 additions and 1 deletions

View file

@ -7,6 +7,7 @@ People who are interested in running a Synapse worker setup should know that **o
- we've added support for [Stream writers](#stream-writers-support) - we've added support for [Stream writers](#stream-writers-support)
- we've added support for [multiple federation sender workers](#multiple-federation-sender-workers-support) - we've added support for [multiple federation sender workers](#multiple-federation-sender-workers-support)
- we've added support for [multiple pusher workers](#multiple-pusher-workers-support) - we've added support for [multiple pusher workers](#multiple-pusher-workers-support)
- we've added support for [running background tasks on a worker](#background-tasks-can-run-on-a-worker)
- we've restored support for [`appservice` workers](#appservice-worker-support-is-back) - we've restored support for [`appservice` workers](#appservice-worker-support-is-back)
- we've restored support for [`user_dir` workers](#user-directory-worker-support-is-back) - we've restored support for [`user_dir` workers](#user-directory-worker-support-is-back)
- see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes) - see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes)
@ -40,6 +41,13 @@ From now on, you can have as many as you want to help with your federation traff
Until now, we only supported a single `pusher` worker (`matrix_synapse_workers_pusher_workers_count` could either be `0` or `1`). Until now, we only supported a single `pusher` worker (`matrix_synapse_workers_pusher_workers_count` could either be `0` or `1`).
From now on, you can have as many as you want to help with pushing notifications out. From now on, you can have as many as you want to help with pushing notifications out.
### Background tasks can run on a worker
From now on, you can put [background task processing on a worker](https://matrix-org.github.io/synapse/latest/workers.html#background-tasks).
With `matrix_synapse_workers_preset: one-of-each`, you'll get one `background` worker automatically.
You can also control the `background` workers count with `matrix_synapse_workers_background_workers_count`. Only `0` or `1` workers of this type are supported by Synapse.
### Appservice worker support is back ### Appservice worker support is back
We previously had an `appservice` worker type, which [Synapse deprecated in v1.59.0](https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types). So did we, at the time. We previously had an `appservice` worker type, which [Synapse deprecated in v1.59.0](https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types). So did we, at the time.
@ -66,7 +74,7 @@ Below we'll discuss **potential backward incompatibilities**.
- **Worker names** (container names, systemd services, worker configuration files) **have changed**. Workers are now labeled sequentially (e.g. `matrix-synapse-worker_generic_worker-18111` -> `matrix-synapse-worker-generic-0`). The playbook will handle these changes automatically. - **Worker names** (container names, systemd services, worker configuration files) **have changed**. Workers are now labeled sequentially (e.g. `matrix-synapse-worker_generic_worker-18111` -> `matrix-synapse-worker-generic-0`). The playbook will handle these changes automatically.
- Due to increased worker types support above, people who use `matrix_synapse_workers_preset: one-of-each` should be aware that with these changes, **the playbook will deploy 8 additional workers** (6 stream writers, 1 `appservice` worker, 1 `user_dir` worker). This **may increase RAM/CPU usage**, etc. If you find your server struggling, consider disabling some workers with the appropriate `matrix_synapse_workers_*_workers_count` variables. - Due to increased worker types support above, people who use `matrix_synapse_workers_preset: one-of-each` should be aware that with these changes, **the playbook will deploy 9 additional workers** (6 stream writers, 1 `appservice` worker, 1 `user_dir` worker, 1 background task worker). This **may increase RAM/CPU usage**, etc. If you find your server struggling, consider disabling some workers with the appropriate `matrix_synapse_workers_*_workers_count` variables.
- **Metric endpoints have also changed** (`/metrics/synapse/worker/generic_worker-18111` -> `/metrics/synapse/worker/generic-worker-0`). If you're [collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server), consider revisiting our [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server) docs and updating your Prometheus configuration. **If you're collecting metrics to the integrated Prometheus server** (not enabled by default), **your Prometheus configuration will be updated automatically**. Old data (from before this change) may stick around though. - **Metric endpoints have also changed** (`/metrics/synapse/worker/generic_worker-18111` -> `/metrics/synapse/worker/generic-worker-0`). If you're [collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server), consider revisiting our [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server) docs and updating your Prometheus configuration. **If you're collecting metrics to the integrated Prometheus server** (not enabled by default), **your Prometheus configuration will be updated automatically**. Old data (from before this change) may stick around though.

View file

@ -398,6 +398,7 @@ matrix_synapse_workers_presets:
media_repository_workers_count: 0 media_repository_workers_count: 0
appservice_workers_count: 0 appservice_workers_count: 0
user_dir_workers_count: 0 user_dir_workers_count: 0
background_workers_count: 0
stream_writer_events_stream_workers_count: 0 stream_writer_events_stream_workers_count: 0
stream_writer_typing_stream_workers_count: 0 stream_writer_typing_stream_workers_count: 0
stream_writer_to_device_stream_workers_count: 0 stream_writer_to_device_stream_workers_count: 0
@ -411,6 +412,7 @@ matrix_synapse_workers_presets:
media_repository_workers_count: 1 media_repository_workers_count: 1
appservice_workers_count: 1 appservice_workers_count: 1
user_dir_workers_count: 1 user_dir_workers_count: 1
background_workers_count: 1
stream_writer_events_stream_workers_count: 1 stream_writer_events_stream_workers_count: 1
stream_writer_typing_stream_workers_count: 1 stream_writer_typing_stream_workers_count: 1
stream_writer_to_device_stream_workers_count: 1 stream_writer_to_device_stream_workers_count: 1
@ -554,6 +556,15 @@ matrix_synapse_workers_user_dir_workers_metrics_range_start: 19661
# `update_user_directory_from_worker` is meant to point to a worker, which is dedicated to updating the user directory and servicing some user directory URL endpoints (`matrix_synapse_workers_user_dir_worker_client_server_endpoints`). # `update_user_directory_from_worker` is meant to point to a worker, which is dedicated to updating the user directory and servicing some user directory URL endpoints (`matrix_synapse_workers_user_dir_worker_client_server_endpoints`).
matrix_synapse_update_user_directory_from_worker: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list | length > 0) else '' }}" matrix_synapse_update_user_directory_from_worker: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list | length > 0) else '' }}"
# matrix_synapse_workers_background_workers_count can only be 0 or 1. More instances are not supported.
# Our implementation uses a generic worker and assigns Synapse to perform background work on this worker using the `run_background_tasks_on` Synapse option.
matrix_synapse_workers_background_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['background_workers_count'] }}"
matrix_synapse_workers_background_workers_metrics_range_start: 19700
# matrix_synapse_run_background_tasks_on populates the `run_background_tasks_on` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
# `run_background_tasks_on` is meant to point to a worker, which is dedicated to processing background tasks.
matrix_synapse_run_background_tasks_on: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'background') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'background') | list | length > 0) else '' }}"
# Default list of workers to spawn. # Default list of workers to spawn.
# #
# Unless you populate this manually, this list is dynamically generated # Unless you populate this manually, this list is dynamically generated

View file

@ -100,6 +100,20 @@
register: "matrix_synapse_workers_list_results_user_dir_workers" register: "matrix_synapse_workers_list_results_user_dir_workers"
loop: "{{ range(0, matrix_synapse_workers_user_dir_workers_count | int) | list }}" loop: "{{ range(0, matrix_synapse_workers_user_dir_workers_count | int) | list }}"
# This type of worker can only have a count of 1, at most
- name: Build background workers
ansible.builtin.set_fact:
worker:
id: "background-{{ item }}"
name: "matrix-synapse-worker-background-{{ item }}"
type: 'background'
app: 'generic_worker'
webserving: false
port: 0
metrics_port: "{{ matrix_synapse_workers_background_workers_metrics_range_start + item }}"
register: "matrix_synapse_workers_list_results_background_workers"
loop: "{{ range(0, matrix_synapse_workers_background_workers_count | int) | list }}"
- name: Build media_repository workers - name: Build media_repository workers
ansible.builtin.set_fact: ansible.builtin.set_fact:
worker: worker:
@ -130,6 +144,8 @@
matrix_synapse_workers_list_results_user_dir_workers.results matrix_synapse_workers_list_results_user_dir_workers.results
+ +
matrix_synapse_workers_list_results_media_repository_workers.results matrix_synapse_workers_list_results_media_repository_workers.results
+
matrix_synapse_workers_list_results_background_workers.results
}} }}
- ansible.builtin.set_fact: - ansible.builtin.set_fact:

View file

@ -20,6 +20,7 @@
with_items: with_items:
- "matrix_synapse_workers_appservice_workers_count" - "matrix_synapse_workers_appservice_workers_count"
- "matrix_synapse_workers_user_dir_workers_count" - "matrix_synapse_workers_user_dir_workers_count"
- "matrix_synapse_workers_background_workers_count"
- "matrix_synapse_workers_stream_writer_typing_stream_workers_count" - "matrix_synapse_workers_stream_writer_typing_stream_workers_count"
- "matrix_synapse_workers_stream_writer_to_device_stream_workers_count" - "matrix_synapse_workers_stream_writer_to_device_stream_workers_count"
- "matrix_synapse_workers_stream_writer_account_data_stream_workers_count" - "matrix_synapse_workers_stream_writer_account_data_stream_workers_count"

View file

@ -2886,6 +2886,9 @@ update_user_directory_from_worker: {{ matrix_synapse_update_user_directory_from_
# data). If not provided this defaults to the main process. # data). If not provided this defaults to the main process.
# #
#run_background_tasks_on: worker1 #run_background_tasks_on: worker1
{% if matrix_synapse_run_background_tasks_on != '' %}
run_background_tasks_on: {{ matrix_synapse_run_background_tasks_on | to_json }}
{% endif %}
# A shared secret used by the replication APIs to authenticate HTTP requests # A shared secret used by the replication APIs to authenticate HTTP requests
# from workers. # from workers.

View file

@ -102,6 +102,8 @@ matrix_synapse_known_worker_types: |
['appservice'] ['appservice']
+ +
['user_dir'] ['user_dir']
+
['background']
) | unique ) | unique
}} }}