Restore support for appservice and user_dir workers
This commit is contained in:
Slavi Pantaleev
parent
8cd7c1ec2f
commit
5f3f460cda
25
CHANGELOG.md
25
CHANGELOG.md
|
|
@ -1,12 +1,14 @@
|
|||
# 2022-09-15
|
||||
|
||||
## (Potential Backward Compatibility Break) Major changes to Synapse workers
|
||||
## (Potential Backward Compatibility Break) Major improvements to Synapse workers
|
||||
|
||||
People who are interested in running a Synapse worker setup should know that **our Synapse worker implementation is much more powerful now**:
|
||||
|
||||
- we've added support for [Stream writers](#stream-writers-support)
|
||||
- we've added support for [multiple federation sender workers](#multiple-federation-sender-workers-support)
|
||||
- we've added support for [multiple pusher workers](#multiple-pusher-workers-support)
|
||||
- we've restored support for [`appservice` workers](#appservice-worker-support-is-back)
|
||||
- we've restored support for [`user_dir` workers](#user-directory-worker-support-is-back)
|
||||
- see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes)
|
||||
|
||||
### Stream writers support
|
||||
|
|
@ -38,6 +40,25 @@ From now on, you can have as many as you want to help with your federation traff
|
|||
Until now, we only supported a single `pusher` worker (`matrix_synapse_workers_pusher_workers_count` could either be `0` or `1`).
|
||||
From now on, you can have as many as you want to help with pushing notifications out.
|
||||
|
||||
### Appservice worker support is back
|
||||
|
||||
We previously had an `appservice` worker type, which [Synapse deprecated in v1.59.0](https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types). So did we, at the time.
|
||||
|
||||
The new way to implement such workers is by using a `generic_worker` and dedicating it to the task of talking to Application Services.
|
||||
From now on, we have support for this.
|
||||
|
||||
With `matrix_synapse_workers_preset: one-of-each`, you'll get one `appserice` worker automatically.
|
||||
You can also control the `appserice` workers count with `matrix_synapse_workers_appserice_workers_count`. Only `0` or `1` workers of this type are supported by Synapse.
|
||||
|
||||
### User Directory worker support is back
|
||||
|
||||
We previously had a `user_dir` worker type, which [Synapse deprecated in v1.59.0](https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types). So did we, at the time.
|
||||
|
||||
The new way to implement such workers is by using a `generic_worker` and dedicating it to the task of serving the user directory.
|
||||
From now on, we have support for this.
|
||||
|
||||
With `matrix_synapse_workers_preset: one-of-each`, you'll get one `user_dir` worker automatically.
|
||||
You can also control the `user_dir` workers count with `matrix_synapse_workers_user_dir_workers_count`. Only `0` or `1` workers of this type are supported by Synapse.
|
||||
|
||||
### Potential Backward Incompatibilities after these Synapse worker changes
|
||||
|
||||
|
|
@ -45,6 +66,8 @@ Below we'll discuss **potential backward incompatibilities**.
|
|||
|
||||
- **Worker names** (container names, systemd services, worker configuration files) **have changed**. Workers are now labeled sequentially (e.g. `matrix-synapse-worker_generic_worker-18111` -> `matrix-synapse-worker-generic-0`). The playbook will handle these changes automatically.
|
||||
|
||||
- Due to increased worker types support above, people who use `matrix_synapse_workers_preset: one-of-each` should be aware that with these changes, **the playbook will deploy 8 additional workers** (6 stream writers, 1 `appservice` worker, 1 `user_dir` worker). This **may increase RAM/CPU usage**, etc. If you find your server struggling, consider disabling some workers with the appropriate `matrix_synapse_workers_*_workers_count` variables.
|
||||
|
||||
- **Metric endpoints have also changed** (`/metrics/synapse/worker/generic_worker-18111` -> `/metrics/synapse/worker/generic-worker-0`). If you're [collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server), consider revisiting our [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server) docs and updating your Prometheus configuration. **If you're collecting metrics to the integrated Prometheus server** (not enabled by default), **your Prometheus configuration will be updated automatically**. Old data (from before this change) may stick around though.
|
||||
|
||||
- **the format of `matrix_synapse_workers_enabled_list` has changed**. You were never advised to use this variable for directly creating workers (we advise people to control workers using `matrix_synapse_workers_preset` or by tweaking `matrix_synapse_workers_*_workers_count` variables only), but some people may have started using the `matrix_synapse_workers_enabled_list` variable to gain more control over workers. If you're one of them, you'll need to adjust its value. See `roles/matrix-synapse/defaults/main.yml` for more information on the new format. The playbook will also do basic validation and complain if you got something wrong.
|
||||
|
|
|
|||
|
|
@ -1766,7 +1766,7 @@ matrix_nginx_proxy_synapse_stream_writer_account_data_stream_worker_client_serve
|
|||
matrix_nginx_proxy_synapse_stream_writer_receipts_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoints }}"
|
||||
matrix_nginx_proxy_synapse_stream_writer_presence_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints }}"
|
||||
matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers_media_repository_endpoints|default([]) }}"
|
||||
matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}"
|
||||
matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_worker_client_server_endpoints|default([]) }}"
|
||||
|
||||
matrix_nginx_proxy_systemd_wanted_services_list: |
|
||||
{{
|
||||
|
|
|
|||
|
|
@ -133,8 +133,9 @@ server {
|
|||
{% endif %}
|
||||
|
||||
{% if user_dir_workers | length > 0 %}
|
||||
# FIXME: obsolete if matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled is set
|
||||
# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappuser_dir
|
||||
# https://matrix-org.github.io/synapse/latest/workers.html#updating-the-user-directory
|
||||
# If matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled is set, requests may not reach here,
|
||||
# but could be captured early on (see `matrix-domain.conf.j2`) and forwarded elsewhere (to an identity server, etc.).
|
||||
{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_user_dir_locations, 'user_dir_workers_upstream') }}
|
||||
{% endif %}
|
||||
{# Workers redirects END #}
|
||||
|
|
|
|||
|
|
@ -394,9 +394,9 @@ matrix_synapse_workers_presets:
|
|||
little-federation-helper:
|
||||
generic_workers_count: 0
|
||||
pusher_workers_count: 0
|
||||
appservice_workers_count: 0
|
||||
federation_sender_workers_count: 1
|
||||
media_repository_workers_count: 0
|
||||
appservice_workers_count: 0
|
||||
user_dir_workers_count: 0
|
||||
stream_writer_events_stream_workers_count: 0
|
||||
stream_writer_typing_stream_workers_count: 0
|
||||
|
|
@ -407,14 +407,10 @@ matrix_synapse_workers_presets:
|
|||
one-of-each:
|
||||
generic_workers_count: 1
|
||||
pusher_workers_count: 1
|
||||
# appservice workers are deprecated since Synapse v1.59. This will be removed.
|
||||
appservice_workers_count: 0
|
||||
federation_sender_workers_count: 1
|
||||
media_repository_workers_count: 1
|
||||
# Disabled until https://github.com/matrix-org/synapse/issues/8787 is resolved.
|
||||
# user_dir workers are deprecated since Synapse v1.59. This will be removed.
|
||||
# See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
|
||||
user_dir_workers_count: 0
|
||||
appservice_workers_count: 1
|
||||
user_dir_workers_count: 1
|
||||
stream_writer_events_stream_workers_count: 1
|
||||
stream_writer_typing_stream_workers_count: 1
|
||||
stream_writer_to_device_stream_workers_count: 1
|
||||
|
|
@ -513,12 +509,6 @@ matrix_synapse_federation_pusher_instances: []
|
|||
# Adjusting this value manually is generally not necessary.
|
||||
matrix_synapse_start_pushers: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list | length > 0) }}"
|
||||
|
||||
# matrix_synapse_workers_appservice_workers_count can only be 0 or 1. More instances are not supported.
|
||||
# appservice workers are deprecated since Synapse v1.59. This will be removed.
|
||||
# See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
|
||||
matrix_synapse_workers_appservice_workers_count: 0
|
||||
matrix_synapse_workers_appservice_workers_metrics_range_start: 19300
|
||||
|
||||
# matrix_synapse_workers_federation_sender_workers_count controls the number of federation sender workers to spawn.
|
||||
# See https://matrix-org.github.io/synapse/latest/workers.html#synapseappfederation_sender
|
||||
matrix_synapse_workers_federation_sender_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['federation_sender_workers_count'] }}"
|
||||
|
|
@ -543,13 +533,27 @@ matrix_synapse_workers_media_repository_workers_metrics_range_start: 19551
|
|||
# Adjusting this value manually is generally not necessary.
|
||||
matrix_synapse_enable_media_repo: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0) }}"
|
||||
|
||||
# Disabled until https://github.com/matrix-org/synapse/issues/8787 is resolved.
|
||||
# user_dir workers are deprecated since Synapse v1.59. This will be removed.
|
||||
# See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
|
||||
matrix_synapse_workers_user_dir_workers_count: 0
|
||||
# matrix_synapse_workers_appservice_workers_count can only be 0 or 1. More instances are not supported.
|
||||
# appservice workers were deprecated since Synapse v1.59 (see: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types).
|
||||
# Our implementation uses generic worker services and assigns them to perform appservice work using the `notify_appservices_from_worker` Synapse option.
|
||||
matrix_synapse_workers_appservice_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['appservice_workers_count'] }}"
|
||||
matrix_synapse_workers_appservice_workers_metrics_range_start: 19300
|
||||
|
||||
# matrix_synapse_notify_appservices_from_worker populates the `notify_appservices_from_worker` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
|
||||
# `notify_appservices_from_worker` is meant to point to a worker, which is dedicated to sending output traffic to Application Services.
|
||||
matrix_synapse_notify_appservices_from_worker: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'appservice') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'appservice') | list | length > 0) else '' }}"
|
||||
|
||||
# matrix_synapse_workers_user_dir_workers_count can only be 0 or 1. More instances are not supported.
|
||||
# user_dir workers were deprecated since Synapse v1.59 (see: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types).
|
||||
# Our implementation uses generic worker services and assigns them to perform appservice work using the `update_user_directory_from_worker` Synapse option.
|
||||
matrix_synapse_workers_user_dir_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['user_dir_workers_count'] }}"
|
||||
matrix_synapse_workers_user_dir_workers_port_range_start: 18661
|
||||
matrix_synapse_workers_user_dir_workers_metrics_range_start: 19661
|
||||
|
||||
# matrix_synapse_update_user_directory_from_worker populates the `update_user_directory_from_worker` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
|
||||
# `update_user_directory_from_worker` is meant to point to a worker, which is dedicated to updating the user directory and servicing some user directory URL endpoints (`matrix_synapse_workers_user_dir_worker_client_server_endpoints`).
|
||||
matrix_synapse_update_user_directory_from_worker: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list | length > 0) else '' }}"
|
||||
|
||||
# Default list of workers to spawn.
|
||||
#
|
||||
# Unless you populate this manually, this list is dynamically generated
|
||||
|
|
|
|||
|
|
@ -58,8 +58,6 @@
|
|||
matrix_synapse_federation_sender_instances: "{{ matrix_synapse_federation_sender_instances + [item.ansible_facts.worker.name] }}"
|
||||
with_items: "{{ matrix_synapse_workers_list_results_federation_sender_workers.results }}"
|
||||
|
||||
- debug: var="matrix_synapse_federation_sender_instances"
|
||||
|
||||
# This type of worker can only have a count of 1, at most
|
||||
- name: Build pusher workers
|
||||
ansible.builtin.set_fact:
|
||||
|
|
@ -81,13 +79,27 @@
|
|||
id: "appservice-{{ item }}"
|
||||
name: "matrix-synapse-worker-appservice-{{ item }}"
|
||||
type: 'appservice'
|
||||
app: 'appservice'
|
||||
app: 'generic_worker'
|
||||
webserving: false
|
||||
port: 0
|
||||
metrics_port: "{{ matrix_synapse_workers_appservice_workers_metrics_range_start + item }}"
|
||||
register: "matrix_synapse_workers_list_results_appservice_workers"
|
||||
loop: "{{ range(0, matrix_synapse_workers_appservice_workers_count | int) | list }}"
|
||||
|
||||
# This type of worker can only have a count of 1, at most
|
||||
- name: Build user_dir workers
|
||||
ansible.builtin.set_fact:
|
||||
worker:
|
||||
id: "user-dir-{{ item }}"
|
||||
name: "matrix-synapse-worker-user-dir-{{ item }}"
|
||||
type: 'user_dir'
|
||||
app: 'generic_worker'
|
||||
webserving: true
|
||||
port: "{{ matrix_synapse_workers_user_dir_workers_port_range_start + item }}"
|
||||
metrics_port: "{{ matrix_synapse_workers_user_dir_workers_metrics_range_start + item }}"
|
||||
register: "matrix_synapse_workers_list_results_user_dir_workers"
|
||||
loop: "{{ range(0, matrix_synapse_workers_user_dir_workers_count | int) | list }}"
|
||||
|
||||
- name: Build media_repository workers
|
||||
ansible.builtin.set_fact:
|
||||
worker:
|
||||
|
|
@ -115,6 +127,8 @@
|
|||
+
|
||||
matrix_synapse_workers_list_results_appservice_workers.results
|
||||
+
|
||||
matrix_synapse_workers_list_results_user_dir_workers.results
|
||||
+
|
||||
matrix_synapse_workers_list_results_media_repository_workers.results
|
||||
}}
|
||||
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@
|
|||
matrix_synapse_webserving_workers_systemd_services_list: "{{ matrix_synapse_webserving_workers_systemd_services_list + [matrix_synapse_worker_details.name + '.service'] }}"
|
||||
when: matrix_synapse_worker_details.webserving | bool
|
||||
|
||||
# Inject stream writers and various other background workers into the instance map.
|
||||
# Inject stream writers into the instance map.
|
||||
- ansible.builtin.set_fact:
|
||||
matrix_synapse_instance_map: "{{ matrix_synapse_instance_map | combine({matrix_synapse_worker_details.name: {'host': matrix_synapse_worker_details.name, 'port': matrix_synapse_worker_details.replication_port}}) }}"
|
||||
when: matrix_synapse_worker_details.type in matrix_synapse_known_instance_map_eligible_worker_types
|
||||
|
|
|
|||
|
|
@ -12,24 +12,14 @@
|
|||
- "matrix_synapse_database_password"
|
||||
- "matrix_synapse_database_database"
|
||||
|
||||
- name: Fail if asking to configure deprecated workers (appservice, userdir)
|
||||
ansible.builtin.fail:
|
||||
msg: >-
|
||||
`{{ item }}` cannot be more than 0.
|
||||
This type of worker has been deprecated since Synapse v1.59.
|
||||
Please remove your `{{ item }}` configuration to solve this problem.
|
||||
See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
|
||||
when: "vars[item]|int != 0"
|
||||
with_items:
|
||||
- "matrix_synapse_workers_appservice_workers_count"
|
||||
- "matrix_synapse_workers_user_dir_workers_count"
|
||||
|
||||
- name: Fail if asking for more than 1 instance of single-instance workers
|
||||
ansible.builtin.fail:
|
||||
msg: >-
|
||||
`{{ item }}` cannot be more than 1. This is a single-instance worker.
|
||||
when: "vars[item]|int > 1"
|
||||
with_items:
|
||||
- "matrix_synapse_workers_appservice_workers_count"
|
||||
- "matrix_synapse_workers_user_dir_workers_count"
|
||||
- "matrix_synapse_workers_stream_writer_typing_stream_workers_count"
|
||||
- "matrix_synapse_workers_stream_writer_to_device_stream_workers_count"
|
||||
- "matrix_synapse_workers_stream_writer_account_data_stream_workers_count"
|
||||
|
|
|
|||
|
|
@ -2874,6 +2874,14 @@ instance_map: {{ matrix_synapse_instance_map | to_json }}
|
|||
# typing: worker1
|
||||
stream_writers: {{ matrix_synapse_stream_writers | to_json }}
|
||||
|
||||
{% if matrix_synapse_notify_appservices_from_worker != '' %}
|
||||
notify_appservices_from_worker: {{ matrix_synapse_notify_appservices_from_worker | to_json }}
|
||||
{% endif %}
|
||||
|
||||
{% if matrix_synapse_update_user_directory_from_worker != '' %}
|
||||
update_user_directory_from_worker: {{ matrix_synapse_update_user_directory_from_worker | to_json }}
|
||||
{% endif %}
|
||||
|
||||
# The worker that is used to run background tasks (e.g. cleaning up expired
|
||||
# data). If not provided this defaults to the main process.
|
||||
#
|
||||
|
|
|
|||
|
|
@ -64,6 +64,11 @@ matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoi
|
|||
matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints:
|
||||
- ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
|
||||
|
||||
# matrix_synapse_workers_user_dir_worker_client_server_endpoints contains the endpoints serviced by the `type = user_dir` (`app = generic_worker`) worker.
|
||||
# See: https://matrix-org.github.io/synapse/latest/workers.html#updating-the-user-directory
|
||||
matrix_synapse_workers_user_dir_worker_client_server_endpoints:
|
||||
- ^/_matrix/client/(r0|v3|unstable)/user_directory/search$
|
||||
|
||||
# matrix_synapse_workers_known_stream_writer_stream_types contains the list of stream writer stream types that the playbook recognizes.
|
||||
# This is used for validation purposes. If adding support for a new type, besides adding it to this list,
|
||||
# don't forget to actually configure it where appropriate (see worker.yaml.j2`, the nginx proxy configuration, etc).
|
||||
|
|
@ -79,13 +84,25 @@ matrix_synapse_workers_webserving_stream_writer_types: ['typing', 'to_device', '
|
|||
matrix_synapse_webserving_workers_systemd_services_list: []
|
||||
|
||||
# matrix_synapse_known_worker_types contains the list of known worker types.
|
||||
#
|
||||
# A worker type is different than a worker app (e.g. `generic_worker`).
|
||||
# For example, the `stream_writer` worker type is served by the `generic_worker` app, but is a separate type that we recognize.
|
||||
#
|
||||
# Some other types (`appservice` and `user_dir`) used to be Synapse worker apps, which got subsequently deprecated.
|
||||
# We still allow these types of workers and map them to the `generic_worker` app,
|
||||
# which is why we make sure they're part of the list below.
|
||||
# We use the `unique` filter because they're part of `matrix_synapse_workers_avail_list` too (for now; scheduled for removal).
|
||||
matrix_synapse_known_worker_types: |
|
||||
{{
|
||||
(
|
||||
matrix_synapse_workers_avail_list
|
||||
+
|
||||
['stream_writer']
|
||||
+
|
||||
['appservice']
|
||||
+
|
||||
['user_dir']
|
||||
) | unique
|
||||
}}
|
||||
|
||||
# matrix_synapse_known_instance_map_eligible_worker_types contains the list of worker types that are to be injected into `matrix_synapse_instance_map`.
|
||||
|
|
|
|||
|
|
@ -90,7 +90,9 @@ matrix_synapse_workers_generic_worker_endpoints:
|
|||
# - ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
|
||||
|
||||
# User directory search requests
|
||||
- ^/_matrix/client/(r0|v3|unstable)/user_directory/search$
|
||||
# Any worker can handle these, but we have a dedicated user_dir worker for this,
|
||||
# so we'd like for other generic workers to not try and capture these requests.
|
||||
# - ^/_matrix/client/(r0|v3|unstable)/user_directory/search$
|
||||
|
||||
# Additionally, the following REST endpoints can be handled for GET requests:
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue