diff --git a/CHANGELOG.md b/CHANGELOG.md index 92fdbc49..9d416714 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,19 @@ +# 2023-03-12 + +## synapse-auto-compressor support + +Thanks to [Aine](https://gitlab.com/etke.cc) of [etke.cc](https://etke.cc/), the playbook can now set up [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state)'s `synapse_auto_compressor` tool to run periodically. + +If enabled, `synapse_auto_compressor` runs on a schedule and compresses your Synapse database's `state_groups` table. It was possible to run `rust-synapse-compress-state` manually via the playbook even before - see [Compressing state with rust-synapse-compress-state](docs/maintenance-synapse.md#compressing-state-with-rust-synapse-compress-state). However, using `synapse_auto_compressor` is better, because: + +- it runs on a more up-to-date version of `rust-synapse-compress-state` +- it's a set-it-and-forget-it tool that you can enable and never have to deal with manual compression anymore + +This tool needs to be enabled manually, for now. In the future, we're considering enabling it by default for all Synapse installations. + +See our [Setting up synapse-auto-compressor](docs/configuring-playbook-synapse-auto-compressor.md) documentation to get started. + + # 2023-03-07 ## Sliding Sync Proxy (Element X) support diff --git a/README.md b/README.md index 185c4085..ff93cd1f 100644 --- a/README.md +++ b/README.md @@ -163,6 +163,7 @@ Various services that don't fit any other category. | Name | Default? | Description | Documentation | | ---- | -------- | ----------- | ------------- | | [sliding-sync](https://github.com/matrix-org/sliding-sync)| x | Sliding Sync support for clients which require it (e.g. Element X) | [Link](docs/configuring-playbook-sliding-sync-proxy.md) | +| [synapse_auto_compressor](https://github.com/matrix-org/rust-synapse-compress-state/#automated-tool-synapse_auto_compressor) | x | A cli tool that automatically compresses `state_groups` database table in background. | [Link](docs/configuring-playbook-synapse-auto-compressor.md) | | [synapse-simple-antispam](https://github.com/t2bot/synapse-simple-antispam) (advanced) | x | A spam checker module | [Link](docs/configuring-playbook-synapse-simple-antispam.md) | | [Matrix Corporal](https://github.com/devture/matrix-corporal) (advanced) | x | Reconciliator and gateway for a managed Matrix server | [Link](docs/configuring-playbook-matrix-corporal.md) | | [Etherpad](https://etherpad.org) | x | An open source collaborative text editor | [Link](docs/configuring-playbook-etherpad.md) | diff --git a/docs/configuring-playbook-synapse-auto-compressor.md b/docs/configuring-playbook-synapse-auto-compressor.md new file mode 100644 index 00000000..108b6a11 --- /dev/null +++ b/docs/configuring-playbook-synapse-auto-compressor.md @@ -0,0 +1,36 @@ +# Setting up synapse_auto_compressor + +The playbook can install and configure [synapse_auto_compressor](https://github.com/matrix-org/rust-synapse-compress-state/#automated-tool-synapse_auto_compressor) for you. + +It's a CLI tool that automatically compresses Synapse's `state_groups` database table in the background. + +See the project's [documentation](https://github.com/matrix-org/rust-synapse-compress-state/#automated-tool-synapse_auto_compressor) to learn what it does and why it might be useful to you. + + +## Adjusting the playbook configuration + +Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file: + +```yaml +matrix_synapse_auto_compressor_enabled: true +``` + + +## Installing + +After configuring the playbook, run the [installation](installing.md) command again: + +``` +ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start +``` + + +## Usage + +After installation, `synapse_auto_compressor` will run automatically every day at `00:00:00` (as defined in `matrix_synapse_auto_compressor_calendar` by default). + +## Manually start the tool + +For testing your setup it can be helpful to not wait until 00:00. If you want to run the tool immediately, log onto the server +and run `systemctl start matrix-synapse-auto-compressor`. Running this command will not return control to your terminal until the compression run is done, which may take a long time. +Consider using [tmux](https://en.wikipedia.org/wiki/Tmux) if your SSH connection is unstable. diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index 68a2658a..a65858f0 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -195,6 +195,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins ### Other specialized services +- [Setting up synapse-auto-compressor](configuring-playbook-synapse-auto-compressor.md) for compressing the database on Synapse homeservers (optional) + - [Setting up the Sliding Sync Proxy](configuring-playbook-sliding-sync-proxy.md) for clients which require Sliding Sync support (like Element X) (optional) - [Setting up the Sygnal push gateway](configuring-playbook-sygnal.md) (optional) diff --git a/docs/maintenance-synapse.md b/docs/maintenance-synapse.md index a51811ff..7c3ecc1b 100644 --- a/docs/maintenance-synapse.md +++ b/docs/maintenance-synapse.md @@ -29,7 +29,9 @@ After deleting data, you may wish to run a [`FULL` Postgres `VACUUM`](./maintena [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state) can be used to optimize some `_state` tables used by Synapse. If your server participates in large rooms this is the most effective way to reduce the size of your database. -This tool should be safe to use (even when Synapse is running), but it's always a good idea to [make Postgres backups](./maintenance-postgres.md#backing-up-postgresql) first. +**Note**: besides running the `rust-synapse-compress-state` tool manually, you can also enable its `synapse-auto-compressor` tool by [Setting up synapse-auto-compressor](configuring-playbook-synapse-auto-compressor.md). The automatic tool will run on a schedule every day and you won't have to compress state manually ever again. + +`rust-synapse-compress-state` should be safe to use (even when Synapse is running), but it's always a good idea to [make Postgres backups](./maintenance-postgres.md#backing-up-postgresql) first. To ask the playbook to run rust-synapse-compress-state, execute: diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 85fbfd2a..18ad756c 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -338,6 +338,8 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-synapse-s3-storage-provider-migrate.timer', 'priority': 5000, 'groups': ['matrix']}] if matrix_synapse_ext_synapse_s3_storage_provider_enabled else []) + + ([{'name': 'matrix-synapse-auto-compressor.timer', 'priority': 5000, 'groups': ['matrix', 'synapse-auto-compressor']}] if matrix_synapse_auto_compressor_enabled else []) + + ([{'name': 'matrix-synapse-admin.service', 'priority': 4000, 'groups': ['matrix', 'synapse-admin']}] if matrix_synapse_admin_enabled else []) + ([{'name': 'matrix-synapse-reverse-proxy-companion.service', 'priority': 1500, 'groups': ['matrix', 'homeservers', 'synapse', 'reverse-proxies']}] if matrix_synapse_reverse_proxy_companion_enabled else []) @@ -2716,15 +2718,15 @@ devture_postgres_managed_databases_auto: | }] if (matrix_synapse_enabled and matrix_synapse_database_host == devture_postgres_connection_hostname) else []) + ([{ - 'name': matrix_dendrite_federationapi_database, + 'name': matrix_dendrite_federation_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_keyserver_database, + 'name': matrix_dendrite_key_server_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_mediaapi_database, + 'name': matrix_dendrite_media_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ @@ -2732,15 +2734,15 @@ devture_postgres_managed_databases_auto: | 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_syncapi_database, + 'name': matrix_dendrite_sync_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_userapi_database, + 'name': matrix_dendrite_user_api_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ - 'name': matrix_dendrite_pushserver_database, + 'name': matrix_dendrite_push_server_database, 'username': matrix_dendrite_database_user, 'password': matrix_dendrite_database_password, },{ @@ -3326,6 +3328,43 @@ matrix_synapse_app_service_config_files_auto: "{{ matrix_homeserver_app_service_ # ###################################################################### + +###################################################################### +# +# matrix-synapse-auto-compressor +# +###################################################################### + +# Not enabled by default, for now +matrix_synapse_auto_compressor_enabled: false + +matrix_synapse_auto_compressor_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}" + +matrix_synapse_auto_compressor_container_additional_networks: | + {{ + ([devture_postgres_container_network] if devture_postgres_enabled and devture_postgres_container_network != matrix_synapse_auto_compressor_container_network else []) + }} + +matrix_synapse_auto_compressor_database_username: "{{ matrix_synapse_database_user if matrix_synapse_enabled else '' }}" +matrix_synapse_auto_compressor_database_password: "{{ matrix_synapse_database_password if matrix_synapse_enabled else '' }}" +matrix_synapse_auto_compressor_database_hostname: "{{ matrix_synapse_database_host if matrix_synapse_enabled else '' }}" +matrix_synapse_auto_compressor_database_port: "{{ matrix_synapse_database_port if matrix_synapse_enabled else '5432' }}" +matrix_synapse_auto_compressor_database_name: "{{ matrix_synapse_database_database if matrix_synapse_enabled else '' }}" + +matrix_synapse_auto_compressor_systemd_required_services_list: | + {{ + ['docker.service'] + + + ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled else []) + }} + +###################################################################### +# +# /matrix-synapse-auto-compressor +# +###################################################################### + + ###################################################################### # # matrix-synapse-reverse-proxy-companion diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index a2b7f2b8..b8964aa5 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -84,6 +84,7 @@ - custom/matrix-cactus-comments - custom/matrix-rageshake - custom/matrix-synapse + - custom/matrix-synapse-auto-compressor - custom/matrix-synapse-reverse-proxy-companion - custom/matrix-dendrite - custom/matrix-conduit diff --git a/requirements.yml b/requirements.yml index 05c5aa70..e841cb4e 100644 --- a/requirements.yml +++ b/requirements.yml @@ -31,7 +31,7 @@ version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 - src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git - version: v1.5.0-2 + version: v1.5.0-3 - src: git+https://gitlab.com/etke.cc/roles/prometheus_postgres_exporter.git version: v0.11.1-2 diff --git a/roles/custom/matrix-dendrite/defaults/main.yml b/roles/custom/matrix-dendrite/defaults/main.yml index 8c2dded4..35793f49 100644 --- a/roles/custom/matrix-dendrite/defaults/main.yml +++ b/roles/custom/matrix-dendrite/defaults/main.yml @@ -6,7 +6,7 @@ matrix_dendrite_enabled: true matrix_dendrite_docker_image: "{{ matrix_dendrite_docker_image_name_prefix }}matrixdotorg/dendrite-monolith:{{ matrix_dendrite_docker_image_tag }}" matrix_dendrite_docker_image_name_prefix: "docker.io/" -matrix_dendrite_docker_image_tag: "v0.11.1" +matrix_dendrite_docker_image_tag: "v0.12.0" matrix_dendrite_docker_image_force_pull: "{{ matrix_dendrite_docker_image.endswith(':latest') }}" matrix_dendrite_base_path: "{{ matrix_base_data_path }}/dendrite" @@ -150,13 +150,13 @@ matrix_dendrite_database_str: "postgresql://{{ matrix_dendrite_database_user }}: matrix_dendrite_database_hostname: '' matrix_dendrite_database_user: "dendrite" matrix_dendrite_database_password: "itsasecret" -matrix_dendrite_federationapi_database: "dendrite_federationapi" -matrix_dendrite_keyserver_database: "dendrite_keyserver" -matrix_dendrite_mediaapi_database: "dendrite_mediaapi" +matrix_dendrite_federation_api_database: "dendrite_federationapi" +matrix_dendrite_key_server_database: "dendrite_keyserver" +matrix_dendrite_media_api_database: "dendrite_mediaapi" matrix_dendrite_room_database: "dendrite_room" -matrix_dendrite_syncapi_database: "dendrite_syncapi" -matrix_dendrite_userapi_database: "dendrite_userapi" -matrix_dendrite_pushserver_database: "dendrite_pushserver" +matrix_dendrite_sync_api_database: "dendrite_syncapi" +matrix_dendrite_user_api_database: "dendrite_userapi" +matrix_dendrite_push_server_database: "dendrite_pushserver" matrix_dendrite_mscs_database: "dendrite_mscs" matrix_dendrite_client_api_turn_uris: [] @@ -206,8 +206,15 @@ matrix_dendrite_configuration_extension: "{{ matrix_dendrite_configuration_exten # You most likely don't need to touch this variable. Instead, see `matrix_dendrite_configuration_yaml`. matrix_dendrite_configuration: "{{ matrix_dendrite_configuration_yaml | from_yaml | combine(matrix_dendrite_configuration_extension, recursive=True) }}" -matrix_dendrite_userapi_auto_join_rooms: [] +matrix_dendrite_user_api_auto_join_rooms: [] # statistics reporting configuration. These statistics contain the server # name, number of active users and some information on your deployment config. matrix_dendrite_report_stats: false + +# Contorls whether thumbnails for media content are generated dynamically +matrix_dendrite_media_api_dynamic_thumbnails: false +matrix_dendrite_media_api_max_thumbnail_generators: 10 + +# Controls whether the full-text search engine is enabled +matrix_dendrite_sync_api_search_enabled: false diff --git a/roles/custom/matrix-dendrite/tasks/validate_config.yml b/roles/custom/matrix-dendrite/tasks/validate_config.yml index 53c17498..1f1b3e41 100644 --- a/roles/custom/matrix-dendrite/tasks/validate_config.yml +++ b/roles/custom/matrix-dendrite/tasks/validate_config.yml @@ -27,3 +27,10 @@ - {'old': 'matrix_dendrite_rate_limiting_enabled', 'new': 'matrix_dendrite_client_api_rate_limiting_enabled'} - {'old': 'matrix_dendrite_rate_limiting_threshold', 'new': 'matrix_dendrite_client_api_rate_limiting_threshold'} - {'old': 'matrix_dendrite_rate_limiting_cooloff_ms', 'new': 'matrix_dendrite_client_api_rate_limiting_cooloff_ms'} + - {'old': 'matrix_dendrite_mediaapi_database', 'new': 'matrix_dendrite_media_api_database'} + - {'old': 'matrix_dendrite_syncapi_database', 'new': 'matrix_dendrite_sync_api_database'} + - {'old': 'matrix_dendrite_userapi_database', 'new': 'matrix_dendrite_user_api_database'} + - {'old': 'matrix_dendrite_userapi_auto_join_rooms', 'new': 'matrix_dendrite_user_api_auto_join_rooms'} + - {'old': 'matrix_dendrite_federationapi_database', 'new': 'matrix_dendrite_federation_api_database'} + - {'old': 'matrix_dendrite_pushserver_database', 'new': 'matrix_dendrite_push_server_database'} + - {'old': 'matrix_dendrite_keyserver_database', 'new': 'matrix_dendrite_key_server_database'} diff --git a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 index 005f9bd6..89821de1 100644 --- a/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 +++ b/roles/custom/matrix-dendrite/templates/dendrite/dendrite.yaml.j2 @@ -223,7 +223,7 @@ federation_api: external_api: listen: http://0.0.0.0:8072 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_federationapi_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_federation_api_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -266,7 +266,7 @@ key_server: listen: http://0.0.0.0:7779 connect: http://key_server:7779 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_keyserver_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_key_server_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -279,7 +279,7 @@ media_api: external_api: listen: http://0.0.0.0:8074 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_mediaapi_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_media_api_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -293,10 +293,10 @@ media_api: max_file_size_bytes: {{ matrix_dendrite_max_file_size_bytes|to_json }} # Whether to dynamically generate thumbnails if needed. - dynamic_thumbnails: false + dynamic_thumbnails: {{ matrix_dendrite_media_api_dynamic_thumbnails|to_json }} # The maximum number of simultaneous thumbnail generators to run. - max_thumbnail_generators: 10 + max_thumbnail_generators: {{ matrix_dendrite_media_api_max_thumbnail_generators|to_json }} # A list of thumbnail sizes to be generated for media content. thumbnail_sizes: @@ -342,7 +342,7 @@ sync_api: external_api: listen: http://0.0.0.0:8073 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_syncapi_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_sync_api_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -355,7 +355,7 @@ sync_api: # Configuration for the full-text search engine. search: # Whether or not search is enabled. - enabled: false + enabled: {{ matrix_dendrite_sync_api_search_enabled|to_json }} # The path where the search index will be created in. index_path: "/matrix-media-store-parent/searchindex" # The language most likely to be used on the server - used when indexing, to @@ -376,7 +376,7 @@ user_api: listen: http://0.0.0.0:7781 connect: http://user_api:7781 account_database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_userapi_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_user_api_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 @@ -385,7 +385,7 @@ user_api: # is considered to be valid in milliseconds. # The default lifetime is 3600000ms (60 minutes). # openid_token_lifetime_ms: 3600000 - auto_join_rooms: {{ matrix_dendrite_userapi_auto_join_rooms | to_json }} + auto_join_rooms: {{ matrix_dendrite_user_api_auto_join_rooms | to_json }} # Not in dendrite-config.yaml, but is in build/docker/config/dendrite.yaml # Configuration for the Push Server API. @@ -394,7 +394,7 @@ push_server: listen: http://localhost:7782 connect: http://localhost:7782 database: - connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_pushserver_database }}?sslmode=disable + connection_string: {{ matrix_dendrite_database_str }}/{{ matrix_dendrite_push_server_database }}?sslmode=disable max_open_conns: 10 max_idle_conns: 2 conn_max_lifetime: -1 diff --git a/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml new file mode 100644 index 00000000..7b5ea54d --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/defaults/main.yml @@ -0,0 +1,64 @@ +--- + +# synapse_auto_compressor tool +# Project source code URL: https://github.com/matrix-org/rust-synapse-compress-state + +matrix_synapse_auto_compressor_enabled: true + +matrix_synapse_auto_compressor_version: v0.1.3 + +matrix_synapse_auto_compressor_base_path: "{{ matrix_base_data_path }}/synapse-auto-compressor" +matrix_synapse_auto_compressor_container_src_files_path: "{{ matrix_synapse_auto_compressor_base_path }}/container-src" + +matrix_synapse_auto_compressor_container_image_self_build: false +matrix_synapse_auto_compressor_container_image_self_build_repo: "https://github.com/matrix-org/rust-synapse-compress-state.git" +matrix_synapse_auto_compressor_container_image_self_build_version: "{{ 'main' if matrix_synapse_auto_compressor_version == 'latest' else matrix_synapse_auto_compressor_version }}" + +matrix_synapse_auto_compressor_container_image: "{{ matrix_synapse_auto_compressor_container_image_name_prefix }}etke.cc/rust-synapse-compress-state:{{ matrix_synapse_auto_compressor_version }}" +matrix_synapse_auto_compressor_container_image_name_prefix: "{{ 'localhost/' if matrix_synapse_auto_compressor_container_image_self_build else 'registry.gitlab.com/' }}" +matrix_synapse_auto_compressor_container_image_force_pull: "{{ matrix_synapse_auto_compressor_container_image.endswith(':latest') }}" + +# The base container network. It will be auto-created by this role if it doesn't exist already. +matrix_synapse_auto_compressor_container_network: matrix-synapse-auto-compressor + +# A list of additional container networks that the container would be connected to. +# The role does not create these networks, so make sure they already exist. +# Use this to expose this container to another reverse proxy, which runs in a different container network. +matrix_synapse_auto_compressor_container_additional_networks: [] + +# A list of extra arguments to pass to the container +matrix_synapse_auto_compressor_container_extra_arguments: [] + +# List of systemd services that matrix-synapse-auto-compressor.service depends on +matrix_synapse_auto_compressor_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-synapse-auto-compressor.service wants +matrix_synapse_auto_compressor_systemd_wanted_services_list: [] + +matrix_synapse_auto_compressor_database_username: 'synapse' +matrix_synapse_auto_compressor_database_password: '' +matrix_synapse_auto_compressor_database_hostname: '' +matrix_synapse_auto_compressor_database_port: 5432 +matrix_synapse_auto_compressor_database_name: 'synapse' + +# connection string to synapse database (postgres only) +matrix_synapse_auto_compressor_synapse_database: 'postgres://{{ matrix_synapse_auto_compressor_database_username | urlencode() }}:{{ matrix_synapse_auto_compressor_database_password | urlencode() }}@{{ matrix_synapse_auto_compressor_database_hostname }}:{{ matrix_synapse_auto_compressor_database_port }}/{{ matrix_synapse_auto_compressor_database_name }}' + +# systemd calendar configuration for the compressor job +matrix_synapse_auto_compressor_calendar: "*-*-* 00:00:00" + +# The number of state groups to work on at once. +# All of the entries from state_groups_state are requested from the database for state groups that are worked on. +# Therefore small chunk sizes may be needed on machines with low memory. +# Note: if the compressor fails to find space savings on the chunk as a whole +# (which may well happen in rooms with lots of backfill in) then the entire chunk is skipped. +matrix_synapse_auto_compressor_chunk_size: 500 + +# CHUNKS_TO_COMPRESS chunks of size CHUNK_SIZE will be compressed. +# The higher this number is set to, the longer the compressor will run for. +matrix_synapse_auto_compressor_chunks_to_compress: 100 + +matrix_synapse_auto_compressor_command: "synapse_auto_compressor -p $POSTGRES_LOCATION -c {{ matrix_synapse_auto_compressor_chunk_size }} -n {{ matrix_synapse_auto_compressor_chunks_to_compress }}" + +# Controls the POSTGRES_LOCATION environment variable +matrix_synapse_auto_compressor_environment_variable_postgres_location: "{{ matrix_synapse_auto_compressor_synapse_database }}" diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml new file mode 100644 index 00000000..be4d5aad --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/install.yml @@ -0,0 +1,72 @@ +--- + +- name: Ensure synapse-auto-compressor paths exist + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + when: item.when | bool + with_items: + - path: "{{ matrix_synapse_auto_compressor_base_path }}" + when: true + - path: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" + when: "{{ matrix_synapse_auto_compressor_container_image_self_build }}" + +- name: Ensure synapse-auto-compressor labels installed + ansible.builtin.template: + src: "{{ role_path }}/templates/env.j2" + dest: "{{ matrix_synapse_auto_compressor_base_path }}/env" + mode: 0640 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure synapse-auto-compressor image is pulled + community.docker.docker_image: + name: "{{ matrix_synapse_auto_compressor_container_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_synapse_auto_compressor_container_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_synapse_auto_compressor_container_image_force_pull }}" + when: "not matrix_synapse_auto_compressor_container_image_self_build | bool" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure synapse-auto-compressor repository is present on self-build + ansible.builtin.git: + repo: "{{ matrix_synapse_auto_compressor_container_image_self_build_repo }}" + version: "{{ matrix_synapse_auto_compressor_container_image_self_build_version }}" + dest: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" + force: "yes" + become: true + become_user: "{{ matrix_user_username }}" + register: matrix_synapse_auto_compressor_git_pull_results + when: "matrix_synapse_auto_compressor_container_image_self_build | bool" + +- name: Ensure synapse-auto-compressor image is built + community.docker.docker_image: + name: "{{ matrix_synapse_auto_compressor_container_image }}" + source: build + force_source: "{{ matrix_synapse_auto_compressor_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mailer_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_synapse_auto_compressor_container_src_files_path }}" + pull: true + when: "matrix_synapse_auto_compressor_container_image_self_build | bool" + +- name: Ensure matrix-synapse-auto-compressor container network is created + community.general.docker_network: + name: "{{ matrix_synapse_auto_compressor_container_network }}" + driver: bridge + +- name: Ensure matrix-synapse-auto-compressor systemd service and timer are installed + ansible.builtin.template: + src: "{{ role_path }}/templates/matrix-synapse-auto-compressor.{{ item }}.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-auto-compressor.{{ item }}" + mode: 0644 + with_items: + - service + - timer diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml new file mode 100644 index 00000000..2cce1f12 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/main.yml @@ -0,0 +1,20 @@ +--- + +- block: + - when: matrix_synapse_auto_compressor_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - when: matrix_synapse_auto_compressor_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/install.yml" + tags: + - setup-all + - setup-synapse-auto-compressor + - install-all + - install-synapse-auto-compressor + +- block: + - when: not matrix_synapse_auto_compressor_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/uninstall.yml" + tags: + - setup-all + - setup-synapse-auto-compressor diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/uninstall.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/uninstall.yml new file mode 100644 index 00000000..075f3c10 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/uninstall.yml @@ -0,0 +1,29 @@ +--- +- name: Check existence of matrix-synapse-auto-compressor service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-auto-compressor.service" + register: matrix_synapse_auto_compressor_service_stat + +- when: matrix_synapse_auto_compressor_service_stat.stat.exists | bool + block: + - name: Ensure matrix-synapse-auto-compressor is stopped + ansible.builtin.service: + name: matrix-synapse-auto-compressor + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-synapse-auto-compressor.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-auto-compressor.service" + state: absent + + - name: Ensure matrix-synapse-auto-compressor.timer doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-synapse-auto-compressor.timer" + state: absent + + - name: Ensure Matrix synapse-auto-compressor paths don't exist + ansible.builtin.file: + path: "{{ matrix_synapse_auto_compressor_base_path }}" + state: absent diff --git a/roles/custom/matrix-synapse-auto-compressor/tasks/validate_config.yml b/roles/custom/matrix-synapse-auto-compressor/tasks/validate_config.yml new file mode 100644 index 00000000..b0dbfab0 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/tasks/validate_config.yml @@ -0,0 +1,10 @@ +--- + +- name: Fail if required matrix-synapse-auto-compressor settings not defined + ansible.builtin.fail: + msg: > + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - matrix_synapse_auto_compressor_database_hostname + - matrix_synapse_auto_compressor_database_password diff --git a/roles/custom/matrix-synapse-auto-compressor/templates/env.j2 b/roles/custom/matrix-synapse-auto-compressor/templates/env.j2 new file mode 100644 index 00000000..27fb1dd8 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/templates/env.j2 @@ -0,0 +1 @@ +POSTGRES_LOCATION={{ matrix_synapse_auto_compressor_environment_variable_postgres_location }} diff --git a/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 new file mode 100644 index 00000000..f530d5b2 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2 @@ -0,0 +1,46 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Synapse State Auto Compressor +{% for service in matrix_synapse_auto_compressor_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_synapse_auto_compressor_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=oneshot +Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-auto-compressor 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-auto-compressor 2>/dev/null || true' + +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \ + --rm \ + --name=matrix-synapse-auto-compressor \ + --log-driver=none \ + --cap-drop=ALL \ + --read-only \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --network={{ matrix_synapse_auto_compressor_container_network }} \ + --env-file={{ matrix_synapse_auto_compressor_base_path }}/env \ + --entrypoint=/bin/sh \ + {% for arg in matrix_synapse_auto_compressor_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_synapse_auto_compressor_container_image }} \ + -c '{{ matrix_synapse_auto_compressor_command }}' + +{% for network in matrix_synapse_auto_compressor_container_additional_networks %} +ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-synapse-auto-compressor +{% endfor %} + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach matrix-synapse-auto-compressor + +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-synapse-auto-compressor 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-auto-compressor 2>/dev/null || true' +SyslogIdentifier=matrix-synapse-auto-compressor + +[Install] +WantedBy=multi-user.target diff --git a/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.timer.j2 b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.timer.j2 new file mode 100644 index 00000000..1d7b16d3 --- /dev/null +++ b/roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.timer.j2 @@ -0,0 +1,9 @@ +[Unit] +Description=Synapse State Auto Compressor Timer + +[Timer] +Unit=matrix-synapse-auto-compressor.service +OnCalendar={{ matrix_synapse_auto_compressor_calendar }} + +[Install] +WantedBy=timers.target diff --git a/roles/custom/matrix-synapse/defaults/main.yml b/roles/custom/matrix-synapse/defaults/main.yml index 41ad0bc4..a5c7106f 100644 --- a/roles/custom/matrix-synapse/defaults/main.yml +++ b/roles/custom/matrix-synapse/defaults/main.yml @@ -4,7 +4,7 @@ matrix_synapse_enabled: true -matrix_synapse_version: v1.78.0 +matrix_synapse_version: v1.79.0 matrix_synapse_username: '' matrix_synapse_uid: '' diff --git a/roles/custom/matrix-synapse/vars/main.yml b/roles/custom/matrix-synapse/vars/main.yml index 46277c87..faaf6ed6 100644 --- a/roles/custom/matrix-synapse/vars/main.yml +++ b/roles/custom/matrix-synapse/vars/main.yml @@ -163,7 +163,9 @@ matrix_synapse_workers_generic_worker_endpoints: - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$ - ^/_matrix/client/v1/rooms/.*/timestamp_to_event$ + - ^/_matrix/client/(api/v1|r0|v3|unstable/.*)/rooms/.*/aliases - ^/_matrix/client/(api/v1|r0|v3|unstable)/search$ + - ^/_matrix/client/(r0|v3|unstable)/user/.*/filter(/|$) # Encryption requests # Note that ^/_matrix/client/(r0|v3|unstable)/keys/upload/ requires `worker_main_http_uri` @@ -184,6 +186,7 @@ matrix_synapse_workers_generic_worker_endpoints: - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/join/ + - ^/_matrix/client/(api/v1|r0|v3|unstable)/knock/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/ # Start of intentionally-ignored-endpoints