This commit is contained in:
p5t2vspoqqw 2019-07-12 15:45:19 +02:00
commit 5054fff88b
18 changed files with 415 additions and 30 deletions

View file

@ -1,3 +1,31 @@
# 2019-07-08
## Synapse Maintenance docs and synapse-janitor support are available
The playbook can now help you with Synapse's maintenance.
There's a new documentation page about [Synapse maintenance](./docs/maintenance-synapse.md) and another section on [Postgres vacuuming](./docs/maintenance-postgres.md#vacuuming-postgresql).
Among other things, if your Postgres database has grown significantly over time, you may wish to [ask the playbook to purge unused data with synapse-janitor](./docs/maintenance-synapse.md#purging-unused-data-with-synapse-janitor) for you.
## (BC Break) Rename run control variables
Some internal playbook control variables have been renamed.
This change **only affects people who run this playbook's roles from another playbook**.
If you're using this playbook as-is, you're not affected and don't need to do anything.
The following variables have been renamed:
- from `run_import_postgres` to `run_postgres_import`
- from `run_import_sqlite_db` to `run_postgres_import_sqlite_db`
- from `run_upgrade_postgres` to `run_postgres_upgrade`
- from `run_import_media_store` to `run_synapse_import_media_store`
- from `run_register_user` to `run_synapse_register_user`
- from `run_update_user_password` to `run_synapse_update_user_password`
# 2019-07-04
## Synapse no longer logs to text files

View file

@ -20,6 +20,8 @@
- [Maintenance / upgrading services](maintenance-upgrading-services.md)
- [Maintenance / Synapse](maintenance-synapse.md)
- [Maintenance / PostgreSQL](maintenance-postgres.md)
- [Maintenance and Troubleshooting](maintenance-and-troubleshooting.md)

View file

@ -28,6 +28,8 @@ matrix_mailer_relay_auth_username: "another.sender@example.com"
matrix_mailer_relay_auth_password: "some-password"
```
**Note**: only the secure submission protocol (using `STARTTLS`, usually on port `587`) is supported. **SMTPS** (encrypted SMTP, usually on port `465`) **is not supported**.
## Troubleshooting

View file

@ -6,6 +6,8 @@ Table of contents:
- [Getting a database terminal](#getting-a-database-terminal), for when you wish to execute SQL queries
- [Vacuuming PostgreSQL](#vacuuming-postgresql), for when you wish to run a Postgres [VACUUM](https://www.postgresql.org/docs/current/sql-vacuum.html) (optimizing disk space)
- [Backing up PostgreSQL](#backing-up-postgresql), for when you wish to make a backup
- [Upgrading PostgreSQL](#upgrading-postgresql), for upgrading to new major versions of PostgreSQL. Such **manual upgrades are sometimes required**.
@ -18,6 +20,19 @@ You can use the `/usr/local/bin/matrix-postgres-cli` tool to get interactive ter
If you are using an [external Postgres server](configuring-playbook-external-postgres.md), the above tool will not be available.
## Vacuuming PostgreSQL
To perform a `FULL` Postgres [VACUUM](https://www.postgresql.org/docs/current/sql-vacuum.html), run the playbook with `--tags=run-postgres-vacuum`.
Example:
```bash
ansible-playbook -i inventory/hosts setup.yml --tags=run-postgres-vacuum,start
```
**Note**: this will automatically stop Synapse temporarily and restart it later. You'll also need plenty of available disk space in your Postgres data directory (usually `/matrix/postgres/data`).
## Backing up PostgreSQL
To make a back up of the current PostgreSQL database, make sure it's running and then execute a command like this on the server:

View file

@ -0,0 +1,72 @@
# Synapse maintenance
This document shows you how to perform various maintenance tasks related to the Synapse chat server.
Table of contents:
- [Purging unused data with synapse-janitor](#purging-unused-data-with-synapse-janitor), for when you wish to delete unused data from the Synapse database
- [Purging old data with the Purge History API](#purging-old-data-with-the-purge-history-api), for when you wish to delete in-use (but old) data from the Synapse database
- [Compressing state with rust-synapse-compress-state](#compressing-state-with-rust-synapse-compress-state), for when you wish to compress some Synapse state tables using the [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state) tool
## Purging unused data with synapse-janitor
When you **leave** and **forget** a room, Synapse can clean up its data, but currently doesn't.
This **unused and unreachable data** remains in your database forever.
There are external tools (like [synapse-janitor](https://github.com/xwiki-labs/synapse_scripts)), which are meant to solve this problem.
To ask the playbook to run synapse-janitor, execute:
```bash
ansible-playbook -i inventory/hosts setup.yml --tags=run-postgres-synapse-janitor,start
```
**Note**: this will automatically stop Synapse temporarily and restart it later.
### Vacuuming Postgres
Running synapse-janitor potentially deletes a lot of data from the Postgres database.
However, disk space only ever gets released after a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql).
It's easiest if you ask the playbook to run both synapse-janitor and a `VACUUM FULL` in one call:
```bash
ansible-playbook -i inventory/hosts setup.yml --tags=run-postgres-synapse-janitor,run-postgres-vacuum,start
```
**Note**: this will automatically stop Synapse temporarily and restart it later. You'll also need plenty of available disk space in your Postgres data directory (usually `/matrix/postgres/data`).
## Purging old data with the Purge History API
If [purging unused and unreachable data](#purging-unused-data-with-synapse-janitor) is not enough for you, you can start deleting in-use (but old) data.
**This is destructive** (especially for non-federated rooms), because it means **people will no longer have access to history past a certain point**.
Synapse provides a [Purge History API](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/purge_history_api.rst) that you can use to purge on a per-room basis.
To make use of this API, **you'll need an admin access token** first. You can find your access token in the setting of some clients (like riot-web).
Alternatively, you can log in and obtain a new access token like this:
```
curl \
--data '{"identifier": {"type": "m.id.user", "user": "YOUR_MATRIX_USERNAME" }, "password": "YOUR_MATRIX_PASSWORD", "type": "m.login.password", "device_id": "Synapse-Purge-History-API"}' \
https://matrix.DOMAIN/_matrix/client/r0/login
```
Follow the [Purge History API](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/purge_history_api.rst) documentation page for the actual purging instructions.
Don't forget that disk space only ever gets released after a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql) - something the playbook can help you with.
## Compressing state with rust-synapse-compress-state
[rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state) can be used to optimize some `_state` tables used by Synapse.
Unfortunately, at this time the playbook can't help you run this **experimental tool**.
Since it's also experimental, you may wish to stay away from it, or at least [make Postgres backups](./maintenance-postgres.md#backing-up-postgresql) first.

View file

@ -6,7 +6,7 @@
- [Python](https://www.python.org/) being installed on the server. Most distributions install Python by default, but some don't (e.g. Ubuntu 18.04) and require manual installation (something like `apt-get install python`).
- a `cron`-like tool installed on the server such as `cron` or `anacron` to automatically schedule the Let's Encrypt SSL certificates. *This can be ignored if you use your own SSL certificates.*
- a `cron`-like tool installed on the server such as `cron` or `anacron` to automatically schedule the Let's Encrypt SSL certificates's renewal. *This can be ignored if you use your own SSL certificates.*
- the [Ansible](http://ansible.com/) program being installed on your own computer. It's used to run this playbook and configures your server for you. Take a look at [our guide about Ansible](ansible.md) for more information, as well as [version requirements](ansible.md#supported-ansible-versions) and alternative ways to run Ansible.

View file

@ -43,13 +43,15 @@ matrix_docker_network: "matrix"
matrix_well_known_matrix_server_enabled: true
# Variables to Control which parts of our roles run.
run_postgres_import: true
run_postgres_upgrade: true
run_postgres_import_sqlite_db: true
run_postgres_synapse_janitor: true
run_postgres_vacuum: true
run_synapse_register_user: true
run_synapse_update_user_password: true
run_synapse_import_media_store: true
run_setup: true
run_import_postgres: true
run_upgrade_postgres: true
run_self_check: true
run_start: true
run_stop: true
run_register_user: true
run_update_user_password: true
run_import_sqlite_db: true
run_import_media_store: true
run_self_check: true

View file

@ -69,7 +69,7 @@ matrix_appservice_discord_configuration_yaml: |
disableJoinLeaveNotifications: false
# Authentication configuration for the discord bot.
auth:
clientID: {{ matrix_appservice_discord_client_id }}
clientID: {{ matrix_appservice_discord_client_id|string }}
botToken: {{ matrix_appservice_discord_bot_token }}
logging:
# What level should the logger output to the console at.

View file

@ -59,8 +59,19 @@
- name: Generate Appservice IRC passkey if it doesn't exist
shell: /usr/bin/openssl genpkey -out {{ matrix_appservice_irc_data_path }}/passkey.pem -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:2048
become: true
become_user: "{{ matrix_user_username }}"
when: "not irc_passkey_file.stat.exists"
# In the past, we used to generate the passkey.pem file with root, so permissions may not be okay.
# Fix it.
- name: (Migration) Ensure Appservice IRC passkey permissions are okay
file:
path: "{{ matrix_appservice_irc_data_path }}/passkey.pem"
mode: 0644
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_username }}"
# Ideally, we'd like to generate the final registration.yaml file by ourselves.
#
# However, the IRC bridge supports multiple servers, which leads to multiple

View file

@ -3,7 +3,7 @@
matrix_mautrix_telegram_enabled: true
matrix_mautrix_telegram_docker_image: "tulir/mautrix-telegram:v0.5.2"
matrix_mautrix_telegram_docker_image: "tulir/mautrix-telegram:v0.6.0"
matrix_mautrix_telegram_docker_image_force_pull: "{{ matrix_mautrix_telegram_docker_image.endswith(':latest') }}"
matrix_mautrix_telegram_base_path: "{{ matrix_base_data_path }}/mautrix-telegram"
@ -196,6 +196,8 @@ matrix_mautrix_telegram_configuration_yaml: |
inline_images: false
# Maximum size of image in megabytes before sending to Telegram as a document.
image_as_file_size: 10
# Maximum size of Telegram documents in megabytes to bridge.
max_document_size: 100
# Whether to bridge Telegram bot messages as m.notices or m.texts.
bot_messages_as_notices: true
@ -295,6 +297,40 @@ matrix_mautrix_telegram_configuration_yaml: |
api_hash: {{ matrix_mautrix_telegram_api_hash }}
# (Optional) Create your own bot at https://t.me/BotFather
bot_token: disabled
# Telethon connection options.
connection:
# The timeout in seconds to be used when connecting.
timeout: 120
# How many times the reconnection should retry, either on the initial connection or when
# Telegram disconnects us. May be set to a negative or null value for infinite retries, but
# this is not recommended, since the program can get stuck in an infinite loop.
retries: 5
# The delay in seconds to sleep between automatic reconnections.
retry_delay: 1
# The threshold below which the library should automatically sleep on flood wait errors
# (inclusive). For instance, if a FloodWaitError for 17s occurs and flood_sleep_threshold
# is 20s, the library will sleep automatically. If the error was for 21s, it would raise
# the error instead. Values larger than a day (86400) will be changed to a day.
flood_sleep_threshold: 60
# How many times a request should be retried. Request are retried when Telegram is having
# internal issues, when there is a FloodWaitError less than flood_sleep_threshold, or when
# there's a migrate error. May take a negative or null value for infinite retries, but this
# is not recommended, since some requests can always trigger a call fail (such as searching
# for messages).
request_retries: 5
# Device info sent to Telegram.
device_info:
# "auto" = OS name+version.
device_model: auto
# "auto" = Telethon version.
system_version: auto
# "auto" = mautrix-telegram version.
app_version: auto
lang_code: en
system_lang_code: en
# Custom server to connect to.
server:
# Set to true to use these server settings. If false, will automatically
@ -306,6 +342,7 @@ matrix_mautrix_telegram_configuration_yaml: |
ip: 149.154.167.40
# The port to connect to. 443 may not work, 80 is better and both are equally secure.
port: 80
# Telethon proxy configuration.
# You must install PySocks from pip for proxies to work.
proxy:

View file

@ -103,11 +103,11 @@
state: absent
when: "not matrix_nginx_proxy_enabled|bool"
# When Let's Encrypt is not used at all, remove all cronjobs in that cron file.
- name: Ensure matrix-ssl-lets-encrypt-renew cronjob removed
cron:
user: root
cron_file: matrix-ssl-lets-encrypt
name: matrix-ssl-lets-encrypt-certificates-renew
state: absent
when: "matrix_ssl_retrieval_method != 'lets-encrypt'"

View file

@ -28,3 +28,5 @@ matrix_postgres_container_extra_arguments: []
#
# Takes an "<ip>:<port>" or "<port>" value (e.g. "127.0.0.1:5432"), or empty string to not expose.
matrix_postgres_container_postgres_bind_port: ""
matrix_postgres_tool_synapse_janitor: "https://raw.githubusercontent.com/xwiki-labs/synapse_scripts/0b3f035951932ceb396631de3fc701043b9723bc/synapse_janitor.sql"

View file

@ -15,16 +15,26 @@
- setup-postgres
- import_tasks: "{{ role_path }}/tasks/import_postgres.yml"
when: run_import_postgres|bool
when: run_postgres_import|bool
tags:
- import-postgres
- import_tasks: "{{ role_path }}/tasks/import_sqlite_db.yml"
when: run_import_sqlite_db|bool
when: run_postgres_import_sqlite_db|bool
tags:
- import-sqlite-db
- import_tasks: "{{ role_path }}/tasks/upgrade_postgres.yml"
when: run_upgrade_postgres|bool
when: run_postgres_upgrade|bool
tags:
- upgrade-postgres
- import_tasks: "{{ role_path }}/tasks/run_synapse_janitor.yml"
when: run_postgres_synapse_janitor|bool
tags:
- run-postgres-synapse-janitor
- import_tasks: "{{ role_path }}/tasks/run_vacuum.yml"
when: run_postgres_vacuum|bool
tags:
- run-postgres-vacuum

View file

@ -0,0 +1,110 @@
---
# Pre-checks
- name: Fail if Postgres not enabled
fail:
msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot run synapse-janitor."
when: "not matrix_postgres_enabled|bool"
# Defaults
- name: Set postgres_start_wait_time, if not provided
set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time|default('') == ''"
- name: Set postgres_synapse_janitor_wait_time, if not provided
set_fact:
postgres_synapse_janitor_wait_time: "{{ 7 * 86400 }}"
when: "postgres_synapse_janitor_wait_time|default('') == ''"
- name: Set postgres_synapse_janitor_tool_path, if not provided
set_fact:
postgres_synapse_janitor_tool_path: "{{ matrix_postgres_base_path }}/synapse_janitor.sql"
when: "postgres_synapse_janitor_tool_path|default('') == ''"
# Actual janitor work
- name: Download synapse-janitor tool
get_url:
url: "{{ matrix_postgres_tool_synapse_janitor }}"
dest: "{{ postgres_synapse_janitor_tool_path }}"
force: true
mode: 0550
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_username }}"
- name: Ensure matrix-postgres is started
service:
name: matrix-postgres
state: started
daemon_reload: yes
- name: Wait a bit, so that Postgres can start
wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
- import_tasks: tasks/util/detect_existing_postgres_version.yml
- name: Abort, if no existing Postgres version detected
fail:
msg: "Could not find existing Postgres installation"
when: "not matrix_postgres_detected_existing|bool"
- name: Generate Postgres database synapse-janitor command
set_fact:
matrix_postgres_synapse_janitor_command: >-
/usr/bin/docker run --rm --name matrix-postgres-synapse-janitor
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
--mount type=bind,src={{ postgres_synapse_janitor_tool_path }},dst=/synapse_janitor.sql,ro=true
{{ matrix_postgres_docker_image_latest }}
psql -v ON_ERROR_STOP=1 -h matrix-postgres {{ matrix_synapse_database_database }} -f /synapse_janitor.sql
- name: Note about Postgres purging alternative
debug:
msg: >-
Running synapse-janitor with the following Postgres command: `{{ matrix_postgres_synapse_janitor_command }}`.
If this crashes, you can stop all processes (`systemctl stop matrix-*`),
start Postgres only (`systemctl start matrix-postgres`)
and manually run the above command directly on the server.
- name: Populate service facts
service_facts:
- set_fact:
matrix_postgres_synapse_was_running: "{{ ansible_facts.services['matrix-synapse.service']|default(none) is not none and ansible_facts.services['matrix-synapse.service'].state == 'running' }}"
- name: Ensure matrix-synapse is stopped
service:
name: matrix-synapse
state: stopped
daemon_reload: yes
- name: Run synapse-janitor
command: "{{ matrix_postgres_synapse_janitor_command }}"
async: "{{ postgres_synapse_janitor_wait_time }}"
poll: 10
register: matrix_postgres_synapse_janitor_result
# Intentionally show the results
- debug: var="matrix_postgres_synapse_janitor_result"
- name: Ensure matrix-synapse is started, if it previously was
service:
name: matrix-synapse
state: started
daemon_reload: yes
when: "matrix_postgres_synapse_was_running|bool"
- name: Delete synapse-janitor tool
file:
path: "{{ postgres_synapse_janitor_tool_path }}"
state: absent

View file

@ -0,0 +1,90 @@
---
# Pre-checks
- name: Fail if Postgres not enabled
fail:
msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot run vacuum."
when: "not matrix_postgres_enabled|bool"
# Defaults
- name: Set postgres_start_wait_time, if not provided
set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time|default('') == ''"
- name: Set postgres_vacuum_wait_time, if not provided
set_fact:
postgres_vacuum_wait_time: "{{ 7 * 86400 }}"
when: "postgres_vacuum_wait_time|default('') == ''"
# Actual vacuuming work
- name: Ensure matrix-postgres is started
service:
name: matrix-postgres
state: started
daemon_reload: yes
- name: Wait a bit, so that Postgres can start
wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
- import_tasks: tasks/util/detect_existing_postgres_version.yml
- name: Abort, if no existing Postgres version detected
fail:
msg: "Could not find existing Postgres installation"
when: "not matrix_postgres_detected_existing|bool"
- name: Generate Postgres database vacuum command
set_fact:
matrix_postgres_vacuum_command: >-
/usr/bin/docker run --rm --name matrix-postgres-synapse-vacuum
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
{{ matrix_postgres_docker_image_latest }}
psql -v ON_ERROR_STOP=1 -h matrix-postgres {{ matrix_synapse_database_database }} -c 'VACUUM FULL VERBOSE'
- name: Note about Postgres vacuum alternative
debug:
msg: >-
Running vacuum with the following Postgres command: `{{ matrix_postgres_vacuum_command }}`.
If this crashes, you can stop all processes (`systemctl stop matrix-*`),
start Postgres only (`systemctl start matrix-postgres`)
and manually run the above command directly on the server.
- name: Populate service facts
service_facts:
- set_fact:
matrix_postgres_synapse_was_running: "{{ ansible_facts.services['matrix-synapse.service']|default(none) is not none and ansible_facts.services['matrix-synapse.service'].state == 'running' }}"
- name: Ensure matrix-synapse is stopped
service:
name: matrix-synapse
state: stopped
daemon_reload: yes
- name: Run Postgres vacuum command
command: "{{ matrix_postgres_vacuum_command }}"
async: "{{ postgres_vacuum_wait_time }}"
poll: 10
register: matrix_postgres_synapse_vacuum_result
# Intentionally show the results
- debug: var="matrix_postgres_synapse_vacuum_result"
- name: Ensure matrix-synapse is started, if it previously was
service:
name: matrix-synapse
state: started
daemon_reload: yes
when: "matrix_postgres_synapse_was_running|bool"

View file

@ -1,7 +1,7 @@
---
#
# Generic tasks, no matter what kind of server we're using (internal/external)
# Tasks related to setting up an internal postgres server
#
- import_tasks: "{{ role_path }}/tasks/migrate_postgres_data_directory.yml"
@ -32,8 +32,6 @@
force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_postgres_docker_image_force_pull }}"
when: matrix_postgres_enabled|bool
# We always create these directories, even if an external Postgres is used,
# because we store environment variable files there.
- name: Ensure Postgres paths exist
file:
path: "{{ item }}"
@ -70,9 +68,12 @@
mode: 0750
when: matrix_postgres_enabled|bool
#
# Tasks related to setting up an internal postgres server
#
- name: Ensure matrix-postgres-update-user-password-hash script created
template:
src: "{{ role_path }}/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2"
dest: "/usr/local/bin/matrix-postgres-update-user-password-hash"
mode: 0750
when: matrix_postgres_enabled|bool
- name: Ensure matrix-postgres.service installed
template:
@ -127,9 +128,12 @@
msg: "Note: You are not using a local PostgreSQL database, but some old data remains from before in `{{ matrix_postgres_data_path }}`. Feel free to delete it."
when: "not matrix_postgres_enabled|bool and matrix_postgres_data_path_stat.stat.exists"
- name: Ensure matrix-postgres-update-user-password-hash script created
template:
src: "{{ role_path }}/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2"
dest: "/usr/local/bin/matrix-postgres-update-user-password-hash"
mode: 0750
when: matrix_postgres_enabled|bool
- name: Remove Postgres scripts
file:
path: "/usr/local/bin/{{ item }}"
state: absent
with_items:
- matrix-postgres-cli
- matrix-make-user-admin
- matrix-postgres-update-user-password-hash
when: "not matrix_postgres_enabled|bool"

View file

@ -1,6 +1,6 @@
matrix_riot_web_enabled: true
matrix_riot_web_docker_image: "bubuntux/riot-web:v1.2.2"
matrix_riot_web_docker_image: "bubuntux/riot-web:v1.2.4"
matrix_riot_web_docker_image_force_pull: "{{ matrix_riot_web_docker_image.endswith(':latest') }}"
matrix_riot_web_data_path: "{{ matrix_base_data_path }}/riot-web"

View file

@ -15,12 +15,12 @@
- setup-synapse
- import_tasks: "{{ role_path }}/tasks/import_media_store.yml"
when: run_import_media_store|bool
when: run_synapse_import_media_store|bool
tags:
- import-media-store
- import_tasks: "{{ role_path }}/tasks/register_user.yml"
when: run_register_user|bool
when: run_synapse_register_user|bool
tags:
- register-user
@ -39,7 +39,7 @@
- self-check
- import_tasks: "{{ role_path }}/tasks/update_user_password.yml"
when: run_update_user_password|bool
when: run_synapse_update_user_password|bool
tags:
- update-user-password