Commit graph

367 commits

Author SHA1 Message Date
Slavi Pantaleev d6c4d41c2b Define instanceId property on workers
This give us the possibility to run multiple instances of
workers that that don't expose a port.

Right now, we don't support that, but in the future we could
run multiple `federation_sender` or `pusher` workers, without
them fighting over naming (previously, they'd all be named
something like `matrix-synapse-worker-pusher-0`, because
they'd all define `port` as `0`).
2021-02-18 18:19:51 +02:00
Slavi Pantaleev d33483b8ce Document that Synapse pusher worker instances are shardable
Related to:
- https://github.com/matrix-org/synapse/pull/9407
- https://github.com/matrix-org/synapse/pull/7855
2021-02-16 17:45:41 +02:00
Slavi Pantaleev daae74b074 Merge branch 'master' into synapse-workers 2021-02-16 17:31:40 +02:00
Slavi Pantaleev 521160c12f Upgrade Synapse (v1.26.0 -> v1.27.0) 2021-02-16 17:30:48 +02:00
Slavi Pantaleev 61e427d690 Do not let people enable more than 1 federation_sender worker 2021-02-15 11:37:03 +02:00
Slavi Pantaleev 85a05f38e8 Allow Synapse worker list to be generated dynamically
This leads to much easier management and potential safety
features (validation). In the future, we could try to avoid port
conflicts as well, but it didn't seem worth the effort to do it now.
Our port ranges seem large enough.

This can also pave the way for a "presets" feature
(similar to `matrix_nginx_proxy_ssl_presets`) which makes it even easier
for people to configure worker counts.
2021-02-15 11:25:35 +02:00
Slavi Pantaleev 43059bb040 Fix metrics listeners for Synapse workers
`::` leads to errors like:

> socket.gaierror: [Errno -9] Address family for hostname not supported
2021-02-15 11:19:07 +02:00
Slavi Pantaleev 453a4ec2d8 Relocate tasks related to Synapse workers 2021-02-15 11:18:47 +02:00
Slavi Pantaleev 5cfeae806b Merge branch 'master' into synapse-workers 2021-02-14 13:00:57 +02:00
Slavi Pantaleev 7e8e95a09a Make S3-mounting path configurable
This will make data migration easier.
2021-02-09 22:05:07 +02:00
Béla Becker 2edc9cb83c Name the Synapse database on state compression import
Fixes:
https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/833
2021-01-28 17:54:02 +01:00
Slavi Pantaleev e7f3f7c431 Enable /devices endpoint for generic workers 2021-01-27 22:18:47 +02:00
Slavi Pantaleev 1cd2a218de Merge branch 'master' into synapse-workers 2021-01-27 21:41:54 +02:00
Slavi Pantaleev c6feb0b99e Upgrade Synapse (v1.25.0 -> v1.26.0) 2021-01-27 21:41:47 +02:00
Slavi Pantaleev d98a1ceadd Merge branch 'master' into synapse-workers 2021-01-27 10:27:17 +02:00
Slavi Pantaleev 512f42aa76 Do not report docker kill/rm attempts as errors
These are just defensive cleanup tasks that we run.
In the good case, there's nothing to kill or remove, so they trigger an
error like this:

> Error response from daemon: Cannot kill container: something: No such container: something

and:

> Error: No such container: something

People often ask us if this is a problem, so instead of always having to
answer with "no, this is to be expected", we'd rather eliminate it now
and make logs cleaner.

In the event that:
- a container is really stuck and needs cleanup using kill/rm
- and cleanup fails, and we fail to report it because of error
suppression (`2>/dev/null`)

.. we'd still get an error when launching ("container name already in use .."),
so it shouldn't be too hard to investigate.
2021-01-27 10:22:46 +02:00
Slavi Pantaleev a535226210 Stop/disable unnecessary worker services before deleting them 2021-01-25 15:20:37 +02:00
Slavi Pantaleev 66cdc7bf5a Clean up worker.yaml generation a bit and make it more flexible 2021-01-25 13:02:01 +02:00
Slavi Pantaleev 1462409b34 Fix worker listening addresses
Not specifying bind addresses for the worker resulted in this warning:

> synapse.app - 47 - WARNING - None - Failed to listen on 0.0.0.0, continuing because listening on [::]

Additionally, metrics listening only on 127.0.0.1 seems like a no-op.
Only having it accessible from within the container is likely not what
we intend. Changed that to all interfaces as well.

Whether it actually gets exposed or not depends on the systemd service
and `matrix_synapse_workers_container_host_bind_address`.
2021-01-25 12:29:47 +02:00
Slavi Pantaleev 01747c8cc4 Prevent Synapse warning about enabling metric listeners with enable_metrics: false
> synapse.app.generic_worker - 606 - WARNING - None - Metrics listener configured, but enable_metrics is not True!
2021-01-25 12:24:12 +02:00
Slavi Pantaleev 70796703d3 Run Synapse workers in their own containers
This switches the `docker exec` method of spawning
Synapse workers inside the `matrix-synapse` container with
dedicated containers for each worker.

We also have dedicated systemd services for each worker,
so this are now:
- more consistent with everything else (we don't use systemd
instantiated services anywhere)
- we don't need the "parse systemd instance name into worker name +
port" part
- we don't need to keep track of PIDs manually
- we don't need jq (less depenendencies)
- workers dying would be restarted by systemd correctly, like any other
service
- `docker ps` shows each worker separately and we can observe resource
usage
2021-01-25 12:14:46 +02:00
Slavi Pantaleev da50fb27a0 Whitelist /_matrix/key requests for going to generic workers on the federation port 2021-01-25 09:46:50 +02:00
Slavi Pantaleev 63301b0ef1 Improvements around Synapse worker/metrics ports exposure
There was a `matrix_nginx_proxy_enabled|default(False)` check, but:
- it didn't seem to work reliably for some reason (hmm)
- referring to a `matrix_nginx_proxy_*` variable from within the
  `matrix-synapse` role is not ideal
- exposing always happened on `127.0.0.1`, which may not be good enough
  for some rarer setups (where the own webserver is external to the host)
2021-01-25 08:25:43 +02:00
Slavi Pantaleev 5ca68210cd Do not handle /_matrix/federation on client-server port, nor /_matrix/client stuff on federation port
I guess it didn't hurt to do it until now, but it's not great serving
federation APIs on the client-server API port, etc.

matrix-corporal doesn't work yet (still something to be solved in the
future), but its firewalling operations will also be sabotaged
by Client-Server APIs being served on the federation port (it's a way to get around its firewalling).
2021-01-24 22:22:57 +02:00
Slavi Pantaleev cc5cf0d725 Load roles/matrix-synapse/vars/workers.yml earlier to not break --tags=setup-nginx-proxy
If we load it at runtime, during matrix-synapse role execution,
it's good enough for matrix-synapse and all roles after that,
but.. it breaks when someone uses `--tags=setup-nginx-proxy` alone.

The downside of including this vars file like this in `setup.yml`
is that the variables contained in it cannot be overriden by the user
(in their inventory's `vars.yml`).
... but it's not like overriding these variables was possible anyway
when including them at runtime.
2021-01-24 20:19:55 +02:00
Marcel Partap edc21f15e5 Restrict publishing worker (metrics) ports to localhost 2021-01-24 08:53:09 +01:00
Marcel Partap 183adec3d8 Merge remote-tracking branch 'origin/master' into synapse-workers 2021-01-23 15:04:11 +01:00
Marcel Partap c8f051a42d Track workers endpoint list in repo instead of regenerating on user side 2021-01-23 14:44:36 +01:00
Marcel Partap f2c7d79238 Drop probably incorrect comment from synapse homeserver.yaml.j2 2021-01-23 14:44:36 +01:00
Slavi Pantaleev 95346f3117 Reorganize Postgres access (breaking change)
In short, this makes Synapse a 2nd class citizen,
preparing for a future where it's just one-of-many homeserver software
options.

We also no longer have a default Postgres superuser password,
which improves security.

The changelog explains more as to why this was done
and how to proceed from here.
2021-01-22 13:26:12 +02:00
Slavi Pantaleev 1692a28fe4 Work around annoying Docker warning about undefined $HOME
> WARNING: Error loading config file: .dockercfg: $HOME is not defined

.. which appeared in Docker 20.10.
2021-01-15 00:23:01 +02:00
Slavi Pantaleev 05ca9357a8 Add .service suffix to systemd units list
We'll be adding `.timer` units later on, so it's good to be
more explicit.
2021-01-14 23:02:10 +02:00
Slavi Pantaleev b15da29ebb Bump Synapse to v1.25.0 for ARM 2021-01-14 10:41:47 +02:00
Slavi Pantaleev 24100342e1 Tell people that federation_ip_range_blacklist is gone
Related to d5945c6e78
2021-01-13 13:47:51 +02:00
Slavi Pantaleev d5945c6e78 Upgrade Synapse (v1.24.0 -> v1.25.0) for amd64 2021-01-13 13:02:49 +02:00
Marcel Partap cd8100544b Merge remote-tracking branch 'origin/master' into synapse-workers
Sync with upstream
2021-01-08 20:58:50 +01:00
Slavi Pantaleev 6e1dfb62f0 Rename some doc files and commands related to importing
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.

Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.

We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
2020-12-14 01:51:00 +02:00
Slavi Pantaleev d08b27784f Fix systemd services autostart problem with Docker 20.10
The Docker 19.04 -> 20.10 upgrade contains the following change
in `/usr/lib/systemd/system/docker.service`:

```
-BindsTo=containerd.service
-After=network-online.target firewalld.service containerd.service
+After=network-online.target firewalld.service containerd.service multi-user.target
-Requires=docker.socket
+Requires=docker.socket containerd.service
Wants=network-online.target
```

The `multi-user.target` requirement in `After` seems to be in conflict
with our `WantedBy=multi-user.target` and `After=docker.service` /
`Requires=docker.service` definitions, causing the following error on
startup for all of our systemd services:

> Job matrix-synapse.service/start deleted to break ordering cycle starting with multi-user.target/start

A workaround which appears to work is to add `DefaultDependencies=no`
to all of our services.
2020-12-10 11:43:20 +02:00
Slavi Pantaleev 245b749946 Upgrade Synapse for ARM (v1.23.0 -> v1.24.0)
Continuation of aa86e0dac6, now that ARM images are out.
2020-12-09 20:54:18 +02:00
Slavi Pantaleev aa86e0dac6 Upgrade Synapse (v1.23.0 -> v1.24.0)
Because the ARM images are not pushed yet, we hold back to v1.23.0
for now.
2020-12-09 13:31:10 +02:00
Slavi Pantaleev c07c927d9f Automatically enable openid listeners when ma1sd enabled
ma1sd requires the openid endpoints for certain functionality.
Example: 90b2b5301c/src/main/java/io/kamax/mxisd/auth/AccountManager.java (L67-L99)

If federation is disabled, we still need to expose these openid APIs on the
federation port.

Previously, we were doing similar magic for Dimension.
As per its documentation, when running unfederated, one is to enable
the openid listener as well. As per their recommendation, people
are advised to do enable it on the Client-Server API port
and use the `federationUrl` variable to override where the federation
port is (making federation requests go to the Client-Server API).

Because ma1sd always uses the federation port (unless you do some
DNS overwriting magic using its configuration -- which we'd rather not
do), it's better if we just default to putting the `openid` listener
where it belongs - on the federation port.

With this commit, we retain the "automatically enable openid APIs" thing
we've been doing for Dimension, but move it to the federation port instead.
We also now do the same thing when ma1sd is enabled.
2020-12-08 16:59:20 +02:00
Marcel Partap b6b95fe742 synapse workers-doc-to-yaml script: compatibility++ with non-gnu awk 2020-12-02 23:22:02 +01:00
Marcel Partap 3156d96619 synapse workers-doc-to-yaml.awk: escape slash for non-gnu awk versions 2020-12-02 00:29:20 +01:00
Marcel Partap e892ac464f synapse workers: untangle config template and specify bind address
.. to mitigate log noise - WARNING:
Failed to listen on 0.0.0.0, continuing because listening on [::]
2020-12-01 23:49:23 +01:00
Marcel Partap f201bca519 synapse workers: define and expose METRICS port for each worker
As seen on TV:
https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md#monitoring-workers
2020-12-01 22:49:15 +01:00
Marcel Partap af08f18779 synapse workers default config: disable user_dir worker for now
(until https://github.com/matrix-org/synapse/issues/8787 is resolved)
2020-12-01 22:22:04 +01:00
Marcel Partap 414b812a29 synapse role workers setup: make configs clean action remote compatible
Many people probably didn't even know this - that ansible can be
quite a bit picky about what it will be willing to work with remotely.

Thanks @maxklenk !
2020-12-01 22:20:27 +01:00
Marcel Partap d5932ca393 synapse role workers setup: execute the endpoint extraction locally
Thanks @maxklenk !
2020-12-01 22:18:42 +01:00
Marcel Partap b73ac965ac Merge remote-tracking branch 'origin/master' into synapse-workers 2020-12-01 21:24:26 +01:00
Slavi Pantaleev be5263f397 Move self-building git repository URLs to variables (stop hardcoding) 2020-11-28 21:34:14 +02:00
Slavi Pantaleev 75f9fde7a4 Remove some more -v usage
Continuation of 1fca917ad1.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/722
2020-11-25 10:49:59 +02:00
Slavi Pantaleev 1fca917ad1 Replace some -v instances with --mount
`-v` magically creates the source destination as a directory,
if it doesn't exist already. We'd like to avoid this magic
and the potential breakage that it might cause.

We'd rather fail while Docker tries to find things to `--mount`
than have it automatically create directories and fail anyway,
while having contaminated the filesystem.

There's a lot more `-v` instances remaining to be fixed later on.
This is just some start.

Things like `matrix_synapse_container_additional_volumes` and
`matrix_nginx_proxy_container_additional_volumes` were not changed to
use `--mount`, as options for each one are passed differently
(`ro` is `ro`, but `rw` doesn't exist and `slave` is `bind-propagation=slave`).
To avoid breaking people's custom volume mounts, we keep it as it is for now.

A deficiency with `--mount` is that it lacks the `z` option (SELinux
ownership changes), and some of our `-v` instances use that. I'm not
sure how supported SELinux is for us right now, but it might be,
and breaking that would not be a good idea.
2020-11-24 10:26:05 +02:00
Slavi Pantaleev b627d93cdc Update homeserver.yaml to keep up with Synapse v1.23.0
Related to #724 (Github Pull Request)
2020-11-18 16:57:50 +02:00
transcaffeine c58a7e03c7
synapse: update to 1.23.0 2020-11-18 14:16:46 +01:00
Slavi Pantaleev 5eed874199 Improve self-building experience (avoid conflict with pullable images)
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/716

This patch makes us use more fully-qualified container image names
(either prefixed with docker.io/ or with localhost/).

The latter happens when self-building is enabled.

We've recently had issues where if an image was removed manually
and the service was restarted (making `docker run` fetch it from Docker Hub, etc.),
we'd end up with a pulled image, even though we're aiming for a self-built one.
Re-running the playbook would then not do a rebuild, because:
- the image with that name already exists (even though it's something
else)
- we sometimes had conditional logic where we'd build only if the git
repo changed

By explicitly changing the name of the images (prefixing with localhost/),
we avoid such confusion and the possibility that we'd automatically pul something
which is not what we expect.

Also, I've removed that condition where building would happen on git
changes only. We now always build (unless an image with that name
already exists). We just force-build when the git repo changes.
2020-11-14 23:00:49 +02:00
Marcel Partap 4678c5d7bd Merge remote-tracking branch 'origin/master' into synapse-workers
Also, replace vague FIXME by a proper NOTE on the complete
story of the user_dir endpoints..
2020-11-11 21:26:08 +01:00
Marcel Partap f3d2797d9c synapse workers: make awk script invocation handle paths with spaces
(quoting ftw)
2020-11-10 22:40:48 +01:00
Marcel Partap 1e971312e8 synapse workers: handle auth fallback endpoint on main process only
(allegedly breaks with SSO enabled)
2020-11-10 21:23:19 +01:00
Slavi Pantaleev 5c91e56898 Upgrade Synapse (v1.22.0 -> v1.22.1) 2020-10-30 19:35:55 +02:00
Marcel Partap cce90b187a synapse workers: fix undefined variable cases when removing workers 2020-10-28 23:09:21 +01:00
Marcel Partap e078e29ef8 synapse workers: fix self name in workers-doc-to-yaml.awk script 2020-10-28 08:39:31 +01:00
Marcel Partap 2d1b9f2dbf synapse workers: reworkings + get endpoints from upstream docs via awk
(yes, a bit awkward and brittle… xD)
2020-10-28 07:13:19 +01:00
Slavi Pantaleev 70f0b97a0a Upgrade Synapse (v1.21.2 -> v1.22.0) 2020-10-27 14:24:02 +02:00
Marcel Partap 87bd64ce9e Merge remote-tracking branch 'origin/master' into synapse-workers 2020-10-23 23:45:07 +02:00
Marcel Partap a4125d5446 synapse workers: polishing, cleansing and installation of jq dependency 2020-10-23 20:49:53 +02:00
Marcel Partap 501efee07e synapse workers: supply systemd with actual worker PIDs (requires jq)
also, worker.yaml.j2:
  - hone worker_name
  - remove worker_pid_file entry (would only be used if worker_daemonize
    set to true; also, synapse only knows about the container namespace
    and thus can not provide the required host-view PID)
2020-10-22 20:53:41 +02:00
Slavi Pantaleev f7ecc7a2a5 Upgrade Synapse (v1.21.1 -> v1.21.2) 2020-10-15 17:42:52 +03:00
Slavi Pantaleev 5abd511368 Upgrade Synapse (v1.21.0 -> v1.21.1) 2020-10-13 13:08:25 +03:00
Aaron Raimist 78529cbd47
Upgrade Synapse (v1.20.1 -> v1.21.0) 2020-10-12 23:59:34 -05:00
Marcel Partap d2e61af224 Add worker_name to synapse worker config template
& restrict federation listener; frontend_proxy / user_dir don't need it
2020-10-11 21:52:08 +02:00
Marcel Partap 36e9be6092 matrix_synapse_workers_{avail,enabled}_list: sort non-generic workers
.. alphabetically and put those not documented as multi-instance
capable on ports ending on zero.
2020-10-11 21:44:42 +02:00
Marcel Partap e9241f5fb9 Improve synapse-workers systemd service template
Is the PID magic gonna work? or will it need an ExecStartPost hack..
2020-10-11 21:09:19 +02:00
Marcel Partap 40024e9b81 Prevent workers failing if their config doesn't exist
- cherry-pick "Ensure worker config exists in systemd service (#7528)"
  from synapse d74cdc1a42e8b487d74c214b1d0ca575429d546a:
  "check that the worker config file exists instead of silently failing."
2020-10-11 21:09:19 +02:00
Marcel Partap 93a8ea7e4a Merge remote-tracking branch 'master' into feature/add-worker-support 2020-10-11 20:59:05 +02:00
Slavi Pantaleev e68450f094 Upgrade Synapse (v1.20.0 -> v1.20.1) 2020-09-24 18:43:54 +03:00
Slavi Pantaleev dd217137b6 Upgrade Synapse (v1.19.3 -> v1.20.0) 2020-09-22 19:28:07 +03:00
Slavi Pantaleev 65e22a6888 Upgrade Synapse (v1.19.2 -> v1.19.3) 2020-09-18 17:37:04 +03:00
Slavi Pantaleev e10e3e354d Upgrade Synapse (v1.19.1 -> v1.19.2) 2020-09-16 16:35:17 +03:00
Max Klenk 4fdfc0a34f
add missing ratelimiting options required for load testing 2020-09-11 09:46:20 +02:00
Max Klenk a25a429a52
add redis support 2020-09-10 13:39:00 +02:00
Max Klenk 06bc430c7c
refactor to use new workers and routes they serve 2020-08-28 13:53:39 +02:00
Max Klenk 53ccc783b7
remove duplicated key 2020-08-27 15:26:46 +02:00
Max Klenk 567d0318b0
Merge branch 'synapse-workers' into feature/add-worker-support 2020-08-27 15:22:12 +02:00
Slavi Pantaleev 6e9600ffec Upgrade Synapse (v1.19.0 -> v1.19.1) 2020-08-27 12:59:11 +03:00
Slavi Pantaleev daf13107a0 Add support for rust-synapse-compress-state 2020-08-21 13:53:39 +03:00
Slavi Pantaleev 9952ec6c16 Upgrade Synapse (v1.18.0 -> v1.19.0) 2020-08-17 17:02:40 +03:00
Slavi Pantaleev f78a5d4ee8 Upgrade Synapse (v1.17.0 -> v1.18.0) 2020-07-30 14:21:44 +03:00
Slavi Pantaleev c6ab1c6a90 Riot is now Element
Fixes #586 (Github Issue)
2020-07-17 11:31:20 +03:00
Slavi Pantaleev 200f912c04 Upgrade Synapse (v1.16.1 -> v1.17.0)
Fixes #579 (Github Issue).
2020-07-13 14:08:50 +03:00
Slavi Pantaleev eff55e4d00 Upgrade Synapse (v1.16.0 -> v1.16.1) 2020-07-10 14:33:18 +03:00
Slavi Pantaleev 928982cffe Upgrade Synapse (v1.15.2 -> v1.16.0) 2020-07-08 14:08:46 +03:00
Slavi Pantaleev 18ab677a96 Remove useless file 2020-07-08 00:22:47 +03:00
shadow 6293f1bdb0 Run all API self checks in check_mode 2020-07-04 15:24:33 +02:00
Slavi Pantaleev f758ee90cb
Add |to_json to some values 2020-07-04 09:31:52 +03:00
Justin Croonenberghs 35c2655fa4 Removed troublesome #s 2020-07-03 19:01:03 -05:00
Justin Croonenberghs 1f21f0c09a Add variables for reCAPTCHA validation 2020-07-03 18:33:25 -05:00
Aaron Raimist 78382b0ce4
Upgrade Synapse (1.15.1 -> 1.15.2) 2020-07-02 10:38:25 -05:00
Slavi Pantaleev 10bc85962e Upgrade Synapse (1.15.0 -> 1.15.1) 2020-06-16 13:55:27 +03:00
Slavi Pantaleev 6538ae34f5 Upgrade Synapse (v1.14 -> v1.15)
Fixes #539 (Github Issue).
2020-06-11 16:02:01 +03:00
Slavi Pantaleev 10b3ceff72 Make Matrix federation port configurable
Fixes #523 (Github Issue).
2020-06-09 08:29:03 +03:00