Commit graph

148 commits

Author SHA1 Message Date
Marcel Partap 414b812a29 synapse role workers setup: make configs clean action remote compatible
Many people probably didn't even know this - that ansible can be
quite a bit picky about what it will be willing to work with remotely.

Thanks @maxklenk !
2020-12-01 22:20:27 +01:00
Marcel Partap d5932ca393 synapse role workers setup: execute the endpoint extraction locally
Thanks @maxklenk !
2020-12-01 22:18:42 +01:00
Marcel Partap b73ac965ac Merge remote-tracking branch 'origin/master' into synapse-workers 2020-12-01 21:24:26 +01:00
Slavi Pantaleev be5263f397 Move self-building git repository URLs to variables (stop hardcoding) 2020-11-28 21:34:14 +02:00
Slavi Pantaleev 75f9fde7a4 Remove some more -v usage
Continuation of 1fca917ad1.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/722
2020-11-25 10:49:59 +02:00
Slavi Pantaleev 1fca917ad1 Replace some -v instances with --mount
`-v` magically creates the source destination as a directory,
if it doesn't exist already. We'd like to avoid this magic
and the potential breakage that it might cause.

We'd rather fail while Docker tries to find things to `--mount`
than have it automatically create directories and fail anyway,
while having contaminated the filesystem.

There's a lot more `-v` instances remaining to be fixed later on.
This is just some start.

Things like `matrix_synapse_container_additional_volumes` and
`matrix_nginx_proxy_container_additional_volumes` were not changed to
use `--mount`, as options for each one are passed differently
(`ro` is `ro`, but `rw` doesn't exist and `slave` is `bind-propagation=slave`).
To avoid breaking people's custom volume mounts, we keep it as it is for now.

A deficiency with `--mount` is that it lacks the `z` option (SELinux
ownership changes), and some of our `-v` instances use that. I'm not
sure how supported SELinux is for us right now, but it might be,
and breaking that would not be a good idea.
2020-11-24 10:26:05 +02:00
Slavi Pantaleev 5eed874199 Improve self-building experience (avoid conflict with pullable images)
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/716

This patch makes us use more fully-qualified container image names
(either prefixed with docker.io/ or with localhost/).

The latter happens when self-building is enabled.

We've recently had issues where if an image was removed manually
and the service was restarted (making `docker run` fetch it from Docker Hub, etc.),
we'd end up with a pulled image, even though we're aiming for a self-built one.
Re-running the playbook would then not do a rebuild, because:
- the image with that name already exists (even though it's something
else)
- we sometimes had conditional logic where we'd build only if the git
repo changed

By explicitly changing the name of the images (prefixing with localhost/),
we avoid such confusion and the possibility that we'd automatically pul something
which is not what we expect.

Also, I've removed that condition where building would happen on git
changes only. We now always build (unless an image with that name
already exists). We just force-build when the git repo changes.
2020-11-14 23:00:49 +02:00
Marcel Partap f3d2797d9c synapse workers: make awk script invocation handle paths with spaces
(quoting ftw)
2020-11-10 22:40:48 +01:00
Marcel Partap cce90b187a synapse workers: fix undefined variable cases when removing workers 2020-10-28 23:09:21 +01:00
Marcel Partap 2d1b9f2dbf synapse workers: reworkings + get endpoints from upstream docs via awk
(yes, a bit awkward and brittle… xD)
2020-10-28 07:13:19 +01:00
Marcel Partap a4125d5446 synapse workers: polishing, cleansing and installation of jq dependency 2020-10-23 20:49:53 +02:00
Marcel Partap 501efee07e synapse workers: supply systemd with actual worker PIDs (requires jq)
also, worker.yaml.j2:
  - hone worker_name
  - remove worker_pid_file entry (would only be used if worker_daemonize
    set to true; also, synapse only knows about the container namespace
    and thus can not provide the required host-view PID)
2020-10-22 20:53:41 +02:00
Max Klenk 567d0318b0
Merge branch 'synapse-workers' into feature/add-worker-support 2020-08-27 15:22:12 +02:00
Slavi Pantaleev daf13107a0 Add support for rust-synapse-compress-state 2020-08-21 13:53:39 +03:00
Slavi Pantaleev 9952ec6c16 Upgrade Synapse (v1.18.0 -> v1.19.0) 2020-08-17 17:02:40 +03:00
shadow 6293f1bdb0 Run all API self checks in check_mode 2020-07-04 15:24:33 +02:00
Slavi Pantaleev 10b3ceff72 Make Matrix federation port configurable
Fixes #523 (Github Issue).
2020-06-09 08:29:03 +03:00
Slavi Pantaleev f56a9a0f5f
Merge pull request #524 from cnvandijk/fix-executable-path
Remove hardcoded paths to commands on the host machine
2020-05-28 15:39:25 +03:00
Slavi Pantaleev 8bae39050e Update settings for Synapse v1.14.0 2020-05-28 15:23:05 +03:00
Chris van Dijk 6e3b877dc2 Remove hardcoded command paths in playbook shell usage 2020-05-27 23:14:56 +02:00
Slavi Pantaleev 8fb3ce6f6d Upgrade Synapse (v1.12.4 -> v1.13.0) 2020-05-19 21:35:32 +03:00
Chris van Dijk 7585bcc4ac Allow the matrix user username and groupname to be configured separately
No migration steps should be required.
2020-05-01 19:59:32 +02:00
Chris van Dijk cf0e56e92b Consistent usage of matrix username and groupname, not uid and gid
This should be a no-op, no migration necessary.
2020-05-01 19:39:17 +02:00
Slavi Pantaleev 9cc0c5955d Use |quote in some command calls 2020-04-24 09:59:30 +03:00
Marcel Partap cf452fdf0a Fix corner-cases found through testing (aka ansible is nuts) 2020-04-19 19:05:03 +02:00
Marcel Partap 353bc7c362 Add initial support for synapse workers
· needs documentation; no checks yet for port clashes or typos in worker name
· according to https://github.com/matrix-org/synapse/wiki/Workers-setup-with-nginx#results
  about 90% of requests go to the synchrotron endpoint
· thus, the synchrotron worker is especially suited to be load-balanced
· most of the other workers are documented to support only a single instance
· https://github.com/matrix-org/synapse/blob/master/docs/workers.md
2020-04-19 19:05:03 +02:00
Slavi Pantaleev ac60115190 Fix git installation bug during synapse-simple-antispam installation
Fixes #424 (Github Issue).
2020-03-29 10:11:08 +03:00
Christian Wolf 51c271905d Removed bug in enhanced waiting 2020-03-28 13:00:56 +01:00
Christian Wolf 8c9b5ea6dd Removed a few syntax bugs in Archlinux configuration 2020-03-28 13:00:01 +01:00
Christian Wolf 26bc66117d Removed redunadant waiting 2020-03-28 11:42:41 +01:00
Christian Wolf d84b2868b7 Added basic changes to make it compatible with Archlinux 2020-03-28 11:39:15 +01:00
mooomooo eebc6e13f8 Made directory variables for /etc/systemd/system , /etc/cron.d , /usr/local/bin 2020-03-24 11:27:58 -07:00
Slavi Pantaleev 027056e027 Fix weird path creation
Fixes #403 (Github Issue).
2020-03-18 18:24:30 +02:00
Slavi Pantaleev 3cee815baf Fix some typos 2020-03-15 11:34:35 +02:00
Slavi Pantaleev 2b85fde103 Rename some variables for consistency 2020-03-15 10:15:27 +02:00
Horvath Gergely 3c8535c3bc check ansible version for self-building in every role 2020-03-08 19:17:10 +01:00
Horvath Gergely a5d94eec0b refactor variable names 2020-03-08 00:28:14 +01:00
Horvath Gergely 310aa685f9 refactor based on Slavi's requests 2020-03-08 00:24:00 +01:00
Horvath Gergely a096eafb45 add possibility to install synapse on raspberry pi 2020-02-17 21:48:48 +01:00
Slavi Pantaleev 2c04384e8e Synchronize config with the one from Synapse 1.9.0
Related to #355.
2020-01-23 15:47:53 +02:00
Dan Arnfield 86eff45e8b uri.follow_redirects is now a string field 2020-01-22 15:36:54 -06:00
Michael Haak 5b213e6ad0 Replace constructs appending elements with variables to matrix_synapse_container_extra_arguments. Fixes issue https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/304 2019-11-09 23:16:12 +01:00
Slavi Pantaleev 73c90e9513 Try to make Synapse config/key generation respect uid/gid
Attempt at fixing #268 (Github Issue)
2019-09-23 18:08:36 +03:00
Slavi Pantaleev 68ed2ebefa Add support for Synapse Simple Antispam
Fixes #255 (Github Issue).
2019-09-09 08:13:10 +03:00
Slavi Pantaleev 10a9deba4a Make Synapse configuration extensible 2019-08-22 09:49:22 +03:00
Slavi Pantaleev b440d5b73c Remove some fact definitions during runtime 2019-08-22 08:00:02 +03:00
Slavi Pantaleev 631a14bf0c Rename run control variables for consistency 2019-07-08 09:38:36 +03:00
Slavi Pantaleev 7d3adc4512 Automatically force-pull :latest images
We do use some `:latest` images by default for the following services:
- matrix-dimension
- Goofys (in the matrix-synapse role)
- matrix-bridge-appservice-irc
- matrix-bridge-appservice-discord
- matrix-bridge-mautrix-facebook
- matrix-bridge-mautrix-whatsapp

It's terribly unfortunate that those software projects don't release
anything other than `:latest`, but that's how it is for now.

Updating that software requires that users manually do `docker pull`
on the server. The playbook didn't force-repull images that it already
had.

With this patch, it starts doing so. Any image tagged `:latest` will be
force re-pulled by the playbook every time it's executed.

It should be noted that even though we ask the `docker_image` module to
force-pull, it only reports "changed" when it actually pulls something
new. This is nice, because it lets people know exactly when something
gets updated, as opposed to giving the indication that it's always
updating the images (even though it isn't).
2019-06-10 14:30:28 +03:00
Slavi Pantaleev 70487061f4 Prefer --mount instead of -v for mounting volumes
This doesn't replace all usage of `-v`, but it's a start.

People sometimes troubleshoot by deleting files (especially bridge
config files). Restarting Synapse with a missing registration.yaml file
for a given bridge, causes the `-v
/something/registration.yaml:/something/registration.yaml:ro` option
to force-create `/something/registration.yaml` as a directory.

When a path that's provided to the `-v` option is missing, Docker
auto-creates that path as a directory.
This causes more breakage and confusion later on.

We'd rather fail, instead of magically creating directories.
Using `--mount`, instead of `-v` is the solution to this.

From Docker's documentation:

> When you use --mount with type=bind, the host-path must refer to an existing path on the host.
> The path will not be created for you and the service will fail with an error if the path does not exist.
2019-05-29 09:59:50 +03:00
Slavi Pantaleev ab59cc50bd Add support for more flexible container port exposing
Fixes #171 (Github Issue).
2019-05-25 07:41:08 +09:00
Slavi Pantaleev a8b633561d Upgrade Synapse (v0.99.4 -> v0.99.5.1) 2019-05-23 09:23:04 +09:00
Dan Arnfield 9c23d877fe Fix docker_image option for ansible < 2.8 2019-05-22 05:43:33 -05:00
Dan Arnfield db15791819 Add source option to docker_image to fix deprecation warning 2019-05-21 10:29:12 -05:00
Dan Arnfield 3982f114af Fix CONDITIONAL_BARE_VARS deprecation warning in ansible 2.8 2019-05-21 10:25:59 -05:00
Slavi Pantaleev affb99003c Improve Synapse variable naming consistency 2019-05-21 12:09:38 +09:00
Slavi Pantaleev fc7ba153b1 Make matrix-synapse role respect matrix_synapse_enabled flag 2019-05-21 10:46:49 +09:00
Slavi Pantaleev 8d654aecdd Improve file naming consistency 2019-05-21 09:57:48 +09:00
Slavi Pantaleev e3b4622ac8 Split Synapse extension tasks into install/uninstall files 2019-05-18 06:36:54 +09:00
Slavi Pantaleev 663d1add92 Move matrix-appservice-discord into a separate role 2019-05-18 01:14:12 +09:00
Slavi Pantaleev 3339e37ce9 Move matrix-appservice-irc into a separate role 2019-05-16 09:07:40 +09:00
Slavi Pantaleev 43fd3cc274 Move mautrix-facebook into a separate role 2019-05-15 09:34:31 +09:00
Slavi Pantaleev bb816df557 Move mautrix telegram and whatsapp into separate roles
The goal is to move each bridge into its own separate role.
This commit starts off the work on this with 2 bridges:
- mautrix-telegram
- mautrix-whatsapp

Each bridge's role (including these 2) is meant to:

- depend only on the matrix-base role

- integrate nicely with the matrix-synapse role (if available)

- integrate nicely with the matrix-nginx-proxy role (if available and if
required). mautrix-telegram bridge benefits from integrating with
it.

- not break if matrix-synapse or matrix-nginx-proxy are not used at all

This has been provoked by #174 (Github Issue).
2019-05-14 23:47:22 +09:00
Hugues Morisset a82d5ed281 Add tulir mautrix-facebook (https://github.com/tulir/mautrix-facebook) 2019-05-08 17:11:07 +02:00
Lyubomir Popov a206b65ed7 Use the '-p' non-interactive option to generate password hash instead of 'expect' 2019-05-03 11:02:17 +03:00
Lyubomir Popov 134faa3139 Add the ability to update user passwords with ansible (when using the matrix-postgres container). 2019-04-30 16:30:26 +03:00
Slavi Pantaleev 892abdc700 Do not refer to Synapse as "Matrix Synapse" 2019-04-23 10:20:56 +03:00
Slavi Pantaleev 39566aa7fe Generate a Synapse signing key file, if missing
The code used to check for a `homeserver.yaml` file and generate
a configuration (+ key) only if such a configuration file didn't exist.

Certain rare cases (setting up with one server name and then
changing to another) lead to `homeserver.yaml` being there,
but a `matrix.DOMAIN.signing.key` file missing (because the domain
changed).
A new signing key file would never get generated, because `homeserver.yaml`'s
existence used to be (incorrectly) satisfactory for us.

From now on, we don't mix things up like that.
We don't care about `homeserver.yaml` anymore, but rather
about the actual signing key.

The rest of the configuration (`homeserver.yaml` and
`matrix.DOMAIN.log.config`) is rebuilt by us in any case, so whether
it exists or not is irrelevant and doesn't need checking.
2019-04-23 10:06:42 +03:00
Slavi Pantaleev af1c9ae59d Do not force firewalld on people
In most cases, there's not really a need to touch the system
firewall, as Docker manages iptables by itself
(see https://docs.docker.com/network/iptables/).

All ports exposed by Docker containers are automatically whitelisted
in iptables and wired to the correct container.

This made installing firewalld and whitelisting ports pointless,
as far as this playbook's services are concerned.

People that wish to install firewalld (for other reasons), can do so
manually from now on.

This is inspired by and fixes #97 (Github Issue).
2019-04-03 11:37:20 +03:00
Slavi Pantaleev 73af8f7bbb Make self-check not validate self-signed certificates
By default, `--tags=self-check` no longer validates certificates
when `matrix_ssl_retrieval_method` is set to `self-signed`.

Besides this default, people can also enable/disable validation using the
individual role variables manually.

Fixes #124 (Github Issue)
2019-03-22 09:41:08 +02:00
Lorrin Nelson ceba99eed3 Make federation self-check conditional on matrix_synapse_federation_enabled 2019-03-13 22:33:52 -07:00
Slavi Pantaleev 2d56ff0afa Skip some uninstall tasks if not necessary to run 2019-03-13 07:40:51 +02:00
Plailect f6de3fd668
Start appservice-irc as non-root 2019-03-12 13:17:51 -04:00
Slavi Pantaleev 390ec8a599 Skip some tasks when not necessary to run them 2019-03-08 12:14:58 +02:00
Slavi Pantaleev 85c5adfd69 Minor consistency improvements 2019-03-05 09:20:36 +02:00
Slavi Pantaleev a310a01818 Use non-root and no-capability containers during Discord setup
Related to #105 (Github Pull Request).
2019-03-05 09:10:51 +02:00
Slavi Pantaleev f037f63a07
Merge pull request #105 from Lionstiger/matrix-discord-bridge
Add Support for matrix-appservice-discord
2019-03-05 06:39:46 +00:00
Lionstiger 278484656b ensure systemd reloaded after bridge installation 2019-03-04 15:12:37 +01:00
Lionstiger 4aeeb5cf31 Autogenerate Discord invite link
Generates the link required to add the Bridge to a Discord server.
2019-03-03 19:33:16 +01:00
Lionstiger 835c349275 Add matrix-appservice-discord bridge
Bridge is setup to work on the matrix side with this, but the discord invite link is not automatically generated.
2019-03-03 18:22:52 +01:00
Slavi Pantaleev 45618679f5 Reload systemd services when they get updated
Fixes #69 (Github Issue)
2019-03-03 11:55:15 +02:00
Slavi Pantaleev a43bcd81fe Rename some variables 2019-02-28 11:51:09 +02:00
Slavi Pantaleev 433780384e Do not use docker_container module
Using `docker_container` with a `cap_drop` argument requires
Ansible >=2.7.

We want to support older versions too (2.4), so we either need to
stop invoking it with `cap_drop` (insecure), or just stop using
the module altogether.

Since it was suffering from other bugs too (not deleting containers
on failure), we've decided to remove `docker_container` usage completely.
2019-02-25 10:42:27 +02:00
Slavi Pantaleev 350b25690d Add Riot v1.0 (v1.0.1) support 2019-02-16 11:48:17 +02:00
Slavi Pantaleev eb08e20418 Upgrade Synapse (0.99.0 -> 0.99.1) and sync config
`matrix_synapse_no_tls` is now implicit, so we've gotten rid of it.

The `homeserver.yaml.j2` template has been synchronized with the
configuration generated by Synapse v0.99.1 (some new options
are present, etc.)
2019-02-14 18:40:55 +02:00
Slavi Pantaleev 40f3793af7 Upgrade Synapse to v0.99 and simplify dummy TLS cert logic 2019-02-06 09:17:55 +02:00
Slavi Pantaleev f6ebd4ce62 Initial work on Synapse 0.99/1.0 preparation 2019-02-05 12:09:46 +02:00
Aaron Raimist 1f0cc92b33
Use IPv4 localhost everywhere (or almost everywhere) 2019-02-04 09:49:45 -06:00
Slavi Pantaleev a9fae8e3b1 Revert "Use native OpenSSL module to generate passkey.pem"
This reverts commit 0dac5ea508.

Relying on pyOpenSSL is the Ansible way of doing things, but is
impractical and annoying for users.

`openssl` is easily available on most servers, even by default.
We'd better use that.
2019-01-31 20:45:14 +02:00
Plailect 0dac5ea508
Use native OpenSSL module to generate passkey.pem 2019-01-31 11:38:54 -05:00
Plailect 3a4a671dd7
Add support for matrix-appservice-irc 2019-01-31 00:37:23 -05:00
Slavi Pantaleev bf10331456 Make mautrix-whatsapp run as non-root and w/o capabilities 2019-01-28 15:55:58 +02:00
Slavi Pantaleev 8a3f942d93 Make mautrix-telegram run as non-root and w/o capabilities 2019-01-28 15:40:16 +02:00
Slavi Pantaleev 3e8a4159e6 Uncomment unintentionally-commented logic 2019-01-28 14:25:03 +02:00
Slavi Pantaleev 9438402f61 Drop capabilities in a few more places
Continuation of 316d653d3e
2019-01-28 11:43:32 +02:00
Slavi Pantaleev 299a8c4c7c Make (most) containers start as non-root
This makes all containers (except mautrix-telegram and
mautrix-whatsapp), start as a non-root user.

We do this, because we don't trust some of the images.
In any case, we'd rather not trust ALL images and avoid giving
`root` access at all. We can't be sure they would drop privileges
or what they might do before they do it.

Because Postfix doesn't support running as non-root,
it had to be replaced by an Exim mail server.

The matrix-nginx-proxy nginx container image is patched up
(by replacing its main configuration) so that it can work as non-root.
It seems like there's no other good image that we can use and that is up-to-date
(https://hub.docker.com/r/nginxinc/nginx-unprivileged is outdated).

Likewise for riot-web (https://hub.docker.com/r/bubuntux/riot-web/),
we patch it up ourselves when starting (replacing the main nginx
configuration).
Ideally, it would be fixed upstream so we can simplify.
2019-01-27 20:25:13 +02:00
Slavi Pantaleev f4f06ae068 Make matrix-nginx-proxy role independent of others
The matrix-nginx-proxy role can now be used independently.
This makes it consistent with all other roles, with
the `matrix-base` role remaining as their only dependency.

Separating matrix-nginx-proxy was relatively straightforward, with
the exception of the Mautrix Telegram reverse-proxying configuration.
Mautrix Telegram, being an extension/bridge, does not feel important enough
to justify its own special handling in matrix-nginx-proxy.

Thus, we've introduced the concept of "additional configuration blocks"
(`matrix_nginx_proxy_proxy_matrix_additional_server_configuration_blocks`),
where any module can register its own custom nginx server blocks.

For such dynamic registration to work, the order of role execution
becomes important. To make it possible for each module participating
in dynamic registration to verify that the order of execution is
correct, we've also introduced a `matrix_nginx_proxy_role_executed`
variable.

It should be noted that this doesn't make the matrix-synapse role
dependent on matrix-nginx-proxy. It's optional runtime detection
and registration, and it only happens in the matrix-synapse role
when `matrix_mautrix_telegram_enabled: true`.
2019-01-17 13:32:46 +02:00
Slavi Pantaleev c10182e5a6 Make roles more independent of one another
With this change, the following roles are now only dependent
on the minimal `matrix-base` role:
- `matrix-corporal`
- `matrix-coturn`
- `matrix-mailer`
- `matrix-mxisd`
- `matrix-postgres`
- `matrix-riot-web`
- `matrix-synapse`

The `matrix-nginx-proxy` role still does too much and remains
dependent on the others.

Wiring up the various (now-independent) roles happens
via a glue variables file (`group_vars/matrix-servers`).
It's triggered for all hosts in the `matrix-servers` group.

According to Ansible's rules of priority, we have the following
chain of inclusion/overriding now:
- role defaults (mostly empty or good for independent usage)
- playbook glue variables (`group_vars/matrix-servers`)
- inventory host variables (`inventory/host_vars/matrix.<your-domain>`)

All roles default to enabling their main component
(e.g. `matrix_mxisd_enabled: true`, `matrix_riot_web_enabled: true`).
Reasoning: if a role is included in a playbook (especially separately,
in another playbook), it should "work" by default.

Our playbook disables some of those if they are not generally useful
(e.g. `matrix_corporal_enabled: false`).
2019-01-16 18:05:48 +02:00
Slavi Pantaleev 51312b8250 Split playbook into multiple roles
As suggested in #63 (Github issue), splitting the
playbook's logic into multiple roles will be beneficial for
maintainability.

This patch realizes this split. Still, some components
affect others, so the roles are not really independent of one
another. For example:
- disabling mxisd (`matrix_mxisd_enabled: false`), causes Synapse
and riot-web to reconfigure themselves with other (public)
Identity servers.

- enabling matrix-corporal (`matrix_corporal_enabled: true`) affects
how reverse-proxying (by `matrix-nginx-proxy`) is done, in order to
put matrix-corporal's gateway server in front of Synapse

We may be able to move away from such dependencies in the future,
at the expense of a more complicated manual configuration, but
it's probably not worth sacrificing the convenience we have now.

As part of this work, the way we do "start components" has been
redone now to use a loop, as suggested in #65 (Github issue).
This should make restarting faster and more reliable.
2019-01-12 18:01:10 +02:00