Commit graph

128 commits

Author SHA1 Message Date
rakshazi 2f887f292c
added "matrix_%SERVICE%_version" variable to all roles, use it in "matrix_%SERVICE%_docker_image" var (preserving backward-compatibility) 2021-02-20 19:08:28 +02:00
Slavi Pantaleev 5cfeae806b Merge branch 'master' into synapse-workers 2021-02-14 13:00:57 +02:00
Slavi Pantaleev c4a05b760a Make mautrix bridges not overwrite their config
If they do, our next playbook runs would simply revert it
and report "changed" for that task.

There's no benefit to letting the bridge spew a new config file.

This does not apply to the mautrix whatsapp bridge, because that one
is written in Go (not Python) and takes different flags. There's no
equivalent flag there.
2021-02-03 13:23:18 +02:00
Slavi Pantaleev a49dab76f8 Merge branch 'master' into synapse-workers 2021-01-27 15:49:16 +02:00
Slavi Pantaleev e3290d8bcb Remove |to_json causing trouble
Fixes a regression introduced in f6097fbba1, which was cauing Synapse
to die with this error message:

> ValueError: sender_localpart needs characters which are not URL encoded.
2021-01-27 15:48:35 +02:00
Slavi Pantaleev a31c9603fa Merge branch 'master' into synapse-workers 2021-01-27 15:43:56 +02:00
Slavi Pantaleev f6097fbba1 E2BE not working for mautrix bridges
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/806
2021-01-27 15:43:33 +02:00
Slavi Pantaleev d98a1ceadd Merge branch 'master' into synapse-workers 2021-01-27 10:27:17 +02:00
Slavi Pantaleev 512f42aa76 Do not report docker kill/rm attempts as errors
These are just defensive cleanup tasks that we run.
In the good case, there's nothing to kill or remove, so they trigger an
error like this:

> Error response from daemon: Cannot kill container: something: No such container: something

and:

> Error: No such container: something

People often ask us if this is a problem, so instead of always having to
answer with "no, this is to be expected", we'd rather eliminate it now
and make logs cleaner.

In the event that:
- a container is really stuck and needs cleanup using kill/rm
- and cleanup fails, and we fail to report it because of error
suppression (`2>/dev/null`)

.. we'd still get an error when launching ("container name already in use .."),
so it shouldn't be too hard to investigate.
2021-01-27 10:22:46 +02:00
Slavi Pantaleev 869727a402 Add comment to mautrix-facebook bridge regarding alembic migrations 2021-01-27 10:17:48 +02:00
Slavi Pantaleev d3ecc6f017 Fix bridges failing to upload media when Synapse workers are enabled 2021-01-25 13:55:08 +02:00
Slavi Pantaleev 024a23ed17 Upgrade mautrix-facebook to the new Postgres-only version
I had intentionally held it back in 39ea3496a4
until:
- it received more testing (there were a few bugs during the
migration, but now it seems OK)
- this migration guide was written
2021-01-20 10:12:51 +02:00
Slavi Pantaleev 39ea3496a4 Downgrade/lock mautrix-facebook to pre-mobile times
The `mobile` branch got merged to `master`, which ends up becoming
`:latest`. It's a "rewrite" of the bridge's backend and only
supports a Postgres database.

We'd like to go back (well, forward) to `:latest`, but that will take
a little longer, because:
- we need to handle and document things for people still on SQLite
(especially those with external Postgres, who are likely on SQLite for
bridges)
- I'd rather test the new builds (and migration) a bit before
releasing it to others and possibly breaking their bridge

Brave ones who are already using the bridge with Postgres
can jump on `:latest` and report their experience.
2021-01-19 18:44:15 +02:00
Slavi Pantaleev 1692a28fe4 Work around annoying Docker warning about undefined $HOME
> WARNING: Error loading config file: .dockercfg: $HOME is not defined

.. which appeared in Docker 20.10.
2021-01-15 00:23:01 +02:00
Slavi Pantaleev 05ca9357a8 Add .service suffix to systemd units list
We'll be adding `.timer` units later on, so it's good to be
more explicit.
2021-01-14 23:02:10 +02:00
Slavi Pantaleev a2a4218e95 Make mautrix-python-based bridges E2EE happier
Fixes a problem like this:
> File "/usr/lib/python3.8/site-packages/mautrix/bridge/e2ee.py", line 79, in __init__
> raise RuntimeError("Unsupported database scheme")

mautrix-python's e2ee.py module expects to find `postgres://` instead of
`postgresql://`.
2020-12-23 15:39:12 +02:00
Stuart Mumford 019a4d7dcd Use role relative paths for things 2020-12-23 11:34:48 +00:00
Slavi Pantaleev be0c599565 Feed more slashes to mautrix bridges when using SQLite
This makes the `sqlite://` URI match what we were using before
and what the config expects.
2020-12-23 13:33:25 +02:00
Slavi Pantaleev 15f4cc924d Rename variables (_database_db_name -> _database_name) 2020-12-22 17:10:02 +02:00
Slavi Pantaleev bbc08722c5 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-facebook 2020-12-14 16:14:23 +02:00
Slavi Pantaleev b9a04a7f95 Rename some remaining matrix_*_postgres_* vars back to matrix_*_database_*
Looks like there are some that I missed in 087dbe4ddc
2020-12-14 14:42:18 +02:00
Slavi Pantaleev 087dbe4ddc Rename matrix_*_postgres_* back to matrix_*_database_*
I was thinking that it makes sense to be more specific,
and using `_postgres_` also separated these variables
from the `_database_` variables that ended up in bridge configuration.

However, @jdreichmann makes a good point
(https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/740#discussion_r542281102)
that we don't need to be so specific and can allow for other engines (like MySQL) to use these variables.
2020-12-14 13:02:47 +02:00
Slavi Pantaleev 43d6ff2af8 Fix sqlite usage for mautrix-facebook/mautrix-telegram
Regression since 2d99ade72f and 9bf8ce878e, respectively.

When SQLite is to be used, these bridges expect an `sqlite://`
connection string, and not a plain file name (path), like Appservice
Discord and mautrix-whatsapp do.
2020-12-14 12:30:10 +02:00
Slavi Pantaleev a3406a182b Move some things around 2020-12-14 12:04:47 +02:00
Slavi Pantaleev 2d99ade72f Add (Postgres + SQLite) support to matrix-mautrix-facebook bridge 2020-12-14 11:50:42 +02:00
transcaffeine d9f4914e0d
WIP: postgres: create databases for all services
If a service is enabled, a database for it is created in postgres with a uniqque password. The service can then use this database for data storage instead of relying on sqlite.
2020-12-10 18:26:22 +01:00
Slavi Pantaleev d08b27784f Fix systemd services autostart problem with Docker 20.10
The Docker 19.04 -> 20.10 upgrade contains the following change
in `/usr/lib/systemd/system/docker.service`:

```
-BindsTo=containerd.service
-After=network-online.target firewalld.service containerd.service
+After=network-online.target firewalld.service containerd.service multi-user.target
-Requires=docker.socket
+Requires=docker.socket containerd.service
Wants=network-online.target
```

The `multi-user.target` requirement in `After` seems to be in conflict
with our `WantedBy=multi-user.target` and `After=docker.service` /
`Requires=docker.service` definitions, causing the following error on
startup for all of our systemd services:

> Job matrix-synapse.service/start deleted to break ordering cycle starting with multi-user.target/start

A workaround which appears to work is to add `DefaultDependencies=no`
to all of our services.
2020-12-10 11:43:20 +02:00
Slavi Pantaleev 04da1bddf7 Update matrix-mautrix-facebook config a bit
This also disables presence if it's disabled for Synapse.
2020-12-01 11:55:18 +02:00
Slavi Pantaleev be5263f397 Move self-building git repository URLs to variables (stop hardcoding) 2020-11-28 21:34:14 +02:00
Slavi Pantaleev 5eed874199 Improve self-building experience (avoid conflict with pullable images)
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/716

This patch makes us use more fully-qualified container image names
(either prefixed with docker.io/ or with localhost/).

The latter happens when self-building is enabled.

We've recently had issues where if an image was removed manually
and the service was restarted (making `docker run` fetch it from Docker Hub, etc.),
we'd end up with a pulled image, even though we're aiming for a self-built one.
Re-running the playbook would then not do a rebuild, because:
- the image with that name already exists (even though it's something
else)
- we sometimes had conditional logic where we'd build only if the git
repo changed

By explicitly changing the name of the images (prefixing with localhost/),
we avoid such confusion and the possibility that we'd automatically pul something
which is not what we expect.

Also, I've removed that condition where building would happen on git
changes only. We now always build (unless an image with that name
already exists). We just force-build when the git repo changes.
2020-11-14 23:00:49 +02:00
Scott Crossen fa5d85426b Renamed systemd descriptions for all bridges 2020-10-13 16:40:30 -07:00
Slavi Pantaleev 440569c47b Update mautrix-facebook configuration 2020-06-10 09:20:06 +03:00
Slavi Pantaleev b0b744dede Add periodic reconnection configuration to mautrix-facebook bridge 2020-06-05 15:21:41 +03:00
Slavi Pantaleev c00a7c2fe9 Update mautrix-facebook configuration (invitation, backfilling support) 2020-06-03 09:35:31 +03:00
Slavi Pantaleev 5da31ba579 Move configuration templates outside of defaults/main.yml files 2020-06-03 09:33:28 +03:00
Chris van Dijk 6334f6c1ea Remove hardcoded command paths in systemd unit files
Depending on the distro, common commands like sleep and chown may either
be located in /bin or /usr/bin.

Systemd added path lookup to ExecStart in v239, allowing only the
command name to be put in unit files and not the full path as
historically required. At least Ubuntu 18.04 LTS is however still on
v237 so we should maintain portability for a while longer.
2020-05-27 23:14:54 +02:00
Slavi Pantaleev a353217cf3 Add delivery_receipts option for matrix-bridge-mautrix-facebook 2020-05-26 09:49:12 +03:00
Chris van Dijk 7585bcc4ac Allow the matrix user username and groupname to be configured separately
No migration steps should be required.
2020-05-01 19:59:32 +02:00
Slavi Pantaleev 4d260c0dd5 Add encryption configuration defaults to Mautrix bridges
Related to #451 (Github Issue).
2020-04-17 09:44:48 +03:00
Horvath Gergely 3d096a2b13 make selfbuild be forced on (and only) git master change 2020-04-06 18:36:17 +02:00
mooomooo eebc6e13f8 Made directory variables for /etc/systemd/system , /etc/cron.d , /usr/local/bin 2020-03-24 11:27:58 -07:00
Slavi Pantaleev 027056e027 Fix weird path creation
Fixes #403 (Github Issue).
2020-03-18 18:24:30 +02:00
Slavi Pantaleev 3cee815baf Fix some typos 2020-03-15 11:34:35 +02:00
Slavi Pantaleev 2b85fde103 Rename some variables for consistency 2020-03-15 10:15:27 +02:00
Slavi Pantaleev 8fe97abe7d Wire matrix_container_images_self_build to self_build variables via group_vars/matrix_servers
This keeps the roles cleaner and more independent of matrix-base,
which may be important for people building their own playbook
out of the individual roles and not using the matrix-base role.
2020-03-15 10:10:41 +02:00
Horvath Gergely 30fd1ab2d6 add forgotten support for variable for matrix-bridge-mautrix-facebook role 2020-03-14 20:40:31 +01:00
Horvath Gergely 3c8535c3bc check ansible version for self-building in every role 2020-03-08 19:17:10 +01:00
Horvath Gergely a5d94eec0b refactor variable names 2020-03-08 00:28:14 +01:00
Horvath Gergely 310aa685f9 refactor based on Slavi's requests 2020-03-08 00:24:00 +01:00
Horvath Gergely 8c1e00a6cd add mautrix-facebook support for raspberry pi 2020-02-21 07:56:28 +01:00
Slavi Pantaleev bd38861179 Add support for automatic Double Puppeting for all Mautrix bridges 2020-01-12 20:28:36 +02:00
Slavi Pantaleev 53186ffa1c Sync configuration with upstream for all Mautrix bridges 2020-01-12 19:10:05 +02:00
Marcel Partap 7ead77fdb7 Remove spurious 'mxisd' string from mautrix bridge templates comment 2020-01-08 22:39:21 +01:00
Michael Haak 5b213e6ad0 Replace constructs appending elements with variables to matrix_synapse_container_extra_arguments. Fixes issue https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/304 2019-11-09 23:16:12 +01:00
Slavi Pantaleev c4eebc5355 Add information about community-grouping for the Facebook bridge 2019-08-24 12:14:17 +03:00
Slavi Pantaleev 82bb55ae7a Use new default port config for mautrix-facebook 2019-07-28 18:42:42 +03:00
Slavi Pantaleev b0162d6f75 Use mautrix-facebook Docker image from new official registry 2019-07-28 18:40:55 +03:00
Slavi Pantaleev 37c8b96d06 Use stricter regex in bridges' registration.yaml
I've been thinking of doing before, but haven't.

Now that the Whatsapp bridge does it (since 4797469383),
it makes sense to do it for all other bridges as well.
(Except for the IRC bridge - that one manages most of registration.yaml by itself)
2019-06-24 07:50:51 +03:00
Slavi Pantaleev 668f98a2d3 Escape domain in bridge registration regex 2019-06-19 10:40:59 +03:00
Slavi Pantaleev 2902b53267 Minor fixes for consistency 2019-06-15 09:42:40 +03:00
Slavi Pantaleev 00383a73ac Make running --tags=setup-synapse only not fail to register bridges
Until now, if `--tags=setup-synapse` was used, bridge tasks would not
run and bridges would fail to register with the `matrix-synapse` role.
This means that Synapse's configuration would be generated with an empty
list of appservices (`app_service_config_files: []`).

.. and then bridges would fail, because Synapse would not be aware of
there being any bridges.

From now on, bridges always run their init tasks and always register
with Synapse.

For the Telegram bridge, the same applies to registering with
matrix-nginx-proxy. Previously, running `--tags=setup-nginx-proxy` would
get rid of the Telegram endpoint configuration for the same reason.
Not anymore.
2019-06-14 10:19:52 +03:00
Slavi Pantaleev 7d3adc4512 Automatically force-pull :latest images
We do use some `:latest` images by default for the following services:
- matrix-dimension
- Goofys (in the matrix-synapse role)
- matrix-bridge-appservice-irc
- matrix-bridge-appservice-discord
- matrix-bridge-mautrix-facebook
- matrix-bridge-mautrix-whatsapp

It's terribly unfortunate that those software projects don't release
anything other than `:latest`, but that's how it is for now.

Updating that software requires that users manually do `docker pull`
on the server. The playbook didn't force-repull images that it already
had.

With this patch, it starts doing so. Any image tagged `:latest` will be
force re-pulled by the playbook every time it's executed.

It should be noted that even though we ask the `docker_image` module to
force-pull, it only reports "changed" when it actually pulls something
new. This is nice, because it lets people know exactly when something
gets updated, as opposed to giving the indication that it's always
updating the images (even though it isn't).
2019-06-10 14:30:28 +03:00
Slavi Pantaleev a9953dd641 Make Facebook/Telegram bridges not log to files
We log to journald anyway. There's no need for double-logging.

It should not that matrix-synapse logs to journald and to files,
but that's likely to change in the future as well.
Because Synapse's logs are insanely verbose right now (and may get
dropped by journald), it's more reliable to have file-logging too.

As Synapse matures and gets more stable, logging should hopefully
get less, we should be able to only use journald and stop writing to
files for it as well.
2019-06-07 15:48:13 +03:00
Slavi Pantaleev 18baeabdf2 Do not create Facebook bridge directories with recurse: true
I'm not sure what I had in mind when I added this earlier,
but I think we'd better go without it.
2019-06-07 15:18:29 +03:00
Slavi Pantaleev bf446b6e15 Fix double mv command 2019-06-07 15:06:21 +03:00
Slavi Pantaleev 172b0fa88c Separate Facebook bridge configuration and data
Using a separate directory allows easier backups
(only need to back up the Ansible playbook configuration and the
bridge's `./data` directory).

The playbook takes care of migrating an existing database file
from the base directory into the `./data` directory.

In the future, we can also mount the configuration read-only,
to ensure the bridge won't touch it.
For now, mautrix-facebook is keen on rebuilding the `config.yaml`
file on startup though, so this will have to wait.
2019-06-07 14:52:38 +03:00
Slavi Pantaleev 330648a3e0 Make Facebook bridge configuration playbook-managed
Related to #193, but for the Facebook bridge.
(other bridges can be changed to do the same later).

This patch makes the bridge configuration entirely managed by the
Ansible playbook. The bridge's `config.yaml` and `registration.yaml`
configuration files are regenerated every time the playbook runs.

This allows us to apply updates to those files and to avoid
people having to manage the configuration files manually on the server.

-------------------------------------------------------------

A deficiency of the current approach to dumping YAML configuration in
`config.yaml` is that we strip all comments from it.
Later on, when the bridge actually starts, it will load and redump
(this time with comments), which will make the `config.yaml` file
change.

Subsequent playbook runs will report "changed" for the
"Ensure mautrix-facebook config.yaml installed" task, which is a little
strange.

We might wish to improve this in the future, if possible.

Still, it's better to have a (usually) somewhat meaningless "changed"
task than to what we had -- never rebuilding the configuration.
2019-06-07 14:05:53 +03:00
Slavi Pantaleev d6d6c152a3 Delay bridge startup to ensure Synapse is up
Bridges start matrix-synapse.service as a dependency, but
Synapse is sometimes slow to start, while bridges are quick to
hit it and die (if unavailable).

They'll auto-restart later, but .. this still breaks `--tags=start`,
which doesn't wait long enough for such a restart to happen.

This attempts to slow down bridge startup enough to ensure Synapse
is up and no failures happen at all.
2019-06-07 12:15:37 +03:00
Slavi Pantaleev 70487061f4 Prefer --mount instead of -v for mounting volumes
This doesn't replace all usage of `-v`, but it's a start.

People sometimes troubleshoot by deleting files (especially bridge
config files). Restarting Synapse with a missing registration.yaml file
for a given bridge, causes the `-v
/something/registration.yaml:/something/registration.yaml:ro` option
to force-create `/something/registration.yaml` as a directory.

When a path that's provided to the `-v` option is missing, Docker
auto-creates that path as a directory.
This causes more breakage and confusion later on.

We'd rather fail, instead of magically creating directories.
Using `--mount`, instead of `-v` is the solution to this.

From Docker's documentation:

> When you use --mount with type=bind, the host-path must refer to an existing path on the host.
> The path will not be created for you and the service will fail with an error if the path does not exist.
2019-05-29 09:59:50 +03:00
Slavi Pantaleev be2812bc8f Remove unnecessary variables
Continuation of 54a281a425.
Related to #188 (Github Pull Request).
2019-05-24 08:01:24 +09:00
Jason Locklin 54a281a425 Config validation not required
Validate_config was copied from the telegram bridge code, but doesn't
apply to the facebook bridge.
2019-05-23 15:02:44 -04:00
Dan Arnfield 9c23d877fe Fix docker_image option for ansible < 2.8 2019-05-22 05:43:33 -05:00
Dan Arnfield db15791819 Add source option to docker_image to fix deprecation warning 2019-05-21 10:29:12 -05:00
Dan Arnfield 3982f114af Fix CONDITIONAL_BARE_VARS deprecation warning in ansible 2.8 2019-05-21 10:25:59 -05:00
Slavi Pantaleev 3250df6765 Make bridge uninstallation stop services
Fixes #155 (Github Issue)
2019-05-21 11:27:09 +09:00
Slavi Pantaleev 8d654aecdd Improve file naming consistency 2019-05-21 09:57:48 +09:00
Slavi Pantaleev 13c4e7e5b6 Merge branch 'master' into separate-bridge-roles 2019-05-16 09:45:06 +09:00
Slavi Pantaleev 43fd3cc274 Move mautrix-facebook into a separate role 2019-05-15 09:34:31 +09:00