Commit graph

1290 commits

Author SHA1 Message Date
Slavi Pantaleev e64758c119 Add missing restart task
Should have been part of 149872e00c
2020-12-22 13:24:53 +02:00
Slavi Pantaleev 149872e00c Add support for automatic (Postgres -> SQLite) migration to mx-puppet-discord 2020-12-22 11:10:10 +02:00
Slavi Pantaleev 9b4bf73587 Fix undefined variable reference 2020-12-22 11:08:07 +02:00
Slavi Pantaleev 6488e11d69 Relocate some tasks 2020-12-22 10:52:36 +02:00
Slavi Pantaleev 0f4649a45c Merge branch 'master' into postgres-per-default 2020-12-16 03:35:39 +02:00
Slavi Pantaleev a4b8baee49 Fix inability to send (Matrix -> Discord) messages via appservice-discord
Revert "Correct inabillity for appservice-discord to connect"
This reverts commit 673e19f830.

While certain things do work even with such a local URL, sending
messages leads to an error like this:

> [DiscordBot] verbose: DiscordAPIError: Invalid Form Body
> avatar_url: Not a well formed URL.

Fixes https://github.com/Half-Shot/matrix-appservice-discord/issues/649

The sample configuration file for appservice-discord
c29cfc72f5/config/config.sample.yaml (L8)
explicitly says that we need a public URL.
2020-12-16 03:35:13 +02:00
Slavi Pantaleev a197968b7f Make matrix-registration use Postgres by default
Now that 0.7.2 is out, the Docker image supports Postgres
and we can do the (SQLite -> Postgres) migration.

I've also found out that we needed to fix up the `tokens.ex_date` column
data type a bit to prevent matrix-registration from raising exceptions
when comparing `datetime.now()` with `ex_date` coming from the database.

Example:

> File "/usr/local/lib/python3.8/site-packages/matrix_registration/tokens.py", line 58, in valid
> expired = self.ex_date < datetime.now()
> TypeError: can't compare offset-naive and offset-aware datetimes
2020-12-15 23:19:56 +02:00
Slavi Pantaleev 1bd5c240e5 Add support for executing additional DB migration statements
In cases where pgloader is not enough and we need to do some additional
migration work after it, we can now use
`additional_psql_statements_list` and
`additional_psql_statements_db_name`.

This is to be used when migrating `matrix-registration`'s data at the
very least.
2020-12-15 23:18:29 +02:00
Slavi Pantaleev 3289298ac7 Merge branch 'master' into postgres-per-default 2020-12-15 22:02:52 +02:00
Slavi Pantaleev 69f71f48a6 Upgrade matrix-registration (v0.7.1 -> 0.7.2) and use official image
This switches us to a container image maintained by the
matrix-registration developer.

0.7.2 also supports a `base_url` configuration option we can use to
make it easier to reverse-proxy at a different base URL.

We still keep some workarounds, because of this issue:
https://github.com/ZerataX/matrix-registration/issues/47
2020-12-15 22:02:06 +02:00
Slavi Pantaleev e2ba46bf01 Fix Jinja2 syntax error (else if -> elif) 2020-12-14 22:40:37 +02:00
Slavi Pantaleev dd797ba6a7 Fix Postgres database importing/upgrading conflicts
We were running into conflicts, because having initialized
the roles (users) and databases, trying to import leads to
errors (role XXX already exists, etc.).

We were previously ignoring the Synapse database (`homeserver`)
when upgrading/importing, because that one gets created by default
whenever the container starts.

For our additional databases, it's a similar situation now.
It's not created by default as soon as Postgres starts with an empty
database, but rather we create it as part of running the playbook.

So we either need to skip those role/database creation statements
while upgrading/importing, or to avoid creating the additional database
and rely on the import for that. I've gone for the former, because
it's already similar to what we were doing and it's simpler
(it lets `setup_postgres.yml` be the same in all scenarios).
2020-12-14 22:28:20 +02:00
Slavi Pantaleev 2a502db239 Add (SQLite + Postgres) support and automatic migration to matrix-dimension 2020-12-14 21:01:47 +02:00
Slavi Pantaleev 0790a7b2a8 Add support for matrix_dimension_systemd_{required,wanted}_services_list
We were referencing them from `group_vars/matrix_servers` since
recently, but there were no such variables and they weren't being put to
use.
2020-12-14 20:31:07 +02:00
Slavi Pantaleev 374f43735a Separate matrix-dimension install/uninstall tasks 2020-12-14 20:05:31 +02:00
Slavi Pantaleev 8d74593878 Prepare matrix-registation for (SQLite + Postgres) support
Auto-migration and everything seems to work. It's just that
matrix-registration cannot load the Python modules required
for talking to a Postgres database.

Tracked here: https://github.com/ZerataX/matrix-registration/issues/44

Until this gets fixed, we'll continue default to 'sqlite'.
2020-12-14 18:58:37 +02:00
Slavi Pantaleev 516ccb2b2b Separate matrix-registration install/uninstall tasks 2020-12-14 18:12:14 +02:00
transcaffeine 13d8a9b39c
hint supported automatic migration nedb->postgres 2020-12-14 16:33:40 +01:00
Slavi Pantaleev af3ea67bba Add (SQLite + Postgres) support and automatic migration to matrix-ma1sd 2020-12-14 17:16:25 +02:00
Slavi Pantaleev 0ca48f3532 Separate matrix-ma1sd install/uninstall tasks 2020-12-14 16:57:51 +02:00
Slavi Pantaleev 7248eb3c11 Fix syntax error in roles/matrix-bridge-appservice-irc/defaults/main.yml 2020-12-14 16:25:44 +02:00
Slavi Pantaleev cba973d6b5 Enable automatic (SQLite -> Postgres) migration for matrix-appservice-discord 2020-12-14 16:25:22 +02:00
Slavi Pantaleev 13f84e2ad5 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-whatsapp 2020-12-14 16:21:01 +02:00
Slavi Pantaleev 86a8091768 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-telegram 2020-12-14 16:19:54 +02:00
Slavi Pantaleev 3ba8520266 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-hangouts 2020-12-14 16:18:38 +02:00
Slavi Pantaleev bbc08722c5 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-facebook 2020-12-14 16:14:23 +02:00
Slavi Pantaleev c1431b28f0 Make use of matrix_postgres_db_migration_request.caller 2020-12-14 16:13:57 +02:00
Slavi Pantaleev ac37091d01 Enable automatic (SQLite -> Postgres) migration for matrix-reminder-bot 2020-12-14 16:03:40 +02:00
Slavi Pantaleev dc7850e83c Fix wording and variable names a bit 2020-12-14 16:03:40 +02:00
Slavi Pantaleev bc376c2fb2 Add database migration utility to matrix-postgres role 2020-12-14 16:03:40 +02:00
transcaffeine 54da61f81b
add postgres support mx-appservice-[slack|irc] with fallback to nedb in role and migration notice 2020-12-14 14:08:35 +01:00
Slavi Pantaleev e3a0c9adda Add (Postgres + SQLite) support to matrix-reminder-bot
This has been tested and appears to work.
2020-12-14 15:02:11 +02:00
Slavi Pantaleev dde1c9f899 Fix indentation causing YAML syntax error 2020-12-14 14:53:35 +02:00
Slavi Pantaleev aa828ff9f6 Separate matrix-reminder-bot install/uninstall tasks 2020-12-14 14:50:04 +02:00
Slavi Pantaleev b9a04a7f95 Rename some remaining matrix_*_postgres_* vars back to matrix_*_database_*
Looks like there are some that I missed in 087dbe4ddc
2020-12-14 14:42:18 +02:00
transcaffeine 5d70bc1376
add postgres support for mx-puppet-* with fallback to sqlite in role and migration notice 2020-12-14 13:22:58 +01:00
Slavi Pantaleev 087dbe4ddc Rename matrix_*_postgres_* back to matrix_*_database_*
I was thinking that it makes sense to be more specific,
and using `_postgres_` also separated these variables
from the `_database_` variables that ended up in bridge configuration.

However, @jdreichmann makes a good point
(https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/740#discussion_r542281102)
that we don't need to be so specific and can allow for other engines (like MySQL) to use these variables.
2020-12-14 13:02:47 +02:00
Slavi Pantaleev ce21ea3640 Add (Postgres + SQLite) support to matrix-mautrix-hangouts bridge
I don't use this bridge, so this is completely untested.
2020-12-14 12:34:59 +02:00
Slavi Pantaleev 43d6ff2af8 Fix sqlite usage for mautrix-facebook/mautrix-telegram
Regression since 2d99ade72f and 9bf8ce878e, respectively.

When SQLite is to be used, these bridges expect an `sqlite://`
connection string, and not a plain file name (path), like Appservice
Discord and mautrix-whatsapp do.
2020-12-14 12:30:10 +02:00
Slavi Pantaleev 6c77eae969 Add (Postgres + SQLite) support to matrix-mautrix-whatsapp bridge
I don't use this bridge, so this is completely untested.
2020-12-14 12:24:37 +02:00
Slavi Pantaleev 9bf8ce878e Add (Postgres + SQLite) support to matrix-mautrix-telegram bridge
I don't use this bridge, so this is completely untested.
2020-12-14 12:06:28 +02:00
Slavi Pantaleev a3406a182b Move some things around 2020-12-14 12:04:47 +02:00
Slavi Pantaleev 2d99ade72f Add (Postgres + SQLite) support to matrix-mautrix-facebook bridge 2020-12-14 11:50:42 +02:00
Slavi Pantaleev 5dba0c038b Make --tags=import-generic-sqlite-db commands not pass a sensitive connection string around
Instead of passing the connection string, we can now pass a name of a
variable, which contains a connection string.

Both are supported for having extra flexibility.
2020-12-14 11:47:00 +02:00
Slavi Pantaleev d91aa5a060 Do not introduce sub-variables exposing implementation details 2020-12-14 10:52:07 +02:00
Slavi Pantaleev f1e85f7112 Don't mention Postgres roles, just say users 2020-12-14 10:04:37 +02:00
Slavi Pantaleev 4617984b9f Add (SQLite -> Postgres) migration instructions 2020-12-14 02:24:32 +02:00
Slavi Pantaleev cb969c6ca2 Add --tags=import-generic-sqlite-db (pgloader import)
This can be used by various bridges, etc., to import an SQLite
(or some other supported) database into Postgres.
2020-12-14 02:23:29 +02:00
Slavi Pantaleev c66c084027 Merge branch 'master' into postgres-per-default 2020-12-14 01:51:15 +02:00
Slavi Pantaleev 6e1dfb62f0 Rename some doc files and commands related to importing
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.

Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.

We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
2020-12-14 01:51:00 +02:00