Commit graph

67 commits

Author SHA1 Message Date
Slavi Pantaleev ad1425eee4 Add pgloader self-building support (for ARM) 2020-12-23 09:08:54 +02:00
Slavi Pantaleev 8675dedbdb Add support for automatic (nedb -> Postgres) migration to matrix-appservice-slack 2020-12-22 19:56:52 +02:00
Slavi Pantaleev 715bdf2c64 Add support for automatic (nedb -> Postgres) migration to mx-appservice-irc 2020-12-22 19:32:43 +02:00
Slavi Pantaleev 1bd5c240e5 Add support for executing additional DB migration statements
In cases where pgloader is not enough and we need to do some additional
migration work after it, we can now use
`additional_psql_statements_list` and
`additional_psql_statements_db_name`.

This is to be used when migrating `matrix-registration`'s data at the
very least.
2020-12-15 23:18:29 +02:00
Slavi Pantaleev dd797ba6a7 Fix Postgres database importing/upgrading conflicts
We were running into conflicts, because having initialized
the roles (users) and databases, trying to import leads to
errors (role XXX already exists, etc.).

We were previously ignoring the Synapse database (`homeserver`)
when upgrading/importing, because that one gets created by default
whenever the container starts.

For our additional databases, it's a similar situation now.
It's not created by default as soon as Postgres starts with an empty
database, but rather we create it as part of running the playbook.

So we either need to skip those role/database creation statements
while upgrading/importing, or to avoid creating the additional database
and rely on the import for that. I've gone for the former, because
it's already similar to what we were doing and it's simpler
(it lets `setup_postgres.yml` be the same in all scenarios).
2020-12-14 22:28:20 +02:00
transcaffeine 13d8a9b39c
hint supported automatic migration nedb->postgres 2020-12-14 16:33:40 +01:00
Slavi Pantaleev af3ea67bba Add (SQLite + Postgres) support and automatic migration to matrix-ma1sd 2020-12-14 17:16:25 +02:00
Slavi Pantaleev c1431b28f0 Make use of matrix_postgres_db_migration_request.caller 2020-12-14 16:13:57 +02:00
Slavi Pantaleev dc7850e83c Fix wording and variable names a bit 2020-12-14 16:03:40 +02:00
Slavi Pantaleev bc376c2fb2 Add database migration utility to matrix-postgres role 2020-12-14 16:03:40 +02:00
Slavi Pantaleev 5dba0c038b Make --tags=import-generic-sqlite-db commands not pass a sensitive connection string around
Instead of passing the connection string, we can now pass a name of a
variable, which contains a connection string.

Both are supported for having extra flexibility.
2020-12-14 11:47:00 +02:00
Slavi Pantaleev cb969c6ca2 Add --tags=import-generic-sqlite-db (pgloader import)
This can be used by various bridges, etc., to import an SQLite
(or some other supported) database into Postgres.
2020-12-14 02:23:29 +02:00
Slavi Pantaleev c66c084027 Merge branch 'master' into postgres-per-default 2020-12-14 01:51:15 +02:00
Slavi Pantaleev 6e1dfb62f0 Rename some doc files and commands related to importing
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.

Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.

We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
2020-12-14 01:51:00 +02:00
Slavi Pantaleev 183d2a10db Ensure matrix-postgres.service is started before creating additional users/databases 2020-12-14 00:59:59 +02:00
Slavi Pantaleev 46a4034d3e Use "password" for additional Postgres databases, not "pass"
Being more explicit sounds better.
2020-12-14 00:43:03 +02:00
Slavi Pantaleev 3a037a5993 Ensure additional databases contain all the keys that we expect 2020-12-14 00:39:38 +02:00
Slavi Pantaleev da4cb2f639 Do not use the postgresql_user/postgresql_db modules
While these modules are really nice and helpful, we can't use them
for at least 2 reasons:

- for us, Postgres runs in a container on a private Docker network
(`--network=matrix`) without usually being exposed to the host.
These modules execute on the host so they won't be able to reach it.

- these modules require `psycopg2`, so we need to install it before
using it. This might or might not be its own can of worms.
2020-12-14 00:31:38 +02:00
Slavi Pantaleev bbc09d013b Do not execute additional databases creation code if not necessary
The tasks in `create_additional_databases.yml` will likely
ensure `matrix-postgres.service` is started, etc.

If no additional databases are defined, we'd rather not execute that
file and all these tasks that it may do in the future.
2020-12-13 23:46:05 +02:00
Slavi Pantaleev c765ceb270 Prevent weird loop error
> Invalid data passed to 'loop', it requires a list, got this instead: matrix_postgres_additional_databases. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.

Well, or working around it, as I've done in this commit (which seems
more sane than `wantlist=True` stuff).
2020-12-13 22:56:56 +02:00
Slavi Pantaleev e2952f16f7 Determine matrix-postgres IP address without relying on jq
To avoid needing to have `jq` installed on the machine, we could:
- try to run jq in a Docker container using some small image providing
that
- better yet, avoid `jq` altogether
2020-12-13 22:45:48 +02:00
Slavi Pantaleev f47e8a97e6 Make use of matrix_host_command_docker instead of hardcoding 2020-12-13 22:38:35 +02:00
Slavi Pantaleev 0641106370 Allow username of additional Postgres databases to be different
We'll most likely use one that matches the database name, but
it's better to have it configurable.
2020-12-13 22:37:04 +02:00
Slavi Pantaleev 527d5f57d5 Relocate Postgres additional database creation logic
Moving it above the "uninstalling" set of tasks is better.
Extracting it out to another file at the same time, for readability,
especially given that it will probably have to become more complex in
the future (potentially installing `jq`, etc.)
2020-12-13 22:37:04 +02:00
Slavi Pantaleev 77a5c7cf3c Merge branch 'master' into postgres-per-default 2020-12-13 21:04:15 +02:00
Slavi Pantaleev 47613e5a27 Remove synapse-janitor support
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/746
2020-12-11 23:24:42 +02:00
transcaffeine d9f4914e0d
WIP: postgres: create databases for all services
If a service is enabled, a database for it is created in postgres with a uniqque password. The service can then use this database for data storage instead of relying on sqlite.
2020-12-10 18:26:22 +01:00
Slavi Pantaleev 796c752b60 Ensure Postgres passwords are not longer than 99 characters
Complements https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/732
2020-11-26 09:51:48 +02:00
Slavi Pantaleev 75f9fde7a4 Remove some more -v usage
Continuation of 1fca917ad1.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/722
2020-11-25 10:49:59 +02:00
Slavi Pantaleev 6a72e3fa54 Try to make importing SQLite from older Synapse version work
If the SQLite database was from an older version of Synapse, it appears
that Synapse would try to run migrations on it first, before importing.
This was failing, because the file wasn't writable.

Hopefully, this fixes the problem.
2020-10-07 08:54:46 +03:00
Dan Arnfield 3a3383fada Add support for postgres 13 2020-09-30 16:50:59 -05:00
Slavi Pantaleev a456e3a9e7 Surface certain messages at the end of playbook execution
Fixes #106 (Github Issue).
2020-09-01 13:12:35 +03:00
Slavi Pantaleev e3dca2f66f Try to avoid Docker logs growing too much for one-off containers
We recently had a report of the Postgres backup container's log file
growing the size of /var/lib/docker until it ran out of disk space.

Trying to prevent similar problems in the future.
2020-09-01 09:03:48 +03:00
Chris van Dijk 6e3b877dc2 Remove hardcoded command paths in playbook shell usage 2020-05-27 23:14:56 +02:00
Ugurtan 5ace3f4a1c
fix for importing sqllite database
the current version fails the import, because the volume for the media is missing. It still fails if you have the optional shared secret password provider is enabled, so that might need another mount. Commenting out the password provider in the hoimeserver.yaml during the run works as well.
2020-05-27 18:13:36 +02:00
Slavi Pantaleev 8fb3ce6f6d Upgrade Synapse (v1.12.4 -> v1.13.0) 2020-05-19 21:35:32 +03:00
Chris van Dijk 7585bcc4ac Allow the matrix user username and groupname to be configured separately
No migration steps should be required.
2020-05-01 19:59:32 +02:00
Slavi Pantaleev 9991d545dd Add note about synapse-janitor and require explicit confirmation for usage
Fixes #465 (Github Issue).

Related to https://github.com/xwiki-labs/synapse_scripts/pull/12.
2020-04-21 21:48:06 +03:00
mooomooo eebc6e13f8 Made directory variables for /etc/systemd/system , /etc/cron.d , /usr/local/bin 2020-03-24 11:27:58 -07:00
Slavi Pantaleev 704d2ff861
Merge pull request #351 from prasket/matrix-change-user-admin-status
Matrix change user admin status
2020-01-23 10:35:28 +02:00
prasket 83e7a8d9e6 corrected hardcoded matrix domain and added code to remove old matrix-make-user-admin script per suggestion. 2020-01-21 18:29:17 -08:00
Slavi Pantaleev 9d3d538a2d Use C collation for Postgres to appease Synapse 2020-01-21 00:13:47 +02:00
prasket e99497bb60 created new file with change admin options and removed make admin file. Updated name references as well throughout the project. 2020-01-19 18:39:56 -08:00
Slavi Pantaleev e32aaacaa7 Make gzipped SQL dumps by default during --upgrade-postgres 2019-10-05 11:42:08 +03:00
Slavi Pantaleev 29526e7bb1 Add support for backing up / importing all Postgres databases/users 2019-10-05 11:42:08 +03:00
Slavi Pantaleev 9c438a3870 Add support for Postgres v12 2019-10-04 08:51:36 +03:00
Slavi Pantaleev 1784bde226 Ensure Postgres data ownership is correct
Fixes #257 (Github Issue)
2019-09-02 10:01:31 +03:00
Slavi Pantaleev 0ca21d80d7 Add Synapse Maintenance docs and synapse-janitor integration 2019-07-08 09:38:36 +03:00
Slavi Pantaleev 631a14bf0c Rename run control variables for consistency 2019-07-08 09:38:36 +03:00
Slavi Pantaleev e805044b80 Delete scripts when uninstalling Postgres 2019-07-08 09:38:36 +03:00