Commit graph

120 commits

Author SHA1 Message Date
Slavi Pantaleev 5cfeae806b Merge branch 'master' into synapse-workers 2021-02-14 13:00:57 +02:00
Slavi Pantaleev 7d39e5153a Upgrade Postgres minor versions 2021-02-14 09:12:29 +02:00
Julian Foad d1f28d17bb Allow psql args to be given to matrix-postgres-cli
This passes any arguments given to 'matrix-postgres-cli' to the 'psql' command.

Examples:
  $ # start an interactive shell connected to a given db
  $ sudo matrix-postgres-cli -d synapse
  $ # run a query, non-interactively
  $ sudo matrix-postgres-cli -d synapse -c 'SELECT group_id FROM groups;'
2021-02-03 12:59:21 +00:00
o8F0LY 0a0c9a4efc Add double quotes to avoid synatx errors 2021-01-30 22:54:51 +01:00
Slavi Pantaleev d98a1ceadd Merge branch 'master' into synapse-workers 2021-01-27 10:27:17 +02:00
Slavi Pantaleev 512f42aa76 Do not report docker kill/rm attempts as errors
These are just defensive cleanup tasks that we run.
In the good case, there's nothing to kill or remove, so they trigger an
error like this:

> Error response from daemon: Cannot kill container: something: No such container: something

and:

> Error: No such container: something

People often ask us if this is a problem, so instead of always having to
answer with "no, this is to be expected", we'd rather eliminate it now
and make logs cleaner.

In the event that:
- a container is really stuck and needs cleanup using kill/rm
- and cleanup fails, and we fail to report it because of error
suppression (`2>/dev/null`)

.. we'd still get an error when launching ("container name already in use .."),
so it shouldn't be too hard to investigate.
2021-01-27 10:22:46 +02:00
Marcel Partap 183adec3d8 Merge remote-tracking branch 'origin/master' into synapse-workers 2021-01-23 15:04:11 +01:00
Slavi Pantaleev a56cb34850 Notify people if /matrix/postgres/data-auto-upgrade-backup exists 2021-01-23 14:14:45 +02:00
Slavi Pantaleev a2422c458a Notify of remaining matrix-postgres local data in a better way 2021-01-23 14:04:51 +02:00
Slavi Pantaleev f085362149 Fix some Postgres CLI scripts to target the correct database
Fixes a regression introduced in 95346f3117.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/814

Using `matrix_synapse_` variables in the `matrix-postgres` role is not
ideal, but.. this script belongs neither here, nor there.
We'll have it be like that for now.
2021-01-23 11:38:34 +02:00
Slavi Pantaleev f9968b6981 Fix matrix_postgres_connection_password length check 2021-01-22 21:22:58 +02:00
Slavi Pantaleev 88addd71fc Fix Postgres imports going to the matrix DB by default
Well, they still do go to that DB by default,
but our docs give a better command to users, which would do the right
thing.
2021-01-22 17:39:08 +02:00
Slavi Pantaleev bef0702fea Wait some more when starting Postgres during setup on ARM 2021-01-22 16:21:30 +02:00
Slavi Pantaleev f9c1d62435 Fix Postgres database (-alpine) failing to start on ARM32 2021-01-22 13:52:55 +02:00
Slavi Pantaleev 95346f3117 Reorganize Postgres access (breaking change)
In short, this makes Synapse a 2nd class citizen,
preparing for a future where it's just one-of-many homeserver software
options.

We also no longer have a default Postgres superuser password,
which improves security.

The changelog explains more as to why this was done
and how to proceed from here.
2021-01-22 13:26:12 +02:00
Slavi Pantaleev 1692a28fe4 Work around annoying Docker warning about undefined $HOME
> WARNING: Error loading config file: .dockercfg: $HOME is not defined

.. which appeared in Docker 20.10.
2021-01-15 00:23:01 +02:00
Slavi Pantaleev 05ca9357a8 Add .service suffix to systemd units list
We'll be adding `.timer` units later on, so it's good to be
more explicit.
2021-01-14 23:02:10 +02:00
Marcel Partap cd8100544b Merge remote-tracking branch 'origin/master' into synapse-workers
Sync with upstream
2021-01-08 20:58:50 +01:00
Slavi Pantaleev fb83eccf99 Relocate SQL template file 2021-01-03 00:58:31 +02:00
Slavi Pantaleev 86da489b9b Never fail when stopping systemd service during (SQLite -> Postgres) migration
We need to suppress systemd service-stopping requests in certain rare
cases like https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/771

That issue seems to describe a case, where a migration from mxisd to
ma1sd was happening (DB files had just been moved), and then we were
attemping to stop `matrix-ma1sd.service` so we could import that database into
Postgres. However, there's neither `matrix-mxisd.service`, nor
`matrix-ma1sd.service` after `migrate_mxisd.yml` had just run, so
stopping `matrix-ma1sd.service` was failing.
2020-12-29 10:31:20 +02:00
Slavi Pantaleev 21662af3be Archive database only after additional_psql_statements_list had executed 2020-12-23 14:12:11 +02:00
Stuart Mumford 019a4d7dcd Use role relative paths for things 2020-12-23 11:34:48 +00:00
Slavi Pantaleev ad1425eee4 Add pgloader self-building support (for ARM) 2020-12-23 09:08:54 +02:00
Slavi Pantaleev 8675dedbdb Add support for automatic (nedb -> Postgres) migration to matrix-appservice-slack 2020-12-22 19:56:52 +02:00
Slavi Pantaleev 715bdf2c64 Add support for automatic (nedb -> Postgres) migration to mx-appservice-irc 2020-12-22 19:32:43 +02:00
Slavi Pantaleev 1bd5c240e5 Add support for executing additional DB migration statements
In cases where pgloader is not enough and we need to do some additional
migration work after it, we can now use
`additional_psql_statements_list` and
`additional_psql_statements_db_name`.

This is to be used when migrating `matrix-registration`'s data at the
very least.
2020-12-15 23:18:29 +02:00
Slavi Pantaleev dd797ba6a7 Fix Postgres database importing/upgrading conflicts
We were running into conflicts, because having initialized
the roles (users) and databases, trying to import leads to
errors (role XXX already exists, etc.).

We were previously ignoring the Synapse database (`homeserver`)
when upgrading/importing, because that one gets created by default
whenever the container starts.

For our additional databases, it's a similar situation now.
It's not created by default as soon as Postgres starts with an empty
database, but rather we create it as part of running the playbook.

So we either need to skip those role/database creation statements
while upgrading/importing, or to avoid creating the additional database
and rely on the import for that. I've gone for the former, because
it's already similar to what we were doing and it's simpler
(it lets `setup_postgres.yml` be the same in all scenarios).
2020-12-14 22:28:20 +02:00
transcaffeine 13d8a9b39c
hint supported automatic migration nedb->postgres 2020-12-14 16:33:40 +01:00
Slavi Pantaleev af3ea67bba Add (SQLite + Postgres) support and automatic migration to matrix-ma1sd 2020-12-14 17:16:25 +02:00
Slavi Pantaleev c1431b28f0 Make use of matrix_postgres_db_migration_request.caller 2020-12-14 16:13:57 +02:00
Slavi Pantaleev dc7850e83c Fix wording and variable names a bit 2020-12-14 16:03:40 +02:00
Slavi Pantaleev bc376c2fb2 Add database migration utility to matrix-postgres role 2020-12-14 16:03:40 +02:00
Slavi Pantaleev 5dba0c038b Make --tags=import-generic-sqlite-db commands not pass a sensitive connection string around
Instead of passing the connection string, we can now pass a name of a
variable, which contains a connection string.

Both are supported for having extra flexibility.
2020-12-14 11:47:00 +02:00
Slavi Pantaleev f1e85f7112 Don't mention Postgres roles, just say users 2020-12-14 10:04:37 +02:00
Slavi Pantaleev cb969c6ca2 Add --tags=import-generic-sqlite-db (pgloader import)
This can be used by various bridges, etc., to import an SQLite
(or some other supported) database into Postgres.
2020-12-14 02:23:29 +02:00
Slavi Pantaleev c66c084027 Merge branch 'master' into postgres-per-default 2020-12-14 01:51:15 +02:00
Slavi Pantaleev 6e1dfb62f0 Rename some doc files and commands related to importing
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.

Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.

We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
2020-12-14 01:51:00 +02:00
Slavi Pantaleev 183d2a10db Ensure matrix-postgres.service is started before creating additional users/databases 2020-12-14 00:59:59 +02:00
Slavi Pantaleev 46a4034d3e Use "password" for additional Postgres databases, not "pass"
Being more explicit sounds better.
2020-12-14 00:43:03 +02:00
Slavi Pantaleev 3a037a5993 Ensure additional databases contain all the keys that we expect 2020-12-14 00:39:38 +02:00
Slavi Pantaleev da4cb2f639 Do not use the postgresql_user/postgresql_db modules
While these modules are really nice and helpful, we can't use them
for at least 2 reasons:

- for us, Postgres runs in a container on a private Docker network
(`--network=matrix`) without usually being exposed to the host.
These modules execute on the host so they won't be able to reach it.

- these modules require `psycopg2`, so we need to install it before
using it. This might or might not be its own can of worms.
2020-12-14 00:31:38 +02:00
Slavi Pantaleev bbc09d013b Do not execute additional databases creation code if not necessary
The tasks in `create_additional_databases.yml` will likely
ensure `matrix-postgres.service` is started, etc.

If no additional databases are defined, we'd rather not execute that
file and all these tasks that it may do in the future.
2020-12-13 23:46:05 +02:00
Slavi Pantaleev c765ceb270 Prevent weird loop error
> Invalid data passed to 'loop', it requires a list, got this instead: matrix_postgres_additional_databases. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.

Well, or working around it, as I've done in this commit (which seems
more sane than `wantlist=True` stuff).
2020-12-13 22:56:56 +02:00
Slavi Pantaleev e2952f16f7 Determine matrix-postgres IP address without relying on jq
To avoid needing to have `jq` installed on the machine, we could:
- try to run jq in a Docker container using some small image providing
that
- better yet, avoid `jq` altogether
2020-12-13 22:45:48 +02:00
Slavi Pantaleev f47e8a97e6 Make use of matrix_host_command_docker instead of hardcoding 2020-12-13 22:38:35 +02:00
Slavi Pantaleev 0641106370 Allow username of additional Postgres databases to be different
We'll most likely use one that matches the database name, but
it's better to have it configurable.
2020-12-13 22:37:04 +02:00
Slavi Pantaleev 527d5f57d5 Relocate Postgres additional database creation logic
Moving it above the "uninstalling" set of tasks is better.
Extracting it out to another file at the same time, for readability,
especially given that it will probably have to become more complex in
the future (potentially installing `jq`, etc.)
2020-12-13 22:37:04 +02:00
Slavi Pantaleev dac0d3a682 Add default matrix_postgres_additional_databases 2020-12-13 21:07:16 +02:00
Slavi Pantaleev 77a5c7cf3c Merge branch 'master' into postgres-per-default 2020-12-13 21:04:15 +02:00
Slavi Pantaleev 47613e5a27 Remove synapse-janitor support
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/746
2020-12-11 23:24:42 +02:00