Commit graph

111 commits

Author SHA1 Message Date
Slavi Pantaleev ccabc82d4c Use more fully-qualified container images
This is both for consistency with 93cc71cb69976c
and for making things more obvious.
2020-11-14 23:01:11 +02:00
Slavi Pantaleev 6a72e3fa54 Try to make importing SQLite from older Synapse version work
If the SQLite database was from an older version of Synapse, it appears
that Synapse would try to run migrations on it first, before importing.
This was failing, because the file wasn't writable.

Hopefully, this fixes the problem.
2020-10-07 08:54:46 +03:00
Dan Arnfield 3a3383fada Add support for postgres 13 2020-09-30 16:50:59 -05:00
Slavi Pantaleev a456e3a9e7 Surface certain messages at the end of playbook execution
Fixes #106 (Github Issue).
2020-09-01 13:12:35 +03:00
Slavi Pantaleev e3dca2f66f Try to avoid Docker logs growing too much for one-off containers
We recently had a report of the Postgres backup container's log file
growing the size of /var/lib/docker until it ran out of disk space.

Trying to prevent similar problems in the future.
2020-09-01 09:03:48 +03:00
Dan Arnfield 20eea648a5 Update postgres versions (12.3 -> 12.4, etc) 2020-08-16 14:41:40 -05:00
Chris van Dijk 6e3b877dc2 Remove hardcoded command paths in playbook shell usage 2020-05-27 23:14:56 +02:00
Chris van Dijk 6334f6c1ea Remove hardcoded command paths in systemd unit files
Depending on the distro, common commands like sleep and chown may either
be located in /bin or /usr/bin.

Systemd added path lookup to ExecStart in v239, allowing only the
command name to be put in unit files and not the full path as
historically required. At least Ubuntu 18.04 LTS is however still on
v237 so we should maintain portability for a while longer.
2020-05-27 23:14:54 +02:00
Ugurtan 5ace3f4a1c
fix for importing sqllite database
the current version fails the import, because the volume for the media is missing. It still fails if you have the optional shared secret password provider is enabled, so that might need another mount. Commenting out the password provider in the hoimeserver.yaml during the run works as well.
2020-05-27 18:13:36 +02:00
Dan Arnfield ee3944bcdb Update postgres (12.2 -> 12.3, etc) 2020-05-21 11:40:40 -05:00
Slavi Pantaleev 8fb3ce6f6d Upgrade Synapse (v1.12.4 -> v1.13.0) 2020-05-19 21:35:32 +03:00
Chris van Dijk 7585bcc4ac Allow the matrix user username and groupname to be configured separately
No migration steps should be required.
2020-05-01 19:59:32 +02:00
Slavi Pantaleev 9991d545dd Add note about synapse-janitor and require explicit confirmation for usage
Fixes #465 (Github Issue).

Related to https://github.com/xwiki-labs/synapse_scripts/pull/12.
2020-04-21 21:48:06 +03:00
Chris van Dijk 88c1fbf6aa Fix hardcoded values in postgres script matrix-change-user-admin-status 2020-04-17 00:54:38 +02:00
Chris van Dijk d83236ea0e Fix inconsistent whitespace 2020-04-17 00:53:26 +02:00
Mickaël Cornière 82989b662a Fix bad network and bad path to postgre env 2020-03-29 23:49:09 +02:00
mooomooo eebc6e13f8 Made directory variables for /etc/systemd/system , /etc/cron.d , /usr/local/bin 2020-03-24 11:27:58 -07:00
Dan Arnfield e36de7e627 Update postgres (12.1 -> 12.2, etc) 2020-03-18 06:50:51 -05:00
David Gnedt c55682d099 Update synapse-janitor to support current synapse database schema 2020-03-06 17:48:16 +01:00
Slavi Pantaleev 704d2ff861
Merge pull request #351 from prasket/matrix-change-user-admin-status
Matrix change user admin status
2020-01-23 10:35:28 +02:00
prasket 83e7a8d9e6 corrected hardcoded matrix domain and added code to remove old matrix-make-user-admin script per suggestion. 2020-01-21 18:29:17 -08:00
Slavi Pantaleev 9d3d538a2d Use C collation for Postgres to appease Synapse 2020-01-21 00:13:47 +02:00
prasket e99497bb60 created new file with change admin options and removed make admin file. Updated name references as well throughout the project. 2020-01-19 18:39:56 -08:00
Dan Arnfield 4a60f385d1 Update postgres versions (12.0 -> 12.1, etc) 2019-11-21 09:38:37 -06:00
Slavi Pantaleev e32aaacaa7 Make gzipped SQL dumps by default during --upgrade-postgres 2019-10-05 11:42:08 +03:00
Slavi Pantaleev 29526e7bb1 Add support for backing up / importing all Postgres databases/users 2019-10-05 11:42:08 +03:00
Slavi Pantaleev 9c438a3870 Add support for Postgres v12 2019-10-04 08:51:36 +03:00
Slavi Pantaleev 1784bde226 Ensure Postgres data ownership is correct
Fixes #257 (Github Issue)
2019-09-02 10:01:31 +03:00
Dan Arnfield dc11704c11 Bump postgres versions (11.5, 10.10, 9.6.15) 2019-08-09 06:03:26 -05:00
Slavi Pantaleev 0ca21d80d7 Add Synapse Maintenance docs and synapse-janitor integration 2019-07-08 09:38:36 +03:00
Slavi Pantaleev 631a14bf0c Rename run control variables for consistency 2019-07-08 09:38:36 +03:00
Slavi Pantaleev e805044b80 Delete scripts when uninstalling Postgres 2019-07-08 09:38:36 +03:00
Dan Arnfield 1eaa7b6967 Update postgres versions to latest 2019-06-24 13:11:23 -05:00
Slavi Pantaleev 7d3adc4512 Automatically force-pull :latest images
We do use some `:latest` images by default for the following services:
- matrix-dimension
- Goofys (in the matrix-synapse role)
- matrix-bridge-appservice-irc
- matrix-bridge-appservice-discord
- matrix-bridge-mautrix-facebook
- matrix-bridge-mautrix-whatsapp

It's terribly unfortunate that those software projects don't release
anything other than `:latest`, but that's how it is for now.

Updating that software requires that users manually do `docker pull`
on the server. The playbook didn't force-repull images that it already
had.

With this patch, it starts doing so. Any image tagged `:latest` will be
force re-pulled by the playbook every time it's executed.

It should be noted that even though we ask the `docker_image` module to
force-pull, it only reports "changed" when it actually pulls something
new. This is nice, because it lets people know exactly when something
gets updated, as opposed to giving the indication that it's always
updating the images (even though it isn't).
2019-06-10 14:30:28 +03:00
Slavi Pantaleev 4f87f7e43e
Explain matrix_postgres_container_postgres_bind_port a little more
Previously, it only mentioned exposing for psql-usage purposes.

Realistically, it can be used for much more. Especially given that
psql can be easily accessed via our matrix-postgres-cli script,
without exposing the container port.
2019-06-10 08:24:37 +03:00
Aaron Raimist 6fce809d10
Add config option to be able to access database outside of container 2019-06-09 20:35:35 -05:00
Slavi Pantaleev 3567d9adba Fix typo 2019-06-07 16:07:01 +03:00
Dan Arnfield 9c23d877fe Fix docker_image option for ansible < 2.8 2019-05-22 05:43:33 -05:00
Dan Arnfield db15791819 Add source option to docker_image to fix deprecation warning 2019-05-21 10:29:12 -05:00
Dan Arnfield 3982f114af Fix CONDITIONAL_BARE_VARS deprecation warning in ansible 2.8 2019-05-21 10:25:59 -05:00
Slavi Pantaleev ae7c8d1524 Use SyslogIdentifier to improve logging
Reasoning is the same as for matrix-org/synapse#5023.

For us, the journal used to contain `docker` for all services, which
is not very helpful when looking at them all together (`journalctl -f`).
2019-05-16 09:43:46 +09:00
Dan Arnfield 6163ba5bb1 Bump postgres versions 2019-05-10 08:02:32 -05:00
Hugues De Keyzer c451025134 Fix indentation in templates
Use Jinja2 lstrip_blocks option in templates to ensure consistent
indentation in generated files.
2019-05-07 21:23:35 +02:00
Slavi Pantaleev e0b7b4dc61
Merge pull request #159 from TheLastProject/feature/docker_add_hosts
Add the possibility to pass extra flags to the docker container
2019-05-05 10:22:59 +03:00
Sylvia van Os 75b1528d13 Add the possibility to pass extra flags to the docker container 2019-04-30 16:35:18 +02:00
Lyubomir Popov 134faa3139 Add the ability to update user passwords with ansible (when using the matrix-postgres container). 2019-04-30 16:30:26 +03:00
Slavi Pantaleev 6f6dff3e2b Update some Docker images 2019-03-03 12:27:43 +02:00
Slavi Pantaleev 45618679f5 Reload systemd services when they get updated
Fixes #69 (Github Issue)
2019-03-03 11:55:15 +02:00
Slavi Pantaleev a43bcd81fe Rename some variables 2019-02-28 11:51:09 +02:00
Slavi Pantaleev 433780384e Do not use docker_container module
Using `docker_container` with a `cap_drop` argument requires
Ansible >=2.7.

We want to support older versions too (2.4), so we either need to
stop invoking it with `cap_drop` (insecure), or just stop using
the module altogether.

Since it was suffering from other bugs too (not deleting containers
on failure), we've decided to remove `docker_container` usage completely.
2019-02-25 10:42:27 +02:00
Slavi Pantaleev 0be7b25c64 Make (most) containers run with a read-only filesystem 2019-01-29 18:52:02 +02:00
Slavi Pantaleev 9438402f61 Drop capabilities in a few more places
Continuation of 316d653d3e
2019-01-28 11:43:32 +02:00
Slavi Pantaleev 316d653d3e Drop capabilities in containers
We run containers as a non-root user (no effective capabilities).

Still, if a setuid binary is available in a container image, it could
potentially be used to give the user the default capabilities that the
container was started with. For Docker, the default set currently is:
- "CAP_CHOWN"
- "CAP_DAC_OVERRIDE"
- "CAP_FSETID"
- "CAP_FOWNER"
- "CAP_MKNOD"
- "CAP_NET_RAW"
- "CAP_SETGID"
- "CAP_SETUID"
- "CAP_SETFCAP"
- "CAP_SETPCAP"
- "CAP_NET_BIND_SERVICE"
- "CAP_SYS_CHROOT"
- "CAP_KILL"
- "CAP_AUDIT_WRITE"

We'd rather prevent such a potential escalation by dropping ALL
capabilities.

The problem is nicely explained here: https://github.com/projectatomic/atomic-site/issues/203
2019-01-28 11:22:54 +02:00
Slavi Pantaleev 299a8c4c7c Make (most) containers start as non-root
This makes all containers (except mautrix-telegram and
mautrix-whatsapp), start as a non-root user.

We do this, because we don't trust some of the images.
In any case, we'd rather not trust ALL images and avoid giving
`root` access at all. We can't be sure they would drop privileges
or what they might do before they do it.

Because Postfix doesn't support running as non-root,
it had to be replaced by an Exim mail server.

The matrix-nginx-proxy nginx container image is patched up
(by replacing its main configuration) so that it can work as non-root.
It seems like there's no other good image that we can use and that is up-to-date
(https://hub.docker.com/r/nginxinc/nginx-unprivileged is outdated).

Likewise for riot-web (https://hub.docker.com/r/bubuntux/riot-web/),
we patch it up ourselves when starting (replacing the main nginx
configuration).
Ideally, it would be fixed upstream so we can simplify.
2019-01-27 20:25:13 +02:00
Slavi Pantaleev 1a80058a2a Indent (non-YAML) using tabs
Fixes #83 (Github issue)
2019-01-26 09:37:29 +02:00
Slavi Pantaleev a4e7ad5566 Use async Ansible task for importing Postgres
A long-running import task may hit the SSH timeout value
and die. Using async is supposed to improve reliability
in such scenarios.
2019-01-21 08:34:49 +02:00
Slavi Pantaleev 0392822aa7 Show Postgres import command and mention manual importing 2019-01-21 08:33:10 +02:00
Slavi Pantaleev 8d186e5194 Fix Postgres import when Postgres had never started
If this is a brand new server and Postgres had never started,
detecting it before we even start it is not possible.

This moves the logic, so that it happens later on, when Postgres
would have had the chance to start and possibly initialize
a new empty database.

Fixes #82 (Github issue)
2019-01-21 07:32:19 +02:00
Slavi Pantaleev c10182e5a6 Make roles more independent of one another
With this change, the following roles are now only dependent
on the minimal `matrix-base` role:
- `matrix-corporal`
- `matrix-coturn`
- `matrix-mailer`
- `matrix-mxisd`
- `matrix-postgres`
- `matrix-riot-web`
- `matrix-synapse`

The `matrix-nginx-proxy` role still does too much and remains
dependent on the others.

Wiring up the various (now-independent) roles happens
via a glue variables file (`group_vars/matrix-servers`).
It's triggered for all hosts in the `matrix-servers` group.

According to Ansible's rules of priority, we have the following
chain of inclusion/overriding now:
- role defaults (mostly empty or good for independent usage)
- playbook glue variables (`group_vars/matrix-servers`)
- inventory host variables (`inventory/host_vars/matrix.<your-domain>`)

All roles default to enabling their main component
(e.g. `matrix_mxisd_enabled: true`, `matrix_riot_web_enabled: true`).
Reasoning: if a role is included in a playbook (especially separately,
in another playbook), it should "work" by default.

Our playbook disables some of those if they are not generally useful
(e.g. `matrix_corporal_enabled: false`).
2019-01-16 18:05:48 +02:00
Slavi Pantaleev b80d44afaa Stop Postgres before finding files to move over 2019-01-12 18:16:08 +02:00
Slavi Pantaleev 51312b8250 Split playbook into multiple roles
As suggested in #63 (Github issue), splitting the
playbook's logic into multiple roles will be beneficial for
maintainability.

This patch realizes this split. Still, some components
affect others, so the roles are not really independent of one
another. For example:
- disabling mxisd (`matrix_mxisd_enabled: false`), causes Synapse
and riot-web to reconfigure themselves with other (public)
Identity servers.

- enabling matrix-corporal (`matrix_corporal_enabled: true`) affects
how reverse-proxying (by `matrix-nginx-proxy`) is done, in order to
put matrix-corporal's gateway server in front of Synapse

We may be able to move away from such dependencies in the future,
at the expense of a more complicated manual configuration, but
it's probably not worth sacrificing the convenience we have now.

As part of this work, the way we do "start components" has been
redone now to use a loop, as suggested in #65 (Github issue).
This should make restarting faster and more reliable.
2019-01-12 18:01:10 +02:00