Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/963
This also simplifies Prerequisites, which is great.
It'd be nice if we were doing these checks in some optional manner
and reporting them as helpful messages (using
`matrix_playbook_runtime_results`), but that's more complicated.
I'd rather drop these checks completely.
don't spawn an extra container
run pg_dumpall within matrix-postgres instead, ensures correct version
store under /matrix so a backup of the folder will contain a DB dump
use absolute paths just in case something in the ENV is messed up
In short, this makes Synapse a 2nd class citizen,
preparing for a future where it's just one-of-many homeserver software
options.
We also no longer have a default Postgres superuser password,
which improves security.
The changelog explains more as to why this was done
and how to proceed from here.
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/756
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/737
I feel like timers are somewhat more complicated and dirty (compared to
cronjobs), but they come with these benefits:
- log output goes to journald
- on newer systemd distros, you can see when the timer fired, when it
will fire, etc.
- we don't need to rely on cron (reducing our dependencies to just
systemd + Docker)
Cronjobs work well, but it's one more dependency that needs to be
installed. We were even asking people to install it manually
(in `docs/prerequisites.md`), which could have gone unnoticed.
Once in a while someone says "my SSL certificates didn't renew"
and it's likely because they forgot to install a cron daemon.
Switching to systemd timers means that installation is simpler
and more unified.
The different configurations are now all lower case, for consistent
naming.
`matrix_nginx_proxy_ssl_config` is now called
`matrix_nginx_proxy_ssl_preset`. The different options for "modern",
"intermediate" and "old" are stored in the main.yml file, instead of
being hardcoded in the configuration files. This will improve the
maintainability of the code.
The "custom" preset was removed. Now if one of the variables is set, it
will use it instead of the preset. This will allow to mix and match more
easily, for example using all the intermediate options but only
supporting TLSv1.2. This will also provide better backward
compatibility.
While it's kind of nice having it, it's also somewhat raw
and unnecessary.
Having a good default and not even mentioning it seems better
for most users.
People who need a more exposed bridge (rare) can use
override the default configuration using
`matrix_mautrix_signal_configuration_extension_yaml`.
We've had people who get the impression that well-known = bad,
DNS SRV = good, and who try to use DNS SRV for server delegation.
While it's true that DNS SRV can be superior for high-availability
scenarios, it's much harder to set up and comes with its own potential
downsides.
Using the well-known method is more straightforward and is enough
for almost all of us. Throwing people into the deep for no good
reason is not nice. Hopefully wording is better after this patch.
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.
Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.
We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
ma1sd requires the openid endpoints for certain functionality.
Example: 90b2b5301c/src/main/java/io/kamax/mxisd/auth/AccountManager.java (L67-L99)
If federation is disabled, we still need to expose these openid APIs on the
federation port.
Previously, we were doing similar magic for Dimension.
As per its documentation, when running unfederated, one is to enable
the openid listener as well. As per their recommendation, people
are advised to do enable it on the Client-Server API port
and use the `federationUrl` variable to override where the federation
port is (making federation requests go to the Client-Server API).
Because ma1sd always uses the federation port (unless you do some
DNS overwriting magic using its configuration -- which we'd rather not
do), it's better if we just default to putting the `openid` listener
where it belongs - on the federation port.
With this commit, we retain the "automatically enable openid APIs" thing
we've been doing for Dimension, but move it to the federation port instead.
We also now do the same thing when ma1sd is enabled.
This may be a bit premature, because the bridge didn't work for me
the last time I tried it (RC3).
Some bugs have been fixed to make our config compatible with v1.0.0
though, so it may work for some people (especially those starting
fresh).
I'm not for shipping potentially broken things, but given that we were
using `docker.io/halfshot/matrix-appservice-discord:latest` and that
points to v1.0.0 already (with no other tag we can use), our setup was
already broken in any case.
Now, at least it has some chance of running.
This supersedes/fixes-up this Pull Request:
https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/719
The Jitsi Web and JVB containers now (in build 5142) always
start by bulding their own default configuration
(`config.js` and `sip-communicator.properties`, respectively).
The fact that we were generating these files ourselves was no longer of use,
because our configuration was thrown away in favor of the one created
by the containers on startup.
With this commit, we're completely redoing things. We no longer
generate these configuration files. We try to pass the proper
environment variables, so that Jitsi services can generate the
configuration files themselves.
Besides that, we try to use the "custom configuration" mechanism
provided by Jitsi Web and Jitsi JVB (`custom-config.js` and
`custom-sip-communicator.properties`, respectively), so that
we and our users can inject additional configuration.
Some configuration options we had are gone now. Others are no longer
controllable via variables and need to be injected using
the `_config_extension` variables that we provide.
The validation logic that is part of the role should take care
to inform people about how to upgrade (if they're using some custom
configuration, which needs special care now). Most users should not
have to do anything special though.
This broke in 63a49bb2dc.
Proxying the OpenID Connect endpoints is now possible,
but needs to be enabled explicitly now.
Supersedes #702 (Github Pull Request).
This patch builds up on the idea from that Pull Request,
but does things in a cleaner way.
Interestingly, no one has reported this failure before #662 (Github
Issue).
It doesn't make sense to keep saying that we support such old Ansible
versions, when we're not even testing on anything close to those.
Time is also passing and such versions are getting more and more
ancient. It's time we bumped our requirements to something that is more
likely to work.
Change anonymized to more proper term as server vice this is not anonymized. Server name is the first parameter that is collected. And if server happens to be for individual use these statistics would be at personal level without any anonymising.
We recently had a report of the Postgres backup container's log file
growing the size of /var/lib/docker until it ran out of disk space.
Trying to prevent similar problems in the future.
If a Postgres dump contains ALTER TABLE ... OWNER_TO <username>
statements which set the owner to a username different from
'synapse' the post Postgres import task will fail complaining
about lack of role.
Changing the matrix_postgres_connection_username group var has no
effect. However, the ALTER TABLE statements (and accompanying comments)
can be rewritten to change the username to 'synapse', which permits the
import task to succeed.
From a sample of 1, having the owner set in this was causes no
discernable side effects on the homeserver.
One could also remove the two variables from the docs completely,
because they are set by the playbook automatically.
Error: javax.net.ssl.SSLPeerUnverifiedException: Certificate for
<matrix.<your-domain>> doesn't match any of the subject alternative
names: [<your-domain>]
Fixes#577 (Github Issue).
`matrix_container_images_self_build` was not really doing anything
anymore. It previously was influencing `matrix_*_self_build` variables,
but it's no longer the case since some time ago.
Individual `matrix_*_self_build` variables are still available.
People that would like to toggle self-building for a specific component
ought to use those.
These variables are also controlled automatically (via
`group_vars/matrix_servers`) depending on `matrix_architecture`.
In other words, self-building is being done automatically for
all components when they don't have a prebuilt image for the specified
architecture. Some components only support `amd64`, while others also
have images for other architectures.
This flag is necessary for Ansible to ask for the sudo password when using the non-root option as documented in the `hosts` file. Otherwise, Ansible errors out with `missing sudo password`.
We do this for 2 reasons:
- so we can control things which are not controllable using environment
variables (for example `stunServers` in jitsi/web, since we don't wish
to use the hardcoded Google STUN servers if our own Coturn is enabled)
- so playbook variable changes will properly rebuild the configuration.
When using Jitsi environment variables, the configuration is only built
once (the first time) and never rebuilt again. This is not the
consistent with the rest of the playbook and with how Ansible operates.
We're not perfect at it (yet), because we still let the Jitsi containers
generate some files on their own, but we are closer and it should be
good enough for most things.
Related to #415 (Github Pull Request).
I am new to synapse. Thanks so much for these playbooks!
I wasn't sure how you actual activate dimensions after setting these variables. Should you re run
ansible-playbook -i inventory/hosts setup.yml --tags=setup-all
ansible-playbook -i inventory/hosts setup.yml --tags=start
? If so perhaps you could tell the readers to do this somewhere in the help file
Thanks =]
This fixes an incorrect indentation in the database specification for
appservice-irc which caused matrix-appservice-irc to refuse to start
with the remarkably unhelpful error message:
```
ERROR:CLI Failed to run bridge.
```
This also updates doc links to the new matrixdotorg repo because the
tedomum repo contains out-of-date documentation.
Don't mention systemd-journald adjustment anymore, because
we've changed log levels to WARNING and Synapse is not chatty by default
anymore.
The "excessive log messages may get dropped on CentOS" issue no longer
applies to most users and we shouldn't bother them with it.
Continuation of #234 (Github Pull Request).
I had unintentionally updated the documentation for the feature,
saying the page is available at `https://matrix.DOMAIN/nginx_status`.
Looks like it wasn't the case, going against my expectations.
I'm correcting this with this patch.
The status page is being made available on both HTTP and HTTPS.
Serving over HTTP is likely necessary for services like
Longview
(https://www.linode.com/docs/platform/longview/longview-app-for-nginx/)
It's been pointed out that DEBUG logs could contain sensitive
information (access tokens, etc.), which makes them unsuitable
for sharing with others. INFO should be enough.
Also discussed previously in #213 (Github Pull Request).
shared-secret-auth and rest-auth logging is still at `INFO`
intentionally, as user login events seem more important to keep.
Those modules typically don't spam as much.
We do restart Synapse explicitly, but some other services
(bridges, matrix-corporal, ..) may not restart sometimes.
It's best to restart all services explicitly.
Somewhat related to #213 (Github Pull Request).
We've been moving in the opposite direction for quite a long time.
All services should just leave logging to systemd's journald.
Well, `config.yaml` has been playbook-managed for a long time.
It's now extended to match the default sample config of the Discord
bridge.
With this patch, we also make `registration.yaml` playbook-managed,
which leads us to consistency with all other bridges.
Along with that, we introduce `./config` and `./data` separation,
like we do for the other bridges.
With most people on Synapse v0.99+ and Synapse v1.0 now available,
we should no longer try to be backward compatible with Synapse 0.34,
because this just complicates the instructions for no good reason.
Changes to the original are:
- it tells people to stop and disable services, so that:
- services won't be running while you are copying files
- services won't accidentally start again later
- it does the file-copying in 1 step
- it does copying before running `--tags=setup-all`, so that existing files (SSL certificates, etc.) can be reused. Otherwise, the playbook starts from a blank slate, retrieves them anew, generates new signing keys anew, etc. Only to have those replaced by your own old backup later.
- it mentions DNS changes
- combines `--tags=setup-all,start` into a single step, thanks to the files being already copied
This doesn't replace all usage of `-v`, but it's a start.
People sometimes troubleshoot by deleting files (especially bridge
config files). Restarting Synapse with a missing registration.yaml file
for a given bridge, causes the `-v
/something/registration.yaml:/something/registration.yaml:ro` option
to force-create `/something/registration.yaml` as a directory.
When a path that's provided to the `-v` option is missing, Docker
auto-creates that path as a directory.
This causes more breakage and confusion later on.
We'd rather fail, instead of magically creating directories.
Using `--mount`, instead of `-v` is the solution to this.
From Docker's documentation:
> When you use --mount with type=bind, the host-path must refer to an existing path on the host.
> The path will not be created for you and the service will fail with an error if the path does not exist.
The bit about the matrix-make-user-admin script was messed up (it wasn't actually a code block so the "<username>" was hidden). For me at least it seems like the ``` syntax is much harder to accidentally mess up.
Fixes#129 (Github Issue).
Unfortunately, we rely on `service_facts`, which is only available
in Ansible >= 2.5.
There's little reason to stick to an old version such as Ansible 2.4:
- some time has passed since we've raised version requirements - it's
time to move into the future (a little bit)
- we've recently (in 82b4640072) improved the way one can run
Ansible in a Docker container
From now on, Ansible >= 2.5 is required.
Inspired by #128 (Github Issue), we've created a new Docker image
to replace https://hub.docker.com/r/qmxme/ansible
Adding dnspython or dig to `qmxme/ansible` doesn't seem like a good
idea (that might be accepted by them), given that it's specific to our
use case. That's why we'll be maintaining our own image from now on.
When using Let's Encrypt SSL certificates, a cronjob is set up to
automatically renew them. Though it does require a `cron`-compatible
program on the server.
This fixes the error that is caused by the `/etc/cron.d` directory
not existing and the `ansible-cron` module trying to write out a
file there -- without checking if the directory exists first.
It didn't mention `matrix_appservice_discord_client_id` and
`matrix_appservice_discord_bot_token`, which makes it hard for
beginners.
Related to #105 (Github Pull Request).
`matrix_synapse_no_tls` is now implicit, so we've gotten rid of it.
The `homeserver.yaml.j2` template has been synchronized with the
configuration generated by Synapse v0.99.1 (some new options
are present, etc.)
We had something like that on the Server Delegation how-to page,
but it's better if we have it on the SSL certificates page.
Relocated there and improved linking.
Fixes#94 (Github Issue)
This reverts commit 0dac5ea508.
Relying on pyOpenSSL is the Ansible way of doing things, but is
impractical and annoying for users.
`openssl` is easily available on most servers, even by default.
We'd better use that.
This makes all containers (except mautrix-telegram and
mautrix-whatsapp), start as a non-root user.
We do this, because we don't trust some of the images.
In any case, we'd rather not trust ALL images and avoid giving
`root` access at all. We can't be sure they would drop privileges
or what they might do before they do it.
Because Postfix doesn't support running as non-root,
it had to be replaced by an Exim mail server.
The matrix-nginx-proxy nginx container image is patched up
(by replacing its main configuration) so that it can work as non-root.
It seems like there's no other good image that we can use and that is up-to-date
(https://hub.docker.com/r/nginxinc/nginx-unprivileged is outdated).
Likewise for riot-web (https://hub.docker.com/r/bubuntux/riot-web/),
we patch it up ourselves when starting (replacing the main nginx
configuration).
Ideally, it would be fixed upstream so we can simplify.
With this change, the following roles are now only dependent
on the minimal `matrix-base` role:
- `matrix-corporal`
- `matrix-coturn`
- `matrix-mailer`
- `matrix-mxisd`
- `matrix-postgres`
- `matrix-riot-web`
- `matrix-synapse`
The `matrix-nginx-proxy` role still does too much and remains
dependent on the others.
Wiring up the various (now-independent) roles happens
via a glue variables file (`group_vars/matrix-servers`).
It's triggered for all hosts in the `matrix-servers` group.
According to Ansible's rules of priority, we have the following
chain of inclusion/overriding now:
- role defaults (mostly empty or good for independent usage)
- playbook glue variables (`group_vars/matrix-servers`)
- inventory host variables (`inventory/host_vars/matrix.<your-domain>`)
All roles default to enabling their main component
(e.g. `matrix_mxisd_enabled: true`, `matrix_riot_web_enabled: true`).
Reasoning: if a role is included in a playbook (especially separately,
in another playbook), it should "work" by default.
Our playbook disables some of those if they are not generally useful
(e.g. `matrix_corporal_enabled: false`).
As suggested in #63 (Github issue), splitting the
playbook's logic into multiple roles will be beneficial for
maintainability.
This patch realizes this split. Still, some components
affect others, so the roles are not really independent of one
another. For example:
- disabling mxisd (`matrix_mxisd_enabled: false`), causes Synapse
and riot-web to reconfigure themselves with other (public)
Identity servers.
- enabling matrix-corporal (`matrix_corporal_enabled: true`) affects
how reverse-proxying (by `matrix-nginx-proxy`) is done, in order to
put matrix-corporal's gateway server in front of Synapse
We may be able to move away from such dependencies in the future,
at the expense of a more complicated manual configuration, but
it's probably not worth sacrificing the convenience we have now.
As part of this work, the way we do "start components" has been
redone now to use a loop, as suggested in #65 (Github issue).
This should make restarting faster and more reliable.
This change is provoked by a few different things:
- #54 (Github Pull Request), which rightfully says that we need a
way to support ALL mxisd configuration options easily
- the upcoming mxisd 1.3.0 release, which drops support for
property-style configuration (dot-notation), forcing us to
redo the way we generate the configuration file
With this, mxisd is much more easily configurable now
and much more easily maintaneable by us in the future
(no need to introduce additional playbook variables and logic).
After having multiple people report issues with retrieving
SSL certificates, we've finally discovered the culprit to be
Ansible 2.5.1 (default and latest version on Ubuntu 18.04 LTS).
As silly as it is, certain distributions ("LTS" even) are 13 bugfix
versions of Ansible behind.
From now on, we try to auto-detect buggy Ansible versions and tell the
user. We also provide some tips for how to upgrade Ansible or
run it from inside a Docker container.
My testing shows that Ansible 2.4.0 and 2.4.6 are OK.
All other intermediate 2.4.x versions haven't been tested, but we
trust they're OK too.
From the 2.5.x releases, only 2.5.0 and 2.5.1 seem to be affected.
Ansible 2.5.2 corrects the problem with `include_tasks` + `with_items`.
This is a simplification and a way to make it consistent with
how we do Postgres imports (see 6d89319822), using
files coming from the server, not from the local machine.
By encouraging people NOT to use local files,
we potentially avoid problems such as #34 (Github issue),
where people would download `media_store` to their Mac's filesystem
and case-sensitivity issues will actually corrupt it.
By not encouraging local files usage, it's less likely that
people would copy (huge) directories to their local machine like that.
This is a simplification and a way to make it consistent with
how we do Postgres imports (see 6d89319822), using
files coming from the server, not from the local machine.
Adds support for managing certificates manually and for
having the playbook generate self-signed certificates for you.
With this, Let's Encrypt usage is no longer required.
Fixes Github issue #50.
Relay hostnames that have MX records are looked up by postfix
and the MX record's payload is used instead.
This special behavior may be undesirable, so we make sure to
point it out.
This is provoked by Github issue #46.
No client had made use of the well-known mechanism
so far, so the set up performed by this playbook was not tested
and turned out to be a little deficient.
Even though /.well-known/matrix/client is usually requested with a
simple request (no preflight), it's still considered cross-origin
and [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)
applies. Thus, the file always needs to be served with the appropriate
`Access-Control-Allow-Origin` header.
Github issue #46 attempts to fix it at the "reverse-proxying" layer,
which may work, but would need to be done for every server.
It's better if it's done "upstream", so that all reverse-proxy
configurations can benefit.
We've had some people get confused into installing
Matrix Corporal and having pain with that.
With this documentation change, we try to make it clearer
that it's an advanced feature not to be touched unless
you know what you're doing.
On a similar note, we also make sure other things are properly
labeled as "(optional)" and/or "(advanced)".
teutat3s
