Commit graph

22 commits

Author SHA1 Message Date
Slavi Pantaleev 226c550ffa Add support for stream writer Synapse workers
As stream writer workers are also powered by the `generic_worker`
Synapse app, this necessitated that we provide means for distinguishing
between them and regular `generic_workers`.

I've also taken the time to optimize nginx configuration generation
(more Jinja2 macro usage, less duplication).

Worker names have also changed.
Workers are now named sequentially like this:
- `matrix-synapse-worker-0-generic`
- `matrix-synapse-worker-1-stream-writer-typing`
- `matrix-synapse-worker-2-pusher`

instead of `matrix-synapse-worker_generic_worker-18111` (indexed with a
port number).

People who modify `matrix_synapse_workers_enabled_list` directly will
need to adjust their configuration.
2022-09-15 08:10:04 +03:00
Aine c71fea70d3
matrix-prometheus feedback 2022-06-26 12:01:57 +03:00
Aine 1542e8bca0
Update roles/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
2022-06-26 06:59:46 +00:00
Aine 574f57c82c
expose prometheus process args 2022-06-26 08:41:22 +03:00
Slavi Pantaleev 0364c6c634 Suppress old container cleanup (kill/rm) failures
People often report and ask about these "failures".
More-so previously, when the `docker kill/rm` output was collected,
but it still happens now when people do `systemctl status
matrix-something` and notice that it says "FAILURE".

Suppressing to avoid further time being wasted on saying "this is
expected".
2022-04-11 09:05:33 +03:00
Slavi Pantaleev 86c36523df Replace ExecStopPost with ExecStop
Reverts b1b4ba501f, 90c9801c56, a3c84f78ca, ..

I haven't really traced it (yet), but on some servers, I'm observing
`ansible-playbook ... --tags=start` completing very slowly, waiting
to stop services. I can't reproduce this on all Matrix servers I manage.
I suspect that either the systemd version is to blame or that some
specific service is not responding well to some `docker kill/rm` command.

`ExecStop` seems to work great in all cases and it's what we've been
using for a very long time, so I'm reverting to that.
2022-02-05 12:13:36 +02:00
HarHarLinks 321ed9b609 Merge remote-tracking branch 'origin/master' into hookshot 2022-01-14 19:26:31 +01:00
HarHarLinks 87871040df add hookshot metrics to internal prometheus 2022-01-11 00:56:51 +01:00
Slavi Pantaleev b1b4ba501f Replace ExecStop with ExecStopPost
ExecStopPost should allow us to clean up (docker kill + docker rm)
even if the ExecStart (docker run ..) command failed, and not just after
a graceful service stop was initiated.

Source: https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStopPost=
2022-01-04 17:27:25 +02:00
Stuart Mumford 2aa457efcc Use a prom variable and not a synapse role variable 2021-07-02 15:41:36 +00:00
Stuart Mumford 09ee5ce52e we index from 0 apparently 2021-06-30 21:32:19 +00:00
Stuart Mumford 3d063f6ace make them show as jobs in grafana 2021-06-30 21:30:18 +00:00
Stuart Mumford 7b52e6ad5e Add worker metrics to prometheus exporter 2021-06-30 20:52:49 +00:00
Raymond Coetzee 4e2780ff88 Add support for a prometheus postgres exporter
This commit introduces a new role that downloads and installs the
prometheus community postgres exporter  https://github.com/prometheus-community/postgres_exporter.
A new credential is added to matrix_postgres_additional_databases that
allows the exporter access to the database to gather statistics.
A new dashboard was added to the grafana role, with some refactoring
to enable the dashboard only if the new role is enabled.
I've included some basic instructions for how to enable the role in
the Docs section.

In terms of testing, I've tested enabling the role, and disabling
it to make sure it cleans up the container and systemd role.
2021-05-27 20:13:29 +01:00
Slavi Pantaleev c8ab200cb1 Break dependency between matrix-prometheus and (matrix-prometheus-node-exporter, matrix-synapse) 2021-02-12 11:59:24 +02:00
Slavi Pantaleev 18e31526a8 Rename some variables 2021-02-12 11:59:24 +02:00
Slavi Pantaleev 85a260daaf Make --tags=setup-prometheus not break, relying on matrix-base facts 2021-02-12 11:59:24 +02:00
Slavi Pantaleev df3dd1c824 Use --read-only FS for metrics-related containers
It seems like it doesn't cause any issues for any of these services.
2021-02-12 11:59:24 +02:00
Peetz0r 144a5e6198 Register docker network info and use it for prometheus-node-exporter
Using the hardcoded IP did break while I was
messing with IPv6 stuff on the other branch
2021-02-10 22:54:42 +01:00
Peetz0r 76d7e84be5 Make prometheus-node-exporter a bit more capable
By running it in a more privileged container with access to the host network stack and such
2021-02-10 22:54:14 +01:00
Peetz0r e525970b39 Prometheus Node Exporter
Basic system stats, to show stuff the synapse metrics
can't show such as resource usage by bridges, etc

Seems to work fine as well.

This too has only been tested on debian amd64 so far
2021-02-10 22:54:14 +01:00
Peetz0r 13ef9e85cf Prometheus
Initial attempt. Seems to work fine.

Only tested on debian amd64 so far
2021-02-10 22:54:14 +01:00