Prometheus and Grafana on stats.<domain> #838

Peetz0r · 2021-01-29T11:36:03Z

The Node Exporter is missing some data because it isn't fully effective inside the container, but most of it works.

Tested in debian 10 amd64 (in a vm) and on my RockPro64 running debian testing/bullseye/11.

Demo: https://stats.glitch.im/

pushytoxin · 2021-01-29T13:41:43Z

If this adds a new subdomain, docs/configuring-dns.md should be updated as well

pushytoxin · 2021-01-29T13:55:57Z

You can also extend the list at docs/self-building.md since you added self-building

aaronraimist · 2021-01-30T06:55:01Z

Maybe there should be two tables for the DNS settings in docs/configuring-dns.md. One that has the components that are enabled by default and one for things that aren't. Right now it is a bit confusing because at the top of that table it says "General outline of DNS settings you need to do" even though half of the table is optional, off by default stuff.

Peetz0r · 2021-01-30T07:10:17Z

If this adds a new subdomain, docs/configuring-dns.md should be updated as well

Done!

Maybe there should be two tables for the DNS settings in docs/configuring-dns.md. One that has the components that are enabled by default and one for things that aren't. Right now it is a bit confusing because at the top of that table it says "General outline of DNS settings you need to do" even though half of the table is optional, off by default stuff.

Done!

You can also extend the list at docs/self-building.md since you added self-building

Enabling it in the first place was actually a mistake. I am new to Docker and quite rusty on Ansible, so I started just copypasting stuff from other services. Only to find out in the process how it actually works.

I found out that all images I used support amd64, arm32 and arm64. I also find out that self-building didn't even actually work. So I disabled it.

I also tested it on my (arm64) rockpro64 and everything works just as well as it did on my amd64 VM and deployed it to my main personal homeserver :)

Demo at https://stats.glitch.im/ (should be online 24/7, feel free poke me if it breaks or anything)

Peetz0r · 2021-01-30T08:49:33Z

I rebased and edited the commits enabling and then disabling self-building so it's as if it never happened :)

spantaleev · 2021-01-31T08:21:52Z

What do you think about renaming the domain name for Grafana from stats.DOMAIN to stats.matrix.DOMAIN?

While jitsi.DOMAIN and dimension.DOMAIN are also generic, it's probably unlikely that one is hosting something else called "Dimension" for the domain, or that one is hosting a 2nd Jitsi server, so naming these jitsi.matrix.DOMAIN and dimension.matrix.DOMAIN is unnecessary.

stats. on the other hand sounds quite generic. Especially right now, it's probably difficult to use the same Prometheus/Grafana installation for non-Matrix stuff as well, so I wonder if it should be so "top-level".

On the other hand:

it's easier that way
it may very well be possible to make use of this installation for non-Matrix things
people who have a problem with stats.DOMAIN can override it

Peetz0r · 2021-01-31T17:40:34Z

Those are all good improvements I think.

Like I said, most of this is new to me, I am learning as I go :)

pushytoxin · 2021-02-01T16:46:32Z

I'm getting confused by some of the variables, I think there needs to be a cleanup for better defaults.

Is there any reason for the synapse metrics to be presented through web proxy if Grafana and Prometheus are enabled?
Why would someone run Grafana without Prometheus on the homeserver?
Why would someone run Prometheus without Grafana on the homeserver?

There might be users who collect the metrics on an external time series DB. They probably don't need Grafana nor Prometheus in the homeserver.

There should be a separate variable for the stats.DOMAIN Grafana dashboard, which turns on metrics internally on port 9100, and Prometheus+Grafana
There should be a separate variable for exposing /_synapse/metrics in the web proxy

Then for additional fine tuning, the admin user or the node-exporter can be configured. The admin user should also be disabled if possible unless explicitly set up by the user.

spantaleev · 2021-02-03T08:21:19Z

Yeah, this is opening pandora's box in a way..

One could also say that we need to hook Prometheus to the SMTP server, so people can set up Alerts, etc.

Then there's the people who will use a separate Prometheus installation and separate Grafana. They'd want to just expose /_synapse/metrics somehow. And they'd want to do it privately (behind HTTP Basic Auth, etc.). We already have support for this, but it's not documented on this page (it might be on some other docs page):

matrix_synapse_metrics_enabled: true

matrix_nginx_proxy_proxy_synapse_metrics: true
matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled: true
# The HTTP Basic Auth user is always `prometheus`
matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key: "password"

Then, #456 is coming along and each worker exposes its own metrics. I'm not sure how those should be handled.

I think we shouldn't get overly obsessed with this and just try to launch something. Even now, it seems good enough.

Documentation around edge cases can be improved. People who want SMTP support for alerts or want to monitor workers, etc., can also jump in and contribute in the future.

Initial attempt. Seems to work fine. Only tested on debian amd64 so far

Basic system stats, to show stuff the synapse metrics can't show such as resource usage by bridges, etc Seems to work fine as well. This too has only been tested on debian amd64 so far

Also includes the dashboards for Synapse and for Node Exporter. Again has only been tested on debian amd64 so far, but the grafana docker image is available for arm64 and arm32. Nice.

By running it in a more privileged container with access to the host network stack and such

Using the hardcoded IP did break while I was messing with IPv6 stuff on the other branch

Peetz0r · 2021-02-10T22:14:40Z

Rebased on current master, and updated grafana and prometheus-node-exporter to latest versions.

Yes, we might need more documentation to expose how flexible this can be. But for the common usecase (just exploring performance metrics of your new shiny homeserver) it works just fine.

The quotes around "host" for both `--pid` and `--net` were causing trouble for me: > docker: --pid: invalid PID mode. and: > docker: Error response from daemon: network "host" not found. I've also changed the `-v` call to `--mount` for consistency with the rest of the playbook.

We mainly switch the anonymous metrics viewing variable to false, along with other wording changes.

It seems like it doesn't cause any issues for any of these services.

…e-exporter, matrix-synapse)

.. and other documentation improvements.

spantaleev · 2021-02-12T12:11:30Z

You rebasing made my changes disappear, but I rebased as well ;)

Our poll in the support room indicated that people prefer stats.DOMAIN over stats.matrix.DOMAIN, so we'll go with that as the default.

I've made a few minor changes and merged this! I've gotten rid of the explicit matrix_synapse_metrics_enabled: true, making the configuration a little simpler for most people. I've also documented how one can use an external Prometheus/Grafana.

Something else we might wish to do in the future is to adjust matrix_grafana_dashboard_download_urls dynamically based on the enabled services. People who don't use Synapse (when Dendrite comes along) won't need a Synapse dashboard at all, etc. So not only should we not download it, but maybe we should even try to clean up after ourselves. It doesn't seem too important now, and I'd rather not block this PR any more (because #456 is to be merged soon.. and #818 is waiting upon #456 as well, etc.), so I got it merged as-is. I believe it works great for most of us anyway!

Thanks a lot for the huge amount of work on this, @Peetz0r! 👍

Peetz0r force-pushed the stats branch from 317089f to 316d5e3 Compare January 30, 2021 08:47

Peetz0r added 12 commits February 10, 2021 22:54

Prometheus

13ef9e8

Initial attempt. Seems to work fine. Only tested on debian amd64 so far

Prometheus Node Exporter

e525970

Basic system stats, to show stuff the synapse metrics can't show such as resource usage by bridges, etc Seems to work fine as well. This too has only been tested on debian amd64 so far

Grafana

eb5aa93

Also includes the dashboards for Synapse and for Node Exporter. Again has only been tested on debian amd64 so far, but the grafana docker image is available for arm64 and arm32. Nice.

Grafana nginx proxy config

989100b

Documentation for graphs

a10e324

Make prometheus-node-exporter a bit more capable

76d7e84

By running it in a more privileged container with access to the host network stack and such

Mention stats subdomain in docs

1079272

Mention new images in docks

8aafb44

Document security and privacy considerations for stats.

da82d67

Register docker network info and use it for prometheus-node-exporter

144a5e6

Using the hardcoded IP did break while I was messing with IPv6 stuff on the other branch

Update Grafana 7.3.7 => 7.4.0

3a77261

Update Prometheus Node Exporter 1.0.1 => 1.1.0

fde222a

Peetz0r force-pushed the stats branch from dbe6ce3 to fde222a Compare February 10, 2021 22:11

Peetz0r and others added 6 commits February 12, 2021 11:59

Split DNS table in default and optional parts

9531d13

Fix Grafana dashboard/datasource label

3ce9712

Improve metrics docs page a bit

1d7d99c

We mainly switch the anonymous metrics viewing variable to false, along with other wording changes.

Use --read-only FS for metrics-related containers

df3dd1c

It seems like it doesn't cause any issues for any of these services.

Make --tags=setup-prometheus not break, relying on matrix-base facts

85a260d

spantaleev added 6 commits February 12, 2021 11:59

Rename some variables

18e3152

Split install/uninstall tasks in matrix-prometheus

6842102

Break dependency between matrix-prometheus and (matrix-prometheus-nod…

c8ab200

…e-exporter, matrix-synapse)

Minor docs updates

eb9aac0

Do not auto-expose metrics on matrix.DOMAIN/_synapse/metrics

2b47258

.. and other documentation improvements.

Announce Prometheus/Grafana

890e4ad

spantaleev merged commit 2ac2b02 into spantaleev:master Feb 12, 2021

ptman mentioned this pull request Feb 17, 2021

Add Prometheus and Grafana to display metrics #4

Closed

spantaleev mentioned this pull request Feb 19, 2021

Synapse workers #456

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prometheus and Grafana on stats.<domain> #838

Prometheus and Grafana on stats.<domain> #838

Peetz0r commented Jan 29, 2021 •

edited

Loading

pushytoxin commented Jan 29, 2021

pushytoxin commented Jan 29, 2021

aaronraimist commented Jan 30, 2021 •

edited

Loading

Peetz0r commented Jan 30, 2021

Peetz0r commented Jan 30, 2021

spantaleev commented Jan 31, 2021

Peetz0r commented Jan 31, 2021

pushytoxin commented Feb 1, 2021 •

edited

Loading

spantaleev commented Feb 3, 2021

Peetz0r commented Feb 10, 2021

spantaleev commented Feb 12, 2021

Prometheus and Grafana on stats.<domain> #838

Prometheus and Grafana on stats.<domain> #838

Conversation

Peetz0r commented Jan 29, 2021 • edited Loading

pushytoxin commented Jan 29, 2021

pushytoxin commented Jan 29, 2021

aaronraimist commented Jan 30, 2021 • edited Loading

Peetz0r commented Jan 30, 2021

Peetz0r commented Jan 30, 2021

spantaleev commented Jan 31, 2021

Peetz0r commented Jan 31, 2021

pushytoxin commented Feb 1, 2021 • edited Loading

spantaleev commented Feb 3, 2021

Peetz0r commented Feb 10, 2021

spantaleev commented Feb 12, 2021

Peetz0r commented Jan 29, 2021 •

edited

Loading

aaronraimist commented Jan 30, 2021 •

edited

Loading

pushytoxin commented Feb 1, 2021 •

edited

Loading