Adding more buckets to loki_memcache_request_duration_seconds histogr…

…am (#9431)

**What this PR does / why we need it**:
Users using extstore (with NVMe drives) may experience latencies higher
than the ~250ms max as previously defined.

---------

Signed-off-by: Danny Kopping <danny.kopping@grafana.com>

CHANGELOG.md

@@ -27,6 +27,7 @@
 * [8732](https://github.com/grafana/loki/pull/8732) **abaguas**: azure: respect retry config before cancelling the context
 * [9206](https://github.com/grafana/loki/pull/9206) **dannykopping**: Ruler: log rule evaluation detail.
 * [9184](https://github.com/grafana/loki/pull/9184) **periklis**: Bump dskit to introduce IPv6 support for memberlist
+* [9431](https://github.com/grafana/loki/pull/9431) **dannykopping**: Add more buckets to `loki_memcache_request_duration_seconds` metric; latencies can increase if using memcached with NVMe
 
 ##### Fixes
 

pkg/storage/chunk/cache/memcached.go

@@ -65,8 +65,13 @@ func NewMemcached(cfg MemcachedConfig, client MemcachedClient, name string, reg
 				Namespace: "loki",
 				Name:      "memcache_request_duration_seconds",
 				Help:      "Total time spent in seconds doing memcache requests.",
-				// Memcached requests are very quick: smallest bucket is 16us, biggest is 1s
-				Buckets:     prometheus.ExponentialBuckets(0.000016, 4, 8),
+				// 16us, 64us, 256us, 1.024ms, 4.096ms, 16.384ms, 65.536ms, 150ms, 250ms, 500ms, 1s
+				Buckets: append(prometheus.ExponentialBuckets(0.000016, 4, 7), []float64{
+					(150 * time.Millisecond).Seconds(),
+					(250 * time.Millisecond).Seconds(),
+					(500 * time.Millisecond).Seconds(),
+					(time.Second).Seconds(),
+				}...),
 				ConstLabels: prometheus.Labels{"name": name},
 			}, []string{"method", "status_code"}),
 		),