From 3486889db9a563ff6ba70a90c869f9fc913686ec Mon Sep 17 00:00:00 2001 From: Ed Welch Date: Thu, 13 Aug 2020 13:07:32 -0400 Subject: [PATCH] Release 1.6.0 (#2502) * Updating changelog and release notes for 1.6.0 * updating version in documentation --- CHANGELOG.md | 288 +++++++++++++++++- README.md | 31 +- docs/sources/clients/aws/ec2/_index.md | 4 +- docs/sources/clients/aws/ecs/_index.md | 2 +- docs/sources/clients/aws/ecs/ecs-task.json | 2 +- docs/sources/clients/aws/eks/values.yaml | 2 +- docs/sources/clients/promtail/installation.md | 2 +- docs/sources/configuration/_index.md | 13 +- docs/sources/installation/docker.md | 20 +- docs/sources/operations/loki-canary.md | 2 +- docs/sources/operations/upgrade.md | 89 +++++- 11 files changed, 410 insertions(+), 45 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 40b3143f5bc4..ebab3563ae6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,286 @@ -## Unreleased (Master) +## 1.6.0 (2020-08-13) + +It's the second thursday of the eighth month of the year which means it's time for another Loki Release!! + +Before we highlight important features and changes, congratulations to [@adityacs](https://github.com/adityacs), who is the newest member of the Loki team! +Aditya has been regularly contributing to the Loki project for the past year, with each contribution better than the last. +Many of the items on the following list were thanks to his hard work. Thank you, Aditya, and welcome to the team! + +I think we might have set a new record with 189 PR's in this release! + +### Important Notes + +**Please Note** There are several changes in this release which might require your attention! + +* The NET_BIND_SERVICE capability was removed from the Loki process in the docker image, it's no longer possible to run Loki with the supplied image on a port less than 1024 +* If you run microservices, there is an important rollout sequence to prevent query errors. +* Scrape configs have changed for Promtail in both Helm and Ksonnet affecting two labels: `instance` -> `pod` and `container_name` -> `container`. +* Almost all of the Loki Canary metrics were renamed. +* A few command line flags where changed (although they are likely not commonly used) +* If you use ksonnet and run on GCS and Bigtable you may see an error in your config as a default value was removed. +* If you are using boltdb-shipper, you will likekly need to add a new schema_config entry. + +Check the [upgrade guide](https://github.com/grafana/loki/blob/master/docs/sources/operations/upgrade.md#160) for detailed information on all these changes. + +### Notable Features and Fixes + +#### Query language enhancements + +* [2150](https://github.com/grafana/loki/pull/2150) introduces `bytes_rate`, which calculates the per second byte rate of a log stream, and `bytes_over_time`, which returns the byte size of a log stream. +* [2182](https://github.com/grafana/loki/pull/2182) introduces a long list of comparison operators, which will let you write queries like `count_over_time({foo="bar"}[1m]) > 10`. Check out the PR for a more detailed description. + +#### Loki performance improvements + +* [2216](https://github.com/grafana/loki/pull/2216), [2218](https://github.com/grafana/loki/pull/2218), and [2219](https://github.com/grafana/loki/pull/2219) all improve how memory is allocated and reused for queries. +* [2239](https://github.com/grafana/loki/pull/2239) is a huge improvement for certain cases in which a query covers a large number of streams that all overlap in time. Overlapping data is now internally cached while Loki works to sort all the streams into the proper time order. +* [2293](https://github.com/grafana/loki/pull/2293) was a big refactor to how Loki internally processes log queries vs. metric queries, creating separate code paths to further optimize metric queries. Metric query performance is now 2 to 10 times faster. + +If you are using the query-frontend: + +* [2441](https://github.com/grafana/loki/pull/2441) improves how label queries can be split and queried in parallel +* [2123](https://github.com/grafana/loki/pull/2123) allows queries to the `series` API to be split by time and parallelized; and last but most significant +* [1927](https://github.com/grafana/loki/pull/1927) allows for a much larger range of queries to be sharded and performed in parallel. Query sharding is a topic in itself, but as a rough summary, this type of sharding is not time dependent and leverages how data is already stored by Loki to be able to split queries up into 16 separate pieces to be queried at the same time. + +#### Promtail + +* [2296](https://github.com/grafana/loki/pull/2296) allows Promtail to expose the Loki Push API. With this, you can push from any client to Promtail as if it were Loki, and Promtail can then forward those logs to another Promtail or to Loki. There are some good use cases for this with the Loki Docker Logging Driver; if you want an easier way to configure pipelines or expose metrics collection, point your Docker drivers at a Promtail instance. +* [2282](https://github.com/grafana/loki/pull/2282) contains an example Amazon Lambda where you can use a fan-in approach and ingestion timestamping in Promtail to work around `out of order` issues with multiple Lambdas processing the same log stream. This is one way to get logs from a high-cardinality source without adding a high-cardinality label. +* [2060](https://github.com/grafana/loki/pull/2060) introduces the `Replace` stage, which lets you find and replace or remove text inside a log line. Combined with [2422](https://github.com/grafana/loki/pull/2422) and [2480](https://github.com/grafana/loki/pull/2480), you can now find and replace sensitive data in a log line like a password or email address and replace it with ****, or hash the value to prevent readability, while still being able to trace the value through your logs. Last on the list of pipeline additions, +* [2496](https://github.com/grafana/loki/pull/2496) adds a `Drop` pipeline stage, which lets you drop log lines based on several criteria options including regex matching content, line length, or the age of the log line. The last two are useful to prevent sending to Loki logs that you know would be rejected based on configured limits in the Loki server. + +#### Logstash output plugin + +* [1822](https://github.com/grafana/loki/pull/1822) added a Logstash output plugin for Loki. If you have an existing Logstash install, you can now use this plugin to send your logs to Loki to make it easier to try out, or use Loki alongside an existing logging installation. + +#### Loki Canary + +* [2344](https://github.com/grafana/loki/pull/2344) improved the canaries capabilities for checking for data integrity, including spot checking for logs over a longer time window and running metric queries to verify count_over_time accuracy. + +#### Logcli + +* [2470](https://github.com/grafana/loki/pull/2470) allows you to color code your log lines based on their stream labels for a nice visual indicator of streams. +* [2497](https://github.com/grafana/loki/pull/2497) expands on the series API query to Loki with the`--analyze-labels` flag, which can show you a detailed breakdown of your label key and value combinations. This is very useful for finding improper label usage in Loki or labels with high cardinality. +* [2482](https://github.com/grafana/loki/pull/2482), in which LogCLI will automatically batch requests to Loki to allow making queries with a `--limit=` far larger than the server side limit defined in Loki. LogCLI will dispatch the request in a series of queries configured by the `--batch=` parameter (which defaults to 1000) until the requested limit is reached! + +#### Misc + +* [2453](https://github.com/grafana/loki/pull/2453) improves the error messages when a query times out, as `Context Deadline Exceeded` wasn’t the most intuitive. +* [2336](https://github.com/grafana/loki/pull/2336) provides two new flags that will print the entire Loki config object at startup. Be warned there are a lot of config options, and many won’t apply to your setup (such as storage configs you aren’t using), but this can be a really useful tool when troubleshooting. Sticking with the theme of best for last, +* [2224](https://github.com/grafana/loki/pull/2224) and [2288](https://github.com/grafana/loki/pull/2288) improve support for running Loki with a shared Ring using memberlist while not requiring Consul or Etcd. We need to follow up soon with some better documentation or a blog post on this! + + +### Dependencies + +* Go Version: 1.14.2 +* Cortex Version: 7014ff11ed70d9d59ad29d0a95e73999c436c47c + +### All Changes + +#### Loki +* [2484](https://github.com/grafana/loki/pull/2484) **slim-bean**: Loki: fix batch iterator error when all chunks overlap and chunk time ranges are greater than query time range +* [2483](https://github.com/grafana/loki/pull/2483) **sandeepsukhani**: download boltdb files parallelly during reads +* [2472](https://github.com/grafana/loki/pull/2472) **owen-d**: series endpoint uses normal splits +* [2466](https://github.com/grafana/loki/pull/2466) **owen-d**: BatchIter edge cases +* [2463](https://github.com/grafana/loki/pull/2463) **sandeepsukhani**: revendor cortex to latest master +* [2457](https://github.com/grafana/loki/pull/2457) **adityacs**: Fix panic in cassandra storage while registering metrics +* [2453](https://github.com/grafana/loki/pull/2453) **slim-bean**: Loki: Improve error messages on query timeout or cancel +* [2450](https://github.com/grafana/loki/pull/2450) **adityacs**: Fixes panic in runtime_config +* [2449](https://github.com/grafana/loki/pull/2449) **jvrplmlmn**: Replace usage of sync/atomic with uber-go/atomic +* [2441](https://github.com/grafana/loki/pull/2441) **cyriltovena**: Split label names queries in the frontend. +* [2427](https://github.com/grafana/loki/pull/2427) **owen-d**: Revendor cortex +* [2392](https://github.com/grafana/loki/pull/2392) **owen-d**: avoid mutating config while parsing -config.file +* [2346](https://github.com/grafana/loki/pull/2346) **cyriltovena**: Fixes LogQL grouping +* [2336](https://github.com/grafana/loki/pull/2336) **slim-bean**: Loki: add -print-config-stderr flag to dump loki's runtime config to stderr +* [2330](https://github.com/grafana/loki/pull/2330) **slim-bean**: Loki: Use a new context to update the ring state after a failed chunk transfer +* [2328](https://github.com/grafana/loki/pull/2328) **slim-bean**: Loki: Transfer one chunk at a time per series during chunk transfers +* [2327](https://github.com/grafana/loki/pull/2327) **adityacs**: Fix data race in ingester +* [2323](https://github.com/grafana/loki/pull/2323) **cyriltovena**: Improve object key parsing for boltdb shipper. +* [2306](https://github.com/grafana/loki/pull/2306) **cyriltovena**: Fixes buffered iterator skipping very long lines. +* [2302](https://github.com/grafana/loki/pull/2302) **cyriltovena**: Improve entry deduplication. +* [2294](https://github.com/grafana/loki/pull/2294) **cyriltovena**: Remove NET_BIND_SERVICE capability requirement. +* [2293](https://github.com/grafana/loki/pull/2293) **cyriltovena**: Improve metric queries by computing samples at the edges. +* [2288](https://github.com/grafana/loki/pull/2288) **periklis**: Add support for memberlist dns-based discovery +* [2268](https://github.com/grafana/loki/pull/2268) **owen-d**: lock fix for flaky test +* [2266](https://github.com/grafana/loki/pull/2266) **cyriltovena**: Update to latest cortex. +* [2264](https://github.com/grafana/loki/pull/2264) **adityacs**: Fix ingester results for series query +* [2261](https://github.com/grafana/loki/pull/2261) **sandeepsukhani**: create smaller unique files from boltdb shipper and other code refactorings +* [2254](https://github.com/grafana/loki/pull/2254) **slim-bean**: Loki: Series API will return all series with no match or empty matcher +* [2252](https://github.com/grafana/loki/pull/2252) **owen-d**: avoids further time splitting in querysharding mware +* [2250](https://github.com/grafana/loki/pull/2250) **slim-bean**: Loki: Remove redundant log warning +* [2249](https://github.com/grafana/loki/pull/2249) **owen-d**: avoids recording stats in the sharded engine +* [2248](https://github.com/grafana/loki/pull/2248) **cyriltovena**: Add performance profile flags for logcli. +* [2239](https://github.com/grafana/loki/pull/2239) **cyriltovena**: Cache overlapping blocks +* [2224](https://github.com/grafana/loki/pull/2224) **periklis**: Replace memberlist service in favor of cortex provided service +* [2223](https://github.com/grafana/loki/pull/2223) **adityacs**: Add Error method for step evaluators +* [2219](https://github.com/grafana/loki/pull/2219) **cyriltovena**: Reuse slice for the range vector allocations. +* [2218](https://github.com/grafana/loki/pull/2218) **cyriltovena**: Reuse buffer for hash computation in the engine. +* [2216](https://github.com/grafana/loki/pull/2216) **cyriltovena**: Improve point allocations for each steps in the logql engine. +* [2211](https://github.com/grafana/loki/pull/2211) **sandeepsukhani**: query tee proxy with support for comparison of responses +* [2206](https://github.com/grafana/loki/pull/2206) **sandeepsukhani**: disable index dedupe when rf > 1 and current or upcoming index type is boltdb-shipper +* [2204](https://github.com/grafana/loki/pull/2204) **owen-d**: bumps cortex & fixes conflicts +* [2191](https://github.com/grafana/loki/pull/2191) **periklis**: Add flag to disable tracing activation +* [2189](https://github.com/grafana/loki/pull/2189) **owen-d**: Fix vector-scalar comparisons +* [2182](https://github.com/grafana/loki/pull/2182) **owen-d**: Logql comparison ops +* [2178](https://github.com/grafana/loki/pull/2178) **cyriltovena**: Fixes path prefix in the querier. +* [2166](https://github.com/grafana/loki/pull/2166) **sandeepsukhani**: enforce requirment for periodic config for index tables to be 24h when using boltdb shipper +* [2161](https://github.com/grafana/loki/pull/2161) **cyriltovena**: Fix error message for max tail connections. +* [2156](https://github.com/grafana/loki/pull/2156) **sandeepsukhani**: boltdb shipper download failure handling and some refactorings +* [2150](https://github.com/grafana/loki/pull/2150) **cyriltovena**: Bytes aggregations +* [2136](https://github.com/grafana/loki/pull/2136) **cyriltovena**: Fixes Iterator boundaries +* [2123](https://github.com/grafana/loki/pull/2123) **adityacs**: Fix Series API slowness +* [1927](https://github.com/grafana/loki/pull/1927) **owen-d**: Feature/querysharding ii +* [2032](https://github.com/grafana/loki/pull/2032) **tivvit**: Added support for tail to query frontend + +#### Promtail +* [2496](https://github.com/grafana/loki/pull/2496) **slim-bean**: Promtail: Drop stage +* [2475](https://github.com/grafana/loki/pull/2475) **slim-bean**: Promtail: force the log level on any Loki Push API target servers to match Promtail's log level. +* [2474](https://github.com/grafana/loki/pull/2474) **slim-bean**: Promtail: use --client.external-labels for all clients +* [2471](https://github.com/grafana/loki/pull/2471) **owen-d**: Fix/promtail yaml config +* [2464](https://github.com/grafana/loki/pull/2464) **slim-bean**: Promtail: Bug: loki push api, clone labels before handling +* [2438](https://github.com/grafana/loki/pull/2438) **rfratto**: pkg/promtail: propagate a logger rather than using util.Logger globally +* [2432](https://github.com/grafana/loki/pull/2432) **pyr0hu**: Promtail: Allow empty replace values for replace stage +* [2422](https://github.com/grafana/loki/pull/2422) **wardbekker**: Template: Added a sha256 template function for obfuscating / anonymize PII data in e.g. the replace stage +* [2414](https://github.com/grafana/loki/pull/2414) **rfratto**: Add RegisterFlagsWithPrefix to config structs +* [2386](https://github.com/grafana/loki/pull/2386) **cyriltovena**: Add regex function to promtail template stage. +* [2345](https://github.com/grafana/loki/pull/2345) **adityacs**: Refactor Promtail target manager code +* [2301](https://github.com/grafana/loki/pull/2301) **flixr**: Promtail: support unix timestamps with fractional seconds +* [2296](https://github.com/grafana/loki/pull/2296) **slim-bean**: Promtail: Loki Push API +* [2282](https://github.com/grafana/loki/pull/2282) **owen-d**: Lambda-Promtail +* [2242](https://github.com/grafana/loki/pull/2242) **carlpett**: Set user agent on outgoing http requests +* [2196](https://github.com/grafana/loki/pull/2196) **cyriltovena**: Adds default -config.file for the promtail docker images. +* [2127](https://github.com/grafana/loki/pull/2127) **bastjan**: Update go-syslog to accept non-UTF8 encoding in syslog message +* [2111](https://github.com/grafana/loki/pull/2111) **adityacs**: Fix Promtail journal seeking known position +* [2105](https://github.com/grafana/loki/pull/2105) **fatpat**: promtail: Add Entry variable to template +* [1118](https://github.com/grafana/loki/pull/1118) **shuttie**: promtail: fix high CPU usage on large kubernetes clusters. +* [2060](https://github.com/grafana/loki/pull/2060) **adityacs**: Feature: Replace stage in pipeline +* [2087](https://github.com/grafana/loki/pull/2087) **adityacs**: Set JournalTarget Priority value to keyword + +#### Logcli +* [2497](https://github.com/grafana/loki/pull/2497) **slim-bean**: logcli: adds --analyize-labels to logcli series command and changes how labels are provided to the command +* [2482](https://github.com/grafana/loki/pull/2482) **slim-bean**: Logcli: automatically batch requests +* [2470](https://github.com/grafana/loki/pull/2470) **adityacs**: colored labels output for logcli +* [2235](https://github.com/grafana/loki/pull/2235) **pstibrany**: logcli: Remove single newline from the raw line before printing. +* [2126](https://github.com/grafana/loki/pull/2126) **cyriltovena**: Validate local storage config for the logcli +* [2083](https://github.com/grafana/loki/pull/2083) **adityacs**: Support querying labels on time range in logcli + +#### Docs +* [2473](https://github.com/grafana/loki/pull/2473) **owen-d**: fixes lambda-promtail relative doc link +* [2454](https://github.com/grafana/loki/pull/2454) **oddlittlebird**: Create CODEOWNERS +* [2439](https://github.com/grafana/loki/pull/2439) **till**: Docs: updated "Upgrading" for docker driver +* [2437](https://github.com/grafana/loki/pull/2437) **wardbekker**: DOCS: clarified globbing behaviour of __path__ of the doublestar library +* [2431](https://github.com/grafana/loki/pull/2431) **endu**: fix dead link +* [2425](https://github.com/grafana/loki/pull/2425) **RichiH**: Change conduct contact email address +* [2420](https://github.com/grafana/loki/pull/2420) **petuhovskiy**: Fix docker driver doc +* [2418](https://github.com/grafana/loki/pull/2418) **cyriltovena**: Add logstash to clients page with FrontMatter. +* [2402](https://github.com/grafana/loki/pull/2402) **cyriltovena**: More fixes for the website. +* [2400](https://github.com/grafana/loki/pull/2400) **tontongg**: Fix URL to LogQL documentation +* [2398](https://github.com/grafana/loki/pull/2398) **robbymilo**: Docs - update links, readme +* [2397](https://github.com/grafana/loki/pull/2397) **coderanger**: 📝 Note that entry_parser is deprecated. +* [2396](https://github.com/grafana/loki/pull/2396) **dnsmichi**: Docs: Fix Fluentd title (visible in menu) +* [2391](https://github.com/grafana/loki/pull/2391) **cyriltovena**: Update fluentd docs and fixes links for the website. +* [2390](https://github.com/grafana/loki/pull/2390) **cyriltovena**: Fluent bit docs +* [2389](https://github.com/grafana/loki/pull/2389) **cyriltovena**: Docker driver doc +* [2385](https://github.com/grafana/loki/pull/2385) **abowloflrf**: Update logo link in README.md +* [2378](https://github.com/grafana/loki/pull/2378) **robbymilo**: Sync docs to website +* [2360](https://github.com/grafana/loki/pull/2360) **owen-d**: Makes timestamp parsing docs clearer +* [2358](https://github.com/grafana/loki/pull/2358) **rille111**: Documentation: Add example for having separate pvc for loki, using helm +* [2357](https://github.com/grafana/loki/pull/2357) **owen-d**: Storage backend examples +* [2338](https://github.com/grafana/loki/pull/2338) **cyriltovena**: Add a complete tutorial on how to ship logs from AWS EKS. +* [2335](https://github.com/grafana/loki/pull/2335) **cyriltovena**: Improve documentation of the metric stage. +* [2331](https://github.com/grafana/loki/pull/2331) **cyriltovena**: Add a tutorial to forward AWS ECS logs to Loki. +* [2321](https://github.com/grafana/loki/pull/2321) **cyriltovena**: Tutorial to run Promtail on AWS EC2 +* [2318](https://github.com/grafana/loki/pull/2318) **adityacs**: Configuration documentation improvements +* [2317](https://github.com/grafana/loki/pull/2317) **owen-d**: remove DynamoDB chunk store doc +* [2308](https://github.com/grafana/loki/pull/2308) **wardbekker**: Added a link to the replace parsing stage +* [2305](https://github.com/grafana/loki/pull/2305) **rafaelpissolatto**: Fix schema_config store value +* [2285](https://github.com/grafana/loki/pull/2285) **adityacs**: Fix local.md doc +* [2284](https://github.com/grafana/loki/pull/2284) **owen-d**: Update local.md +* [2279](https://github.com/grafana/loki/pull/2279) **Fra-nk**: Documentation: Refine LogQL documentation +* [2273](https://github.com/grafana/loki/pull/2273) **RichiH**: Fix typo +* [2247](https://github.com/grafana/loki/pull/2247) **carlpett**: docs: Fix missing quotes +* [2233](https://github.com/grafana/loki/pull/2233) **vyzigold**: docs: Add readmes to individual helm charts +* [2220](https://github.com/grafana/loki/pull/2220) **oddlittlebird**: Docs: Local install edits +* [2217](https://github.com/grafana/loki/pull/2217) **fredr**: docs: BoltDB typo +* [2215](https://github.com/grafana/loki/pull/2215) **fredr**: docs: Correct loki address for docker-compose +* [2172](https://github.com/grafana/loki/pull/2172) **cyriltovena**: Update old link for pipeline stages. +* [2163](https://github.com/grafana/loki/pull/2163) **slim-bean**: docs: fix an error in the example log line and byte counter metrics +* [2160](https://github.com/grafana/loki/pull/2160) **slim-bean**: Fix some errors in the upgrade guide to 1.5.0 and add some missing notes discovered by users. +* [2152](https://github.com/grafana/loki/pull/2152) **eamonryan**: Fix typo in promtail ClusterRole +* [2139](https://github.com/grafana/loki/pull/2139) **adityacs**: Fix configuration docs +* [2137](https://github.com/grafana/loki/pull/2137) **RichiH**: Propose new governance +* [2136](https://github.com/grafana/loki/pull/2136) **cyriltovena**: Fixes Iterator boundaries +* [2125](https://github.com/grafana/loki/pull/2125) **theMercedes**: Update logql.md +* [2112](https://github.com/grafana/loki/pull/2112) **nileshcs**: Documentation: Outdated fluentd image name, UID details, link update +* [2092](https://github.com/grafana/loki/pull/2092) **i-takizawa**: docs: make visible + +#### Build +* [2467](https://github.com/grafana/loki/pull/2467) **slim-bean**: Update Loki build image + +#### Ksonnet +* [2460](https://github.com/grafana/loki/pull/2460) **Duologic**: refactor: use $.core.v1.envVar +* [2452](https://github.com/grafana/loki/pull/2452) **slim-bean**: ksonnet: Reduce querier parallelism to a more sane default value and remove the default setting for storage_backend +* [2377](https://github.com/grafana/loki/pull/2377) **Duologic**: refactor: moved jaeger-agent-mixin +* [2373](https://github.com/grafana/loki/pull/2373) **slim-bean**: Ksonnet: Add a Pod Disruption Budget to Loki Ingesters +* [2185](https://github.com/grafana/loki/pull/2185) **cyriltovena**: Refactor mixin routes and add series API. +* [2162](https://github.com/grafana/loki/pull/2162) **slim-bean**: ksonnet: Fix up datasources and variables in Loki Operational +* [2091](https://github.com/grafana/loki/pull/2091) **beorn7**: Keep scrape config in line with the new Prometheus scrape config + +#### Docker logging driver +* [2435](https://github.com/grafana/loki/pull/2435) **cyriltovena**: Add more precisions on the docker driver installed on the daemon. +* [2343](https://github.com/grafana/loki/pull/2343) **jdfalk**: loki-docker-driver: Change "ignoring empty line" to debug logging +* [2295](https://github.com/grafana/loki/pull/2295) **cyriltovena**: Remove mount in the docker driver. +* [2199](https://github.com/grafana/loki/pull/2199) **cyriltovena**: Docker driver relabeling +* [2116](https://github.com/grafana/loki/pull/2116) **cyriltovena**: Allows to change the log driver mode and buffer size. + +#### Logstash output plugin +* [2415](https://github.com/grafana/loki/pull/2415) **cyriltovena**: Set service values via --set for logstash. +* [2410](https://github.com/grafana/loki/pull/2410) **adityacs**: logstash code refactor and doc improvements +* [1822](https://github.com/grafana/loki/pull/1822) **adityacs**: Loki Logstash Plugin + +#### Loki canary +* [2413](https://github.com/grafana/loki/pull/2413) **slim-bean**: Loki-Canary: Backoff retries on query failures, add histograms for query performance. +* [2369](https://github.com/grafana/loki/pull/2369) **slim-bean**: Loki Canary: One more round of improvements to query for missing websocket entries up to max-wait +* [2350](https://github.com/grafana/loki/pull/2350) **slim-bean**: Canary tweaks +* [2344](https://github.com/grafana/loki/pull/2344) **slim-bean**: Loki-Canary: Add query spot checking and metric count checking +* [2259](https://github.com/grafana/loki/pull/2259) **ombre8**: Canary: make stream configurable + +#### Fluentd +* [2407](https://github.com/grafana/loki/pull/2407) **cyriltovena**: bump fluentd version to release a new gem. +* [2399](https://github.com/grafana/loki/pull/2399) **tarokkk**: fluentd: Make fluentd version requirements permissive +* [2179](https://github.com/grafana/loki/pull/2179) **takanabe**: Improve fluentd plugin development experience +* [2171](https://github.com/grafana/loki/pull/2171) **takanabe**: Add server TLS certificate verification + +#### Fluent Bit +* [2375](https://github.com/grafana/loki/pull/2375) **cyriltovena**: Fixes the fluentbit batchwait backward compatiblity. +* [2367](https://github.com/grafana/loki/pull/2367) **dojci**: fluent-bit: Add more loki client configuration options +* [2365](https://github.com/grafana/loki/pull/2365) **dojci**: fluent-bit: Fix fluent-bit exit callback when buffering is enabled +* [2290](https://github.com/grafana/loki/pull/2290) **cyriltovena**: Fixes the lint issue merged to master. +* [2286](https://github.com/grafana/loki/pull/2286) **adityacs**: Fix fluent-bit newline and tab characters +* [2142](https://github.com/grafana/loki/pull/2142) **dojci**: Add FIFO queue persistent buffering for fluent bit output plugin +* [2089](https://github.com/grafana/loki/pull/2089) **FrederikNS**: Allow configuring more options for output configuration + +#### Helm +* [2406](https://github.com/grafana/loki/pull/2406) **steven-sheehy**: Helm: Fix regression in chart name +* [2379](https://github.com/grafana/loki/pull/2379) **StevenReitsma**: production/helm: Add emptyDir volume type to promtail PSP +* [2366](https://github.com/grafana/loki/pull/2366) **StevenReitsma**: production/helm: Add projected and downwardAPI volume types to PodSecurityPolicy (#2355) +* [2258](https://github.com/grafana/loki/pull/2258) **Synehan**: helm: add annotations to service monitor +* [2241](https://github.com/grafana/loki/pull/2241) **chauffer**: Kubernetes manifests: Remove namespace from cluster-wide resources +* [2238](https://github.com/grafana/loki/pull/2238) **vhrosales**: helm: Add loadBalancerIP option to loki chart +* [2205](https://github.com/grafana/loki/pull/2205) **joschi36**: BUG: add missing namespace in ingress object +* [2197](https://github.com/grafana/loki/pull/2197) **cyriltovena**: Render loki datasources even if Grafana is disabled. +* [2141](https://github.com/grafana/loki/pull/2141) **cyriltovena**: Adds the ability to have a pull secrets for Promtail. +* [2099](https://github.com/grafana/loki/pull/2099) **allout58**: helm/loki-stack: Support Prometheus on a sub-path in Grafana config +* [2086](https://github.com/grafana/loki/pull/2086) **osela**: helm/loki-stack: render loki datasource only if grafana is enabled +* [2091](https://github.com/grafana/loki/pull/2091) **beorn7**: Keep scrape config in line with the new Prometheus scrape config + +#### Build +* [2371](https://github.com/grafana/loki/pull/2371) **cyriltovena**: Fixes helm publish that needs now to add repo. +* [2341](https://github.com/grafana/loki/pull/2341) **slim-bean**: Build: Fix CI helm test +* [2309](https://github.com/grafana/loki/pull/2309) **cyriltovena**: Test again arm32 on internal ci. +* [2307](https://github.com/grafana/loki/pull/2307) **cyriltovena**: Removes arm32 for now as we're migrating the CI. +* [2287](https://github.com/grafana/loki/pull/2287) **wardbekker**: Change the Grafana image to latest +* [2212](https://github.com/grafana/loki/pull/2212) **roidelapluie**: Remove unhelpful/problematic term in circleci.yml -* [2282](https://github.com/grafana/loki/pull/2282) **owen-d**: introduces a [lambda-promtail](https://github.com/grafana/loki/blob/master/docs/clients/lambda-promtail/README.md) workflow for shipping Cloudwatch logs to Loki. ## 1.5.0 (2020-05-20) @@ -20,7 +300,7 @@ there were substantial breaking config changes in Cortex 1.0 which standardized **The Loki docker image user has changed to no longer be root** -Check the [upgrade guide](https://github.com/grafana/loki/blob/master/docs/operations/upgrade.md#150) for more detailed information on these changes. +Check the [upgrade guide](https://github.com/grafana/loki/blob/master/docs/sources/operations/upgrade.md#150) for more detailed information on these changes. ### Notable Features and Fixes @@ -274,7 +554,7 @@ Over 130 PR's merged for this release, from 40 different contributors!! We cont Before we get into new features, version 1.4.0 brings with it the first (that we are aware of) upgrade dependency. -We have created a dedicated page for upgrading Loki in the [operations section of the docs](https://github.com/grafana/loki/blob/master/docs/operations/upgrade.md#140) +We have created a dedicated page for upgrading Loki in the [operations section of the docs](https://github.com/grafana/loki/blob/master/docs/sources/operations/upgrade.md#140) The docker image tag naming was changed, the starting in 1.4.0 docker images no longer have the `v` prefix: `grafana/loki:1.4.0` diff --git a/README.md b/README.md index 0199648055ba..8b37180aa9cd 100644 --- a/README.md +++ b/README.md @@ -29,9 +29,9 @@ Loki differs from Prometheus by focusing on logs instead of metrics, and deliver ## Getting started -* [Installing Loki](https://github.com/grafana/loki/tree/v1.5.0/docs/installation/README.md) -* [Installing Promtail](https://github.com/grafana/loki/tree/v1.5.0/docs/clients/promtail/installation.md) -* [Getting Started Guide](https://github.com/grafana/loki/tree/v1.5.0/docs/getting-started/README.md) +* [Installing Loki](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/installation/README.md) +* [Installing Promtail](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/clients/promtail/installation.md) +* [Getting Started Guide](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/getting-started/README.md) ## Upgrading @@ -40,6 +40,7 @@ Loki differs from Prometheus by focusing on logs instead of metrics, and deliver ### Documentation * [master](./docs/README.md) +* [v1.6.0](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/_index.md) * [v1.5.0](https://github.com/grafana/loki/tree/v1.5.0/docs/README.md) * [v1.4.1](https://github.com/grafana/loki/tree/v1.4.1/docs/README.md) * [v1.4.0](https://github.com/grafana/loki/tree/v1.4.0/docs/README.md) @@ -48,18 +49,18 @@ Loki differs from Prometheus by focusing on logs instead of metrics, and deliver * [v1.1.0](https://github.com/grafana/loki/tree/v1.1.0/docs/README.md) * [v1.0.0](https://github.com/grafana/loki/tree/v1.0.0/docs/README.md) -Commonly used sections (from the latest release v1.5.0): - -- [API documentation](https://github.com/grafana/loki/tree/v1.5.0/docs/api.md) for alternative ways of getting logs into Loki. -- [Labels](https://github.com/grafana/loki/blob/master/docs/getting-started/labels.md) -- [Operations](https://github.com/grafana/loki/tree/v1.5.0/docs/operations) for important aspects of running Loki. -- [Promtail](https://github.com/grafana/loki/tree/v1.5.0/docs/clients/promtail) is an agent which can tail your log files and push them to Loki. -- [Pipelines](https://github.com/grafana/loki/tree/v1.5.0/docs/clients/promtail/pipelines.md) for detailed log processing pipeline documentation -- [Docker Logging Driver](https://github.com/grafana/loki/tree/v1.5.0/docs/clients/docker-driver) is a docker plugin to send logs directly to Loki from Docker containers. -- [LogCLI](https://github.com/grafana/loki/tree/v1.5.0/docs/getting-started/logcli.md) on how to query your logs without Grafana. -- [Loki Canary](https://github.com/grafana/loki/tree/v1.5.0/docs/operations/loki-canary.md) for monitoring your Loki installation for missing logs. -- [Troubleshooting](https://github.com/grafana/loki/tree/v1.5.0/docs/getting-started/troubleshooting.md) for help around frequent error messages. -- [Loki in Grafana](https://github.com/grafana/loki/tree/v1.5.0/docs/getting-started/grafana.md) for how to set up a Loki datasource in Grafana and query your logs. +Commonly used sections (from the latest release v1.6.0): + +- [API documentation](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/api/_index.md) for alternative ways of getting logs into Loki. +- [Labels](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/getting-started/labels.md) +- [Operations](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/operations/_index.md) for important aspects of running Loki. +- [Promtail](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/clients/promtail/_index.md) is an agent which can tail your log files and push them to Loki. +- [Pipelines](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/clients/promtail/pipelines.md) for detailed log processing pipeline documentation +- [Docker Logging Driver](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/clients/docker-driver/_index.md) is a docker plugin to send logs directly to Loki from Docker containers. +- [LogCLI](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/getting-started/logcli.md) on how to query your logs without Grafana. +- [Loki Canary](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/operations/loki-canary.md) for monitoring your Loki installation for missing logs. +- [Troubleshooting](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/getting-started/troubleshooting.md) for help around frequent error messages. +- [Loki in Grafana](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/getting-started/grafana.md) for how to set up a Loki datasource in Grafana and query your logs. ## Getting Help diff --git a/docs/sources/clients/aws/ec2/_index.md b/docs/sources/clients/aws/ec2/_index.md index cdb13f9ac742..89fd9ecc751f 100644 --- a/docs/sources/clients/aws/ec2/_index.md +++ b/docs/sources/clients/aws/ec2/_index.md @@ -91,7 +91,7 @@ Next we'll download, install and give executable right to [Promtail](../../promt ```bash mkdir /opt/promtail && cd /opt/promtail -curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/promtail-linux-amd64.zip" +curl -O -L "https://github.com/grafana/loki/releases/download/v1.6.0/promtail-linux-amd64.zip" unzip "promtail-linux-amd64.zip" chmod a+x "promtail-linux-amd64" ``` @@ -231,7 +231,7 @@ Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal systemd[1]: Started P Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal systemd[1]: Starting Promtail... Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=warn ts=2020-07-08T15:48:57.559085451Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages" Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=info ts=2020-07-08T15:48:57.559869071Z caller=server.go:179 http=[::]:3100 grpc=[::]:35127 msg="server listening on addresses" -Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=info ts=2020-07-08T15:48:57.56029474Z caller=main.go:67 msg="Starting Promtail" version="(version=1.5.0, branch=HEAD, revision=12c7eab8)" +Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=info ts=2020-07-08T15:48:57.56029474Z caller=main.go:67 msg="Starting Promtail" version="(version=1.6.0, branch=HEAD, revision=12c7eab8)" ``` You can now verify in Grafana that Loki has correctly received your instance logs by using the [LogQL](../../../logql/) query `{zone="us-east-2"}`. diff --git a/docs/sources/clients/aws/ecs/_index.md b/docs/sources/clients/aws/ecs/_index.md index 4aa2ac917568..8c3fc41d7207 100644 --- a/docs/sources/clients/aws/ecs/_index.md +++ b/docs/sources/clients/aws/ecs/_index.md @@ -98,7 +98,7 @@ curl https://raw.githubusercontent.com/grafana/loki/master/docs/aws/ecs/ecs-task ```json { "essential": true, - "image": "grafana/fluent-bit-plugin-loki:1.5.0-amd64", + "image": "grafana/fluent-bit-plugin-loki:1.6.0-amd64", "name": "log_router", "firelensConfiguration": { "type": "fluentbit", diff --git a/docs/sources/clients/aws/ecs/ecs-task.json b/docs/sources/clients/aws/ecs/ecs-task.json index cce553a43ee6..b4d4435a368a 100644 --- a/docs/sources/clients/aws/ecs/ecs-task.json +++ b/docs/sources/clients/aws/ecs/ecs-task.json @@ -2,7 +2,7 @@ "containerDefinitions": [ { "essential": true, - "image": "grafana/fluent-bit-plugin-loki:1.5.0-amd64", + "image": "grafana/fluent-bit-plugin-loki:1.6.0-amd64", "name": "log_router", "firelensConfiguration": { "type": "fluentbit", diff --git a/docs/sources/clients/aws/eks/values.yaml b/docs/sources/clients/aws/eks/values.yaml index d39e0b46a3c0..966e67922075 100644 --- a/docs/sources/clients/aws/eks/values.yaml +++ b/docs/sources/clients/aws/eks/values.yaml @@ -17,7 +17,7 @@ initContainer: image: repository: grafana/promtail - tag: 1.5.0 + tag: 1.6.0 pullPolicy: IfNotPresent ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. diff --git a/docs/sources/clients/promtail/installation.md b/docs/sources/clients/promtail/installation.md index f4f791b9a3d2..7f2baf93e0b2 100644 --- a/docs/sources/clients/promtail/installation.md +++ b/docs/sources/clients/promtail/installation.md @@ -15,7 +15,7 @@ Every release includes binaries for Promtail which can be found on the ```bash # modify tag to most recent version -docker pull grafana/promtail:1.5.0 +docker pull grafana/promtail:1.6.0 ``` ## Helm diff --git a/docs/sources/configuration/_index.md b/docs/sources/configuration/_index.md index 72b72d1cfb25..1d333b5f4dbd 100644 --- a/docs/sources/configuration/_index.md +++ b/docs/sources/configuration/_index.md @@ -1101,11 +1101,16 @@ redis: [enable_tls: | default = false] fifocache: - # Number of entries to cache in-memory. - # CLI flag: -.fifocache.size - [size: | default = 0] + # Maximum memory size of the cache in bytes. A unit suffix (KB, MB, GB) may be + # applied. + # CLI flag: -.fifocache.max-size-bytes + [max_size_bytes: | default = ""] - # The expiry duration for the in-memory cache. + # Maximum number of entries in the cache. + # CLI flag: -.fifocache.max-size-items + [max_size_items: | default = 0] + + # The expiry duration for the cache. # CLI flag: -.fifocache.duration [validity: | default = 0s] ``` diff --git a/docs/sources/installation/docker.md b/docs/sources/installation/docker.md index f7e2bb1d008d..4a45e01681c2 100644 --- a/docs/sources/installation/docker.md +++ b/docs/sources/installation/docker.md @@ -18,17 +18,17 @@ For production, we recommend installing with Tanka or Helm. Copy and paste the commands below into your command line. ```bash -wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml -docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:1.5.0 -config.file=/mnt/config/loki-config.yaml -wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml -docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:1.5.0 -config.file=/mnt/config/promtail-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.6.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml +docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:1.6.0 -config.file=/mnt/config/loki-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.6.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml +docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:1.6.0 -config.file=/mnt/config/promtail-config.yaml ``` When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files. Navigate to http://localhost:3100/metrics to view the metrics and http://localhost:3100/ready for readiness. -As of v1.5.0, image is configured to run by default as user loki with UID `10001` and GID `10001`. You can use a different user, specially if you are using bind mounts, by specifying the UID with a `docker run` command and using `--user=UID` with numeric UID suited to your needs. +As of v1.6.0, image is configured to run by default as user loki with UID `10001` and GID `10001`. You can use a different user, specially if you are using bind mounts, by specifying the UID with a `docker run` command and using `--user=UID` with numeric UID suited to your needs. **Windows** @@ -36,10 +36,10 @@ Copy and paste the commands below into your terminal. Note that you will need to ```bash cd "" -wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml -docker run -v :/mnt/config -p 3100:3100 grafana/loki:1.5.0 --config.file=/mnt/config/loki-config.yaml -wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml -docker run -v :/mnt/config -v /var/log:/var/log grafana/promtail:1.5.0 --config.file=/mnt/config/promtail-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.6.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml +docker run -v :/mnt/config -p 3100:3100 grafana/loki:1.6.0 --config.file=/mnt/config/loki-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.6.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml +docker run -v :/mnt/config -v /var/log:/var/log grafana/promtail:1.6.0 --config.file=/mnt/config/promtail-config.yaml ``` When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files. @@ -51,6 +51,6 @@ Navigate to http://localhost:3100/metrics to view the output. Run the following commands in your command line. They work for Windows or Linux systems. ```bash -wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/production/docker-compose.yaml -O docker-compose.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.6.0/production/docker-compose.yaml -O docker-compose.yaml docker-compose -f docker-compose.yaml up ``` diff --git a/docs/sources/operations/loki-canary.md b/docs/sources/operations/loki-canary.md index cad19f971bfd..607a1ab8711f 100644 --- a/docs/sources/operations/loki-canary.md +++ b/docs/sources/operations/loki-canary.md @@ -116,7 +116,7 @@ Loki Canary is also provided as a Docker container image: ```bash # change tag to the most recent release -$ docker pull grafana/loki-canary:1.5.0 +$ docker pull grafana/loki-canary:1.6.0 ``` ### Kubernetes diff --git a/docs/sources/operations/upgrade.md b/docs/sources/operations/upgrade.md index a2f47ad2f2e0..2e731807c2ec 100644 --- a/docs/sources/operations/upgrade.md +++ b/docs/sources/operations/upgrade.md @@ -12,23 +12,102 @@ On this page we will document any upgrade issues/gotchas/considerations we are a ## Master / Unreleased -#### Documentation changes: +## 1.6.0 -Configuration document has been re-orderd a bit and for all the config, corresponding `CLI` flag is provided. +### IMPORTANT: Ksonnet Port Change and Removal of NET_BIND_SERVICE Capability from docker image -S3 config now supports exapnded config. Example can be found here [s3_expanded_config](../configuration/examples.md#s3-expanded-config) +In 1.5.0 we changed the Loki user to not run as root which created problems binding to port 80. +To address this we updated the docker image to add the NET_BIND_SERVICE capability to the loki process +which allowed Loki to bind to port 80 as a non root user, so long as the underlying system allowed that +linux capability. -### New Ingester GRPC API special rollout procedure in microservices mode +This has proved to be a problem for many reasons and in PR [2294](https://github.com/grafana/loki/pull/2294/files) +the capability was removed. -A new ingester GRPC API has been added allowing to speed up metric queries, to ensure a rollout without query errors make sure you upgrade all ingesters first. +It is now no longer possible for the Loki to be started with a port less than 1024 with the published docker image. + +The default for Helm has always been port 3100, and Helm users should be unaffect unless they changed the default. + +**Ksonnet users however should closely check their configuration, in PR 2294 the loki port was changed from 80 to 3100** + + +### IMPORTANT: If you run Loki in microservices mode, special rollout instructions + +A new ingester GRPC API has been added allowing to speed up metric queries, to ensure a rollout without query errors **make sure you upgrade all ingesters first.** Once this is done you can then proceed with the rest of the deployment, this is to ensure that queriers won't look for an API not yet available. If you roll out everything at once, queriers with this new code will attempt to query ingesters which may not have the new method on the API and queries will fail. This will only affect reads(queries) and not writes and only for the duration of the rollout. +### IMPORTANT: Scrape config changes to both Helm and Ksonnet will affect labels created by Promtail + +PR [2091](https://github.com/grafana/loki/pull/2091) Makes several changes to the promtail scrape config: + +```` +This is triggered by https://github.com/grafana/jsonnet-libs/pull/261 + +The above PR changes the instance label to be actually unique within +a scrape config. It also adds a pod and a container target label +so that metrics can easily be joined with metrics from cAdvisor, KSM, +and the Kubelet. + +This commit adds the same to the Loki scrape config. It also removes +the container_name label. It is the same as the container label +and was already added to Loki previously. However, the +container_name label is deprecated and has disappeared in K8s 1.16, +so that it will soon become useless for direct joining. +```` + +TL;DR + +The following label have been changed in both the Helm and Ksonnet Promtail scrape configs: + +`instance` -> `pod` +`container_name` -> `container` + + +### Experimental boltdb-shipper changes + +PR [2166](https://github.com/grafana/loki/pull/2166) now forces the index to have a period of exactly `24h`: + +Loki will fail to start with an error if the active schema or upcoming schema are not set to a period of `24h` + +You can add a new schema config like this: + +```yaml +schema_config: + configs: + - from: 2020-01-01 <----- This is your current entry, date will be different + store: boltdb-shipper + object_store: aws + schema: v11 + index: + prefix: index_ + period: 168h + - from: [INSERT FUTURE DATE HERE] <----- Add another entry, set a future date + store: boltdb-shipper + object_store: aws + schema: v11 + index: + prefix: index_ + period: 24h <--- This must be 24h +``` +If you are not on `schema: v11` this would be a good oportunity to make that change _in the new schema config_ also. + +**NOTE** If the current time in your timezone is after midnight UTC already, set the date one additional day forward. + +There was also a significant overhaul to how boltdb-shipper internals, this should not be visible to a user but as this +feature is experimental and under development bug are possible! + +The most noticeable change if you look in the storage, Loki no longer updates an existing file and instead creates a +new index file every 15mins, this is an important move to make sure objects in the object store are immutable and +will simplify future operations like compaction and deletion. + ### Breaking CLI flags changes +The following CLI flags where changed to improve consistency, they are not expected to be widely used + ```diff - querier.query_timeout + querier.query-timeout