Merge branch 'main' into trinity--search-after-scroll

quickwit-oss · Jan 25, 2024 · 48a36bd · 48a36bd
2 parents aaa2656 + 9eef3bb
commit 48a36bd
Show file tree

Hide file tree

Showing 109 changed files with 2,028 additions and 1,130 deletions.
diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml
@@ -110,7 +110,7 @@ jobs:
           sudo apt install libsasl2-dev
           sudo apt install libsasl2-2
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: |
             ~/.cargo/git

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -22,6 +22,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 --->
 
+## [0.7.1]
+
+### Added
+
+- Add es _count API (#4410)
+- Add _elastic/_field_caps API (#4350)
+- Make gRPC message size configurable (#4388)
+- Add API endpoint to get some control-plan internal info (#4339)
+- Add Google Cloud Storage Implementation available for storage paths starting with `gs://` (#4344)
+
+### Changed
+
+- Return 404 on index not found in ES Bulk API (#4425)
+- Allow $ and @ characters in field names (#4413)
+
+### Fixed
+- Assign all sources/shards, even if this requires exceeding the indexer #4363 
+- Fix traces doc mapping (service name set as  fast) and update default otel logs index ID to `otel-logs-v0_7` (#4401)
+- Fix parsing multi-line queries (#4409)
+- Fix range query for optional fast field panics with Index out of bounds (#4362)
+
+### Migration from 0.7.0 to 0.7.1
+
+Quickwit 0.7.1 will create the new index `otel-logs-v0_7` which is now used by default when ingesting data with the OTEL gRPC and HTTP API.
+
+In the traces index `otel-traces-v0_7`, the `service_name` field is now fast. No migration is done if `otel-traces-v0_7` already exists. If you want `service_name` field to be fast, you have to delete first the existing `otel-traces-v0_7` index or create your own index.
+
+## [0.7.0]
+
 ### Added
 
 - Elasticsearch-compatible API

diff --git a/config/tutorials/grafana/docker-compose.yml b/config/tutorials/grafana/docker-compose.yml
@@ -10,7 +10,7 @@ networks:
 
 services:
   quickwit:
-    image: quickwit/quickwit:${QUICKWIT_VERSION:-0.7.0}
+    image: quickwit/quickwit:${QUICKWIT_VERSION:-0.7.1}
   grafana:
     image: grafana/grafana-oss:${GRAFANA_VERSION:-9.4.7}
     container_name: grafana

diff --git a/docs/configuration/node-config.md b/docs/configuration/node-config.md
@@ -31,6 +31,7 @@ A commented example is available here: [quickwit.yaml](https://github.com/quickw
 | `data_dir` | Path to directory where data (tmp data, splits kept for caching purpose) is persisted. This is mostly used in indexing. | `QW_DATA_DIR` | `./qwdata` |
 | `metastore_uri` | Metastore URI. Can be a local directory or `s3://my-bucket/indexes` or `postgres://username:password@localhost:5432/metastore`. [Learn more about the metastore configuration](metastore-config.md). | `QW_METASTORE_URI` | `{data_dir}/indexes` |
 | `default_index_root_uri` | Default index root URI that defines the location where index data (splits) is stored. The index URI is built following the scheme: `{default_index_root_uri}/{index-id}` | `QW_DEFAULT_INDEX_ROOT_URI` | `{data_dir}/indexes` |
+| environment variable only | Log level of Quickwit. Can be a direct log level, or a comma separated list of `module_name=level` | `RUST_LOG` | `info` |
 
 ## REST configuration
 

diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md
@@ -43,13 +43,13 @@ Downloads `.tar.gz`:
 <!-- Bellow is the set of links to edit when a new version is released -->
 <RenderIf condition={useDocsVersion().version != 'current'}>
 
-version: 0.7.0 - [Release notes](https://github.com/quickwit-oss/quickwit/releases/tag/v0.7.0) - [Changelog](https://github.com/quickwit-oss/quickwit/blob/main/CHANGELOG.md)
+version: 0.7.1 - [Release notes](https://github.com/quickwit-oss/quickwit/releases/tag/v0.7.1) - [Changelog](https://github.com/quickwit-oss/quickwit/blob/main/CHANGELOG.md)
 License: [AGPL V3](https://github.com/quickwit-oss/quickwit/blob/main/LICENSE.md)
 Downloads `.tar.gz`:
-- [Linux ARM64](https://github.com/quickwit-oss/quickwit/releases/download/v0.7.0/quickwit-v0.7.0-aarch64-unknown-linux-gnu.tar.gz)
-- [Linux x86_64](https://github.com/quickwit-oss/quickwit/releases/download/v0.7.0/quickwit-v0.7.0-x86_64-unknown-linux-gnu.tar.gz)
-- [macOS aarch64](https://github.com/quickwit-oss/quickwit/releases/download/v0.7.0/quickwit-v0.7.0-aarch64-apple-darwin.tar.gz)
-- [macOS x86_64](https://github.com/quickwit-oss/quickwit/releases/download/v0.7.0/quickwit-v0.7.0-x86_64-apple-darwin.tar.gz)
+- [Linux ARM64](https://github.com/quickwit-oss/quickwit/releases/download/v0.7.1/quickwit-v0.7.1-aarch64-unknown-linux-gnu.tar.gz)
+- [Linux x86_64](https://github.com/quickwit-oss/quickwit/releases/download/v0.7.1/quickwit-v0.7.1-x86_64-unknown-linux-gnu.tar.gz)
+- [macOS aarch64](https://github.com/quickwit-oss/quickwit/releases/download/v0.7.1/quickwit-v0.7.1-aarch64-apple-darwin.tar.gz)
+- [macOS x86_64](https://github.com/quickwit-oss/quickwit/releases/download/v0.7.1/quickwit-v0.7.1-x86_64-apple-darwin.tar.gz)
 
 </RenderIf>
 

diff --git a/docs/get-started/quickstart.md b/docs/get-started/quickstart.md
@@ -62,6 +62,8 @@ docker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 127.0.0.1:7280:7280 quickwi
 
 </Tabs>
 
+Tips: you can use the environment variable `RUST_LOG` to control quickwit verbosity.
+
 Check it's working by browsing the [UI at http://localhost:7280](http://localhost:7280) or do a simple GET with cURL:
 
 ```bash

diff --git a/docs/get-started/tutorials/_category_.yaml b/docs/get-started/tutorials/_category_.yaml
@@ -1,3 +1,3 @@
 label: 'Tutorials'
 position: 2
-collapsed: true
+collapsed: false
diff --git a/docs/get-started/tutorials/trace-analytics-with-grafana.md b/docs/get-started/tutorials/trace-analytics-with-grafana.md
@@ -1,5 +1,5 @@
 ---
-title: Trace analytics with Grafana
+title: Logs and Traces with Grafana
 description: A simple tutorial to use Grafana with Quickwit's datasource plugin.
 icon_url: /img/tutorials/quickwit-logo.png
 tags: [grafana, integration]
@@ -36,7 +36,7 @@ Then we create a [Grafana](https://grafana.com/docs/grafana/latest/setup-grafana
     ports:
       - "${MAP_HOST_GRAFANA:-127.0.0.1}:3000:3000"
     environment:
-      GF_INSTALL_PLUGINS: https://github.com/quickwit-oss/quickwit-datasource/releases/download/v0.3.0-beta.2/quickwit-quickwit-datasource-0.3.0-beta.2.zip;quickwit-quickwit-datasource
+      GF_INSTALL_PLUGINS: https://github.com/quickwit-oss/quickwit-datasource/releases/download/v0.3.1/quickwit-quickwit-datasource-0.3.1.zip;quickwit-quickwit-datasource
       GF_AUTH_DISABLE_LOGIN_FORM: "true"
       GF_AUTH_ANONYMOUS_ENABLED: "true"
       GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
@@ -54,21 +54,26 @@ You should be able to access Quickwit's UI on `http://localhost:7280/` and Grafa
 
 In Grafana, head to [Data Sources](http://localhost:3000/connections/datasources). If the plugin is installed correctly you should be able to find Quickwit in the list.
 
-We're going to set up a new Quickwit data source lookig at Quickwit's own OpenTelemetry traces, let's configure the datasource with the following parameters:
+We're going to set up a new Quickwit data source looking at Quickwit's own OpenTelemetry traces, let's configure the datasource with the following parameters:
 
-- URL : `http://quickwit:7280/api/v1/` _This uses the docker service name as the host_
-- Index ID : `otel-traces-v0_6`
-- Timestamp field : `span_start_timestamp_nanos`
-- Timestamp output format : `unix_timestamp_nanos`
+- URL : `http://quickwit:7280/api/v1` _This uses the docker service name as the host_
+- Index ID : `otel-traces-v0_7`
 
 Save and test, you should obtain a confirmation that the datasource is correctly set up.
 
 
 ![Quickwit Plugin configuration success](../../assets/images/grafana-ui-quickwit-datasource-plugin-success.png)
 
+
+You can also set up a new Quickwit data source looking at Quickwit's own OpenTelemetry logs (or your own logs index), let's configure the datasource with the following parameters:
+
+- URL : `http://quickwit:7280/api/v1` _This uses the docker service name as the host_
+- Index ID : `otel-logs-v0_7`
+
+
 ## Creating a dashboard
 
-You can then [create a new dashboard](http://localhost:3000/dashboard/new) and add a visualization : you should be able to choose our new quickwit datasource here.
+You can then [create a new dashboard](http://localhost:3000/dashboard/new) and add a visualization : you should be able to choose the traces quickwit datasource here.
 
 Quickwit sends itself its own traces, so you should already have data to display. Let's configure some panels !
 

diff --git a/docs/get-started/tutorials/tutorial-jaeger.md b/docs/get-started/tutorials/tutorial-jaeger.md
@@ -1,6 +1,6 @@
 ---
-title: Tracing with Jaeger
-sidebar_position: 1
+title: Traces with Jaeger
+sidebar_position: 2
 ---
 
 In this quick start guide, we will set up a Quickwit instance and analyze its own traces with Jaeger using Docker Compose.
@@ -16,7 +16,7 @@ version: "3"
 
 services:
   quickwit:
-    image: quickwit/quickwit:${QW_VERSION:-0.7.0}
+    image: quickwit/quickwit:${QW_VERSION:-0.7.1}
     volumes:
       - ./qwdata:/quickwit/qwdata
     ports:
@@ -69,7 +69,7 @@ You can also use the Quickwit UI at [http://localhost:7280](http://localhost:728
 Here are a couple of query examples:
 - `service_name:quickwit AND events.event_attributes.level:INFO`
 - `span_duration_millis:>100`
-- `resource_attributes.service.version:v0.7.0`
+- `resource_attributes.service.version:v0.7.1`
 - `service_name:quickwit`
 
 That's it! You can level up with the following tutorials to discover all Quickwit features.

diff --git a/docs/log-management/send-logs/send-docker-logs.md b/docs/log-management/send-logs/send-docker-logs.md
@@ -86,7 +86,7 @@ x-default-logging: &logging
 
 services:
   quickwit:
-    image: quickwit/quickwit:${QW_VERSION:-0.7.0}
+    image: quickwit/quickwit:${QW_VERSION:-0.7.1}
     volumes:
       - ./qwdata:/quickwit/qwdata
     ports:

diff --git a/docs/reference/cli.md b/docs/reference/cli.md
@@ -819,28 +819,15 @@ Disables [telemetry](../telemetry.md) when set to any non-empty value.
 
 `QW_DISABLE_TELEMETRY=1 quickwit help`
 
-<!--
-    End of auto-generated CLI docs
--->
+### RUST_LOG
 
-## Environment Variables
+Configure quickwit log level.
 
-### QW_CLUSTER_ENDPOINT
-
-Specifies the address of the cluster to connect to. Management commands `index`, `split` and `source` require the `cluster_endpoint`, which you can set once and for all with the `QW_CLUSTER_ENDPOINT` environment variable.
-
-### QW_CONFIG
-
-Specifies the path to the [quickwit config](../configuration/node-config.md). Commands `run` and `tools` require the `config`, which you can set once and for all with the `QW_CONFIG` environment variable.
-
-*Example*
-
-`export QW_CONFIG=config/quickwit.yaml`
-
-### QW_DISABLE_TELEMETRY
-
-Disables [telemetry](../telemetry.md) when set to any non-empty value.
-
-*Example*
+*Examples*
 
-`QW_DISABLE_TELEMETRY=1 quickwit help`
+```
+# run with higher verbosity
+RUST_LOG=debug quickwit run
+# run with log level info, except for indexing related logs
+RUST_LOG=info,quickwit_indexing=debug quickwit run
+```