Skip to content

Commit

Permalink
Add ingest v2 internal doc (#5107)
Browse files Browse the repository at this point in the history
* Add ingest v2 internal doc

* Add caveats sections in ingest v2

* Add index template docs in internal docs
  • Loading branch information
fmassot authored Jun 20, 2024
1 parent 09888bb commit 511e1dd
Show file tree
Hide file tree
Showing 2 changed files with 56 additions and 0 deletions.
24 changes: 24 additions & 0 deletions docs/internals/ingest-v2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Ingest V2

Ingest V2 is a new ingestion API that is designed to be more efficient and scalable for thousands of indexes than the previous version. It is currently in beta and is not yet enabled by default.

## Enabling Ingest V2

To enable Ingest V2, you need to set the `QW_ENABLE_INGEST_V2` environment variable to `1` on the indexer, control-plane, and metastore services.

You also have to activate the `enable_cooperative_indexing` option in the indexer configuration. The indexer configuration is in the node configuration:

```yaml
version: 0.8
# [...]
indexer:
enable_cooperative_indexing: true
```
See [full configuration example](https://github.com/quickwit-oss/quickwit/blob/main/config/quickwit.yaml).
The only way to use the ingest API V2 is to use the [bulk endpoint](../reference/es_compatible_api#_bulk--batch-ingestion-endpoint) of the Elasticsearch-compatible API. The native Quickwit API is not yet compatible with the ingest V2 API.
## Caveats
The `refresh` parameter is not yet supported on the ingest V2 API.
32 changes: 32 additions & 0 deletions docs/internals/template-index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Index template API

Index templates are a way to create indexes automatically with some given configuration when Quickwit receives documents for an index that doesn't exist yet.

Example of templates: [https://github.com/quickwit-oss/quickwit/tree/main/config/templates](https://github.com/quickwit-oss/quickwit/tree/main/config/templates).

# Curl to run to use the REST API to create Stackoverflow template

```bash
curl -XPOST -H 'Content-Type: application/yaml' 'http://localhost:7280/api/v1/templates' --data-binary @config/templates/stackoverflow.yaml

# Lists templates.
curl 'http://localhost:7280/api/v1/templates'

# Update Stackoverflow template.
curl -XPUT -H 'Content-Type: application/yaml' 'http://localhost:7280/api/v1/templates/stackoverflow' --data-binary @config/templates/stackoverflow.yaml

# Download dataset.
curl -O https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json

# Ingest 10k docs into `stackoverflow-foo` index.
curl -XPOST "http://127.0.0.1:7280/api/v1/stackoverflow-foo/ingest-v2" --data-binary @stackoverflow.posts.transformed-10000.json

# Ingest 10k docs into `stackoverflow-bar` index.
curl -XPOST "http://127.0.0.1:7280/api/v1/stackoverflow-bar/ingest-v2" --data-binary @stackoverflow.posts.transformed-10000.json

# Delete Stackoverflow template.
curl -XDELETE 'http://localhost:7280/api/v1/templates/stackoverflow'

```bash

0 comments on commit 511e1dd

Please sign in to comment.