From 511e1dd638babef968d1f2f80dbd7bed50b3e87d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fran=C3=A7ois=20Massot?= <francois.massot@gmail.com>
Date: Thu, 20 Jun 2024 13:49:08 +0200
Subject: [PATCH] Add ingest v2 internal doc (#5107)

* Add ingest v2 internal doc

* Add caveats sections in ingest v2

* Add index template docs in internal docs
---
 docs/internals/ingest-v2.md      | 24 ++++++++++++++++++++++++
 docs/internals/template-index.md | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
 create mode 100644 docs/internals/ingest-v2.md
 create mode 100644 docs/internals/template-index.md

diff --git a/docs/internals/ingest-v2.md b/docs/internals/ingest-v2.md
new file mode 100644
index 00000000000..c925b2377ee
--- /dev/null
+++ b/docs/internals/ingest-v2.md
@@ -0,0 +1,24 @@
+# Ingest V2
+
+Ingest V2 is a new ingestion API that is designed to be more efficient and scalable for thousands of indexes than the previous version. It is currently in beta and is not yet enabled by default.
+
+## Enabling Ingest V2
+
+To enable Ingest V2, you need to set the `QW_ENABLE_INGEST_V2` environment variable to `1` on the indexer, control-plane, and metastore services.
+
+You also have to activate the `enable_cooperative_indexing` option in the indexer configuration. The indexer configuration is in the node configuration:
+
+```yaml
+version: 0.8
+# [...]
+indexer:
+  enable_cooperative_indexing: true
+```
+
+See [full configuration example](https://github.com/quickwit-oss/quickwit/blob/main/config/quickwit.yaml).
+
+The only way to use the ingest API V2 is to use the [bulk endpoint](../reference/es_compatible_api#_bulk--batch-ingestion-endpoint) of the Elasticsearch-compatible API. The native Quickwit API is not yet compatible with the ingest V2 API.
+
+## Caveats
+
+The `refresh` parameter is not yet supported on the ingest V2 API.
diff --git a/docs/internals/template-index.md b/docs/internals/template-index.md
new file mode 100644
index 00000000000..5f60778c6a7
--- /dev/null
+++ b/docs/internals/template-index.md
@@ -0,0 +1,32 @@
+# Index template API
+
+Index templates are a way to create indexes automatically with some given configuration when Quickwit receives documents for an index that doesn't exist yet.
+
+Example of templates: [https://github.com/quickwit-oss/quickwit/tree/main/config/templates](https://github.com/quickwit-oss/quickwit/tree/main/config/templates).
+
+# Curl to run to use the REST API to create Stackoverflow template
+
+```bash
+curl -XPOST -H 'Content-Type: application/yaml' 'http://localhost:7280/api/v1/templates' --data-binary @config/templates/stackoverflow.yaml
+
+# Lists templates.
+curl 'http://localhost:7280/api/v1/templates'
+
+# Update Stackoverflow template.
+curl -XPUT -H 'Content-Type: application/yaml' 'http://localhost:7280/api/v1/templates/stackoverflow' --data-binary @config/templates/stackoverflow.yaml
+
+# Download dataset.
+curl -O https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json
+
+# Ingest 10k docs into `stackoverflow-foo` index.
+curl -XPOST "http://127.0.0.1:7280/api/v1/stackoverflow-foo/ingest-v2" --data-binary @stackoverflow.posts.transformed-10000.json
+
+# Ingest 10k docs into `stackoverflow-bar` index.
+curl -XPOST "http://127.0.0.1:7280/api/v1/stackoverflow-bar/ingest-v2" --data-binary @stackoverflow.posts.transformed-10000.json
+
+# Delete Stackoverflow template.
+curl -XDELETE 'http://localhost:7280/api/v1/templates/stackoverflow'
+
+```bash
+
+