diff --git a/docs/specifications/edges-buffers-buckets.md b/docs/specifications/edges-buffers-buckets.md index ac87013bde..a90ed08857 100644 --- a/docs/specifications/edges-buffers-buckets.md +++ b/docs/specifications/edges-buffers-buckets.md @@ -1,7 +1,5 @@ # Edges, Buffers and Buckets -![Proposal](../assets/proposal.svg) - > This document describes the concepts of `Edge`, `Buffer` and `Bucket` in a pipeline. ## Edges diff --git a/docs/user-guide/reference/multi-partition.md b/docs/user-guide/reference/multi-partition.md new file mode 100644 index 0000000000..98436bb36d --- /dev/null +++ b/docs/user-guide/reference/multi-partition.md @@ -0,0 +1,22 @@ +# Multi-partitioned Edges + +To achieve higher throughput(> 10K but < 25K), users can create multi-partitioned edges. +Multi-partitioned edges are only supported for pipelines with JetStream as ISB. Please ensure +that the JetStream is provisioned with more nodes to support higher throughput. + +Since partitions are owned by the vertex that reads from it, to create a multi-partitioned edge +we need to configure the vertex that reads from it to have multiple partitions. + +Below is the example snippet to configure a vertex (`cat` vertex) to have multiple partitions, +this means vertex reading from `cat` can read at high TPS. + +```yaml + - name: cat + partitions: 3 + udf: + builtin: + name: cat # A built-in UDF which simply cats the message +``` + + + diff --git a/examples/9-multi-partition-pipeline.yaml b/examples/9-multi-partition-pipeline.yaml new file mode 100644 index 0000000000..fcb6b65900 --- /dev/null +++ b/examples/9-multi-partition-pipeline.yaml @@ -0,0 +1,29 @@ +apiVersion: numaflow.numaproj.io/v1alpha1 +kind: Pipeline +metadata: + name: simple-mp-pipeline +spec: + vertices: + - name: in + source: + # A self data generating source + generator: + rpu: 100 + duration: 1s + keyCount: 5 + value: 5 + - name: cat + partitions: 3 + udf: + builtin: + name: cat # A built-in UDF which simply cats the message + - name: out + partitions: 3 + sink: + # A simple log printing sink + log: {} + edges: + - from: in + to: cat + - from: cat + to: out