Skip to content

Commit

Permalink
chore: docs for multi partitioned edges (#844)
Browse files Browse the repository at this point in the history
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Vigith Maurice <[email protected]>
Co-authored-by: Vigith Maurice <[email protected]>
  • Loading branch information
yhl25 and vigith committed Jul 11, 2023
1 parent 80c75c2 commit f0d04b8
Show file tree
Hide file tree
Showing 3 changed files with 51 additions and 2 deletions.
2 changes: 0 additions & 2 deletions docs/specifications/edges-buffers-buckets.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# Edges, Buffers and Buckets

![Proposal](../assets/proposal.svg)

> This document describes the concepts of `Edge`, `Buffer` and `Bucket` in a pipeline.
## Edges
Expand Down
22 changes: 22 additions & 0 deletions docs/user-guide/reference/multi-partition.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Multi-partitioned Edges

To achieve higher throughput(> 10K but < 25K), users can create multi-partitioned edges.
Multi-partitioned edges are only supported for pipelines with JetStream as ISB. Please ensure
that the JetStream is provisioned with more nodes to support higher throughput.

Since partitions are owned by the vertex that reads from it, to create a multi-partitioned edge
we need to configure the vertex that reads from it to have multiple partitions.

Below is the example snippet to configure a vertex (`cat` vertex) to have multiple partitions,
this means vertex reading from `cat` can read at high TPS.

```yaml
- name: cat
partitions: 3
udf:
builtin:
name: cat # A built-in UDF which simply cats the message
```
29 changes: 29 additions & 0 deletions examples/9-multi-partition-pipeline.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
apiVersion: numaflow.numaproj.io/v1alpha1
kind: Pipeline
metadata:
name: simple-mp-pipeline
spec:
vertices:
- name: in
source:
# A self data generating source
generator:
rpu: 100
duration: 1s
keyCount: 5
value: 5
- name: cat
partitions: 3
udf:
builtin:
name: cat # A built-in UDF which simply cats the message
- name: out
partitions: 3
sink:
# A simple log printing sink
log: {}
edges:
- from: in
to: cat
- from: cat
to: out

0 comments on commit f0d04b8

Please sign in to comment.