docs: running on istio (numaproj#1056)

Signed-off-by: Derek Wang <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>

docs/user-guide/reference/configuration/istio.md

@@ -0,0 +1,51 @@
+# Running on Istio
+
+If you want to get pipeline vertices running on Istio, so that they are able to talk to other services with Istio enabled, one approach is to whitelist the ports that Numaflow uses.
+
+To whitelist the ports, add `traffic.sidecar.istio.io/excludeInboundPorts` and `traffic.sidecar.istio.io/excludeOutboundPorts` annotations to your vertex configuration:
+
+```yaml
+apiVersion: numaflow.numaproj.io/v1alpha1
+kind: Pipeline
+metadata:
+  name: my-pipeline
+spec:
+  vertices:
+    - name: my-vertex
+      metadata:
+        annotations:
+          sidecar.istio.io/inject: "true"
+          traffic.sidecar.istio.io/excludeOutboundPorts: "4222" # Nats JetStream port
+          traffic.sidecar.istio.io/excludeInboundPorts: "2469" # Numaflow vertex metrics port
+      udf:
+        container:
+          image: my-udf-image:latest
+    ...
+```
+
+If you want to apply same configuration to all the vertices, use [Vertex Template](./pipeline-customization.md#vertices):
+
+```yaml
+apiVersion: numaflow.numaproj.io/v1alpha1
+kind: Pipeline
+metadata:
+  name: my-pipeline
+spec:
+  templates:
+    vertex:
+      metadata:
+        annotations:
+          sidecar.istio.io/inject: "true"
+          traffic.sidecar.istio.io/excludeOutboundPorts: "4222" # Nats JetStream port
+          traffic.sidecar.istio.io/excludeInboundPorts: "2469" # Numaflow vertex metrics port
+  vertices:
+    - name: my-vertex-1
+      udf:
+        container:
+          image: my-udf-1-image:latest
+    - name: my-vertex-2
+      udf:
+        container:
+          image: my-udf-2-image:latest
+    ...
+```

docs/user-guide/reference/configuration/pipeline-operations.md → docs/user-guide/reference/pipeline-operations.md

@@ -7,7 +7,7 @@ You might want to make some changes to an existing pipeline, for example, updati
 The scenarios include but are not limited to:
 
 - Topology changes such as adding or removing vertices, or updating the edges between vertices.
-- Updating the [partitions](../multi-partition.md) for a [keyed](../../user-defined-functions/reduce/windowing/windowing.md#keyed) reduce vertex.
+- Updating the [partitions](multi-partition.md) for a [keyed](../user-defined-functions/reduce/windowing/windowing.md#keyed) reduce vertex.
 - Updating the user-defined container image for a vertex, while the new image can not properly handle the unprocessed data in its backlog.
 
 To summarize, if there are unprocessed messages in the pipeline, and the new pipeline spec will change the way how the messages are processed, then you should delete and recreate the pipeline.
@@ -24,7 +24,7 @@ Pausing a pipeline will not cause data loss. It does not clean up the unprocesse
 
 When pausing a pipeline, it will shutdown the source vertex pods first, and then wait for the other vertices to finish the backlog before terminating them.
 
-If there's a [reduce](../../user-defined-functions/reduce/reduce.md) vertex in the pipeline, please make sure it uses [Persistent Volume Claim](../../user-defined-functions/reduce/reduce.md#persistent-volume-claim-pvc) for storage, otherwise the data will be lost.
+If there's a [reduce](../user-defined-functions/reduce/reduce.md) vertex in the pipeline, please make sure it uses [Persistent Volume Claim](../user-defined-functions/reduce/reduce.md#persistent-volume-claim-pvc) for storage, otherwise the data will be lost.
 
 ## Resume a Pipeline
 

docs/user-guide/sinks/user-defined-sinks.md

@@ -18,7 +18,7 @@ spec:
 
 ## Available Environment Variables
 
-Some environment variables are available in the user defined sink Pods:
+Some environment variables are available in the user defined sink container:
 
 - `NUMAFLOW_NAMESPACE` - Namespace.
 - `NUMAFLOW_POD` - Pod name.

docs/user-guide/sources/transformer/overview.md

@@ -85,7 +85,7 @@ spec:
 
 ### Available Environment Variables
 
-Some environment variables are available in the source vertex Pods, they might be useful in you own source data transformer implementation.
+Some environment variables are available in the source transformer container, they might be useful in you own source data transformer implementation.
 
 - `NUMAFLOW_NAMESPACE` - Namespace.
 - `NUMAFLOW_POD` - Pod name.

docs/user-guide/user-defined-functions/map/map.md

@@ -48,6 +48,7 @@ spec:
 ```
 
 ### Streaming Mode
+
 In cases the map function generates more than one outputs (e.g. flat map), the UDF can be
 configured to run in a streaming mode instead of batching which is the default mode.
 In streaming mode, the messages will be pushed to the downstream vertices once generated
@@ -57,8 +58,9 @@ instead of in a batch at the end. The streaming mode can be enabled by setting t
 Note that to maintain data orderliness, we restrict the read batch size to be `1`.
 
 ```yaml
-...
-    - name:  my-vertex
+spec:
+  vertices:
+    - name: my-vertex
       metadata:
         annotations:
           numaflow.numaproj.io/map-stream: "true"
@@ -70,10 +72,9 @@ Check the links below to see the UDF examples in streaming mode for different la
 - [Golang](https://github.com/numaproj/numaflow-go/tree/main/pkg/function/examples/flatmap_stream)
 - [Java](https://github.com/numaproj/numaflow-java/tree/main/examples/src/main/java/io/numaproj/numaflow/examples/function/map/flatmapstream)
 
-
 ### Available Environment Variables
 
-Some environment variables are available in the user defined function Pods, they might be useful in you own UDF implementation.
+Some environment variables are available in the user defined function container, they might be useful in you own UDF implementation.
 
 - `NUMAFLOW_NAMESPACE` - Namespace.
 - `NUMAFLOW_POD` - Pod name.
@@ -85,8 +86,8 @@ Some environment variables are available in the user defined function Pods, they
 
 Configuration data can be provided to the UDF container at runtime multiple ways.
 
-* [`environment variables`](../../reference/configuration/environment-variables.md)
-* `args`
-* `command`
-* [`volumes`](../../reference/configuration/volumes.md)
-* [`init containers`](../../reference/configuration/init-containers.md)
+- [`environment variables`](../../reference/configuration/environment-variables.md)
+- `args`
+- `command`
+- [`volumes`](../../reference/configuration/volumes.md)
+- [`init containers`](../../reference/configuration/init-containers.md)

mkdocs.yml

@@ -85,6 +85,7 @@ nav:
           - user-guide/reference/edge-tuning.md
           - user-guide/reference/autoscaling.md
           - user-guide/reference/conditional-forwarding.md
+          - user-guide/reference/pipeline-operations.md
           - user-guide/reference/join-vertex.md
           - user-guide/reference/multi-partition.md
           - user-guide/reference/side-inputs.md
@@ -96,7 +97,7 @@ nav:
             - user-guide/reference/configuration/init-containers.md
             - user-guide/reference/configuration/sidecar-containers.md
             - user-guide/reference/configuration/pipeline-customization.md
-            - user-guide/reference/configuration/pipeline-operations.md
+            - user-guide/reference/configuration/istio.md
             - user-guide/reference/configuration/max-message-size.md
           - user-guide/reference/kustomize/kustomize.md
           - APIs.md