MonoVertex to bypass ISB for high TPS large payload use cases

[vigith]

Summary

Today, we have to have at least a Source and a Sink. This means we need to write the message from the Source vertex to the ISB so that the Sink vertex can pick it up. For simple processing needs, one could avoid having the ISB altogether and implement Source and Sink in the same Vertex. SourcerSinker cannot support exactly-once semantics if the source does not support it because the Ack will be done after the message has been written on Sink, and we can hit the 2-generals problem.

Use Cases

If you are okay with the at-least-once semantics and want to avoid the cost of writing to ISB when the processing function is cheap to execute.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Tasks

Beta Give feedback

1. feat: Rust k8s model for Numaflow #1898
   +0
2. MonoVertex CRD #1903
   area/controller +1
   
3. exiting is not clean on ctrl-c
4. implement source-sink that bypasses ISB #1901
   6 of 6
   enhancement rust +2
   

   
5. rename sourcer-sinker to mono-vertex
6. feat: autoscaling for MonoVertex #1927
   +0
7. redo metrics using prometheus-rust client #1925
   area/mono-vertex rust +2
   
8. retry on error for sink (user requested retries) #1932
   area/mono-vertex rust +2
9. publish numaflow-rs crate numaflow-rs#77
   +0
   
10. Daemon Server for MonoVertex #1926
    2 of 2
    area/mono-vertex +1
    
11. UI to display MonoVertex #1923
    +0
    
12. move to scratch image from distroless #1924
    opex +1
    
13. Check if all build flags will be populated at runtime for MonoVertex version
14. enable versioning check for rust in numaflow #2033
    2 of 2
    +0
    
15. copy proto files at build time
16. test: add mono vertex e2e tests #1945
    +0
17. MonoVertex Restarts on missing pending information #1929
    area/mono-vertex bug +2
18. performance analysis of MonoVertex #1936
    area/mono-vertex opex performance rust +4
19. ack only after all messages are successfully written to sink #1937
    area/mono-vertex rust +2
    
20. handle drop messages from Transformer #1940
    area/mono-vertex rust +2
    
21. improve build time for Rust compilation #1943
    opex rust +2
    
22. Support MonoVertex for basic CRUD numaplane#193
    crud +1
    
23. Fallback sink support for MonoVertex #1950
    area/mono-vertex rust +2
    
24. Health Status for MonoVertex #1952
    area/mono-vertex +1
    
25. MonoVertex gets autoscaled to zero replicas when the pending is zero and processing rate is not zero #1960
    area/mono-vertex bug +2
26. Seeing pod count as zero in the UI for mono vertex when the actual pod count is one #1959
    area/mono-vertex bug +2
    
27. Document MonoVertex #2090
    documentation +1
    
28. -ve test cases
29. code coverage is missing for "rust" dir #1986
    rust testing +2
    
30. changing scale is causing pod recreation #1988
    area/controller area/mono-vertex bug +3
31. log better latency and processing metrics with histograms
32. chore: disallow MonoVertex replicas less than min or greater than max #1994
    +0
33. support pause feature for monovertex #2005
    area/controller enhancement +2
    
34. metrics for latency per container (source, transformer, sink)
35. metrics.rs has no tests #2038
    area/mono-vertex rust testing +3
    
36. Reorganize Rust Code #2098
    enhancement +1