e2e test to confirm no data loss

[juliev0]

Summary

This can either be a separate test or be incorporated into our existing test, but it should only be for the case of PPND strategy (i.e. not our current DATA_LOSS_PREVENTION=false case).

Our test can perform a topology change going from something like A->B->C to A->C in which the B vertex is slow (so to make sure there's no data loss, we ensure B is drained fully).

What type of Source should we use for A? We can consider http, in which case we need to send to it, or some event driven source such as Kafka.

I created a manual test for this that can be modified as needed. It involves a curl script which sends to an http source, in which the payload is an auto-incrementing number. I think we can improve upon what I did. I can discuss.

Use Cases

When would you use this?

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.