Add support for workflow level timeouts

[fg91]

Currently timeouts can only be configured on the task level (see docs). In many situations it would be useful to have the ability to restrict the runtime of the entire workflow.

[davidmirror-ops]

From Contributors meetup: upvoted by @eapolinario. Further discussion with @hamersaw is recommended before we decide if it should be an RFC/Issue or how to move forward.

[eapolinario]

This is the other issue I mentioned: #5125

[hamersaw]

Strictly adding workflow timeouts should be pretty easy, it can be implemented the same as node-level timeouts. If this is the scope, I don't think there is any need for an RFC. Just need to add configuration and support in the workflow executor. Happy to include some code references if anyone is willing to pick this up.

[eapolinario]

That's great, @hamersaw . Can you clarify what's taken into account in node-level timeouts? For example, do timeouts only account for the time the node is in the running state, right?

[hamersaw]

There are technically two possible timeouts. 'active' would be from when the node starts, 'running' is the runtime of the current attempt. We store timestamps in the CR relating to the first time propeller sees the node, and when the last attempt started running. For workflows we would probably use a single configuration value for the time since the CR was created?

[davidmirror-ops]

@pvditt @fg91 do you think we should create an Issue out of this including some code pointers for contributors to pick it up?