cleanup job for running containers

[christophprokop]

atm if a longer job is running there is no possibility to stop the running containers
CTRL-C does not stop the currently running containers/jobs but just gives you back the command prompt but the jobs keep running
maybe introduce some kind of cleanup job?
regular use case here: you want to add a minor change and rerun

[ammernico]

Took a look into this. Currently, when pressing CTRL+C the process gets terminated with a SIGINT signal by Linux.
But we can use the ctrlc crate or the signal feature from tokio to handle the signal ourself.

The nicest way to clean up containers would probably be implementing drop on the structure which holds the job with the container id.

I already tried this and ran into issues with how tokio currently handles panics. When receiving the CTRL+C signal and trying to unwinding the program with std::process::ExitCode or calling down panic, just the current thread panics and other tokio threads continue to run.

This is wanted by tokio but there are open issues which try to implement custom behaviur for how to handle panics in threads.

tokio-rs/tokio#2002
tokio-rs/tokio#4516

The feature can already be used in tokio unstable. But the unstable version brings many other changes/issues and I would recommend waiting for it to become stable.

https://docs.rs/tokio/1.40.0/tokio/runtime/struct.Builder.html#method.unhandled_panic

[primeos-work]

Nice catch. The current code use a bit of a fire-and-forget approach where the containers get started but then run to completion with no way to stop them (IIRC) - butido only streams the logs, updates the current status, saves artifacts, etc.

My idea with @ammernico was to use the RAII pattern to ensure that all containers that a butido-build command starts do also get stopped.

Unfortunately it wasn't quite that easy as Ctrl+c aborts the program so we need to implement a graceful shutdown. Ideally we can shutdown the Tokio runtime but if that doesn't work we probably have to implement that part ourselves (i.e., we should at least check if Ctrl-c was pressed during the butido-build part where the containers are running).

Some "random" notes:

• https://docs.rs/tokio/latest/tokio/signal/fn.ctrl_c.html
• drop(runtime)?
• Alternatively https://docs.rs/tokio/latest/tokio/runtime/struct.Runtime.html#method.shutdown_background, because the drop likely isn't allowed)
• https://docs.rs/tokio/latest/tokio/runtime/struct.Runtime.html#method.shutdown_timeout also an option
• Optional: Force-stop if the user presses Ctrl-c again (+ ideally resource leak warning due to the running containers)
• We probably have to replace the tokio::main attribute macro to get access to the runtime: https://docs.rs/tokio/latest/tokio/attr.main.html
  • A handle probably isn't enough: https://docs.rs/tokio/0.2.25/tokio/runtime/struct.Handle.html#method.current
• Alternatives
  • https://docs.rs/tokio-util/latest/tokio_util/sync/struct.CancellationToken.html (but seems overkill as it requires lots of changes to the current code)
  • https://docs.rs/tokio-graceful-shutdown/latest/tokio_graceful_shutdown/