Run all tests with user-specified runner

[perazz]

Fix #1041

@urbanjost what do you think?

[perazz]

Summary of updated behavior:

fpm run --example     -> do not run - dump available names
fpm run               -> do not run - dump available names
fpm test              -> *run all*
fpm run --example '*' -> run all
fpm test '*'          -> run all
fpm run '*'           -> run all
fpm run '*' --runner 'time'           -> run all
fpm run --example '*' --runner 'time' -> run all 
fpm test '*' --runner 'time'          -> run all

cc: @gnikit @henilp105 @fortran-lang/fpm @jacobwilliams @urbanjost

[perazz]

@jacobwilliams @lockstockandbarrel there are no outstanding comments. So I will wait for one or two days, and then merge.

[gnikit]

Summary of updated behavior:

fpm run --example     -> do not run - dump available names
fpm run               -> do not run - dump available names
fpm test              -> *run all*
fpm run --example '*' -> run all
fpm test '*'          -> run all
fpm run '*'           -> run all
fpm run '*' --runner 'time'           -> run all
fpm run --example '*' --runner 'time' -> run all 
fpm test '*' --runner 'time'          -> run all

@perazz is this list of commands and actions up to date?

[perazz]

@gnikit exactly

[gnikit]

Ok cool. Then I have some thoughts on this, about name expectations and ease of use. I'll try and post them later today.

[gnikit]

The only thing that doesn't sit right with me is that

• fpm run
• fpm run --example and
• fpm test

describe actions running and testing, but when one does fpm run they actually get a list. IMHO this is inconsistent with what a user would expect.
For me the behaviour of test and run (with/out --examples) should be consistent.

Command	Result
fpm run	Run all apps
fpm run --examples	Run all examples
fpm test	Run all tests

Usage of --all / '*' should also run all, although the --all option is somewhat redundant now (I would still keep it).
Usage of --runner should not affect the target selection, e.g. fpm run and fpm run --runner 'time' should both run all targets
To get a list of the available targets one should do: fpm run --list, fpm run --example --list or fpm test --list.

These are my 2cents @perazz and @urbanjost let me know what you think.

[perazz]

The former logic was: you wanted to run all tests (simple programs) by default to ease CI runs, while if you have many apps or examples, in general you should know what you're trying to run (they should be more complete programs etc.)

However, I also like the consistency approach.

[gnikit]

LGTM, nice work @perazz

[urbanjost]

Some thoughts and features I recall being debated in the past .. should running all be a consistent default? running by globbing is supported. Should regex be allowed too? should running by name allow globbing per name? Currently (I think) globbing is a single expression (need to confirm, not on platform a moment) running by list of names listing available choices gives short names with --list; but --runner with no command gives long ger names. Is there some better choice or is that OK? launching via runner command should pick same targets as without --runner option all messages going to stderr so, for example, a list can be piped into another command should be expected behavior complementing that shouldl be a quiet mode so only program output or listing is produced option to pause between programs deciding if a program termination should stop execution of a list of programs deciding if compilation failure (of perhaps a platform-specific code) should stop compilation and loading of other applications getting a tally of program exit codes, particularly for test (allowing it to act as a unit test framework) allowing specifying other directories to auto-build and auto-run by specifying a directory like -dir test|app|example|anything_I_want or if a name contains a slash or backslash should examples be built via --example or as a name like test? That is, why "fpm test" but "fpm --example"? Is --all now redundant? It is clearest what the expected behavior is. IMO I like the consistency and function of all; particularly since the current behavior is to run if only one program found, list if there are choices. It is not bad but I think the default being to run all is better. Out of the things that came to mind it is possible to get fancy with --runner and create a pause between programs but I think I would still like a switch specifically for that; and --example being changed to simply example is probably a big enough change that there would be calls for backward compatibility; but something like "fpm mydir/" or fpm name(s) --dir mydir would be nice.I have codes just for measuring timing and performance I do not always want to run in my test directory, and so I use globbing and a pattern of names currently and it would be nice to just be able to type "fpm test" and just run the unit and regression tests; and fpm timing/ to just build/run programs in the timing/ directory, for example. Just food for thought. I like a change towards consistency so I am good with the change; not sure if it will considered disruptive or not.

…

On 06/19/2024 5:26 AM EDT gnikit ***@***.***> wrote: Ok cool. Then I have some thoughts on this, about name expectations and ease of use. I'll try and post them later today. — Reply to this email directly, view it on GitHub #1046 (comment), or unsubscribe https://github.com/notifications/unsubscribe-auth/AHDWN3L5LW6NMZQB7Q6HMV3ZIFFFRAVCNFSM6AAAAABI5OS6XOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZYGIYDAMJWGM. You are receiving this because you were mentioned.Message ID: ***@***.***>

[perazz]

@urbanjost thanks, there's a lot of ideas here. I will merge this PR as they deserve separate investigation.

I like these two the most:

running by globbing is supported. Should regex be allowed too?

getting a tally of program exit codes, particularly for test (allowing it to act as a unit test framework)