OLD Extending JUnit's Standard Behaviour

There are numerous aspects of JUnit's core behavior that could be extensible via a first-class plugin or extension model.

Please note that there is considerable overlap between this page and API for Running and Reporting Tests.

Test Method Invocation Wrapping

Wrapping single tests or groups of tests with some set-up and/or tear-down, like the ExternalResource rule in JUnit 4.

Test Collection

Have a different way of collecting tests (like in ClasspathSuite).

• If we do this, please have a standard annotation to specify that a test class is just a collection of other tests (maybe @ExplicitSuite). If I have a SmokeTests source file in my package that selects tests annotated with @Smoke and someone tells their IDE to run all tests in the package, the tests included in SmokeTests msut not be run twice (once individually, and once as part of the suite). - @kcooney

Test File Format

Parse a different file format to specify tests (eg. Fit tests, cucumber tests).

• Gherkin specification support would also be nice to have - @mlschechter

Extension Points to Replace Runners

Provide extension points that enable behavior that currently requires a runner, thereby completely eliminating the need for custom runners. - @kcooney & @sbrannen

• parameterization
• ordering/sorting
• filtering
• dependency injection

1. test instance field injection
2. test method argument resolution
3. possibly support dependency injection and parameterization simply with extension points that allow the extension to instantiate the test class or post-process the test instance. The latter is analogous to the prepareTestInstance() callback in Spring's TestExecutionListener.

• This is a "must have" feature. - @sbrannen
• rule-like objects that act on the class level and the method level
• This is a "must have" feature. - @sbrannen
• I think this is where the intent is also to address the current limitation that execution/behavior/reporting currently provided via a runner cannot be aggregated; for example, adding Spring contexts to tests using JUnitParams. - @mlschechter
  • I am not convinced that finding a way to aggregate runners is the best solution; I am not even sure it is possible. IMHO too much behavior is buried in ParentRunner, and more focused extension points that provide better separation of concerns could make writing a custom runner unnecessary in most cases (including Spring). - @kcooney
  • I agree with your assessment of ParentRunner; I was looking for confirmation that this would be the place where we could add requirements that would clarify that the extension mechanism would be capable of what the above use case describes. - @mlschechter
• test generation
• Programmatically add test instances from generator classes. Making generators would allow deferred tests registration. - @ttddyy
• feature auto detection (mainly for third party library)
  • In spring framework, it has a factory loading mechanism(SpringFactoriesLoader) which detects spring.factories file from classpath resources. External jar libraries can provide spring.factories file in its jar file for spring to pick up. Spring detects it and plugs library logic into its lifecycle. For example, spring-boot uses this mechanism for autoconfiguration, actuator, etc. Also spring test-context-framework uses it for TestExecutionListener detection. This may be too aggressive but would be a good extension point especially for third party libraries. - @ttddyy

Lifecycle Events

Allowing an extension to have a long-lived service that stops when it is notified that the test run is ending.

First-class Support for Integration Tests

Provide first-class support for executing integration tests in contrast to JUnit's historical focus on unit tests (i.e., test instance state is not shared across test method invocations).

• better support for integration tests - kcooney
  • possibly can be done by third parties if we added support for dependency injection and class/test-run life cycle events

Conditional Test Discovery and Execution

What if the discovery and execution of individual tests were conditional at run-time, with subsequent tests executed or ignored based on programmatic logic instead of a predetermined list of test methods per test class?

What if the decision about what next test to instantiate and execute could be based on factors such as the outcome of the previous test?

• One thing that holds back junit-quickcheck from full-fledged QuickCheck-ness is its lack of shrinking. When a theory fails on a given set of inputs, a shrinking QuickCheck would continue to execute the theory with progressively "smaller" inputs (whatever an input's "size" means, based on its type) until the theory passes or until the inputs can get no smaller. Then, for that theory you have a (hopefully) much smaller counterexample to diagnose your code with.
• It would be great if the thing that executes junit-quickcheck theories could have the opportunity to inject a new execution of a failing theory with smaller inputs into the execution "plan" of the JUnit run.