Running JUnit 3 test suite with JUnit Vintage engine with Gradle

[ulrikls]

Running a JUnit 3 test annotated to be run with the AllTests runner using JUnit 5 may not actually run the test.

Steps to reproduce

In our project we have quite a few JUnit 3 tests looking like:

@RunWith(AllTests.class)
public class JUnit3Test extends TestCase {

    public static TestSuite suite()
    {
        var suite = new TestSuite("JUnit 3 style test suite");
        suite.addTest(new JUnit3Test("JUnit 3 style test") {
            @Override
            protected void runTest() {
                testSomething();
            }
        });
        return suite;
    }

    public void testSomething() {
        ...
    }

    ...
}

Currently, we're executing them using JUnit 4, but I'm trying to move to JUnit 5. When I change to useJUnitPlatform() in the Gradle build file these tests are no longer executed.

There's a full working example in https://github.com/ulrikls/junit-vintage-alltests

Any of these changes will successfully execute the JUnit 3 test:

• Switching to JUnit 4 - i.e. useJUnit() instead of useJUnitPlatform().
• Changing the name of the test from JUnit 3 style test to testSomeLibraryMethodReturnsTrue.
• Removing the @RunWith(AllTests.class) annotation and the suite() method.

I don't know if this example is even supposed to work in JUnit 5, there doesn't seem to be a lot of documentation on running JUnit 3 tests with JUnit 5. But migrating all our JUnit 3 tests is also not trivial, as the tests currently do various stuff in the suite() method.

Context

• Used versions (Jupiter/Vintage/Platform): 5.8.2 / 4.13.2
• Build Tool/IDE: Gradle 7.4.2

[sbrannen]

The following all-in-one test class runs fine for me in the Eclipse IDE using JUnit 3, JUnit 4, and JUnit 5.

package example;

import org.junit.runner.RunWith;
import org.junit.runners.AllTests;

import junit.framework.TestCase;
import junit.framework.TestSuite;

@RunWith(AllTests.class)
public class JUnit3Test extends TestCase {

	public JUnit3Test(String name) {
		super(name);
	}

	public static TestSuite suite() {
		var suite = new TestSuite("JUnit 3 style test suite");
		suite.addTest(new JUnit3Test("JUnit 3 style test") {
			@Override
			protected void runTest() {
				testSomeLibraryMethodReturnsTrue();
			}
		});
		return suite;
	}

	public void testSomeLibraryMethodReturnsTrue() {
		Library classUnderTest = new Library();
		assertTrue("someLibraryMethod should return 'true'", classUnderTest.someLibraryMethod());
	}

	public static class Library {
		public boolean someLibraryMethod() {
			return true;
		}
	}

}

Thus, I suspect it is an issue with Gradle.

@marcphilipp, thoughts?

[marcphilipp]

Sorry for the very late reply. This issue seems to have fallen through the cracks. 😕
Thanks for keeping the reproducer up to date! 👍

Thus, I suspect it is an issue with Gradle.

I suspect so as well. I somewhat inelegantly verified the test is actually executed when run via Gradle by adding System.exit(42); to the test method body. As expected that causes the :lib:test task to fail "with non-zero exit value 42".

Changing the name of the test from JUnit 3 style test to testSomeLibraryMethodReturnsTrue.

I debugged your reproducer for a bit and this hints at the problem. The Vintage engine creates TestDescriptor instances for JUnit 3/4's Description objects. For the Description of the test (JUnit 3 style test(junit.JUnit3Test$1)) it tries to find a method called "JUnit 3 style test" in the junit.JUnit3Test$1 class. Since there's no such method, it uses a ClassSource as the TestSource of the TestDescriptor, indicating the test is somewhere in that class. Gradle, however, checks whether the TestSource is a ClassSource to determine which of its internal TestDescriptorInternal objects to create. In this case it creates a DefaultTestClassDescriptor which is treated like an empty test class and thus not reported further down the line. Please raise an issue with Gradle since the Vintage engine is not at fault here.