Update TCK Process to specify that Jakarta EE Platform implementations only need to pass TCK tests that can be run against a Platform (e.g. EE containers) implementation to be considered Jakarta EE Platform compatible.

[scottmarlow]

Describe the bug
As per #1679 (comment) feedback, I think the (Jakarta EE) TCK Process requirements should be updated to not require Standalone SE TCKs to include Platform tests:

Jakarta EE Platform implementations only need to pass TCK tests that can be run against a Platform (e.g. EE containers) implementation to be considered Jakarta EE Platform compatible. Jakarta EE Platform implementations do not need to run TCK tests that do not run against a Platform implementation.

Expected behavior

The current Jakarta EE development process involves completing (included) Specification releases in a series of waves that are respectful of the needed compilation dependencies.

If we follow the new TCK Process changes with the EE 11 development process, each Specification wave TCK needs to include testing on a EE 11 Platform implementation which cannot happen as waves 1-6 will definitely not have an EE 11 Platform implementation to test against.

I recommend that we either correct the TCK Process or revert the #1679 change so that Standalone Specifications can proceed through the ballot process without having an EE 11 Platform implementation to test against.

Correcting the TCK Process is not difficult, the hard thing for me is that my feedback seemed to be ignored on the #1679 which perhaps is due to me not being clear enough with my feedback.

[Emily-Jiang]

@scottmarlow I don't quite understand your concern here. The updated process is to prevent from only SE style tcks available per spec. It does not say that all TCKs must pass to release a spec. The integration tests are normally in platform tck suite and it will run at the later stage. There is nothing new here. Take CDI as an example, it has integration tests with EJB etc. As long as that tests pass, the spec can be released.

[scottmarlow]

@scottmarlow I don't quite understand your concern here. The updated process is to prevent from only SE style TCKs available per spec.

A specific example is the https://download.eclipse.org/jakartaee/persistence/3.1/jakarta-persistence-tck-3.1.0.zip SE style TCK which as per the 1.4 TCK Process change is now required to be changed to run against standalone or in a platform (or profile).

The problem is the or which from a Specification ballot perspective, both conditions must be tested which is currently not possible.

There are other problems as well such as the Persistence Provider development teams (e.g. Hibernate ORM, EclipseLink, OpenJPA, others) should be able to continue to only run the SE style TCK. The same is true for all of the other SE style TCKs.

It does not say that all TCKs must pass to release a spec. The integration tests are normally in platform tck suite and it will run at the later stage.

The Specification ballot process has traditionally ensured that relevant Java SE style TCKs are passed before the specification ballot is complete. The TCK Process 1.4 changed that by introducing Platform/profile TCK tests into Standalone Specifications like Persistence and others.

There is nothing new here. Take CDI as an example, it has integration tests with EJB etc. As long as that tests pass, the spec can be released.

Agreed that the CDI TCK is an example of including Platform/Profile TCK tests and the teams have always tested that as needed but CDI wasn't ever required to include Platform/Profile TCK tests. Another example is the https://download.eclipse.org/jakartaee/batch/2.1/jakarta.batch.official.tck-2.1.1.zip TCK which includes some Platform/Profile tests which the Batch Specification team also tests as needed during the initial Batch Specification Ballot process but IMO we should not require the same of the other Standalone Specifications as that would hurt the development process that those other Standalone Specifications have always followed (which we should be respectful of).

What I mean about being respectful of the Standalone Specification development process, I mean that we should appreciate what the few people that contribute to those Specifications do and how they do it.

[Emily-Jiang]

@scottmarlow my comments are inline.

@scottmarlow I don't quite understand your concern here. The updated process is to prevent from only SE style TCKs available per spec.

A specific example is the https://download.eclipse.org/jakartaee/persistence/3.1/jakarta-persistence-tck-3.1.0.zip SE style TCK which as per the 1.4 TCK Process change is now required to be changed to run against standalone or in a platform (or profile).

The problem is the or which from a Specification ballot perspective, both conditions must be tested which is currently not possible.

The or does not imply both.

There are other problems as well such as the Persistence Provider development teams (e.g. Hibernate ORM, EclipseLink, OpenJPA, others) should be able to continue to only run the SE style TCK. The same is true for all of the other SE style TCKs.

It does not say that all TCKs must pass to release a spec. The integration tests are normally in platform tck suite and it will run at the later stage.

The Specification ballot process has traditionally ensured that relevant Java SE style TCKs are passed before the specification ballot is complete. The TCK Process 1.4 changed that by introducing Platform/profile TCK tests into Standalone Specifications like Persistence and others.

The TCK process just states that Type 2 and Type 3 must exist but it does not mandate the tests must be run against a profile. The statement "The type 3 tests must be executed in compatible implementations of the profiles to validate the individual specification implementation functions properly in the profile implementation: platform, web profile or core profile." is for certifying profile implementations. There is no mention about individual spec certification. If the new persistence only contains Java SE style tests, we want to prevent this from happening. It is a correct showstopper for it to pass the ballot.

There is nothing new here. Take CDI as an example, it has integration tests with EJB etc. As long as that tests pass, the spec can be released.

Agreed that the CDI TCK is an example of including Platform/Profile TCK tests and the teams have always tested that as needed but CDI wasn't ever required to include Platform/Profile TCK tests. Another example is the https://download.eclipse.org/jakartaee/batch/2.1/jakarta.batch.official.tck-2.1.1.zip TCK which includes some Platform/Profile tests which the Batch Specification team also tests as needed during the initial Batch Specification Ballot process but IMO we should not require the same of the other Standalone Specifications as that would hurt the development process that those other Standalone Specifications have always followed (which we should be respectful of).

What I mean about being respectful of the Standalone Specification development process, I mean that we should appreciate what the few people that contribute to those Specifications do and how they do it.

I understand the situation here, but I don't think we should push out a spec without any proof that runtimes can run the TCKs to demonstrate they do pass the TCKs.

[lukasj]

The updated process is to prevent from only SE style tcks available per spec.

jsonp testing is better example than persistence, in terms of specific test, think ie about following test:

  @Test
  public void jsonNumber21Test() {
      assertEquals(Json.createValue(1), Json.createValue(Byte.valueOf((byte) 1)));
  }

Is this generic and purely jsonp API specific thing independent on the environment something that - as of current wording - has to be run by everyone under all possible container/profile configurations? What if the language gets updated to say sth like:

if provider P passes this test then P is certified
if provider P is certified AND container C finds/can use provider P then container C is certified

where provider P passes this test means running a test using a test runner and/or test configuration spec & platform teams agrees on, ie use JUnit test runner in classpath mode
and container C finds/can use provider P means P is available in Servlet, EJB, ... or through JPMS should the spec team require support for JPMS. In the other words, it could be said that TCK certification process sees SE/classpath and SE/modulepath as two different "containers" where the SE/modulepath is not required to be supported by the implementations of standalone specifications included in the implementation of the platform (... or something along those lines)

From standalone spec point of view, this would require the spec team to produce "generic tests" running on Java SE in classpath mode (container is Java SE restricted to classpath + other limitations agreed upon should there be any). Should the spec want, it can add specific tests for loading the provider on JPMS (+some other specific functionality), or have more "container" specific tests which would be by default skipped during platform container/profile run.

...not sure what I wrote makes sense, but the idea is to

• remove some redundancy in test runs, ie AFACT the test above either passes everywhere or fails everywhere, so why to not run it just once
• allow standalone specs adoption of SE features which the platform sees as out of scope and give them an option to certify implementations willing to opt-in supporting these features, ie distinguish Jakarta EE Platform certified and Jakarta EE certified (?)
• platform & spec team share common ground for testing common, profile-independent functionality to avoid duplicity and redundancy in running tests

[Emily-Jiang]

The updated process is to prevent from only SE style tcks available per spec.

jsonp testing is better example than persistence, in terms of specific test, think ie about following test:

  @Test
  public void jsonNumber21Test() {
      assertEquals(Json.createValue(1), Json.createValue(Byte.valueOf((byte) 1)));
  }

Is this generic and purely jsonp API specific thing independent on the environment something that - as of current wording - has to be run by everyone under all possible container/profile configurations? What if the language gets updated to say sth like:

if provider P passes this test then P is certified if provider P is certified AND container C finds/can use provider P then container C is certified

where provider P passes this test means running a test using a test runner and/or test configuration spec & platform teams agrees on, ie use JUnit test runner in classpath mode and container C finds/can use provider P means P is available in Servlet, EJB, ... or through JPMS should the spec team require support for JPMS. In the other words, it could be said that TCK certification process sees SE/classpath and SE/modulepath as two different "containers" where the SE/modulepath is not required to be supported by the implementations of standalone specifications included in the implementation of the platform (... or something along those lines)

From standalone spec point of view, this would require the spec team to produce "generic tests" running on Java SE in classpath mode (container is Java SE restricted to classpath + other limitations agreed upon should there be any). Should the spec want, it can add specific tests for loading the provider on JPMS (+some other specific functionality), or have more "container" specific tests which would be by default skipped during platform container/profile run.

...not sure what I wrote makes sense, but the idea is to

* remove some redundancy in test runs, ie AFACT the test above either passes everywhere or fails everywhere, so why to not run it just once

If the tests are Junit tests running in SE mode, it is not feasible to run it in the container mode. These tests are Type 1 and they are not runnable in the container mode.
I think Jakarta EE is more focused on EE not SE. Therefore, it is key to have integration tests to certify the spec behaves as speced.

* allow standalone specs adoption of SE features which the platform sees as out of scope and give them an option to certify implementations willing to opt-in supporting these features, ie distinguish _Jakarta EE Platform certified_ and _Jakarta EE certified_ (?)

Java SE only tests are not required to be executed by Jakarta EE certified impls anyway.

* platform & spec team share common ground for testing common, profile-independent functionality to avoid duplicity and redundancy in running tests

I don't think there is any redundancy here since one test is only executed once by a given runtime. I would rather my runtime to execute all TCK tests to give my customer reassurance rather than worrying about the tests have been run elsewhere.

[scottmarlow]

I don't think there is any redundancy here since one test is only executed once by a given runtime. I would rather my runtime to execute all TCK tests to give my customer reassurance rather than worrying about the tests have been run elsewhere.

I mentioned a few times already that others gave (Jakarta EE Platform mailing list) feedback that the only Jakarta EE Platform/Profile TCK testing that is worthwhile, is when the tests are deployed to an EE container/profile.

I think a change to what was added in the TCK Process 1.4 document is needed that allows:

1. Jakarta EE Platform/Profile implementations to only be required to run TCK tests that deploy to the Jakarta EE Platform/Profile implementation.
2. Java SE Specifications are required to provide TCK tests that run on Java SE. If they want to also provide EE mode tests, that is also welcome but not required.
3. The Platform TCK contributors (which includes everyone) may add Jakarta EE Platform/Profile tests that complement the (Java SE style Specification) Java SE only TCK tests such that the tests are deployed on Jakarta EE Platform/Profile implementations.

IMO, #1 covers your customer needs.

[Emily-Jiang]

I don't think there is any redundancy here since one test is only executed once by a given runtime. I would rather my runtime to execute all TCK tests to give my customer reassurance rather than worrying about the tests have been run elsewhere.

I mentioned a few times already that others gave (Jakarta EE Platform mailing list) feedback that the only Jakarta EE Platform/Profile TCK testing that is worthwhile, is when the tests are deployed to an EE container/profile.

I think a change to what was added in the TCK Process 1.4 document is needed that allows:

1. Jakarta EE Platform/Profile implementations to only be required to run TCK tests that deploy to the Jakarta EE Platform/Profile implementation.

2. Java SE Specifications are required to provide TCK tests that run on Java SE.  If they want to also provide EE mode tests, that is also welcome but not required.

I disagree with this. If the spec has just SE tests, how can you ensure the integration was done correctly. You can pull in the Weld to a runtime without integrating it. Can you just claim that runtime certifies against CDI 4.0. You need to run some tests to tell the users that the integration was done correctly. A runtime could disable some behavior and continue claiming the certification. I don't think so.

3. The Platform TCK contributors (which includes everyone) may add Jakarta EE Platform/Profile tests that complement the (Java SE style Specification) Java SE only TCK tests such that the tests are deployed on Jakarta EE Platform/Profile implementations.

Disagree here. Ideally, most TCKs can be executed in the platform/profile such as CDI.

IMO, #1 covers your customer needs.

Can you please bring your use case here and state what stops you from releasing a spec?

[lukasj]

If the tests are Junit tests running in SE mode, it is not feasible to run it in the container mode. These tests are Type 1 and they are not runnable in the container mode. I think Jakarta EE is more focused on EE not SE. Therefore, it is key to have integration tests to certify the spec behaves as speced.

My point is about what the test tests, not about how the test itself is implemented nor which test framework does it use. It is also unclear to me, what does Java SE mode mean. Is it different from existing standalone mode? If so, then how? Or does it just extend it? Is there any consensus across the group on what rules are applicable to each execution mode, ie which Java SE features test execution in given mode may or may not use? What got to me on the standalone spec end over past few years was only sort of informal "you can't test X because the platform does not want to require it".

* allow standalone specs adoption of SE features which the platform sees as out of scope and give them an option to certify implementations willing to opt-in supporting these features, ie distinguish _Jakarta EE Platform certified_ and _Jakarta EE certified_ (?)

Java SE only tests are not required to be executed by Jakarta EE certified impls anyway.

How are Java SE only tests defined? Is it that each and every JUnit test is Java SE only test and each and every JavaTest test is a container test? Or is Java SE only tests every test which does not require deployment?

* platform & spec team share common ground for testing common, profile-independent functionality to avoid duplicity and redundancy in running tests

I don't think there is any redundancy here since one test is only executed once by a given runtime.

Quite often it is that one test is executed once in each container provided by the runtime, even if there is no reason to do so nor to run the test through deployed artifact from functional point of view. For this case it may be good idea to have a trusted execution mode which runs execution-mode independent tests only, so container or SE-only tests can do just what's important for them.

I would rather my runtime to execute all TCK tests to give my customer reassurance rather than worrying about the tests have been run elsewhere.

So what you want is the ability to execute standalone spec-defined tests without tests for Java SE-only scoped functionality as part of EE Platform TCK run, correct?

[scottmarlow]

@scottmarlow my comments are inline.

@scottmarlow I don't quite understand your concern here. The updated process is to prevent from only SE style TCKs available per spec.

A specific example is the https://download.eclipse.org/jakartaee/persistence/3.1/jakarta-persistence-tck-3.1.0.zip SE style TCK which as per the 1.4 TCK Process change is now required to be changed to run against standalone or in a platform (or profile).
The problem is the or which from a Specification ballot perspective, both conditions must be tested which is currently not possible.

The or does not imply both.

That is the problem though as the (Type 3) or expression states that the Standalone (component) Specification must provide tests that can run against Standalone or Platform/Profile TCK tests but how would the Spec ballot mentor coordinate that as an or, which side of the or would they require for the initial Spec ballot and what does that mean for the Platform/Profile tests that likely would not be tested?

[Emily-Jiang]

@scottmarlow my comments are inline.

@scottmarlow I don't quite understand your concern here. The updated process is to prevent from only SE style TCKs available per spec.

A specific example is the https://download.eclipse.org/jakartaee/persistence/3.1/jakarta-persistence-tck-3.1.0.zip SE style TCK which as per the 1.4 TCK Process change is now required to be changed to run against standalone or in a platform (or profile).
The problem is the or which from a Specification ballot perspective, both conditions must be tested which is currently not possible.

The or does not imply both.

That is the problem though as the (Type 3) or expression states that the Standalone (component) Specification must provide tests that can run against Standalone or Platform/Profile TCK tests but how would the Spec ballot mentor coordinate that as an or, which side of the or would they require for the initial Spec ballot and what does that mean for the Platform/Profile tests that likely would not be tested?

For component spec release, It works if one of the sides is tested. For Platform/Profile release, they must pass all Platform or Profile tcks.

[scottmarlow]

Can you please bring your use case here and state what stops you from releasing a spec?

As an example, assume that we want to release Persistence Spec which doesn't even have their own TCK yet. How would the SPEC Ballot mentor know if the Persistence TCK tests are Type 3 TCK or not.

The SPEC Ballot mentor might also ask if the Persistence Spec plans to have a Type 1 TCK which must be excluded and not required to be executed as only Type 3 tests are allowed.

Exactly what is the SPEC Mentor going to do with the Persistence SPEC Ballot? Wait until the Persistence Spec has a TCK that can be run against EE 11 implementations or Persistence Providers?

[Emily-Jiang]

Can you please bring your use case here and state what stops you from releasing a spec?

As an example, assume that we want to release Persistence Spec which doesn't even have their own TCK yet. How would the SPEC Ballot mentor know if the Persistence TCK tests are Type 3 TCK or not.

The SPEC Ballot mentor might also ask if the Persistence Spec plans to have a Type 1 TCK which must be excluded and not required to be executed as only Type 3 tests are allowed.

Exactly what is the SPEC Mentor going to do with the Persistence SPEC Ballot? Wait until the Persistence Spec has a TCK that can be run against EE 11 implementations or Persistence Providers?

You won't be able to release the spec without a TCK. How can you be sure the spec is correctly written? This is a no-go scenario.

[scottmarlow]

Can you please bring your use case here and state what stops you from releasing a spec?

As an example, assume that we want to release Persistence Spec which doesn't even have their own TCK yet. How would the SPEC Ballot mentor know if the Persistence TCK tests are Type 3 TCK or not.
The SPEC Ballot mentor might also ask if the Persistence Spec plans to have a Type 1 TCK which must be excluded and not required to be executed as only Type 3 tests are allowed.
Exactly what is the SPEC Mentor going to do with the Persistence SPEC Ballot? Wait until the Persistence Spec has a TCK that can be run against EE 11 implementations or Persistence Providers?

You won't be able to release the spec without a TCK. How can you be sure the spec is correctly written? This is a no-go scenario.

By default the Standalone TCK will be required and produced by the Platform TCK same as always except it will be refactored but still the current plan is for the Standalone Persistence TCK (produced by Platform TCK) to only be able to run in Java SE mode (e.g. type 1).

The Platform TCK will run the Persistence TCK tests against the Platform/profile but still it sounds like the TCK Process needs to change as its not clear how the Spec Ballot Mentor will deal with the fact that the Persistence Spec has only passed a type 1 TCK.

[Emily-Jiang]

Can you please bring your use case here and state what stops you from releasing a spec?

As an example, assume that we want to release Persistence Spec which doesn't even have their own TCK yet. How would the SPEC Ballot mentor know if the Persistence TCK tests are Type 3 TCK or not.
The SPEC Ballot mentor might also ask if the Persistence Spec plans to have a Type 1 TCK which must be excluded and not required to be executed as only Type 3 tests are allowed.
Exactly what is the SPEC Mentor going to do with the Persistence SPEC Ballot? Wait until the Persistence Spec has a TCK that can be run against EE 11 implementations or Persistence Providers?

You won't be able to release the spec without a TCK. How can you be sure the spec is correctly written? This is a no-go scenario.

By default the Standalone TCK will be required and produced by the Platform TCK same as always except it will be refactored but still the current plan is for the Standalone Persistence TCK (produced by Platform TCK) to only be able to run in Java SE mode (e.g. type 1).

The Platform TCK will run the Persistence TCK tests against the Platform/profile but still it sounds like the TCK Process needs to change as its not clear how the Spec Ballot Mentor will deal with the fact that the Persistence Spec has only passed a type 1 TCK.

In order for Jakarta Persistence to be included in any Profile or Platform release, type 2 or 3 TCKs must be provided. One way to achieve this is to rework Type 1 to Type 2 or 3. If the TCKs use the Arquillian framework, these TCKs are automatically Type 3 TCKs. The current TCK process is to prevent from Jakarta EE specifications only producing Type 1 only TCKs as runtimes have no way to ensure everything integrated correctly.
Please free to contribute towards the TCK process improvement @scottmarlow .

[scottmarlow]

In order for Jakarta Persistence to be included in any Profile or Platform release, type 2 or 3 TCKs must be provided.

The Platform TCK will have the Persistence TCK tests that are either type 2 or type 3. However, the Persistence Specification may have Type 1 tests (in EE 11 or maybe EE 12) that can be extended by the Platform TCK (e.g. such that the Platform TCK is adjusted in future EE releases for changes in tests that the Platform TCK extends).

The current TCK process is to prevent from Jakarta EE specifications only producing Type 1 only TCKs as runtimes have no way to ensure everything integrated correctly.

One alternative is in the title of this issue Update TCK Process to specify that Jakarta EE Platform implementations only need to pass TCK tests that can be run against a Platform (e.g. EE containers) implementation to be considered Jakarta EE Platform compatible. We can update the title to refer to Profile or Platform.

Please free to contribute towards the TCK process improvement @scottmarlow .

+100