Is a drastic reduction in number of tests from vulkan-cts-1.3.7.3 to vulkan-cts-1.3.8.0 expected and intentional?

[billhollings]

vulkan-cts-1.3.8.0 seems to have only about two-thirds the number of tests that vulkan-cts-1.3.7.3 has.

From a recent comparison running tests on MoltenVK:

dEQP Core vulkan-cts-1.3.7.3-0-gd71a36db16d98313c431829432a136dbda692a08 (0xd71a36db) starting..
Test run totals:
  Passed:        404332/4006625 (10.1%)
  Failed:        11140/4006625 (0.3%)
  Not supported: 3591092/4006625 (89.6%)
  Warnings:      61/4006625 (0.0%)
  Waived:        0/4006625 (0.0%)

dEQP Core vulkan-cts-1.3.8.0-0-gd13582ab1e753a41b2cedf8e3c504e1cb92edfe5 (0xd13582ab) starting..
Test run totals:
  Passed:        392275/2702883 (14.5%)
  Failed:        9919/2702883 (0.4%)
  Not supported: 2300634/2702883 (85.1%)
  Warnings:      55/2702883 (0.0%)
  Waived:        0/2702883 (0.0%)

We seem to have an overall reduction of 1.3 M tests.

I found these comments in the vulkan-cts-1.3.8.0 release notes:

Remove duplicate pipeline protected access tests

Which seems to account for about 320 K tests.

Simplify BorderSwizzle TestNode

Which seems to account for about 280 K tests.

Reduce the amount of variations in VK_EXT_host_image_copy tests

Which seems to account for about 108 K tests.

Remove duplicate shader object protected image tests

Which seems to account for about 861 K tests.

There seems to be a large reduction in dEQP-VK.pipeline.*.extended_dynamic_state.* tests, but I can't see a clear pattern to those removed tests, nor do a see a clear note about this in the release notes.

These seem to account for about 145 K tests.

But this is all assuming that the test reductions are driven by these notes. I would have expected such a large change in overall number to be mentioned in the release notes, since the trend has always been to increase the test count from release to release.

So, just to confirm, is this overall drastic reduction correct, and intentional? On a full run, should I expect "only" 2.7M tests? Or is it possible that I have somehow pulled or built CTS in a manner that has lost a large number of tests?

[AlexGalazin-IMG]

Sounds about right. In addition to what you listed
1fa0038 removed about 1.09m (1088703) tests
faf4fbb removed about 139k (138866) tests

[billhollings]

Sounds about right. In addition to what you listed 1fa0038 removed about 1.09m (1088703) tests faf4fbb removed about 139k (138866) tests

Ah. Great. Good work in consolidating!