Broken plugin verification and builds for partial versions of IDE

[Pakisan]

What happened?

Plugin verification is totally broken

configuration:

    pluginVerification {
        failureLevel = listOf(
            VerifyPluginTask.FailureLevel.INVALID_PLUGIN,
            VerifyPluginTask.FailureLevel.COMPATIBILITY_PROBLEMS,
            VerifyPluginTask.FailureLevel.NOT_DYNAMIC,
            VerifyPluginTask.FailureLevel.COMPATIBILITY_WARNINGS,
        )

        ides {
            ides(listOf(
                "2022.3",
                "2022.3.1",
                "2022.3.2",
                "2022.3.3",
                "2023.1",
                "2023.1.1",
                "2023.1.2",
                "2023.1.3",
                "2023.1.4",
                "2023.1.5",
                "2023.2",
                "2023.2.1",
                "2023.2.2",
                "2023.2.3",
                "2023.2.4",
                "2023.2.5",
                "2023.3",
                "2023.3.1",
                "2023.3.2",
                "2023.3.3",
                "2023.3.4",
                "2023.3.5",
                "2023.3.6",
                "2023.3.7",
                "2024.1",
                "2024.1.1",
                "2024.1.2",
                "2024.1.3",
                "2024.1.4",
                "2024.1.5",
                "2024.1.6",
                "2024.2",
                "2024.2.0.1",
                "2024.2.0.2"
            ))
        }
    }

execution:

./gradlew :verifyPlugin

execution result:

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-223.8617.56: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-223.7571.182: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-223.8836.41: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-231.9011.34: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-223.8214.52: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-231.8109.175: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-231.8770.65: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-242.20224.419: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-232.8660.185: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-231.9392.1: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-231.9225.16: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-231.9161.38: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-232.9559.62: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-232.10072.27: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-232.9921.47: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-233.11799.241: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-233.13135.103: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-232.10203.10: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-232.10227.8: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-233.11799.300: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-233.14475.28: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-233.14015.106: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-233.14808.21: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-241.15989.150: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-241.14494.240: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-233.15026.9: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-233.15325.26: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-241.17011.79: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-241.17890.1: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-241.18034.62: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-241.18968.26: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-241.19072.14: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-242.20224.300: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.6.0+jre17 against IC-242.20224.387: Compatible
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart
[verifyPlugin full.txt](https://github.com/user-attachments/files/16799650/verifyPlugin.full.txt)

Incompatible build

Based on plugin verification report, looks like I can publish it without worries, but in fact is not true

Fact 1

Checking plugin inside 2024.1.6 via :runIde, was successful

While after installation of fresh downloaded 2024.1.6 with installation of plugin from disks, reproduces incompatibility

Fact 2

Verification report is not same, as verification results on plugin page

Fact 3

Looks like new version of Gradle plugin broke compatibility, because previous version build with v1 was ok

Fact 4

Changing id("org.jetbrains.intellij.platform") version "2.0.1" to id("org.jetbrains.intellij") version "1.17.4" builds compatible
build with old gradle plugin.txt
build with new gradle plugin.txt
plugin

Relevant log output or stack trace

No response

Steps to reproduce

Clone repo https://github.com/asyncapi/jasyncapi-idea-plugin, switch to /release/2.6.0

Choose IDEA version 2024.1.6

• run ./gradlew :verifyPlugin
• run ./gradlew :runIde

Inside created IDE everything is working smoothly

Download 2024.1.6 and install plugin manually, check IDEA log file

2024-08-29 16:49:47,014 [   9229] SEVERE - #c.i.c.c.CompletionProgressIndicator - Companion
java.lang.NoSuchFieldError: Companion
...

Gradle IntelliJ Plugin version

2.0.1

Gradle version

8.10

Operating System

None

Link to build, i.e. failing GitHub Action job

No response

[YannCebron]

Regarding specifying every single bugfix update: this is (usually) not necessary. Bugfix updates are backwards-compatible by design.

    "2022.3",
                "2022.3.1",
                "2022.3.2",
                "2022.3.3",

In this case, checking against latest available bugfix version (2022.3.3) would be sufficient (and speedup this task considerably, apart from saving many GBs of space).

[YannCebron]

Regarding differences between local run and results from Marketplace: please check the local version of Plugin Verifier matches the one used on Marketplace.