-
Notifications
You must be signed in to change notification settings - Fork 271
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support dependency verification #1779
Comments
Search https://github.com/gradle/gradle/issues?q=is%3Aissue+is%3Aopen+transform+constraint returns a couple of bugs, these 2 look somewhat similar but not really: |
The same exception gradle/gradle#10951 |
From the stacktrace, which I've provided we can see that the error originated from In the logs we see dozens of exceptions, because This only confirms my idea, that the cause is somewhere among the usages of
|
On the second issue, it may be that Gradle's mechanism for bootstrapping (generating, resolving) hashes of artifacts in Line 17 in 33af9bd
As I've outlined in the XML file, Gradle generates hashes only for Ivy XML file created from that custom IvyModule. But not from the artifact I tried to set <?xml version="1.0" encoding="UTF-8"?>
<verification-metadata xmlns="https://schema.gradle.org/dependency-verification" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://schema.gradle.org/dependency-verification https://schema.gradle.org/dependency-verification/dependency-verification-1.3.xsd">
<configuration>
<verify-metadata>true</verify-metadata>
<verify-signatures>false</verify-signatures>
<keyring-format>armored</keyring-format>
<key-servers>
<key-server uri="https://keyserver.ubuntu.com"/>
</key-servers>
</configuration>
<components>
<!-- This is a clipped file, there were many other components here. -->
<component group="bundledPlugin" name="Tomcat" version="2023.3.8+84">
<artifact name="ivy-2023.3.8+84.xml">
<md5 value="0315fbe6c213b753f79381b0e5a18b02" origin="Generated by Gradle"/>
<sha1 value="1a6e7dd7eaff3a3c9ed514b454a3a4ac0f985402" origin="Generated by Gradle"/>
<sha256 value="9197a957995f195ac2ff3e56cf1720e195d61401223577ce513653cb918388dd" origin="Generated by Gradle"/>
<sha512 value="bba0126c3d9251ac5bbe765b4daab6a467c54a38b8096130389d8af453a495f080543dff2168d22100fcfa11d51e5e21c7d24498b86e7e45a54899a4e2f8ad1e" origin="Generated by Gradle"/>
</artifact>
<!-- I think here should be another entry for that .directory, but it does not get generated. This is what the error is about. -->
</component>
<!-- See below examples of how other artifacts look like. -->
<component group="com.google.guava" name="guava" version="32.1.3-jre">
<artifact name="guava-32.1.3-jre.jar">
<md5 value="adc3cf557a48d15cb71be90948558923" origin="Generated by Gradle"/>
<sha1 value="0f306708742ce2bf0fb0901216183bc14073feae" origin="Generated by Gradle"/>
<sha256 value="6d4e2b5a118aab62e6e5e29d185a0224eed82c85c40ac3d33cf04a270c3b3744" origin="Generated by Gradle"/>
<sha512 value="f575321aa79046f2c12bdf9895db7e235cbaaa8e913389ae48ec4bb5f387d6bf066ed98c5f2c7854bf0c56bb38b59b005ca3c16d68e314743491a223a18cee47" origin="Generated by Gradle"/>
</artifact>
<artifact name="guava-32.1.3-jre.module">
<md5 value="e6794493e3bcab67b0c6b8448f4e92bf" origin="Generated by Gradle"/>
<sha1 value="4c853bdbbb6f44e3a1674fb4ca931e4720f158b1" origin="Generated by Gradle"/>
<sha256 value="f5fff7642c12e7627bc14289fd267e2602c17f9590e23522c3e63107f61c2942" origin="Generated by Gradle"/>
<sha512 value="8ce0d85f4466999263bea74b62f583a6dcd7008531a7a4c89f4387adbae605986a48735161d96ad1d329ef43dee15812baae4ec4cf5bb086efddaa0e2209b2aa" origin="Generated by Gradle"/>
</artifact>
</component>
<component group="com.google.guava" name="guava-parent" version="26.0-android">
<artifact name="guava-parent-26.0-android.pom">
<md5 value="cb15100f42049979dd296ae701e4ef4e" origin="Generated by Gradle"/>
<sha1 value="a2c0df489614352b7e8e503e274bd1dee5c42a64" origin="Generated by Gradle"/>
<sha256 value="f8698ab46ca996ce889c1afc8ca4f25eb8ac6b034dc898d4583742360016cc04" origin="Generated by Gradle"/>
<sha512 value="1d786f14fbfa5c90eedcc160d1e0a71acb2141f372049b22ce62b0bd1e883c17cc24a59dc8b00e5037e959cccdb54d4d8dc8f252302d4bb7ce82dfdaff764476" origin="Generated by Gradle"/>
</artifact>
</component>
<component group="com.jetbrains.intellij.idea" name="ideaIU" version="2023.3.8">
<artifact name="ideaIU-2023.3.8.pom">
<md5 value="23def2f158ab129eb5355e6fbe4997e2" origin="Generated by Gradle"/>
<sha1 value="95f09d04f4652c8b9a80c1f72baed43ba7be2d0f" origin="Generated by Gradle"/>
<sha256 value="713cf2aaf872406cc0d5c03a6985382a0c9819cc080ed8f4024411a99118ad0f" origin="Generated by Gradle"/>
<sha512 value="8d58edca55fc09f4bd8e49372e870ba912b6590a56dd5204db3c5d7f3d38d0dba74ea0deee35083a148f453789c38c07bd0f2246c43645a6a602c7952aeec93a" origin="Generated by Gradle"/>
</artifact>
<artifact name="ideaIU-2023.3.8.zip">
<md5 value="d90a44d9ce142e1bd45f74cbc811858e" origin="Generated by Gradle"/>
<sha1 value="7cb6ca200b78b921b30a70b5057b952b5bfebd2a" origin="Generated by Gradle"/>
<sha256 value="27b011801301c0e7dfb903b447a2dacd5803d9734e45a97b0f6c5947c57a5699" origin="Generated by Gradle"/>
<sha512 value="b4c3771c8155f5620be264867c6f93096d52b9c550d47371f687de2bf6062775a2353022ecdb8f3c0937c2716cb0522945f007fbd1c735ea768c2587698c1e2e" origin="Generated by Gradle"/>
</artifact>
</component>
</components>
</verification-metadata> |
Actually these two problems are the same issue. It turns out that the same error That is why hashes are missing for bundledPlugin dependencies in From Gradle sources and the stack trace it looks like it is using https://github.com/gradle/gradle/blob/81c7139e837079a44aa45244268771d978c9f452/subprojects/core/src/main/java/org/gradle/internal/hash/ChecksumHasher.java#L25 Which will not be able to hash a directory simply because of FileInputStream. It seems to me that in order to support dependency verification this plugin should specify a file (not clear which file if it a dir with jars) instead of a directory for custom artifacts like bundledPlugin. Probably the same is true for bundledModule but I have not tested this one. One option would be to structure those plugins in the next way:
That way Gradle will hash the ivy.xml & jars separately. It looks like right now this transformer does a similar thing -- collects jars from Ivy artifacts having only dirs in them. Line 39 in 33af9bd
|
For bundledModule artifacts hashes are generated correctly because they point to jars not dirs, at least in this case: <component group="bundledModule" name="intellij.java.unscramble" version="242.23339.11+552">
<artifact name="/home/sasha/.gradle/caches/8.10.2/transforms/5e431020a32f377d747dae792b1dcf82/transformed/ideaIU-242-EAP-SNAPSHOT/plugins/java/lib/modules/intellij.java.unscramble.jar-242.23339.11+552.jar">
<md5 value="65e1b87751a3be825fdbe9ec0c69d9c5" origin="Generated by Gradle"/>
<sha1 value="0ebaaca1276243d5057226cee9682c6478eadae4" origin="Generated by Gradle"/>
<sha256 value="c02c8b42882e911544c1310905d1c6f4418a2b06743d3c17dc9e138124f13337" origin="Generated by Gradle"/>
<sha512 value="4387fe6c5d359da2cb7f91ae9e473fb5ac70efcbe8a20a949684b72875d2bde98158822b79fede2ecdd2e9469a53724f5d9e03c76298f6265377294730e81bfe" origin="Generated by Gradle"/>
</artifact>
<artifact name="ivy-242.23339.11+552.xml">
<md5 value="dda50acdc8a64d89fe8ffa5a203f4872" origin="Generated by Gradle"/>
<sha1 value="52d59dec394da2cf85cb4c7644e3f8a6999144fe" origin="Generated by Gradle"/>
<sha256 value="599ab6979e994c82488b8c88a9d9450044f8d609a5a7ec26084ce1f7aadcc51c" origin="Generated by Gradle"/>
<sha512 value="40c8c096ad4146ba3305e6827fb0a320b5f1715478bd021ba18d68f6376423aa5b1227e369d50aec34ffdf8a68e597f27f438ed905d6798ae89f84051403ad42" origin="Generated by Gradle"/>
</artifact>
</component> But, if I enable dependency locking, I still see the error in the report. If locking is disabled, modules verification (but not bundled plugins) works fine.
We have the hash for
|
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
My PR seems to fix the issue, except metadata verification for because this plugin generates Ivy XML files on the fly and whenever ExtractorTransformer runs (See #1778) it creates a new absolute path to the unzipped IDE in the transforms cache, which them gts mentioned a newly generated Ivy XML files. Since the xml files have an absolute path mentioned, their (xml files) hashes will also be different on different envs, because of that
<ivy-module version="2.0">
<info organisation="com.jetbrains.localhost-only.bundledPlugin" module="Git4Idea" revision="242.23339.11"/>
<configurations>
<conf name="default" visibility="public"/>
</configurations>
<publications>
<artifact name="git4idea-rt.jar" type="home/sasha/.gradle/caches/8.10.2/transforms/28ecd40df1b8c2cf9e48725c9c2de2bb/transformed/ideaIU-242-EAP-SNAPSHOT/plugins/vcs-git/lib/" ext="jar" conf="default"/>
<artifact name="vcs-git.jar" type="home/sasha/.gradle/caches/8.10.2/transforms/28ecd40df1b8c2cf9e48725c9c2de2bb/transformed/ideaIU-242-EAP-SNAPSHOT/plugins/vcs-git/lib/" ext="jar" conf="default"/>
</publications>
<dependencies>
<dependency org="com.jetbrains.localhost-only.bundledPlugin" name="com.jetbrains.performancePlugin" rev="242.23339.11"/>
<dependency org="com.jetbrains.localhost-only.bundledPlugin" name="org.jetbrains.plugins.terminal" rev="242.23339.11"/>
<dependency org="com.jetbrains.localhost-only.bundledModule" name="intellij.platform.collaborationTools" rev="242.23339.11"/>
<dependency org="com.jetbrains.localhost-only.bundledModule" name="intellij.platform.ide.newUiOnboarding" rev="242.23339.11"/>
<dependency org="com.jetbrains.localhost-only.bundledModule" name="intellij.platform.coverage" rev="242.23339.11"/>
</dependencies>
</ivy-module> To fix this we probably should make the ExtractorTransformer (or create a new, preferably not, because we already have a problem with the extractor running too often) generate Ivy XML files directly in the unzipped IDE, so that it becomes like a local Ivy repository. In that case we probably can use relative paths, which will not change. This could also significantly improve performance due to Line 138 in 3d6e84e
Then here Line 71 in dbc44d9
Or here: Line 131 in dbc44d9
When the location of the transform path becomes available we can register a new Ivy repository (or update artifactPath of the existing one) for that location. But this will not work with |
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
The ideas from the previous comment were implemented in #1785 |
The only case where PGP signature and hash generation still fails is anything stored in |
…n. Previously it was failing with "Failed to create MD5 hash for file".
…n. Previously it was failing with "Failed to create MD5 hash for file".
What happened?
Gradle dependency verification does not work with this plugin.
Multiple issues may be at play (see #1779 (comment) for correction & clarification):
<verify-metadata>true</verify-metadata>
andpgp
option in CLI. It fails withFailed to create MD5 hash for file '/home/sasha/.gradle/caches/8.10.2/transforms/2f4561d46bc638866dfa4bcb9a564517/transformed/ideaIU-2023.3.8/plugins/JPA' as it does not exist.
with may or may not be a bug in Gradle.<verify-metadata>false</verify-metadata>
and nopgp
option in CLI. Hashes for the actual artifacts are missing for dependencies created by this plugin, likebundledPlugin:*
. Only hashes for ivy.xml are being generated by Gradle.Context information
I have a strong suspicion that the cause is somewhere among the usages of
org.jetbrains.intellij.platform.gradle.utils.UtilsKt#safelyCreatePlugin
.Now a little guide on how to use dependency verification:
gradle/verification-metadata.xml
gradle/verification-metadata.xml
to the state above.<verify-metadata>true</verify-metadata>
.<verify-signatures>false</verify-signatures>
ingradle/verification-metadata.xml
and running the next command:gradle/verification-metadata.xml
do not forget to revert it before the next run.<verify-signatures>true</verify-signatures>
ingradle/verification-metadata.xml
so do not forget to change it back because it will be creating errors even if you do not list pgp in the command line.gradle/verification-metadata.xml
do not forget to revert it before the next run.Relevant log output or stack trace
Below.
Steps to reproduce
Repo
https://github.com/AlexanderBartash/intellij-platform-plugin-template/tree/dep-verification-bug
Also see ignored tests in #1780
Without Signatures
First try to use it without pgp:
DO NOT change
gradle/verification-metadata.xml
manually!In the report for this failure I see:
And in the
verification-metadata.xml
file I see some possible discrepancies:Now if you add bundledModule & bundledPlugin to ignore rules it will work. But it defeats the purpose since now it won't verify intellij artifacts. It is just to demonstrate that it can work in general.
With Signatures
Revert
gradle/verification-metadata.xml
back to the original state.Try now with pgp.
With pgp added it is even worse than without. Because here even if you also add bundledModule & bundledPlugin to ignore rules it will NOT work, because this command is supposed to bootstrap a list of trusted keys, it is too big to create manually from scratch. And the below errors interrupt that process. It may be actually a bug in Gradle, because it seems like it confuses a dir for a file, or maybe the plugin just registers is wrong, or dirs are not supposed to be there at all.
Do not forget to revert
gradle/verification-metadata.xml
after using the pgp option because it changes<verify-signatures>false</verify-signatures>
totrue
.with-pgp.log
Gradle IntelliJ Plugin version
3.1.0
Gradle version
8.10.2
Operating System
Linux
Link to build, i.e. failing GitHub Action job
No response
The text was updated successfully, but these errors were encountered: