[vcpkg-tool] Implement downloads from json file

[Thomas1664]

See microsoft/vcpkg#24199 (comment)

Slightly modified example after @autoantwort:

[
        {
            "provider": "gitlab",
            "repo": "microsoft/vscode",
            "ref": "v1.33.1",
            "sha256": "...",
            "head-ref": "master",
            "patches": [
                "1.patch",
                "2.patch"
            ]
        },
        {
            "provider": "archive",
            "cmake-out-var": "code",
            "url": "https://..._.zip",
            "sha256": "..."
        }
    ]

CMake functions to consider:

• vcpkg_from_bitbucket
• vcpkg_from_git
  • Additional arguments: LFS
• vcpkg_from_github
  • Additional arguments: FILE_DISAMBIGUATOR, AUTHORIZATION_TOKEN, GITHUB_HOST
• vcpkg_from_gitlab
  • Additional arguments: FILE_DISAMBIGUATOR, GITLAB_URL
• vcpkg_from_sourceforge
  • Additional arguments: WORKING_DIRECTORY, FILENAME, NO_REMOVE_ONE_LEVEL
• vcpkg_download_distfile
  • Additional arguments: HEADERS, ALWAYS_REDOWNLOAD, QUIET, URLS

[Neumann-A]

Probably needs https://github.com/orgs/open-vcpkg/projects/1?pane=issue&itemId=52127920 first.

Is it really necessary to implement this in the tool itself?

[autoantwort]

Is it really necessary to implement this in the tool itself?

Would be nicer for --only-downloads and then you can download and extract the sources before executing the portfile.cmake to lower installation times.

[Neumann-A]

Would be nicer for --only-downloads and then you can download and extract the sources before executing the portfile.cmake to lower installation times.

Ok let me ask the question in another way. How does the tool know what to download when the information is only given in the portfile or a file included by the portfile.

[autoantwort]

I would add fields to the vcpkg.json file if the tool gets responsible for downloading the sources.

[Neumann-A]

I would add fields to the vcpkg.json file if the tool gets responsible for downloading the sources.

I am speaking of the current situation and a plan to transition to a new format. So how does the tool currently determine what to downlad? As far as I am aware it runs portfile with special flags and that is it.

I also wouldn't merge that information into vcpkg.json but store it somewhere else. (i.e. downloads.json)

[Thomas1664]

I am speaking of the current situation and a plan to transition to a new format. So how does the tool currently determine what to downlad? As far as I am aware it runs portfile with special flags and that is it.

Old versions would still have this information in the portfile. We have to either parse the CMake file for that information (already done for SBOM) or fall back to the current mechanism.

Probably needs https://github.com/orgs/open-vcpkg/projects/1?pane=issue&itemId=52127920 first.

I think this is an alternative to https://github.com/orgs/open-vcpkg/projects/1?pane=issue&itemId=52127920

[Neumann-A]

The point is you always have to run both ways since the portfile is allowed to do anything. So even if you define something like donwload.json you still require the parsing logic. As such providing a buildin way is just an optimization for downloading stuff.

That being said I am not against it. I just don't see it being implemented in the tool first when you have to run both ways anyway. I just see it as a script port first.

But maybe we should define a json schema for it first:

"downloads": 
[
        "code_src" : {
            "type": "github",
            "repo": "microsoft/vscode",
            "ref": "v${VERSION}", -> make some "CMake/Manifest" variables expand in the json
            "sha256": "...", or sha512
            "head-ref": "master", (do we really need this`?)
            "patches": [
                "1.patch",
                "2.patch"
            ],
           ("remove" : [ "list of dirs/files to remove"])
        },
       "file" : {
            "type": "file",
            "url": "https://..._.zip",
            "sha256": "..."
            ("extraction-destination": "a_target_path_relative_within_buildtrees")
        }
    ] 

"downloads" -> Array of JSON "download" objects (or a relative filepath to be included?), The name of the object is the name of the corresponding CMake output variable and needs to be unique.
Does the array need a ordering sequence if it also gets extracted or does the patching step wait until everything is downloaded?

[Thomas1664]

The point is you always have to run both ways since the portfile is allowed to do anything. So even if you define something like donwload.json you still require the parsing logic. As such providing a buildin way is just an optimization for downloading stuff.

I think we need to restrict that. All functions that are executed in downloads mode could error out if the file download.json exists.

[Thomas1664]

Does anyone have an idea on how to integrate this feature into the rest of the build process? I think the most efficient way is after abi hashing. This allows us to parallelize the download process in the future. Also many other package managers download and extract all packages in one go before installing them.

[autoantwort]

I thing the best solution would be to create a task based system and use that for almost everything

[Thomas1664]

I thing the best solution would be to create a task based system and use that for almost everything

I like this idea. Maybe we want to use this library?

The problem is that things are cached all over the place and there are many connections between different components.