How do I use nested vcpkg manifests in submodules/sub-projects?

[mikepricedev]

├── CMakeLists.txt
├── vcpkg.json
├── vcpkg_installed
├── extern
│   ├── CMakeLists.txt
│   ├── sub_project
│   │   ├── CMakeLists.txt
│   │   ├── vcpkg.json
│   │   ├── vcpkg_installed

In the above folder structure the "sub_project" submodule uses vcpkg and a vcpkg.json manifest to manage its deps. The root project also uses vcpkg and a vcpkg.json manifest to manage deps.

However, when the project is configured the sub_project's CMakeLists.txt vcpkg wiring no longer finds packages installed in sub_project/vcpkg_installed dir.
Steps taken:

1. vcpkg install ran in sub_project dir. vcpkg_installed dir was created as expected.
2. PATHS option added to find_package in submodule cmake. (in my case using the frozen lib)

find_package(frozen REQUIRED
  PATHS "${CMAKE_CURRENT_SOURCE_DIR}/vcpkg_installed/${VCPKG_TARGET_TRIPLET}"
)

3. Step 2 finds the library (frozen), but when compiling generates: 'frozen/map.h' file not found #include <frozen/map.h> not an issue in sub_project standalone.
4. The following correctly prints the include dirs.

get_target_property(frozeninc frozen::frozen INTERFACE_INCLUDE_DIRECTORIES)
message(STATUS "frozen INTERFACE_INCLUDE_DIRECTORIES: ${frozeninc}")

Note: using vcpkg as a submodule technique in both i.e.

set(
  CMAKE_TOOLCHAIN_FILE "${CMAKE_CURRENT_SOURCE_DIR}/extern/vcpkg/scripts/buildsystems/vcpkg.cmake"
  CACHE STRING "Vcpkg toolchain file")

[NancyLi1013]

Hi @mikepricedev

Thanks for posting this issue.

You might forget to add target_link_libraries(main PRIVATE frozen::frozen frozen::frozen-headers) in your CMakeLists.txt.

Could you please refer to the usage of frozen to rebuild your project?

 find_package(frozen CONFIG REQUIRED)
 target_link_libraries(main PRIVATE frozen::frozen frozen::frozen-headers)

[NancyLi1013]

@mikepricedev

Please let me know if the above suggestion can work for you.

[mikepricedev]

@NancyLi1013
Thank you so much for taking a look. Before I go further I realized I am making an assumption that I haven't found any specific documentation. Does vcpkg support project's using vcpkg.json that have submodules that also use vcpkg.json? I'm using the following technique to add my submodules to my root project's CMakeLists.txt.

add_subdirectory(extern/sub_project)

In my case, Frozen is a dep of the sub_project managed in the sub_project by a vcpkg.json manifest. When I build the sub_project project independently (not as a submodule) the #include <frozen/map.h> is not an issue.

It is only when the sub_project is built as a submodule of another project using vcpkg. Both projects have vcpkg.json mainfest files. If I hoist the dependencies from the sub_project and paste them into the root project's vcpkg.json dependencies . There is no issue.

I also noticed something else particular. The sub_project also has a dep to "boost-smart-ptr". When I build the sub_project independenly boost is found in the vcpkg_installed dir. When built as a submodule the dep is found from my system's copy of "boost". I do not have a system copy of "frozen".

[PhoebeHui]

@mikepricedev, does this still be a question for you?

From manifests.md, when invoked from any subdirectory of the directory containing vcpkg.json, vcpkg install with no package arguments will install all manifest dependencies into /vcpkg_installed/.

[cyraid]

I'm running into this problem too.. But I'm relying on visual studio 2019 cmake support to use vcpkg. It seems to be ignoring the sub directory/sub directory.

Is there support for add_subdirectory and sub directories with their own vcpkg.json? (And just using the main project without opening separate visual studio 2019 instances with their own cmakelists.txt).

[strega-nil]

@cyraid there is currently no support for having multiple top level manifests. We have considered some ways to solve this, but none of them very good. If you're interested in solving this issue, please open a discussion topic about it. Otherwise, writing a top-level vcpkg.json that contains all of the dependencies your subdirectories need, or alternatively using vcpkg to manage these dependencies as well, are the two solutions we see right now.

[cyraid]

Hi @strega-nil ! Thank you for the reply! Where would you guys like to discuss this? I mean, I don't have much time to talk about it but my quick input would be to add a ADD_TARGET_VCPKG in the sub directory inside that CMakeLists.txt. So, ADD_LIBRARY(MyPlugin), and then ADD_TARGET_VCPKG(MyPlugin). This way you could add the appropriate includes and properties etc. for that target. This would of course also call vcpkg in that directory too, but perhaps with the same cache directory and out from CMakeSettings.json?

[strega-nil]

@cyraid https://github.com/microsoft/vcpkg/discussions

[kentf]

We'd also like to have nested vcpkg.json support.

We have many projects in a small monorepo. Some of the projects require some larger dependencies (OpenCV, Qt), while some only requires header only libraries.

The projects can be added/excluded from the solution by CMake option toggles. It would be nice to only acquire/build, say Qt and OpenCV, if the projects that uses those dependencies are being configured and built.

[ras0219-msft]

You can use features for this: https://github.com/microsoft/vcpkg/blob/master/docs/users/manifests.md#vcpkg_manifest_features

[ras0219-msft]

Vcpkg doesn't support separate vcpkg.json files in this manner; instead, you'll need to add the inner dependencies to the outer vcpkg.json file. For CMake, only the vcpkg.json adjacent to the top-level CMakeLists.txt is considered.

[ras0219-msft]

To be crisp about what's unsupported: vcpkg does not support multiple vcpkg.json files consumed by the same CMakeLists.txt source tree. This can be resolved today either by:

1. If there is only one dependency needing vcpkg, simply copy the vcpkg.json from the dependency up to the root CMakeLists.txt and then proceed with the standard vcpkg integration.
2. If there are multiple dependencies needing vcpkg, merge their dependency arrays together into a parent vcpkg.json. This is a simple concatenation assuming they are not using more complicated features like overrides (in which case the user needs to thoughtfully choose which version they actually want to use)
3. Instead of consuming the dependencies as submodules/add_subdirectory(), consume them as vcpkg packages. This will get you binary caching and the full suite of vcpkg dependency management functions. You can use either overlay ports or custom registries to do this.

As for the philosophy behind it: it is not trivial in the general case to merge manifests together. We are not saying it will never be supported, but rather that we have not yet worked through all the design implications.

[sketch34]

This is still an issue, just bumping for visibility and any news.

Specifically, I just encountered this issue again with a top-level non-vcpkg project consuming a submodule that uses vcpkg. It seems like the workarounds are required, although option (3) is not possible in my current case.

[petersteneteg]

At some point I wrote custom cmake "toolchain" file similar to the vcpkg/scripts/buildsystems/vcpkg.cmake one.
It will traverse my current build setup (a fairly complex plugin based with with lots of optional parts) and look for "sub" vcpkg.json files. It will then go ahead and merge those and with the top level one write that in the build dir. Then I set the VCPKG_MANIFEST_DIR to the build dir and call into the regular vcpkg toolchain.

I have some limitation on what can happen in the "sub" vcpkg.json files to make it easier to merge. but generally it seems to work quite well.

[derekcyruschow-catapult]

@petersteneteg that sounds great! Are you will to share your toolchain file please?

[petersteneteg]

@derekcyruschow-catapult it is quite specific to our workflow, but you might get some inspiration https://github.com/inviwo/inviwo/blob/71f4dd278595367a84d1c1cb8317fe9113becb4d/cmake/vcpkg-manifest-generation-toolchain.cmake

[laurentcau]

CMakePresets.json uses an include section to solve this issue.

{
        "version": 4,
	"include": [
		"Public/SDK/CMakePresets.json"
	  ]
}

Could that be an option to solve the issue for vcpkg.json too ?

[JasonDictos]

Was just musing over how to solve this as well, we can hijack the fact that vcpkg ignores keys with $ in their name, then with this bash script and a top level "$include": "" value in the vcpkg manifest, this script will install each dependency:

#!/bin/bash

MANIFEST=$1

function pop() {
  popd > /dev/null
}
trap pop EXIT

if [[ ! -f $MANIFEST ]]; then
  echo "usage: ports.sh <manifest.json>"
  exit 1
fi

MANIFEST=$(realpath "$MANIFEST")

readarray -t DEPS < <(cat $MANIFEST | jq -r ".dependencies" | tr -d '[],:{}" ' | sed '/^$/d')

pushd /tmp > /dev/null

for item in ${DEPS[@]}; do
  vcpkg install $item
done

INCLUDE=$(cat $MANIFEST | jq -r '."$include"')
if [[ -f "$INCLUDE" ]]; then
  echo "Found include $INCLUDE"
  $0 "$INCLUDE"
fi                  

My vcpkg.json this works on:

{
  "name": "sfio",
  "version-string": "1.0.0",
  "$include": "libeen-cpp/vcpkg.json",
  "dependencies": [
    "liburing4cpp",
    "docopt",
    "cli11",
    "boost-lockfree",
    "boost-interprocess",
    "boost-icl",
    "boost-iostreams",
    "boost-crc",
    "boost-accumulators",
    "prometheus-cpp",
    "mio",
    "libmeazure"
  ]
}

Doesn't support vcpkg features or what not, but could be modified to...