Fix building docs from subprojects by not using CMAKE_SOURCE_DIR

[chewi]

The generate_documentation function currently uses CMAKE_SOURCE_DIR to find documentation assets at the SDK top-level, but when building from a subproject like sdk/core/azure-core, the variable points to that directory instead.

Fix this by defining SDK_ROOT_DIR, which is based on PROJECT_SOURCE_DIR. This should always work as long as each subproject calling the function is always 3 levels down, which is currently the case.

I've tested this against the whole SDK and when building 5 of the subprojects separately.

Pull Request Checklist

Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:

See the detailed list in the contributing guide.

• C++ Guidelines -- N/A
• Doxygen docs
• Unit tests -- N/A
• No unwanted commits/changes
• Descriptive title/description
  • PR is single purpose
  • Related issue listed
• Comments in source
• No typos
• Update changelog -- N/A
• Not work-in-progress
• External references or docs updated
• Self review of PR done
• Any breaking changes?

[github-actions]

Thank you for your contribution @chewi! We will review the pull request and get back to you soon.

[ahsonkhan]

@chewi thank you for making this change.

but when building from a subproject like sdk/core/azure-core, the variable points to that directory instead

Can you explain why the variable pointing to the SDK top-level is an issue in that case?
What happens here, which is different from what you expect?

[chewi]

but when building from a subproject like sdk/core/azure-core, the variable points to that directory instead

Can you explain why the variable pointing to the SDK top-level is an issue in that case? What happens here, which is different from what you expect?

Sorry, I didn't word that well. When building from sdk/core/azure-core, it's pointing at sdk/core/azure-core, which isn't where eng/docs/api/assets/header.html etc is located. I guess you want me to say that in the commit. 🙂

[LarryOsterman]

but when building from a subproject like sdk/core/azure-core, the variable points to that directory instead

Can you explain why the variable pointing to the SDK top-level is an issue in that case? What happens here, which is different from what you expect?

Sorry, I didn't word that well. When building from sdk/core/azure-core, it's pointing at sdk/core/azure-core, which isn't where eng/docs/api/assets/header.html etc is located. I guess you want me to say that in the commit. 🙂

In general, I'm in favor if this change, but I'm curious about the need to build documentation for individual subprojects in the subproject itself.

Generally, our documentation is generated at package publish time, and when our publishing pipelines run, they build the repo from the root and then extract the documentation for each package. What is the scenario for building documentation from the package specific directory?

[chewi]

I'm a Flatcar Linux developer at work (for MS) and a Gentoo Linux developer at home. Flatcar is based on Gentoo. Flatcar needs to add a package that uses some of the SDK, and it would good for others to benefit, so I have been packaging those parts of the SDK for Gentoo. Once I'd figured out the best approach, it turned out to be fairly simple. I would prefer to keep them as separate packages so that users don't need to pull in parts of the SDK that they don't need. I also wasn't sure how I would version a package for the whole SDK given that the components are released and versioned separately. It is good practise in Gentoo packages to allow the API documentation to be optionally installed, hence my desire for this to work from the subprojects.

I am aware that the API and ABI may not be particularly stable. I note the lack of SOVERSIONs. I have no immediate plans to add any consumers beyond this one package though, and if any follow later, it will probably be few enough for me to fix up any compatibility issues.

I'll get those changes to you tomorrow.

[ahsonkhan]

Thanks @chewi

When building from sdk/core/azure-core, it's pointing at sdk/core/azure-core, which isn't where eng/docs/api/assets/header.html etc is located

I would like to understand what you experience when building the docs for the subproject, to better understand the scope of impact here.

1. How exactly are you building sdk/core/azure-core and its docs to experience the issue you are describing? Can you share the few repro steps/commands you are running (and from where), so I can try it myself.
2. The main thing I want to understand is what happens because the various docs assets like header.html aren't where they are expected to be. Do the generated doxygen docs not render properly, or do they fail to build, or is it something else entirely?

[ahsonkhan]

LGTM
Thanks!

[LarryOsterman]

I'm a Flatcar Linux developer at work (for MS) and a Gentoo Linux developer at home. Flatcar is based on Gentoo. Flatcar needs to add a package that uses some of the SDK, and it would good for others to benefit, so I have been packaging those parts of the SDK for Gentoo. Once I'd figured out the best approach, it turned out to be fairly simple. I would prefer to keep them as separate packages so that users don't need to pull in parts of the SDK that they don't need. I also wasn't sure how I would version a package for the whole SDK given that the components are released and versioned separately. It is good practise in Gentoo packages to allow the API documentation to be optionally installed, hence my desire for this to work from the subprojects.

I am aware that the API and ABI may not be particularly stable. I note the lack of SOVERSIONs. I have no immediate plans to add any consumers beyond this one package though, and if any follow later, it will probably be few enough for me to fix up any compatibility issues.

I'll get those changes to you tomorrow.

Is here a reason you cannot use the vcpkg package manager on gentoo, which already contains the azure sdk for C++?

[LarryOsterman]

I'm a Flatcar Linux developer at work (for MS) and a Gentoo Linux developer at home. Flatcar is based on Gentoo. Flatcar needs to add a package that uses some of the SDK, and it would good for others to benefit, so I have been packaging those parts of the SDK for Gentoo. Once I'd figured out the best approach, it turned out to be fairly simple. I would prefer to keep them as separate packages so that users don't need to pull in parts of the SDK that they don't need. I also wasn't sure how I would version a package for the whole SDK given that the components are released and versioned separately. It is good practise in Gentoo packages to allow the API documentation to be optionally installed, hence my desire for this to work from the subprojects.
I am aware that the API and ABI may not be particularly stable. I note the lack of SOVERSIONs. I have no immediate plans to add any consumers beyond this one package though, and if any follow later, it will probably be few enough for me to fix up any compatibility issues.
I'll get those changes to you tomorrow.

Is here a reason you cannot use the vcpkg package manager on gentoo, which already contains the azure sdk for C++?

Oh, and fwiw, the API for published packages should be relatively stable at the source level. This is not true for types in the _internal or _detail namespaces - there are absolutely no stability guarantees on those. But from one minor version to another, clients of the SDK should just need to be recompiled with the new SDK without changing source code[1].

The ABI is as stable as any C++ ABI is, which means absolutely not stable - there is essentially no mechanism for delivering C++ binaries that is not either (a) distributed by source or (b) actually C code. The number of variables that need to be held stable to distribute C++ binaries is rather staggering (the C++ ABI can change from one compiler version to another in the same compiler, similarly, compiler command line switches can dramatically change the code generation).

So attempts to distribute the SDK as a binary package are almost certainly going to have significant challenges.

[1] This is not strictly true - for example, we recently deprecated a static member that was (to our knowledge) unused, and the use of that member could cause significant misbehavior in an application that wasn't prepared to deal with the consequences of that behavior.

[chewi]

Is here a reason you cannot use the vcpkg package manager on gentoo, which already contains the azure sdk for C++?

I'm not sure which way you're asking that.

I am a package maintainer trying to package software that uses the SDK. I cannot ask end users to use a different package manager to install the dependencies of that package.

If you mean I could use vcpkg within our Gentoo packages, then that is not feasible because regular Gentoo packages are not supposed to fetch artifacts "live". They are supposed to come from static files (usually tarballs) that are fetchable using a regular HTTP client. They can then be built and installed offline later. Call us old-fashioned, but this is what our users want.

It's also not clear whether you meant using vcpkg to fetch sources or binaries. I gather it can do both. If you meant binaries, Gentoo is a source-based distribution where the end users build packages from source. Although we reluctantly use third party binaries where necessary, that wouldn't be in the spirit of the distribution. Some other distributions have much stricter rules. Thankfully building with CMake turned out to be very easy. 🙂

Thank you for the information regarding stability. That is good news because our package manager has a feature that can automatically rebuild consumers of a library when it has changed "subslot" (a label we assign to indicate an ABI version). For most libraries, this is infrequent, but I can base it on the library's full version number so that consumers always rebuild.

[chewi]

1. How exactly are you building sdk/core/azure-core and its docs to experience the issue you are describing? Can you share the few repro steps/commands you are running (and from where), so I can try it myself.
2. The main thing I want to understand is what happens because the various docs assets like header.html aren't where they are expected to be. Do the generated doxygen docs not render properly, or do they fail to build, or is it something else entirely?

They fail to build. This is a slight simplification of what we do. It would happen with just make too, but this makes it clearer.

$ cd sdk/identity/azure-identity
$ AZURE_SDK_DISABLE_AUTO_VCPKG=yes cmake -DBUILD_DOCUMENTATION=yes -DWARNINGS_AS_ERRORS=no .
…
…
$ make azure-identity-docs
[100%] Generate documentation for azure-identity with Doxygen Version 1.12.0
error: tag HTML_HEADER: header file '/tmp/azure-sdk-for-cpp-6b9e1cc69187ec71b4c9a652c89ec927b3953ccf/sdk/identity/azure-identity/eng/docs/api/assets/header.html' does not exist
Exiting...
make[3]: *** [CMakeFiles/azure-identity-docs.dir/build.make:72: CMakeFiles/azure-identity-docs] Error 1
make[2]: *** [CMakeFiles/Makefile2:112: CMakeFiles/azure-identity-docs.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:119: CMakeFiles/azure-identity-docs.dir/rule] Error 2
make: *** [Makefile:182: azure-identity-docs] Error 2

[LarryOsterman]

Is here a reason you cannot use the vcpkg package manager on gentoo, which already contains the azure sdk for C++?

I'm not sure which way you're asking that.

Mostly it's because I'm not familiar with the process you're dealing with.

And in this case, I was suggesting that you use vcpkg to retrieve the Azure SDK packages you want to consume (and as I mentioned, for C++, source is really the only reliable option), build those packages in your build pipelines, and then publish the built packages.

That allows you to use vcpkg's dependency management to avoid some of the many challenges associated with building the C++ SDK from the repo while allowing you to be able to repackage the built binaries for gentoo. The only negative on that solution is that you don't really have the ability to build documentation as a part of that process, because we publish our documentation separately as a part of our official release process.

Using vcpkg also means that you have a measure of stability w.r.t. the release status of our packages - cloning our repo means you get all the packages in the repo at whatever state they are in, and at a given point of time, the repo may have some packages that are in release, some packages which are in beta, and some packages that are not ready for any kind of public consumption. An alternative is to clone the repo at the appropriate label for each of the packages which you want to consume.

[chewi]

Using vcpkg also means that you have a measure of stability w.r.t. the release status of our packages - cloning our repo means you get all the packages in the repo at whatever state they are in, and at a given point of time, the repo may have some packages that are in release, some packages which are in beta, and some packages that are not ready for any kind of public consumption. An alternative is to clone the repo at the appropriate label for each of the packages which you want to consume.

Since I'm packaging the components separately, they will follow your release versioning and use the source tarballs from those specific commits, so not to worry there.

Since you're curious, this is what one of our packages looks like. This file would be called dev-cpp/azure-identity/azure-identity-1.9.0.ebuild. Normally we would fetch the tarball from a URL based on that version number, but I noticed that some components share the same commit when they are released (e.g. azure-storage-common 12.7.0 and azure-storage-blobs 12.12.0), so I figured it made sense for these packages to share tarballs where appropriate.

# Copyright 1999-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=8

inherit cmake

COMMIT="d151ac9691dc40a0d159642c5975f9eb70335a6f"
MY_P="azure-sdk-for-cpp-${COMMIT}"
DESCRIPTION="Azure SDK for C++"
HOMEPAGE="https://azure.github.io/azure-sdk-for-cpp/"
SRC_URI="https://github.com/Azure/azure-sdk-for-cpp/archive/${COMMIT}.tar.gz -> ${MY_P}.tar.gz"
S="${WORKDIR}/${MY_P}/sdk/identity/${PN}"
LICENSE="MIT"
SLOT="0"
KEYWORDS="~amd64"
IUSE="doc"
RESTRICT="test" # Too many online tests.

RDEPEND="
        dev-cpp/azure-core
        dev-libs/openssl:=
        net-misc/curl[ssl]
"
DEPEND="
        ${RDEPEND}
"
BDEPEND="
        virtual/pkgconfig
        doc? ( app-text/doxygen )
"

src_configure() {
        local mycmakeargs=(
                -DBUILD_DOCUMENTATION=$(usex doc)
                -DWARNINGS_AS_ERRORS=no
        )

        AZURE_SDK_DISABLE_AUTO_VCPKG=yes \
        cmake_src_configure
}

src_install() {
        cmake_src_install
        use doc && dodoc -r "${BUILD_DIR}"/docs/html
}