Doubts regarding compiling llvm (clang, lld) to wasm

[anutosh491]

I see this

llvm-project/clang/lib/Interpreter/CMakeLists.txt

Lines 1 to 48 in e8b95a0

	set(LLVM_LINK_COMPONENTS
	core
	native
	MC
	Option
	OrcJit
	OrcDebugging
	OrcShared
	OrcTargetProcess
	Support
	Target
	TargetParser
	)
	if (EMSCRIPTEN AND "lld" IN_LIST LLVM_ENABLE_PROJECTS)
	set(WASM_SRC Wasm.cpp)
	set(WASM_LINK lldWasm)
	endif()
	add_clang_library(clangInterpreter
	DeviceOffload.cpp
	CodeCompletion.cpp
	IncrementalExecutor.cpp
	IncrementalParser.cpp
	Interpreter.cpp
	InterpreterValuePrinter.cpp
	InterpreterUtils.cpp
	Value.cpp
	${WASM_SRC}
	PARTIAL_SOURCES_INTENDED
	DEPENDS
	intrinsics_gen
	ClangDriverOptions
	LINK_LIBS
	clangAST
	clangAnalysis
	clangBasic
	clangCodeGen
	clangDriver
	clangFrontend
	clangFrontendTool
	clangLex
	clangParse
	clangSema
	clangSerialization
	${WASM_LINK}

Well this generates the following link.txt

 │ │ $BUILD_PREFIX/opt/emsdk/upstream/emscripten/emar qc ../../../../lib/libclangInterpreter.a CMakeFiles/obj.clangInterpreter.dir/DeviceOffload.cpp.o CMakeFile
 │ │ s/obj.clangInterpreter.dir/CodeCompletion.cpp.o CMakeFiles/obj.clangInterpreter.dir/IncrementalExecutor.cpp.o CMakeFiles/obj.clangInterpreter.dir/Increment
 │ │ alParser.cpp.o CMakeFiles/obj.clangInterpreter.dir/Interpreter.cpp.o CMakeFiles/obj.clangInterpreter.dir/InterpreterUtils.cpp.o CMakeFiles/obj.clangInterpr
 │ │ eter.dir/Value.cpp.o CMakeFiles/obj.clangInterpreter.dir/Wasm.cpp.o
 │ │ $BUILD_PREFIX/opt/emsdk/upstream/emscripten/emranlib ../../../../lib/libclangInterpreter.a

Here I can see the source file object being used .

But i am not able to understand the linking between clangInterpreter and the other link_libs (like clangAST) and components (like for eg Support). Where and How does this take place ?

I am working on a very similar structure where there are

1. llvm components
2. static libraries generated when compiling against wasm (like libclangInterpreter.a)
3. There are source files

So I need to compile my project (let's say libXX.a) .
I am using the similar mechanism as above (only add_llvm_library instead of add_clang_library)

So when I build xx against emscripten, I end up getting lot of undefined symbols (which are supposed to be provided by the static libraries like libclangInterpreter.a) .... which tells me that the linking against the static libraries and components is not taking place as expected.

Hence if I understand how the above works I shall try to replicate it for my usecase too

[DavidSpickett]

I am not sure what link.txt is, perhaps this is specific to the emscripten system, but you can compare against a native build using ninja. I found this in build/build.ninja:

#############################################
# Link the static library lib/libclangInterpreter.a

build lib/libclangInterpreter.a: CXX_STATIC_LIBRARY_LINKER__clangInterpreter_Release tools/clang/lib/Interpreter/CMakeFiles/obj.clangInterpreter.dir/DeviceOffload.cpp.o tools/clang/lib/Interpreter/CMakeFiles/obj.clangInterpreter.dir/CodeCompletion.cpp.o tools/clang/lib/Interpreter/CMakeFiles/obj.clangInterpreter.dir/IncrementalExecutor.cpp.o tools/clang/lib/Interpreter/CMakeFiles/obj.clangInterpreter.dir/IncrementalParser.cpp.o tools/clang/lib/Interpreter/CMakeFiles/obj.clangInterpreter.dir/Interpreter.cpp.o tools/clang/lib/Interpreter/CMakeFiles/obj.clangInterpreter.dir/InterpreterValuePrinter.cpp.o tools/clang/lib/Interpreter/CMakeFiles/obj.clangInterpreter.dir/InterpreterUtils.cpp.o tools/clang/lib/Interpreter/CMakeFiles/obj.clangInterpreter.dir/Value.cpp.o || include/llvm/IR/intrinsics_gen lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64CodeGen.a lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Disassembler.a lib/libLLVMAArch64Info.a lib/libLLVMAArch64Utils.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMAnalysis.a lib/libLLVMAsmParser.a lib/libLLVMAsmPrinter.a lib/libLLVMBinaryFormat.a lib/libLLVMBitReader.a lib/libLLVMBitWriter.a lib/libLLVMBitstreamReader.a lib/libLLVMCFGuard.a lib/libLLVMCGData.a lib/libLLVMCodeGen.a lib/libLLVMCodeGenTypes.a lib/libLLVMCore.a lib/libLLVMCoroutines.a lib/libLLVMCoverage.a lib/libLLVMDebugInfoBTF.a lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoDWARF.a lib/libLLVMDebugInfoMSF.a lib/libLLVMDebugInfoPDB.a lib/libLLVMDemangle.a lib/libLLVMExecutionEngine.a lib/libLLVMExtensions.a lib/libLLVMFrontendDriver.a lib/libLLVMFrontendHLSL.a lib/libLLVMFrontendOffloading.a lib/libLLVMFrontendOpenMP.a lib/libLLVMGlobalISel.a lib/libLLVMHipStdPar.a lib/libLLVMIRPrinter.a lib/libLLVMIRReader.a lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a lib/libLLVMJITLink.a lib/libLLVMLTO.a lib/libLLVMLinker.a lib/libLLVMMC.a lib/libLLVMMCDisassembler.a lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a lib/libLLVMObject.a lib/libLLVMOption.a lib/libLLVMOrcDebugging.a lib/libLLVMOrcJIT.a lib/libLLVMOrcShared.a lib/libLLVMOrcTargetProcess.a lib/libLLVMPasses.a lib/libLLVMProfileData.a lib/libLLVMRemarks.a lib/libLLVMRuntimeDyld.a lib/libLLVMSandboxIR.a lib/libLLVMScalarOpts.a lib/libLLVMSelectionDAG.a lib/libLLVMSupport.a lib/libLLVMSymbolize.a lib/libLLVMTarget.a lib/libLLVMTargetParser.a lib/libLLVMTextAPI.a lib/libLLVMTextAPIBinaryReader.a lib/libLLVMTransformUtils.a lib/libLLVMVectorize.a lib/libLLVMWindowsDriver.a lib/libLLVMipo.a lib/libclangAPINotes.a lib/libclangARCMigrate.a lib/libclangAST.a lib/libclangASTMatchers.a lib/libclangAnalysis.a lib/libclangBasic.a lib/libclangCodeGen.a lib/libclangCrossTU.a lib/libclangDriver.a lib/libclangEdit.a lib/libclangExtractAPI.a lib/libclangFormat.a lib/libclangFrontend.a lib/libclangFrontendTool.a lib/libclangIndex.a lib/libclangInstallAPI.a lib/libclangLex.a lib/libclangParse.a lib/libclangRewrite.a lib/libclangRewriteFrontend.a lib/libclangSema.a lib/libclangSerialization.a lib/libclangStaticAnalyzerCheckers.a lib/libclangStaticAnalyzerCore.a lib/libclangStaticAnalyzerFrontend.a lib/libclangSupport.a lib/libclangToolingCore.a lib/libclangToolingInclusions.a tools/clang/clang-tablegen-targets tools/clang/include/clang/Driver/ClangDriverOptions tools/clang/lib/Interpreter/obj.clangInterpreter

The cmake functions/macros used in the file you linked are defined in clang/cmake/modules/AddClang.cmake and llvm/cmake/modules/AddLLVM.cmake (but I think you've been digging in there already).

Perhaps you could make your setup work as a native build first and then try wasm again? To see if the problem is the commands that cmake is generating.

[anutosh491]

Okay maybe I can provide some more context

1. link.txt is nothing but the place that technically has the link specific instructions (basically it stores the instructions that you see while building with make Verbose=1). This file just contains the exact command that CMake will use to link a target and hence CMake generates this file based on the configuration specified in CMakeLists.txt

So when you see something like this

 │ │ [ 82%] Linking CXX static library ../../../../lib/libclangInterpreter.a
 │ │ cd /home/runner/work/recipes/recipes/output/bld/rattler-build_llvm_1729598973/work/build/tools/clang/lib/Interpreter && $BUILD_PREFIX/bin/cmake -P CMakeFil
 │ │ es/clangInterpreter.dir/cmake_clean_target.cmake
 │ │ cd /home/runner/work/recipes/recipes/output/bld/rattler-build_llvm_1729598973/work/build/tools/clang/lib/Interpreter && $BUILD_PREFIX/bin/cmake -E cmake_li
 │ │ nk_script CMakeFiles/clangInterpreter.dir/link.txt --verbose=1
 │ │ $BUILD_PREFIX/opt/emsdk/upstream/emscripten/emar qc ../../../../lib/libclangInterpreter.a CMakeFiles/obj.clangInterpreter.dir/DeviceOffload.cpp.o CMakeFile
 │ │ s/obj.clangInterpreter.dir/CodeCompletion.cpp.o CMakeFiles/obj.clangInterpreter.dir/IncrementalExecutor.cpp.o CMakeFiles/obj.clangInterpreter.dir/Increment
 │ │ alParser.cpp.o CMakeFiles/obj.clangInterpreter.dir/Interpreter.cpp.o CMakeFiles/obj.clangInterpreter.dir/InterpreterUtils.cpp.o CMakeFiles/obj.clangInterpr
 │ │ eter.dir/Value.cpp.o CMakeFiles/obj.clangInterpreter.dir/Wasm.cpp.o
 │ │ $BUILD_PREFIX/opt/emsdk/upstream/emscripten/emranlib ../../../../lib/libclangInterpreter.a
 │ │ make[2]: Leaving directory '/home/runner/work/recipes/recipes/output/bld/rattler-build_llvm_1729598973/work/build'

The emar and emranlib instructions are stored there.

I am maintaining the llvm recipe for emscripten-forge (a place that hosts recipes built against emscripten ) and the above logs are from there

2. The project is an established one (CppInterOp ... provides wrappers over clang-repl) and I want to compile it against emscripten so that I can run clang-repl in the browser.

Now I have this segment in the CMakeLists.txt

set(LLVM_LINK_COMPONENTS
 ${LLVM_TARGETS_TO_BUILD}
 BinaryFormat
 Core
 Object
 OrcJit
 Support
)

set(link_libs
 clangInterpreter
 clangAST
 clangBasic
 clangFrontend
 clangLex
 clangSema
 )

add_llvm_library(clangCppInterOp
 DISABLE_LLVM_LINK_LLVM_DYLIB
 CppInterOp.cpp
 ${DLM}
 LINK_LIBS
 ${link_libs}
)

I just can't understand why the linking between clangCppInterOp and clangInterpreter isn't happening properly (leading to undefined symbols). Now I see similar steps being encountered while building llvm (clang, lld) against emscripten too .... but as I mentioned above I don't think there are any errors and libclangInterpreter.a has access to necessary symbols out of link_libs like libclangAST.a