Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CGData] LLD for MachO #90166

Merged
merged 3 commits into from
Sep 15, 2024
Merged

[CGData] LLD for MachO #90166

merged 3 commits into from
Sep 15, 2024

Conversation

kyulee-com
Copy link
Contributor

@kyulee-com kyulee-com commented Apr 26, 2024

It reads raw CG data encoded in the custom section (__llvm_outline) in object files and merges them into the indexed codegen data file specified by -codegen-data-generate-path={path}.

This depends on #90074.
This is a patch for https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.

kyulee-com added a commit that referenced this pull request Sep 10, 2024
This commit introduces support for outlining functions across modules
using codegen data generated from previous codegen. The codegen data
currently manages the outlined hash tree, which records outlining
instances that occurred locally in the past.
    
The machine outliner now operates in one of three modes:

1. CGDataMode::None: This is the default outliner mode that uses the
suffix tree to identify (local) outlining candidates within a module.
This mode is also used by (full)LTO to maintain optimal behavior with
the combined module.
2. CGDataMode::Write (`-codegen-data-generate`): This mode is identical
to the default mode, but it also publishes the stable hash sequences of
instructions in the outlined functions into a local outlined hash tree.
It then encodes this into the `__llvm_outline` section, which will be
dead-stripped at link time.
3. CGDataMode::Read (`-codegen-data-use-path={.cgdata}`): This mode
reads a codegen data file (.cgdata) and initializes a global outlined
hash tree. This tree is used to generate global outlining candidates.
Note that the codegen data file has been post-processed with the raw
`__llvm_outline` sections from all native objects using the
`llvm-cgdata` tool (or a linker, `LLD`, or a new ThinLTO pipeline
later).

This depends on #105398. After
this PR, LLD (#90166) and Clang
(#90304) will follow for each
client side support.
This is a patch for
https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.
@llvmbot
Copy link
Collaborator

llvmbot commented Sep 11, 2024

@llvm/pr-subscribers-lld-macho

@llvm/pr-subscribers-lld

Author: Kyungwoo Lee (kyulee-com)

Changes

It reads raw CG data encoded in the custom section (__llvm_outline) in object files and merges them into the indexed codegen data file specified by -codegen-data-generate-path={path}.

This depends on #90074.
This is a patch for https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.


Full diff: https://github.com/llvm/llvm-project/pull/90166.diff

7 Files Affected:

  • (modified) lld/MachO/Config.h (+1)
  • (modified) lld/MachO/Driver.cpp (+39)
  • (modified) lld/MachO/InputSection.h (+1)
  • (modified) lld/MachO/Options.td (+2)
  • (modified) lld/test/CMakeLists.txt (+1)
  • (added) lld/test/MachO/cgdata-generate.s (+94)
  • (modified) lld/test/lit.cfg.py (+1)
diff --git a/lld/MachO/Config.h b/lld/MachO/Config.h
index 5fca3f16a897ed..8f6da6330d7ad4 100644
--- a/lld/MachO/Config.h
+++ b/lld/MachO/Config.h
@@ -210,6 +210,7 @@ struct Configuration {
   std::vector<SectionAlign> sectionAlignments;
   std::vector<SegmentProtection> segmentProtections;
   bool ltoDebugPassManager = false;
+  llvm::StringRef codegenDataGeneratePath;
   bool csProfileGenerate = false;
   llvm::StringRef csProfilePath;
   bool pgoWarnMismatch;
diff --git a/lld/MachO/Driver.cpp b/lld/MachO/Driver.cpp
index 73b83123f49b4d..e4c908f0b0aaf9 100644
--- a/lld/MachO/Driver.cpp
+++ b/lld/MachO/Driver.cpp
@@ -36,6 +36,7 @@
 #include "llvm/ADT/StringRef.h"
 #include "llvm/BinaryFormat/MachO.h"
 #include "llvm/BinaryFormat/Magic.h"
+#include "llvm/CGData/CodeGenDataWriter.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/LTO/LTO.h"
 #include "llvm/Object/Archive.h"
@@ -1322,6 +1323,38 @@ static void gatherInputSections() {
   }
 }
 
+static void codegenDataGenerate() {
+  TimeTraceScope timeScope("Generating codegen data");
+
+  OutlinedHashTreeRecord globalOutlineRecord;
+  for (ConcatInputSection *isec : inputSections) {
+    if (isec->getSegName() == segment_names::data &&
+        isec->getName() == section_names::outlinedHashTree) {
+      // Read outlined hash tree from each section
+      OutlinedHashTreeRecord localOutlineRecord;
+      auto *data = isec->data.data();
+      localOutlineRecord.deserialize(data);
+
+      // Merge it to the global hash tree.
+      globalOutlineRecord.merge(localOutlineRecord);
+    }
+  }
+
+  CodeGenDataWriter Writer;
+  if (!globalOutlineRecord.empty())
+    Writer.addRecord(globalOutlineRecord);
+
+  std::error_code EC;
+  auto fileName = config->codegenDataGeneratePath;
+  assert(!fileName.empty());
+  raw_fd_ostream Output(fileName, EC, sys::fs::OF_None);
+  if (EC)
+    error("fail to create raw_fd_ostream");
+
+  if (auto E = Writer.write(Output))
+    error("fail to write CGData");
+}
+
 static void foldIdenticalLiterals() {
   TimeTraceScope timeScope("Fold identical literals");
   // We always create a cStringSection, regardless of whether dedupLiterals is
@@ -1759,6 +1792,8 @@ bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
     config->ignoreAutoLinkOptions.insert(arg->getValue());
   config->strictAutoLink = args.hasArg(OPT_strict_auto_link);
   config->ltoDebugPassManager = args.hasArg(OPT_lto_debug_pass_manager);
+  config->codegenDataGeneratePath =
+      args.getLastArgValue(OPT_codegen_data_generate_path);
   config->csProfileGenerate = args.hasArg(OPT_cs_profile_generate);
   config->csProfilePath = args.getLastArgValue(OPT_cs_profile_path);
   config->pgoWarnMismatch =
@@ -2103,6 +2138,10 @@ bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
     }
 
     gatherInputSections();
+
+    if (!config->codegenDataGeneratePath.empty())
+      codegenDataGenerate();
+
     if (config->callGraphProfileSort)
       priorityBuilder.extractCallGraphProfile();
 
diff --git a/lld/MachO/InputSection.h b/lld/MachO/InputSection.h
index 4e238d8ef77793..7ef0e31066f372 100644
--- a/lld/MachO/InputSection.h
+++ b/lld/MachO/InputSection.h
@@ -354,6 +354,7 @@ constexpr const char objcMethname[] = "__objc_methname";
 constexpr const char objcNonLazyCatList[] = "__objc_nlcatlist";
 constexpr const char objcNonLazyClassList[] = "__objc_nlclslist";
 constexpr const char objcProtoList[] = "__objc_protolist";
+constexpr const char outlinedHashTree[] = "__llvm_outline";
 constexpr const char pageZero[] = "__pagezero";
 constexpr const char pointers[] = "__pointers";
 constexpr const char rebase[] = "__rebase";
diff --git a/lld/MachO/Options.td b/lld/MachO/Options.td
index cbd28bbceef786..8c5c6a853f3cc7 100644
--- a/lld/MachO/Options.td
+++ b/lld/MachO/Options.td
@@ -162,6 +162,8 @@ def no_objc_category_merging : Flag<["-"], "no_objc_category_merging">,
     Group<grp_lld>;
 def lto_debug_pass_manager: Flag<["--"], "lto-debug-pass-manager">,
     HelpText<"Debug new pass manager">, Group<grp_lld>;
+def codegen_data_generate_path : Joined<["--"], "codegen-data-generate-path=">,
+    HelpText<"Codegen data file path">, Group<grp_lld>;
 def cs_profile_generate: Flag<["--"], "cs-profile-generate">,
     HelpText<"Perform context sensitive PGO instrumentation">, Group<grp_lld>;
 def cs_profile_path: Joined<["--"], "cs-profile-path=">,
diff --git a/lld/test/CMakeLists.txt b/lld/test/CMakeLists.txt
index 5d4a2757c529b8..abc8ea75da180b 100644
--- a/lld/test/CMakeLists.txt
+++ b/lld/test/CMakeLists.txt
@@ -48,6 +48,7 @@ if (NOT LLD_BUILT_STANDALONE)
     llvm-ar
     llvm-as
     llvm-bcanalyzer
+    llvm-cgdata
     llvm-config
     llvm-cvtres
     llvm-dis
diff --git a/lld/test/MachO/cgdata-generate.s b/lld/test/MachO/cgdata-generate.s
new file mode 100644
index 00000000000000..0f91070dd9cc0c
--- /dev/null
+++ b/lld/test/MachO/cgdata-generate.s
@@ -0,0 +1,94 @@
+# UNSUPPORTED: system-windows
+# REQUIRES: aarch64
+
+# RUN: rm -rf %t; split-file %s %t
+
+# Synthesize raw cgdata without the header (24 byte) from the indexed cgdata.
+# RUN: llvm-cgdata --convert --format binary %t/raw-1.cgtext -o %t/raw-1.cgdata
+# RUN: od -t x1 -j 24 -An %t/raw-1.cgdata | tr -d '\n\r\t' | sed 's/ \+/ /g; s/^ *//; s/ *$//; s/ /,0x/g; s/^/0x/' > %t/raw-1-bytes.txt
+# RUN: sed "s/<RAW_1_BYTES>/$(cat %t/raw-1-bytes.txt)/g" %t/merge-1-template.s > %t/merge-1.s
+# RUN: llvm-cgdata --convert --format binary %t/raw-2.cgtext -o %t/raw-2.cgdata
+# RUN: od -t x1 -j 24 -An %t/raw-2.cgdata | tr -d '\n\r\t' | sed 's/ \+/ /g; s/^ *//; s/ *$//; s/ /,0x/g; s/^/0x/' > %t/raw-2-bytes.txt
+# RUN: sed "s/<RAW_2_BYTES>/$(cat %t/raw-2-bytes.txt)/g" %t/merge-2-template.s > %t/merge-2.s
+
+# RUN: llvm-mc -filetype obj -triple arm64-apple-darwin %t/merge-1.s -o %t/merge-1.o
+# RUN: llvm-mc -filetype obj -triple arm64-apple-darwin %t/merge-2.s -o %t/merge-2.o
+# RUN: llvm-mc -filetype obj -triple arm64-apple-darwin %t/main.s -o %t/main.o
+
+# This checks if the codegen data from the linker is identical to the merged codegen data
+# from each object file, which is obtained using the llvm-cgdata tool.
+# RUN: %no-arg-lld -dylib -arch arm64 -platform_version ios 14.0 15.0 -o %t/out \
+# RUN: %t/merge-1.o %t/merge-2.o %t/main.o --codegen-data-generate-path=%t/out-cgdata
+# RUN: llvm-cgdata --merge %t/merge-1.o %t/merge-2.o %t/main.o -o %t/merge-cgdata
+# RUN: diff %t/out-cgdata %t/merge-cgdata
+
+# Merge order doesn't matter. `main.o` is dropped due to missing __llvm_outline.
+# RUN: llvm-cgdata --merge %t/merge-2.o %t/merge-1.o -o %t/merge-cgdata-shuffle
+# RUN: diff %t/out-cgdata %t/merge-cgdata-shuffle
+
+# We can also generate the merged codegen data from the executable that is not dead-stripped.
+# RUN: llvm-objdump -h %t/out| FileCheck %s
+CHECK: __llvm_outline
+# RUN: llvm-cgdata --merge %t/out -o %t/merge-cgdata-exe
+# RUN: diff %t/merge-cgdata-exe %t/merge-cgdata
+
+# Dead-strip will remove __llvm_outline sections from the final executable.
+# But the codeden data is still correctly produced from the linker.
+# RUN: %no-arg-lld -dylib -arch arm64 -platform_version ios 14.0 15.0 -o %t/out-strip \
+# RUN: %t/merge-1.o %t/merge-2.o %t/main.o -dead_strip --codegen-data-generate-path=%t/out-cgdata-strip
+# RUN: llvm-cgdata --merge %t/merge-1.o %t/merge-2.o %t/main.o -o %t/merge-cgdata-strip
+# RUN: diff %t/out-cgdata-strip %t/merge-cgdata-strip
+# RUN: diff %t/out-cgdata-strip %t/merge-cgdata
+
+# Ensure no __llvm_outline section remains in the executable.
+# RUN: llvm-objdump -h %t/out-strip | FileCheck %s --check-prefix=STRIP
+STRIP-NOT: __llvm_outline
+
+#--- raw-1.cgtext
+:outlined_hash_tree
+0:
+  Hash:            0x0
+  Terminals:       0
+  SuccessorIds:    [ 1 ]
+1:
+  Hash:            0x1
+  Terminals:       0
+  SuccessorIds:    [ 2 ]
+2:
+  Hash:            0x2
+  Terminals:       4
+  SuccessorIds:    [  ]
+...
+
+#--- merge-1-template.s
+.section __DATA,__llvm_outline
+_data:
+.byte <RAW_1_BYTES>
+
+#--- raw-2.cgtext
+:outlined_hash_tree
+0:
+  Hash:            0x0
+  Terminals:       0
+  SuccessorIds:    [ 1 ]
+1:
+  Hash:            0x1
+  Terminals:       0
+  SuccessorIds:    [ 2 ]
+2:
+  Hash:            0x3
+  Terminals:       5
+  SuccessorIds:    [  ]
+...
+
+#--- merge-2-template.s
+.section __DATA,__llvm_outline
+_data:
+.byte <RAW_2_BYTES>
+
+#--- main.s
+.globl _main
+
+.text
+_main:
+  ret
diff --git a/lld/test/lit.cfg.py b/lld/test/lit.cfg.py
index d309c2ad4ee284..859094e2b57dbc 100644
--- a/lld/test/lit.cfg.py
+++ b/lld/test/lit.cfg.py
@@ -40,6 +40,7 @@
 tool_patterns = [
     "llc",
     "llvm-as",
+    "llvm-cgdata",
     "llvm-mc",
     "llvm-nm",
     "llvm-objdump",

lld/MachO/Driver.cpp Outdated Show resolved Hide resolved
lld/MachO/Options.td Outdated Show resolved Hide resolved
lld/test/MachO/cgdata-generate.s Outdated Show resolved Hide resolved
lld/test/MachO/cgdata-generate.s Outdated Show resolved Hide resolved
VitaNuo pushed a commit to VitaNuo/llvm-project that referenced this pull request Sep 12, 2024
This commit introduces support for outlining functions across modules
using codegen data generated from previous codegen. The codegen data
currently manages the outlined hash tree, which records outlining
instances that occurred locally in the past.
    
The machine outliner now operates in one of three modes:

1. CGDataMode::None: This is the default outliner mode that uses the
suffix tree to identify (local) outlining candidates within a module.
This mode is also used by (full)LTO to maintain optimal behavior with
the combined module.
2. CGDataMode::Write (`-codegen-data-generate`): This mode is identical
to the default mode, but it also publishes the stable hash sequences of
instructions in the outlined functions into a local outlined hash tree.
It then encodes this into the `__llvm_outline` section, which will be
dead-stripped at link time.
3. CGDataMode::Read (`-codegen-data-use-path={.cgdata}`): This mode
reads a codegen data file (.cgdata) and initializes a global outlined
hash tree. This tree is used to generate global outlining candidates.
Note that the codegen data file has been post-processed with the raw
`__llvm_outline` sections from all native objects using the
`llvm-cgdata` tool (or a linker, `LLD`, or a new ThinLTO pipeline
later).

This depends on llvm#105398. After
this PR, LLD (llvm#90166) and Clang
(llvm#90304) will follow for each
client side support.
This is a patch for
https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.
@ellishg
Copy link
Contributor

ellishg commented Sep 12, 2024

-fprofile-generate supports modifiers like %p, %h, %m, %t, and %c to allow generating raw profile filenames based on the run instance. This is especially useful when we want to dump a bunch of raw profiles to a directory.

https://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation

Do you think we should support something similar given that we can pass a directory to the clang frontend flag in #90304? If we build several binaries, we still want to disambiguate the .cgdata files, even if we want to dump them in the same directory. Maybe %n could expand to the binary name and %b could expand to some build number, which could be anywhere in the path. I'm also not sure if it's better to implement this in Clang or LLVM.

@kyulee-com
Copy link
Contributor Author

-fprofile-generate supports modifiers like %p, %h, %m, %t, and %c to allow generating raw profile filenames based on the run instance. This is especially useful when we want to dump a bunch of raw profiles to a directory.

https://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation

Do you think we should support something similar given that we can pass a directory to the clang frontend flag in #90304? If we build several binaries, we still want to disambiguate the .cgdata files, even if we want to dump them in the same directory. Maybe %n could expand to the binary name and %b could expand to some build number, which could be anywhere in the path. I'm also not sure if it's better to implement this in Clang or LLVM.

Thank you for the suggestion! I largely copied the Clang flags from IRPGO. Unlike IRPGO, which needs to handle many raw profile data files emitted by runtimes, here we deal with a single codegen data file at build time, similar to how a linker handles a single executable file. Therefore, I think the build system can easily configure this additional flag with the correct path for different binaries. So, I don't think there is a need to support a directory path or these modifiers for simplicity. I'm considering simplifying the Clang flag to avoid this confusion. What do you think?

kyulee-com added a commit to kyulee-com/llvm-project that referenced this pull request Sep 12, 2024
kyulee-com added a commit to kyulee-com/llvm-project that referenced this pull request Sep 12, 2024
kyulee-com added a commit to kyulee-com/llvm-project that referenced this pull request Sep 12, 2024
@ellishg
Copy link
Contributor

ellishg commented Sep 12, 2024

-fprofile-generate supports modifiers like %p, %h, %m, %t, and %c to allow generating raw profile filenames based on the run instance. This is especially useful when we want to dump a bunch of raw profiles to a directory.
https://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation
Do you think we should support something similar given that we can pass a directory to the clang frontend flag in #90304? If we build several binaries, we still want to disambiguate the .cgdata files, even if we want to dump them in the same directory. Maybe %n could expand to the binary name and %b could expand to some build number, which could be anywhere in the path. I'm also not sure if it's better to implement this in Clang or LLVM.

Thank you for the suggestion! I largely copied the Clang flags from IRPGO. Unlike IRPGO, which needs to handle many raw profile data files emitted by runtimes, here we deal with a single codegen data file at build time, similar to how a linker handles a single executable file. Therefore, I think the build system can easily configure this additional flag with the correct path for different binaries. So, I don't think there is a need to support a directory path or these modifiers for simplicity. I'm considering simplifying the Clang flag to avoid this confusion. What do you think?

I see. If we don't support writing profiles to a directory, then I don't think it's necessary to support these modifiers. I do, however, see the value in writing to a default file (default.cgdata) if no path is specified, similar to a.out. But I don't see the point in a user specifying a directory without specifying the filename.

@kyulee-com
Copy link
Contributor Author

-fprofile-generate supports modifiers like %p, %h, %m, %t, and %c to allow generating raw profile filenames based on the run instance. This is especially useful when we want to dump a bunch of raw profiles to a directory.
https://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation
Do you think we should support something similar given that we can pass a directory to the clang frontend flag in #90304? If we build several binaries, we still want to disambiguate the .cgdata files, even if we want to dump them in the same directory. Maybe %n could expand to the binary name and %b could expand to some build number, which could be anywhere in the path. I'm also not sure if it's better to implement this in Clang or LLVM.

Thank you for the suggestion! I largely copied the Clang flags from IRPGO. Unlike IRPGO, which needs to handle many raw profile data files emitted by runtimes, here we deal with a single codegen data file at build time, similar to how a linker handles a single executable file. Therefore, I think the build system can easily configure this additional flag with the correct path for different binaries. So, I don't think there is a need to support a directory path or these modifiers for simplicity. I'm considering simplifying the Clang flag to avoid this confusion. What do you think?

I see. If we don't support writing profiles to a directory, then I don't think it's necessary to support these modifiers. I do, however, see the value in writing to a default file (default.cgdata) if no path is specified, similar to a.out. But I don't see the point in a user specifying a directory without specifying the filename.

I already updated #90304 accordingly to remove a directory support while stilling setting a default file (default.cgdata) when it's not specified.

@kyulee-com
Copy link
Contributor Author

Can someone take another look? I think I've addressed the comments and this code should be straightforward.

Copy link
Contributor

@thevinster thevinster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

TimeTraceScope timeScope("Generating codegen data");

OutlinedHashTreeRecord globalOutlineRecord;
for (ConcatInputSection *isec : inputSections) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: You can remove the braces in the for loop. Though in practice, I've seen both conventions be used here. So up to you if you think that helps with readability.

@kyulee-com kyulee-com merged commit 00c0b1a into llvm:main Sep 15, 2024
8 checks passed
@kyulee-com kyulee-com deleted the lld branch September 15, 2024 03:38
@llvm-ci
Copy link
Collaborator

llvm-ci commented Sep 15, 2024

LLVM Buildbot has detected a new failure on builder llvm-clang-aarch64-darwin running on doug-worker-5 while building lld at step 6 "test-build-unified-tree-check-all".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/190/builds/5762

Here is the relevant piece of the build log for the reference
Step 6 (test-build-unified-tree-check-all) failure: test (failure)
******************** TEST 'lld :: MachO/cgdata-generate.s' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 4: rm -rf /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp; split-file /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/lld/test/MachO/cgdata-generate.s /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp
+ rm -rf /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp
+ split-file /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/lld/test/MachO/cgdata-generate.s /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp
RUN: at line 7: /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/llvm-cgdata --convert --format binary /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1.cgtext -o /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1.cgdata
+ /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/llvm-cgdata --convert --format binary /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1.cgtext -o /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1.cgdata
RUN: at line 8: od -t x1 -j 24 -An /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1.cgdata | tr -d '\n\r\t' | sed 's/ \+/ /g; s/^ *//; s/ *$//; s/ /,0x/g; s/^/0x/' > /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1-bytes.txt
+ tr -d '\n\r\t'
+ od -t x1 -j 24 -An /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1.cgdata
+ sed 's/ \+/ /g; s/^ *//; s/ *$//; s/ /,0x/g; s/^/0x/'
RUN: at line 9: sed "s/<RAW_BYTES>/$(cat /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1-bytes.txt)/g" /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-template.s > /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-1.s
++ cat /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-1-bytes.txt
+ sed 's/<RAW_BYTES>/0x03,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x02,0x,0x00,0x,0x00,0x,0x00,0x,0x02,0x,0x00,0x,0x00,0x,0x00,0x,0x02,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x04,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00/g' /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-template.s
RUN: at line 10: /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/llvm-cgdata --convert --format binary /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2.cgtext -o /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2.cgdata
+ /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/llvm-cgdata --convert --format binary /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2.cgtext -o /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2.cgdata
RUN: at line 11: od -t x1 -j 24 -An /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2.cgdata | tr -d '\n\r\t' | sed 's/ \+/ /g; s/^ *//; s/ *$//; s/ /,0x/g; s/^/0x/' > /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2-bytes.txt
+ tr -d '\n\r\t'
+ sed 's/ \+/ /g; s/^ *//; s/ *$//; s/ /,0x/g; s/^/0x/'
+ od -t x1 -j 24 -An /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2.cgdata
RUN: at line 12: sed "s/<RAW_BYTES>/$(cat /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2-bytes.txt)/g" /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-template.s > /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-2.s
++ cat /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/raw-2-bytes.txt
+ sed 's/<RAW_BYTES>/0x03,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x02,0x,0x00,0x,0x00,0x,0x00,0x,0x02,0x,0x00,0x,0x00,0x,0x00,0x,0x03,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x05,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00/g' /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-template.s
RUN: at line 14: /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/llvm-mc -filetype obj -triple arm64-apple-darwin /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-1.s -o /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-1.o
+ /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/llvm-mc -filetype obj -triple arm64-apple-darwin /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-1.s -o /Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-1.o
/Users/buildbot/buildbot-root/aarch64-darwin/build/tools/lld/test/MachO/Output/cgdata-generate.s.tmp/merge-1.s:3:12: error: invalid hexadecimal number
.byte 0x03,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x01,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x02,0x,0x00,0x,0x00,0x,0x00,0x,0x02,0x,0x00,0x,0x00,0x,0x00,0x,0x02,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x,0x04,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00,0x,0x00
           ^

--

********************


@llvm-ci
Copy link
Collaborator

llvm-ci commented Sep 15, 2024

LLVM Buildbot has detected a new failure on builder bolt-x86_64-ubuntu-shared running on bolt-worker while building lld at step 6 "test-build-bolt-check-bolt".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/151/builds/2294

Here is the relevant piece of the build log for the reference
Step 6 (test-build-bolt-check-bolt) failure: test (failure)
...
16.349 [285/18/1302] Building CXX object tools/lld/MachO/CMakeFiles/lldMachO.dir/Driver.cpp.o
16.367 [284/18/1303] Building CXX object tools/clang/lib/Index/CMakeFiles/obj.clangIndex.dir/IndexingContext.cpp.o
16.387 [283/18/1304] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/Checker.cpp.o
16.444 [282/18/1305] Building CXX object tools/lld/MachO/CMakeFiles/lldMachO.dir/InputFiles.cpp.o
16.495 [281/18/1306] Building CXX object tools/clang/lib/Index/CMakeFiles/obj.clangIndex.dir/IndexingAction.cpp.o
16.507 [280/18/1307] Building CXX object tools/clang/lib/FrontendTool/CMakeFiles/obj.clangFrontendTool.dir/ExecuteCompilerInvocation.cpp.o
16.531 [279/18/1308] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/CheckerRegistryData.cpp.o
16.536 [278/18/1309] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/CommonBugCategories.cpp.o
16.557 [277/18/1310] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/ConstraintManager.cpp.o
16.566 [276/18/1311] Linking CXX shared library lib/liblldMachO.so.20.0git
FAILED: lib/liblldMachO.so.20.0git 
: && /usr/bin/c++ -fPIC -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -fno-lifetime-dse -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-uninitialized -Wno-nonnull -Wno-class-memaccess -Wno-redundant-move -Wno-pessimizing-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wno-misleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=lld -Wl,--color-diagnostics   -Wl,--gc-sections -shared -Wl,-soname,liblldMachO.so.20.0git -o lib/liblldMachO.so.20.0git tools/lld/MachO/CMakeFiles/lldMachO.dir/Arch/ARM64.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Arch/ARM64Common.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Arch/ARM64_32.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Arch/X86_64.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/ConcatOutputSection.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Driver.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/DriverUtils.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Dwarf.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/EhFrame.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/ExportTrie.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/ICF.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/InputFiles.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/InputSection.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/LTO.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/MapFile.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/MarkLive.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/ObjC.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/OutputSection.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/OutputSegment.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Relocations.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/BPSectionOrderer.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/SectionPriorities.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Sections.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/SymbolTable.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Symbols.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/SyntheticSections.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Target.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/UnwindInfoSection.cpp.o tools/lld/MachO/CMakeFiles/lldMachO.dir/Writer.cpp.o  -Wl,-rpath,"\$ORIGIN/../lib:/home/worker/bolt-worker2/bolt-x86_64-ubuntu-shared/build/lib:"  lib/liblldCommon.so.20.0git  lib/libLLVMX86CodeGen.so.20.0git  lib/libLLVMX86AsmParser.so.20.0git  lib/libLLVMX86Desc.so.20.0git  lib/libLLVMX86Disassembler.so.20.0git  lib/libLLVMX86Info.so.20.0git  lib/libLLVMAArch64CodeGen.so.20.0git  lib/libLLVMAArch64AsmParser.so.20.0git  lib/libLLVMAArch64Disassembler.so.20.0git  lib/libLLVMRISCVCodeGen.so.20.0git  lib/libLLVMRISCVAsmParser.so.20.0git  lib/libLLVMRISCVDisassembler.so.20.0git  lib/libLLVMLTO.so.20.0git  lib/libLLVMOption.so.20.0git  lib/libLLVMPasses.so.20.0git  lib/libLLVMAArch64Desc.so.20.0git  lib/libLLVMAArch64Info.so.20.0git  lib/libLLVMAArch64Utils.so.20.0git  lib/libLLVMRISCVDesc.so.20.0git  lib/libLLVMRISCVInfo.so.20.0git  lib/libLLVMObjCARCOpts.so.20.0git  lib/libLLVMBitWriter.so.20.0git  lib/libLLVMProfileData.so.20.0git  lib/libLLVMDebugInfoDWARF.so.20.0git  lib/libLLVMObject.so.20.0git  lib/libLLVMTextAPI.so.20.0git  lib/libLLVMMC.so.20.0git  lib/libLLVMBitReader.so.20.0git  lib/libLLVMCore.so.20.0git  lib/libLLVMBinaryFormat.so.20.0git  lib/libLLVMTargetParser.so.20.0git  lib/libLLVMSupport.so.20.0git  lib/libLLVMDemangle.so.20.0git  -Wl,-rpath-link,/home/worker/bolt-worker2/bolt-x86_64-ubuntu-shared/build/lib && :
ld.lld: error: undefined symbol: llvm::OutlinedHashTree::size(bool) const
>>> referenced by Driver.cpp
>>>               tools/lld/MachO/CMakeFiles/lldMachO.dir/Driver.cpp.o:(lld::macho::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool))

ld.lld: error: undefined symbol: llvm::CodeGenDataWriter::addRecord(llvm::OutlinedHashTreeRecord&)
>>> referenced by Driver.cpp
>>>               tools/lld/MachO/CMakeFiles/lldMachO.dir/Driver.cpp.o:(lld::macho::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool))

ld.lld: error: undefined symbol: llvm::CodeGenDataWriter::write(llvm::raw_fd_ostream&)
>>> referenced by Driver.cpp
>>>               tools/lld/MachO/CMakeFiles/lldMachO.dir/Driver.cpp.o:(lld::macho::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool))

ld.lld: error: undefined symbol: llvm::OutlinedHashTreeRecord::deserialize(unsigned char const*&)
>>> referenced by Driver.cpp
>>>               tools/lld/MachO/CMakeFiles/lldMachO.dir/Driver.cpp.o:(lld::macho::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool))

ld.lld: error: undefined symbol: llvm::OutlinedHashTree::merge(llvm::OutlinedHashTree const*)
>>> referenced by Driver.cpp
>>>               tools/lld/MachO/CMakeFiles/lldMachO.dir/Driver.cpp.o:(lld::macho::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool))
collect2: error: ld returned 1 exit status
16.572 [276/17/1312] Building CXX object tools/clang/lib/Index/CMakeFiles/obj.clangIndex.dir/IndexBody.cpp.o
16.576 [276/16/1313] Building CXX object tools/clang/lib/Frontend/CMakeFiles/obj.clangFrontend.dir/InterfaceStubFunctionsConsumer.cpp.o
16.590 [276/15/1314] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/AnalysisManager.cpp.o
16.641 [276/14/1315] Building CXX object tools/clang/lib/Index/CMakeFiles/obj.clangIndex.dir/IndexTypeSourceInfo.cpp.o
16.739 [276/13/1316] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/BugReporter.cpp.o
16.776 [276/12/1317] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/CallEvent.cpp.o
16.788 [276/11/1318] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/CallDescription.cpp.o
16.803 [276/10/1319] Building CXX object tools/clang/lib/InstallAPI/CMakeFiles/obj.clangInstallAPI.dir/Frontend.cpp.o
16.809 [276/9/1320] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/CheckerContext.cpp.o
16.847 [276/8/1321] Building CXX object tools/clang/lib/InstallAPI/CMakeFiles/obj.clangInstallAPI.dir/Visitor.cpp.o
16.868 [276/7/1322] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/BugReporterVisitors.cpp.o
16.898 [276/6/1323] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/CheckerHelpers.cpp.o
16.908 [276/5/1324] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/BugSuppression.cpp.o
16.951 [276/4/1325] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/CheckerManager.cpp.o
16.978 [276/3/1326] Building CXX object tools/clang/lib/StaticAnalyzer/Core/CMakeFiles/obj.clangStaticAnalyzerCore.dir/CoreEngine.cpp.o
17.732 [276/2/1327] Building CXX object tools/clang/lib/Frontend/CMakeFiles/obj.clangFrontend.dir/CompilerInvocation.cpp.o
18.200 [276/1/1328] Building CXX object tools/clang/lib/Driver/CMakeFiles/obj.clangDriver.dir/ToolChains/HIPUtility.cpp.o

kyulee-com pushed a commit that referenced this pull request Sep 15, 2024
kyulee-com added a commit to kyulee-com/llvm-project that referenced this pull request Sep 15, 2024
It reads raw CG data encoded in the custom section (__llvm_outline) in
object files and merges them into the indexed codegen data file
specified by `-codegen-data-generate-path={path}`.

This depends on llvm#90074.
This is a patch for
https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.
kyulee-com added a commit that referenced this pull request Sep 15, 2024
It reads raw CG data encoded in the custom section (__llvm_outline) in
object files and merges them into the indexed codegen data file
specified by -codegen-data-generate-path={path}.

This depends on #90074.
This is a patch for
https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants