[libc] newheadergen: script adjusted for cmake

[aaryanshukla]

• added entrypoints and headerfile parameters depending on target
• fixed nits in yaml files causing errors
• tested with new cmake config
• cmake patch will be seperate

[llvmbot]

@llvm/pr-subscribers-libc

Author: None (aaryanshukla)

Changes

• added entrypoints and headerfile parameters depending on target
• fixed nits in yaml files causing errors
• tested with new cmake config
• cmake patch will be seperate

---

Full diff: https://github.com/llvm/llvm-project/pull/98825.diff

6 Files Affected:

• (added) libc/newhdrgen/header2.py (+96)
• (added) libc/newhdrgen/yaml/features.yaml (+8)
• (modified) libc/newhdrgen/yaml/pthread.yaml (+3-28)
• (modified) libc/newhdrgen/yaml/sys_random.yaml (+1-1)
• (modified) libc/newhdrgen/yaml/time.yaml (+1-1)
• (modified) libc/newhdrgen/yaml_to_classes.py (+64-27)

diff --git a/libc/newhdrgen/header2.py b/libc/newhdrgen/header2.py
new file mode 100644
index 0000000000000..e115a7e8ef358
--- /dev/null
+++ b/libc/newhdrgen/header2.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+#
+# ===- HeaderFile Class for libc function headers  -----------*- python -*--==#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==-------------------------------------------------------------------------==#
+
+
+class HeaderFile:
+    def __init__(self, name):
+        self.name = name
+        self.macros = []
+        self.types = []
+        self.enumerations = []
+        self.objects = []
+        self.functions = []
+        self.includes = []
+
+    def add_macro(self, macro):
+        self.macros.append(macro)
+
+    def add_type(self, type_):
+        self.types.append(type_)
+
+    def add_enumeration(self, enumeration):
+        self.enumerations.append(enumeration)
+
+    def add_object(self, object):
+        self.objects.append(object)
+
+    def add_function(self, function):
+        self.functions.append(function)
+
+    def add_include(self, include):
+        self.includes.append(include)
+
+    def __str__(self):
+        content = []
+
+        content.append(
+            f"//===-- C standard declarations for {self.name} ------------------------------===//"
+        )
+        content.append("//")
+        content.append(
+            "// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions."
+        )
+        content.append("// See https://llvm.org/LICENSE.txt for license information.")
+        content.append("// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception")
+        content.append("//")
+        content.append(
+            "//===----------------------------------------------------------------------===//\n"
+        )
+
+        header_guard = f"__LLVM_LIBC_DECLARATIONS_{self.name.upper()[:-2]}_H"
+        content.append(f"#ifndef {header_guard}")
+        content.append(f"#define {header_guard}\n")
+
+        content.append("#ifndef __LIBC_ATTRS")
+        content.append("#define __LIBC_ATTRS")
+        content.append("#endif\n")
+
+        content.append("#ifdef __cplusplus")
+        content.append('extern "C" {')
+        content.append("#endif\n")
+
+        for include in self.includes:
+            content.append(str(include))
+
+        for macro in self.macros:
+            content.append(f"{macro}\n")
+
+        for type_ in self.types:
+            content.append(f"{type_}")
+
+        if self.enumerations:
+            combined_enum_content = ",\n  ".join(
+                str(enum) for enum in self.enumerations
+            )
+            content.append(f"\nenum {{\n  {combined_enum_content},\n}};")
+
+        for function in self.functions:
+            content.append(f"{function} __LIBC_ATTRS;\n")
+
+        for object in self.objects:
+            content.append(f"{object} __LIBC_ATTRS;\n")
+
+        content.append("#ifdef __cplusplus")
+        content.append("}")
+        content.append("#endif\n")
+
+        content.append(f"#endif")
+
+        return "\n".join(content)
diff --git a/libc/newhdrgen/yaml/features.yaml b/libc/newhdrgen/yaml/features.yaml
new file mode 100644
index 0000000000000..86bc0acfe89ed
--- /dev/null
+++ b/libc/newhdrgen/yaml/features.yaml
@@ -0,0 +1,8 @@
+header: features.h
+standards: 
+  - stdc
+macros: []
+types: []
+enums: []
+objects: []
+functions: []
diff --git a/libc/newhdrgen/yaml/pthread.yaml b/libc/newhdrgen/yaml/pthread.yaml
index 14a562082d5de..f22767eb1b752 100644
--- a/libc/newhdrgen/yaml/pthread.yaml
+++ b/libc/newhdrgen/yaml/pthread.yaml
@@ -13,33 +13,8 @@ types:
   - type_name: __pthread_start_t
   - type_name: __pthread_once_func_t
   - type_name: __atfork_callback_t
-enums:
-  - name: PTHREAD_CREATE_JOINABLE
-    value: 0x0
-  - name: PTHREAD_CREATE_DETACHED
-    value: 0x1
-  - name: PTHREAD_MUTEX_NORMAL
-    value: 0x0
-  - name: PTHREAD_MUTEX_ERRORCHECK
-    value: 0x1
-  - name: PTHREAD_MUTEX_RECURSIVE
-    value: 0x2
-  - name: PTHREAD_MUTEX_DEFAULT
-    value: 0x0
-  - name: PTHREAD_PROCESS_PRIVATE
-    value: 0x0
-  - name: PTHREAD_PROCESS_SHARED
-    value: 0x1
-  - name: PTHREAD_MUTEX_STALLED
-    value: 0x0
-  - name: PTHREAD_MUTEX_ROBUST
-    value: 0x1
-  - name: PTHREAD_RWLOCK_PREFER_READER_NP
-    value: 0
-  - name: PTHREAD_RWLOCK_PREFER_WRITER_NP
-    value: 1
-  - name: PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP
-    value: 2
+  - type_name: pthread_rwlock_t
+enums: []
 functions:
   - name: pthread_atfork
     standards: 
@@ -184,7 +159,7 @@ functions:
   - name: pthread_exit
     standards: 
       - POSIX
-    return_type: __Noreturn void
+    return_type: _Noreturn void
     arguments:
       - type: void *
   - name: pthread_getname_np
diff --git a/libc/newhdrgen/yaml/sys_random.yaml b/libc/newhdrgen/yaml/sys_random.yaml
index 233fb2c7988cb..6d84056d7dd71 100644
--- a/libc/newhdrgen/yaml/sys_random.yaml
+++ b/libc/newhdrgen/yaml/sys_random.yaml
@@ -4,7 +4,7 @@ types:
   - type_name: ssize_t
   - type_name: size_t
 enums: []
-objects:
+objects: []
 functions:
   - name: getrandom
     standards: 
diff --git a/libc/newhdrgen/yaml/time.yaml b/libc/newhdrgen/yaml/time.yaml
index e7f8de65eeb75..220d4328dbbdb 100644
--- a/libc/newhdrgen/yaml/time.yaml
+++ b/libc/newhdrgen/yaml/time.yaml
@@ -15,7 +15,7 @@ functions:
       - stdc
     return_type: char *
     arguments:
-      - type: struct tm *
+      - type: const struct tm *
   - name: asctime_r
     standard: 
       - stdc
diff --git a/libc/newhdrgen/yaml_to_classes.py b/libc/newhdrgen/yaml_to_classes.py
index 6bccda8e03640..d88ba95ee3d6b 100644
--- a/libc/newhdrgen/yaml_to_classes.py
+++ b/libc/newhdrgen/yaml_to_classes.py
@@ -8,12 +8,11 @@
 #
 # ==-------------------------------------------------------------------------==#
 
-
 import yaml
 import argparse
-
 from pathlib import Path
 from header import HeaderFile
+from header2 import HeaderFile as HeaderFile2
 from class_implementation.classes.macro import Macro
 from class_implementation.classes.type import Type
 from class_implementation.classes.function import Function
@@ -22,18 +21,20 @@
 from class_implementation.classes.object import Object
 
 
-def yaml_to_classes(yaml_data):
+def yaml_to_classes(yaml_data, header_class, entry_points=None):
     """
     Convert YAML data to header classes.
 
     Args:
         yaml_data: The YAML data containing header specifications.
+        header_class: The class to use for creating the header (HeaderFile or HeaderFile2).
+        entry_points: A list of specific function names to include in the header.
 
     Returns:
         HeaderFile: An instance of HeaderFile populated with the data.
     """
     header_name = yaml_data.get("header")
-    header = HeaderFile(header_name)
+    header = header_class(header_name)
 
     for macro_data in yaml_data.get("macros", []):
         header.add_macro(Macro(macro_data["macro_name"], macro_data["macro_value"]))
@@ -49,12 +50,15 @@ def yaml_to_classes(yaml_data):
         )
 
     functions = yaml_data.get("functions", [])
+    if entry_points:
+        entry_points_set = set(entry_points)
+        functions = [f for f in functions if f["name"] in entry_points_set]
     sorted_functions = sorted(functions, key=lambda x: x["name"])
     guards = []
     guarded_function_dict = {}
     for function_data in sorted_functions:
         guard = function_data.get("guard", None)
-        if guard == None:
+        if guard is None:
             arguments = [arg["type"] for arg in function_data["arguments"]]
             attributes = function_data.get("attributes", None)
             standards = function_data.get("standards", None)
@@ -105,19 +109,21 @@ def yaml_to_classes(yaml_data):
     return header
 
 
-def load_yaml_file(yaml_file):
+def load_yaml_file(yaml_file, header_class, entry_points):
     """
     Load YAML file and convert it to header classes.
 
     Args:
-        yaml_file: The path to the YAML file.
+        yaml_file: Path to the YAML file.
+        header_class: The class to use for creating the header (HeaderFile or HeaderFile2).
+        entry_points: A list of specific function names to include in the header.
 
     Returns:
-        HeaderFile: An instance of HeaderFile populated with the data from the YAML file.
+        HeaderFile: An instance of HeaderFile populated with the data.
     """
     with open(yaml_file, "r") as f:
         yaml_data = yaml.safe_load(f)
-    return yaml_to_classes(yaml_data)
+    return yaml_to_classes(yaml_data, header_class, entry_points)
 
 
 def fill_public_api(header_str, h_def_content):
@@ -207,7 +213,14 @@ def increase_indent(self, flow=False, indentless=False):
     print(f"Added function {new_function.name} to {yaml_file}")
 
 
-def main(yaml_file, h_def_file, output_dir, add_function=None):
+def main(
+    yaml_file,
+    output_dir=None,
+    h_def_file=None,
+    add_function=None,
+    entry_points=None,
+    export_decls=False,
+):
     """
     Main function to generate header files from YAML and .h.def templates.
 
@@ -216,41 +229,50 @@ def main(yaml_file, h_def_file, output_dir, add_function=None):
         h_def_file: Path to the .h.def template file.
         output_dir: Directory to output the generated header file.
         add_function: Details of the function to be added to the YAML file (if any).
+        entry_points: A list of specific function names to include in the header.
+        export_decls: Flag to use HeaderFile2 for exporting declarations.
     """
-
     if add_function:
         add_function_to_yaml(yaml_file, add_function)
 
-    header = load_yaml_file(yaml_file)
-
-    with open(h_def_file, "r") as f:
-        h_def_content = f.read()
+    header_class = HeaderFile2 if export_decls else HeaderFile
+    header = load_yaml_file(yaml_file, header_class, entry_points)
 
     header_str = str(header)
-    final_header_content = fill_public_api(header_str, h_def_content)
 
-    output_file_name = Path(h_def_file).stem
-    output_file_path = Path(output_dir) / output_file_name
-
-    with open(output_file_path, "w") as f:
-        f.write(final_header_content)
+    if output_dir:
+        output_file_path = Path(output_dir)
+        if output_file_path.is_dir():
+            output_file_path /= f"{Path(yaml_file).stem}.h"
+    else:
+        output_file_path = Path(f"{Path(yaml_file).stem}.h")
+
+    if not export_decls and h_def_file:
+        with open(h_def_file, "r") as f:
+            h_def_content = f.read()
+        final_header_content = fill_public_api(header_str, h_def_content)
+        with open(output_file_path, "w") as f:
+            f.write(final_header_content)
+    else:
+        with open(output_file_path, "w") as f:
+            f.write(header_str)
 
     print(f"Generated header file: {output_file_path}")
 
 
 if __name__ == "__main__":
-    parser = argparse.ArgumentParser(
-        description="Generate header files from YAML and .h.def templates"
-    )
+    parser = argparse.ArgumentParser(description="Generate header files from YAML")
     parser.add_argument(
         "yaml_file", help="Path to the YAML file containing header specification"
     )
-    parser.add_argument("h_def_file", help="Path to the .h.def template file")
     parser.add_argument(
         "--output_dir",
-        default=".",
         help="Directory to output the generated header file",
     )
+    parser.add_argument(
+        "--h_def_file",
+        help="Path to the .h.def template file (required if not using --export_decls)",
+    )
     parser.add_argument(
         "--add_function",
         nargs=6,
@@ -264,6 +286,21 @@ def main(yaml_file, h_def_file, output_dir, add_function=None):
         ),
         help="Add a function to the YAML file",
     )
+    parser.add_argument(
+        "--e", action="append", help="Entry point to include", dest="entry_points"
+    )
+    parser.add_argument(
+        "--export-decls",
+        action="store_true",
+        help="Flag to use HeaderFile2 for exporting declarations",
+    )
     args = parser.parse_args()
 
-    main(args.yaml_file, args.h_def_file, args.output_dir, args.add_function)
+    main(
+        args.yaml_file,
+        args.output_dir,
+        args.h_def_file,
+        args.add_function,
+        args.entry_points,
+        args.export_decls,
+    )

[michaelrj-google]

this file should have a name and license header that explain that it's for the GPU --export-decls version of headers.

[michaelrj-google]

why did these get removed?

[aaryanshukla]

Was already in the h.def prior so we were duplicating enums.

[michaelrj-google]

for the GPU version we don't need anything but the function and object declarations.

[michaelrj-google]

why is this set made?

[aaryanshukla]

The line after when checking for the names, sets have a faster operation time for checking because we are just seeing if the func name is in the set rather than scanning through all of entry_points everytime

[michaelrj-google]

Overall looking good, just a few final changes

[michaelrj-google]

This class should have a different name from HeaderFile

[michaelrj-google]

You should also use a more descriptive name than HeaderFile2 here.

[michaelrj-google]

LGTM