Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libc++] Fix deployment target Lit features #94791

Merged
merged 4 commits into from
Jun 21, 2024

Conversation

ldionne
Copy link
Member

@ldionne ldionne commented Jun 7, 2024

We were not making any distinction between e.g. the "Apple-flavored" libc++ built from trunk and the system-provided standard library on Apple platforms. For example, any test that would be XFAILed on a back-deployment target would unexpectedly pass when run on that deployment target against the tip of trunk Apple-flavored libc++. In reality, that test would be expected to pass because we're running against the latest libc++, even if it is Apple-flavored.

To solve this issue, we introduce a new feature that describes whether the Standard Library in use is the one provided by the system by default, and that notion is different from the underlying standard library flavor. We also refactor the existing Lit features to make a distinction between availability markup and the library we're running against at runtime, which otherwise limit the flexibility of what we can express in the test suite. Finally, we refactor some of the back-deployment versions that were incorrect (such as thinking that LLVM 10 was introduced in macOS 11, when in reality macOS 11 was synced with LLVM 11).

Fixes #82107

@ldionne ldionne requested review from a team as code owners June 7, 2024 19:17
@llvmbot llvmbot added libc++ libc++ C++ Standard Library. Not GNU libstdc++. Not libc++abi. libc++abi libc++abi C++ Runtime Library. Not libc++. libunwind labels Jun 7, 2024
@llvmbot
Copy link
Collaborator

llvmbot commented Jun 7, 2024

@llvm/pr-subscribers-libunwind

@llvm/pr-subscribers-libcxx

Author: Louis Dionne (ldionne)

Changes

We were not making any distinction between e.g. the "Apple-flavored" libc++ built from trunk and the system-provided standard library on Apple platforms. For example, any test that would be XFAILed on a back-deployment target would unexpectedly pass when run on that deployment target against the tip of trunk Apple-flavored libc++. In reality, that test would be expected to pass because we're running against the latest libc++, even if it is Apple-flavored.

To solve this issue, we introduce a new feature that describes whether the Standard Library in use is the one provided by the system by default, and that notion is different from the underlying standard library flavor.


Patch is 75.03 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/94791.diff

94 Files Affected:

  • (modified) libcxx/test/configs/apple-libc++-backdeployment.cfg.in (+1)
  • (modified) libcxx/test/libcxx/language.support/cxa_deleted_virtual.pass.cpp (+1-1)
  • (modified) libcxx/test/libcxx/strings/basic.string/string.capacity/PR53170.pass.cpp (+1-1)
  • (modified) libcxx/test/libcxx/strings/basic.string/string.capacity/allocation_size.pass.cpp (+1-1)
  • (modified) libcxx/test/libcxx/thread/thread.condition/PR30202_notify_from_pthread_created_thread.pass.cpp (+1-1)
  • (modified) libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.pass.cpp (+1-1)
  • (modified) libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.signals.pass.cpp (+1-1)
  • (modified) libcxx/test/std/depr/depr.c.headers/stdlib_h.aligned_alloc.compile.pass.cpp (+1-1)
  • (modified) libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/generic_category.pass.cpp (+1-1)
  • (modified) libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/system_category.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.mods/refresh.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/file_size.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/hard_link_count.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/last_write_time.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.copy_file/copy_file.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directories/create_directories.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory_with_attributes.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.file_size/file_size.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.remove_all/toctou.pass.cpp (+3-3)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/bool.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/double.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/float.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/int.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/long.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/long_double.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/long_long.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/pointer.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/short.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/unsigned_int.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/unsigned_long.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/unsigned_long_long.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/unsigned_short.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream_extractors/streambuf.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/get.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/get_chart.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/get_pointer_size.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/get_pointer_size_chart.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/get_streambuf.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/get_streambuf_chart.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/getline_pointer_size.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/getline_pointer_size_chart.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/ignore.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/peek.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/read.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/input.streams/istream.unformatted/seekg_off.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minmax_showbase.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/minus1.pass.cpp (+1-1)
  • (modified) libcxx/test/std/input.output/iostreams.base/ios.base/ios.types/ios_Init/ios_Init.multiple.pass.cpp (+1-1)
  • (modified) libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.array/sized_delete_array_fsizeddeallocation.pass.cpp (+1-1)
  • (modified) libcxx/test/std/language.support/support.dynamic/new.delete/new.delete.single/sized_delete_fsizeddeallocation.pass.cpp (+1-1)
  • (modified) libcxx/test/std/language.support/support.exception/uncaught/uncaught_exceptions.pass.cpp (+2-2)
  • (modified) libcxx/test/std/language.support/support.runtime/cstdlib.aligned_alloc.compile.pass.cpp (+1-1)
  • (modified) libcxx/test/std/language.support/support.runtime/ctime.timespec.compile.pass.cpp (+1-1)
  • (modified) libcxx/test/std/localization/codecvt_unicode.pass.cpp (+1-1)
  • (modified) libcxx/test/std/localization/locale.categories/category.ctype/facet.ctype.special/facet.ctype.char.statics/table_size.pass.cpp (+2-2)
  • (modified) libcxx/test/std/localization/locale.categories/category.numeric/locale.num.get/facet.num.get.members/get_long.pass.cpp (+1-1)
  • (modified) libcxx/test/std/localization/locales/locale/locale.cons/name_construction.pass.cpp (+1-1)
  • (modified) libcxx/test/std/numerics/rand/rand.device/ctor.pass.cpp (+2-2)
  • (modified) libcxx/test/std/strings/basic.string/string.capacity/max_size.pass.cpp (+1-1)
  • (modified) libcxx/test/std/strings/basic.string/string.capacity/over_max_size.pass.cpp (+2-2)
  • (modified) libcxx/test/std/strings/basic.string/string.capacity/reserve_size.pass.cpp (+1-1)
  • (modified) libcxx/test/std/thread/futures/futures.async/async_race.38682.pass.cpp (+1-1)
  • (modified) libcxx/test/std/thread/futures/futures.future_error/what.pass.cpp (+1-1)
  • (modified) libcxx/test/std/thread/thread.condition/notify_all_at_thread_exit_lwg3343.pass.cpp (+2-2)
  • (modified) libcxx/test/std/utilities/memory/temporary.buffer/overaligned.pass.cpp (+1-1)
  • (modified) libcxx/utils/ci/buildkite-pipeline.yml (+2-2)
  • (modified) libcxx/utils/ci/run-buildbot (+1-1)
  • (modified) libcxx/utils/libcxx/test/features.py (+8-8)
  • (modified) libcxx/utils/libcxx/test/params.py (+13)
  • (modified) libcxxabi/test/catch_function_01.pass.cpp (+1-1)
  • (modified) libcxxabi/test/catch_function_03.pass.cpp (+1-1)
  • (modified) libcxxabi/test/catch_member_data_pointer_01.pass.cpp (+1-1)
  • (modified) libcxxabi/test/catch_member_function_pointer_02.pass.cpp (+1-1)
  • (modified) libcxxabi/test/catch_member_pointer_nullptr.pass.cpp (+1-1)
  • (modified) libcxxabi/test/catch_multi_level_pointer.pass.cpp (+1-1)
  • (modified) libcxxabi/test/catch_null_pointer_to_object_pr64953.pass.cpp (+2-2)
  • (modified) libcxxabi/test/catch_pointer_nullptr.pass.cpp (+1-1)
  • (modified) libcxxabi/test/catch_ptr_02.pass.cpp (+1-1)
  • (modified) libcxxabi/test/configs/apple-libc++abi-backdeployment.cfg.in (+1)
  • (modified) libcxxabi/test/cxa_vec_new_overflow_PR41395.pass.cpp (+1-1)
  • (modified) libcxxabi/test/dynamic_cast.pass.cpp (+2-2)
  • (modified) libcxxabi/test/exception_object_alignment.pass.cpp (+1-1)
  • (modified) libcxxabi/test/forced_unwind1.pass.cpp (+2-2)
  • (modified) libcxxabi/test/forced_unwind2.pass.cpp (+2-2)
  • (modified) libcxxabi/test/incomplete_type.sh.cpp (+1-1)
  • (modified) libcxxabi/test/test_aux_runtime_op_array_new.pass.cpp (+1-1)
  • (modified) libcxxabi/test/test_demangle.pass.cpp (+3-3)
  • (modified) libcxxabi/test/test_exception_address_alignment.pass.cpp (+1-1)
  • (modified) libcxxabi/test/uncaught_exception.pass.cpp (+2-2)
  • (modified) libcxxabi/test/uncaught_exceptions.pass.cpp (+1-1)
  • (modified) libunwind/test/configs/apple-libunwind-backdeployment.cfg.in (+1)
  • (modified) libunwind/test/libunwind_01.pass.cpp (+1-1)
diff --git a/libcxx/test/configs/apple-libc++-backdeployment.cfg.in b/libcxx/test/configs/apple-libc++-backdeployment.cfg.in
index 42594461f10e7..9843c4a9ad70d 100644
--- a/libcxx/test/configs/apple-libc++-backdeployment.cfg.in
+++ b/libcxx/test/configs/apple-libc++-backdeployment.cfg.in
@@ -55,6 +55,7 @@ config.substitutions.append(('%{exec}',
 ))
 
 config.stdlib = 'apple-libc++'
+config.using_system_stdlib = True
 
 libcxx.test.config.configure(
     libcxx.test.params.DEFAULT_PARAMETERS + BACKDEPLOYMENT_PARAMETERS,
diff --git a/libcxx/test/libcxx/language.support/cxa_deleted_virtual.pass.cpp b/libcxx/test/libcxx/language.support/cxa_deleted_virtual.pass.cpp
index 723cd9f33b263..11b72fa9ed401 100644
--- a/libcxx/test/libcxx/language.support/cxa_deleted_virtual.pass.cpp
+++ b/libcxx/test/libcxx/language.support/cxa_deleted_virtual.pass.cpp
@@ -11,7 +11,7 @@
 // Test exporting the symbol: "__cxa_deleted_virtual" in macosx
 // But don't expect the symbol to be exported in previous versions.
 //
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12|13}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12|13}}
 
 struct S { virtual void f() = delete; virtual ~S() {} };
 int main(int, char**) {
diff --git a/libcxx/test/libcxx/strings/basic.string/string.capacity/PR53170.pass.cpp b/libcxx/test/libcxx/strings/basic.string/string.capacity/PR53170.pass.cpp
index 8415214efa8b8..2b46d4bc2fb08 100644
--- a/libcxx/test/libcxx/strings/basic.string/string.capacity/PR53170.pass.cpp
+++ b/libcxx/test/libcxx/strings/basic.string/string.capacity/PR53170.pass.cpp
@@ -22,7 +22,7 @@
 // Reported as https://llvm.org/PR53170.
 
 // reserve(n) used to shrink the string until https://llvm.org/D117332 was shipped.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.9|10.10|10.11|10.12|10.13|10.14|10.15|11.0|12.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.9|10.10|10.11|10.12|10.13|10.14|10.15|11.0|12.0}}
 
 #include <string>
 #include <stdexcept>
diff --git a/libcxx/test/libcxx/strings/basic.string/string.capacity/allocation_size.pass.cpp b/libcxx/test/libcxx/strings/basic.string/string.capacity/allocation_size.pass.cpp
index 1110e3d3ec568..7ece456acd553 100644
--- a/libcxx/test/libcxx/strings/basic.string/string.capacity/allocation_size.pass.cpp
+++ b/libcxx/test/libcxx/strings/basic.string/string.capacity/allocation_size.pass.cpp
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.13|10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.13|10.15|11.0}}
 
 // <string>
 
diff --git a/libcxx/test/libcxx/thread/thread.condition/PR30202_notify_from_pthread_created_thread.pass.cpp b/libcxx/test/libcxx/thread/thread.condition/PR30202_notify_from_pthread_created_thread.pass.cpp
index b1a3f86e86437..b7652b074bde8 100644
--- a/libcxx/test/libcxx/thread/thread.condition/PR30202_notify_from_pthread_created_thread.pass.cpp
+++ b/libcxx/test/libcxx/thread/thread.condition/PR30202_notify_from_pthread_created_thread.pass.cpp
@@ -14,7 +14,7 @@
 // UNSUPPORTED: c++03
 
 // PR30202 was fixed starting in macosx10.13.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12}}
 
 // <condition_variable>
 
diff --git a/libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.pass.cpp b/libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.pass.cpp
index 47741d0851e89..6f08dde2f3132 100644
--- a/libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.pass.cpp
+++ b/libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.pass.cpp
@@ -11,7 +11,7 @@
 // Until 58a0a70fb2f1, this_thread::sleep_for could sometimes get interrupted
 // by signals and this test would fail spuriously. Disable the test on the
 // corresponding system libraries.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11}}
 
 // ALLOW_RETRIES: 3
 
diff --git a/libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.signals.pass.cpp b/libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.signals.pass.cpp
index ad62e0c113281..12376695fac34 100644
--- a/libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.signals.pass.cpp
+++ b/libcxx/test/libcxx/thread/thread.threads/thread.thread.this/sleep_for.signals.pass.cpp
@@ -14,7 +14,7 @@
 // Until 58a0a70fb2f1, this_thread::sleep_for misbehaves when interrupted by
 // a signal, as tested here. Disable the test on the corresponding system
 // libraries.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11}}
 
 // ALLOW_RETRIES: 3
 
diff --git a/libcxx/test/std/depr/depr.c.headers/stdlib_h.aligned_alloc.compile.pass.cpp b/libcxx/test/std/depr/depr.c.headers/stdlib_h.aligned_alloc.compile.pass.cpp
index c0409307a8f4f..7ddf03fff4f4e 100644
--- a/libcxx/test/std/depr/depr.c.headers/stdlib_h.aligned_alloc.compile.pass.cpp
+++ b/libcxx/test/std/depr/depr.c.headers/stdlib_h.aligned_alloc.compile.pass.cpp
@@ -13,7 +13,7 @@
 
 // ::aligned_alloc is provided by the C library, but it's marked as unavailable
 // until macOS 10.15
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
 
 // ::aligned_alloc is not implemented on Windows
 // XFAIL: target={{.+}}-windows-{{.+}}
diff --git a/libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/generic_category.pass.cpp b/libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/generic_category.pass.cpp
index 7283fdc769d86..35c2726fda2a0 100644
--- a/libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/generic_category.pass.cpp
+++ b/libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/generic_category.pass.cpp
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12}}
 
 // <system_error>
 
diff --git a/libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/system_category.pass.cpp b/libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/system_category.pass.cpp
index 02a1baf599983..0f622289248b7 100644
--- a/libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/system_category.pass.cpp
+++ b/libcxx/test/std/diagnostics/syserr/syserr.errcat/syserr.errcat.objects/system_category.pass.cpp
@@ -12,7 +12,7 @@
 
 // const error_category& system_category();
 
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12}}
 
 #include <system_error>
 #include <cassert>
diff --git a/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.mods/refresh.pass.cpp b/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.mods/refresh.pass.cpp
index 4f5002e068c19..f2d5e7dd0e563 100644
--- a/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.mods/refresh.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.mods/refresh.pass.cpp
@@ -11,7 +11,7 @@
 
 // The string reported on errors changed, which makes those tests fail when run
 // against already-released libc++'s.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // <filesystem>
 
diff --git a/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/file_size.pass.cpp b/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/file_size.pass.cpp
index e1407858f66ac..11295855d3ca0 100644
--- a/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/file_size.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/file_size.pass.cpp
@@ -11,7 +11,7 @@
 
 // The string reported on errors changed, which makes those tests fail when run
 // against already-released libc++'s.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // Starting in Android N (API 24), SELinux policy prevents the shell user from
 // creating a FIFO file.
diff --git a/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/hard_link_count.pass.cpp b/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/hard_link_count.pass.cpp
index 514bbb27c2c47..f40b06e106e12 100644
--- a/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/hard_link_count.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/hard_link_count.pass.cpp
@@ -11,7 +11,7 @@
 
 // The string reported on errors changed, which makes those tests fail when run
 // against already-released libc++'s.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // Starting in Android N (API 24), SELinux policy prevents the shell user from
 // creating a hard link.
diff --git a/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/last_write_time.pass.cpp b/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/last_write_time.pass.cpp
index 41e2ee668f525..72039bb06921f 100644
--- a/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/last_write_time.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/class.directory_entry/directory_entry.obs/last_write_time.pass.cpp
@@ -11,7 +11,7 @@
 
 // The string reported on errors changed, which makes those tests fail when run
 // against already-released libc++'s.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // <filesystem>
 
diff --git a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.copy_file/copy_file.pass.cpp b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.copy_file/copy_file.pass.cpp
index e81beed4f2adf..3209e16d00612 100644
--- a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.copy_file/copy_file.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.copy_file/copy_file.pass.cpp
@@ -12,7 +12,7 @@
 
 // The string reported on errors changed, which makes those tests fail when run
 // against already-released libc++'s.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // Starting in Android N (API 24), SELinux policy prevents the shell user from
 // creating a FIFO file.
diff --git a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directories/create_directories.pass.cpp b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directories/create_directories.pass.cpp
index 9955e08710c56..3658a24b72b28 100644
--- a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directories/create_directories.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directories/create_directories.pass.cpp
@@ -12,7 +12,7 @@
 // UNSUPPORTED: availability-filesystem-missing
 
 // This test requires the dylib support introduced in D92769.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // <filesystem>
 
diff --git a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory.pass.cpp b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory.pass.cpp
index 21f283cae86d5..cdedf0d461260 100644
--- a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory.pass.cpp
@@ -12,7 +12,7 @@
 // UNSUPPORTED: availability-filesystem-missing
 
 // This test requires the dylib support introduced in http://llvm.org/D92769.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // <filesystem>
 
diff --git a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory_with_attributes.pass.cpp b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory_with_attributes.pass.cpp
index a0ec52643e8c0..dee6690997fb1 100644
--- a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory_with_attributes.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.create_directory/create_directory_with_attributes.pass.cpp
@@ -12,7 +12,7 @@
 // UNSUPPORTED: availability-filesystem-missing
 
 // This test requires the dylib support introduced in http://llvm.org/D92769.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // <filesystem>
 
diff --git a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.file_size/file_size.pass.cpp b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.file_size/file_size.pass.cpp
index 99d303bc4244d..648598bb10124 100644
--- a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.file_size/file_size.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.file_size/file_size.pass.cpp
@@ -13,7 +13,7 @@
 
 // The string reported on errors changed, which makes those tests fail when run
 // against already-released libc++'s.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // <filesystem>
 
diff --git a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp
index d43bc5eea5121..459224d41953a 100644
--- a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.last_write_time/last_write_time.pass.cpp
@@ -13,7 +13,7 @@
 
 // The string reported on errors changed, which makes those tests fail when run
 // against already-released libc++'s.
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx{{10.15|11.0}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx{{10.15|11.0}}
 
 // <filesystem>
 
diff --git a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.remove_all/toctou.pass.cpp b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.remove_all/toctou.pass.cpp
index 5248ba24b4038..5335f8070ffa4 100644
--- a/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.remove_all/toctou.pass.cpp
+++ b/libcxx/test/std/input.output/filesystems/fs.op.funcs/fs.op.remove_all/toctou.pass.cpp
@@ -31,9 +31,9 @@
 
 // This test requires a dylib containing the fix shipped in https://reviews.llvm.org/D118134.
 // We use UNSUPPORTED instead of XFAIL because the test might not fail reliably.
-// UNSUPPORTED: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14|15}}
-// UNSUPPORTED: stdlib=apple-libc++ && target={{.+}}-apple-macosx11.0
-// UNSUPPORTED: stdlib=apple-libc++ && target={{.+}}-apple-macosx12.{{0|1|2}}
+// UNSUPPORTED: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14|15}}
+// UNSUPPORTED: stdlib=system && target={{.+}}-apple-macosx11.0
+// UNSUPPORTED: stdlib=system && target={{.+}}-apple-macosx12.{{0|1|2}}
 
 // Windows doesn't support the necessary APIs to mitigate this issue.
 // XFAIL: target={{.+}}-windows-{{.+}}
diff --git a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/bool.pass.cpp b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/bool.pass.cpp
index c7d7a4e1f83dd..4e915d340b31b 100644
--- a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/bool.pass.cpp
+++ b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/bool.pass.cpp
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
 
 // <istream>
 
diff --git a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/double.pass.cpp b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/double.pass.cpp
index 7bb793082ee93..c7f2476cf82f0 100644
--- a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/double.pass.cpp
+++ b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/double.pass.cpp
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
 
 // <istream>
 
diff --git a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/float.pass.cpp b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/float.pass.cpp
index cf4070a569a8f..1acc771a6af11 100644
--- a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/float.pass.cpp
+++ b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/float.pass.cpp
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
 
 // <istream>
 
diff --git a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/int.pass.cpp b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/int.pass.cpp
index 6bfd64abd41f6..dfd26975840ee 100644
--- a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/int.pass.cpp
+++ b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/int.pass.cpp
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
+// XFAIL: stdlib=system && target={{.+}}-apple-macosx10.{{9|10|11|12|13|14}}
 
 // <istream>
 
diff --git a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/long.pass.cpp b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/long.pass.cpp
index 8eca4d4299105..440afad4afb1b 100644
--- a/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/long.pass.cpp
+++ b/libcxx/test/std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/long.pass.cpp
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: stdlib=apple-libc++ && target={{.+}}-apple...
[truncated]

Copy link

github-actions bot commented Jun 7, 2024

✅ With the latest revision this PR passed the C/C++ code formatter.

@ldionne ldionne force-pushed the review/fix-backdeployment-features branch from db6ebcc to 783f282 Compare June 18, 2024 01:22
We were not making any distinction between e.g. the "Apple-flavored"
libc++ built from trunk and the system-provided standard library on
Apple platforms. For example, any test that would be XFAILed on a
back-deployment target would unexpectedly pass when run on that
deployment target against the tip of trunk Apple-flavored libc++.
In reality, that test would be expected to pass because we're running
against the latest libc++, even if it is Apple-flavored.

To solve this issue, we introduce a new feature that describes whether
the Standard Library in use is the one provided by the system by default,
and that notion is different from the underlying standard library flavor.
@ldionne ldionne force-pushed the review/fix-backdeployment-features branch from 783f282 to bf507d2 Compare June 18, 2024 14:57
Copy link

github-actions bot commented Jun 18, 2024

⚠️ Python code formatter, darker found issues in your code. ⚠️

You can test this locally with the following command:
darker --check --diff -r 65b0301943e64d7841e11f047a1a9fbd15f28037...36a6dcb6beea56e7c0de53a5f255d9e3f09256cf libcxx/utils/libcxx/test/features.py libcxx/utils/libcxx/test/params.py
View the diff from darker here.
--- features.py	2024-06-21 11:56:45.000000 +0000
+++ features.py	2024-06-21 11:59:19.761163 +0000
@@ -687,18 +687,18 @@
 # For example, tests for availability markup should be expected to produce diagnostics
 # regardless of whether we're running against a system library, as long as we're using
 # a libc++ flavor that enables availability markup. Similarly, a test could fail when
 # run against the system library of an older version of FreeBSD, even though FreeBSD
 # doesn't provide availability markup at the time of writing this.
-for version in ('11', '12', '13', '14', '15', '16', '17', '18', '19'):
+for version in ("11", "12", "13", "14", "15", "16", "17", "18", "19"):
     DEFAULT_FEATURES.append(
         Feature(
             name="using-built-library-before-llvm-{}".format(version),
             when=lambda cfg: BooleanExpression.evaluate(
                 "stdlib=system && _target-before-llvm-{}".format(version),
                 cfg.available_features,
-            )
+            ),
         )
     )
 
 DEFAULT_FEATURES += [
     # Tests that require std::filesystem support in the built library

@ldionne ldionne force-pushed the review/fix-backdeployment-features branch from bf507d2 to 5f74799 Compare June 18, 2024 18:13
@ldionne ldionne merged commit db8c7e0 into llvm:main Jun 21, 2024
4 of 6 checks passed
@ldionne ldionne deleted the review/fix-backdeployment-features branch June 21, 2024 14:31
AlexisPerry pushed a commit to llvm-project-tlp/llvm-project that referenced this pull request Jul 9, 2024
We were not making any distinction between e.g. the "Apple-flavored"
libc++ built from trunk and the system-provided standard library on
Apple platforms. For example, any test that would be XFAILed on a
back-deployment target would unexpectedly pass when run on that
deployment target against the tip of trunk Apple-flavored libc++. In
reality, that test would be expected to pass because we're running
against the latest libc++, even if it is Apple-flavored.

To solve this issue, we introduce a new feature that describes whether
the Standard Library in use is the one provided by the system by
default, and that notion is different from the underlying standard
library flavor. We also refactor the existing Lit features to make a
distinction between availability markup and the library we're running
against at runtime, which otherwise limit the flexibility of what we can
express in the test suite. Finally, we refactor some of the
back-deployment versions that were incorrect (such as thinking that LLVM
10 was introduced in macOS 11, when in reality macOS 11 was synced with
LLVM 11).

Fixes llvm#82107
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
libc++abi libc++abi C++ Runtime Library. Not libc++. libc++ libc++ C++ Standard Library. Not GNU libstdc++. Not libc++abi. libunwind
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate backdeployment tests.
3 participants