[Intrinsics][PreISelInstrinsicLowering] llvm.memcpy.inline length no longer needs to be constant #98281

asb · 2024-07-10T08:09:43Z

Following on from the discussion in
https://discourse.llvm.org/t/rfc-introducing-an-llvm-memset-pattern-inline-intrinsic/79496 and the equivalent change for llvm.memset.inline (#95397), this removes the requirement that the length of llvm.memcpy.inline is constant. PreISelInstrinsicLowering will expand llvm.memcpy.inline with non-constant lengths, while the codegen path for constant lengths is left unaltered.

llvmbot · 2024-07-10T08:10:12Z

@llvm/pr-subscribers-llvm-transforms
@llvm/pr-subscribers-llvm-ir

@llvm/pr-subscribers-llvm-analysis

Author: Alex Bradbury (asb)

Changes

Following on from the discussion in
https://discourse.llvm.org/t/rfc-introducing-an-llvm-memset-pattern-inline-intrinsic/79496 and the equivalent change for llvm.memset.inline (#95397), this removes the requirement that the length of llvm.memcpy.inline is constant. PreISelInstrinsicLowering will expand llvm.memcpy.inline with non-constant lengths, while the codegen path for constant lengths is left unaltered.

Full diff: https://github.com/llvm/llvm-project/pull/98281.diff

7 Files Affected:

(modified) llvm/docs/LangRef.rst (+1-1)
(modified) llvm/include/llvm/IR/IntrinsicInst.h (-3)
(modified) llvm/include/llvm/IR/Intrinsics.td (+1-2)
(modified) llvm/lib/Analysis/Lint.cpp (+2-18)
(modified) llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp (+16)
(added) llvm/test/Transforms/PreISelIntrinsicLowering/X86/memcpy-inline-non-constant-len.ll (+34)
(modified) llvm/test/Verifier/intrinsic-immarg.ll (-8)

diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index ae39217dc8ff8..21ded802ebdf0 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -15026,7 +15026,7 @@ Arguments:
 """"""""""
 
 The first argument is a pointer to the destination, the second is a
-pointer to the source. The third argument is a constant integer argument
+pointer to the source. The third argument is an integer argument
 specifying the number of bytes to copy, and the fourth is a
 boolean indicating a volatile access.
 
diff --git a/llvm/include/llvm/IR/IntrinsicInst.h b/llvm/include/llvm/IR/IntrinsicInst.h
index a2ecf625ff61a..fe3f92da400f8 100644
--- a/llvm/include/llvm/IR/IntrinsicInst.h
+++ b/llvm/include/llvm/IR/IntrinsicInst.h
@@ -1296,9 +1296,6 @@ class MemMoveInst : public MemTransferInst {
 /// This class wraps the llvm.memcpy.inline intrinsic.
 class MemCpyInlineInst : public MemCpyInst {
 public:
-  ConstantInt *getLength() const {
-    return cast<ConstantInt>(MemCpyInst::getLength());
-  }
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const IntrinsicInst *I) {
     return I->getIntrinsicID() == Intrinsic::memcpy_inline;
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 65a9b68b5229d..615ece332a1d6 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -966,7 +966,6 @@ def int_memcpy  : Intrinsic<[],
 // Memcpy semantic that is guaranteed to be inlined.
 // In particular this means that the generated code is not allowed to call any
 // external function.
-// The third argument (specifying the size) must be a constant.
 def int_memcpy_inline
     : Intrinsic<[],
       [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty, llvm_i1_ty],
@@ -974,7 +973,7 @@ def int_memcpy_inline
        NoCapture<ArgIndex<0>>, NoCapture<ArgIndex<1>>,
        NoAlias<ArgIndex<0>>, NoAlias<ArgIndex<1>>,
        WriteOnly<ArgIndex<0>>, ReadOnly<ArgIndex<1>>,
-       ImmArg<ArgIndex<2>>, ImmArg<ArgIndex<3>>]>;
+       ImmArg<ArgIndex<3>>]>;
 
 def int_memmove : Intrinsic<[],
                             [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
diff --git a/llvm/lib/Analysis/Lint.cpp b/llvm/lib/Analysis/Lint.cpp
index 496308a0c247a..a44d5a3bbe462 100644
--- a/llvm/lib/Analysis/Lint.cpp
+++ b/llvm/lib/Analysis/Lint.cpp
@@ -290,7 +290,8 @@ void Lint::visitCallBase(CallBase &I) {
 
       // TODO: Check more intrinsics
 
-    case Intrinsic::memcpy: {
+    case Intrinsic::memcpy:
+    case Intrinsic::memcpy_inline: {
       MemCpyInst *MCI = cast<MemCpyInst>(&I);
       visitMemoryReference(I, MemoryLocation::getForDest(MCI),
                            MCI->getDestAlign(), nullptr, MemRef::Write);
@@ -311,23 +312,6 @@ void Lint::visitCallBase(CallBase &I) {
             "Undefined behavior: memcpy source and destination overlap", &I);
       break;
     }
-    case Intrinsic::memcpy_inline: {
-      MemCpyInlineInst *MCII = cast<MemCpyInlineInst>(&I);
-      const uint64_t Size = MCII->getLength()->getValue().getLimitedValue();
-      visitMemoryReference(I, MemoryLocation::getForDest(MCII),
-                           MCII->getDestAlign(), nullptr, MemRef::Write);
-      visitMemoryReference(I, MemoryLocation::getForSource(MCII),
-                           MCII->getSourceAlign(), nullptr, MemRef::Read);
-
-      // Check that the memcpy arguments don't overlap. The AliasAnalysis API
-      // isn't expressive enough for what we really want to do. Known partial
-      // overlap is not distinguished from the case where nothing is known.
-      const LocationSize LS = LocationSize::precise(Size);
-      Check(AA->alias(MCII->getSource(), LS, MCII->getDest(), LS) !=
-                AliasResult::MustAlias,
-            "Undefined behavior: memcpy source and destination overlap", &I);
-      break;
-    }
     case Intrinsic::memmove: {
       MemMoveInst *MMI = cast<MemMoveInst>(&I);
       visitMemoryReference(I, MemoryLocation::getForDest(MMI),
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index 8572cdc160456..19950f3eb67ba 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -230,6 +230,21 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(Function &F) const {
 
       break;
     }
+    case Intrinsic::memcpy_inline: {
+      // Only expand llvm.memcpy.inline with non-constant length in this
+      // codepath, leaving the current SelectionDAG expansion for constant
+      // length memcpy intrinsics undisturbed.
+      auto *Memcpy = cast<MemCpyInlineInst>(Inst);
+      if (isa<ConstantInt>(Memcpy->getLength()))
+        break;
+
+      Function *ParentFunc = Memcpy->getFunction();
+      const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
+      expandMemCpyAsLoop(Memcpy, TTI);
+      Changed = true;
+      Memcpy->eraseFromParent();
+      break;
+    }
     case Intrinsic::memmove: {
       auto *Memmove = cast<MemMoveInst>(Inst);
       Function *ParentFunc = Memmove->getFunction();
@@ -291,6 +306,7 @@ bool PreISelIntrinsicLowering::lowerIntrinsics(Module &M) const {
     default:
       break;
     case Intrinsic::memcpy:
+    case Intrinsic::memcpy_inline:
     case Intrinsic::memmove:
     case Intrinsic::memset:
     case Intrinsic::memset_inline:
diff --git a/llvm/test/Transforms/PreISelIntrinsicLowering/X86/memcpy-inline-non-constant-len.ll b/llvm/test/Transforms/PreISelIntrinsicLowering/X86/memcpy-inline-non-constant-len.ll
new file mode 100644
index 0000000000000..5fe1f55ec272b
--- /dev/null
+++ b/llvm/test/Transforms/PreISelIntrinsicLowering/X86/memcpy-inline-non-constant-len.ll
@@ -0,0 +1,34 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
+; RUN: opt -mtriple=x86_64-pc-linux-gnu -passes=pre-isel-intrinsic-lowering -S -o - %s | FileCheck %s
+
+; Constant length memcpy.inline should be left unmodified.
+define void @memcpy_32(ptr %dst, ptr %src) nounwind {
+; CHECK-LABEL: define void @memcpy_32(
+; CHECK-SAME: ptr [[DST:%.*]], ptr [[SRC:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    tail call void @llvm.memcpy.inline.p0.p0.i64(ptr [[DST]], ptr [[SRC]], i64 32, i1 false)
+; CHECK-NEXT:    ret void
+;
+  tail call void @llvm.memcpy.inline.p0.p0.i64(ptr %dst, ptr %src, i64 32, i1 0)
+  ret void
+}
+
+define void @memcpy_x(ptr %dst, ptr %src, i64 %x) nounwind {
+; CHECK-LABEL: define void @memcpy_x(
+; CHECK-SAME: ptr [[DST:%.*]], ptr [[SRC:%.*]], i64 [[X:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp ne i64 [[X]], 0
+; CHECK-NEXT:    br i1 [[TMP1]], label %[[LOOP_MEMCPY_EXPANSION:.*]], label %[[POST_LOOP_MEMCPY_EXPANSION:.*]]
+; CHECK:       [[LOOP_MEMCPY_EXPANSION]]:
+; CHECK-NEXT:    [[LOOP_INDEX:%.*]] = phi i64 [ 0, [[TMP0:%.*]] ], [ [[TMP5:%.*]], %[[LOOP_MEMCPY_EXPANSION]] ]
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i8, ptr [[SRC]], i64 [[LOOP_INDEX]]
+; CHECK-NEXT:    [[TMP3:%.*]] = load i8, ptr [[TMP2]], align 1
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i8, ptr [[DST]], i64 [[LOOP_INDEX]]
+; CHECK-NEXT:    store i8 [[TMP3]], ptr [[TMP4]], align 1
+; CHECK-NEXT:    [[TMP5]] = add i64 [[LOOP_INDEX]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = icmp ult i64 [[TMP5]], [[X]]
+; CHECK-NEXT:    br i1 [[TMP6]], label %[[LOOP_MEMCPY_EXPANSION]], label %[[POST_LOOP_MEMCPY_EXPANSION]]
+; CHECK:       [[POST_LOOP_MEMCPY_EXPANSION]]:
+; CHECK-NEXT:    ret void
+;
+  tail call void @llvm.memcpy.inline.p0.p0.i64(ptr %dst, ptr %src, i64 %x, i1 0)
+  ret void
+}
diff --git a/llvm/test/Verifier/intrinsic-immarg.ll b/llvm/test/Verifier/intrinsic-immarg.ll
index b1b9f7ee4be11..ad70b17e5fb72 100644
--- a/llvm/test/Verifier/intrinsic-immarg.ll
+++ b/llvm/test/Verifier/intrinsic-immarg.ll
@@ -36,14 +36,6 @@ define void @memcpy_inline_is_volatile(ptr %dest, ptr %src, i1 %is.volatile) {
   ret void
 }
 
-define void @memcpy_inline_variable_size(ptr %dest, ptr %src, i32 %size) {
-  ; CHECK: immarg operand has non-immediate parameter
-  ; CHECK-NEXT: i32 %size
-  ; CHECK-NEXT: call void @llvm.memcpy.inline.p0.p0.i32(ptr %dest, ptr %src, i32 %size, i1 true)
-  call void @llvm.memcpy.inline.p0.p0.i32(ptr %dest, ptr %src, i32 %size, i1 true)
-  ret void
-}
-
 declare void @llvm.memmove.p0.p0.i32(ptr nocapture, ptr nocapture, i32, i1)
 define void @memmove(ptr %dest, ptr %src, i1 %is.volatile) {
   ; CHECK: immarg operand has non-immediate parameter

llvm/test/Transforms/PreISelIntrinsicLowering/X86/memcpy-inline-non-constant-len.ll

….inline is preserved Equivalent change ot the one requested in the review for #98281.

… doesn't drop volatile flag As suggested in the discussion for #98281.

….inline is preserved Equivalent change ot the one requested in the review for llvm#98281.

… doesn't drop volatile flag As suggested in the discussion for llvm#98281.

…longer needs to be constant Following on from the discussion in https://discourse.llvm.org/t/rfc-introducing-an-llvm-memset-pattern-inline-intrinsic/79496 and the equivalent change for llvm.memset.inline (llvm#95397), this removes the requirement that the length of llvm.memcpy.inline is constant. PreISelInstrinsicLowering will expand llvm.memcpy.inline with non-constant lengths, while the codegen path for constant lengths is left unaltered.

llvm-ci · 2024-07-16T13:19:52Z

LLVM Buildbot has detected a new failure on builder mlir-rocm-mi200 running on mi200-buildbot while building llvm at step 6 "test-build-check-mlir-build-only-check-mlir".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/177/builds/1673