Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HLSL][SPIRV][DXIL] Implement WaveActiveSum intrinsic #112400

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[HLSL][SPIRV][DXIL] Implement WaveActiveSum intrinsic #112400

wants to merge 1 commit into from
+495 −0

Conversation

inbelic
Copy link
Contributor

@inbelic inbelic commented Oct 15, 2024 •
edited
Loading 

      - link builtin in hlsl_intrinsics
      - add codegen for spirv intrinsic and two directx intrinsics to retain
        signedness information of the operands in CGBuiltin.cpp
      - add semantic analysis in SemaHLSL.cpp
      - add lowering of spirv intrinsic to spirv backend in
        SPIRVInstructionSelector.cpp
      - add lowering of directx intrinsics to WaveActiveOp dxil op in
    DXIL.td

      - add test cases to illustrate passespendent pr merges.

Resolves #70106

@inbelic inbelic force-pushed the inbelic/wave-active-sum branch 2 times, most recently from c541955 to f214ba6 Compare October 16, 2024 21:47
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 16, 2024 •
edited
Loading

✅ With the latest revision this PR passed the C/C++ code formatter.

@inbelic inbelic force-pushed the inbelic/wave-active-sum branch 5 times, most recently from 751f953 to aa2ae99 Compare October 18, 2024 18:12
@inbelic inbelic changed the title [HLSL] Implement WaveActiveSum intrinsic [HLSL][SPIRV] Implement WaveActiveSum intrinsic Oct 18, 2024
@inbelic inbelic marked this pull request as ready for review October 18, 2024 21:09
@llvmbot llvmbot added clang Clang issues not falling into any other category backend:X86 clang:frontend Language frontend issues, e.g. anything involving "Sema" clang:headers Headers provided by Clang, e.g. for intrinsics clang:codegen backend:DirectX HLSL HLSL Language Support backend:SPIR-V llvm:ir labels Oct 18, 2024
@llvmbot
Copy link
Collaborator

llvmbot commented Oct 18, 2024 •
edited
Loading

@llvm/pr-subscribers-llvm-ir
@llvm/pr-subscribers-backend-directx

@llvm/pr-subscribers-hlsl

Author: Finn Plummer (inbelic)

Changes 
  - add clang builtin to Builtins.td
  - link builtin in hlsl_intrinsics
  - add codegen for spirv intrinsic and two directx intrinsics to retain
    signedness information of the operands in CGBuiltin.cpp
  - add semantic analysis in SemaHLSL.cpp
  - add lowering of spirv intrinsic to spirv backend in
    SPIRVInstructionSelector.cpp

  - add test cases to illustrate passes

Note that this defines the dx intrinsics but does not implement the
DirectX lowering to DXIL. This will implemented in a second pr when
the dependent pr merges.

Part 1 of #70106

Patch is 21.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/112400.diff

11 Files Affected:

  • (modified) clang/include/clang/Basic/Builtins.td (+6)
  • (modified) clang/include/clang/Basic/DiagnosticSemaKinds.td (+3)
  • (modified) clang/lib/CodeGen/CGBuiltin.cpp (+34)
  • (modified) clang/lib/Headers/hlsl/hlsl_intrinsics.h (+99)
  • (modified) clang/lib/Sema/SemaHLSL.cpp (+31)
  • (added) clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl (+48)
  • (added) clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl (+28)
  • (modified) llvm/include/llvm/IR/IntrinsicsDirectX.td (+2)
  • (modified) llvm/include/llvm/IR/IntrinsicsSPIRV.td (+1)
  • (modified) llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp (+33)
  • (added) llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveSum.ll (+41)
diff --git a/clang/include/clang/Basic/Builtins.td b/clang/include/clang/Basic/Builtins.td
index 382fb6b7a3c031..7205f10b01c2b9 100644
--- a/clang/include/clang/Basic/Builtins.td
+++ b/clang/include/clang/Basic/Builtins.td
@@ -4749,6 +4749,12 @@ def HLSLWaveActiveCountBits : LangBuiltin<"HLSL_LANG"> {
   let Prototype = "unsigned int(bool)";
 }
 
+def HLSLWaveActiveSum : LangBuiltin<"HLSL_LANG"> {
+  let Spellings = ["__builtin_hlsl_wave_active_sum"];
+  let Attributes = [NoThrow, Const];
+  let Prototype = "void (...)";
+}
+
 def HLSLWaveGetLaneIndex : LangBuiltin<"HLSL_LANG"> {
   let Spellings = ["__builtin_hlsl_wave_get_lane_index"];
   let Attributes = [NoThrow, Const];
diff --git a/clang/include/clang/Basic/DiagnosticSemaKinds.td b/clang/include/clang/Basic/DiagnosticSemaKinds.td
index c458a62d9be48c..95093723c2f495 100644
--- a/clang/include/clang/Basic/DiagnosticSemaKinds.td
+++ b/clang/include/clang/Basic/DiagnosticSemaKinds.td
@@ -9232,6 +9232,9 @@ def err_typecheck_expect_scalar_or_vector : Error<
   "a vector of such type is required">;
 def err_typecheck_expect_any_scalar_or_vector : Error<
   "invalid operand of type %0 where a scalar or vector is required">;
+def err_typecheck_expect_scalar_or_vector_not_type : Error<
+  "invalid operand of type %0 where %1 or "
+  "a vector of such type is not allowed">;
 def err_typecheck_expect_flt_or_vector : Error<
   "invalid operand of type %0 where floating, complex or "
   "a vector of such types is required">;
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 12f99d9f1178a9..46646492b30d31 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -18638,6 +18638,23 @@ static Intrinsic::ID getDotProductIntrinsic(CGHLSLRuntime &RT, QualType QT) {
   return RT.getUDotIntrinsic();
 }
 
+// Return wave active sum that corresponds to the QT scalar type
+static Intrinsic::ID getWaveActiveSumIntrinsic(llvm::Triple::ArchType Arch,
+                                               CGHLSLRuntime &RT, QualType QT) {
+  switch (Arch) {
+  case llvm::Triple::spirv:
+    return llvm::Intrinsic::spv_wave_active_sum;
+  case llvm::Triple::dxil: {
+    if (QT->isUnsignedIntegerType())
+      return llvm::Intrinsic::dx_wave_active_usum;
+    return llvm::Intrinsic::dx_wave_active_sum;
+  }
+  default:
+    llvm_unreachable("Intrinsic WaveActiveSum"
+                     " not supported by target architecture");
+  }
+}
+
 Value *CodeGenFunction::EmitHLSLBuiltinExpr(unsigned BuiltinID,
                                             const CallExpr *E,
                                             ReturnValueSlot ReturnValue) {
@@ -18883,6 +18900,23 @@ case Builtin::BI__builtin_hlsl_elementwise_isinf: {
         /*ReturnType=*/Op0->getType(), CGM.getHLSLRuntime().getStepIntrinsic(),
         ArrayRef<Value *>{Op0, Op1}, nullptr, "hlsl.step");
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    // Due to the use of variadic arguments, explicitly retreive argument
+    Value *OpExpr = EmitScalarExpr(E->getArg(0));
+    llvm::FunctionType *FT = llvm::FunctionType::get(
+        OpExpr->getType(), ArrayRef{OpExpr->getType()}, false);
+    Intrinsic::ID IID = getWaveActiveSumIntrinsic(
+        getTarget().getTriple().getArch(), CGM.getHLSLRuntime(),
+        E->getArg(0)->getType());
+
+    // Get overloaded name
+    std::string Name =
+        Intrinsic::getName(IID, ArrayRef{OpExpr->getType()}, &CGM.getModule());
+    return EmitRuntimeCall(CGM.CreateRuntimeFunction(FT, Name, {},
+                                                     /*Local=*/false,
+                                                     /*AssumeConvergent=*/true),
+                           ArrayRef{OpExpr}, "hlsl.wave.active.sum");
+  }
   case Builtin::BI__builtin_hlsl_wave_get_lane_index: {
     // We don't define a SPIR-V intrinsic, instead it is a SPIR-V built-in
     // defined in SPIRVBuiltins.td. So instead we manually get the matching name
diff --git a/clang/lib/Headers/hlsl/hlsl_intrinsics.h b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
index 30dce60b3ff702..6b2843ca40c2e5 100644
--- a/clang/lib/Headers/hlsl/hlsl_intrinsics.h
+++ b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
@@ -2177,6 +2177,105 @@ __attribute__((convergent)) double3 WaveReadLaneAt(double3, int32_t);
 _HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_read_lane_at)
 __attribute__((convergent)) double4 WaveReadLaneAt(double4, int32_t);
 
+//===----------------------------------------------------------------------===//
+// WaveActiveSum builtins
+//===----------------------------------------------------------------------===//
+
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half WaveActiveSum(half);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half2 WaveActiveSum(half2);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half3 WaveActiveSum(half3);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half4 WaveActiveSum(half4);
+
+#ifdef __HLSL_ENABLE_16_BIT
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t WaveActiveSum(int16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t2 WaveActiveSum(int16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t3 WaveActiveSum(int16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t4 WaveActiveSum(int16_t4);
+
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t WaveActiveSum(uint16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t2 WaveActiveSum(uint16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t3 WaveActiveSum(uint16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t4 WaveActiveSum(uint16_t4);
+#endif
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int WaveActiveSum(int);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int2 WaveActiveSum(int2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int3 WaveActiveSum(int3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int4 WaveActiveSum(int4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint WaveActiveSum(uint);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint2 WaveActiveSum(uint2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint3 WaveActiveSum(uint3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint4 WaveActiveSum(uint4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t WaveActiveSum(int64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t2 WaveActiveSum(int64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t3 WaveActiveSum(int64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t4 WaveActiveSum(int64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t WaveActiveSum(uint64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t2 WaveActiveSum(uint64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t3 WaveActiveSum(uint64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t4 WaveActiveSum(uint64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float WaveActiveSum(float);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float2 WaveActiveSum(float2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float3 WaveActiveSum(float3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float4 WaveActiveSum(float4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double WaveActiveSum(double);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double2 WaveActiveSum(double2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double3 WaveActiveSum(double3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double4 WaveActiveSum(double4);
+
 //===----------------------------------------------------------------------===//
 // sign builtins
 //===----------------------------------------------------------------------===//
diff --git a/clang/lib/Sema/SemaHLSL.cpp b/clang/lib/Sema/SemaHLSL.cpp
index 698fdbed0484e5..7ad4ccc551c7af 100644
--- a/clang/lib/Sema/SemaHLSL.cpp
+++ b/clang/lib/Sema/SemaHLSL.cpp
@@ -1767,6 +1767,23 @@ static bool CheckAnyScalarOrVector(Sema *S, CallExpr *TheCall,
   return false;
 }
 
+static bool CheckNotScalarType(Sema *S, CallExpr *TheCall, QualType Scalar,
+                               unsigned ArgIndex) {
+  assert(TheCall->getNumArgs() >= ArgIndex);
+  QualType ArgType = TheCall->getArg(ArgIndex)->getType();
+  auto *VTy = ArgType->getAs<VectorType>();
+  // is the scalar or vector<scalar>
+  if (S->Context.hasSameUnqualifiedType(ArgType, Scalar) ||
+      (VTy &&
+       S->Context.hasSameUnqualifiedType(VTy->getElementType(), Scalar))) {
+    S->Diag(TheCall->getArg(0)->getBeginLoc(),
+            diag::err_typecheck_expect_scalar_or_vector_not_type)
+        << ArgType << Scalar;
+    return true;
+  }
+  return false;
+}
+
 static bool CheckBoolSelect(Sema *S, CallExpr *TheCall) {
   assert(TheCall->getNumArgs() == 3);
   Expr *Arg1 = TheCall->getArg(1);
@@ -2002,6 +2019,20 @@ bool SemaHLSL::CheckBuiltinFunctionCall(unsigned BuiltinID, CallExpr *TheCall) {
     TheCall->setType(ArgTyA);
     break;
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    if (SemaRef.checkArgCount(TheCall, 1))
+      return true;
+
+    // Ensure input expr type is a scalar/vector and the same as the return type
+    if (CheckAnyScalarOrVector(&SemaRef, TheCall, 0))
+      return true;
+    if (CheckNotScalarType(&SemaRef, TheCall, getASTContext().BoolTy, 0))
+      return true;
+    ExprResult Expr = TheCall->getArg(0);
+    QualType ArgTyExpr = Expr.get()->getType();
+    TheCall->setType(ArgTyExpr);
+    break;
+  }
   // Note these are llvm builtins that we want to catch invalid intrinsic
   // generation. Normal handling of these builitns will occur elsewhere.
   case Builtin::BI__builtin_elementwise_bitreverse: {
diff --git a/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
new file mode 100644
index 00000000000000..e5e463a6e9beea
--- /dev/null
+++ b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
@@ -0,0 +1,48 @@
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   dxil-pc-shadermodel6.3-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-DXIL
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   spirv-pc-vulkan-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-SPIRV
+
+// Test basic lowering to runtime function call.
+
+// CHECK-LABEL: test_int
+int test_int(int expr) {
+  // CHECK-SPIRV: %[[#entry_tok:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i32([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.sum.i32([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.sum.i32([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i32([[TY]]) #[[#attr:]]
+
+// CHECK-LABEL: test_uint64_t
+uint64_t test_uint64_t(uint64_t expr) {
+  // CHECK-SPIRV: %[[#entry_tok1:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i64([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok1]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.usum.i64([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.usum.i64([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i64([[TY]]) #[[#attr:]]
+
+// Test basic lowering to runtime function call with array and float value.
+
+// CHECK-LABEL: test_floatv4
+float4 test_floatv4(float4 expr) {
+  // CHECK-SPIRV: %[[#entry_tok2:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.spv.wave.active.sum.v4f32([[TY1]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok2]]) ]
+  // CHECK-DXIL:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.dx.wave.active.sum.v4f32([[TY1]] %[[#]])
+  // CHECK:  ret [[TY1]] %[[RET1]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY1]] @llvm.dx.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+// CHECK-SPIRV: declare [[TY1]] @llvm.spv.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+
+// CHECK: attributes #[[#attr]] = {{{.*}} convergent {{.*}}}
diff --git a/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
new file mode 100644
index 00000000000000..e475df8994f3e7
--- /dev/null
+++ b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
@@ -0,0 +1,28 @@
+// RUN: %clang_cc1 -finclude-default-header -triple dxil-pc-shadermodel6.6-library %s -emit-llvm-only -disable-llvm-passes -verify
+
+int test_too_few_arg() {
+  return __builtin_hlsl_wave_active_sum();
+  // expected-error@-1 {{too few arguments to function call, expected 1, have 0}}
+}
+
+float2 test_too_many_arg(float2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0, p0);
+  // expected-error@-1 {{too many arguments to function call, expected 1, have 2}}
+}
+
+bool test_expr_bool_type_check(bool p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool' where 'bool' or a vector of such type is not allowed}}
+}
+
+bool2 test_expr_bool_vec_type_check(bool2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool2' (aka 'vector<bool, 2>') where 'bool' or a vector of such type is not allowed}}
+}
+
+struct S { float f; };
+
+S test_expr_struct_type_check(S p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'S' where a scalar or vector is required}}
+}
diff --git a/llvm/include/llvm/IR/IntrinsicsDirectX.td b/llvm/include/llvm/IR/IntrinsicsDirectX.td
index 27a437a83be6dd..0ad6c5e1aa2351 100644
--- a/llvm/include/llvm/IR/IntrinsicsDirectX.td
+++ b/llvm/include/llvm/IR/IntrinsicsDirectX.td
@@ -85,6 +85,8 @@ def int_dx_umad : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>, LLV
 def int_dx_normalize : DefaultAttrsIntrinsic<[LLVMMatchType<0>], [llvm_anyfloat_ty], [IntrNoMem]>;
 def int_dx_rsqrt  : DefaultAttrsIntrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>], [IntrNoMem]>;
 def int_dx_wave_getlaneindex : DefaultAttrsIntrinsic<[llvm_i32_ty], [], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_usum : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
 def int_dx_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
 def int_dx_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
 def int_dx_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/include/llvm/IR/IntrinsicsSPIRV.td b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
index 6df2eb156a0774..b90190b8963dbe 100644
--- a/llvm/include/llvm/IR/IntrinsicsSPIRV.td
+++ b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
@@ -83,6 +83,7 @@ let TargetPrefix = "spv" in {
     DefaultAttrsIntrinsic<[LLVMVectorElementType<0>],
     [llvm_anyint_ty, LLVMScalarOrSameVectorWidth<0, LLVMVectorElementType<0>>],
     [IntrNoMem, Commutative] >;
+  def int_spv_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
   def int_spv_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
   def int_spv_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
   def int_spv_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
index d9377fe4b91a1a..54ab680d27b88e 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
@@ -164,6 +164,9 @@ class SPIRVInstructionSelector : public InstructionSelector {
   bool selectIntegerDot(Register ResVReg, const SPIRVType *ResType,
                         MachineInstr &I) const;
 
+  bool selectWaveActiveSum(Register ResVReg, const SPIRVType *ResType,
+                           MachineInstr &I) const;
+
   void renderImm32(MachineInstrBuilder &MIB, const MachineInstr &I,
                    int OpIdx) const;
   void renderFImm64(MachineInstrBuilder &MIB, const MachineInstr &I,
@@ -1782,6 +1785,34 @@ bool SPIRVInstructionSelector::selectWaveReadLaneAt(Register ResVReg,
       .addUse(I.getOperand(3).getReg());
 }
 
+bool SPIRVInstructionSelector::selectWaveActiveSum(Register ResVReg,
+                                                   const SPIRVType *ResType,
+                                                   MachineInstr &I) const {
+  assert(I.getNumOperands() == 3);
+  assert(I.getOperand(2).isReg());
+  MachineBasicBlock &BB = *I.getParent();
+  Register InputRegister = I.getOperand(2).getReg();
+  SPIRVType *InputType = GR.getSPIRVTypeForVReg(InputRegister);
+
+  if (!InputType)
+    report_fatal_error("Input Type could not be determined.");
+
+  // IntTy is used to define the execution scope, set to 3 to denote a
+  // cross-lane interaction equivalent to a SPIR-V subgroup.
+  SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
+
+  // Retreive the operation to use based on input type
+  bool IsFloatTy = GR.isScalarOrVectorOfType(InputRegister, SPIRV::OpTypeFloat);
+  auto Opcode =
+      IsFloatTy ? SPIRV::OpGroupNonUniformFAdd : SPIRV::OpGroupNonUniformIAdd;
+  return BuildMI(BB, I, I.getDebugLoc(), TII.get(Opcode))
+      .addDef(ResVReg)
+      .addUse(GR.getSPIRVTypeID(ResType))
+      .addUse(GR.getOrCreateConstInt(3, I, IntTy, TII))
+      .addImm(0) // group operation set to 0 to denote a reduction operation
+      .addUse(I.getOperand(2).getReg());
+}
+
 bool SPIRVInstructionSelector::selectBitreverse(Register ResVReg,
                                                 const SPIRVType *ResType,
                                                 MachineInstr &I) const {
@@ -2559,6 +2590,8 @@ bool SPIRVInstructionSelector::selectIntrinsic(Register ResVReg,
   } break;
   case Intrinsic::spv_saturate:
     return selectSaturate(ResVReg, ResType, I);
+  case Intrinsic::spv_wave_active_sum:
+    return selectWaveActiveSum(ResVReg, ResType, I);
   case Intrinsic::spv_wave_is_first_lane: {
     SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
     return BuildMI(BB, I, I.getDebugLoc(),
diff --git a/llvm/test/CodeGen/SP...
[truncated]

@llvmbot
Copy link
Collaborator

llvmbot commented Oct 18, 2024

@llvm/pr-subscribers-backend-spir-v

Author: Finn Plummer (inbelic)

Changes 
  - add clang builtin to Builtins.td
  - link builtin in hlsl_intrinsics
  - add codegen for spirv intrinsic and two directx intrinsics to retain
    signedness information of the operands in CGBuiltin.cpp
  - add semantic analysis in SemaHLSL.cpp
  - add lowering of spirv intrinsic to spirv backend in
    SPIRVInstructionSelector.cpp

  - add test cases to illustrate passes

Note that this defines the dx intrinsics but does not implement the
DirectX lowering to DXIL. This will implemented in a second pr when
the dependent pr merges.

Part 1 of #70106

Patch is 21.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/112400.diff

11 Files Affected:

  • (modified) clang/include/clang/Basic/Builtins.td (+6)
  • (modified) clang/include/clang/Basic/DiagnosticSemaKinds.td (+3)
  • (modified) clang/lib/CodeGen/CGBuiltin.cpp (+34)
  • (modified) clang/lib/Headers/hlsl/hlsl_intrinsics.h (+99)
  • (modified) clang/lib/Sema/SemaHLSL.cpp (+31)
  • (added) clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl (+48)
  • (added) clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl (+28)
  • (modified) llvm/include/llvm/IR/IntrinsicsDirectX.td (+2)
  • (modified) llvm/include/llvm/IR/IntrinsicsSPIRV.td (+1)
  • (modified) llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp (+33)
  • (added) llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveSum.ll (+41)
diff --git a/clang/include/clang/Basic/Builtins.td b/clang/include/clang/Basic/Builtins.td
index 382fb6b7a3c031..7205f10b01c2b9 100644
--- a/clang/include/clang/Basic/Builtins.td
+++ b/clang/include/clang/Basic/Builtins.td
@@ -4749,6 +4749,12 @@ def HLSLWaveActiveCountBits : LangBuiltin<"HLSL_LANG"> {
   let Prototype = "unsigned int(bool)";
 }
 
+def HLSLWaveActiveSum : LangBuiltin<"HLSL_LANG"> {
+  let Spellings = ["__builtin_hlsl_wave_active_sum"];
+  let Attributes = [NoThrow, Const];
+  let Prototype = "void (...)";
+}
+
 def HLSLWaveGetLaneIndex : LangBuiltin<"HLSL_LANG"> {
   let Spellings = ["__builtin_hlsl_wave_get_lane_index"];
   let Attributes = [NoThrow, Const];
diff --git a/clang/include/clang/Basic/DiagnosticSemaKinds.td b/clang/include/clang/Basic/DiagnosticSemaKinds.td
index c458a62d9be48c..95093723c2f495 100644
--- a/clang/include/clang/Basic/DiagnosticSemaKinds.td
+++ b/clang/include/clang/Basic/DiagnosticSemaKinds.td
@@ -9232,6 +9232,9 @@ def err_typecheck_expect_scalar_or_vector : Error<
   "a vector of such type is required">;
 def err_typecheck_expect_any_scalar_or_vector : Error<
   "invalid operand of type %0 where a scalar or vector is required">;
+def err_typecheck_expect_scalar_or_vector_not_type : Error<
+  "invalid operand of type %0 where %1 or "
+  "a vector of such type is not allowed">;
 def err_typecheck_expect_flt_or_vector : Error<
   "invalid operand of type %0 where floating, complex or "
   "a vector of such types is required">;
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 12f99d9f1178a9..46646492b30d31 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -18638,6 +18638,23 @@ static Intrinsic::ID getDotProductIntrinsic(CGHLSLRuntime &RT, QualType QT) {
   return RT.getUDotIntrinsic();
 }
 
+// Return wave active sum that corresponds to the QT scalar type
+static Intrinsic::ID getWaveActiveSumIntrinsic(llvm::Triple::ArchType Arch,
+                                               CGHLSLRuntime &RT, QualType QT) {
+  switch (Arch) {
+  case llvm::Triple::spirv:
+    return llvm::Intrinsic::spv_wave_active_sum;
+  case llvm::Triple::dxil: {
+    if (QT->isUnsignedIntegerType())
+      return llvm::Intrinsic::dx_wave_active_usum;
+    return llvm::Intrinsic::dx_wave_active_sum;
+  }
+  default:
+    llvm_unreachable("Intrinsic WaveActiveSum"
+                     " not supported by target architecture");
+  }
+}
+
 Value *CodeGenFunction::EmitHLSLBuiltinExpr(unsigned BuiltinID,
                                             const CallExpr *E,
                                             ReturnValueSlot ReturnValue) {
@@ -18883,6 +18900,23 @@ case Builtin::BI__builtin_hlsl_elementwise_isinf: {
         /*ReturnType=*/Op0->getType(), CGM.getHLSLRuntime().getStepIntrinsic(),
         ArrayRef<Value *>{Op0, Op1}, nullptr, "hlsl.step");
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    // Due to the use of variadic arguments, explicitly retreive argument
+    Value *OpExpr = EmitScalarExpr(E->getArg(0));
+    llvm::FunctionType *FT = llvm::FunctionType::get(
+        OpExpr->getType(), ArrayRef{OpExpr->getType()}, false);
+    Intrinsic::ID IID = getWaveActiveSumIntrinsic(
+        getTarget().getTriple().getArch(), CGM.getHLSLRuntime(),
+        E->getArg(0)->getType());
+
+    // Get overloaded name
+    std::string Name =
+        Intrinsic::getName(IID, ArrayRef{OpExpr->getType()}, &CGM.getModule());
+    return EmitRuntimeCall(CGM.CreateRuntimeFunction(FT, Name, {},
+                                                     /*Local=*/false,
+                                                     /*AssumeConvergent=*/true),
+                           ArrayRef{OpExpr}, "hlsl.wave.active.sum");
+  }
   case Builtin::BI__builtin_hlsl_wave_get_lane_index: {
     // We don't define a SPIR-V intrinsic, instead it is a SPIR-V built-in
     // defined in SPIRVBuiltins.td. So instead we manually get the matching name
diff --git a/clang/lib/Headers/hlsl/hlsl_intrinsics.h b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
index 30dce60b3ff702..6b2843ca40c2e5 100644
--- a/clang/lib/Headers/hlsl/hlsl_intrinsics.h
+++ b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
@@ -2177,6 +2177,105 @@ __attribute__((convergent)) double3 WaveReadLaneAt(double3, int32_t);
 _HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_read_lane_at)
 __attribute__((convergent)) double4 WaveReadLaneAt(double4, int32_t);
 
+//===----------------------------------------------------------------------===//
+// WaveActiveSum builtins
+//===----------------------------------------------------------------------===//
+
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half WaveActiveSum(half);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half2 WaveActiveSum(half2);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half3 WaveActiveSum(half3);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half4 WaveActiveSum(half4);
+
+#ifdef __HLSL_ENABLE_16_BIT
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t WaveActiveSum(int16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t2 WaveActiveSum(int16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t3 WaveActiveSum(int16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t4 WaveActiveSum(int16_t4);
+
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t WaveActiveSum(uint16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t2 WaveActiveSum(uint16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t3 WaveActiveSum(uint16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t4 WaveActiveSum(uint16_t4);
+#endif
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int WaveActiveSum(int);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int2 WaveActiveSum(int2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int3 WaveActiveSum(int3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int4 WaveActiveSum(int4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint WaveActiveSum(uint);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint2 WaveActiveSum(uint2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint3 WaveActiveSum(uint3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint4 WaveActiveSum(uint4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t WaveActiveSum(int64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t2 WaveActiveSum(int64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t3 WaveActiveSum(int64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t4 WaveActiveSum(int64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t WaveActiveSum(uint64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t2 WaveActiveSum(uint64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t3 WaveActiveSum(uint64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t4 WaveActiveSum(uint64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float WaveActiveSum(float);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float2 WaveActiveSum(float2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float3 WaveActiveSum(float3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float4 WaveActiveSum(float4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double WaveActiveSum(double);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double2 WaveActiveSum(double2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double3 WaveActiveSum(double3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double4 WaveActiveSum(double4);
+
 //===----------------------------------------------------------------------===//
 // sign builtins
 //===----------------------------------------------------------------------===//
diff --git a/clang/lib/Sema/SemaHLSL.cpp b/clang/lib/Sema/SemaHLSL.cpp
index 698fdbed0484e5..7ad4ccc551c7af 100644
--- a/clang/lib/Sema/SemaHLSL.cpp
+++ b/clang/lib/Sema/SemaHLSL.cpp
@@ -1767,6 +1767,23 @@ static bool CheckAnyScalarOrVector(Sema *S, CallExpr *TheCall,
   return false;
 }
 
+static bool CheckNotScalarType(Sema *S, CallExpr *TheCall, QualType Scalar,
+                               unsigned ArgIndex) {
+  assert(TheCall->getNumArgs() >= ArgIndex);
+  QualType ArgType = TheCall->getArg(ArgIndex)->getType();
+  auto *VTy = ArgType->getAs<VectorType>();
+  // is the scalar or vector<scalar>
+  if (S->Context.hasSameUnqualifiedType(ArgType, Scalar) ||
+      (VTy &&
+       S->Context.hasSameUnqualifiedType(VTy->getElementType(), Scalar))) {
+    S->Diag(TheCall->getArg(0)->getBeginLoc(),
+            diag::err_typecheck_expect_scalar_or_vector_not_type)
+        << ArgType << Scalar;
+    return true;
+  }
+  return false;
+}
+
 static bool CheckBoolSelect(Sema *S, CallExpr *TheCall) {
   assert(TheCall->getNumArgs() == 3);
   Expr *Arg1 = TheCall->getArg(1);
@@ -2002,6 +2019,20 @@ bool SemaHLSL::CheckBuiltinFunctionCall(unsigned BuiltinID, CallExpr *TheCall) {
     TheCall->setType(ArgTyA);
     break;
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    if (SemaRef.checkArgCount(TheCall, 1))
+      return true;
+
+    // Ensure input expr type is a scalar/vector and the same as the return type
+    if (CheckAnyScalarOrVector(&SemaRef, TheCall, 0))
+      return true;
+    if (CheckNotScalarType(&SemaRef, TheCall, getASTContext().BoolTy, 0))
+      return true;
+    ExprResult Expr = TheCall->getArg(0);
+    QualType ArgTyExpr = Expr.get()->getType();
+    TheCall->setType(ArgTyExpr);
+    break;
+  }
   // Note these are llvm builtins that we want to catch invalid intrinsic
   // generation. Normal handling of these builitns will occur elsewhere.
   case Builtin::BI__builtin_elementwise_bitreverse: {
diff --git a/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
new file mode 100644
index 00000000000000..e5e463a6e9beea
--- /dev/null
+++ b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
@@ -0,0 +1,48 @@
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   dxil-pc-shadermodel6.3-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-DXIL
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   spirv-pc-vulkan-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-SPIRV
+
+// Test basic lowering to runtime function call.
+
+// CHECK-LABEL: test_int
+int test_int(int expr) {
+  // CHECK-SPIRV: %[[#entry_tok:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i32([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.sum.i32([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.sum.i32([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i32([[TY]]) #[[#attr:]]
+
+// CHECK-LABEL: test_uint64_t
+uint64_t test_uint64_t(uint64_t expr) {
+  // CHECK-SPIRV: %[[#entry_tok1:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i64([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok1]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.usum.i64([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.usum.i64([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i64([[TY]]) #[[#attr:]]
+
+// Test basic lowering to runtime function call with array and float value.
+
+// CHECK-LABEL: test_floatv4
+float4 test_floatv4(float4 expr) {
+  // CHECK-SPIRV: %[[#entry_tok2:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.spv.wave.active.sum.v4f32([[TY1]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok2]]) ]
+  // CHECK-DXIL:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.dx.wave.active.sum.v4f32([[TY1]] %[[#]])
+  // CHECK:  ret [[TY1]] %[[RET1]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY1]] @llvm.dx.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+// CHECK-SPIRV: declare [[TY1]] @llvm.spv.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+
+// CHECK: attributes #[[#attr]] = {{{.*}} convergent {{.*}}}
diff --git a/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
new file mode 100644
index 00000000000000..e475df8994f3e7
--- /dev/null
+++ b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
@@ -0,0 +1,28 @@
+// RUN: %clang_cc1 -finclude-default-header -triple dxil-pc-shadermodel6.6-library %s -emit-llvm-only -disable-llvm-passes -verify
+
+int test_too_few_arg() {
+  return __builtin_hlsl_wave_active_sum();
+  // expected-error@-1 {{too few arguments to function call, expected 1, have 0}}
+}
+
+float2 test_too_many_arg(float2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0, p0);
+  // expected-error@-1 {{too many arguments to function call, expected 1, have 2}}
+}
+
+bool test_expr_bool_type_check(bool p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool' where 'bool' or a vector of such type is not allowed}}
+}
+
+bool2 test_expr_bool_vec_type_check(bool2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool2' (aka 'vector<bool, 2>') where 'bool' or a vector of such type is not allowed}}
+}
+
+struct S { float f; };
+
+S test_expr_struct_type_check(S p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'S' where a scalar or vector is required}}
+}
diff --git a/llvm/include/llvm/IR/IntrinsicsDirectX.td b/llvm/include/llvm/IR/IntrinsicsDirectX.td
index 27a437a83be6dd..0ad6c5e1aa2351 100644
--- a/llvm/include/llvm/IR/IntrinsicsDirectX.td
+++ b/llvm/include/llvm/IR/IntrinsicsDirectX.td
@@ -85,6 +85,8 @@ def int_dx_umad : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>, LLV
 def int_dx_normalize : DefaultAttrsIntrinsic<[LLVMMatchType<0>], [llvm_anyfloat_ty], [IntrNoMem]>;
 def int_dx_rsqrt  : DefaultAttrsIntrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>], [IntrNoMem]>;
 def int_dx_wave_getlaneindex : DefaultAttrsIntrinsic<[llvm_i32_ty], [], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_usum : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
 def int_dx_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
 def int_dx_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
 def int_dx_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/include/llvm/IR/IntrinsicsSPIRV.td b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
index 6df2eb156a0774..b90190b8963dbe 100644
--- a/llvm/include/llvm/IR/IntrinsicsSPIRV.td
+++ b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
@@ -83,6 +83,7 @@ let TargetPrefix = "spv" in {
     DefaultAttrsIntrinsic<[LLVMVectorElementType<0>],
     [llvm_anyint_ty, LLVMScalarOrSameVectorWidth<0, LLVMVectorElementType<0>>],
     [IntrNoMem, Commutative] >;
+  def int_spv_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
   def int_spv_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
   def int_spv_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
   def int_spv_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
index d9377fe4b91a1a..54ab680d27b88e 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
@@ -164,6 +164,9 @@ class SPIRVInstructionSelector : public InstructionSelector {
   bool selectIntegerDot(Register ResVReg, const SPIRVType *ResType,
                         MachineInstr &I) const;
 
+  bool selectWaveActiveSum(Register ResVReg, const SPIRVType *ResType,
+                           MachineInstr &I) const;
+
   void renderImm32(MachineInstrBuilder &MIB, const MachineInstr &I,
                    int OpIdx) const;
   void renderFImm64(MachineInstrBuilder &MIB, const MachineInstr &I,
@@ -1782,6 +1785,34 @@ bool SPIRVInstructionSelector::selectWaveReadLaneAt(Register ResVReg,
       .addUse(I.getOperand(3).getReg());
 }
 
+bool SPIRVInstructionSelector::selectWaveActiveSum(Register ResVReg,
+                                                   const SPIRVType *ResType,
+                                                   MachineInstr &I) const {
+  assert(I.getNumOperands() == 3);
+  assert(I.getOperand(2).isReg());
+  MachineBasicBlock &BB = *I.getParent();
+  Register InputRegister = I.getOperand(2).getReg();
+  SPIRVType *InputType = GR.getSPIRVTypeForVReg(InputRegister);
+
+  if (!InputType)
+    report_fatal_error("Input Type could not be determined.");
+
+  // IntTy is used to define the execution scope, set to 3 to denote a
+  // cross-lane interaction equivalent to a SPIR-V subgroup.
+  SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
+
+  // Retreive the operation to use based on input type
+  bool IsFloatTy = GR.isScalarOrVectorOfType(InputRegister, SPIRV::OpTypeFloat);
+  auto Opcode =
+      IsFloatTy ? SPIRV::OpGroupNonUniformFAdd : SPIRV::OpGroupNonUniformIAdd;
+  return BuildMI(BB, I, I.getDebugLoc(), TII.get(Opcode))
+      .addDef(ResVReg)
+      .addUse(GR.getSPIRVTypeID(ResType))
+      .addUse(GR.getOrCreateConstInt(3, I, IntTy, TII))
+      .addImm(0) // group operation set to 0 to denote a reduction operation
+      .addUse(I.getOperand(2).getReg());
+}
+
 bool SPIRVInstructionSelector::selectBitreverse(Register ResVReg,
                                                 const SPIRVType *ResType,
                                                 MachineInstr &I) const {
@@ -2559,6 +2590,8 @@ bool SPIRVInstructionSelector::selectIntrinsic(Register ResVReg,
   } break;
   case Intrinsic::spv_saturate:
     return selectSaturate(ResVReg, ResType, I);
+  case Intrinsic::spv_wave_active_sum:
+    return selectWaveActiveSum(ResVReg, ResType, I);
   case Intrinsic::spv_wave_is_first_lane: {
     SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
     return BuildMI(BB, I, I.getDebugLoc(),
diff --git a/llvm/test/CodeGen/SP...
[truncated]

@llvmbot
Copy link
Collaborator

llvmbot commented Oct 18, 2024

@llvm/pr-subscribers-backend-x86

Author: Finn Plummer (inbelic)

Changes 
  - add clang builtin to Builtins.td
  - link builtin in hlsl_intrinsics
  - add codegen for spirv intrinsic and two directx intrinsics to retain
    signedness information of the operands in CGBuiltin.cpp
  - add semantic analysis in SemaHLSL.cpp
  - add lowering of spirv intrinsic to spirv backend in
    SPIRVInstructionSelector.cpp

  - add test cases to illustrate passes

Note that this defines the dx intrinsics but does not implement the
DirectX lowering to DXIL. This will implemented in a second pr when
the dependent pr merges.

Part 1 of #70106

Patch is 21.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/112400.diff

11 Files Affected:

  • (modified) clang/include/clang/Basic/Builtins.td (+6)
  • (modified) clang/include/clang/Basic/DiagnosticSemaKinds.td (+3)
  • (modified) clang/lib/CodeGen/CGBuiltin.cpp (+34)
  • (modified) clang/lib/Headers/hlsl/hlsl_intrinsics.h (+99)
  • (modified) clang/lib/Sema/SemaHLSL.cpp (+31)
  • (added) clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl (+48)
  • (added) clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl (+28)
  • (modified) llvm/include/llvm/IR/IntrinsicsDirectX.td (+2)
  • (modified) llvm/include/llvm/IR/IntrinsicsSPIRV.td (+1)
  • (modified) llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp (+33)
  • (added) llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveSum.ll (+41)
diff --git a/clang/include/clang/Basic/Builtins.td b/clang/include/clang/Basic/Builtins.td
index 382fb6b7a3c031..7205f10b01c2b9 100644
--- a/clang/include/clang/Basic/Builtins.td
+++ b/clang/include/clang/Basic/Builtins.td
@@ -4749,6 +4749,12 @@ def HLSLWaveActiveCountBits : LangBuiltin<"HLSL_LANG"> {
   let Prototype = "unsigned int(bool)";
 }
 
+def HLSLWaveActiveSum : LangBuiltin<"HLSL_LANG"> {
+  let Spellings = ["__builtin_hlsl_wave_active_sum"];
+  let Attributes = [NoThrow, Const];
+  let Prototype = "void (...)";
+}
+
 def HLSLWaveGetLaneIndex : LangBuiltin<"HLSL_LANG"> {
   let Spellings = ["__builtin_hlsl_wave_get_lane_index"];
   let Attributes = [NoThrow, Const];
diff --git a/clang/include/clang/Basic/DiagnosticSemaKinds.td b/clang/include/clang/Basic/DiagnosticSemaKinds.td
index c458a62d9be48c..95093723c2f495 100644
--- a/clang/include/clang/Basic/DiagnosticSemaKinds.td
+++ b/clang/include/clang/Basic/DiagnosticSemaKinds.td
@@ -9232,6 +9232,9 @@ def err_typecheck_expect_scalar_or_vector : Error<
   "a vector of such type is required">;
 def err_typecheck_expect_any_scalar_or_vector : Error<
   "invalid operand of type %0 where a scalar or vector is required">;
+def err_typecheck_expect_scalar_or_vector_not_type : Error<
+  "invalid operand of type %0 where %1 or "
+  "a vector of such type is not allowed">;
 def err_typecheck_expect_flt_or_vector : Error<
   "invalid operand of type %0 where floating, complex or "
   "a vector of such types is required">;
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 12f99d9f1178a9..46646492b30d31 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -18638,6 +18638,23 @@ static Intrinsic::ID getDotProductIntrinsic(CGHLSLRuntime &RT, QualType QT) {
   return RT.getUDotIntrinsic();
 }
 
+// Return wave active sum that corresponds to the QT scalar type
+static Intrinsic::ID getWaveActiveSumIntrinsic(llvm::Triple::ArchType Arch,
+                                               CGHLSLRuntime &RT, QualType QT) {
+  switch (Arch) {
+  case llvm::Triple::spirv:
+    return llvm::Intrinsic::spv_wave_active_sum;
+  case llvm::Triple::dxil: {
+    if (QT->isUnsignedIntegerType())
+      return llvm::Intrinsic::dx_wave_active_usum;
+    return llvm::Intrinsic::dx_wave_active_sum;
+  }
+  default:
+    llvm_unreachable("Intrinsic WaveActiveSum"
+                     " not supported by target architecture");
+  }
+}
+
 Value *CodeGenFunction::EmitHLSLBuiltinExpr(unsigned BuiltinID,
                                             const CallExpr *E,
                                             ReturnValueSlot ReturnValue) {
@@ -18883,6 +18900,23 @@ case Builtin::BI__builtin_hlsl_elementwise_isinf: {
         /*ReturnType=*/Op0->getType(), CGM.getHLSLRuntime().getStepIntrinsic(),
         ArrayRef<Value *>{Op0, Op1}, nullptr, "hlsl.step");
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    // Due to the use of variadic arguments, explicitly retreive argument
+    Value *OpExpr = EmitScalarExpr(E->getArg(0));
+    llvm::FunctionType *FT = llvm::FunctionType::get(
+        OpExpr->getType(), ArrayRef{OpExpr->getType()}, false);
+    Intrinsic::ID IID = getWaveActiveSumIntrinsic(
+        getTarget().getTriple().getArch(), CGM.getHLSLRuntime(),
+        E->getArg(0)->getType());
+
+    // Get overloaded name
+    std::string Name =
+        Intrinsic::getName(IID, ArrayRef{OpExpr->getType()}, &CGM.getModule());
+    return EmitRuntimeCall(CGM.CreateRuntimeFunction(FT, Name, {},
+                                                     /*Local=*/false,
+                                                     /*AssumeConvergent=*/true),
+                           ArrayRef{OpExpr}, "hlsl.wave.active.sum");
+  }
   case Builtin::BI__builtin_hlsl_wave_get_lane_index: {
     // We don't define a SPIR-V intrinsic, instead it is a SPIR-V built-in
     // defined in SPIRVBuiltins.td. So instead we manually get the matching name
diff --git a/clang/lib/Headers/hlsl/hlsl_intrinsics.h b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
index 30dce60b3ff702..6b2843ca40c2e5 100644
--- a/clang/lib/Headers/hlsl/hlsl_intrinsics.h
+++ b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
@@ -2177,6 +2177,105 @@ __attribute__((convergent)) double3 WaveReadLaneAt(double3, int32_t);
 _HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_read_lane_at)
 __attribute__((convergent)) double4 WaveReadLaneAt(double4, int32_t);
 
+//===----------------------------------------------------------------------===//
+// WaveActiveSum builtins
+//===----------------------------------------------------------------------===//
+
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half WaveActiveSum(half);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half2 WaveActiveSum(half2);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half3 WaveActiveSum(half3);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half4 WaveActiveSum(half4);
+
+#ifdef __HLSL_ENABLE_16_BIT
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t WaveActiveSum(int16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t2 WaveActiveSum(int16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t3 WaveActiveSum(int16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t4 WaveActiveSum(int16_t4);
+
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t WaveActiveSum(uint16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t2 WaveActiveSum(uint16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t3 WaveActiveSum(uint16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t4 WaveActiveSum(uint16_t4);
+#endif
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int WaveActiveSum(int);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int2 WaveActiveSum(int2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int3 WaveActiveSum(int3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int4 WaveActiveSum(int4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint WaveActiveSum(uint);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint2 WaveActiveSum(uint2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint3 WaveActiveSum(uint3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint4 WaveActiveSum(uint4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t WaveActiveSum(int64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t2 WaveActiveSum(int64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t3 WaveActiveSum(int64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t4 WaveActiveSum(int64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t WaveActiveSum(uint64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t2 WaveActiveSum(uint64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t3 WaveActiveSum(uint64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t4 WaveActiveSum(uint64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float WaveActiveSum(float);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float2 WaveActiveSum(float2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float3 WaveActiveSum(float3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float4 WaveActiveSum(float4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double WaveActiveSum(double);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double2 WaveActiveSum(double2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double3 WaveActiveSum(double3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double4 WaveActiveSum(double4);
+
 //===----------------------------------------------------------------------===//
 // sign builtins
 //===----------------------------------------------------------------------===//
diff --git a/clang/lib/Sema/SemaHLSL.cpp b/clang/lib/Sema/SemaHLSL.cpp
index 698fdbed0484e5..7ad4ccc551c7af 100644
--- a/clang/lib/Sema/SemaHLSL.cpp
+++ b/clang/lib/Sema/SemaHLSL.cpp
@@ -1767,6 +1767,23 @@ static bool CheckAnyScalarOrVector(Sema *S, CallExpr *TheCall,
   return false;
 }
 
+static bool CheckNotScalarType(Sema *S, CallExpr *TheCall, QualType Scalar,
+                               unsigned ArgIndex) {
+  assert(TheCall->getNumArgs() >= ArgIndex);
+  QualType ArgType = TheCall->getArg(ArgIndex)->getType();
+  auto *VTy = ArgType->getAs<VectorType>();
+  // is the scalar or vector<scalar>
+  if (S->Context.hasSameUnqualifiedType(ArgType, Scalar) ||
+      (VTy &&
+       S->Context.hasSameUnqualifiedType(VTy->getElementType(), Scalar))) {
+    S->Diag(TheCall->getArg(0)->getBeginLoc(),
+            diag::err_typecheck_expect_scalar_or_vector_not_type)
+        << ArgType << Scalar;
+    return true;
+  }
+  return false;
+}
+
 static bool CheckBoolSelect(Sema *S, CallExpr *TheCall) {
   assert(TheCall->getNumArgs() == 3);
   Expr *Arg1 = TheCall->getArg(1);
@@ -2002,6 +2019,20 @@ bool SemaHLSL::CheckBuiltinFunctionCall(unsigned BuiltinID, CallExpr *TheCall) {
     TheCall->setType(ArgTyA);
     break;
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    if (SemaRef.checkArgCount(TheCall, 1))
+      return true;
+
+    // Ensure input expr type is a scalar/vector and the same as the return type
+    if (CheckAnyScalarOrVector(&SemaRef, TheCall, 0))
+      return true;
+    if (CheckNotScalarType(&SemaRef, TheCall, getASTContext().BoolTy, 0))
+      return true;
+    ExprResult Expr = TheCall->getArg(0);
+    QualType ArgTyExpr = Expr.get()->getType();
+    TheCall->setType(ArgTyExpr);
+    break;
+  }
   // Note these are llvm builtins that we want to catch invalid intrinsic
   // generation. Normal handling of these builitns will occur elsewhere.
   case Builtin::BI__builtin_elementwise_bitreverse: {
diff --git a/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
new file mode 100644
index 00000000000000..e5e463a6e9beea
--- /dev/null
+++ b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
@@ -0,0 +1,48 @@
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   dxil-pc-shadermodel6.3-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-DXIL
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   spirv-pc-vulkan-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-SPIRV
+
+// Test basic lowering to runtime function call.
+
+// CHECK-LABEL: test_int
+int test_int(int expr) {
+  // CHECK-SPIRV: %[[#entry_tok:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i32([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.sum.i32([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.sum.i32([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i32([[TY]]) #[[#attr:]]
+
+// CHECK-LABEL: test_uint64_t
+uint64_t test_uint64_t(uint64_t expr) {
+  // CHECK-SPIRV: %[[#entry_tok1:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i64([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok1]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.usum.i64([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.usum.i64([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i64([[TY]]) #[[#attr:]]
+
+// Test basic lowering to runtime function call with array and float value.
+
+// CHECK-LABEL: test_floatv4
+float4 test_floatv4(float4 expr) {
+  // CHECK-SPIRV: %[[#entry_tok2:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.spv.wave.active.sum.v4f32([[TY1]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok2]]) ]
+  // CHECK-DXIL:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.dx.wave.active.sum.v4f32([[TY1]] %[[#]])
+  // CHECK:  ret [[TY1]] %[[RET1]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY1]] @llvm.dx.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+// CHECK-SPIRV: declare [[TY1]] @llvm.spv.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+
+// CHECK: attributes #[[#attr]] = {{{.*}} convergent {{.*}}}
diff --git a/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
new file mode 100644
index 00000000000000..e475df8994f3e7
--- /dev/null
+++ b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
@@ -0,0 +1,28 @@
+// RUN: %clang_cc1 -finclude-default-header -triple dxil-pc-shadermodel6.6-library %s -emit-llvm-only -disable-llvm-passes -verify
+
+int test_too_few_arg() {
+  return __builtin_hlsl_wave_active_sum();
+  // expected-error@-1 {{too few arguments to function call, expected 1, have 0}}
+}
+
+float2 test_too_many_arg(float2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0, p0);
+  // expected-error@-1 {{too many arguments to function call, expected 1, have 2}}
+}
+
+bool test_expr_bool_type_check(bool p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool' where 'bool' or a vector of such type is not allowed}}
+}
+
+bool2 test_expr_bool_vec_type_check(bool2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool2' (aka 'vector<bool, 2>') where 'bool' or a vector of such type is not allowed}}
+}
+
+struct S { float f; };
+
+S test_expr_struct_type_check(S p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'S' where a scalar or vector is required}}
+}
diff --git a/llvm/include/llvm/IR/IntrinsicsDirectX.td b/llvm/include/llvm/IR/IntrinsicsDirectX.td
index 27a437a83be6dd..0ad6c5e1aa2351 100644
--- a/llvm/include/llvm/IR/IntrinsicsDirectX.td
+++ b/llvm/include/llvm/IR/IntrinsicsDirectX.td
@@ -85,6 +85,8 @@ def int_dx_umad : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>, LLV
 def int_dx_normalize : DefaultAttrsIntrinsic<[LLVMMatchType<0>], [llvm_anyfloat_ty], [IntrNoMem]>;
 def int_dx_rsqrt  : DefaultAttrsIntrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>], [IntrNoMem]>;
 def int_dx_wave_getlaneindex : DefaultAttrsIntrinsic<[llvm_i32_ty], [], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_usum : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
 def int_dx_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
 def int_dx_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
 def int_dx_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/include/llvm/IR/IntrinsicsSPIRV.td b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
index 6df2eb156a0774..b90190b8963dbe 100644
--- a/llvm/include/llvm/IR/IntrinsicsSPIRV.td
+++ b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
@@ -83,6 +83,7 @@ let TargetPrefix = "spv" in {
     DefaultAttrsIntrinsic<[LLVMVectorElementType<0>],
     [llvm_anyint_ty, LLVMScalarOrSameVectorWidth<0, LLVMVectorElementType<0>>],
     [IntrNoMem, Commutative] >;
+  def int_spv_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
   def int_spv_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
   def int_spv_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
   def int_spv_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
index d9377fe4b91a1a..54ab680d27b88e 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
@@ -164,6 +164,9 @@ class SPIRVInstructionSelector : public InstructionSelector {
   bool selectIntegerDot(Register ResVReg, const SPIRVType *ResType,
                         MachineInstr &I) const;
 
+  bool selectWaveActiveSum(Register ResVReg, const SPIRVType *ResType,
+                           MachineInstr &I) const;
+
   void renderImm32(MachineInstrBuilder &MIB, const MachineInstr &I,
                    int OpIdx) const;
   void renderFImm64(MachineInstrBuilder &MIB, const MachineInstr &I,
@@ -1782,6 +1785,34 @@ bool SPIRVInstructionSelector::selectWaveReadLaneAt(Register ResVReg,
       .addUse(I.getOperand(3).getReg());
 }
 
+bool SPIRVInstructionSelector::selectWaveActiveSum(Register ResVReg,
+                                                   const SPIRVType *ResType,
+                                                   MachineInstr &I) const {
+  assert(I.getNumOperands() == 3);
+  assert(I.getOperand(2).isReg());
+  MachineBasicBlock &BB = *I.getParent();
+  Register InputRegister = I.getOperand(2).getReg();
+  SPIRVType *InputType = GR.getSPIRVTypeForVReg(InputRegister);
+
+  if (!InputType)
+    report_fatal_error("Input Type could not be determined.");
+
+  // IntTy is used to define the execution scope, set to 3 to denote a
+  // cross-lane interaction equivalent to a SPIR-V subgroup.
+  SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
+
+  // Retreive the operation to use based on input type
+  bool IsFloatTy = GR.isScalarOrVectorOfType(InputRegister, SPIRV::OpTypeFloat);
+  auto Opcode =
+      IsFloatTy ? SPIRV::OpGroupNonUniformFAdd : SPIRV::OpGroupNonUniformIAdd;
+  return BuildMI(BB, I, I.getDebugLoc(), TII.get(Opcode))
+      .addDef(ResVReg)
+      .addUse(GR.getSPIRVTypeID(ResType))
+      .addUse(GR.getOrCreateConstInt(3, I, IntTy, TII))
+      .addImm(0) // group operation set to 0 to denote a reduction operation
+      .addUse(I.getOperand(2).getReg());
+}
+
 bool SPIRVInstructionSelector::selectBitreverse(Register ResVReg,
                                                 const SPIRVType *ResType,
                                                 MachineInstr &I) const {
@@ -2559,6 +2590,8 @@ bool SPIRVInstructionSelector::selectIntrinsic(Register ResVReg,
   } break;
   case Intrinsic::spv_saturate:
     return selectSaturate(ResVReg, ResType, I);
+  case Intrinsic::spv_wave_active_sum:
+    return selectWaveActiveSum(ResVReg, ResType, I);
   case Intrinsic::spv_wave_is_first_lane: {
     SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
     return BuildMI(BB, I, I.getDebugLoc(),
diff --git a/llvm/test/CodeGen/SP...
[truncated]

@llvmbot
Copy link
Collaborator

llvmbot commented Oct 18, 2024

@llvm/pr-subscribers-clang

Author: Finn Plummer (inbelic)

Changes 
  - add clang builtin to Builtins.td
  - link builtin in hlsl_intrinsics
  - add codegen for spirv intrinsic and two directx intrinsics to retain
    signedness information of the operands in CGBuiltin.cpp
  - add semantic analysis in SemaHLSL.cpp
  - add lowering of spirv intrinsic to spirv backend in
    SPIRVInstructionSelector.cpp

  - add test cases to illustrate passes

Note that this defines the dx intrinsics but does not implement the
DirectX lowering to DXIL. This will implemented in a second pr when
the dependent pr merges.

Part 1 of #70106

Patch is 21.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/112400.diff

11 Files Affected:

  • (modified) clang/include/clang/Basic/Builtins.td (+6)
  • (modified) clang/include/clang/Basic/DiagnosticSemaKinds.td (+3)
  • (modified) clang/lib/CodeGen/CGBuiltin.cpp (+34)
  • (modified) clang/lib/Headers/hlsl/hlsl_intrinsics.h (+99)
  • (modified) clang/lib/Sema/SemaHLSL.cpp (+31)
  • (added) clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl (+48)
  • (added) clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl (+28)
  • (modified) llvm/include/llvm/IR/IntrinsicsDirectX.td (+2)
  • (modified) llvm/include/llvm/IR/IntrinsicsSPIRV.td (+1)
  • (modified) llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp (+33)
  • (added) llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveSum.ll (+41)
diff --git a/clang/include/clang/Basic/Builtins.td b/clang/include/clang/Basic/Builtins.td
index 382fb6b7a3c031..7205f10b01c2b9 100644
--- a/clang/include/clang/Basic/Builtins.td
+++ b/clang/include/clang/Basic/Builtins.td
@@ -4749,6 +4749,12 @@ def HLSLWaveActiveCountBits : LangBuiltin<"HLSL_LANG"> {
   let Prototype = "unsigned int(bool)";
 }
 
+def HLSLWaveActiveSum : LangBuiltin<"HLSL_LANG"> {
+  let Spellings = ["__builtin_hlsl_wave_active_sum"];
+  let Attributes = [NoThrow, Const];
+  let Prototype = "void (...)";
+}
+
 def HLSLWaveGetLaneIndex : LangBuiltin<"HLSL_LANG"> {
   let Spellings = ["__builtin_hlsl_wave_get_lane_index"];
   let Attributes = [NoThrow, Const];
diff --git a/clang/include/clang/Basic/DiagnosticSemaKinds.td b/clang/include/clang/Basic/DiagnosticSemaKinds.td
index c458a62d9be48c..95093723c2f495 100644
--- a/clang/include/clang/Basic/DiagnosticSemaKinds.td
+++ b/clang/include/clang/Basic/DiagnosticSemaKinds.td
@@ -9232,6 +9232,9 @@ def err_typecheck_expect_scalar_or_vector : Error<
   "a vector of such type is required">;
 def err_typecheck_expect_any_scalar_or_vector : Error<
   "invalid operand of type %0 where a scalar or vector is required">;
+def err_typecheck_expect_scalar_or_vector_not_type : Error<
+  "invalid operand of type %0 where %1 or "
+  "a vector of such type is not allowed">;
 def err_typecheck_expect_flt_or_vector : Error<
   "invalid operand of type %0 where floating, complex or "
   "a vector of such types is required">;
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 12f99d9f1178a9..46646492b30d31 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -18638,6 +18638,23 @@ static Intrinsic::ID getDotProductIntrinsic(CGHLSLRuntime &RT, QualType QT) {
   return RT.getUDotIntrinsic();
 }
 
+// Return wave active sum that corresponds to the QT scalar type
+static Intrinsic::ID getWaveActiveSumIntrinsic(llvm::Triple::ArchType Arch,
+                                               CGHLSLRuntime &RT, QualType QT) {
+  switch (Arch) {
+  case llvm::Triple::spirv:
+    return llvm::Intrinsic::spv_wave_active_sum;
+  case llvm::Triple::dxil: {
+    if (QT->isUnsignedIntegerType())
+      return llvm::Intrinsic::dx_wave_active_usum;
+    return llvm::Intrinsic::dx_wave_active_sum;
+  }
+  default:
+    llvm_unreachable("Intrinsic WaveActiveSum"
+                     " not supported by target architecture");
+  }
+}
+
 Value *CodeGenFunction::EmitHLSLBuiltinExpr(unsigned BuiltinID,
                                             const CallExpr *E,
                                             ReturnValueSlot ReturnValue) {
@@ -18883,6 +18900,23 @@ case Builtin::BI__builtin_hlsl_elementwise_isinf: {
         /*ReturnType=*/Op0->getType(), CGM.getHLSLRuntime().getStepIntrinsic(),
         ArrayRef<Value *>{Op0, Op1}, nullptr, "hlsl.step");
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    // Due to the use of variadic arguments, explicitly retreive argument
+    Value *OpExpr = EmitScalarExpr(E->getArg(0));
+    llvm::FunctionType *FT = llvm::FunctionType::get(
+        OpExpr->getType(), ArrayRef{OpExpr->getType()}, false);
+    Intrinsic::ID IID = getWaveActiveSumIntrinsic(
+        getTarget().getTriple().getArch(), CGM.getHLSLRuntime(),
+        E->getArg(0)->getType());
+
+    // Get overloaded name
+    std::string Name =
+        Intrinsic::getName(IID, ArrayRef{OpExpr->getType()}, &CGM.getModule());
+    return EmitRuntimeCall(CGM.CreateRuntimeFunction(FT, Name, {},
+                                                     /*Local=*/false,
+                                                     /*AssumeConvergent=*/true),
+                           ArrayRef{OpExpr}, "hlsl.wave.active.sum");
+  }
   case Builtin::BI__builtin_hlsl_wave_get_lane_index: {
     // We don't define a SPIR-V intrinsic, instead it is a SPIR-V built-in
     // defined in SPIRVBuiltins.td. So instead we manually get the matching name
diff --git a/clang/lib/Headers/hlsl/hlsl_intrinsics.h b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
index 30dce60b3ff702..6b2843ca40c2e5 100644
--- a/clang/lib/Headers/hlsl/hlsl_intrinsics.h
+++ b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
@@ -2177,6 +2177,105 @@ __attribute__((convergent)) double3 WaveReadLaneAt(double3, int32_t);
 _HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_read_lane_at)
 __attribute__((convergent)) double4 WaveReadLaneAt(double4, int32_t);
 
+//===----------------------------------------------------------------------===//
+// WaveActiveSum builtins
+//===----------------------------------------------------------------------===//
+
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half WaveActiveSum(half);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half2 WaveActiveSum(half2);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half3 WaveActiveSum(half3);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half4 WaveActiveSum(half4);
+
+#ifdef __HLSL_ENABLE_16_BIT
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t WaveActiveSum(int16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t2 WaveActiveSum(int16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t3 WaveActiveSum(int16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t4 WaveActiveSum(int16_t4);
+
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t WaveActiveSum(uint16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t2 WaveActiveSum(uint16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t3 WaveActiveSum(uint16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t4 WaveActiveSum(uint16_t4);
+#endif
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int WaveActiveSum(int);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int2 WaveActiveSum(int2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int3 WaveActiveSum(int3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int4 WaveActiveSum(int4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint WaveActiveSum(uint);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint2 WaveActiveSum(uint2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint3 WaveActiveSum(uint3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint4 WaveActiveSum(uint4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t WaveActiveSum(int64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t2 WaveActiveSum(int64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t3 WaveActiveSum(int64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t4 WaveActiveSum(int64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t WaveActiveSum(uint64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t2 WaveActiveSum(uint64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t3 WaveActiveSum(uint64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t4 WaveActiveSum(uint64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float WaveActiveSum(float);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float2 WaveActiveSum(float2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float3 WaveActiveSum(float3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float4 WaveActiveSum(float4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double WaveActiveSum(double);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double2 WaveActiveSum(double2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double3 WaveActiveSum(double3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double4 WaveActiveSum(double4);
+
 //===----------------------------------------------------------------------===//
 // sign builtins
 //===----------------------------------------------------------------------===//
diff --git a/clang/lib/Sema/SemaHLSL.cpp b/clang/lib/Sema/SemaHLSL.cpp
index 698fdbed0484e5..7ad4ccc551c7af 100644
--- a/clang/lib/Sema/SemaHLSL.cpp
+++ b/clang/lib/Sema/SemaHLSL.cpp
@@ -1767,6 +1767,23 @@ static bool CheckAnyScalarOrVector(Sema *S, CallExpr *TheCall,
   return false;
 }
 
+static bool CheckNotScalarType(Sema *S, CallExpr *TheCall, QualType Scalar,
+                               unsigned ArgIndex) {
+  assert(TheCall->getNumArgs() >= ArgIndex);
+  QualType ArgType = TheCall->getArg(ArgIndex)->getType();
+  auto *VTy = ArgType->getAs<VectorType>();
+  // is the scalar or vector<scalar>
+  if (S->Context.hasSameUnqualifiedType(ArgType, Scalar) ||
+      (VTy &&
+       S->Context.hasSameUnqualifiedType(VTy->getElementType(), Scalar))) {
+    S->Diag(TheCall->getArg(0)->getBeginLoc(),
+            diag::err_typecheck_expect_scalar_or_vector_not_type)
+        << ArgType << Scalar;
+    return true;
+  }
+  return false;
+}
+
 static bool CheckBoolSelect(Sema *S, CallExpr *TheCall) {
   assert(TheCall->getNumArgs() == 3);
   Expr *Arg1 = TheCall->getArg(1);
@@ -2002,6 +2019,20 @@ bool SemaHLSL::CheckBuiltinFunctionCall(unsigned BuiltinID, CallExpr *TheCall) {
     TheCall->setType(ArgTyA);
     break;
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    if (SemaRef.checkArgCount(TheCall, 1))
+      return true;
+
+    // Ensure input expr type is a scalar/vector and the same as the return type
+    if (CheckAnyScalarOrVector(&SemaRef, TheCall, 0))
+      return true;
+    if (CheckNotScalarType(&SemaRef, TheCall, getASTContext().BoolTy, 0))
+      return true;
+    ExprResult Expr = TheCall->getArg(0);
+    QualType ArgTyExpr = Expr.get()->getType();
+    TheCall->setType(ArgTyExpr);
+    break;
+  }
   // Note these are llvm builtins that we want to catch invalid intrinsic
   // generation. Normal handling of these builitns will occur elsewhere.
   case Builtin::BI__builtin_elementwise_bitreverse: {
diff --git a/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
new file mode 100644
index 00000000000000..e5e463a6e9beea
--- /dev/null
+++ b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
@@ -0,0 +1,48 @@
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   dxil-pc-shadermodel6.3-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-DXIL
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   spirv-pc-vulkan-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-SPIRV
+
+// Test basic lowering to runtime function call.
+
+// CHECK-LABEL: test_int
+int test_int(int expr) {
+  // CHECK-SPIRV: %[[#entry_tok:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i32([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.sum.i32([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.sum.i32([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i32([[TY]]) #[[#attr:]]
+
+// CHECK-LABEL: test_uint64_t
+uint64_t test_uint64_t(uint64_t expr) {
+  // CHECK-SPIRV: %[[#entry_tok1:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i64([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok1]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.usum.i64([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.usum.i64([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i64([[TY]]) #[[#attr:]]
+
+// Test basic lowering to runtime function call with array and float value.
+
+// CHECK-LABEL: test_floatv4
+float4 test_floatv4(float4 expr) {
+  // CHECK-SPIRV: %[[#entry_tok2:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.spv.wave.active.sum.v4f32([[TY1]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok2]]) ]
+  // CHECK-DXIL:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.dx.wave.active.sum.v4f32([[TY1]] %[[#]])
+  // CHECK:  ret [[TY1]] %[[RET1]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY1]] @llvm.dx.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+// CHECK-SPIRV: declare [[TY1]] @llvm.spv.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+
+// CHECK: attributes #[[#attr]] = {{{.*}} convergent {{.*}}}
diff --git a/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
new file mode 100644
index 00000000000000..e475df8994f3e7
--- /dev/null
+++ b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
@@ -0,0 +1,28 @@
+// RUN: %clang_cc1 -finclude-default-header -triple dxil-pc-shadermodel6.6-library %s -emit-llvm-only -disable-llvm-passes -verify
+
+int test_too_few_arg() {
+  return __builtin_hlsl_wave_active_sum();
+  // expected-error@-1 {{too few arguments to function call, expected 1, have 0}}
+}
+
+float2 test_too_many_arg(float2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0, p0);
+  // expected-error@-1 {{too many arguments to function call, expected 1, have 2}}
+}
+
+bool test_expr_bool_type_check(bool p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool' where 'bool' or a vector of such type is not allowed}}
+}
+
+bool2 test_expr_bool_vec_type_check(bool2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool2' (aka 'vector<bool, 2>') where 'bool' or a vector of such type is not allowed}}
+}
+
+struct S { float f; };
+
+S test_expr_struct_type_check(S p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'S' where a scalar or vector is required}}
+}
diff --git a/llvm/include/llvm/IR/IntrinsicsDirectX.td b/llvm/include/llvm/IR/IntrinsicsDirectX.td
index 27a437a83be6dd..0ad6c5e1aa2351 100644
--- a/llvm/include/llvm/IR/IntrinsicsDirectX.td
+++ b/llvm/include/llvm/IR/IntrinsicsDirectX.td
@@ -85,6 +85,8 @@ def int_dx_umad : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>, LLV
 def int_dx_normalize : DefaultAttrsIntrinsic<[LLVMMatchType<0>], [llvm_anyfloat_ty], [IntrNoMem]>;
 def int_dx_rsqrt  : DefaultAttrsIntrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>], [IntrNoMem]>;
 def int_dx_wave_getlaneindex : DefaultAttrsIntrinsic<[llvm_i32_ty], [], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_usum : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
 def int_dx_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
 def int_dx_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
 def int_dx_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/include/llvm/IR/IntrinsicsSPIRV.td b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
index 6df2eb156a0774..b90190b8963dbe 100644
--- a/llvm/include/llvm/IR/IntrinsicsSPIRV.td
+++ b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
@@ -83,6 +83,7 @@ let TargetPrefix = "spv" in {
     DefaultAttrsIntrinsic<[LLVMVectorElementType<0>],
     [llvm_anyint_ty, LLVMScalarOrSameVectorWidth<0, LLVMVectorElementType<0>>],
     [IntrNoMem, Commutative] >;
+  def int_spv_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
   def int_spv_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
   def int_spv_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
   def int_spv_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
index d9377fe4b91a1a..54ab680d27b88e 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
@@ -164,6 +164,9 @@ class SPIRVInstructionSelector : public InstructionSelector {
   bool selectIntegerDot(Register ResVReg, const SPIRVType *ResType,
                         MachineInstr &I) const;
 
+  bool selectWaveActiveSum(Register ResVReg, const SPIRVType *ResType,
+                           MachineInstr &I) const;
+
   void renderImm32(MachineInstrBuilder &MIB, const MachineInstr &I,
                    int OpIdx) const;
   void renderFImm64(MachineInstrBuilder &MIB, const MachineInstr &I,
@@ -1782,6 +1785,34 @@ bool SPIRVInstructionSelector::selectWaveReadLaneAt(Register ResVReg,
       .addUse(I.getOperand(3).getReg());
 }
 
+bool SPIRVInstructionSelector::selectWaveActiveSum(Register ResVReg,
+                                                   const SPIRVType *ResType,
+                                                   MachineInstr &I) const {
+  assert(I.getNumOperands() == 3);
+  assert(I.getOperand(2).isReg());
+  MachineBasicBlock &BB = *I.getParent();
+  Register InputRegister = I.getOperand(2).getReg();
+  SPIRVType *InputType = GR.getSPIRVTypeForVReg(InputRegister);
+
+  if (!InputType)
+    report_fatal_error("Input Type could not be determined.");
+
+  // IntTy is used to define the execution scope, set to 3 to denote a
+  // cross-lane interaction equivalent to a SPIR-V subgroup.
+  SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
+
+  // Retreive the operation to use based on input type
+  bool IsFloatTy = GR.isScalarOrVectorOfType(InputRegister, SPIRV::OpTypeFloat);
+  auto Opcode =
+      IsFloatTy ? SPIRV::OpGroupNonUniformFAdd : SPIRV::OpGroupNonUniformIAdd;
+  return BuildMI(BB, I, I.getDebugLoc(), TII.get(Opcode))
+      .addDef(ResVReg)
+      .addUse(GR.getSPIRVTypeID(ResType))
+      .addUse(GR.getOrCreateConstInt(3, I, IntTy, TII))
+      .addImm(0) // group operation set to 0 to denote a reduction operation
+      .addUse(I.getOperand(2).getReg());
+}
+
 bool SPIRVInstructionSelector::selectBitreverse(Register ResVReg,
                                                 const SPIRVType *ResType,
                                                 MachineInstr &I) const {
@@ -2559,6 +2590,8 @@ bool SPIRVInstructionSelector::selectIntrinsic(Register ResVReg,
   } break;
   case Intrinsic::spv_saturate:
     return selectSaturate(ResVReg, ResType, I);
+  case Intrinsic::spv_wave_active_sum:
+    return selectWaveActiveSum(ResVReg, ResType, I);
   case Intrinsic::spv_wave_is_first_lane: {
     SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
     return BuildMI(BB, I, I.getDebugLoc(),
diff --git a/llvm/test/CodeGen/SP...
[truncated]

@llvmbot
Copy link
Collaborator

llvmbot commented Oct 18, 2024

@llvm/pr-subscribers-clang-codegen

Author: Finn Plummer (inbelic)

Changes 
  - add clang builtin to Builtins.td
  - link builtin in hlsl_intrinsics
  - add codegen for spirv intrinsic and two directx intrinsics to retain
    signedness information of the operands in CGBuiltin.cpp
  - add semantic analysis in SemaHLSL.cpp
  - add lowering of spirv intrinsic to spirv backend in
    SPIRVInstructionSelector.cpp

  - add test cases to illustrate passes

Note that this defines the dx intrinsics but does not implement the
DirectX lowering to DXIL. This will implemented in a second pr when
the dependent pr merges.

Part 1 of #70106

Patch is 21.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/112400.diff

11 Files Affected:

  • (modified) clang/include/clang/Basic/Builtins.td (+6)
  • (modified) clang/include/clang/Basic/DiagnosticSemaKinds.td (+3)
  • (modified) clang/lib/CodeGen/CGBuiltin.cpp (+34)
  • (modified) clang/lib/Headers/hlsl/hlsl_intrinsics.h (+99)
  • (modified) clang/lib/Sema/SemaHLSL.cpp (+31)
  • (added) clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl (+48)
  • (added) clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl (+28)
  • (modified) llvm/include/llvm/IR/IntrinsicsDirectX.td (+2)
  • (modified) llvm/include/llvm/IR/IntrinsicsSPIRV.td (+1)
  • (modified) llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp (+33)
  • (added) llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveSum.ll (+41)
diff --git a/clang/include/clang/Basic/Builtins.td b/clang/include/clang/Basic/Builtins.td
index 382fb6b7a3c031..7205f10b01c2b9 100644
--- a/clang/include/clang/Basic/Builtins.td
+++ b/clang/include/clang/Basic/Builtins.td
@@ -4749,6 +4749,12 @@ def HLSLWaveActiveCountBits : LangBuiltin<"HLSL_LANG"> {
   let Prototype = "unsigned int(bool)";
 }
 
+def HLSLWaveActiveSum : LangBuiltin<"HLSL_LANG"> {
+  let Spellings = ["__builtin_hlsl_wave_active_sum"];
+  let Attributes = [NoThrow, Const];
+  let Prototype = "void (...)";
+}
+
 def HLSLWaveGetLaneIndex : LangBuiltin<"HLSL_LANG"> {
   let Spellings = ["__builtin_hlsl_wave_get_lane_index"];
   let Attributes = [NoThrow, Const];
diff --git a/clang/include/clang/Basic/DiagnosticSemaKinds.td b/clang/include/clang/Basic/DiagnosticSemaKinds.td
index c458a62d9be48c..95093723c2f495 100644
--- a/clang/include/clang/Basic/DiagnosticSemaKinds.td
+++ b/clang/include/clang/Basic/DiagnosticSemaKinds.td
@@ -9232,6 +9232,9 @@ def err_typecheck_expect_scalar_or_vector : Error<
   "a vector of such type is required">;
 def err_typecheck_expect_any_scalar_or_vector : Error<
   "invalid operand of type %0 where a scalar or vector is required">;
+def err_typecheck_expect_scalar_or_vector_not_type : Error<
+  "invalid operand of type %0 where %1 or "
+  "a vector of such type is not allowed">;
 def err_typecheck_expect_flt_or_vector : Error<
   "invalid operand of type %0 where floating, complex or "
   "a vector of such types is required">;
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 12f99d9f1178a9..46646492b30d31 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -18638,6 +18638,23 @@ static Intrinsic::ID getDotProductIntrinsic(CGHLSLRuntime &RT, QualType QT) {
   return RT.getUDotIntrinsic();
 }
 
+// Return wave active sum that corresponds to the QT scalar type
+static Intrinsic::ID getWaveActiveSumIntrinsic(llvm::Triple::ArchType Arch,
+                                               CGHLSLRuntime &RT, QualType QT) {
+  switch (Arch) {
+  case llvm::Triple::spirv:
+    return llvm::Intrinsic::spv_wave_active_sum;
+  case llvm::Triple::dxil: {
+    if (QT->isUnsignedIntegerType())
+      return llvm::Intrinsic::dx_wave_active_usum;
+    return llvm::Intrinsic::dx_wave_active_sum;
+  }
+  default:
+    llvm_unreachable("Intrinsic WaveActiveSum"
+                     " not supported by target architecture");
+  }
+}
+
 Value *CodeGenFunction::EmitHLSLBuiltinExpr(unsigned BuiltinID,
                                             const CallExpr *E,
                                             ReturnValueSlot ReturnValue) {
@@ -18883,6 +18900,23 @@ case Builtin::BI__builtin_hlsl_elementwise_isinf: {
         /*ReturnType=*/Op0->getType(), CGM.getHLSLRuntime().getStepIntrinsic(),
         ArrayRef<Value *>{Op0, Op1}, nullptr, "hlsl.step");
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    // Due to the use of variadic arguments, explicitly retreive argument
+    Value *OpExpr = EmitScalarExpr(E->getArg(0));
+    llvm::FunctionType *FT = llvm::FunctionType::get(
+        OpExpr->getType(), ArrayRef{OpExpr->getType()}, false);
+    Intrinsic::ID IID = getWaveActiveSumIntrinsic(
+        getTarget().getTriple().getArch(), CGM.getHLSLRuntime(),
+        E->getArg(0)->getType());
+
+    // Get overloaded name
+    std::string Name =
+        Intrinsic::getName(IID, ArrayRef{OpExpr->getType()}, &CGM.getModule());
+    return EmitRuntimeCall(CGM.CreateRuntimeFunction(FT, Name, {},
+                                                     /*Local=*/false,
+                                                     /*AssumeConvergent=*/true),
+                           ArrayRef{OpExpr}, "hlsl.wave.active.sum");
+  }
   case Builtin::BI__builtin_hlsl_wave_get_lane_index: {
     // We don't define a SPIR-V intrinsic, instead it is a SPIR-V built-in
     // defined in SPIRVBuiltins.td. So instead we manually get the matching name
diff --git a/clang/lib/Headers/hlsl/hlsl_intrinsics.h b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
index 30dce60b3ff702..6b2843ca40c2e5 100644
--- a/clang/lib/Headers/hlsl/hlsl_intrinsics.h
+++ b/clang/lib/Headers/hlsl/hlsl_intrinsics.h
@@ -2177,6 +2177,105 @@ __attribute__((convergent)) double3 WaveReadLaneAt(double3, int32_t);
 _HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_read_lane_at)
 __attribute__((convergent)) double4 WaveReadLaneAt(double4, int32_t);
 
+//===----------------------------------------------------------------------===//
+// WaveActiveSum builtins
+//===----------------------------------------------------------------------===//
+
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half WaveActiveSum(half);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half2 WaveActiveSum(half2);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half3 WaveActiveSum(half3);
+_HLSL_16BIT_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) half4 WaveActiveSum(half4);
+
+#ifdef __HLSL_ENABLE_16_BIT
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t WaveActiveSum(int16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t2 WaveActiveSum(int16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t3 WaveActiveSum(int16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int16_t4 WaveActiveSum(int16_t4);
+
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t WaveActiveSum(uint16_t);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t2 WaveActiveSum(uint16_t2);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t3 WaveActiveSum(uint16_t3);
+_HLSL_AVAILABILITY(shadermodel, 6.0)
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint16_t4 WaveActiveSum(uint16_t4);
+#endif
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int WaveActiveSum(int);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int2 WaveActiveSum(int2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int3 WaveActiveSum(int3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int4 WaveActiveSum(int4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint WaveActiveSum(uint);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint2 WaveActiveSum(uint2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint3 WaveActiveSum(uint3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint4 WaveActiveSum(uint4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t WaveActiveSum(int64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t2 WaveActiveSum(int64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t3 WaveActiveSum(int64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) int64_t4 WaveActiveSum(int64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t WaveActiveSum(uint64_t);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t2 WaveActiveSum(uint64_t2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t3 WaveActiveSum(uint64_t3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) uint64_t4 WaveActiveSum(uint64_t4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float WaveActiveSum(float);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float2 WaveActiveSum(float2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float3 WaveActiveSum(float3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) float4 WaveActiveSum(float4);
+
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double WaveActiveSum(double);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double2 WaveActiveSum(double2);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double3 WaveActiveSum(double3);
+_HLSL_BUILTIN_ALIAS(__builtin_hlsl_wave_active_sum)
+__attribute((convergent)) double4 WaveActiveSum(double4);
+
 //===----------------------------------------------------------------------===//
 // sign builtins
 //===----------------------------------------------------------------------===//
diff --git a/clang/lib/Sema/SemaHLSL.cpp b/clang/lib/Sema/SemaHLSL.cpp
index 698fdbed0484e5..7ad4ccc551c7af 100644
--- a/clang/lib/Sema/SemaHLSL.cpp
+++ b/clang/lib/Sema/SemaHLSL.cpp
@@ -1767,6 +1767,23 @@ static bool CheckAnyScalarOrVector(Sema *S, CallExpr *TheCall,
   return false;
 }
 
+static bool CheckNotScalarType(Sema *S, CallExpr *TheCall, QualType Scalar,
+                               unsigned ArgIndex) {
+  assert(TheCall->getNumArgs() >= ArgIndex);
+  QualType ArgType = TheCall->getArg(ArgIndex)->getType();
+  auto *VTy = ArgType->getAs<VectorType>();
+  // is the scalar or vector<scalar>
+  if (S->Context.hasSameUnqualifiedType(ArgType, Scalar) ||
+      (VTy &&
+       S->Context.hasSameUnqualifiedType(VTy->getElementType(), Scalar))) {
+    S->Diag(TheCall->getArg(0)->getBeginLoc(),
+            diag::err_typecheck_expect_scalar_or_vector_not_type)
+        << ArgType << Scalar;
+    return true;
+  }
+  return false;
+}
+
 static bool CheckBoolSelect(Sema *S, CallExpr *TheCall) {
   assert(TheCall->getNumArgs() == 3);
   Expr *Arg1 = TheCall->getArg(1);
@@ -2002,6 +2019,20 @@ bool SemaHLSL::CheckBuiltinFunctionCall(unsigned BuiltinID, CallExpr *TheCall) {
     TheCall->setType(ArgTyA);
     break;
   }
+  case Builtin::BI__builtin_hlsl_wave_active_sum: {
+    if (SemaRef.checkArgCount(TheCall, 1))
+      return true;
+
+    // Ensure input expr type is a scalar/vector and the same as the return type
+    if (CheckAnyScalarOrVector(&SemaRef, TheCall, 0))
+      return true;
+    if (CheckNotScalarType(&SemaRef, TheCall, getASTContext().BoolTy, 0))
+      return true;
+    ExprResult Expr = TheCall->getArg(0);
+    QualType ArgTyExpr = Expr.get()->getType();
+    TheCall->setType(ArgTyExpr);
+    break;
+  }
   // Note these are llvm builtins that we want to catch invalid intrinsic
   // generation. Normal handling of these builitns will occur elsewhere.
   case Builtin::BI__builtin_elementwise_bitreverse: {
diff --git a/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
new file mode 100644
index 00000000000000..e5e463a6e9beea
--- /dev/null
+++ b/clang/test/CodeGenHLSL/builtins/WaveActiveSum.hlsl
@@ -0,0 +1,48 @@
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   dxil-pc-shadermodel6.3-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-DXIL
+// RUN: %clang_cc1 -std=hlsl2021 -finclude-default-header -triple \
+// RUN:   spirv-pc-vulkan-compute %s -emit-llvm -disable-llvm-passes -o - | \
+// RUN:   FileCheck %s --check-prefixes=CHECK,CHECK-SPIRV
+
+// Test basic lowering to runtime function call.
+
+// CHECK-LABEL: test_int
+int test_int(int expr) {
+  // CHECK-SPIRV: %[[#entry_tok:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i32([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.sum.i32([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.sum.i32([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i32([[TY]]) #[[#attr:]]
+
+// CHECK-LABEL: test_uint64_t
+uint64_t test_uint64_t(uint64_t expr) {
+  // CHECK-SPIRV: %[[#entry_tok1:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET:.*]] = call [[TY:.*]] @llvm.spv.wave.active.sum.i64([[TY]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok1]]) ]
+  // CHECK-DXIL:  %[[RET:.*]] = call [[TY:.*]] @llvm.dx.wave.active.usum.i64([[TY]] %[[#]])
+  // CHECK:  ret [[TY]] %[[RET]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY]] @llvm.dx.wave.active.usum.i64([[TY]]) #[[#attr:]]
+// CHECK-SPIRV: declare [[TY]] @llvm.spv.wave.active.sum.i64([[TY]]) #[[#attr:]]
+
+// Test basic lowering to runtime function call with array and float value.
+
+// CHECK-LABEL: test_floatv4
+float4 test_floatv4(float4 expr) {
+  // CHECK-SPIRV: %[[#entry_tok2:]] = call token @llvm.experimental.convergence.entry()
+  // CHECK-SPIRV:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.spv.wave.active.sum.v4f32([[TY1]] %[[#]]) [ "convergencectrl"(token %[[#entry_tok2]]) ]
+  // CHECK-DXIL:  %[[RET1:.*]] = call [[TY1:.*]] @llvm.dx.wave.active.sum.v4f32([[TY1]] %[[#]])
+  // CHECK:  ret [[TY1]] %[[RET1]]
+  return WaveActiveSum(expr);
+}
+
+// CHECK-DXIL: declare [[TY1]] @llvm.dx.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+// CHECK-SPIRV: declare [[TY1]] @llvm.spv.wave.active.sum.v4f32([[TY1]]) #[[#attr]]
+
+// CHECK: attributes #[[#attr]] = {{{.*}} convergent {{.*}}}
diff --git a/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
new file mode 100644
index 00000000000000..e475df8994f3e7
--- /dev/null
+++ b/clang/test/SemaHLSL/BuiltIns/WaveActiveSum-errors.hlsl
@@ -0,0 +1,28 @@
+// RUN: %clang_cc1 -finclude-default-header -triple dxil-pc-shadermodel6.6-library %s -emit-llvm-only -disable-llvm-passes -verify
+
+int test_too_few_arg() {
+  return __builtin_hlsl_wave_active_sum();
+  // expected-error@-1 {{too few arguments to function call, expected 1, have 0}}
+}
+
+float2 test_too_many_arg(float2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0, p0);
+  // expected-error@-1 {{too many arguments to function call, expected 1, have 2}}
+}
+
+bool test_expr_bool_type_check(bool p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool' where 'bool' or a vector of such type is not allowed}}
+}
+
+bool2 test_expr_bool_vec_type_check(bool2 p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'bool2' (aka 'vector<bool, 2>') where 'bool' or a vector of such type is not allowed}}
+}
+
+struct S { float f; };
+
+S test_expr_struct_type_check(S p0) {
+  return __builtin_hlsl_wave_active_sum(p0);
+  // expected-error@-1 {{invalid operand of type 'S' where a scalar or vector is required}}
+}
diff --git a/llvm/include/llvm/IR/IntrinsicsDirectX.td b/llvm/include/llvm/IR/IntrinsicsDirectX.td
index 27a437a83be6dd..0ad6c5e1aa2351 100644
--- a/llvm/include/llvm/IR/IntrinsicsDirectX.td
+++ b/llvm/include/llvm/IR/IntrinsicsDirectX.td
@@ -85,6 +85,8 @@ def int_dx_umad : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>, LLV
 def int_dx_normalize : DefaultAttrsIntrinsic<[LLVMMatchType<0>], [llvm_anyfloat_ty], [IntrNoMem]>;
 def int_dx_rsqrt  : DefaultAttrsIntrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>], [IntrNoMem]>;
 def int_dx_wave_getlaneindex : DefaultAttrsIntrinsic<[llvm_i32_ty], [], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
+def int_dx_wave_active_usum : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
 def int_dx_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
 def int_dx_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
 def int_dx_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/include/llvm/IR/IntrinsicsSPIRV.td b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
index 6df2eb156a0774..b90190b8963dbe 100644
--- a/llvm/include/llvm/IR/IntrinsicsSPIRV.td
+++ b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
@@ -83,6 +83,7 @@ let TargetPrefix = "spv" in {
     DefaultAttrsIntrinsic<[LLVMVectorElementType<0>],
     [llvm_anyint_ty, LLVMScalarOrSameVectorWidth<0, LLVMVectorElementType<0>>],
     [IntrNoMem, Commutative] >;
+  def int_spv_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
   def int_spv_wave_is_first_lane : DefaultAttrsIntrinsic<[llvm_i1_ty], [], [IntrConvergent]>;
   def int_spv_wave_readlane : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>, llvm_i32_ty], [IntrConvergent, IntrNoMem]>;
   def int_spv_sign : DefaultAttrsIntrinsic<[LLVMScalarOrSameVectorWidth<0, llvm_i32_ty>], [llvm_any_ty], [IntrNoMem]>;
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
index d9377fe4b91a1a..54ab680d27b88e 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
@@ -164,6 +164,9 @@ class SPIRVInstructionSelector : public InstructionSelector {
   bool selectIntegerDot(Register ResVReg, const SPIRVType *ResType,
                         MachineInstr &I) const;
 
+  bool selectWaveActiveSum(Register ResVReg, const SPIRVType *ResType,
+                           MachineInstr &I) const;
+
   void renderImm32(MachineInstrBuilder &MIB, const MachineInstr &I,
                    int OpIdx) const;
   void renderFImm64(MachineInstrBuilder &MIB, const MachineInstr &I,
@@ -1782,6 +1785,34 @@ bool SPIRVInstructionSelector::selectWaveReadLaneAt(Register ResVReg,
       .addUse(I.getOperand(3).getReg());
 }
 
+bool SPIRVInstructionSelector::selectWaveActiveSum(Register ResVReg,
+                                                   const SPIRVType *ResType,
+                                                   MachineInstr &I) const {
+  assert(I.getNumOperands() == 3);
+  assert(I.getOperand(2).isReg());
+  MachineBasicBlock &BB = *I.getParent();
+  Register InputRegister = I.getOperand(2).getReg();
+  SPIRVType *InputType = GR.getSPIRVTypeForVReg(InputRegister);
+
+  if (!InputType)
+    report_fatal_error("Input Type could not be determined.");
+
+  // IntTy is used to define the execution scope, set to 3 to denote a
+  // cross-lane interaction equivalent to a SPIR-V subgroup.
+  SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
+
+  // Retreive the operation to use based on input type
+  bool IsFloatTy = GR.isScalarOrVectorOfType(InputRegister, SPIRV::OpTypeFloat);
+  auto Opcode =
+      IsFloatTy ? SPIRV::OpGroupNonUniformFAdd : SPIRV::OpGroupNonUniformIAdd;
+  return BuildMI(BB, I, I.getDebugLoc(), TII.get(Opcode))
+      .addDef(ResVReg)
+      .addUse(GR.getSPIRVTypeID(ResType))
+      .addUse(GR.getOrCreateConstInt(3, I, IntTy, TII))
+      .addImm(0) // group operation set to 0 to denote a reduction operation
+      .addUse(I.getOperand(2).getReg());
+}
+
 bool SPIRVInstructionSelector::selectBitreverse(Register ResVReg,
                                                 const SPIRVType *ResType,
                                                 MachineInstr &I) const {
@@ -2559,6 +2590,8 @@ bool SPIRVInstructionSelector::selectIntrinsic(Register ResVReg,
   } break;
   case Intrinsic::spv_saturate:
     return selectSaturate(ResVReg, ResType, I);
+  case Intrinsic::spv_wave_active_sum:
+    return selectWaveActiveSum(ResVReg, ResType, I);
   case Intrinsic::spv_wave_is_first_lane: {
     SPIRVType *IntTy = GR.getOrCreateSPIRVIntegerType(32, I, TII);
     return BuildMI(BB, I, I.getDebugLoc(),
diff --git a/llvm/test/CodeGen/SP...
[truncated]

@inbelic
[HLSL][SPIRV][DXIL] Implement WaveActiveSum intrinsic
cb507cc 
- add clang builtin to Builtins.td
  - link builtin in hlsl_intrinsics
  - add codegen for spirv intrinsic and two directx intrinsics to retain
    signedness information of the operands in CGBuiltin.cpp
  - add semantic analysis in SemaHLSL.cpp
  - add lowering of spirv intrinsic to spirv backend in
    SPIRVInstructionSelector.cpp
  - add lowering of directx intrinsics to WaveActiveOp dxil op in
DXIL.td

  - add test cases to illustrate passes
inbelic
inbelic commented Oct 29, 2024
View reviewed changes
llvm/include/llvm/IR/IntrinsicsDirectX.td
@@ -85,6 +85,8 @@ def int_dx_umad : DefaultAttrsIntrinsic<[llvm_anyint_ty], [LLVMMatchType<0>, LLV
def int_dx_normalize : DefaultAttrsIntrinsic<[LLVMMatchType<0>], [llvm_anyfloat_ty], [IntrNoMem]>;
def int_dx_rsqrt : DefaultAttrsIntrinsic<[llvm_anyfloat_ty], [LLVMMatchType<0>], [IntrNoMem]>;
def int_dx_wave_getlaneindex : DefaultAttrsIntrinsic<[llvm_i32_ty], [], [IntrConvergent, IntrNoMem]>;
def int_dx_wave_active_sum : DefaultAttrsIntrinsic<[llvm_any_ty], [LLVMMatchType<0>], [IntrConvergent, IntrNoMem]>;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could further be split into int_dx_wave_active_fsum and int_dx_wave_active_ssum to denote the difference in float/signed integer. They were merged into one as the separation is not required for lowering to WaveActiveOp. Would appreciate if reviewers weighed in. Could increase readability/documentation.

@inbelic inbelic changed the title [HLSL][SPIRV] Implement WaveActiveSum intrinsic [HLSL][SPIRV][DXIL] Implement WaveActiveSum intrinsic Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backend:DirectX backend:SPIR-V backend:X86 clang:codegen clang:frontend Language frontend issues, e.g. anything involving "Sema" clang:headers Headers provided by Clang, e.g. for intrinsics clang Clang issues not falling into any other category HLSL HLSL Language Support llvm:ir
Projects
HLSL Support
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Implement the WaveActiveSum HLSL Function
2 participants
@inbelic @llvmbot