Skip to content

Commit

Permalink
Add new customization points in RendererServices for reporting errors…
Browse files Browse the repository at this point in the history
…, warnings, printf, and fprintf.

These customization points accept a fmtlib style format specifier (vs. printf style) whose argument types are represented as EncodedTypes (encodedtypes.h) and can be subsequently decoded via a utility function.

Removes dependence on ShadingContext and OSL::ErrorHandler for logging error, warning, printf, and fprintf calls
for use on CPU and device during Shader execution.  ShadingContext and OSL::ErrorHandler remain utilized during shader compilation/JIT.

Journal Buffer approach:  each thread gets its own chain of pages within the shared journal buffer for recording errors, warnings, etc.  By maintaining a separate chain per thread we can record errors (etc.) in a non-blocking fashion. Shade index is recorded for use by error reporter to sort messages but it is not used at present.
Using a Journal buffer is up to a Renderer.  The Renderer use the new customization points (rs_errorfmt, rs_filefmt, rs_printfmt, rs_warningfmt or their virtual function equivalents) to interact with a journal::Writer and journal buffer owned by its RenderState.  The Renderer would of have to allocate and initialized the journal buffer with the requested page size and number of threads before executing a shader. After a shader is done executing, the contents of the journal buffer can be transfered back to the host and processed with a journal::Reader, which reports them through a journal::Reporter's virtual interface. Intent is for Renderers's to use the Journal buffer and provide their own overriden version of the journal::Reporter.  Testshade provides example usage.  For legacy purposes, the default virtual RendererServices will route errors to ShadingSystem's OSL::ErrorHandler, but intent is for Renderer's to switch over to using a Journal buffer or their own custom logging.

NOTE:  OSL library functions as well use Renderer Service free functions can directly call OSL::filefmt, OSL::printfmt, OSL::errorfmt, OSL::warningfmt which wrappers to the underlying free functions (who may record the messages into a journal buffer).

Added a new hashcode datatype in typespec.cpp; represented as "h" in function arguments in builtindecl.
To print closures added a new OSL function osl_closure_to_ustringhash to return a ustringhash_pod value directly.

Shader execute functions now require a zero-based thread_index to identify the calling thread; used for journalling.
RendererServices::get_texture_info parameter ShadingContext * changed to ShaderGlobals * (aka ExecutionContext).

ShaderGlobals now tracks both shade_index and thread_index.
As we look to hide/remove ShaderGlobals in the future, we still need a context object to replace the role the ShaderGlobals struct is currently serving (just minus the actual shader global variables).  As a first step towards this goal, we introduce the concept of an ExecutionContext and OpaqueExecutionContext.  These just alias to ShaderGlobals currently, but start to establish the idea of hiding its contents.  Instead of direct memory access to the ExecutionContext/ShaderGlobals new getter functions have been added that accept an OpaqueExecutionContext and provide the requested data.  When inlined there should be no performance penalty, and reliance on the ShaderGlobals implementation detail will be reduced.  IE:
    inline const Vec3 & get_P(OpaqueExecContextPtr oec);
    template<typename RenderStateT> RenderStateT* get_rs(OpaqueExecContextPtr oec);
    inline int get_raytype(OpaqueExecContextPtr oec);
Updated free function renderer services and opcolor to use the ExecContext, but left the rest alone to minimize size of this PR.

We introduce opfmt.cpp that provides shim function for llvm_gen to call, where the shim adapts datatypes from llvm to c++ forwarding calls to the free function renderer service error, warning, print, fileprint functions.

Split llvm_gen_printf into llvm_gen_printf_legacy, for string format calls or when using optix, and the new llvm_gen_print_fmt which will convert the printf style specifier to the fmtlib format that the Renderer Service customization points are expecting.

fprintf by default no longer writes out to a file, is routed to journal::Reporter::report_file_print which could be overriden (example in TestShade to maintain existing behavior).  However for security reasons, we do not recommend allowing OSL shader's direct file system but instead just an annotated log entry.

To maintain existing behavior of disallowing output of repeated errors/warnings over a history window, we have provided 2 classes that could be used during reporting: TrackRecentlyReported and Report2ErrorHandler.  TrackRecentlyReported can turn on/off limitting errors and warnings and control the history capacity.  Report2ErrorHandler simply reports errors to an existing OSL::ErrorHandler to allow for easy integration.  However we expect Renderers to not use Report2ErrorHandler and provide their own to directly interact with their logging systems.

Signed-off-by: Steena Monteiro <[email protected]>
  • Loading branch information
steenax86 committed Jun 22, 2023
1 parent f2e33c0 commit 6441f10
Show file tree
Hide file tree
Showing 83 changed files with 3,664 additions and 1,981 deletions.
5 changes: 0 additions & 5 deletions src/cmake/compiler.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -182,11 +182,6 @@ if (CMAKE_COMPILER_IS_GNUCC AND NOT (CMAKE_COMPILER_IS_CLANG OR CMAKE_COMPILER_I
add_compile_options ("-Wno-error=strict-overflow")
add_compile_options ("-Wno-unused-local-typedefs")
add_compile_options ("-Wno-unused-result")
add_compile_options ("-Wno-attributes")
add_compile_options ("-Wno-error=sign-compare")
add_compile_options ("-Wno-error=class-memaccess")
add_compile_options ("-Wno-error=unused-variable")
add_compile_options ("-Wno-error=switch")
if (NOT ${GCC_VERSION} VERSION_LESS 6.0)
add_compile_options ("-Wno-error=misleading-indentation")
endif ()
Expand Down
14 changes: 14 additions & 0 deletions src/cmake/testing.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,17 @@ macro ( TESTSUITE )
add_one_testsuite ("${_testname}.opt" "${_testsrcdir}"
ENV TESTSHADE_OPT=2 )
endif ()
# Run the same test again with aggressive -O2 runtime
# optimization, triggered by setting TESTSHADE_OPT env variable.
# Skip OptiX-only tests and those with a NOOPTIMIZE marker file.
if (NOT _testname MATCHES "optix"
AND NOT EXISTS "${_testsrcdir}/NOSCALAR"
AND NOT EXISTS "${_testsrcdir}/BATCHED_REGRESSION"
AND NOT EXISTS "${_testsrcdir}/NOOPTIMIZE"
AND NOT EXISTS "${_testsrcdir}/NORSBITCODE")
add_one_testsuite ("${_testname}.opt.rs_bitcode" "${_testsrcdir}"
ENV TESTSHADE_OPT=2 TESTSHADE_RS_BITCODE=1)
endif ()
# When building for OptiX support, also run it in OptiX mode
# if there is an OPTIX marker file in the directory.
# If an environment variable $TESTSUITE_OPTIX is nonzero, then
Expand Down Expand Up @@ -346,6 +357,9 @@ macro (osl_add_all_tests)
struct-nested struct-nested-assign struct-nested-deep
ternary
testshade-expr
test-fmt-arrays test-fmt-fileprint
test-fmt-cxpf test-fmt-noise test-fmt-matrixcolor
test-fmt-stpf test-fmt-errorwarning test-fmt-errorwarning-repeats
texture-alpha texture-alpha-derivs
texture-blur texture-connected-options
texture-derivs texture-environment texture-errormsg
Expand Down
214 changes: 175 additions & 39 deletions src/include/OSL/encodedtypes.h
Original file line number Diff line number Diff line change
Expand Up @@ -6,61 +6,197 @@

#include <OSL/oslconfig.h>

OSL_NAMESPACE_ENTER


enum class EncodedType : uint8_t
{
kUstringHash,
// Substitute for variable argument list, which is traditionally used with a printf, with arrays
// of EncodedTypes to identify types contained in a blind payload of values.
enum class EncodedType : uint8_t {
// OSL Shaders could encode these types
kUstringHash = 0,
kInt32,
kFloat,
kDouble, // TODO: remove kDouble

// OSL library functions or renderer services encode additional native types
kInt64,
kDouble,
kUInt32,
kUInt64,
kPointer,
kTypeDesc,
kCount
};

// Decode will use each EncodedType in the array to interpret the contents of the arg_values
// parameter alongwith a fmtlib specifier identified by the ustring that the format_hash represents.
// Contents of decoded_str are written over (not appended).
// Returns # of bytes read from arg_values
int
decode_message(uint64_t format_hash, int32_t arg_count,
const EncodedType* arg_types, const uint8_t* arg_values,
std::string& decoded_str);

namespace pvt {

constexpr inline uint32_t
size_of_encoded_type(EncodedType et)
{
constexpr uint32_t SizeByEncodedType[] = {
sizeof(OSL::ustringhash), sizeof(int32_t), sizeof(float),
sizeof(int64_t), sizeof(double), sizeof(uint32_t),
sizeof(uint64_t), sizeof(void*), sizeof(OSL::TypeDesc),
};
static_assert(sizeof(SizeByEncodedType) / sizeof(SizeByEncodedType[0])
== size_t(EncodedType::kCount),
"Keep array contents lined up with enum");
return SizeByEncodedType[static_cast<int>(et)];
}

template<typename T>
struct TypeEncoder; //Undefined on purpose; all types must have a specialization
struct TypeEncoder; //Undefined on purpose; all types must have a specialization

template<> struct TypeEncoder<OSL::ustringhash> {
using DataType = ustringhash_pod;
static constexpr EncodedType value = EncodedType::kUstringHash;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const OSL::ustringhash& val) { return val.hash(); }
};

namespace {
// MAYBE DONT NEED THIS
// ensure each compilation unit has its own lookup table by putting it in unnamed namespace
constexpr uint32_t SizeByEncodedType[] = {
sizeof(OSL::ustringhash),
sizeof(int32_t),
sizeof(float),
sizeof(double),
};
template<> struct TypeEncoder<OSL::ustring> {
using DataType = ustringhash_pod;
static constexpr EncodedType value = EncodedType::kUstringHash;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const OSL::ustring& val) { return val.hash(); }
};

template<>
struct TypeEncoder<OSL::ustringhash>
{
using DataType = OSL::ustringhash;
static constexpr EncodedType value = EncodedType::kUstringHash;
template<> struct TypeEncoder<const char*> {
using DataType = ustringhash_pod;
static constexpr EncodedType value = EncodedType::kUstringHash;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const char* val) { return ustring(val).hash(); }
};

template<>
struct TypeEncoder<OSL::ustring>
{
using DataType = OSL::ustringhash;
static constexpr EncodedType value = EncodedType::kUstringHash;
template<> struct TypeEncoder<std::string> {
using DataType = ustringhash_pod;
static constexpr EncodedType value = EncodedType::kUstringHash;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const std::string &val) { return ustring(val).hash(); }
};

template<>
struct TypeEncoder<int32_t>
{
using DataType = int32_t;
static constexpr EncodedType value = EncodedType::kInt32;
template<> struct TypeEncoder<int32_t> {
using DataType = int32_t;
static constexpr EncodedType value = EncodedType::kInt32;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const int32_t val) { return val; }
};

template<>
struct TypeEncoder<float>
{
using DataType = float;
static constexpr EncodedType value = EncodedType::kFloat;
template<> struct TypeEncoder<float> {
using DataType = float;
static constexpr EncodedType value = EncodedType::kFloat;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const float val) { return val; }
};

template<>
struct TypeEncoder<double>
{
using DataType = double;
static constexpr EncodedType value = EncodedType::kDouble;

template<> struct TypeEncoder<int64_t> {
using DataType = int64_t;
static constexpr EncodedType value = EncodedType::kInt64;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const int64_t val) { return val; }
};

template<> struct TypeEncoder<double> {
using DataType = double;
static constexpr EncodedType value = EncodedType::kDouble;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const double val) { return val; }
};

template<> struct TypeEncoder<uint64_t> {
using DataType = uint64_t;
static constexpr EncodedType value = EncodedType::kUInt64;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const uint64_t val) { return val; }
};

template<> struct TypeEncoder<uint32_t> {
using DataType = uint32_t;
static constexpr EncodedType value = EncodedType::kUInt32;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const uint32_t val) { return val; }
};

template<typename T> struct TypeEncoder<T*> {
// fmtlib only supports const void *, all other pointers must be
// converted with a cast or helper like fmt::ptr(p)
static_assert(
(std::is_same<void*, T*>::value || std::is_same<const void*, T*>::value),
"formatting of non-void pointers is disallowed, wrap with fmt::ptr");

using DataType = const T*;
static constexpr EncodedType value = EncodedType::kPointer;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const T* val) { return val; }
};

template<> struct TypeEncoder<OSL::TypeDesc> {
// To avoid warnings about non-pod types when we PackArgs
// we will encode into builtin type
using DataType = uint64_t;
static_assert(sizeof(OSL::TypeDesc) == sizeof(DataType), "unexpected");
static constexpr EncodedType value = EncodedType::kTypeDesc;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const OSL::TypeDesc& val)
{
return OSL::bitcast<DataType>(val);
}
};

// Promote thinner types
template<> struct TypeEncoder<int16_t> {
using DataType = int32_t;
static constexpr EncodedType value = EncodedType::kInt32;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const int16_t val) { return DataType(val); }
};

template<> struct TypeEncoder<int8_t> {
using DataType = int32_t;
static constexpr EncodedType value = EncodedType::kInt32;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const int8_t val) { return DataType(val); }
};

template<> struct TypeEncoder<uint16_t> {
using DataType = uint32_t;
static constexpr EncodedType value = EncodedType::kUInt32;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const uint16_t val) { return DataType(val); }
};

template<> struct TypeEncoder<uint8_t> {
using DataType = uint32_t;
static constexpr EncodedType value = EncodedType::kUInt32;
static_assert(size_of_encoded_type(value) == sizeof(DataType),
"unexpected");
static DataType Encode(const uint8_t val) { return DataType(val); }
};
} // namespace pvt


OSL_NAMESPACE_EXIT
Loading

0 comments on commit 6441f10

Please sign in to comment.