Skip to content

Commit

Permalink
Oss release 27.0.128 created 2022-12-15-04-15
Browse files Browse the repository at this point in the history
see CHANGELOG.txt for details

Original commit sha: b1b416d169d06d528133f8b3d33e5bc478d981f0

Co-authored-by: Carsten Rohn <[email protected]>
Co-authored-by: Daniel Haas <[email protected]>
Co-authored-by: Tobias Hammer <[email protected]>
Co-authored-by: Bernhard Kisslinger <[email protected]>
Co-authored-by: Martin Veith <[email protected]>
Co-authored-by: Violin Yanev <[email protected]>
Co-authored-by: Jonathan Conrad <[email protected]>
Co-authored-by: Mohamed Sharaf-El-Deen <[email protected]>
Co-authored-by: Markus Keppler <[email protected]>
  • Loading branch information
10 people authored and violinyanev committed Dec 15, 2022
1 parent b2f32c3 commit a42c7cf
Show file tree
Hide file tree
Showing 35 changed files with 1,047 additions and 265 deletions.
15 changes: 15 additions & 0 deletions CHANGELOG.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
27.0.128
-------------------
General changes
------------------------------------------------------------------------
- set file buffer size to 8kB for Integrity OS
- ramses-scene-viewer:
- added filter for node visibility
- filter resources by hash name
- fixed resource hash name formatting (use same format as in the logs)
- added option to sort by uncompressed size
- export shader sources to file or clipboard
- ramses-packet-player:
- added details for scene actions
- Added ramses::DisplayConfig::setScenePriority()

27.0.127
-------------------
Bugfixes
Expand Down
4 changes: 1 addition & 3 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ CMAKE_MINIMUM_REQUIRED(VERSION 3.10)

SET(RAMSES_VERSION_MAJOR 27)
SET(RAMSES_VERSION_MINOR 0)
SET(RAMSES_VERSION_PATCH 127)
SET(RAMSES_VERSION_PATCH 128)
SET(RAMSES_VERSION_POSTFIX "")

CMAKE_POLICY(SET CMP0048 NEW)
Expand Down Expand Up @@ -75,8 +75,6 @@ OPTION(ramses-sdk_USE_LINUX_DEV_PTP "Enable support for synchronized ptp time on
OPTION(ramses-sdk_BUILD_WITH_LTO "Build all targets with link time optimization enabled (not supported on all platforms)" OFF)
option(ramses-sdk_ENABLE_COVERAGE "Build withcode coverage enabled (gcc: gcov, clang: source based coverage)" OFF)

set(ramses_CPACK_GENERATOR "TGZ" CACHE STRING "Passed to CPack to determine the type of package to be created. Default is TGZ.")

SET(ramses-sdk_CONSOLE_LOGLEVEL "" CACHE STRING "Console log level.")
SET_PROPERTY(CACHE ramses-sdk_CONSOLE_LOGLEVEL PROPERTY STRINGS "off" "fatal" "error" "warn" "info" "debug" "trace" "")

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@

ext.globalRepositoryDefinition = {
google()
jcenter()
mavenCentral()
}
4 changes: 2 additions & 2 deletions doc/main.dox
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ To understand the different Ramses concepts and general architecture, see @ref G
<!-- TODO Violin we should improve our presentation of example snippets - that's 80% of what people see from the docs -->
If you prefer to learn by example, have a look at the @ref CodeExamples "list of examples".

# Debugging and tools
# Profiling, debugging and tools

For details on specific tools, debugging options and log analysis, visit the @ref ToolsAndDebugging "tools and debugging section".
For details on specific tools, debugging options and log analysis, visit the @ref ToolsAndDebugging "profiling, debugging and tools section".

# Developer section

Expand Down
24 changes: 21 additions & 3 deletions doc/tools/00_MainPage.dox
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,28 @@
// -------------------------------------------------------------------------

/**
@page ToolsAndDebugging Tools and debugging
@page ToolsAndDebugging Profiling, debugging, and tools

In addition to the core libraries, Ramses offers binary utilities for resource and shader
baking, as well as runtime logging based on DLT.
Ramses provides logs and various tools to help find issues in your application.
In the sections below you will find hints where to find those and how to use them.

# Debugging your scene

The easiest way to find problems in your content/scene is to use the validation layer of the
Ramses client. The validation is an optional layer of Ramses which shows hints of possible
undefined behavior, performance problems, or missing data which can cause your content to render
not as expected. This is similar to how Vulkan works - you can enable the validation to find issues,
and disable it for production to save resources.

Each ramses::StatusObject derived class has a validate() method. Call it to find issues in that object.
You can do this for an entire scene by calling validate() on a ramses::Scene, ramses::RamsesRenderer or any
other object. Validation works recursively, i.e. a RamsesClient will validate all its scenes, a ramses::Scene will
validate all its scene objects, and so on.

# Profiling and performance

For a detailed overview of the various ways to profile your content and optimize performance, refer to the @ref Profiling
section.

# Resource and shader tools

Expand Down
177 changes: 177 additions & 0 deletions doc/tools/profiling/10_Profiling.dox
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
// -------------------------------------------------------------------------
// Copyright (C) 2022 BMW AG
// -------------------------------------------------------------------------
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
// -------------------------------------------------------------------------

/**
@page Profiling Performance and profiling

# Overview

There are multiple ways to find bottlenecks in your assets, your code using Ramses, or Ramses itself.

# Inspecting the contents of your scene and optimizing it

The first source of performance problems in a 3D application is usually the content. Having large textures or geometry,
heavy shaders, or simply rendering suboptimally - those are all things that can be easily solved in the content/scene directly
by a skilled technical artist.

Ramses provides a tool to perform such analysis, called the ramses scene viewer (see its documentation in @ref SceneViewer).

# Looking at the Ramses periodic performance logs

A great source of information about what's going on in the Ramses threads (specifically when having network involved) are
the ramses periodic logs (RPER). See below how
to read and use these logs.

## General periodic logs

Any Ramses application (regardless if it has a client, a renderer, or both) has a periodic log whichs looks like this:

20210212-16:33:50.770 | Info | RPER | Version: 27.0.6 Hash:cca2a0b60f Commit:28143 Type:Release Env:(unknown) SyncT:1613144030770ms (dtSteady:2023 - dtSync:2023 -> 0) PUp:31679 RUp:31575 RInit:1 RParallel:1
20210212-16:33:50.775 | Info | RPER | Connected Participant(s): 00000000-0000-0000-0000-000000000003, 00000000-0000-0000-0000-000000000004, 00000000-0000-0000-0000-000000000002, 00000000-0000-0000-0000-000000000005
20210212-16:33:50.775 | Info | RPER | Dcsm(7AB1-B4A4D03AA15F) C LP:false LC:false CP[] C[]

First line:
- Ramses version, commithash, commit no, build type
- RInit: 1 instance of ramses initialized (framework)
- RParallel: 1 instance of ramses running now

Second line: Participants connected to this instance of ramses

Third line: Dcsm related stats

## Client periodic logs

A Ramses client typically reports logs like this:

20210212-16:33:50.778 | Info | RPER | Client: 1 scene(s): 123 Subscribed
20210212-16:33:50.778 | Info | RPER | msgIn (0) msgO (0) res+ (27/32/29) res- (0) resNr (218/245/231) resF (0) resFS (0)
20210212-16:33:50.782 | Info | RPER | scene: 123 flush (60/61/60) obj+ (50/75/62) obj- (0) objNr (2670/2720/2695) actG (1560/1671/1615) actGS (30988/33020/32004) actO (1560/1671/1615) actSkp (0) ar# (173/197/185) aras (53460/54325/53892) arms (408272) er# (11) eras (1241) erms (2388) tr# (19/20/19) tras (1099739/1114657/1107198) trms (2097144)


The client-side periodic logger collects the stats values (flush, obj+, etc) every second. However the time interval
for logging output is configurable and by default set to 2 seconds.

The collected values are printed in two fashions depending on the stat value:
- `suX` lists the largest collected values of the last logging period (value1, value2, ..., valueN)
- For all other stats there are typically 3 values printed (min/max/avg):
-* min: the smallest value that was collected since the last log output
-* max: the largest value that was collected since the last log output
-* avg: the average of all collected values since the last log output: ((value1 + value2 + ..valueN) / n)

If the smallest and the largest value are equal (min == max), only 1 value will be printed: (value)

Examples:

- Time interval is set to 4 seconds. The logger collected 4 values: 20, 25, 21, 55; Logs will print: (20/55/30)
- Time interval is set to 2 seconds, the logger collected 2 values: 42, 42; Logs will print (42)

The logs explained line by line:

First line:
- Client participant id
- List of scenes owned by the client and their status


Second line (General client performance stats):

- msgIn: Number of messages received
- msgO: Number of message sent
- res+: Client Resources created
- res-: Client Resources destroyed
- resNr: Client Resource Count
- resF: Number of Client Resources loaded from File
- resFS: Size of Client Resources loaded from file

Third line and over (Scene related stats):

- scene: scene id
- flush: Number of flushes triggered per second
- obj+: Number of scene objects created (ramses::SceneObject) per second
- obj-: Number of scene objects destroyed (ramses::SceneObject) per second
- objNr: Number of currently existing scene objects (ramses::SceneObject)
- actG: Number of scene actions generated per second
- actGS: Size of scene actions generated per second
- actO: Number of scene actions sent to renderer(s) per second (will be counted for each scene subscriber)
- actSkp: Number of skipped scene actions per second (usually an optimization to avoid empty updates)
- suG: Scene updates generated per second. Number of scene update packages generated for network send (might be more than # of sceneupdates)
- suGS: Scene update generated size per second. Accumulated size of scene update packages generated for network
send
- suX: The size of the largest scene update package within 1 second
- ar#: Number of currently used array resources
- aras: Average size of a single array resource ((totalSize of currently used array resources) / ar#)
- arms: Largest currently used array resource
- er#: Number of currently used Effects
- eras: Average size of a single effect resource ((totalSize of currently used effects) / er#)
- erms: Largest currently used effect resource
- tr#: Number of currently used texture resources
- tras: Average size of a single texture resource ((totalSize of currently used textures) / tr#)
- trms: Largest currently used texture resource

Caveats:

Some stats describe changes/deltas to the scene: flush, obj+, obj-, act*, su* - others describe a snapshot of the current scene state: objNr, ar*, er* tr*
Resource stats (ar*,er*, tr*) are only logged if there was a flush during the logging interval

## Renderer periodic logs

A Ramses application which also contains a renderer component has periodic logs which look like this:

20210212-16:33:48.791 | Info | RPER | Renderer: 1 scene(s): 123 Rendered
Avg framerate: 52.63158 FPS [minFrameTime 381us, maxFrameTime 59404us], drawcallsPerFrame 35, numFrames 107, resUploaded 128 (13207530 B)
FB0: 92
Scene 123: rendered 92, framesFArrived 92, framesFApplied 92, framesFBlocked 0, maxFramesWithNoFApplied 3, maxFramesFBlocked 0, FArrived 1397, FApplied 1397, actions/F (10/215/20.180386), dt/F (1/21295/34.343594), RC+/F (0/28/0.09234073), RC-/F (0/28/0.09520401), RS/F (0/0/0)
Time budgets: sceneResourceUpload 315360000000000us resourceUpload 315360000000000us obRender 315360000000000us
Longest frame(us)[avg]:59190 RendererCommands:749 [387] UpdateClientResources:32 [372] ApplySceneActions:7841 [2781] UpdateSceneResources:159 [114] UpdateEmbeddedCompositingResources:14 [8] UpdateStreamTextures:10 [5] UpdateScenesToBeMapped:13 [8] UpdateResourceCache:1039 [3234] UpdateAnimations:8 [5] UpdateTransformations:290 [138] UpdateDataLinks:44 [26] HandleDisplayEvents:34 [316] DrawScenes:1908 [1393] SwapBuffersNotifyClients:47049 [6854] MaxFramerateSleep:0 [2870]

Read these logs like this:

First line:
- Renderer participant id
- List of scenes known to renderer and status

Second line (General renderer performance stats):
- Number of frames per second
- minimal and maximal frame time within time period
- drawcalls per frame
- Number of frames rendered in time period
- resources uploaded in time period

Third line and over(scene related stats):
- Scene id
- rendered: Number of frames rendered
- FrameFArrived: Number of frames where flushes arrived
- FramesFApplied: Number of frames where flushes applied
- FrameFBlocked: Number of frames where applying a flush was blocked
- maxFramesWithNoFApplied: How many consecutive frame there was no flush applied
- maxFramesFBlocked: How many consecutive frames flushes were blocked from applying
- FArrived: Number of flushes arrived
- FApplied: Number of flushes applied
- actions/F: number of scene actions per flush
- dt/F: flush latency
- RC+/F RC-/F: Number of client resources added/removed per flush
- RS/F: Number of scene resource actions per flush
- RSUploaded: Size of scene resources uploaded

Time budgets: Information about how much time per frame an action may take as a maximum, set by application

Advanced stats:
- Longest Frame: How long did the longest frame take to render alltogether
- Rest: Advanced stats for profiling renderer which need internal understanding of the Ramses renderer.



# Using specialized tools

If the above methods didn't yield the results you expected, or you still think your application can perform better,
you can also use some of the professional tools for profiling:

- NVidia NSight - a great tool by NVidia which can analyze any OpenGL-based application. Limitation: requires an NVidia graphics card.
- Standard profilers like the MSVC Profiler or gprof - great for finding CPU bottlenecks or memory attrition issues.
- Android Profiler - a great all-round tool for finding issues, also in native libs such as Ramses. Works only on Android.

*/
2 changes: 1 addition & 1 deletion external/abseil
Submodule abseil updated 1339 files
31 changes: 8 additions & 23 deletions external/acme2/internal/create_package.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,7 @@
#
############################################################################

set(CPACK_GENERATOR ${ramses_CPACK_GENERATOR})

SET(CPACK_GENERATOR "TGZ")
SET(CPACK_SOURCE_GENERATOR "TGZ")

IF (NOT DEFINED GIT_COMMIT_COUNT OR NOT DEFINED GIT_COMMIT_HASH)
Expand All @@ -27,28 +26,15 @@ ENDIF()
SET(SCM_VERSION "${GIT_COMMIT_COUNT}-${GIT_COMMIT_HASH}")

IF("${CPACK_PACKAGE_NAME}" STREQUAL "")
SET(CPACK_PACKAGE_NAME "${PROJECT_NAME}")
SET(CPACK_PACKAGE_NAME "${PROJECT_NAME}")
ENDIF()

SET(CPACK_PACKAGE_VERSION "${PROJECT_VERSION_STRING}-${SCM_VERSION}")
SET(CPACK_PACKAGE_CONTACT "[email protected]")
SET(CPACK_SOURCE_STRIP_FILES TRUE)
set(CPACK_STRIP_FILES FALSE)
set(CPACK_PACKAGE_VENDOR "ramses")

set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "A distributed 3D rendering framework for embedded systems")
set(CPACK_PACKAGE_DESCRIPTION "A packaged version of ramses. Generated using CPack.")
set(CPACK_PACKAGE_VERSION_MAJOR ${PROJECT_VERSION_MAJOR})
set(CPACK_PACKAGE_VERSION_MINOR ${PROJECT_VERSION_MINOR})
set(CPACK_PACKAGE_VERSION_PATCH ${PROJECT_VERSION_PATCH})
set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE.txt")
set(CPACK_RESOURCE_FILE_README "${CMAKE_CURRENT_SOURCE_DIR}/README.md")
set(CPACK_PACKAGE_ICON "${CMAKE_CURRENT_SOURCE_DIR}/doc/general/images/ramses_logo_with_alpha2.png")

if(ramses_CPACK_GENERATOR STREQUAL "DEB")
# Enables CPack to add proper dependency info to the package, see docs for more info
set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
endif()
#SET(CPACK_RPM_COMPONENT_INSTALL 1)
#SET(CPACK_ARCHIVE_COMPONENT_INSTALL 1)
SET(CPACK_PACKAGE_VERSION "${PROJECT_VERSION_STRING}-${SCM_VERSION}")
SET(CPACK_PACKAGE_CONTACT "${PROJECT_CONTACT}")
SET(CPACK_SOURCE_STRIP_FILES TRUE)
SET(CPACK_STRIP_FILES FALSE)

# Allows providing custom package suffix, use "-<commitsha>" by default
if(RAMSES_CUSTOM_PACKAGE_SUFFIX)
Expand All @@ -60,4 +46,3 @@ endif()
SET(CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}-${PROJECT_VERSION_STRING}-${PACKAGE_SUFFIX}")

INCLUDE(CPack)

Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ namespace ramses_internal
return false;
}
const auto bytesWritten{m_packetWriter.getBytesWritten()};
m_sceneStatistics.statSceneUpdatesGeneratedSize.incCounter(static_cast<decltype(m_sceneStatistics.statSceneUpdatesGeneratedSize)::ValueType>(bytesWritten));
m_sceneStatistics.statSceneUpdatesGeneratedSize.incCounter(static_cast<uint32_t>(bytesWritten));
m_sceneStatistics.statSceneUpdatesGeneratedPackets.incCounter(1);

m_overallSize += bytesWritten;
Expand Down
1 change: 1 addition & 0 deletions framework/Core/Utils/include/Utils/StatisticCollection.h
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include <array>
#include <vector>
#include <atomic>
#include <cstddef>

namespace ramses_internal
{
Expand Down
5 changes: 5 additions & 0 deletions framework/Core/Utils/src/File.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,11 @@ namespace ramses_internal
return false;
}

#if defined(__INTEGRITY)
// optimize fs access (default buffer size is 512)
setvbuf(handle, NULL, _IOFBF, 8192);
#endif

m_handle = handle;
m_isOpen = true;
return true;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@
#define RAMSES_DISPLAYCONFIG_H

#include "RendererAPI/Types.h"
#include "SceneAPI/SceneId.h"
#include "SceneAPI/TextureEnums.h"
#include "SceneAPI/WaylandIviSurfaceId.h"
#include "Math3d/Vector4.h"
#include <unordered_map>

namespace ramses_internal
{
Expand Down Expand Up @@ -105,6 +107,10 @@ namespace ramses_internal
void setSwapInterval(int32_t interval);
int32_t getSwapInterval() const;

void setScenePriority(SceneId sceneId, int32_t priority);
int32_t getScenePriority(SceneId sceneId) const;
const std::unordered_map<SceneId, int32_t>& getScenePriorities() const;

Bool operator==(const DisplayConfig& other) const;
Bool operator!=(const DisplayConfig& other) const;

Expand Down Expand Up @@ -143,6 +149,7 @@ namespace ramses_internal
String m_platformRenderNode;

int32_t m_swapInterval = -1;
std::unordered_map<SceneId, int32_t> m_scenePriorities;
};
}

Expand Down
Loading

0 comments on commit a42c7cf

Please sign in to comment.