What are the best practices for implementing a custom ElementRenderer?

[EsatMan]

I'm attempting to implement a new text block but encountered internal fields and methods after copying the code from DefaultTextBlockRenderer (e.g., SpriteFont.InternalUIDrawCommand, UIElement.LayoutingContext, TextBlock.WorldMatrixInternal, and UIBatch.DrawString). Since Stride doesn't seem to have functionality similar to Unity's assembly definition assets, my options appear to be either using reflection or modifying Stride's source code and building it myself. The latter seems like overkill, and the former isn't an ideal solution. Are there any recommended approaches for implementing a custom ElementRenderer?

[EsatMan]

Hello everyone,

I've just started to explore the process of customizing a new UIElement, which seems like a promising entry point. Although I haven't fully grasped the entire process yet, I'd like to share my current progress in case someone else encounters the same issues I have.

Firstly, the rendering of a UIElement object is handled by UIRenderFeature and ElementRenderer. To render our own UI objects, we need to implement a RenderFeature that inherits from UIRenderFeature. Override the InitializeCore method and, after calling base.InitializeCore();, use RegisterRendererFactory(typeof(MyUIElement), new MyUIElementRenderersFactory(RenderSystem.Services)); to register our own renderer and the UI objects it cares about.

Next, in the editor's GraphicsCompositor, remove the default UIRenderFeature and add the one we've implemented ourselves. Since it inherits from the default UIRenderFeature, there's no need to worry about breaking other functionalities.

Then, we need to implement a Renderer that inherits from ElementRenderer, along with its factory class. The factory class implements the IElementRendererFactory interface. In reality, it doesn't require complex logic; it just maintains an instance of the Renderer and returns it according to the interface's methods. The Renderer is concerned with which UI object class is registered in the initialization method of the UIRenderFeature.

Finally, under this ElementRenderer, there are overridable methods for drawing, which mainly rely on the methods opened by UIBatch for specific drawing operations. I haven't figured this part out yet, but by using reflection to call the official internal implementation, I got the process running. As for how to draw specifically, it seems to be a complex issue that I will continue to study.

[EsatMan]

Hello everyone, I have roughly figured out the entire process. Since my implementation does not take into account the complete business logic and edge cases, I will not be directly sharing a repository (as it would not be directly usable even if shared), but I will explain the process in the following text. I will keep the discussion open in case someone wishes to further communicate.

Due to the fact that the official font implementation of Stride does not meet my needs, and most of its scheduling processes are isolated at the internal level, I need to implement a font management system myself, which can be achieved through a library called FreeType2 (there is a wrapper library for .Net called FreeTypeSharp). Through this library, we can obtain the necessary font information and get the bitmap for each character. What we specifically need to do is to implement a font management class that can track and maintain the layout information and bitmap for each character (by drawing the bitmap on a large texture as an atlas), and other modules can call to get the layout information and texture information of a character through key-value pairs (including char, width, height, style, etc., which can be decided by you). Finally, the text is rendered through the renderer.

In fact, there are still parts of the process that require calling the internal parts, but they can all be achieved through UnsafeAccessor, which means that accessing protected members will not affect performance. If the developers of Stride are reading this article, I sincerely suggest relaxing the protection level of some graphics-related APIs. A large number of internal members and nested classes make it very difficult to insert your own implementation, which is quite restrictive.

public static class StrideUIInternalWrapper
{
    public static void DrawCharacter(this UIBatch batch, Texture texture, in Matrix worldViewProjectionMatrix, in RectangleF sourceRectangle, in Color color, int depthBias, SwizzleMode swizzle)
    {
        UIBatch_Instance_DrawCharacter(batch, texture, worldViewProjectionMatrix, sourceRectangle, color, depthBias, swizzle);
    }

    public static Matrix GetViewProjectionMatrix(this UIBatch batch)
    {
        return UIBatch_Instance_viewProjectionMatrix(batch);
    }

    public static LayoutingContext GetLayoutingContext(this UIElement element)
    {
        return UIElement_Instance_get_LayoutingContext(element);
    }

    public static Matrix GetWorldMatrixInternal(this UIElement element)
    {
        return UIElement_Instance_WorldMatrixInternal(element);
    }

    [UnsafeAccessor(UnsafeAccessorKind.Method, Name = "get_LayoutingContext")]
    private extern static LayoutingContext UIElement_Instance_get_LayoutingContext(UIElement element);

    [UnsafeAccessor(UnsafeAccessorKind.Field, Name = "WorldMatrixInternal")]
    private extern static ref Matrix UIElement_Instance_WorldMatrixInternal(UIElement element);

    [UnsafeAccessor(UnsafeAccessorKind.Field, Name = "viewProjectionMatrix")]
    private extern static ref Matrix UIBatch_Instance_viewProjectionMatrix(UIBatch batch);

    [UnsafeAccessor(UnsafeAccessorKind.Method, Name = "DrawCharacter")]
    private extern static void UIBatch_Instance_DrawCharacter(UIBatch batch, Texture texture, in Matrix worldViewProjectionMatrix, in RectangleF sourceRectangle, in Color color, int depthBias, SwizzleMode swizzle);
}

Below is the specific implementation explanation.

Since the FreeType2 API returns errors through error codes, we can wrap it into an exception-throwing method through extension methods.

public static void err(this FT_Error error)
{
    if (error != FT_Error.FT_Err_Ok)
    {
        throw new Exception($"FreeType Error:{error.ToString()}");
    }
}

We use the FreeType2 API to get several faces from the font file, each face contains all the characters of a specific style in a family. This is exactly where the official implementation of Stride is not good, their implementation assumes that the font only has three styles: regular, bold, and italic, and cannot recognize style identifiers like heavy, light, etc. We can read the string of the style name through the FreeType2 API and classify the faces based on this.

public unsafe class FontGroup : IDisposable
{
    // other staff ...
    private Dictionary<string,List<IntPtr>> FacesPtrByStyleName;

    void IDisposable.Dispose()
    {
        FacesPtrByStyleName.ForEachInEveryList(ptr =>
        {
            var facePtr = (FT_FaceRec_*)ptr;

            FT.FT_Done_Face(facePtr).err();
        });
    }

    public void LoadFontFromFile(string path)
    {
        if (loadedFontPath.Contains(path)) return;

        FT_FaceRec_* FirstFacePtr;
        var pathGlobalPtr = Marshal.StringToHGlobalAnsi(path);
        var pathGlobalPtrInByte = (byte*)pathGlobalPtr;
        FT.FT_New_Face(FontEnvironment.FontLibraryPtr, pathGlobalPtrInByte, 0, &FirstFacePtr).err();
        var totalFaceCount = FirstFacePtr->num_faces;
        for (nint i = 1; i < totalFaceCount; i++)
        {
            FT_FaceRec_* CurrentFacePtr;
            FT.FT_New_Face(FontEnvironment.FontLibraryPtr, pathGlobalPtrInByte, i, &CurrentFacePtr).err();
            RegisterFace(CurrentFacePtr);
        }

        loadedFontPath.Add(path);
        Marshal.FreeHGlobal(pathGlobalPtr);
        unsupportedChars.Clear();

        void RegisterFace(FT_FaceRec_* _facePtr)
        {
            var _faceIntPtr = (IntPtr)_facePtr;
            var styleName = UnsafeUtl.ConvertBytePointerToString(_facePtr->style_name);
            FacesPtrByStyleName.AddToListForce(styleName, _faceIntPtr);
        }
        // other staff ...
    }

}

public unsafe static class UnsafeUtl
{
    public static int GetStringLength(byte* buffer)
    {
        int length = 0;
        while (true)
        {
            byte currentByte = buffer[length];
            if (currentByte == 0) break;
            length++;
        }
        return length;
    }

    public static string ConvertBytePointerToString(byte* buffer)
    {
        int length = GetStringLength(buffer);
        byte[] byteArray = new byte[length];

        for (int i = 0; i < length; i++)
        {
            byteArray[i] = buffer[i];
        }

        return string.Intern(Encoding.UTF8.GetString(byteArray));
    }
}

Then we need to define the query key and layout information for a single character. This part can be designed according to your business needs, and below is my implementation.

public readonly struct CharKey
{
    public string GroupName { get; init; }

    public string StyleName { get; init; }

    public nint Width { get; init; }

    public nint Height { get; init; }
    public char Charactor { get; init; }

    public override bool Equals(object obj)
    {
        if (obj == null || obj is not CharKey) return false;
        return Equals(this, (CharKey)obj);
    }

    public static bool Equals(CharKey left, CharKey right)
    {
        return left.GroupName == right.GroupName && left.StyleName == right.StyleName && left.Width == right.Width && left.Height == right.Height && left.Charactor == right.Charactor;
    }

    public override int GetHashCode()
    {
        return HashCode.Combine(GroupName, StyleName, Width, Height, Charactor);
    }
}

public struct CharInfo
{
    public int Character;

    public FT_Size_Metrics_ SizeMetrics;
    public FT_Glyph_Metrics_ GlyphMetrics;
    public FT_BBox_ BBox;
    public ushort UnitsPerEM;

    public nint LineSpace
    {
        get
        {
            return SizeMetrics.ascender - SizeMetrics.descender + SizeMetrics.height;
        }
    }

    public nint InternalLeading
    {
        get
        {
            return SizeMetrics.ascender - SizeMetrics.descender - UnitsPerEM;
        }
    }

    public nint ExternalLeading { get => SizeMetrics.height; }

    public int AtlasIndex { get; set; }
    public Rectangle Rect { get; set; }

    public unsafe CharInfo(char character, FT_GlyphSlotRec_* ptr)
    {
        Character = character;
        var face = ptr->face;
        var glyph = ptr;

        BBox = face->bbox;
        UnitsPerEM = face->units_per_EM;
        SizeMetrics = face->size->metrics;
        GlyphMetrics = glyph->metrics;
    }
}

Next, we can implement the method to read the character glyph information from the font file. You can notice that I called the method to generate a bitmap at this time, which will be discussed next.

public unsafe class FontGroup : IDisposable
{
    // other staff ...
    public bool GetFontData(char character, nint width, nint height, string styleName, CommandList commandList, out CharInfo result)
    {
        if (unsupportedChars.Contains(character))
        {
            result = default;
            return false;
        }

        var key = new CharKey() { Charactor = character, GroupName = Name, StyleName = styleName, Width = width, Height = height };

        if(loadedCharFontDatas.TryGetValue(key, out var data))
        {
            result = data;
            return true;
        }
        else
        {
            var faceList = FacesPtrByStyleName[styleName];
            foreach(var ptr in faceList)
            {
                var face = (FT_FaceRec_*)ptr;
                FT.FT_Set_Char_Size(face, width * 64, height * 64, 72, 72).err();
                var glyph_index = FT.FT_Get_Char_Index(face, character);
                if (glyph_index == 0) { continue; } // go next
                FT.FT_Load_Glyph(face, glyph_index, FT_LOAD.FT_LOAD_RENDER).err();
                var glyph = face->glyph;
                var info = new CharInfo(character,glyph);
                FontEnvironment.UploadCharacterBitmap(commandList, glyph, ref info);
                result = info;
                loadedCharFontDatas.Add(key, info);
                
                return true;
            }

        }


        unsupportedChars.Add(character);
        result = default;
        return false;
    }
}

For texture management, what we need to do is track a texture atlas, draw the bitmap from FreeType2 to the texture through Stride's graphics API, and save the reference of the texture (the index of the texture list is also fine, as long as you can get this texture through the character information later) and the rectangle of this character on the texture into the character information. The algorithm for inserting graphics can directly use the GuillotinePacker implemented by the official Stride. What I demonstrate here is an implementation written just to run through the process, without considering optimization and edge cases. If you want to implement a complete font management, I suggest reading the source code.

public unsafe class FontTextureManager
{
    private List<Texture> fontAtlas;
    private GuillotinePacker packer;

    public FontTextureManager(GraphicsDevice device)
    {
        fontAtlas = new List<Texture>();
        packer = new GuillotinePacker();
        NewTexture(device);
    }

    public void UploadCharacterBitmap(CommandList commandList, FT_GlyphSlotRec_* glyph, ref CharInfo charInfo)
    {
        var bitmap = glyph->bitmap;
        var width = (int)bitmap.width;
        var height = (int)bitmap.rows;
        var rect = Rectangle.Empty;
        if (!packer.Insert(width, height, ref rect)) // 插入不成功说明当前纹理满了
        {
            NewTexture(commandList.GraphicsDevice);
            if (!packer.Insert(width, height, ref rect))
            {
                throw new InvalidOperationException("The rendered character is too big for the cache texture");
            }
        }
        if (bitmap.width != 0 && bitmap.rows != 0)
        {
            var dataBox = new DataBox((IntPtr)bitmap.buffer, bitmap.pitch, bitmap.pitch * (int)bitmap.rows);
            var region = new ResourceRegion(rect.Left, rect.Top, 0, rect.Right, rect.Bottom, 1);
            UpdateSubresource(commandList, fontAtlas[^1], 0, dataBox, region);
        }
        charInfo.Rect = rect;
        charInfo.AtlasIndex = fontAtlas.Count - 1;
    }

    private void NewTexture(GraphicsDevice device)
    {
        var newTexture = Texture.New2D(device, 2048, 2048, PixelFormat.R8_UNorm, TextureFlags.ShaderResource, 1, GraphicsResourceUsage.Default, TextureOptions.None);
        fontAtlas.Add(newTexture);
        packer.Clear(newTexture.ViewWidth, newTexture.ViewHeight);
    }

    internal void Debug_PrintTexture(CommandList commandList)
    {
        var image = fontAtlas[0].GetDataAsImage(commandList);
        using(FileStream fs = new FileStream(@"C:\Users\WINDOWS\Desktop\outputImage.png", FileMode.OpenOrCreate, FileAccess.ReadWrite))
        {
            image.Save(fs, ImageFileType.Png);
        }
    }

    public Texture GetTextureByIndex(int index)
    {
        return fontAtlas[index];
    }

    [UnsafeAccessor(UnsafeAccessorKind.Method, Name = "UpdateSubresource")]
    extern static void UpdateSubresource(CommandList commandList, GraphicsResource resource, int subResourceIndex, DataBox databox, ResourceRegion region);
}

After completing the above steps, we can come to the implementation of the renderer, and here only the override of the RenderColor method is demonstrated. The implementation of the renderer, in essence, is two parts of work. One part is to convert the world coordinate matrix of the UI element to the rendering view matrix, which can be copied from the official implementation; the other part is to complete the drawing at the original point after the conversion. The official implementation is relatively complex due to optimization needs and actual business logic, and here we only demonstrate the principle, so a very simple drawing logic can be implemented.

public override void RenderColor(UIElement element, UIRenderingContext context)
{
    // part one
    base.RenderColor(element, context);

    var label = (TestLabelElement)element;

    if (label.TextToDisplay == null)
        return;


    var drawCommand = new DrawCommands
    {
        DepthBias = context.DepthBias,
        RealVirtualResolutionRatio = element.GetLayoutingContext().RealVirtualResolutionRatio,
        RequestedFontSize = label.ActualTextSize,
        Batch = Batch,
        SnapText = context.ShouldSnapText && !label.DoNotSnapText,
        WorldMatrix = label.GetWorldMatrixInternal(),
        TextAlignment = label.TextAlignment,
        TextBoxSize = new Vector2(label.ActualWidth, label.ActualHeight),
        Text = label.TextToDisplay,
    };

    // shift the string position so that it is written from the left/top corner of the element
    var leftTopCornerOffset = drawCommand.TextBoxSize / 2;
    var worldMatrix = drawCommand.WorldMatrix;
    worldMatrix.M41 -= worldMatrix.M11 * leftTopCornerOffset.X + worldMatrix.M21 * leftTopCornerOffset.Y;
    worldMatrix.M42 -= worldMatrix.M12 * leftTopCornerOffset.X + worldMatrix.M22 * leftTopCornerOffset.Y;
    worldMatrix.M43 -= worldMatrix.M13 * leftTopCornerOffset.X + worldMatrix.M23 * leftTopCornerOffset.Y;

    // transform the world matrix into the world view project matrix
    var viewProjectionMatrix = Batch.GetViewProjectionMatrix();
    Matrix.Multiply(ref worldMatrix, ref viewProjectionMatrix, out drawCommand.WorldMatrix);
   
    // part two ...
}

private struct DrawCommands
{
    public int DepthBias;
    public Vector2 RealVirtualResolutionRatio;
    public float RequestedFontSize;
    public UIBatch Batch;
    public bool SnapText;
    public Matrix WorldMatrix;
    public TextAlignment TextAlignment;
    public Vector2 TextBoxSize;
    public string Text;
}

The official implementation above has already pointed the current matrix to the upper left corner of the text box, and we simply implement a logic of laying out the text from the upper left corner to the right. Regarding the typesetting knowledge, I suggest directly reading the official documentation of FreeType2, but in simple terms, what the listed code does is to offset the original point of the upper left corner to the baseline based on which the character layout is based. And once the baseline is obtained, we can adjust the specific position of the bitmap through the layout information of the character. All values are divided by 64 because of the data conversion reason of FreeType2.

public override void RenderColor(UIElement element, UIRenderingContext context)
{
    // part one ...

    // part two
    {
        Vector2 pen = Vector2.Zero;
        for (int i = 0; i < drawCommand.Text.Length; i++)
        {
            var c = drawCommand.Text[i];
            var info = FontEnvironment.GetOrCreateFontInfo("Main", "Regular", c, 36, 36, context.GraphicsContext.CommandList);
            Batch.DrawCharacter(FontEnvironment.GetTextureByIndex(info.AtlasIndex), MatrixShiftByPosAndSize(drawCommand.WorldMatrix, new Vector2(pen.X + (info.GlyphMetrics.horiBearingX / 64), (info.SizeMetrics.ascender / 64)- (info.GlyphMetrics.horiBearingY / 64)), new Vector2(info.GlyphMetrics.width/64,info.GlyphMetrics.height/64)), info.Rect, Color.White, drawCommand.DepthBias, SwizzleMode.RRRR);
            pen = new Vector2(pen.X + info.GlyphMetrics.horiAdvance / 64, pen.Y);
        }
    }
    
}

private Matrix MatrixShiftByPosAndSize(Matrix matrix, Vector2 pos, Vector2 size)
{
    matrix.M41 += matrix.M11 * pos.X + matrix.M21 * pos.Y;
    matrix.M42 += matrix.M12 * pos.X + matrix.M22 * pos.Y;
    matrix.M43 += matrix.M13 * pos.X + matrix.M23 * pos.Y;
    matrix.M44 += matrix.M14 * pos.X + matrix.M24 * pos.Y;

    matrix.M11 *= size.X;
    matrix.M12 *= size.X;
    matrix.M13 *= size.X;
    matrix.M14 *= size.X;
    matrix.M21 *= size.Y;
    matrix.M22 *= size.Y;
    matrix.M23 *= size.Y;
    matrix.M24 *= size.Y;

    return matrix;
}

With this, the most basic independent font rendering has been implemented.

[VaclavElias]

This looks interesting. I hope we will get some feedback from the core devs.