ScreenCaptureforHWND sample leaks handles leading to "Out of memory" crash

[ollie-dawes]

Background:

I am working on a C++ screen capture application and have been running into a few issues. One issue is that our applications handle count gradually increases every time we re-create our frame pool for capture. I decided to pull down the ScreenCaptureforHWND sample from this repo to see if I could reproduce the issue and while this sample uses slightly different logic for frame pool creation (Direct3D11CaptureFramePool::Create vs IDirect3D11CaptureFramePoolStatics::Create) I can see the same handle leaking issue (though to a lesser degree).

How to reproduce:

1. Build the ScreenCaptureforHWND sample.
2. Quickly press up and down on the keyboard after focusing the combobox.
3. Observe that the processes handle count increases with time in Task Managers details pane

Now the above steps just show that the sample is leaking handles somewhere but another way to reproduce the problem that will lead to an out of memory error is to replace the following code in SimpleCapture.cpp:

Windows.UI.Composition-Win32-Samples/cpp/ScreenCaptureforHWND/ScreenCaptureforHWND/SimpleCapture.cpp

Lines 51 to 55 in ee50e2e

	m_framePool = Direct3D11CaptureFramePool::Create(
	m_device,
	DirectXPixelFormat::B8G8R8A8UIntNormalized,
	2,
	size);

With a loop:

    for (int i = 0; i < 65000; i++) {
        m_framePool = Direct3D11CaptureFramePool::Create(
            m_device,
            DirectXPixelFormat::B8G8R8A8UIntNormalized,
            2,
            size);
        // This should free all resources associated with m_framePool, but the process handle count
        // will keep incrementing eventually leading to a "out of memory" error.
        m_framePool.Close();
        m_framePool = nullptr;
    }

This will eventually result in a Out of memory error:

GraphicsCapture.dll!00007FFB264E6900: ReturnHr(1) tid(f960) 8007000E Not enough memory resources are available to complete this operation.
    Msg:[std::exception: bad allocation] 
Unhandled exception at 0x00007FFBDE1CCF19 in ScreenCaptureforHWND.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x000000FE01B5D7C8.

and often leads to whatever application was being captured crashing (possibly unrelated and just a result of the memory issue).

This makes me think there is a bug in the Direct3D11CaptureFramePool::Create which would mirror what would at least partially explain the issue I am seeing with IDirect3D11CaptureFramePoolStatics::Create, which I assume is using the same logic under the hood.

System info:

Edition	Windows 10 Pro
Version	22H2
Installed on	‎13/‎02/‎2021
OS build	19045.4170
Experience	Windows Feature Experience Pack 1000.19054.1000.0

[robmikh]

Thanks for the report!