[playground-tests] Flaky EventHubs test in TestEndpointsReturnOk

[radical]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=818717
Build error leg or test failing: Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
Pull request: N/A

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": " Timed out waiting for the consumer message to be logged: 'Hello, from /test sent via producerClient'",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=818717
Error message validated: [ Timed out waiting for the consumer message to be logged: 'Hello, from /test sent via producerClient']
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/26/2024 7:43:41 PM UTC

Report

Build	Definition	Test	Pull Request
825556	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	#6045

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

[radical]

On the last build the consumer shutdown with this exception:

      The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
      System.AggregateException: One or more errors occurred. (The messaging entity 'sb://emulatorns1.eventhubs.emulator.net/hub' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions.  (hub).  For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot)
       ---> Azure.Messaging.EventHubs.EventHubsException(ResourceNotFound): The messaging entity 'sb://emulatorns1.eventhubs.emulator.net/hub' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions.  (hub).  For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot
         at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response, String eventHubName)
         at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateProcessingPreconditions(CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingAsync(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventProcessorClient.StartProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventProcessorClient.StartProcessingAsync(CancellationToken cancellationToken)
         at EventHubsConsumer.Processor.ExecuteAsync(CancellationToken stoppingToken) in /datadisks/disk1/work/ABF7093F/w/B02B09B0/e/playground/AspireEventHub/EventHubsConsumer/Processor.cs:line 44
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

.. specifically - Azure.Messaging.EventHubs.EventHubsException(ResourceNotFound): The messaging entity 'sb://emulatorns1.eventhubs.emulator.net/hub' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions. (hub). For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot.

.. but the test keeps waiting for the message from the consumer, and times out.

We should fix the tests to also fail earlier, so the xunit test reports show the real cause, or closer to it.

cc @davidfowl @eerhardt

[davidfowl]

There's a bug in the consumer, I think this is a resiliency problem.