WinUI 3 tests failing at random in CI

[Arlodotexe]

Describe the bug

The WinUI 3 tests that are run in the CI for build validation are sometimes failing with the following error:

  Passed RequestsAsync (20,5) [4 ms]
  Passed RequestsAsync (5,5) [< 1 ms]
  Passed RequestsAsync (20,5,2500,1000,1000,1000,1000) [6 s]
The active test run was aborted. Reason: Unable to communicate with test host process.
Terminating app with process ID '14816'.
Results File: C:\a\Windows\Windows\TestResults\WinAppSdk.trx

The active Test Run was aborted because the host process exited unexpectedly. Please inspect the call stack above, if available, to get more information about where the exception originated from.
The test running when the crash occurred: 
StackedNotificationsBehaviorExperiment.Tests.StackedNotificationsBehaviorTestClass.NotificationBehaviorSingleTest_StackedNotificationsBehaviorTestClass_Test

This test may, or may not be the source of the crash.

This seems to be happening more or less at random and can sometimes be resolved with a rerun. However, this has been happening more often in the last few weeks and is increasingly becoming a blocker for other work.

Steps to reproduce

Exact repro steps are unknown. Seems to only be happening in the CI and hasn't been reproduced locally.

Expected behavior

Consistent test results, no failure if there's no problem.

Screenshots

No response

Code Platform

• UWP
• WinAppSDK / WinUI 3
• Web Assembly (WASM)
• Android
• iOS
• MacOS
• Linux / GTK

Windows Build Number

• Windows 10 1809 (Build 17763)
• Windows 10 1903 (Build 18362)
• Windows 10 1909 (Build 18363)
• Windows 10 2004 (Build 19041)
• Windows 10 20H2 (Build 19042)
• Windows 10 21H1 (Build 19043)
• Windows 10 21H2 (Build 19044)
• Windows 10 22H2 (Build 19045)
• Windows 11 21H2 (Build 22000)
• Other (specify)

Other Windows Build number

No response

App minimum and target SDK version

• Windows 10, version 1809 (Build 17763)
• Windows 10, version 1903 (Build 18362)
• Windows 10, version 1909 (Build 18363)
• Windows 10, version 2004 (Build 19041)
• Windows 10, version 2104 (Build 20348)
• Windows 11, version 22H2 (Build 22000)
• Other (specify)

Other SDK version

No response

Visual Studio Version

No response

Visual Studio Build Number

No response

Device form factor

No response

Additional context

No response

Help us help you

Yes, I'd like to be assigned to work on this item.

[Arlodotexe]

Looks like the cleanup done in #348 has slowed down the frequency of the test error, but not eliminated it altogether.

[Arlodotexe]

In #453 (comment), we caught the issue happening on a UWP test run. This may not be specific to WinUI 3.

[michael-hawker]

Part of the instability seems to be unexpected exceptions in tests, like #589 where the exception is unhandled and therefore the test process gets hung.

In that case it's using the old setup vs. our new generators/helpers. So, it could be good to look how these behave with the new setups and if we can better ensure that any exceptions are caught within the test setup and reported as test fails explicitly instead of killing the test process.