Internal CLR error. (0x80131506) in threads.cpp

[majocha]

Description

We're observing random crashes when running xUnit tests in the CI targeting net9.0:
The active test run was aborted. Reason: Test host process crashed : Fatal error. Internal CLR error. (0x80131506)

Reproduction Steps

I have no way yet to reproduce this reliably.

Minidump:
879847-testhost.exe-28560-1732638273.dmp

FILE_IN_CAB:  879847-testhost.exe-28560-1732638273.dmp

NTGLOBALFLAG:  0

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  000072e8

PROCESS_NAME:  testhost.dll

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

STACK_TEXT:  
000000d6`e8f7df78 00007ffa`4a98464c     : aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa 00000000`00000000 00000255`e9091050 : ntdll!NtWaitForMultipleObjects+0x14
000000d6`e8f7df80 00007ffa`104cd0f0     : aaaaaaaa`aaaaaaaa 00000000`00000000 aaaaaaaa`aaaaaaaa 00000255`e9091120 : KERNELBASE!WaitForMultipleObjectsEx+0xec
000000d6`e8f7e270 00007ffa`104cce05     : 00000000`00000000 000000d6`e8f7e331 00000255`e9091120 00007ffa`108591b0 : coreclr!Thread::DoAppropriateAptStateWait+0x5c
000000d6`e8f7e2b0 00007ffa`104ccc44     : 000000d6`e8f7e490 aaaaaaaa`aaaaaaaa 00000255`e9091120 aaaaaaaa`aaaaaaaa : coreclr!Thread::DoAppropriateWaitWorker+0x171
000000d6`e8f7e380 00007ffa`104ca37d     : 00007ffa`108590c0 aaaaaaaa`aaaaaaaa 00000000`00000000 aaaaaaaa`aaaaaaaa : coreclr!Thread::DoAppropriateWait+0xb0
000000d6`e8f7e420 00007ffa`104ca4ff     : 000000d6`e8f7e6a0 aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa 000000d6`e8f7e6a0 : coreclr!SyncBlock::Wait+0x1c9
000000d6`e8f7e560 00007ffa`0f902fc5     : 00000000`00000000 aaaaaaaa`aaaaaaaa 00000255`e9091050 aaaaaaaa`aaaaaaaa : coreclr!Monitor_Wait+0x7f
000000d6`e8f7e5d0 00007ffa`0f90fa4f     : 00000255`f2c09f10 aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa aaaaaaaa`aaaaaaaa : System_Private_CoreLib+0x3a2fc5
000000d6`e8f7e6a0 00007ffa`0f928ff6     : 00000255`f2c09ee8 aaaaaaaa`aaaaaaaa 00000000`00000000 00007ff9`b095fda0 : System_Private_CoreLib+0x3afa4f
000000d6`e8f7e740 00007ffa`0f928dee     : 00000255`f2c09e60 aaaaaaaa`aaaaaaaa 00000000`00000000 00000255`e9091050 : System_Private_CoreLib+0x3c8ff6
000000d6`e8f7e7c0 00007ffa`0f928a2e     : 00000255`f343ee98 aaaaaaaa`aaaaaaaa 00000000`00000000 00000255`e9091050 : System_Private_CoreLib+0x3c8dee
000000d6`e8f7e840 00007ffa`0f9288c2     : 00000255`f343ee98 aaaaaaaa`aaaaaaaa 00000000`00000000 00000255`e9091050 : System_Private_CoreLib+0x3c8a2e
000000d6`e8f7e870 00007ff9`b098b4c7     : 00000255`f343ee98 00000000`00000000 aaaaaaaa`aaaaaaaa 00000255`e9091050 : System_Private_CoreLib+0x3c88c2
000000d6`e8f7e8a0 00000255`f343ee98     : 00000000`00000000 aaaaaaaa`aaaaaaaa 00000255`e9091050 000000d6`e8f7e8a0 : 0x00007ff9`b098b4c7
000000d6`e8f7e8a8 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00000255`f343ee98


STACK_COMMAND:  ~0s; .ecxr ; kb

FAULTING_SOURCE_LINE:  D:\a\_work\1\s\src\coreclr\vm\threads.cpp

FAULTING_SOURCE_FILE:  D:\a\_work\1\s\src\coreclr\vm\threads.cpp

FAULTING_SOURCE_LINE_NUMBER:  3184

FAULTING_SOURCE_SRV_COMMAND:  https://raw.githubusercontent.com/dotnet/runtime/9d5a6a9aa463d6d10b0b0ba6d5982cc82f363dc3/src/coreclr/vm/threads.cpp

FAULTING_SOURCE_CODE:  
No source found for 'D:\a\_work\1\s\src\coreclr\vm\threads.cpp'


SYMBOL_NAME:  coreclr!Thread::DoAppropriateAptStateWait+5c

MODULE_NAME: coreclr

IMAGE_NAME:  coreclr.dll

FAILURE_BUCKET_ID:  BREAKPOINT_80000003_coreclr.dll!Thread::DoAppropriateAptStateWait

OS_VERSION:  10.0.20348.1

BUILDLAB_STR:  fe_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  9.0.24.52809

FAILURE_ID_HASH:  {e2358c87-f452-7ecb-c918-08173d45a949}

Followup:     MachineOwner
---------

Expected behavior

No crash.

Actual behavior

testhost crashes with Internal CLR error. (0x80131506)

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

This started after enabling test parallelism, so there is much more pressure on the GC and a lot of threads in use during the test run.

[jkotas]

I have opened 879847-testhost.exe-28560-1732638273.dmp from https://github.com/user-attachments/files/17924725/dumps.zip . I see it crashing with null reference exception at this stacktrace:

coreclr!LoaderAllocator::SetHandleValue+0xb2:
00007ffa`1060ead2 488b4210        mov     rax,qword ptr [rdx+10h] ds:00000000`00000010=????????????????
0:029> k
 # Child-SP          RetAddr               Call Site
00 000000d6`ebbff9d0 00007ffa`1060dc2e     coreclr!LoaderAllocator::SetHandleValue+0xb2 [D:\a\_work\1\s\src\coreclr\vm\loaderallocator.cpp @ 991] 
01 000000d6`ebbffa40 00007ffa`104ed30a     coreclr!LoaderAllocator::FreeHandle+0x1a [D:\a\_work\1\s\src\coreclr\vm\loaderallocator.cpp @ 883] 
02 000000d6`ebbffa70 00007ffa`104ecec2     coreclr!FreeLoaderAllocatorHandlesForTLSData+0xce [D:\a\_work\1\s\src\coreclr\vm\threadstatics.cpp @ 394] 
03 000000d6`ebbffae0 00007ffa`104ec785     coreclr!Thread::OnThreadTerminate+0xa6 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 2838] 
04 000000d6`ebbffb60 00007ffa`104c5225     coreclr!DestroyThread+0x55 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 862] 
05 000000d6`ebbffb90 00007ffa`4c6f4cb0     coreclr!ThreadNative::KickOffThread+0xc5 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 239] 
06 000000d6`ebbffbf0 00007ffa`4d41edcb     kernel32!BaseThreadInitThunk+0x10 [clientcore\base\win32\client\thread.c @ 75] 
07 000000d6`ebbffc20 00000000`00000000     ntdll!RtlUserThreadStart+0x2b [minkernel\ntdll\rtlstrt.c @ 1152] 

This looks like a regression from the statics simplification. cc @davidwrighton

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[psfinaki]

I think we also have a more recent one if it helps / applies :)
899623-testhost.exe-37660-1734609522.dmp