Winlogon.exe causing BSoD

[BattleAngel444]

Hi All

STOP: c0000021a BSOD sometimes at system shutdown

We have been seeing a BSOD sometimes at shutdown during a reboot test
that we are running on several systems. Typically, we see 1 of these
BSODs (screenshot: http://digidreamerz.com/screenshots/2010-02-22_09.43.47.jpg,
process memory dump: http://digidreamerz.com/WER852b.dir00.zip) per
night on one of our systems (we are running the reboot testing on 6
systems, typically 250-300 reboots per night total). After this
happens and we login after restarting the system, we typically get a
process memory dump of winlogon.exe in a temp folder. We have analyzed
it and always see that winlogon.exe is getting an access violation by
trying to write to address 00000000. We have done much debugging and
work to try to figure out why this is happening and we have not had
much luck in figuring out precisely how this happens.

Our systems are running WindowsXP Embedded with SP2 and a few other
patches. They all have the exact same hardware.

We are looking for suggestions on how to figure out what the problem
is, and how we can fix it.

Any help would be greatly appreciated

 

[BattleAngel444]

Hope this helps...

Some additional information on the Winlogon.exe crash:

We have been able to reproduce the problem on a system running a MSDN
checked-build of Winlogon with both SP3 and SP2. Enabling logging
with the checked-build shows that the problem seems to occur when a
Winlogon job (in our case, a group policy system shutdown script) is
dereferenced twice. Looks a lot like a race condition in within
Winlogon.

Here is a snippet of the Winlogon trace:

952.956> Winlogon-Trace: In InternalWinStationNotifyLogoff
952.3348> Winlogon-Trace-Notify: Executing Windows Update : Shutdown
952.3348> Winlogon-Error: [WUInstall] Failed to query WU value (2).
952.3348> Winlogon-Error: [WUInstall] Failed to clean WU value (2).
952.3348> Winlogon-Trace: [WUInstall] Skipping installs - not a
shutdown.
952.3348> Winlogon-Trace: [WUInstall] Skipping installs - not
requested.
952.3348> Winlogon-Trace: [WUInstall] Calling
WUAutoUpdateAtShutdown(0)...
952.3512> Winlogon-Trace-Notify: Executing Finish Machine Group
Policy : Shutdown
952.3512> Winlogon-Trace: ExecuteGPOScripts: Entering bSync = 1
952.752> Winlogon-Trace-Job: No timeout
952.752> Winlogon-Trace-Job: WaitJob (0:6054b) timeout in 92762
952.752> Winlogon-Trace-Job: WaitJob (0:6054b) timeout in 92714
952.752> Winlogon-Trace-Job: Job 0:6054b root process terminated
952.752> Winlogon-Trace-Job: WaitJob (0:6054b) timeout in 926e5
952.752> Winlogon-Trace-Job: Job 0:6054b completed
952.3512> Winlogon-Trace-Job: Deref job 0:6054b, current ref 2
952.3512> Winlogon-Trace: ExecuteGPOScripts: Leaving.
952.752> Winlogon-Trace-Job: Unlinking Job 0:6054b
952.752> Winlogon-Trace-Job: Deref job 0:6054b, current ref 1
952.3512> Winlogon-Trace: StopMachineGPOProcessing: Waiting for
machine group policy thread to terminate.
952.752> Winlogon-Trace-Job: WaitJob (0:6054b) timeout in ffffffff
952.752> Winlogon-Trace-Job: Root-died termination for job 0:6054b
952.752> Winlogon-Trace-Job: WaitJob (0:6054b) timeout in ffffffff
952.752> Winlogon-Trace-Job: Job 0:6054b completed
952.752> Winlogon-Trace-Job: Deref job 0:6054b, current ref 1
952.3512> Winlogon-Trace: StopMachineGPOProcessing: Machine group
policy thread has terminated.
952.1336> Winlogon-Trace-Notify: Executing C:\WINDOWS
\system32\cscdll.dll : Shutdown

~ Crashes after last message

And here is the exception analysis by the kernel debugger:
*******************************************************************************
*
*
* Exception
Analysis *
*
*
*******************************************************************************


FAULTING_IP:
ntdll!DbgBreakPoint+0
001b:7c90120e cc int 3

EXCEPTION_RECORD: 0136fc70 -- (.exr 136fc70)
ExceptionAddress: 0104b97f (winlogon!DerefWinlogonJob+0x00000065)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 00000000
Attempt to write to address 00000000

DEFAULT_BUCKET_ID: NULL_DEREFERENCE

PROCESS_NAME: ntkrnlmp.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx"
referenced memory at "0x%08lx". The memory could not be "%s".

CONTEXT: 0136fc8c -- (.cxr 136fc8c)
eax=00000000 ebx=010858e0 ecx=000ad118 edx=00000007 esi=000acfe0
edi=010858e0
eip=0104b97f esp=0136ff58 ebp=0136ff60 iopl=0 nv up ei pl nz
na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000
efl=00010206
winlogon!DerefWinlogonJob+0x65:
001b:0104b97f 8908 mov dword ptr [eax],ecx ds:
0023:00000000=????????
Resetting default scope

WRITE_ADDRESS: 00000000

BUGCHECK_STR: ACCESS_VIOLATION

LAST_CONTROL_TRANSFER: from 0104bb72 to 0104b97f

STACK_TEXT:
0136ff60 0104bb72 000acfe0 00070000 00099650 winlogon!DerefWinlogonJob
+0x65
0136ffb4 7c80b713 00000000 00070000 00099650 winlogon!JobThread+0x1d3
0136ffec 00000000 0104b99f 00000000 00000000 kernel32!BaseThreadStart
+0x37


FOLLOWUP_IP:
winlogon!DerefWinlogonJob+65
001b:0104b97f 8908 mov dword ptr [eax],ecx

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: winlogon!DerefWinlogonJob+65

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: winlogon

IMAGE_NAME: winlogon.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 4802c677

STACK_COMMAND: .cxr 0x136fc8c ; kb

FAILURE_BUCKET_ID: ACCESS_VIOLATION_winlogon!DerefWinlogonJob+65

BUCKET_ID: ACCESS_VIOLATION_winlogon!DerefWinlogonJob+65

Followup: MachineOwner
---------

0: kd> k
ChildEBP RetAddr
0136f894 7c9652ae ntdll!DbgBreakPoint
0136f8d4 7c9659c1 ntdll!RtlUnhandledExceptionFilter2+0x27b
0136f8e4 7c864031 ntdll!RtlUnhandledExceptionFilter+0x12
0136fb54 7c83ab38 kernel32!UnhandledExceptionFilter+0x1c7
0136fb5c 7c839b21 kernel32!BaseThreadStart+0x4d
0136fb84 7c9032a8 kernel32!_except_handler3+0x61
0136fba8 7c90327a ntdll!ExecuteHandler2+0x26
0136fc58 7c90e46a ntdll!ExecuteHandler+0x24
0136fc58 0104b97f ntdll!KiUserExceptionDispatcher+0xe
0136ff60 0104bb72 winlogon!DerefWinlogonJob+0x65
0136ffb4 7c80b713 winlogon!JobThread+0x1d3
0136ffec 00000000 kernel32!BaseThreadStart+0x37