Analyzing crash dump file from Windows 2003 R2 SP2

bsodserver-crasheswindows-server-2003

My Windows 2003 R2 SP2 server crashes about once a month. I analyzed the crash dump but can't really figure out what it means. Any help is appreciated….

Microsoft (R) Windows Debugger Version 6.2.9200.20512 AMD64 Copyright
(c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Users\noam\Downloads\MEMORY.DMP] Kernel Summary
Dump File: Only kernel address space is available

Symbol search path is:
SRVC:\Windows\Symbolshttp://msdl.microsoft.com/download/symbols
Executable search path is: Windows Server 2003 Kernel Version 3790
(Service Pack 2) MP (4 procs) Free x64 Product: Server, suite:
TerminalServer SingleUserTS Built by: 3790.srv03_sp2_qfe.120821-0338
Machine Name: Kernel base = 0xfffff80001000000 PsLoadedModuleList =
0xfffff800011d8280 Debug session time: Thu Mar 21 10:48:04.909 2013
(UTC – 4:00) System Uptime: 24 days 13:36:09.953 Loading Kernel
Symbols
………………………………………………………
………………………………………….. Loading User
Symbols PEB is paged out (Peb.Ldr = 000007ff`fffdf018). Type ".hh
dbgerr001" for details Loading unloaded module list ….

  • *
  • Bugcheck Analysis *
  • *

Use !analyze -v to get detailed debugging information.

BugCheck 50, {fffffadf28a13000, 0, fffff800012c532e, 0}

*** ERROR: Module load completed but symbols could not be loaded for sptd.sys Probably caused by : sptd.sys ( sptd+5fe75 )

Followup: MachineOwner

3: kd> !analyze -v

  • *
  • Bugcheck Analysis *
  • *

PAGE_FAULT_IN_NONPAGED_AREA (50) Invalid system memory was referenced.
This cannot be protected by try-except, it must be protected by a
Probe. Typically the address is just plain bad or it is pointing at
freed memory. Arguments: Arg1: fffffadf28a13000, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff800012c532e, If non-zero, the instruction address which
referenced the bad memory address. Arg4: 0000000000000000, (reserved)

Debugging Details:

READ_ADDRESS: fffffadf28a13000

FAULTING_IP: nt!PspGetSetContextInternal+203 fffff800`012c532e
488b58f8 mov rbx,qword ptr [rax-8]

MM_INTERNAL_CODE: 0

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0x50

PROCESS_NAME: w3wp.exe

CURRENT_IRQL: 1

TRAP_FRAME: fffffadf1fb32710 — (.trap 0xfffffadf1fb32710) NOTE: The
trap frame does not contain all registers. Some register values may be
zeroed or incorrect. rax=fffffadf28a13007 rbx=0000000000000000
rcx=0000000000000001 rdx=0000000000000000 rsi=0000000000000000
rdi=0000000000000000 rip=fffff800012c532e rsp=fffffadf1fb328a0
rbp=fffffadf228d0b10 r8=0000000000000000 r9=0000000000000000
r10=0000000000000000 r11=0000000000000000 r12=0000000000000000
r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0
nv up ei ng nz ac pe nc nt!PspGetSetContextInternal+0x203:
fffff800012c532e 488b58f8 mov rbx,qword ptr [rax-8]
ds:fffffadf28a12fff=???????????????? Resetting default scope

LAST_CONTROL_TRANSFER: from fffff800010a69dd to fffff8000102eb90

STACK_TEXT: fffffadf1fb32638 fffff800010a69dd : 0000000000000050
fffffadf28a13000 0000000000000000 fffffadf1fb32710 :
nt!KeBugCheckEx fffffadf1fb32640 fffff8000102d759 :
0000000000000000 0000000000000000 0000000000000000
0000000000000000 : nt!MmAccessFault+0xa1f fffffadf1fb32710
fffff800012c532e : 0000000000000000 0000000000000000
fffffadf1fb33c70 fffffadf228d05c0 : nt!KiPageFault+0x119
fffffadf1fb328a0 fffff800010425bb : fffffadf35821040
fffffadf1fb32fa0 fffffadf35821088 0000000000000000 :
nt!PspGetSetContextInternal+0x203 fffffadf1fb32df0 fffff80001028081
: fffffadf385ce018 fffff80000810310 0000000000000001
fffffadf385ce018 : nt!PspGetSetContextSpecialApc+0xab
fffffadf1fb32f00 fffff80001027c8d : fffffadf358210b8
0000000000000000 0000000000000000 fffffadf35c06558 :
nt!KiDeliverApc+0x215 fffffadf1fb32fa0 fffffadf28b33e75 :
fffffadf28b3b91c fffffadf35c06558 fffffadf35c06440
0000000000000000 : nt!KiApcInterrupt+0xdd fffffadf1fb33138
fffffadf28b3b91c : fffffadf35c06558 fffffadf35c06440
0000000000000000 0000000000000000 : sptd+0x5fe75 fffffadf1fb33140
fffffadf38ddd2de : fffffadf1fb33240 fffff8000102550f
0000000000000000 fffffadf1fb33248 : sptd+0x6791c fffffadf1fb33220
fffffadf1fb33240 : fffff8000102550f 0000000000000000
fffffadf1fb33248 fffffadf385cc050 : 0xfffffadf38ddd2de
fffffadf1fb33228 fffff8000102550f : 0000000000000000
fffffadf1fb33248 fffffadf385cc050 0000000000000000 :
0xfffffadf1fb33240 fffffadf1fb33230 fffff800011ad8fd :
0001100000000000 fffffadf385cc050 0000000180392000
0000000000000002 : nt!IopfCompleteRequest+0x9c8 fffffadf1fb332a0
fffffadf358a8ca0 : 0000000000001000 0000000a1f1e5000
fffffadf385d3040 fffffadf289da2b7 : nt!ExFreePoolWithTag+0x67b
fffffadf1fb33360 0000000000001000 : 0000000a1f1e5000
fffffadf385d3040 fffffadf289da2b7 fffffadf385d4cb0 :
0xfffffadf358a8ca0 fffffadf1fb33368 0000000a1f1e5000 :
fffffadf385d3040 fffffadf289da2b7 fffffadf385d4cb0
fffffadf1fb33860 : 0x1000 fffffadf1fb33370 fffffadf385d3040 :
fffffadf289da2b7 fffffadf385d4cb0 fffffadf1fb33860
fffffadf358a8f68 : 0x0000000a1f1e5000 fffffadf1fb33378
fffffadf289da2b7 : fffffadf385d4cb0 fffffadf1fb33860
fffffadf358a8f68 fffffadf358a8ca0 : 0xfffffadf385d3040
fffffadf1fb33380 fffffadf28943f92 : fffffadf358a8ca0
fffffadf37eee640 fffffadf358a8f68 fffffadf385d3190 :
ftdisk!FtDiskReadWrite+0x1e7 fffffadf1fb333d0 fffffadf2874f361 :
0000000000000000 fffffadf2676915c fffffadf37b811c8
fffffadf358a8ca0 : volsnap!VolSnapRead+0xa2 fffffadf1fb33410
fffffadf2875e582 : fffffadf00000000 fffffadf00001000
fffffadf37eee640 fffffadf00000000 : Ntfs!NtfsPagingFileIo+0x202
fffffadf1fb33510 fffffadf28751e2e : fffffadf1fb336a0
fffffadf358a8ca0 fffffadf1fb33601 fffffadf1fb336e0 :
Ntfs!NtfsCommonRead+0x4d3 fffffadf1fb336a0 2444c7c88b48d233 :
44c7487346635328 15ff000000982024 762d8b4800002f0d
8b481445ff00008d : Ntfs!NtfsFsdRead+0x262 fffffadf289da3a7
44c7487346635328 : 15ff000000982024 762d8b4800002f0d
8b481445ff00008d 4800002f0a15ffcd : 0x2444c7c88b48d233
fffffadf289da3af 15ff000000982024 : 762d8b4800002f0d
8b481445ff00008d 4800002f0a15ffcd 8b1875f08b48c085 :
0x44c7487346635328 fffffadf289da3b7 762d8b4800002f0d :
8b481445ff00008d 4800002f0a15ffcd 8b1875f08b48c085
4d8b28458b442c55 : 0x15ff000000982024 fffffadf289da3bf
8b481445ff00008d : 4800002f0a15ffcd 8b1875f08b48c085
4d8b28458b442c55 483055ff1845ff24 : 0x762d8b4800002f0d
fffffadf289da3c7 4800002f0a15ffcd : 8b1875f08b48c085
4d8b28458b442c55 483055ff1845ff24 c62474f08b48c085 :
0x8b481445ff00008d fffffadf289da3cf 8b1875f08b48c085 :
4d8b28458b442c55 483055ff1845ff24 c62474f08b48c085
3807058d48017a46 : 0x4800002f0a15ffcd fffffadf289da3d7
4d8b28458b442c55 : 483055ff1845ff24 c62474f08b48c085
3807058d48017a46 46c6007846c60000 : 0x8b1875f08b48c085
fffffadf289da3df 483055ff1845ff24 : c62474f08b48c085
3807058d48017a46 46c6007846c60000 c748004246c60079 :
0x4d8b28458b442c55 fffffadf289da3e7 c62474f08b48c085 :
3807058d48017a46 46c6007846c60000 c748004246c60079
8948000000001046 : 0x483055ff1845ff24 fffffadf289da3ef
3807058d48017a46 : 46c6007846c60000 c748004246c60079
8948000000001046 ff184b8d486beb06 : 0xc62474f08b48c085
fffffadf289da3f7 46c6007846c60000 : c748004246c60079
8948000000001046 ff184b8d486beb06 a8bb8000002ea415 :
0x3807058d48017a46 fffffadf289da3ff c748004246c60079 :
8948000000001046 ff184b8d486beb06 a8bb8000002ea415
8b483d7400000000 : 0x46c6007846c60000 fffffadf289da407
8948000000001046 : ff184b8d486beb06 a8bb8000002ea415
8b483d7400000000 878d4c000000b897 : 0xc748004246c60079
fffffadf289da40f ff184b8d486beb06 : a8bb8000002ea415
8b483d7400000000 878d4c000000b897 988b8d4c000000a8 :
0x8948000000001046 fffffadf289da417 a8bb8000002ea415 :
8b483d7400000000 878d4c000000b897 988b8d4c000000a8
4901034a80000000 : 0xff184b8d486beb06 fffffadf289da41f
8b483d7400000000 : 878d4c000000b897 988b8d4c000000a8
4901034a80000000 894908894d08518b : 0xa8bb8000002ea415
fffffadf289da427 878d4c000000b897 : 988b8d4c000000a8
4901034a80000000 894908894d08518b 4b8d4802894c0850 :
0x8b483d7400000000 fffffadf289da42f 988b8d4c000000a8 :
4901034a80000000 894908894d08518b 4b8d4802894c0850
0841894dd0b60f18 : 0x878d4c000000b897 fffffadf289da437
4901034a80000000 : 894908894d08518b 4b8d4802894c0850
0841894dd0b60f18 abe900002cdb15ff : 0x988b8d4c000000a8
fffffadf289da43f 894908894d08518b : 4b8d4802894c0850
0841894dd0b60f18 abe900002cdb15ff 0090b38b48000000 :
0x4901034a80000000 fffffadf289da447 4b8d4802894c0850 :
0841894dd0b60f18 abe900002cdb15ff 0090b38b48000000
b60f184b8d480000 : 0x894908894d08518b fffffadf289da44f
0841894dd0b60f18 : abe900002cdb15ff 0090b38b48000000
b60f184b8d480000 01000000a883c6d0 : 0x4b8d4802894c0850
fffffadf289da457 abe900002cdb15ff : 0090b38b48000000
b60f184b8d480000 01000000a883c6d0 8b4c00002cbb15ff :
0x0841894dd0b60f18 fffffadf289da45f 0090b38b48000000 :
b60f184b8d480000 01000000a883c6d0 8b4c00002cbb15ff
478b48000000b887 : 0xabe900002cdb15ff fffffadf289da467
b60f184b8d480000 : 01000000a883c6d0 8b4c00002cbb15ff
478b48000000b887 468948107e894808 : 0x0090b38b48000000
fffffadf289da46f 01000000a883c6d0 : 8b4c00002cbb15ff
478b48000000b887 468948107e894808 508d4918408b4908 :
0xb60f184b8d480000 fffffadf289da477 8b4c00002cbb15ff :
478b48000000b887 468948107e894808 508d4918408b4908
4c028b2046894808 : 0x01000000a883c6d0 fffffadf289da47f
478b48000000b887 : 468948107e894808 508d4918408b4908
4c028b2046894808 8d48184689306689 : 0x8b4c00002cbb15ff
fffffadf289da487 468948107e894808 : 508d4918408b4908
4c028b2046894808 8d48184689306689 46894800024b1405 :
0x478b48000000b887 fffffadf289da48f 508d4918408b4908 :
4c028b2046894808 8d48184689306689 46894800024b1405
00000098878b4828 : 0x468948107e894808 fffffadf289da497
4c028b2046894808 : 8d48184689306689 46894800024b1405
00000098878b4828 40b60f4138468948 : 0x508d4918408b4908
fffffadf289da49f 8d48184689306689 : 46894800024b1405
00000098878b4828 40b60f4138468948 0338804140468802 :
0x4c028b2046894808 fffffadf289da4a7 46894800024b1405 :
00000098878b4828 40b60f4138468948 0338804140468802
0f00000080be8948 : 0x8d48184689306689 fffffadf289da4af
00000098878b4828 : 40b60f4138468948 0338804140468802
0f00000080be8948 0000889e8948c094 : 0x46894800024b1405
fffffadf289da4b7 40b60f4138468948 : 0338804140468802
0f00000080be8948 0000889e8948c094 0164bb8041468800 :
0x00000098878b4828 fffffadf289da4bf 0338804140468802 :
0f00000080be8948 0000889e8948c094 0164bb8041468800
438b481174000000 : 0x40b60f4138468948 fffffadf289da4c7
0f00000080be8948 : 0000889e8948c094 0164bb8041468800
438b481174000000 000001708b8b4808 : 0x0338804140468802
fffffadf289da4cf 0000889e8948c094 : 0164bb8041468800
438b481174000000 000001708b8b4808 8b480000011090ff :
0x0f00000080be8948 fffffadf289da4d7 0164bb8041468800 :
438b481174000000 000001708b8b4808 8b480000011090ff
d68b48000000b887 : 0x0000889e8948c094 fffffadf289da4df
438b481174000000 : 000001708b8b4808 8b480000011090ff
d68b48000000b887 304e8b4801034880 : 0x0164bb8041468800
fffffadf289da4e7 000001708b8b4808 : 8b480000011090ff
d68b48000000b887 304e8b4801034880 8b483050ff018b48 :
0x438b481174000000 fffffadf289da4ef 8b480000011090ff :
d68b48000000b887 304e8b4801034880 8b483050ff018b48
4024648b4c58246c : 0x000001708b8b4808 fffffadf289da4f7
d68b48000000b887 : 304e8b4801034880 8b483050ff018b48
4024648b4c58246c 7c8b486024748b48 : 0x8b480000011090ff
fffffadf289da4ff 304e8b4801034880 : 8b483050ff018b48
4024648b4c58246c 7c8b486024748b48 b850245c8b486824 :
0xd68b48000000b887 fffffadf289da507 8b483050ff018b48 :
4024648b4c58246c 7c8b486024748b48 b850245c8b486824
48c4834800000103 : 0x304e8b4801034880 fffffadf289da50f
4024648b4c58246c : 7c8b486024748b48 b850245c8b486824
48c4834800000103 ccccccccccccccc3 : 0x8b483050ff018b48
fffffadf289da517 7c8b486024748b48 : b850245c8b486824
48c4834800000103 ccccccccccccccc3 cccccccccccccccc :
0x4024648b4c58246c fffffadf289da51f b850245c8b486824 :
48c4834800000103 ccccccccccccccc3 cccccccccccccccc
828b4838ec8348cc : 0x7c8b486024748b48 fffffadf289da527
48c4834800000103 : ccccccccccccccc3 cccccccccccccccc
828b4838ec8348cc 246c8948000000b8 : 0xb850245c8b486824
fffffadf289da52f ccccccccccccccc3 : cccccccccccccccc
828b4838ec8348cc 246c8948000000b8 107883186a8b4848 :
0x48c4834800000103 fffffadf289da537 cccccccccccccccc :
828b4838ec8348cc 246c8948000000b8 107883186a8b4848
c000000db80f7318 : 0xccccccccccccccc3 fffffadf289da53f
828b4838ec8348cc : 246c8948000000b8 107883186a8b4848
c000000db80f7318 c4834848246c8b48 : 0xcccccccccccccccc
fffffadf289da547 246c8948000000b8 : 107883186a8b4848
c000000db80f7318 c4834848246c8b48 4840245c8948c338 :
0x828b4838ec8348cc

STACK_COMMAND: kb

FOLLOWUP_IP: sptd+5fe75 fffffadf`28b33e75 0000 add
byte ptr [rax],al

SYMBOL_STACK_INDEX: 7

SYMBOL_NAME: sptd+5fe75

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: sptd

IMAGE_NAME: sptd.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4611064e

FAILURE_BUCKET_ID: X64_0x50_sptd+5fe75

BUCKET_ID: X64_0x50_sptd+5fe75

Followup: MachineOwner

3: kd> lmvm sptd start end module name
fffffadf28ad4000 fffffadf28bf2000 sptd (no symbols) 

Loaded symbol image file: sptd.sys
Image path: sptd.sys
Image name: sptd.sys
Timestamp:        Mon Apr 02 09:34:06 2007 (4611064E)
CheckSum:         000D960A
ImageSize:        0011E000
Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Best Answer

I'm not the best OS debugger, but here's what I see:

PAGE_FAULT_IN_NONPAGED_AREA (50) Invalid system memory was referenced.

What often causes this is a "use after free" condition where memory is freed, and then a thread references it as if it were still allocated. When a user-mode program does this, it usually just ends up crashing that program. But when a kernel-mode component, such as a driver, does this, it causes a bug check. It can also be caused when a thread attempts to write to an address that is marked as read-only. But because argument two of the error code specified that the fault occured during a read operation, we can eliminate that possibility.

Other things can cause this bug check as well, such as faulty hardware, and cosmic rays flipping bits in your RAM. But for the sake of discussion we'll disregard those possibilities as well.

If you look at the stack text - this stands out to me:

ExFreePoolWithTag+0x67b fffffadf1fb33360

That function freed pool memory, which is in kernel space. I'm not positive that it freed the memory that was later referenced that caused the machine to crash, but I'm highly suspicious of it. It is typically drivers that allocate and deallocate pool memory. The people who write drivers have to be extremely careful about allocating and deallocating memory, because if you don't do it perfectly, you either cause a memory leak or you crash the machine.

If you look here, you'll see the parameters of your bug check code:

Parameter       Description
---------------------------    
1   Memory address referenced    
2   0: Read operation   1: Write operation    
3   Address that referenced memory (if known)    
4   Reserved

Windbg identified sptd.sys as a probable cause of the crash. I'm guessing Windbg figured that sptd.sys is the culprit because it happened to be loaded in the address that was found in argument 3 of the bug chuck code. (But I could be wrong about that. I'm not sure how Windbg derives that information.) The information is not guaranteed to be accurate in any case, but stpd.sys appears to be a non-Microsoft driver related to CD/DVD burning software like Daemon Tools and Alcohol 120%.

I would definitely start by either upgrading or uninstalling that software.

Edit: Looks like you can find updated versions of sptd.sys here: http://www.disc-tools.com/download/sptd

Related Topic