SqlServer 2008R2 sp1 CHECKDB crashes when spatial index present

checkdbsql-server-2008-r2

We've been having an issue with DBCC CHECKDB crashing with an access violation (null pointer deference) on Sql Server 2008R2 when spatial indexes are present. It is repeatable with DBCC CHECKDB WITH EXTENDED_LOGICAL_CHECKS but also happens in some cases with just DBCC CHECKDB.

This is occurring for us on Sql Server 2008 R2 sp1 standard (and developer edition) under Windows 2008 R2 and under Windows 7 (all 64 bit).

Here's a simple T-SQL script that demonstrates the issue. If you run it in the SSMS you'll see the output and the sql connection getting terminated.

use master
go
if exists(select * from sys.databases where name = 'DbccCrashExample') drop database DbccCrashExample
go
create database DbccCrashExample
GO
use DbccCrashExample
go
CREATE TABLE dbo.GeometryTable
(
    GeometryTableID int NOT NULL,
    Feature geometry NOT NULL,
    CONSTRAINT PK_GeometryTable PRIMARY KEY CLUSTERED (GeometryTableID)
)
GO
insert into dbo.GeometryTable(GeometryTableID, Feature)
select 1, geometry::STGeomFromText('POINT(0 0)', 4326)
go
create spatial index SPATIAL_GeometryTable_Feature on dbo.GeometryTable(Feature) with (BOUNDING_BOX=(0, -2, 1, 2))

-- works
--DBCC CHECKDB

-- fails
DBCC CHECKDB WITH EXTENDED_LOGICAL_CHECKS

One strange thing is that it appears that DBCC has completed without errors, but then a severe error occurs:

CHECKDB found 0 allocation errors and 0 consistency errors in database 'DbccCrashExample'.
Msg 0, Level 11, State 0, Line 0
A severe error occurred on the current command.  The results, if any, should be discarded.
Msg 0, Level 20, State 0, Line 0
A severe error occurred on the current command.  The results, if any, should be discarded.

In the Sql Server error log we get a stack dump like this:

SqlDumpExceptionHandler: Process 59 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is terminating this process.                                                                                        
* *******************************************************************************                                
*                                                                                                                
* BEGIN STACK DUMP:                                                                                              
*   11/20/11 13:23:34 spid 59                                                                                    
*                                                                                                                
*                                                                                                                
*   Exception Address = 0000000000E84A8D Module(sqlservr+0000000000274A8D)                                       
*   Exception Code    = c0000005 EXCEPTION_ACCESS_VIOLATION                                                      
*   Access Violation occurred reading address 0000000000000000                                                   
* Input Buffer 408 bytes -                                                                                       
*             create spatial index SPATIAL_GeometryTable_Feature on dbo.Geo                                      
*  metryTable(Feature) with (BOUNDING_BOX=(0, -2, 1, 2))   -- works --DBCC                                       
*  CHECKDB  -- fails DBCC CHECKDB WITH EXTENDED_LOGICAL_CHECKS                                                   
*                                                                                          

We have opened a paid support case with Microsoft about the issue. I am posting this problem here to share the issue with others. I will post what we hear back from Microsoft on the issue.

Best Answer

Update:

The fix has been released in Sql Server 2008 r2 sp1 cu4.

Cumulative update package 4 for SQL Server 2008 R2 Service Pack 1

More details about the issue are available here:

FIX: Access violation when you run a DBCC CHECKDB command against a database that contains a table that has a spatial index in SQL Server 2008 or in SQL Server 2008 R2

History:

We filed a support case with Microsoft about the issue. Apparently it is a known issue that has been fixed in other versions of Sql Server (2008 SP2 CU7, 2008 SP3 CU3, 2008R2 RTM CU11, 2008R2 SP1 CU4) and will be fixed in Sql Server 2008R2 SP1 CU4 (sp1 Cumulative Update 4).

So the solution for now is to not run "WITH EXTENDED_LOGICAL_CHECKS" or skip "DBCC CHECKDB" altogether until the cumulative update is out in mid December 2011.

Issue Description
-----------------------------

DBCC CHECKDB WITH EXTENDED_LOGICAL_CHECKS fails with below error and it generates an Access Violation dump

CHECKDB found 0 allocation errors and 0 consistency errors in database 'AriesTempForEtlOnly'.
Msg 0, Level 11, State 0, Line 0
A severe error occurred on the current command.  The results, if any, should be discarded.
Msg 0, Level 20, State 0, Line 0
A severe error occurred on the current command.  The results, if any, should be discarded

DBCC CHECKDB runs fine and when you remove spatial indexes it seems to work fine.

Analysis
--------------------
-- Instruction which hit Access Violation

sqlservr!CScaOp_Identifier::Decorate+0x63:
00000000`00954a8d 488b01          mov     rax,qword ptr [rcx] ds:00000000`00000000=????????????????

-- Stack of the thread which hit an AV
Child-SP          RetAddr           Call Site
00000000`0e3d8dd0 00000000`01a40289 sqlservr!CScaOp_Identifier::Decorate+0x63
00000000`0e3d8e20 00000000`01a4007a sqlservr!CScaOp_Identifier::XRelBindSelf+0x16d
00000000`0e3d8eb0 00000000`01a3ccd9 sqlservr!CScaOpArg::XRelBind+0x62
00000000`0e3d8ee0 00000000`01a3cda2 sqlservr!XRelBindProjectList+0x8d
00000000`0e3d8f20 00000000`01a3d781 sqlservr!CRelOp_Project::XRelBind+0x56
00000000`0e3d8f90 00000000`01a3f877 sqlservr!CRelOp_Union::XRelBind+0xa5
00000000`0e3d8ff0 00000000`01a3f990 sqlservr!CRelOp_Query::XRelBind+0x27
00000000`0e3d9020 00000000`00d48d4b sqlservr!CRelOp_Query::XRelBindQuery+0xb4
00000000`0e3d91f0 00000000`008b37a5 sqlservr!CProchdr::FNormQuery+0x3a
00000000`0e3d9230 00000000`00876644 sqlservr!CProchdr::FNormalizeStep+0x13ae
00000000`0e3d9700 00000000`00877259 sqlservr!CSQLSource::FCompile+0xc99
00000000`0e3dbd20 00000000`008770fc sqlservr!CSQLSource::FCompWrapper+0xc1
00000000`0e3dbdf0 00000000`0074ac63 sqlservr!CSQLSource::Transform+0x4de
00000000`0e3dbeb0 00000000`00bce5c9 sqlservr!CSQLSource::Execute+0x449
00000000`0e3dbfe0 00000000`02418255 sqlservr!CSQLSource::SeExecute+0x17c
00000000`0e3dc0a0 00000000`028abc55 sqlservr!ExecXrel+0x1f1
00000000`0e3dc570 00000000`028ab7d5 sqlservr!CheckRowsetDiff::ExecuteXREL+0x195
00000000`0e3dc5f0 00000000`028ad41a sqlservr!CheckRowsetDiff::Execute+0x55
00000000`0e3dc630 00000000`028845eb sqlservr!RowsetDiffExecutor::Execute+0x386
00000000`0e3dd300 00000000`02882661 sqlservr!UtilDbccCheckDatabase+0x1a37
00000000`0e3de670 00000000`028bd0b0 sqlservr!DbccCheckDB+0x2bd
00000000`0e3de6d0 00000000`01bd50a2 sqlservr!DbccCommand::Execute+0xc8
00000000`0e3de7a0 00000000`00749a86 sqlservr!CStmtDbcc::XretExecute+0x8ce
00000000`0e3deb20 00000000`0074b4af sqlservr!CMsqlExecContext::ExecuteStmts+0x375
00000000`0e3dec30 00000000`0074ad6c sqlservr!CMsqlExecContext::FExecute+0x97e
00000000`0e3dedb0 00000000`0076cfa6 sqlservr!CSQLSource::Execute+0x7b5
00000000`0e3deee0 00000000`007965e2 sqlservr!process_request+0x64b
00000000`0e3df540 00000000`006eb450 sqlservr!process_commands+0x4e5
00000000`0e3df750 00000000`006eb116 sqlservr!SOS_Task::Param::Execute+0x12a
00000000`0e3df860 00000000`006eaf5b sqlservr!SOS_Scheduler::RunTask+0x96
00000000`0e3df8c0 00000000`008244fa sqlservr!SOS_Scheduler::ProcessTasks+0x128
00000000`0e3df930 00000000`008247dd sqlservr!SchedulerManager::WorkerEntryPoint+0x2d2
00000000`0e3dfa10 00000000`00c6c0cd sqlservr!SystemThread::RunWorker+0xcc
00000000`0e3dfa50 00000000`008253d2 sqlservr!SystemThreadDispatcher::ProcessWorker+0x2db
00000000`0e3dfb00 00000000`733037d7 sqlservr!SchedulerManager::ThreadEntryPoint+0x173
00000000`0e3dfba0 00000000`73303894 msvcr80!_callthreadstartex+0x17
00000000`0e3dfbd0 00000000`76cc652d msvcr80!_threadstartex+0x84
00000000`0e3dfc00 00000000`773bc521 kernel32!BaseThreadInitThunk+0xd
00000000`0e3dfc30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
  • Hotfix request has been filed to Port from Kilimanjaro RTM CU11 to Kilimanjaro SP1CU4
  • Exception Access Violation in CScaOp_Identifier::Decorate when running CheckTable against a Spatial index.
  • The problem occurs at a line of code in function CScaOp_Identifier::Decorate.
  • Here, we are trying to dereference the structure pLR (LookupResult) but that is set to NULL.
  • issue is fixed in following builds: 2008 SP2 CU7, 2008 SP3 CU3, 2008R2 RTM CU11, 2008R2 SP1 CU4
  • Hotfix request has been filed to Port from Kilimanjaro RTM CU11 to Kilimanjaro SP1CU4