C++ – VS2005 C++ broken vtables

cvisual-studio-2005vtable

I'm currently working on a quite big (and old, sigh) code base, recently upgraded to VS2005 (SP1). Me and my team are changing/updating/replacing modules in this code as we go but we have occasionally been running into problems where the vtables seems broken. I am no expert on vtables but these sure seems to be broken. The errors manifests itself with this error:

Run-Time Check Failure #0 – The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

Of course there can be plenty of other reasons for this error but when debugging (Debug build) I can actually verify that the vtables for the object I want to operate on look strange:

The stack and heap that reference each vtable looks fine and the pointers to the vtables match perfectly to the map file. This indicates to me that this is not a memory overwriting bug or similar, since then it would affect the stack and heap rather than where the vtables are stored. (They are stored in a read only area right?) Anyway, all seems good so far. But when looking at the memory of the vtable I find that all values, if I interpret them as pointers, although they are in the same range (Eg. 0x00f203db 0x00f0f9be 0x00ecdda7 0x00f171e1) does not match any entry in the map file and many of them are not even aligned to 4 bytes. I don't know all the details of how VS2005 builds the vtables, but this looks wrong to me. If this is correct behavior, perhaps somebody can explain this to me?

I guess my question boils down to what can cause this behavior? Is there any know bugs in the linker when having too complex class hierarchies for example? Has anybody seen anything similar before? Currently we are able to get around our crashes by moving functions from the affected class to inline (scary stuff!) but clearly this is not a feasible long term solution.

Thanks for any insight!

Update: I've been asked for more details about the project and of course I will supply this. First however, the question is not entirely related to the ESP value not being saved error. What I am most interested in is why I see the strange values in the vtable. That said, here is some additional info: The solution relies on several external and internal projects but these have not been changed in a long time, all uses the same calling convention. The code where it seems to break is all within the one pretty standard C++ "main" project of the solution. All code is built with the same compiler. The solution also doesn't use any dlls but links with plenty of static libraries:

SHFolder.lib, python25.lib, dxguid.lib, d3d9.lib, d3dx9.lib, dinput8.lib, ddraw.lib, dxerr9.lib, ws2_32.lib, mss32.lib, Winmm.lib, vtuneapi.lib, vttriggers.lib, DbgHelp.lib, kernel32.lib, user32.lib, gdi32.lib, winspool.lib, comdlg32.lib, advapi32.lib, shell32.lib, ole32.lib, oleaut32.lib, uuid.lib, odbc32.lib, odbccp32.lib

Best Answer

I found the problem. Silly really but the class hierarchy that caused the problem had a virtual function called GetObject which conflicted with the windows #define with the same name. The header files included these windows header files in different order, which confused the linker. So, in fact the problem was corrupted vtables, but I didn't expect this to be reason! Well you learn something every day...

However, big thanks to all that replied!

Related Topic