Weird MSC 8.0 error: "The value of ESP was not properly saved across a function call..."

Question

We recently attempted to break apart some of our Visual Studio projects into libraries, and everything seemed to compile and build fine in a test project with one of the library projects as a dependency. However, attempting to run the application gave us the following nasty run-time error message:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function pointer declared with a different calling convention.

We have never even specified calling conventions (__cdecl etc.) for our functions, leaving all the compiler switches on the default. I checked and the project settings are consistent for calling convention across the library and test projects.

Update: One of our devs changed the "Basic Runtime Checks" project setting from "Both (/RTC1, equiv. to /RTCsu)" to "Default" and the run-time vanished, leaving the program running apparently correctly. I do not trust this at all. Was this a proper solution, or a dangerous hack?

Solution 1

This debug error means that the stack pointer register is not returned to its original value after the function call, i.e. that the number of pushes before the function call were not followed by the equal number of pops after the call.

There are 2 reasons for this that I know (both with dynamically loaded libraries). #1 is what VC++ is describing in the error message, but I don't think this is the most often cause of the error (see #2).

1) Mismatched calling conventions:

The caller and the callee do not have a proper agreement on who is going to do what. For example, if you're calling a DLL function that is _stdcall, but you for some reason have it declared as a _cdecl (default in VC++) in your call. This would happen a lot if you're using different languages in different modules etc.

You would have to inspect the declaration of the offending function, and make sure it is not declared twice, and differently.

2) Mismatched types:

The caller and the callee are not compiled with the same types. For example, a common header defines the types in the API and has recently changed, and one module was recompiled, but the other was not--i.e. some types may have a different size in the caller and in the callee.

In that case, the caller pushes the arguments of one size, but the callee (if you're using _stdcall where the callee cleans the stack) pops the different size. The ESP is not, thus, returned to the correct value.

(Of course, these arguments, and others below them, would seem garbled in the called function, but sometimes you can survive that without a visible crash.)

If you have access to all the code, simply recompile it.

Solution 2

I read this in other forum

I was having the same problem, but I just FIXED it. I was getting the same error from the following code:

HMODULE hPowerFunctions = LoadLibrary("Powrprof.dll");
typedef bool (*tSetSuspendStateSig)(BOOL, BOOL, BOOL);

tSetSuspendState SetSuspendState = (tSuspendStateSig)GetProcAddress(hPowerfunctions, "SetSuspendState");

result = SetSuspendState(false, false, false); <---- This line was where the error popped up. 

After some investigation, I changed one of the lines to:

typedef bool (WINAPI*tSetSuspendStateSig)(BOOL, BOOL, BOOL);

which solved the problem. If you take a look in the header file where SetSuspendState is found (powrprof.h, part of the SDK), you will see the function prototype is defined as:

BOOLEAN WINAPI SetSuspendState(BOOLEAN, BOOLEAN, BOOLEAN);

So you guys are having a similar problem. When you are calling a given function from a .dll, its signature is probably off. (In my case it was the missing WINAPI keyword).

Hope that helps any future people! :-)

Cheers.

Solution 3

Silencing the check is not the right solution. You have to figure out what is messed up with your calling conventions.

There are quite a few ways to change the calling convetion of a function without explicitly specifying it. extern "C" will do it, STDMETHODIMP/IFACEMETHODIMP will also do it, other macros might do it as well.

I believe if run your program under WinDBG (http://www.microsoft.com/whdc/devtools/debugging/default.mspx), the runtime should break at the point where you hit that problem. You can look at the call stack and figure out which function has the problem and then look at its definition and the declaration that the caller uses.

Solution 4

I saw this error when the code tried to call a function on an object that was not of the expected type.

So, class hierarchy: Parent with children: Child1 and Child2

Child1* pMyChild = 0;
...
pMyChild = pSomeClass->GetTheObj();// This call actually returned a Child2 object
pMyChild->SomeFunction();          // "...value of ESP..." error occurs here

Solution 5

I was getting similar error for AutoIt APIs which i was calling from VC++ program.

    typedef long (*AU3_RunFn)(LPCWSTR, LPCWSTR);

However, when I changed the declaration which includes WINAPI, as suggested earlier in the thread, problem vanished.

Code without any error looks like:

typedef long (WINAPI *AU3_RunFn)(LPCWSTR, LPCWSTR);

AU3_RunFn _AU3_RunFn;
HINSTANCE hInstLibrary = LoadLibrary("AutoItX3.dll");
if (hInstLibrary)
{
  _AU3_RunFn = (AU3_RunFn)GetProcAddress(hInstLibrary, "AU3_WinActivate");
  if (_AU3_RunFn)
     _AU3_RunFn(L"Untitled - Notepad",L"");
  FreeLibrary(hInstLibrary);
}

View more solutions

Comments

[jyz]

+1 good explanation, would be perfect if you put some code examples to guide him

[Sushi271]

I had the same exception, but none of above were the cases. I was battling it for several hours until I finally narrowed down the problem to a function, that has an argument being pointer to member function of other class. Calling this function caused stack corruption. The solution to this kind of problem can be found here: stackoverflow.com/questions/8676879/…

[Froyo_np]

possibility 3 - mismatched names when getting a function pointer (perhaps via a call to getProcAddress("theWrongFuntionName"). This is what I did! What happened: I bound a pointer to the named function to a function-pointer prototype (via a typedef). Everything looks right - no compile errors, but you're calling the wrong function at runtime. I guess you have to be unlucky enough to mistype a name that actually exists in your dll, but isn't the one you want, otherwise you'll be saved and get null back from getProcAddress().

[IInspectable]

"In my case it was the missing WINAPI keyword" - That's not a keyword. It's a preprocessor symbol, that expands to the calling convention. A question on mismatched calling conventions should at least contain the term "calling convention".

[IInspectable]

Jumping to conclusions, eh? Had it occurred to you, that maybe, just maybe, there is a bug in your code, that is common enough for a library update to implement a fix? Without a minimal reproducible example and thorough analysis of the generated object code, this is just speculation.

[IInspectable]

This code exhibits undefined behavior. There are at least 3 fatal bugs: 1 Accessing an uninitialized array (p). 2 writing past the end of an array (strcat). 3 Returning the address of a local (return p). There are numerous ways to trigger this run-time check. Posting random buggy code that does (sometimes) is not at all helpful, sorry.

[rtischer8277]

@IInspectable The notion that a respectable compiler vendor today would change their code to repair a compiler user’s misbehaving code is without merit. On the other hand, if you can find a flaw or weakness in my natural 3-PC experiment, I would like to know.

[IInspectable]

"if you can find a flaw or weakness in my natural 3-PC experiment, I would like to know" - Uhm... easy. Undefined behavior, in your code, that happens to manifest itself in a reproducible way, with reproducible observable behavior. That would be one obvious explanation, if you don't buy the notion of a compiler vendor changing their support libraries, to address common bugs. None of that is very helpful, though, if we cannot see your minimal reproducible example, that demonstrates the issue. Something like this would do.

[marshal craft]

this was exactly the issue i just was having with compound type or whatever its actual name is. i didn't know where to put WINAPI so just left it out when explicitly loading dll to get a D3D12GetDebugInterface(). I had messed around with arguments but it was exactly as you had said with the winapi.

[Robert]

Agree with this poster. Always try fully rebuilding your project when you get weird unexpected errors from code that you aren't even working on. This kind of thing happens a lot when you build large projects with incremental linking and all of the other utilities of VS. Sometimes it messes up the linking and you get random errors like this one.

[Peter Cordes]

Of course, you need to switch to 64-bit time_t sometime well before 2038 (en.wikipedia.org/wiki/Year_2038_problem) or whenever Windows 32-bit time wraps around, or sooner if your code deals with dates in the future using time_t. And of course, existing binaries will potentially get used for years into the future, so the "well before" is important.

[oli_arborum]

@PeterCordes For sure 64-bit time_t should be used only. However, that "should" doesn't help you when having a DLL using function signatures with 32-bit time_t.

[Peter Cordes]

I commented as a reminder that this solution will stop being viable in a few years, depending on product lifecycles. At some point you have to bite the bullet and discard unmaintainable legacy binary-only crap, or rebuild it from source if you have it. It's useful to know how to make your new software hobble itself to be binary-compatible with old crap (so I upvoted this), but it's worth reminding people that it's not a permanent long-term solution.