To quote the IEBlog:
“Browser Mode” affects the user agent
string, version vector used when
evaluating conditional comments, and
the rendering mode.
It's detailed a bit more on http://msdn.microsoft.com/en-us/library/dd565624(VS.85).aspx.
As you can clearly see, you wouldn't be able to affect all of these things anyway: by the time you tell the browser to act like IE7, it's already acting as IE8.
Perhaps the real question is: Why does it matter that much to you what the browser mode is? The document mode is what you should be most concerned with - everything the browser mode changes as far as the rendering is concerned relate to stuff that is excluded/included but version checking, and users aren't going to look in the developer tools anyway, so they won't care.
Instead of wasting a lot of time getting it to look like pure compatibility mode in the developer tools, you should rather go and make sure that user agent string checking and conditional comments make it so that IE7 and IE8 get the same material to work with, and then leave the EmulateIE7 in.
EDIT:
The problem is your version checks, and as I promised below, I'm going to tell you where the problem is.
If you use the developer tools to debug the menu placement script, you can dig down and see that the execution path for get_x_position differs when the browser reports itself as IE7 or IE8: is_ie5up is set to true for IE7 mode, and false for IE8 mode. This results in very different values being returned.
At this point, we must go back to where this variable is set:
var is_ie5up = (is_ie6up || (is_ie && !is_ie3 && !is_ie4));
As you can see, this depends on the value of is_ie6up, so let's have a look at the surrounding code...
var is_ie8up = (is_ie8 || is_ie9up);
var is_ie7up = (is_ie7 || is_ie8up);
var is_ie7up = (is_ie7);
var is_ie6up = (is_ie6 || is_ie7);
var is_ie5up = (is_ie6up || (is_ie && !is_ie3 && !is_ie4));
var is_ie5_5up = (is_ie6up || (is_ie && !is_ie3 && !is_ie4 && !is_ie5));
...do you spot the flaw (Hint: compare lines 2 and 4 of that snippet)?
That's right: is_ie6up is not set to true unless the browser is exactly IE6 or IE7. The proper line should of course read
var is_ie6up = (is_ie6 || is_ie7up);
...but wait. That's no good either, because line 3 of the snippet changes is_ie7up to only be true if the browser is exactly IE7! So, you need to delete the overwriting of is_ie7up, and fix the setting of is_ie6up.
My guess is that you have the EXACT same problem on the other site: you've messed up the browser checks in much the same way.
The issue has to do with the styles that you are using which IE8 layout engine has trouble interpreting. Unfortunately there is no comprehensive list of style combinations what will trigger this behavior. You have the ability to turn off the automatic recovery on your own machine by going to 'Tools->Internet Options-> Advanced Tab -> Browsing -> and uncheck 'Automatically recovery from layout errors ..."
This is of course only applied to your machine and is of no benefit to your customers if the layout error is triggered. I do not have an answer to your specific issue, but I'd suggest validating your css against the W3C CSS validator:
http://jigsaw.w3.org/css-validator/
Any try removing complex styles one at a time until you find the combination that is triggering the error for your site.
Best Answer
It would seem that IE8, Safari, Firefox, et al will tolerate certain JavaScript syntax errors. IE7 and IE6 (and IE8 in 'compatibility view') will not, and they will also not throw a parse error or any other kind of clue.
Pasting my code into http://www.jslint.com/ revealed a couple of syntax errors that weren't affecting the code's operation in other browsers. So boo on me.