The most extreme I can think of was one reported by Raymond Chen on his "The Old New Thing" blog which I can't find the specific link for, but is discussed in an article by Joel Spolsky:
"I first heard about this from one of the developers of the hit game SimCity, who told me that there was a critical bug in his application: it used memory right after freeing it, a major no-no that happened to work OK on DOS but would not work under Windows where memory that is freed is likely to be snatched up by another running application right away. The testers on the Windows team were going through various popular applications, testing them to make sure they worked OK, but SimCity kept crashing. They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it."
You picked a particularly poor example. The application compatibility layer does not incredibly hamper the OS whatsoever. First of all the ACL is only active when known executables that have problems with the newer versions are running. As an educated guess it would not take more than a millisecond to ID the .exe. Secondly what the ACL does is fairly benign - what it amounts to in simple terms is that it simply uses a different code path for a very narrow set of functions. The cost associated with loading an 'extra library' is so marginal that you would find it hard to measure.
http://www.joelonsoftware.com/articles/APIWar.html
To quote:
"I first heard about this from one of the developers of the hit game SimCity, who told me that there was a critical bug in his application: it used memory right after freeing it, a major no-no that happened to work OK on DOS but would not work under Windows where memory that is freed is likely to be snatched up by another running application right away. The testers on the Windows team were going through various popular applications, testing them to make sure they worked OK, but SimCity kept crashing. They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it."