It's not really possible to write wrappers which can inspect system calls, allow them to be blocked or modified, AND be efficient. Therefore they're not useful for security purposes because of the inherent race conditions.
I ask because when reinstalling a Windows/PC last week, and I needed to copy some missing drivers (network/video) onto a USB flash drive from a working computer. I happened to notice there was a Ubuntu installer on the USB stick, and was very tempted to install Ubuntu!
Summary of the article: AV software hook into Windows system calls. When user-mode calls a kernel function, the AV hook handler first checks that the arguments comply with the security policy. If they do, it passes the arguments to Windows in order to execute the call. However, another user-mode thread can change the data between the time of the check and the execution. The AV will see valid data, but Windows itself will act upon the malicious data.
This is a vulnerability in AV software, not in Windows. If the attack succeeds the system remains just as secure as other operating systems which don't have AV.
How can the attack be prevented? I guess in the same way that the kernel itself prevents such attacks (after all, the kernel has its own checks, which presumably can't be bypassed). There's probably a way to make memory pages read-only for the duration of the system call, so the data can't be changed.
Since hardware virtualization was introduced, the whole system can be wrapped by an attacker. Therefore, well coded malware could use the SSDT vulnerability to virtualize the currently running OS and not even have to remove AV protection afterwards. Tech keeps getting more and more flakey :-/
The first thing I would try is a bit hacky but might work. I'd take a copy of the data structures you are analysing when looking for malware and then copy it back to the original place just before you call the normal windows function.
It would minimise the amount of time the malware had to get the data in.
What you really need is to move the data into kernel space somehow. That would need a tweak to windows though.
The difficulty for the attacker comes in creating the initial attacker and faker threads. Possibly AV can hook the creation of threads and examine new threads for this potential behaviour (this might kill heavily threaded apps though). Then the virus make would have to attack two separate threads.
Details: http://www.lightbluetouchpaper.org/2007/08/06/usenix-woot07-... (Exploiting Concurrency Vulnerabilities in System Call Wrappers)
It's not really possible to write wrappers which can inspect system calls, allow them to be blocked or modified, AND be efficient. Therefore they're not useful for security purposes because of the inherent race conditions.