The driver 'optimizations' in GPUs tend to boil down to hand written replacements for unoptimal/broken shaders and API call sequences on a per application basis.
I expect a network card to not 'interpret' my traffic in a similar way.
As someone who spent a ton of time in this space(working with most mobile GPU vendors from Android E to L) you're talking about things just at the surface.
The way the drivers + backing architecture are built contains tons of IP and things that they would very much not want their competitors seeing.
Our team built our reputation and relationship on keeping that data separate and confidential despite working with just about every vendor out there.
Adding the double blind optimizations leading to absurdly complex drivers that tries to guess how user code tries to guess driver logic... maybe vulkan will help open source GPU drivers in the end.
GPU drivers are very bug ridden and 3D apps can easily cause OS crashes. Apps are tested so they don't trigger those bugs. Witness all the complaint comments on webgl posts about machine crashes - and that's with a thick driver bug workaround layer in browsers.