Are you sure this really negates the problem? (doesn't that essentially make it equivalent to a downsampled input?)
I think a more important question than trying to come up with a work around is to consider whether those cases really matter. If the probability of those "adversarial noises" is low enough, who cares? Those cases becomes a curiosity. We have noisy systems that may fail catastrophically operating everywhere with some low probability noise vector, but we manage those probabilities.
Unless of course it is an adversarial application and the network is made public; but then I think there are other also simple ways of defeating the attack, e.g. some random smoothing (so that the smoothing isn't also attacked), adding some more noise, etc.
There are the practical flaws like "okay, how reliable can this be if it decides that this picture that is clearly a car is instead a camel." But you're right, we can study those empirically and say whether or not they're actually a practical problem.
But I think that the more interesting question is, what the hell is actually going on here? What is the neural network actually detecting, that you can change what it is detecting with such innocuous-to-human-eyes changes? Is there indeed some kind of pattern that is genuinely characteristic of camels or whatever that is invisible to us, but "visible" to the neural net?
The temptation is to say it's just overtraining, picking up some noise that happens to be the same in various training set photos but is genuinely meaningless in the real world. But the fact that multiple different algorithms have the same or similar adversarial examples, and the fact that deep learning neural networks do appear to perform reasonably well in the real world, argues against that.
So if we provisionally think that it's not overtraining, what is it? Could there be a feature of camels that we can't see? Could that tell us something about camels, or vision, or something?
And vice-versa, what if our goal isn't a narrow task like "identify the existence of camels," but instead we're trying to get a computer to identify similar features in camels as we are (perhaps because the goal is not simply to identify camels, but to identify camels and then reason about the camel in some way). How do we get a deep learning neural net to not focus on some invisible feature, and instead learn to identify a camel based on features that humans DO perceive?
Is not about vision, is about emerging patters that arise from noise. If just one piece (deemed important) piece of the pattern differs from the expectation, then the classification fails. Therefore surface blurring does fix the issue, because the number of patterns is reduced (when applied in both the source samples and the user input).
Here is a diff of the borders of the pictures (instead of pixels, which is less useful), the one in the left with surface blur and the one in the right without: http://i.imgur.com/3bihGVA.png (The parts marked as completely different are the white ones)
Yeah, dude, I know. But why is that piece deemed important? Is it truly important? If it is not truly important, then the algorithm is overtrained -- that is, it has found "patterns" that exist in the training set but only by random chance, and which would not exist in a sample set.
But as I mentioned, there seem like there are reasonable reasons to suspect that this is not just overtraining.
What does adding surface blur really fix? Why could you not add similarly subtle network-breaking noise to a blurred image? If you think of blurring as downsampling, it is obvious that you also need downsampled noise to trick the NN.
The noise is not random, one can see exactly where the artifacts were added (the white parts of the image). And for this solution to work the down sampling must happen inside the program, not in the file system, therefore all network-braking noise would have been already discarded.
And for this solution to work the down sampling must happen inside the program, not in the file system, therefore all network-braking noise would have been already discarded.
I see, maybe I just didn't get your point the first time.
Anyway, what I was trying to say is that if you do view the downsampling as part of the network/program, you could apply the optimization procedure mentioned in the paper to a network that blurs its input. I assume that this would then generate network-breaking patterns that are imperceptible to the human eye, in the same way as happens in the paper.
I think a more important question than trying to come up with a work around is to consider whether those cases really matter. If the probability of those "adversarial noises" is low enough, who cares? Those cases becomes a curiosity. We have noisy systems that may fail catastrophically operating everywhere with some low probability noise vector, but we manage those probabilities.
Unless of course it is an adversarial application and the network is made public; but then I think there are other also simple ways of defeating the attack, e.g. some random smoothing (so that the smoothing isn't also attacked), adding some more noise, etc.