> Emulating these mechanisms, particularly the temporal coding inherent in spike timing and synchrony, presents a significant challenge. Consequently, modern neural networks do not rely on temporal dynamics to perform compute, but rather prioritize simplicity and computational efficiency.
Simulating a proper time domain is a very difficult thing to do with practical hardware. It's not that we can't do it - it's that all this timing magic requires additional hyperparameter dimensions that need to be searched over. Finding a set of valid parameters when the space is this vast seems very unlikely. You want to eliminate parameters, not introduce ones.
Also, computational substrates that are efficient to execute can be searched over much more quickly than those that are not. Anything where we need to model a spike that is delivered at a future time immediately chops a few orders of magnitude off the top because you have to keep things like priority queue structures around to serialize events.
Unless hard real time interaction is an actual design goal, I don't know if chasing this rabbit is worth it on the engineering/product side.
The elegance of STDP and how it could enable online, unsupervised learning is still highly alluring to me. I just don't see a path with silicon right now or on the horizon. Purpose built hardware could work but is like taking a really big leap of faith by setting some of the hyperparameters to const in code. The chances of getting this right before running out of money seem low to me.
Hm suppose for argument sake that feeding a batch of data through some moderately large FF architectures takes on the order of 100ms (I realise this depends on a lot parameters - but this seems reasonable for many tasks / networks).
Now suppose instead you have an CTM that allocates 10ms on the standard FF axes, and then multiplies it out by 10 internal “ticks” / recurrent steps?
The exact numbers are contrived, but my point is : couldn’t we conceivably search over that second arch just as easily?
It just boils down to whether the inductive bias of building in some explicit time axis is actually worthwhile, right ?
Simulating a proper time domain is a very difficult thing to do with practical hardware. It's not that we can't do it - it's that all this timing magic requires additional hyperparameter dimensions that need to be searched over. Finding a set of valid parameters when the space is this vast seems very unlikely. You want to eliminate parameters, not introduce ones.
Also, computational substrates that are efficient to execute can be searched over much more quickly than those that are not. Anything where we need to model a spike that is delivered at a future time immediately chops a few orders of magnitude off the top because you have to keep things like priority queue structures around to serialize events.
Unless hard real time interaction is an actual design goal, I don't know if chasing this rabbit is worth it on the engineering/product side.
The elegance of STDP and how it could enable online, unsupervised learning is still highly alluring to me. I just don't see a path with silicon right now or on the horizon. Purpose built hardware could work but is like taking a really big leap of faith by setting some of the hyperparameters to const in code. The chances of getting this right before running out of money seem low to me.