Lecture 4: Programming Spiking Neural Nets
How can we program a SNN and why? I.e. normally we largely train a neural network and let it figure shit out on its own. Why do more explicit programming?
- SNN model of computation
- LIF-based model, notation & conventions
- Turing Tumble revisited
- Resources: time, space, energy.
- Hybrid computations: NMC chip as a co-processor in addition to traditional CPU
- Case study: network flow problem from theory to Loihi implementation.
NMC architectures: what are they good for?
Suitable for:
- Event-driven sensors/actuators (neuromorphic robotics)
- Energy-critical applications that allow for less precision
Not the best for:
- "Deep" classification and pattern recognition
- Applications where precision is more important than energy usage
Is "machine learning" the only application?
A new way of thinking
In analogy with general purpose GPU, we think there may be types of problems may be solved more energy efficiently on neuromorphic chips instead of in an traditional approach.
CPU's are good for serial computations. In order to leverage the potential of GPUs, we need to change our mindset about data structures and think in terms of data streams.
Similarly we need to think different about neuromorphic chips to see how some problems may be solved more energy-efficiently.
SNN as the computational framework for neuromorphic architectures
Q: Why are spiking neural networks are useful model for thinking about computation in neuromorphic chips?
The SNN model can serve as computational model for neuromorphic architectures, because SNNs have the following key neuromorphic aspects:
- Co-located memory & computation
- Spiking behavior -> energy efficiency
- Stochastic sampling or deterministic spikes
Each computational architecture makes some assumptions that abstracts away from some details.
Turing Machines abstract away from details like caching/paging/accessing external memory. In SNNs we assume that only spikes cost energy, and that sub-threshold potentials cost nothing. This is obviously not something that is true, but if the model abstraction is good, this should not fundamentally affect results. In the Turing Machine framework, all kinds of theoretical work is done, e.g. showing that using more tapes does not lead to a higher-than-polynomial efficiency increase.
We are similarly currently interested in finding out whether SNNs as a computational model artificially restrict the kind of problems they can solve. For example, we use the convention that thresholds cannot be negative, and if there's a negative potential we just set it to 0. How does such an assumption play out in actual chips, and does this assumption have theoretical consequences? We should also take these practicalities into account.
Neuronal model: basically a LIP model
Discrete leaky integrate-and-fire model.
- Specific input and readout neurons to inject/extract information
- In nengo we had nodes and probes
- Stochasticity or determinism
- Discrete time steps
- Connections can also have additional delays
- Default delay of d=1
- Leakage: every timestep some fraction of the potential "leaks" away.
Neurons can fire either deterministically or stochastically. For the latter, f could be a sigmoid or any other function that returns probabilities.
Notation
See slide for notation for:
- Circles for regular neuron
- Diamonds for programmed neuron that respond either to a specific singleton or spike train at a particular timestep.
- Octagonal: read-out neurons.
Default values:
- d=1; (so no delay); can be real number
- w=1;
- m=1; in [0,1]
- R=0; [0, inf]
- T=1; in [0,inf]
First example is a way of encoding input to a network. This is not biologically plausible but is a convenient way to encode some input that the system assumes.
Second is mod-K clock neuron. But how can we silence it after six clock pulses? Idea: have an infinity neuron start firing after d=6K+1 with pulses with weight -1.
Exercises
See lecture slides for example networks.
Ex. 1: Array inclusion
Note that the c neurons (comparator?) have no memory.
Search array for inclusion of number i (timepoint) (?)
Take A=[3,2,4], i={1,2,5}. Does network output fire for spiketrain i={1,2,5}
Notice d is determined in terms of A[0:3].
For i=1, none of the spikes of the first neuron arrive at the same time, and the threshold of 2 is not met in the "acceptor" neuron.
It does fire for i=2.
What about a stopping criterion? If we take into account all minimal delays, we know that at max(A)+3=4+3 the acceptor neuron cannot spike anymore.
Ex. 2: Shortest path
Encode path lengths as delays. Then when the output fires, we know the length of the shortest path.
Note that the solution barely takes any computation. The network structure automatically takes care of the computation.
Of course, after firing for the first time, output node will fire again multiple times due to the other paths coming in later.
Challenge: can we make sure each neuron fires exactly once?
General solution: set m=1; T=1+n for all neurons; and R=0 for all neurons, where n is the amount of neurons. Then have a programmed neuron [1] at t=1, that fires once with weight n to all other neurons. Now all other neurons are one below their threshold, so the first time the network will function just like it did before. But after a neuron fires, its threshold will reset to 0. Even if all other neurons would fire into it, it cannot possibly fire again because there are n neurons and the threshold is n+1.
Next challenge: we do not yet know the shortest path itself!
Idea: check how many nodes are in the network. In this case 6. How many bits do we need to encode their "id"? log(6) and round up => 3 neurons.
Solution: tell post-synaptic neuron who caused the first spike by having auxiliary neurons encode in binary the parents. We assume we have access to the computed result.
000 = 0 001 = 1 010 = 2 011 = 3 100 = 4 101 = 5 110 = 6 (111 = 7)
Now, each node A0 will have three additional nodes A1 A2 A3 that register its binary number. E.g. the binary number of the first node A0 is 001. This can be encoded by the following connections:
- A0 -> A1 (w=0)
- A0 -> A2 (w=0)
- A0 -> A3 (w=1)
When we can now read out those neurons, we will see each time which neuron fired to excite the next one. This will provide one unique path because we already ensured that each neuron fires only once.
Is this efficient? We need quite a bit of readout neurons. Is the reading out realistic? What do we need to do to recover information from a chip?
In any case, the result is encoded in the system in a way. The challenge is how to extract that information in a meaningful way, for example when we use this circuit as a coprocessor?
Ex 3: Adding two numbers mod K
Part of assignment.
- Represent numbers = data
- Realize addition = computation
- Interpret outcome
-
For concatenation, outcome must be a number representation again.
-
A is encoded with d=2 relative to clock.
- B with d=3
The clock is a resetting signal here, with delay K.
See slides and recording for in depth discussion.