Q&A Neuromorphic Computing course
Johan did a short overview of lectures.
Questions has to be open-ended now due to the covid situation.
- What would be the best approach to problem X?
- What would be a reason to use a particular learning method e.g. STDP instead of training an ANN and converting it?
- What would be a reason why backpropagation is still hard to ...
Will be pro/con type of questions because they require motivation:
- What would be a benefit of Spinnaker architecture vs. dedicated hardware
- Or analog vs digital etc.
- Explain differences between learning paradigms
- Question about comparing encoding;
- Giving examples
- Why is another complexity theory needed for neuromorphic architectures?
Some more remarks about the exam:
- No drawing needed for the exam.
- No code writing is needed
- If there is an issue send a PM
STDP: rule that allows particular neurons sensitive to particular inputs; tries to learn to be sensitive to particular inputs, process them quickly, and ignore others.
Shortest path problem: Submit information via binary code; does not only send a spike, but have auxilliary low x neurons that send the binary signature along with the spike; so you can trace back what the parent was that instigated the spike.
My questions after reviewing assignments
Nengo assignment 2.1: I noticed that squaring a negative number is less accurate than squaring a positive number.
Why is that?
2.2 What do we see exactly when applying the Oja rule when trying to learn the sum? Why is Oja not suitable for learning addition? What is it suitable for instead?
For which type of applications are different learning rules suitable?
My feedback for a3:
I do think this assignment was interesting, but I started disliking it mainly due to the large amount of time I sunk into it. It would be very helpful for me to discuss this assignment in class in more detail, because I often feel I’m missing systematic way of approaching these types of problems. Perhaps there is none, but right now, it mostly feels like “puzzling” to me and that I’m missing some of the required “thinking tools”.
So the question is: what's a systematic way to approach a problem like this? It feels like puzzling now.
A4: Since we didn't get the comparison with the hardware chip, what was the intended take-away here in performance between software and chip?
As soon as I depart from known examples, I really struggled to find the information I need in the nengo docs to solve issues when I encounter them. This makes me slightly pessimistic about how we can solve and debug issues when we encounter them in the project (assuming we use Nengo).
I'm lacking "Nengo best practices", e.g. storing information in a single Ensemble using multiple connections. There must be plenty of similar "tricks" that I'm unaware of.
Practice questions
Even snel gemaakt, niet goed uitgeschreven.
- Temporal versus rate coding [15 points, suggested about 100 words in total]: a) Explain the difference between rate and temporal coding. When you answer this question, make sure to state the assumptions behind each view [10 points]. b) Give arguments pro and con either of the two [5 points].
Rate: use the frequency in a time window to encode a value to encode a value. Assumption here is that the exact timing of spikes within that window is not very meaningful; at least this is not used in this encoding. Assumption: rate of firing is proportional to strength of the input. Pro: Robust to noise. Con: Encoding "high values" with high firing rates requires a lot of spikes and is not energy efficient.
Temporal: the specific spiking pattern, i.e. the temporal differences between individual spikes, is meaningful and can be used to encode values. For example, a spike may encode a value through the delay with respect to a clock signal. Pro: an individual spike carriers a lot of information Con: As a result, small differences in the timings of spikes and/or measuring errors can radically change/affect the encoded information.
-
Describe three principled differences between Von Neumann and neuromorphic architectures [10 points, suggested 50-100 words in total].
-
computation and memory separated vs. computation and memory co-located
- serial vs massively parallel
- boolean circuits vs. discrete spikes, synaptic strenghts, membrane voltages, timings
-
energy-intensive (volatile memory) vs. energy lean (non-volatile)
-
Name the three currently dominant learning paradigms for deep spiking networks and explain them [10 points, suggested 50-100 words in total].
-
ANN-to-SNN conversion; train ANN (offline); convert to SNN for online inference.
- Backprop in SNN; e.g. try to use the membrane potential as a differntiable signal.
-
Attempts to scale up while still using local STDP learning; e.g. integrate it in a CNN and use STDP "horizontally" i.e. layer-wise.
-
Give an elementary difference between configuring a network and programming a spiking neural network. Assume you have the following two problems: 1) Approximate an NP-hard optimization problem (such as Sudoku) and 2) Finding a shortest path in a graph. For which problem would you use a configuration approach? For which a programming approach?
Programming view: entirely case-specific; when we have computed the result, the resources for that result cannot be amortized. We do not need to recompute the same result. But we cannot re-use the same network either because another problem will require an entirely new network definition and topology. In the configuration view, part of the used resources can be amortized because the defined network can be re-used in many configurations.
Sudoku: configure. The core of the problem are the set of rules and constraints that are supposed to hold for a sudo. When solving sudoku's this problem stays the same, but each time the sudoku "configuration" is different.
Shortest path: programming. The shortest-path problem doesn't have such a reusable core and is a computation that we do once for a specific problem. Because we do not need generality, it makes sense to take the programming approach.
- Explain in general terms how, in the Spinnaker and Loihi neuromorphic architectures, spikes are transmitted from a pre-synaptic to a post-synaptic neuron.
Explain AER; representing spikes with address and timing (implicit).