Lecture 6: Neuromorphic Architectures
Chip design to software stack.
For many new papers, new chips were developed and tested:
- IBM TrueNorth
- Akida
- Neurogrid
- etc. see slides
Neuromorphic Hardware Taxonomy
Learning goal: be able to describe the general categories and varieties of architectures.
We have digital, analog and mixed systems.
Analog: use physical properties of electronic devices as part of the computation
Digital: rely on Boolean logic-based gates.
Mixed: e.g. may have a analog computation in a local core, and then digital communication between cores. Many variants possible.
Two types of digital chips:
- Field Programmable Gate Arrays (FPGAs)
- Application Specific Integrated Circuits (ASIC)
Digital: FPGAs
They are integrated circuits that can be configured after manufacturing.
This offers flexibility and programmability to prototype chips.
So "stop-gap" solution on the way to custom chip implementation.
So in NMC it is used to prototype and test performance gains when moving from simulation to hardware, because making a dedicated chip is obviously expensive.
SpiNNaker
The first large scale digital system; real-time.
Note that the computations are digital, but that the communication between cores is based on spikes.
- Massively parallel; aimed at neuroscience, robotics and computer science
- Fully digital
- Real-time
- 1 million ARM9 Cores
- About 1000 point neurons per core
- Up to 1000 inputs per neuron
- 10 Million connections per second (assuming 10 Hz firing rate)
- Communication between cores uses spike-based messaging.
The main innovation of SpiNNaker is that this spike-based messaging communication scheme allows an incredibly scale-up in the amount of cores that can be used in parallel. This allows for large scale neural network simulations (in theory up to a billion point neurons = 1 mil x 1000 x 1000).
The actual performance is lower due to software limitations. In practice, application simulate perhaps 1 million neurons. (Not sure I got this got this right)
Implementing more complex networks and/or with plasticity on synapses further drops performance.
Digital: ASIC
Full custom chips require longer design and fabrication cost; with the benefit of better performance because they are optimized with a particular application in mind.
TrueNorth is a good and common example. One of the largest neuromorphic chips.
The downside is that they are not as flexible as FPGAs. For example, you are restricted in the type of network you can implement on it.
TrueNorth
TrueNorth is super efficient but less flexible, because it uses a fixed SNN model with limited options for adjusting connectivity. So TrueNorth is an example of ASIC.
- Fully custom chip
- One of the largest neuromorphic chips
- 1 Million neurons
- 256 Million synapses
- 5.4 Billion Transistors
- Realtime
- 73 mW
- Deterministic spikes
- Fixing spiking network model
TrueNorth vs SpiNNaker
They are both fully digital systems that work in real-time, but they show the two extreme different approaches one can take with digital hardware.
The general trade-off is: flexibility in neuron model and network topology vs. performance.
There is also a clear difference in energy:
- ASIC style TrueNorth consumes 25pJ (10^-12J) per connection. (pico Joule)
- FPGA style SpiNNaker consumes 10nJ (10^-9J) per connection (nano Joule)
Analog
Similar to digital hardware, we can separate analog chips into programmable and custom chip implementations.
Instead of FPGA (G for Gate; boolean logic) there are FPAA chips for analogy systems (Field Programmable Analog Arrays) that have programmable components for neurons and synapses.
Q: What's the main issue with analog chips?
A: Analog signals do not scale because they disintegrate and are very noisy. Additionally, there is more R&D on digital hardware.
Analog and NMC: Match made in heaven?
Analog circuits and neuromorphic systems are a good match, because both have these characteristics:
- Conservation of charge
- Amplification
- Thresholding
- Integration
So in theory, we could get these computational primitives for free if we use (fully) analog systems. Originally neuromorphic systems were also defined as analog systems (cf. Mead, 1990).
Issues with analog chips:
- Global asynchrony
- Noisy unreliable components
Neuromorphic systems are promising to deal with issues of analog chips; e.g. they are more robust to noise (cf. "sampling" point from few lectures back).
Scaling up is hard to to above mentioned issues; e.g. signals disintegrate over distance and time; noise. So existing models are mostly small prototypes.
Mixed
Idea is to take the benefits and natural fit of analog systems, but overcome their downsides with digital components. E.g. digital components can be used for programmability and for learning mechanisms.
In order of popularity:
- Use analog circuitry with a digital communication architecture.
- Mixed circuitry
- Analog circuitry, but store memory digitally; less noisy and more reliable
Neurogrid
- Neurogrid is essentially a mixed equivalent of TrueNorth.
- Simulate up to million neurons and 6 billion connections
- Operates in subthreshold mode: gates never turn on or off and this apparently saves energy.
- Primarily an analog chip with digital communication framework
BrainScaleS:
Where Neurogrid is more equivalent to TrueNorth, BrainScaleS is more equivalent to SpiNNaker.
- Wafer-scale with analog components
Wafer-scale integration, WSI for short, is a rarely used system of building very-large integrated circuit networks that use an entire silicon wafer to produce a single "super-chip". Combining large size and reduced packaging, WSI was expected to lead to dramatically reduced costs for some systems, notably massively parallel supercomputers. The name is taken from the term very-large-scale integration, the current state of the art when WSI was being developed.
What's a wafer?
In electronics, a wafer (also called a slice or substrate) is a thin slice of semiconductor, such as a crystalline silicon (c-Si), used for the fabrication of integrated circuits and, in photovoltaics, to manufacture solar cells. (Wikipedia)
- BrainScaleS runs in accelarated time (10.000x).
- The systems before, at least those were it was mentioned, run in real-time.
- 200k neurons per wafer
- In theory up to 20 million neurons simulated
- Local analog computation + digital communication.
What's the advantage of using local analog computations; i.e. offloading computation to a physical medium?
- This way you get some computation for free; you do not have to explicitly simulate every state variable of the neuron.
- So this is very useful when simulating more complex neurons with many paramters, such as Hodgkin-Huxley neurons with many ion channels.
- The more parameters you'd otherwise have to digitally simulate, the more potential benefit there is.
- Apparently physically determined variables can be "estimated at near light speed" (I don't follow this). Anyways, this allows the speedup to 10.000x real-time.
Accelerated architecture enables simulation varying from the millisecond timescales to years. So very suitable for models of learning and development; computational neuroscience.
Supporting Systems: Communication and Programming Tools
One of the problems with neurochips is that in practice a lot of these chips will be used, because there is no ecosystem of tools for neuromorphic computing.
We have plenty of hardware, but too little software.
Nengo and Loihi are major improvements.
We need:
- Communication frameworks
- Supporting softwares
AER Address Event Representation
We have in-chip and between-chip communication. We already saw that for scaling up people rely on a digital communication framework. (I'm assuming AER functions in this digital regime; but multiplexing is in principle possible on analog signals as well)
Inter-chip: Address Event Representation (AER)
Asynchronous communication protocol originally proposed as a means to communicate sparse neural events between chips.
In AER communication, each neuron has an unique address.
Challenge in chip design is that you have limited wires. If you want to simulate with massive connectivity like SNNs, we run into the problem that we have more connections that we can physically implement. So we end up with bottlenecks.
Brain computes in 10-100 Hz range; this is way too high. We want to compute in a slower regime, but still do computations similarly fast. So this is very useful for studying brain processes.
But there's a trick. The limited amount of wires can transmit information faster because this scheme uses digital communication over data busses. So we can bundle many connections in a single high-speed data bus. They can communicate in the order of 100s of Hz; so there's a large difference. This allows us to emulate parallel connections; they are in fact sequential, but they are so fast it appears that they arrive at the same time. This abuses the fact that the brain computes at a lower speed. This is called multiplexing. In this case, time-multiplexing.
In electronics, a multiplexer (or mux; spelled sometimes as multiplexor), also known as a data selector, is a device that selects between several analog or digital input signals and forwards it to a single output line.[1] A multiplexer of $2^n$ inputs has $n$ select lines, which are used to select which input line to send to the output.[2] Multiplexers are mainly used to increase the amount of data that can be sent over the network within a certain amount of time and bandwidth.[1] (Wikipedia)
So this is a form of simulated connectivity.
For a one-to-one connection topology, we can reduce the amount of required wires from N to about log N. (REVIEW; why log?) Can't find this remark anymore in the lecture.
Each spike is represented by:
- Location; explicit encoded of address
- In some limited cases you can only use the location of the neuron from which it originates
- Its timestamp: this is implicit
AER encodes characteristics of neuromorphic computing:
- Parallelity
- Asynchronicity
- Event-based
- Low communication load
Above it is very energy efficient; only few bits needed for encoding packages. Why?
Software
We need:
- Custom hardware synthesis tools
- Mapping tools
- To map NNs on spiking network architectures
- Programming tools
We focus on programming tools, specifically the Nengo API.
Nengo acts as a neural compiler: you specify neuron properties, values to be represented, and then it solves for connection weights between components such that they perform the desired functions.
Based on NEF principles: Neural Engineering Framework (see Stewart 2012).
Neural Engineering Framework (NEF)
TODO: See Deep Learning survey for learning rules
- Represent activity
- Transform inputs
- Dynamics
Representation
A population of neurons collectively represents a time-varying vector of real numbers through non-linear encoding and linear decoding.
Let's say we have some input vector x. Then this input will be represented by
Activity of one unit in the ensemble with respect to that input, with non-linear encoding:
$a_i = G(\alpha_i \textbf{e_i} \dot \textbf{x} + b_i}$)
- Encoding vector e can be interpreted as a tuning curve: to which inputs is the neuron sensitive?
- $\alpha_i$ is a gain parameter that basically dampens or amplifies the effect of the input stream.
- (I find it annoying that they both use a and alpha, and that G does not stand for the gain.)
- non-linearity G (basically your neuron model; e.g. a step function, a rate-based sigmoidal or a LIF).
- b constant background bias.
Typically encoding vector, gains, and biases are determined randomly! No learning happening at this point.
We can now encode input. What about decoding the input from the activations? Turns out that we can linearly decode the input using least squares: see slide for math (bottom line).
For each input we can find decoding vectors $d_i$ such that the reconstructed input is:
$$\hat{x} = \sum_i a_i \textbf{d_i}$$
This is really nice, because we can retrieve non-linear encodings through linear decoding! Math is not super important for us; main point is that we can analytically decode what a population encodes.
Least squares:
$$\textbf{d} = A^{-1}B$$
$$A_{ij}=\sum_x a_i a_j$$
$$B_j = \sum_x a_j \textbf{x}$$
Recall exercise 2 in statistical machine learning, where we derived a closed form solution to optimizing weights of a polynomial using least squares. First of all, you compute the error per datapoint; that's why you have the sums over x. The total error is just the sum of the errors of the individual data points. The closed form solution also has the form $w = A^{-1} T$, analogous to the solution above.
Transformation
We want to be able to do functions over inputs. E.g. we want to encode sensory input to move an arm. That arm movement needs to be a function of the inputs.
Let's say we have two populations A and B, and identity function f(x)=x.
Let's say we A encoded a certain input. We can decode A to vector $\hat{x}$ which retrieves the input to A. We can can then map it again onto population B using some encoding vector over the reconstructed inputs of A. Note that this encoding step ( $\alpha_i \textbf{e} \textbf{x}$ ) is a linear operation! (Is non-linearity G included here?)
So we decode the input of A, encode it again to map it onto B.
Q: Why is this "dummy" step necessary? That is, why can't we just straightforwardly pairwise connect neurons in A to B (a1 with b1, a2 with b2 etc.)?
- There may be different numbers of neurons in populations A and B
- They may have different gain values as well
- The non-linearity G over the input connection will make the encoded values in B different than those in A.
So instead, we assume "an intermediate group of perfectly ideal linear neurons" (Stewart 2012, p. 3) Because these operations are both linear, we can do them in a single step so we can directly express this decoding and encoding step as a single mapping with weights w:
$$w_{ij}=d_i \dot e_j$$
(Note that we do not actually have to use the full weight matrix now! We only need the encoding and decoding vectors which saves memory and speeds up computation )
Here's the cool part: this procedure still works with non-linear functions!
Comparison with SVM; map into higher dimension in which the non-linearity becomes functionally linear. We also encode our input into a high-dimensional space due to having many M+N neurons with MxN connections. As a result, through linear decoding and encoding, we can encode non-linear functions.
Of course this still relies on a mapping and isn't perfect; it's an approximation.
- e is basically a random projection
- We have random variation in the gain values for both ensembles
- This randomness is beneficial for function approximation, because "the function f(x) being approximated ends up being built up out of linear sums of the tuning curves (Figure 1), so a wider variety of tuning curves leads to better function approximation." (Stewart 2012, 4).
- This is also why feeding two inputs into the same neurons amounts to adding them.
The more non-linear the function will be, the worse the decoded result assuming we keep the same amount of neurons. (In hindsight after practicals: I did indeed see worse performance for non-linear functions like raising the input to a certain power)
On accuracy:
This accuracy is also affected by the neuron properties and the encoding method used: for example, if the ei values are all aligned with the standard basis vectors, then nonlinear functions of multiple variables cannot be computed (this is the case for standard connectionist models, which is why they require multiple layers). This allows you to determine what neuron properties would be ideal for particular computations, which can then be used as neurological predictions (Stewart 2012, 4)
Not in lecture but in Stewart 2012 paper on NEF: learning.
Indeed, you can even use the input/output pairs used for training data in standard neural network approaches (the input patterns give the x values to sum over, and the output patterns give the corresponding f(x) values). Importantly, this means that nonlinear functions can be computed with a single layer of connections – no back-propagation of error is required. (Stewart 2012, 4)
Dynamics
Given the first two principles of representation and transformation, we can also do general dynamics i.e. compute a dynamic function that describes a chance of time rather than a value. One example: have a LIF where part of the membrane potential is determined by the potential at t-1 together with some input drive.
A dynamic function has the shape:
$$ dx/dt = A(x) + B(u)$$
Where
- Everything except t is a vector
- x is the value being represented
- u is some input drive
-
A and B are arbitrary functions
-
General form: $dx/dt=A(x)+B(u)$
- Integrator: $dx/dt=u$
- When input u=0, the state will be maintained.
- When input > 0, the value will increase with u every step t; reverse for u < 0
- Oscillator: $dx/dt=[x_2,-x_1]$
Stewart 2012: if you have recurrent connectivity and input drives, then you can define these dynamics on time varying input x. For recursive connections we need to know the neurotransmitter reabsorbtion time $\tau$:
Given this value, we have shown that you can compute the desired dx/dt=A(x)+B(u) function by using the above method to create a set of feedback connection weights that compute τA(x)+x, and setting the input connection to compute τB(u). (5)
So you essentially "derive the connectivity that gives the desired dynamics" using the principles of representation and transformation. (5)
Intel Loihi
Neuromorphic Manycore Processor.
Main feature: On-Chip Learning.
Loihi SNN Model
- Top-down motivated from computational neuroscience
- Two internal state variables:
- synaptic response current
- membrane potential
CUBA LIF neurons with exponential smoothing
CuBa stands for: current-based LIF neurons. They show realistic integration behavior.
A unit in Loihi represents two state variables:
- Synaptic response current
$$u_i(t)= \sum_{j \neq i} w_{i,j}(\alpha_u * \sigma_j)(t) + b_i$$
"sum of filtered input spike trains and a constant bias current" (Davis, 2).
- $w_{ij}$: synaptic weight from neuron i to j
- $\alpha_u(t) = \tau_u^{-1}exp(-t/\tau_u)H(t)$: synaptic filter impulse response
- $\tau_u$ time constant
- H(t) unit step function
All weighted incoming spiking information (no recurrent connection), which are expressed in terms of a Dirac-delta function.
A spike train is mathematically defined as a series of Dirac-delta functions that we sum over (where $t_k$ is the time of the k-th spike in the spike train):
$$\sigma(t)=\sum_k \delta(t-t_k)$$
Then the membrane voltage is the input $u_i(t)$, minus some leakage (left term). There is also a penalty term $\theta$ for firing too much; this is to simulate the refractory period.
$$\hat{v_i}(t) = - \frac{1}{\tau_v}v_i(t) + u_i(t) - \theta_i \sigma_i (t)$$
Solving LASSO benchmark; sparse coding problem.
Example from Loihi white paper.
LASSO problem: we have some input that we represent, and we want to represent this with as little coefficients as possible. That is, we want to find a sparse coding to represent this input.
For LASSO, also see SML-2-Regression_ctd.md
The representation is some linear combination of features with coefficients; and the features come from some feature dictionary.
The Loihi white paper discusses the spiking-locally competitive algorithm
- For each feature we instantiate a neuron
- The activation of that neuron indicates how well a particular input correlates with that feature
- All units are connected to each other with inhibitory connections, so they compete for activity.
- Neurons with the highest activity will win out and will constitute a stable state.
Comparison with analytical solver; Loihi gives a decent solution fast, but does not converge to an exact solution over time.
- Efficiency-accuracy trade-off
- Efficiency possible by exploiting temporal ordering of spikes
- The neurons that spike earliest win the competition and this immediately gives a good approximate solution
- SNN is communication efficient because we have one-to-many communication instead of all-to-all with the vista solver.
Loihi Architecture
- 128 neuromorphic cores (with 3 additional x86 cores)
- each with 1024 primitive spiking units that can be combined into more complex dendritic structures (but then you can simulate less neurons).
- Each core is a learning unit.
- So you have learning engines locally on chip.
- Fully digital and deterministic.
AER:
- hierarchical-mesh hybrid version of the AER protocol for scalability.
- Supports up to 4096 cores in theory
- and scalable to 16384 chips
Computational primitives
One of the main advantages of Loihi is that it offers a lot of computational primitives for neuromorphic computing. Loihi is super flexible which makes this chip very suitable to a broad research field, including computational neuroscientists.
- You can use use stochastic noise
- You can do configurable dendritic tree processing
- What's this?
- "Neurons in the SNN might be decomposed into a tree of compartment units, with the neuron’s input synapses distributed over those compartments. Each compartment supports the same state variables as a neuron, but only the root of the tree (soma compartment) generates spike outputs. The compartments’ state variables are combined in a configurable manner by programming different join functions for each compartment junction." (Davis' Loihi white paper, 5)
- And on-chip learning
- Much more (see p.5 from white paper)
Again, due to this flexibility, you lose some potential performance.
On-chip learning
Each core has its own learning engine, which makes learning very fast as its so close to the hardware.
This learning engine supports STDP out of the box and the learning engines are programmable (meaning you can tell it what to learn). Cf. NMC-3-Learning.md for STDP.
Coding the learning has to be done in C snippets and has to take the following specific form of a sum of products:
$$z := z + \sum_{i=1}^{N_p}S_i \prod_{j=1}^{n_i}(V_{i,j}+C_{i,j}$$
The terms in this formula are the parts you can adjust for the learning.
All components are 4-bit encoded, so there are 16 components you can learn with/on.
From Davis' white paper:
locality constraint:
SNN synaptic weight adaptation rules must satisfy a locality constraint: each weight can only be accessed and modified by the destination neuron, and the rule can only use locally available in formation, such as the spike trains from the presynaptic (source) and postsynaptic (destination) neurons. (4)
This locality constraint is important, because optimization algorithms for traditional neural networks use a global error signal.
The locality constraint imposes a significant challenge on the design of learning algorithms, as most conventional optimization procedures do not satisfy it. Although the development of such decentralized learning algorithms is still in active research, some pioneering work exists showing the promise of this approach. They range from the simple Oja’s rule for finding principal components, to the Widrow-Hoff rule for supervised learning and its generalization to exploit precise spike-timing information,4 to the more complex unsupervised sparse dictionary learning using feedback and event-driven random back-propagation. (4)
Micro-architecture
You see that the components are divided into small compartments that correspond to components of an actual neuron (e.g. soma and axon).
Comparison
- Against ATOM CPU
- on convolutional version of the LASSO problem
- Compared on Energy Delay Product
- balances energy efficiency and efficiency/speed
Slide shows a table. "3x" means that the Atom CPU's EDP is 3x that of Loihi (i.e. it is less efficient)
For small problems, the Atom CPU outperforms; the Loihi does way better for large problems.
This comparison is not very fair, because CPUs are simply not designed to do parallel computation. Ideally you'd want to compare against GPUs or other Neuromorphic chips.
Application: Dynamic Vision Sensors (DVS)
Do not do frame by frame processing, but only monitor changes in inputs with event-based sensors. This is a large gain in energy efficiency.
Inspired by retinal processing.
www.ri.cmu.edu/ri-people/ryad-benosman/
You could use other hardware, but if you use a neuromorphic chip, the event-based processing comes for free.
In general a main reason for using SNNs is that you are either limited on resources (energy or time). If you have the resources and you need precision, then do not use SNNs.