Running average using matrix multiplication

We want a running average over a (T, C) matrix across channels (C), respecting the temporal order (T). Channels e.g. in row 3 should only use information from their respective channel in the "past", row 2 and 1. This trick is handy for doing efficient matrix computations on sequences, for example in NLP tasks for predicting the next word.

import numpy as np

# Rows in temporal order. Row 2 only has information of row 1 available.
# Row 3 has information from row 2 and 1.
T = 3

# We have C "channels" of data
C = 4

I = np.ones((T, T))  # Identity matrix
A = np.tril(I)  # tril stands for "triangle lower"

# (n, k) @ (k, m) -> (n, m)

X = np.random.randint(0, 10, (T, C))

# If you take the matrix multiplication of the data with I
# You replace each channel value with the *sum* of that channel over time T.
summed = I @ X

# But we want to respect the temporal order of T when summing
# We can do this by using the tril of I. As you progress in time (down the rows)
# you'll see a running sum of the current value with past values.
running_sum = A @ X

# If we instead want to have a running average, we need to normalize each row of A
# N.B. keepdims is needed for correct broadcasting against A!
A_norm = A / np.sum(A, axis=1, keepdims=True)

# Now compute the running average respecting the temporal order of rows (T)
running_avg = A_norm @ X

# Do a check
assert all(running_avg[1] == np.mean(X[:2], axis=0))