Closed form solutions
Bayesian inference is an intractable problem in general, but some models have a very convenient structure that make them analytically tractable: we are then able to express the posterior as a finite algebraic combination of elementary functions. Examples are powers, trigonometric functions, powers, divisions etc. Because we can find the solution analytically we say that we have a closed form solution.
What fascinates my and was pointed out by Luca Ambrogioni is that in the end it is a matter of mathematical convention which functions are said to be (sufficiently) elementary; and as a result, which solution counts as "closed form".
In a way an exponential function is quite complex. But because we have clear analytical rules for multiplying, adding, dividing exponential functions anything that reduces to an exponential function can be considered a closed form solution (e.g. in Bayesian inference with Gaussians ). The same holds for sinusoids, which express quite complex relations but which also have clear rules for analytical manipulation.