Multimedia Representations
Representing audio
Audio needs to be discretized.
E.g. for 4-bit PCM the a pressure/amplitude can take on 16 values. So the amount of bits determines the resolution of the audio.
We get these by sampling a continuous wave at time steps. If we do this, our audio is a list of numbers, where each number is a sample. So how can we use this list of number for mining purposes? What are useful representations to capture the characteristics of the original audio?
Static representation
We can construct vectors containing properties of the audio. For example, if we record loudness, pitch, brightness and bandwidth, we can compute their averages over a time period or over the whole recording. This are static variations because the mean abstracts away from temporal variations. We can then work with these vectors similarly to text vectors.
Dynamic representation
Of course, we know that characteristics of sound signals heavily vary over time, and that these variations are meaningful.
So we want to keep the dynamics of the original audio signal. We have some options:
- Break audio into windows
- Classify each window (e.g. loud, medium, soft)
- Count how many windows belong to each class
- Create a vector from these counts
Advanced methods (these may still use windowing):
- Hidden Markov Models
- Recurrent Neural Networks
Representing Images
Pixels; for gray-scale 0-255 for each pixel. RGB we have three channels per pixel.
What's an appropriate representation?
- Vectors of pixel values are not useful, because they do not capture anything about the image and are not comparable
- Vectors of global feature values
- This captures global features of the image, e.g. the number of pixels with a certain quality
- Makes images comparable on that feature
- Vectors of visual word weights
- Analogous to "word" vectors in text representations
- Meaning: we have a set of local features that we get by clustering pixel patches together.
- An image ("sentence/text") representation is a vector of weights/counts for these local features
- Each "word" is a member of a cluster of pixel patches.
- Vectors of neural network values
- Components are nodes in a network, values the activations at those nodes
- Cf. word embeddings approach.