Stochastic Parrot
doi: https://dl.acm.org/doi/pdf/10.1145/3442188.3445922
Background info:
We say seemingly coherent because coherence is in fact in the eye of the beholder. Our human understanding of coherence derives from our ability to recognize interlocutors’ beliefs [ 30, 31 ] and intentions [ 23 , 33 ] within context [ 32 ]. That is, human language use takes place between individuals who share common ground and are mutually aware of that sharing (and its extent), who have communicative intents which they use language to convey, and who model each others’ mental states as they communicate. As such, human communication relies on the interpretation of implicit meaning conveyed between individuals.
Coherence is in the eye of the beholder and our basic assumption when interpreting fluent natural language text is that it is coherent, and that it is grounded in a shared understanding of the world and the communicative intent of the speaker:
Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that. This can seem counter-intuitive given the increasingly fluent qualities of automatically generated text, but we have to account for the fact that our perception of natural language text, regardless of how it was generated, is mediated by our own linguistic competence and our predisposition to interpret communicative acts as conveying coherent meaning and intent, whether or not they do [89, 140]. The problem is, if one side of the communication does not have meaning, then the comprehension of the implicit meaning is an illusion arising from our singular human understanding of language (independent of the model). Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot. (616-617)
This also ties into accountability: part of this coherence assumption is that we assume the utterance corresponds to the communicative intent of someone (or a group/entity) that can be held accountable for what is said.