What's the right objective function for trustworthy AI?
AI systems optimize some objective function, but usually they have some other goal.
A major point of Human Compatible AI (Russell), is to ensure the alignment of AI's objective function with human values.
Also, in my own work (at NS), for assessing the ethical aspects of an application, I added a question that asks developers to reflect on whether the AI application is not only evaluated in terms of some metric (like accuracy or something comparable), but also in terms of the real-world goals and requirements that were set for the AI application. And how? (Think a similar question can be found in the ALTAI of the European Commission).
A clear example of where the objective function is sketchy is "emotion detection".
A classic example of a misplaced objective function is the use of machine learning for emotion detection. The programs are supposed to be able to classify the emotional state of a person based on image recognition that identifies facial expressions and has been trained to link those to labels of emotion such as fear and anger.
But psychologist Lisa Feldman Barrett has criticized the science underlying such a scheme. Emotion recognition systems are not trained to detect emotions, which are complex, nuanced systems of signs, but rather to lump various muscle movements into predetermined bins labeled as this or that emotion. The neural net is merely recreating the rather crude and somewhat suspect reductive categorization upon which it was based.
Much of today's work is focusing on the data, and there has been scrutiny of the scale of architectures, as in the Parrot paper, but the objective function may be the final frontier of ethics.