"Human Compatible" by Stuart Russell
@book
Subtitle: Artificial Intelligence and the problem of control
Stuart Russell is co-author of the famous AI textbook with Peter Norvig.
For reference, see the annotation file for this book. References to annotations are indicated with the A prefix. This method is not 100% future proof, because the numbering changes if I insert new notes in between. (Update: more like 0% future proof because my Epub annotation scripts seems to have messed up the ordering).
This book is worked through in an ethics reading group, guided by Pim Haselager.
Chapter 1: If we succeed
- Against AI denialism (A4)
- Eye for negative societal consequences of dumb algorithms (A5)
This charter introduces the limits of our current definition of intelligence.
def intelligence in A6:
Machines are intelligent to the extent that their actions can be expected to achieve their objectives. " (2020-04-14T08:08:05Z; progress: 0.0524691)
Meaning: these machines are optimizing machines for goals that we encode in them, because they do not have goals of their own. (A7)
This is what Russell calls the standard model of contemporary AI (A8-A11) where learning algorithms are optimizing some objective by minimizing a loss function; the objective being a "purpose put into the machine" (A8)
In the standard model, building AI has, in various shapes and forms, been about building rational machines (A15).
However, many of the dangers of AI stem from blindly optimizing some objective. Some directions:
- The is uncertainty in human objectives and in real life, introducing the need for algorithms that recognize this uncertainty.
- One consequence of this uncertainty would be algorithms deferring decisions to humans; asking permission, accepting correction etc. (A12)
This direction however requires revisiting our current approach to AI, because this means we should not encode definite goals into the AI.
Instead, the focus should be on how to make the AI goals human compatible i.e. align them with human goals and have mechanisms to ensure this alignment.
This is a main problem of the book (A14):
“The problem is right there in the basic definition of AI. We say that machines are intelligent to the extent that their actions can be expected to achieve their objectives, but we have no reliable way to make sure that their objectives are the same as our objectives. What if, instead of allowing machines to pursue their objectives, we insist that they pursue our objectives? Such a machine, if it could be designed, would be not just intelligent but also beneficial to humans.” (2020-04-14T08:13:09Z; progress: 0.058642)
So chapter introduces the necessity of a better and less narrow definition of intelligence, not just as optimizing a preset objective, but as being beneficial (and responsive) to human objectives.
Chapter 2: Intelligence in humans and machines
Broad discussion of the concept of intelligence, more specifically how it has been understood in the tradition so far. This means it focuses on concepts of rationality from Aristotle to modern game theory and AI. Some topics:
- Aristotle and practical rationality
- Bernoulli and utility (A16)
- utilities are inferred from the preferences exhibited in behavior
- Game theory
- Babbage and the Analytical Machine
- Turing machines / universal computing machines.
- Russell argues the Turing test is not intended as a test of intelligence, but a clever rhetorical way to refute Turing's criticasters.
- As a test of intelligence it is very limited: too informal, contingent, depends on understanding of the human mind, does not unpack into guidelines.
- Humans are not perfectly rational and cannot behave fully rational
- Types of agents, with a focus on how they embody the "standard model"
- Purely perceptual (relatively innocent)
-
- decision-making (Russell focuses on issues here)
- Knowledge based systems
- Bayesian networks and "Bayesian rationality"
- Discussion of DeepBlue, AlphaGo, Dota.
- Issue: humans need to define loss functions. Since these are what algorithms optimize, they have far-reaching consequences.
- Russell argues we need a better way to acknowledge the true misclassification costs and also the uncertainty we have about these costs.
- Russell argues against the idea that AGI will only result from research into AGI. Rather, it is more likely to take shape by researchers making conceptual leaps in very specific subfields of AI.
- That is, by very good researchers that do not make ad hoc improvements but really bring the field forward with conceptual innovations, like the mathematical work done on convolutional neural networks that is at the basis of the current AI hype.
Funny observation: from the "standard model" perspective, consciousness is a completely irrelevant issue. Hollywood movies on AI thus completely miss the point.
“All those Hollywood plots about machines mysteriously becoming conscious and hating humans are really missing the point: it’s competence, not consciousness, that matters.” (2020-04-14T08:28:11Z; progress: 0.0679012)
The overall intention seems to be to show that optimizing AI in a strict rational sense (the "standard model") is not sufficient, for various reasons:
- Because rational equilibria in the game-theory sense may not be beneficial
- Because human preferences are not strictly rational, i.e. they may be inconsistent. An AI cannot overcome this issue. The issue is more poignant if the goal is to create beneficial AI that is responsive to human goals.
" A theory of beneficial AI can, in fact, accommodate inconsistency in human preferences, but the inconsistent part of your preferences can never be satisfied and there’s nothing AI can do to help. " (2020-04-14T15:04:12Z; progress: 0.095679)
- Perfect rationality is in itself not achievable due to complexity considerations that are not realistically overcome by just having more computation. Despite hard complexity limitations, clever solutions + increased computation power may result in half decent solutions, not in perfect rationality.
“Running a poorly designed algorithm on a faster computer doesn’t make the algorithm better; it just means you get the wrong answer more quickly. (And with more data there are more opportunities for wrong answers!)” (2020-04-14T15:53:07Z; progress: 0.117284)
- Focus should be more on dealing with uncertainty in goals, loss functions etc.
Chapter 3: How Might AI Progress in the future
- Intelligent assistants and privacy
- Shortcomings of early AI systems: access, content, context.
- Access: lack of sensory awareness about what's going on, e.g. who is the user talking to
- Content: despite having access, they do not understanding the meaning of the input
- Context: "lack the ability to keep track of and reason about the goals, activities, and relationships that constitute daily life"
- General design template for personal assistants, healthcare, education, finances:
- Background knowledge about human activities
- "ability to extract information from streams of perceptual and textual data"
- "a learning process to adapt the assistant to the user's particular circumstances."
- "Smart" (read: dumb) tech as a Trojan horse for AI:
“Despite these shortcomings, smart speakers and cell phone assistants offer just enough value to the user to have entered the homes and pockets of hundreds of millions of people. They are, in a sense, Trojan horses for AI. Because they are there, embedded in so many lives, every tiny improvement in their capabilities is worth billions of dollars.” (2020-04-16T07:48:59Z; progress: 0.194444)
- Related to this: manipulation by screens:
" In the cyber realm, machines already have access to billions of effectors—namely, the displays on all the phones and computers in the world. This partly explains the ability of IT companies to generate enormous wealth with very few employees; it also points to the severe vulnerability of the human race to manipulation via screens. " (2020-04-16T12:43:20Z; progress: 0.265432)
- When will superintelligence arrive? Russell does not make predictions because:
-
- These predictions are usually wrong
-
- There is no clear threshold to superintelligence that will be crossed
-
- The required conceptual breakthroughs are inherently unpredictable
-
This is a sensible position to me.
Q: Given these arguments, why do think that other AI researchers/philosophers nevertheless make predictions about the arrival of AGI?
Especially the second point is quite original and interesting. The threshold that Russell mentions here concerns the threshold between having "partially superintelligent" systems and a "completely general superintelligent AI system". Part of his reasoning is almost self-evident: indeed we'll almost certainly have these specialized systems before a general artificial intelligence and superintelligence. They won't just drop from the sky.
Q: Is Russel talking about Artificial General Intelligence (AGI), superintelligence, or does he combine both in the same argument when speaking of a "completely general superintelligent AI system"? Would that matter for his line of argumentation?
His main argument why the threshold with superintelligence would not be clear is that the issues with superintelligence would also already be present in these "partially superintelligent" systems.
Q: Russell does not, at this point, specifically mention what issues these would be. Which issues would already be present in specialized narrow AIs with superhuman performance?
Q: And do you agree that the threshold between narrow AI with superhuman performance and general superintelligence would not be clear?
My intuition is that Russell's argument may be slightly begging the question, because he chooses his examples of superintelligence such that they are mostly a scaling up of what humans (and thus a human-level AI) can do, thus downplaying potential qualitative differences. He also seems to perform a quite clever rhetorical trick by speaking of "partially superintelligent" AI, suggesting a sort of "in-between" category between narrow AI and AGI/superintelligence. However, when we stick to the more common jargon of the debate, isn't this just well-performing narrow AI? (that just performs better than humans in this one particular field)?
- Russell keeps referring to a quote from Whitehead: “As Alfred North Whitehead wrote in 1911, “Civilization advances by extending the number of important operations which we can perform without thinking about them.”45" (2020-04-16T12:22:46Z; progress: 0.246914)
- Q: What does he mean here exactly?
What's needed for human-level AI?
A method for finding out what is lacking is examining where existing systems break down.
So what conceptual innovations are needed and currently lacking?
- Language and common sense
- Discovering Actions
- Effective methods for generating new concepts, relationships between concepts, to extend on existing knowledge.
- Philosophers: 3xI: intuition, insight, inspiration; resistant to algorithmic explanation.
- AI researchers: there is nothing "essential" here; "Put simply, if a machine learning algorithm can search in a space of hypotheses that includes the possibility of adding definitions for new terms not present in the input, then the algorithm can discover new concepts.” (2020-04-16T11:52:32Z; progress: 0.243827)"
- Potential counter: what's missing is the ability to enlargen or redefine the space of hypotheses.
- Nice description that illustrates what would be required for human-level intelligence, that focuses on finding the right relation between abstractions:
- Effective methods for generating new concepts, relationships between concepts, to extend on existing knowledge.
“For example, is it possible to start from scratch with a robot that knows only that it can send various electric currents to various motors and have it discover for itself the action of standing up? It’s important to understand that I’m not asking whether we can train a robot to stand up, which can be done simply by applying reinforcement learning with a reward for the robot’s head being farther away from the ground.46 Training a robot to stand up requires that the human trainer already knows what standing up means, so that the right reward signal can be defined. What we want is for the robot to discover for itself that standing up is a thing—a useful abstract action, one that achieves the precondition (being upright) for walking or running or shaking hands or seeing over a wall and so forms part of many abstract plans for all kinds of goals.” (2020-04-16T12:21:56Z; progress: 0.25)
This constructing of relevant abstraction hierarchies is the most important required capability:
" I believe this capability is the most important step needed to reach human-level AI. It would, to borrow Whitehead’s phrase again, extend the number of important operations that AI systems can perform without thinking about them." (2020-04-16T12:22:27Z; progress: 0.25)
- Managing mental activity.
- Improvements in AI are more efficient w.r.t. other AI with the same type of computation. Even though AlphaGo defeated Lee Sedol, Sedol probably:
"does no more than a few thousand units of computation per move, but he has a much more flexible computational architecture with many more kinds of units of computation: these include dividing the board into subgames and trying to resolve their interactions; recognizing possible goals to attain and making high-level plans with actions like “keep this group alive” or “prevent my opponent from connecting these two groups”; thinking about how to achieve a specific goal, such as keeping a group alive; and ruling out whole classes of moves because they fail to address a significant threat. " (2020-04-16T12:29:06Z; progress: 0.256173)
- Be more independent of "feature engineering"
Limitations of superintelligence
Incomplete, just a highlight:
- Machines are not human:
A final limitation of machines is that they are not human. This puts them at an intrinsic disadvantage when trying to model and predict one particular class of objects: humans. Our brains are all quite similar, so we can use them to simulate—to experience, if you will—the mental and emotional lives of others. This, for us, comes for free. (If you think about it, machines have an even greater advantage with each other: they can actually run each other’s code!) For example, I don’t need to be an expert on neural sensory systems to know what it feels like when you hit your thumb with a hammer. I can just hit my thumb with a hammer. Machines, on the other hand, have to start almost from scratch in their understanding of humans: they have access only to our external behavior, plus all the neuroscience and psychology literature, and have to develop an understanding of how we work on that basis. In principle, they will be able to do this, but it’s reasonable to suppose that acquiring a human-level or superhuman understanding of humans will take them longer than most other capabilities. " (2020-04-16T12:49:46Z; progress: 0.271605)
This is actually a really argument and I'm not sure if it holds. Does he implicitly admit to some form of essentialism here? Do you need a "human" brain to be able to simulate "human experiences"?
Russell stretches the imagination but argues we should not enter into the region of science fiction. Even to superintelligence there are limits.
Chapter 4: Misuses of AI
The book overall concerns the "control problem" of AI. Later chapters focus on the possibility of us losing control over AI. This chapter focuses on AI itself being used (by humans/organisations) as methods of control.
Indirect control: modify the information environment
Deep fakes and bot armies employ new strategies for changing human behavior: by modifying their information environment
What's new here, compared to e.g. propaganda?
- AI can track online behaviour and personalize the strategy
- get feedback on its success to learn what's effective
Related to this topic of manipulation, Russell coins an interesting concept: true information environment.
"I would like to suggest that everyone should also have the right to mental security—the right to live in a largely true information environment." --- 0.293, 2020-05-07
Direct control: Humans as reinforcement learning algorithms (where governments set the objectives)
"Finally, methods of control can be direct if a government is able to implement rewards and punishments based on behavior. Such a system treats people as reinforcement learning algorithms, training them to optimize the objective set by the state. The temptation for a government, particularly one with a top-down, engineering mind-set, is to reason as follows: it would be better if everyone behaved well, had a patriotic attitude, and contributed to the progress of the country; technology enables measurement of individual behavior, attitudes, and contributions; therefore, everyone will be better off if we set up a technology-based system of monitoring and control based on rewards and punishments." --- 0.293, 2020-05-07
When AI-technology is all about optimizing towards a certain objective, who then sets this objective? And given that we set an objective to optimize, do we in fact achieve what we intended to achieve?
I really enjoyed the following paragraph, where Russell provides three arguments, of different types, against the top-down engineering technocracy mindset.
"First, it ignores the psychic cost of living under a system of intrusive monitoring and coercion; outward harmony masking inner misery is hardly an ideal state. Every act of kindness ceases to be an act of kindness and becomes instead an act of personal score maximization and is perceived as such by the recipient. Or worse, the very concept of a voluntary act of kindness gradually becomes just a fading memory of something people used to do. Visiting an ailing friend in hospital will, under such a system, have no more moral significance and emotional value than stopping at a red light. --- 0.293, 2020-05-07
I think the issue here is not necessarily that there would no longer be acts of kindness; the problem is of how to recognize them as such. Under this regime, the only way to "prove" something you do is in fact an act of kindness, would be to show 1) that you understand how your actions are evaluated and 2) then act against the maxim of score maximization. It would not be sufficient to be altruistic and selfless, a kind act would have to be self-destructive (or one might say, extremely altruistic). Only then (perhaps?) the receiver would see rationally, without relying on empathy and good faith, that some moral and human value is the best explanation for the shown behavior, rather than score optimization.
The paradox is that by trying to optimize desirable human behavior and by quantifying the human values that they promote, you lose the quality of what is sought after, similar to the capitalist perversion of friendship or love when they become part in an economy of (monetary) exchange. Friendship or love are not simply part of a contract, because that would imply you can demand some utility from the other and enforce this demand. I am not denying that friendships and love relationships can have and in fact do have utility. I am rather saying that only the cynical and the sociopathological would argue they are about utility and, as a consequence, can be quantified.
In other words, the true objective would not be captured by any explicitly stated objective. Russell also reaches this conclusion but via a different route, namely by showing many scenarios in which AI optimizing the explicitly stated objective has undesirable consequences. This is a problem for what Russell calls the "standard model" of AI, where "intelligent" machines are optimized a "purpose put in the machine".
Second, the scheme falls victim to the same failure mode as the standard model of AI, in that it assumes that the stated objective is in fact the true, underlying objective. Inevitably, Goodhart’s law will take over, whereby individuals optimize the official measure of outward behavior, just as universities have learned to optimize the “objective” measures of “quality” used by university ranking systems instead of improving their real (but unmeasured) quality. --- 0.293, 2020-05-07
I was unfamiliar with Goodhart's law, but Russell's explanation is exceptionally clear (and the university example is awfully accurate). Marilyn Strathern's paraphrasing is elucidating: "When a measure becomes a target, it ceases to be a good measure."
When you optimize or "game" the measure of your objective, rather than your true goal.
Whereas in economics and game theory it is people gaming the measure (which may in fact be to the detriment of what you are trying to promote), in Russel's argument this problem is inherent to the intelligent machines and algorithms designed to minimize some loss function under the standard model of AI. I found a book called "The Tyranny of Metrics" by Jerry Z. Muller, whose title would have been accurate here as well.
Finally, the imposition of a uniform measure of behavioral virtue misses the point that a successful society may comprise a wide variety of individuals, each contributing in their own way." --- 0.293, 2020-05-07
The latter point is relatively self-evident and adds to the earlier reasoning: can you capture the quality you are trying to promote with reasoning based on a uniform quantitative measure?
Russel's book specifically deals with the problem of specifying objectives in relation to Artificial Intelligence technology that is designed to optimize towards a given goal. You could perhaps generalize his argument to technocracy in general: technocracy firstly assumes you have an (engineer) class of people that know what is best, which is both optimistic and paternalistic, but secondly, even when they do know what is best, it is quite a fundamental issue whether you can also translate that into procedures that actually achieve what you are looking after.
Is false information protected by free speech?
Russell points out that we are very vulnerable to the technology of misinformation:
"Even though we do not expect used-car salespersons and politicians to tell us the truth, we have trouble believing that they are lying as brazenly as they sometimes do. We are, therefore, extremely vulnerable to the technology of misinformation." --- 0.296, 2020-05-07
Russell then points out that there is a tension between freedom of expression, democracy, and battling misinformation: in battling misinformation, there is the risk of censuring free speech which, in turn, is central to democracy. In short: you have the right and freedom to share false information. This tension seems especially relevant in a US-context, where freedom of speech almost has an absolute character.
Russell argues this strong opposition is false, which is from my perspective a very sensible and non-controversial standpoint. For example, I don't think any of us would be convinced by someone claiming "freedom of speech" after making death treats. Do you agree?
Russell explains his position with respect to content hosting platforms such as Facebook:
"Rather than pursuing the idea that there is no freedom of thought without access to true information, democracies seem to have placed a naïve trust in the idea that the truth will win out in the end, and this trust has left us unprotected. Germany is an exception; it recently passed the Network Enforcement Act, which requires content platforms to remove proscribed hate speech and fake news, but this has come under considerable criticism as being unworkable and undemocratic." --- 0.296, 2020-05-07
For reference, it used to be the case that providers who host content are not responsible for the content. They only have the responsibility to remove, let's say, child porn when someone notifies them of that illegal content. This old model is severely under pressure as platforms like Facebook have gained immense societal impact, and also have an increasing say in what people see and when they see it through algorithmic mediation and big data analyses.
Do you agree that making content platforms responsible for the content they host is undesirable, or even undemocratic?
Counter measures against misinformation
- Ground truth systems, like the function of notaries.
"inject sources that are as close as possible to ground truth. A single fact that is certainly true can invalidate any number of sources that are only somewhat trustworthy, if those sources disseminate information contrary to the known fact." --- 0.299, 2020-05-07
Issues: solutions all rely on reputation, but botnets are propagating false information at a rate that reputation systems are likely to fail.
"Ultimately, all such efforts rely on circular reputation systems, in the sense that sources are trusted because trusted sources report them to be trustworthy. If enough false information is propagated, these reputation systems can fail: sources that are actually trustworthy can become untrusted and vice versa, as appears to be occurring today with major media sources such as CNN and Fox News in the United States. Aviv Ovadya, a technologist working against misinformation, has called this the “infopocalypse—a catastrophic failure of the marketplace of ideas.”10" --- 0.299, 2020-05-07
See infopocalypse .
- Disseminating misinformation should not be free, but regulatory penalties are trickier to do right
"impose a cost for purveying false information. Thus, some hotel rating sites accept reviews concerning a particular hotel only from those who have booked and paid for a room at that hotel through the site, while other rating sites accept reviews from anyone" --- 0.302, 2020-05-07
Lethal weapons
Main argument here is that autonomous weapons are scalable weapons of mass destruction.
Technological unemployment
Already pointed out by Aristotle.
Brynjolfsson and McAfee coined the phrase "the Great Decoupling": the benefits of automation end up at the capital and not the labourers (after an initial phase).
Interesting remark about Universal Basic Income, in particular the assumption about human purpose it makes.
Algorithmic bias
TODO: nog niet verwerkt vanuit annotaties.
Brief summary of the intricacies of algorithmic bias:
"One reason for all the concern about automated decisions is the potential for algorithmic bias—the tendency of machine learning algorithms to produce inappropriately biased decisions about loans, housing, jobs, insurance, parole, sentencing, college admission, and so on. The explicit use of criteria such as race in these decisions has been illegal for decades in many countries and is prohibited by Article 9 of the GDPR for a very wide range of applications. That does not mean, of course, that by excluding race from the data we necessarily get racially unbiased decisions. For example, beginning in the 1930s, the government-sanctioned practice of redlining caused certain zip codes in the United States to be off-limits for mortgage lending and other forms of investment, leading to declining real-estate values. It just so happened that those zip codes were largely populated by African Americans." --- 0.348, 2020-05-07
Some observations:
- The first issue is indeed that algorithmic bias is not easily made visible. It can act on (latent) patterns in data that are not explicitly stated. In other words, the data itself can be biased.
- Algorithmic biased decision-making has a self-reinforcing effect.
- Maybe this would be a good sanity-check: if the latent features would be made explicit, would you be discriminating in a way that normally is considered unlawful or unethical?
- For example, consider an algorithm that predicts criminal activity based on financial data.
Bijv. een algoritme kan leren dat de kans op bepaald crimineel gedrag groter is bij grotere armoede. Maar legitimeert dat restrictieve en/of preventieve sancties zonder dat een misdaad is begaan? Armoede is geen misdaad.
- "One can think of such an assault on human dignity in two ways. The first is obvious: by giving machines authority over humans, we relegate ourselves to a second-class status and lose the right to participate in decisions that affect us. (A more extreme form of this is giving machines the authority to kill humans, as discussed earlier in the chapter.) The second is indirect: even if you believe it is not the machines making the decision but those humans who designed and commissioned the machines, the fact that those human designers and commissioners do not consider it worthwhile to weigh the individual circumstances of each human subject in such cases suggests that they attach little value to the lives of others. This is perhaps a symptom of the beginning of a great separation between an elite served by humans and a vast underclass served, and controlled, by machines." --- 0.348, 2020-05-07
Robot decision-making and human dignity
-
"The EU’s GDPR is often said to provide a general “right to an explanation” for any automated decision,38 but the actual language of Article 14 merely requires
meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject.
At present, it is unknown how courts will enforce this clause. It’s possible that the hapless consumer will just be handed a description of the particular deep learning algorithm used to train the classifier that made the decision." --- 0.351, 2020-05-07
Is GDPR art. 14 enforcable?
-
"The mathematical analysis of these definitions of fairness shows that they cannot be achieved simultaneously and that, when enforced, they result in lower prediction accuracy and, in the case of lending decisions, lower profit for the lender. This is perhaps disappointing, but at least it makes clear the trade-offs involved in avoiding algorithmic bias." --- 0.354, 2020-05-07
-
"The question is whether the computer system remains a tool of humans, or humans become tools of the computer system—supplying information and fixing bugs when necessary, but no longer understanding in any depth how the whole thing is working. The answer becomes clear when the system goes down and global chaos ensues until it can be brought back online. For example, a single “computer glitch” on April 3, 2018, caused fifteen thousand flights in Europe to be significantly delayed or canceled.40 When trading algorithms caused the 2010 “flash crash” on the New York Stock Exchange, wiping out \$1 trillion in a few minutes, the only solution was to shut down the exchange. What happened is still not well understood." --- 0.358, 2020-05-07
Is het antwoord zo duidelijk? Dit toont de afhankelijkheid van mensen op (semi-)autonome computersystemen, maar ook andwrsom (wanneer systemen crashwn)
Chapter 5: Overly Intelligent AI
TODO Directly from annotations:
-
"Samuel Butler’s novel Erewhon, published in 1872, developed the theme in far greater depth" --- 0.361, 2020-05-07
-
"I’ll call this the gorilla problem—specifically, the problem of whether humans can maintain their supremacy and autonomy in a world that includes machines with substantially greater intelligence." --- 0.361, 2020-05-07
Def.
-
"In 1847, however, Richard Thornton, editor of the Primitive Expounder, a religious journal, railed against mechanical calculators:2
Mind . . . outruns itself and does away with the necessity of its own existence by inventing machines to do its own thinking. . . . But who knows that such machines when brought to greater perfection, may not think of a plan to remedy all their own defects and then grind out ideas beyond the ken of mortal mind!" --- 0.361, 2020-05-07
First worry about thinking machines (perhaps)
- "The only approach that seems likely to work is to understand why it is that making better AI might be a bad thing. It turns out that we have known the answer for thousands of years." --- 0.370, 2020-05-07
This section gave reasons why banning AI research is not a feasible solution to the Gorilla problem
-
"Until recently, we were shielded from the potentially catastrophic consequences by the limited capabilities of intelligent machines and the limited scope that they have to affect the world." --- 0.373, 2020-05-07
-
"We might call this the King Midas problem: Midas, a legendary king in ancient Greek mythology, got exactly what he asked for—namely, that everything he touched should turn to gold. Too late, he discovered that this included his food, his drink, and his family members, and he died in misery and starvation. The same theme is ubiquitous in human mythology. Wiener cites Goethe’s tale of the sorcerer’s apprentice, who instructs the broom to fetch water—but doesn’t say how much water and doesn’t know how to make the broom stop. A technical way of saying this is that we may suffer from a failure of value alignment—we may, perhaps inadvertently, imbue machines with objectives that are imperfectly aligned with our own." --- 0.373, 2020-05-07
Norbert Wiener realized the impossibility of correctly specifying human values. King Midas expressed his wish and we know the consequences.
- "For example, it might reduce our energy consumption by persuading us to have fewer children, eventually—and inadvertently—achieving the dreams of anti-natalist philosophers who wish to eliminate the noxious impact of humanity on the natural world." --- 0.379, 2020-05-08
Mogelijke nudge door machines
-
"With a bit of practice, you can learn to identify ways in which the achievement of more or less any fixed objective can result in arbitrarily bad outcomes. One of the most common patterns involves omitting something from the objective that you do actually care about. In such cases—as in the examples given above—the AI system will often find an optimal solution that sets the thing you do care about, but forgot to mention, to an extreme value." --- 0.379, 2020-05-08
-
"Unfortunately, with superintelligent systems that can have a global impact, there are no simulators and no do-overs." --- 0.379, 2020-05-08
In het geval van superintelligence zijn de gevolgen heel moeilijk te anticiperen, twrwijl de potentiele impact groot em globaal is. Dat is een giftige cocktail.
- "There’s really not a lot you can do once you’re dead, so we can expect AI systems to act preemptively to preserve their own existence, given more or less any definite objective." --- 0.382, 2020-05-08
AI self-preservation; why "turning off" is not a solution. Would an AI know the concept of its own death?
- "It is important to understand that self-preservation doesn’t have to be any sort of built-in instinct or prime directive in machines. (So Isaac Asimov’s Third Law of Robotics,8 which begins “A robot must protect its own existence,” is completely unnecessary.) There is no need to build self-preservation in because it is an instrumental goal—a goal that is a useful subgoal of almost any original objective.9 Any entity that has a definite objective will automatically act as if it also has instrumental goals." --- 0.385, 2020-05-08
Self-preservation does not need to be given because it is a subgoal for any type of objective
- "In 1965, he wrote what is now his best-known paper, “Speculations Concerning the First Ultraintelligent Machine.”11 The first sentence suggests that Good, alarmed by the nuclear brinkmanship of the Cold War, regarded AI as a possible savior for humanity: “The survival of man depends on the early construction of an ultraintelligent machine.” As the paper proceeds, however, he becomes more circumspect. He introduces the notion of an intelligence explosion, but, like Butler, Turing, and Wiener before him, he worries about losing control" --- 0.388, 2020-05-08
I.J. Good
- "Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside science fiction." --- 0.388, 2020-05-08
Interessante toevoeging
-
"The possibility of an intelligence explosion is often cited as the main source of risk to humanity from AI because it would give us so little time to solve the control problem." --- 0.391, 2020-05-08
-
"“Logically possible” and “I’m willing to bet the future of the human race on it” are, of course, two completely different things. Betting against human ingenuity seems like a losing strategy." --- 0.391, 2020-05-08
Not sure!
- "This view was promulgated by the roboticist and futurist Hans Moravec,14 who writes, “The immensities of cyberspace will be teeming with unhuman superminds, engaged in affairs that are to human concerns as ours are to those of bacteria.” This seems to be a mistake. Value, for humans, is defined primarily by conscious human experience. If there are no humans and no other conscious entities whose subjective experience matters to us, there is nothing of value occurring." --- 0.395, 2020-05-08
Resignation to AI, example
- "The possible responses to this anxiety seem to be to retreat from AI research, to deny that there are risks inherent in developing advanced AI, to understand and mitigate the risks through the design of AI systems that necessarily remain under human control, and to resign—simply to cede the future to intelligent machines." --- 0.395, 2020-05-08
Four possible responses to the anxiety about losing control over AI