On Genies and Bottles: Scientists’ Moral Responsibility and Dangerous Technology R&D | David Koepsell

@ethics @scientific_responsibility

Abstract:

  • Against the idea that science is neutral and that blame for bad use rests solely on technologists and politicians
  • This reasoning is especially weak when you work in an area of science that may have catastrophic impact on whole biospheres
  • Aims to develop ethical principles for science in general, with an "extended moral horizon", based on ethical principles in the field of biomedicine.
  • Effects both on individuals and on the whole of humanity are considered
  • Offers utilitarian calculus for deciding which science projects to pursue and which ones not

Introduction

Presuppositions on moral implications of science:

  • "We can't put the genie back in the bottle"
  • Science have a duty to explore questions, but technologists are not required to release technology into the wild
    • So ethics is for technologists, more than for scientists, seems to be the assumption

Research questions:

  • Do scientists share responsibility of applications of their research by technologists?
  • Does this responsibility involve moral culpability?
  • And "is there any moral duty on the part of individual scientists to simply say "no" to investigating those things whose only or best uses are to cause harms?" (120)

Scientific firewall

It seems that the arguments of scientists claiming neutrality works best with "dual-use" scientific subjects, e.g. nuclear fusion (energy provision or nuclear bomb?).

In bioethics ethical duties for scientists have been established that outweigh the intellectual scientific endeavor to find natural laws. This is in cases where the science may affect the behaviors of many humans.

Bioethics

  • After Nazi experiment on humans: Nuremberg Code
  • But existing codes were regularly ignored by scientists, allowing human subjects "to suffer in the name of science" (123)
    • E.g. Edward Jenner infected a kid with small pox without proper control to test his vaccine (and he was lucky that it worked)
  • Scientists do not always self-regulate and therefore institutions for enforcement were required
  • This regulation currently only applies for science with direct interaction with human subjects

Main point:

  • This should be extended to science that does not directly deal with human individual subjects, but nevertheless affects the whole of humanity.

These examples shown that immorality does not only apply to the technical application of science: "sometimes bad things are done in the name of science itself, well before the application phase of a new technology."

This then counteracts the Presuppositions of the "dual-use" model with "neutral" science and non-neutral application.

Respect, Beneficence, Justice

Intention of the paper is to extend the Belmont report to scientific research that effects humanity as a whole.

The Belmont principles are "thoroughly pragmatic" (124)

  1. Respect for persons
    • Kantian notion: treat people as ends and with human dignity, not as means to an end
  2. Benificence
    • Human subject research must "be conduced with good intentions" (124)
    • Not just for curiosity
  3. Justice
    • Vulnerable population groups must especially be protected (e.g. children or minorities)

These are clearly not limited to bioethics, why are they then not applied that explicitly in other fields of science as well? Probably because they lack huge scandals involving human subjects.

Nice Kurt Vonnegut reference to Ice-9: a catastrophic discovery wiping out humanity that however would not have been controlled by the Belmon Report. But it should have!

Repetition of main questions:

Is the beauty and truth of science justification enough to investigate even the deadliest potential technologies or discoveries? Are there ethical principles that bind individual scientists in the absence of regulatory institutions? Can the Belmont principles be extended to scientists doing research only indirectly involving human subjects where the potential effects of an avenue of study impact humanity as a whole? (126)

Extending the moral horizon

Can scientific inquiry be subject to moral culpability?

  • Perhaps inquiry is a personal freedom, and does limiting it risk being censorship and policing of research
  • "But just because we don’t want government or regulators overseeing the actions of an individual, doesn’t mean that that person’s actions or even intentions, are free from moral scrutiny." (126)
  • You do not limit all research, only particular kinds (cf. how indeed use of artefacts e.g. a knife is a person's responsibility; nevertheless we are not allowed to have automatic guns).
  • "Intentions matter, even when intention alone is not enough to spur public regulation" (127)
    • This point is made in the context of child porn: owning it is already bad even though you didn't abuse a child yourself.
  • The main point is that even though free scientific thought should not be externally constricted, it is not free and immune to moral judgment (127)
  • We should educate scientist to self-regulate and have this extended moral horizon (127)
    • This goes beyond codes of conduct and laws
    • Matter of individual duty

We should take account of a broader moral horizon when considering the ethics and morality of scientists, just as we do with other professionals, and ask whether and when intentions and positive actions on those intentions trigger an individual duty to refrain, and subsequent moral judgment by others, even where the law or regulations ought not to be invoked. (127-128)

Smallpox, Ice-Nine, Dangerous Things

  • The "dual-use" debate is a red herring because any invention, however trivial, can be construed as having a "dual-use"
  • This then weakens the move where you say "that particular use" is not my responsibility, it can also be used for good. The point is, this becomes a useless criterium as an argument for when to take moral responsibility.

In the case where the last stock of small pocks was preserved, with all its dumb consequences... could the Belmont principles have helped?

The "eventual" fallacy and a calculus for determining its soundness

An argument that is often used to justify these sorts of scientific inquiries is that "someone will devise the technologies, and employ them harmfully—eventually. Thus, we should investigate things first (because we have good intentions)." (129-130)

Presuppositions:

  1. "investigators doing the work have good intentions"
  2. "the technology or discovery would eventually be carried out by others"
  3. "once discovered or applied, it can be contained." (130)

  4. This reasoning justifies whatever scientific inquiry with whatever potential consequences

  5. But this reasoning does no longer work when we include humanity as a whole into the Belmont principles

The eventual fallacy can only be an argument for research if

  • The likelihood of the discovery being made by others and being misused ...
    • the likelihood that a cure/fix can be created based on research done right now ...
  • Outweighs the risk of doing research.

The "right now" has emphasis because:

But a generalized threat posed by the potential of someone acquiring knowledge or technology somewhere at some time means that this calculus requires scientific caution. For sufficiently deadly inquiries or applications, scientists should perceive a duty to refrain at least from disseminating certain types of knowledge, if not necessarily from theoretical inquiry alone (unless that inquiry may reasonably be justified by the above calculus). The ‘‘eventual’’ fallacy is committed by simple recourse to the truism that over an infinite time-span, all natural truths will be discovered, and all potential uses and misuses of technology will be developed, so present research on any science or application of technology is morally justified (131)

Conclusion

Of course, mens rea must be considered, but merely citing the ‘‘eventual’’ fallacy will not suffice to defend all scientific inquiry and its resultant dissemination, either through publication or technology. Scientists must not only have a sense that they are morally culpable for the uses of their discoveries where they understand the risk—harm—likelihood calculus, but they must also be liable to be held culpable where harms result from their acts, and where they possess a culpable mens rea.

Individual responsibility hand in hand with institutions for education.

Belmont expansion: "we are all human subjects of certain inquiries" (132)

Respect, beneficence, and justice apply not only to human subjects of particular experiments, but more generally to humanity as a whole. The result of all this should be better trust of scientists and their profession, and a greater realization on the part of scientists that their work proceeds through mutual trust and appreciation between scientists and the public. (133)