The Human in the Loop

Everyone agrees a person must stay in the loop before a machine kills. Through the first weeks of the war in Gaza, one did. He had about twenty seconds, and the question he had been left to answer was whether the voice sounded male.

A course on the thought of Autonomous Warfare.

Philosophy · Ethics

What you will learn

  • State what the human review of a Lavender-marked target actually consisted of, and on whose testimony.
  • Explain why "a human was in the loop" and "a human decided" are not the same claim.
  • Explain why Wiener's two-time-scales argument makes the twenty seconds structural rather than negligent.
  • State what Wiener says happens to a person who is used as an element of a machine.
  • State the reported collateral damage degrees and how they compare to recent US practice.
  • State Anscombe's criterion precisely, including what she says is NOT murder.
  • Describe what Lavender was reported to score, on whom, and on what kind of features.
  • Explain why the objection Kant supplies is not an objection about accuracy.

Course outline

  1. Twenty Seconds — In the first weeks of the war in Gaza an Israeli intelligence officer had about twenty seconds to approve each name a machine put in front of him. What he was checking in those twenty seconds is the…
  2. The Man Who Built the First One — The problem of a machine that decides faster than a person can object was solved, named and warned about by the mathematician who invented automated targeting — in a paper about anti-aircraft guns…
  3. A Number Written in Advance — Before any officer sat down with any name, someone had decided how many civilians a given class of target was worth. Anscombe's 1957 definition of murder turns out to hinge not on that number but on…
  4. A Rating from 1 to 100 — Before the strike and before the directive, a machine scored a population. Every criticism of that scoring in general circulation is an accuracy criticism, and Kant supplies the only one that would…
  5. What "Ninety Percent" Does Not Mean — The technical guts of the thing, and the module where the tracking condition earns its keep. A number everyone repeats turns out not to be a specification at all, and the most damning fact about the…
  6. Nobody Holds Both — The second condition, taken at its word — including the clause everyone skips over, which is the one the black box defeats. Then the thing the whole instrument cannot do, and it is the reason to be…
  7. Rule by Nobody — Who has bought the same arrangement since, and then the sentence to leave holding — that a gap where responsibility should be is not a hole in a form of government but a form of government. And one…

Works read in this course

  • 'Lavender': The AI machine directing Israel's bombing spree in Gaza — Yuval Abraham's investigation for +972 Magazine and Local Call, 3 April 2024, built on six Israeli intelligence officers with first-hand involvement. Co-published with the Guardian.
  • 'A mass assassination factory' — The earlier Abraham investigation, 30 November 2023, which revealed the Gospel — the system that marks buildings rather than people.
  • The Human Use of Human Beings — Wiener's 1950 book for a general readership, revised 1954, written after he had refused to hand his wartime targeting research to a weapons manufacturer.
  • God and Golem, Inc. — Wiener's last book, 1964, on what it means to make a machine that learns. It contains his war-game argument.
  • Mr Truman's Degree — Anscombe's 1957 pamphlet opposing Oxford's honorary degree for the man who ordered the atomic bombings. It contains her definition of murder.
  • Meaningful Human Control over Autonomous Systems — Santoni de Sio and van den Hoven's 2018 paper in Frontiers in Robotics and AI. Open access — the reader can go and read it today, for nothing.
  • Crises of the Republic — Arendt's 1972 collection, containing 'On Violence' and its account of rule by Nobody.

Terms introduced

meaningful human control, the tracking condition, the tracing condition, responsibility gap, moral crumple zone, automation bias, the doctrine of double effect, collateral damage degree, the two time scales

Part of the Simposeum course library.