Ethics for Machines – Kobo eBook

A hospital algorithm ranked patients by how much money they were expected to cost, and everyone involved believed it was ranking them by how sick they were.

The model was Optum’s care-management system, used across American hospitals. In October 2019 a team publishing in Science found that at the same level of illness it was spending about $1,800 a year less on Black patients than on white ones, and that fixing the variable it predicted lifted Black patients among those selected for extra care from 17.7% to 46.5%. Race was never an input. The harm came in through the thing the model was asked to measure.

That is the pattern this book follows, case by case. Two teams audit the same risk score used in Florida courts and publish opposite, mathematically defensible conclusions. Robert Williams is arrested in January 2020 in the driveway of his house in Michigan, in front of his wife and his two daughters, on a facial match to a store video. In England, a grading model lowers 39.1% of teacher assessments and raises 2.2%, with the falls landing on state schools. On an avenue in Tempe, Arizona, a test vehicle sees Elaine Herzberg 5.6 seconds before it hits her, calls her an object, then a vehicle, then a bicycle, with factory emergency braking switched off and a one-second wait written into the software.

By the end you will be able to read a specification the way an auditor does: name the moral claim hidden in an objective function before anything is trained, tell a dataset that is merely incomplete from one that has a past decision baked into its labels, say who pays the error at a given threshold and in what currency, and put a name against the moment a system should be allowed to answer that it does not know.

Most writing on this subject argues about principles and stops there. This one goes to the four or five places where the decision is actually made the target variable, the labels, the category, the threshold, the branch for doubt and shows that each of them was a choice somebody signed.