Penguin Press National Pub date: February 19, 2019 Title: DEEP THINKING Subtitle: 7wenty-Five Ways of Looking at AI By: John Brockman Length: 90,000 words Headline: Science world luminary John Brockman assembles twenty-five of the most important scientific minds, people who have been thinking about the field artificial intelligence for most of their careers for an unparalleled round-table examination about mind, thinking, intelligence and what it means to be human. Description: "Artificial intelligence is today's story—the story behind all other stories. It is the Second Coming and the Apocalypse at the same time: Good AI versus evil AI." —John Brockman More than sixty years ago, mathematician-philosopher Norbert Wiener published a book on the place of machines in society that ended with a warning: “we shall never receive the right answers to our questions unless we ask the right questions.... The hour is very late, and the choice of good and evil knocks at our door.” In the wake of advances in unsupervised, self-improving machine learning, a small but influential community of thinkers is considering Wiener’s words again. In Deep Thinking, John Brockman gathers their disparate visions of where AI might be taking us. The fruit of the long history of Brockman’s profound engagement with the most important scientific minds who have been thinking about AI—from Alison Gopnik and David Deutsch to Frank Wilczek and Stephen Wolfram— Deep Thinking 1s an ideal introduction to the landscape of crucial issues AI presents. The collision between opposing perspectives is salutary and exhilarating; some of these figures, such as computer scientist Stuart Russell, Skype co-founder Jaan Tallinn, and physicist Max Tegmark, are deeply concerned with the threat of AI, including the existential one, while others, notably robotics entrepreneur Rodney Brooks, philosopher Daniel Dennett, and bestselling author Steven Pinker, have a very different view. Serious, searching and authoritative, Deep Thinking lays out the intellectual landscape of one of the most important topics of our time. HOUSE_OVERSIGHT_016221

Participants in The Deep Thinking Project Chris Anderson is an entrepreneur; a roboticist; former editor-in-chief of Wired; co- founder and CEO of 3DR; and author of Zhe Long Tail, Free, and Makers. Rodney Brooks is a computer scientist; Panasonic Professor of Robotics, emeritus, MIT; former director, MIT Computer Science Lab; and founder, chairman, and CTO of Rethink Robotics. He is the author of //esh and Machines. George M. Church is Robert Winthrop Professor of Genetics at Harvard Medical School; Professor of Health Sciences and Technology, Harvard-MIT; and co-author (with Ed Regis) of Regenesis: How Synthetic Biology Will Reinvent Nature and Ourselves. Daniel C. Dennett is University Professor and Austin B. Fletcher Professor of Philosophy and director of the Center for Cognitive Studies at Tufts University. He is the author of a dozen books, including Consciousness Explained and, most recently, From Bacteria to Bach and Back: The Evolution of Minds. David Deutsch is a quantum physicist and a member of the Centre for Quantum Computation at the Clarendon Laboratory, Oxford University. He is the author of Zhe Fabric of Reality and The Beginning of Infinity. Anca Dragan is an assistant professor in the Department of Electrical Engineering and Computer Sciences at UC Berkeley. She co-founded and serves on the steering committee for the Berkeley AI Research (BAIR) Lab and is a co-principal investigator in Berkeley’s Center for Human-Compatible AI. George Dyson is a historian of science and technology and the author of Baidarka: the Kayak, Darwin Among the Machines, Project Orion, and Turing ’s Cathedral. Peter Galison is a science historian, Joseph Pellegrino University Professor and co- founder of the Black Hole Initiative at Harvard University, and the author of Einstein's Clocks and Poincaré ’s Maps: Empires of Time. Neil Gershenfeld is a physicist and director of MIT’s Center for Bits and Atoms. He is the author of AB, co-author (with Alan Gershenfeld & Joel Cutcher-Gershenfeld) of Designing Reality, and founder of the global fab lab network. Alison Gopnik is a developmental psychologist at UC Berkeley; her books include The Philosophical Baby and, most recently, Zhe Gardener and the Carpenter: What the New Science of Child Development Tells Us About the Relationship Between Parents and Children. HOUSE_OVERSIGHT_016222

Tom Griffiths is Henry R. Luce Professor of Information, Technology, Consciousness, and Culture at Princeton University. He is co-author (with Brian Christian) of Algorithms to Live By. W. Daniel “Danny” Hillis is an inventor, entrepreneur, and computer scientist, Judge Widney Professor of Engineering and Medicine at USC, and author of The Pattern on the Stone: The Simple Ideas That Make Computers Work. Caroline A. Jones is a professor of art history in the Department of Architecture at MIT and author of Eyesight Alone: Clement Greenberg’s Modernism and the Bureaucratization of the Senses; Machine in the Studio: Constructing the Postwar American Artist; and The Global Work of Art. David Kaiser is Germeshausen Professor of the History of Science and professor of physics at MIT, and head of its Program in Science, Technology & Society. He is the author of How the Hippies Saved Physics: Science, Counterculture, and the Quantum Revival and American Physics and the Cold War Bubble (forthcoming). Seth Lloyd is a theoretical physicist at MIT, Nam P. Suh Professor in the Department of Mechanical Engineering, and an external professor at the Santa Fe Institute. He is the author of Programming the Universe: A Quantum Computer Scientist Takes on the Cosmos. Hans Ulrich Obrist is artistic director of the Serpentine Gallery, London, and the author of Ways of Curating and Lives of the Artists, Lives of the Architects. Judea Pearl is professor of computer science and director of the Cognitive Systems Laboratory at UCLA. His most recent book, co-authored with Dana Mackenzie, is Zhe Book of Why: The Alex “Sandy” Pentland is Toshiba Professor and professor of media arts and sciences, MIT; director of the Human Dynamics and Connection Science labs and the Media Lab Entrepreneurship Program, and the author of Social Physics. New Science of Cause and Effect. Steven Pinker, a Johnstone Family Professor in the Department of Psychology at Harvard University, is an experimental psychologist who conducts research in visual cognition, psycholinguistics, and social relations. He is the author of eleven books, including The Blank Slate, The Better Angels of Our Nature, and, most recently, Enlightenment Now: The Case for Reason, Science, Humanism, and Progress. Venki Ramakrishnan is a scientist at the Medical Research Council Laboratory of Molecular Biology, Cambridge University; recipient of the Nobel Prize in Chemistry (2009); current president of the Royal Society; and the author of Gene Machine: The Race to Discover the Secrets of the Ribosome. HOUSE_OVERSIGHT_016223

Stuart Russell is a professor of computer science and Smith-Zadeh Professor in Engineering at UC Berkeley. He is the coauthor (with Peter Norvig) of Artificial Intelligence: A Modern Approach. Jaan Tallin, a computer programmer, theoretical physicist, and investor, 1s a co- developer of Skype and Kazaa. Max Tegmark is an MIT physicist and AI researcher; president of the Future of Life Institute; scientific director of the Foundational Questions Institute; and the author of Our Mathematical Universe and Life 3.0: Being Human in the Age of Artificial Intelligence. Frank Wilczek is Herman Feshbach Professor of Physics at MIT, recipient of the 2004 Nobel Prize in physics, and the author of A Beautiful Question: Finding Nature’s Deep Design. Stephen Wolfram 1s a scientist, inventor, and the founder and CEO of Wolfram Research. He is the creator of the symbolic computation program Mathematica and its programming language, Wolfram Language, as well as the knowledge engine Wolfram|Alpha. He is also the author of A New Kind of Science. HOUSE_OVERSIGHT_016224

Deep Thinking Twenty-five Ways of Looking at AI edited by John Brockma Penguin Press — February 19, 2019 HOUSE_OVERSIGHT_016225

Table of Contents Acknowledgments Introduction: On the Promise and Peril of AI by John Brockman Artificial intelligence is today’s story—the story behind all other stories. It is the Second Coming and the Apocalypse at the same time: Good AI versus evil AI. This book comes out of an ongoing conversation with a number of important thinkers, both in the world of Al and beyond it, about what AI is and what it means. Called the Deep Thinking Project, this conversation began in earnest in September 2016, in a meeting at the Mayflower Grace Hotel in Washington, Connecticut with some of the book's contributors. What quickly emerged from that first meeting is that the excitement and fear in the wider culture surrounding AI now has an analogue in the way Norbert Wiener’s ideas regarding “cybernetics” worked their way through the culture, particularly in the 1960's, as artists began to incorporate thinking about new technologies into their work. I witnessed the impact of those ideas at close hand; indeed it’s not too much to say they set me off on my life’s path. With the advent of the digital era beginning in the early 1970s, people stopped talking about Wiener, but today, his Cybernetic Idea has been so widely adopted that it’s internalized to the point where it no longer needs a name. It’s everywhere, it’s in the air, and it’s a fitting a place to begin. Seth Lloyd: Wrong, but More Relevant Than Ever It is exactly in the extension of the cybernetic idea to human beings that Wiener’s conceptions missed their target. Judea Pearl: The Limitations of Opaque Learning Machines Deep learning has its own dynamics, it does its own repair and its own optimization, and it gives you the right results most of the time. But when it doesn’t, you don’t have a clue about what went wrong and what should be fixed. Stuart Russell: The Purpose Put Into the Machine We may face the prospect of superintelligent machines—their actions by definition unpredictable by us and their imperfectly specified objectives conflicting with our own— whose motivation to preserve their existence in order to achieve those objectives may be insuperable. George Dyson: The Third Law Any system simple enough to be understandable will not be complicated enough to behave intelligently, while any system complicated enough to behave intelligently will be too complicated to understand. Daniel C. Dennett: What Can We Do? We don’t need artificial conscious agents. We need intelligent tools. HOUSE_OVERSIGHT_016226

Rodney Brooks: The Inhuman Mess Our Machines Have Gotten Us Into We are ina much more complex situation today than Wiener foresaw, and I am worried that it is much more pernicious than even his worst imagined fears. Frank Wilczek: The Unity of Intelligence The advantages of artificial over natural intelligence appear permanent, while the advantages of natural over artificial intelligence, though substantial at present, appear transient. Max Tegmark: Let’s Aspire to More Than Making Ourselves Obsolete We should analyze what could go wrong with AI to ensure that it goes right. Jaan Tallinn: Dissident Messages Continued progress in AI can precipitate a change of cosmic proportions—a runaway process that will likely kill everyone. Steven Pinker: Tech Prophecy and the Underappreciated Causal Power of Ideas There is no law of complex systems that says that intelligent agents must turn into ruthless megalomaniacs. David Deutsch: Beyond Reward and Punishment Misconceptions about human thinking and human origins are causing corresponding misconceptions about AGI and how it might be created. Tom Griffiths: The Artificial Use of Human Beings Automated intelligent systems that will make good inferences about what people want must have good generative models for human behavior. Anca Dragan: Putting the Human into the AI Equation In the real world, an AI must interact with people and reason about them. People will have to formally enter the AI problem definition somewhere. Chris Anderson: Gradient Descent Just because Al systems sometimes end up in local minima, don’t conclude that this makes them any less like life. Humans—indeed, probably all life-forms—are often stuck in local minima. David Kaiser: “Information” for Wiener, for Shannon, and for Us Many of the central arguments in The Human Use of Human Beings seem closer to the 19th century than the 21st. Wiener seems not to have fully embraced Shannon’s notion of information as consisting of irreducible, meaning-free bits. Neil Gershenfeld: Scaling Although machine making and machine thinking might appear to be unrelated trends, they lie in each other’s futures. HOUSE_OVERSIGHT_016227

W. Daniel Hillis: The First Machine Intelligences Hybrid superintelligences such as nation states and corporations have their own emergent goals and their actions are not always aligned to the interests of the people who created them. Venki Ramakrishnan: Will Computers Become Our Overlords? Our fears about AI reflect the belief that our intelligence is what makes us special. Alex “Sandy” Pentland: The Human Strategy How can we make a good human-artificial ecosystem, something that’s not a machine society but a cyberculture in which we can all live as humans—a culture with a human feel to it? Hans Ulrich Obrist: Making the Invisible Visible: Art Meets AI Many contemporary artists are articulating various doubts about the promises of AI and reminding us not to associate the term “artificial intelligence” solely with positive outcomes. Alison Gopnik: AIs versus Four-Year-Olds Looking at what children do may give programmers useful hints about directions for computer learning. Peter Galison: Algorists Dream of Objectivity By now, the legal, ethical, formal, and economic dimensions of algorithms are all quasi- infinite. George M. Church: The Rights of Machines Probably we should be less concerned about us-versus-them and more concerned about the rights of all sentients in the face of an emerging unprecedented diversity of minds. Caroline A. Jones: The Artistic Use of Cybernetic Beings The work of cybernetically inclined artists concerns the emergent behaviors of life that elude AI in its current condition. Stephen Wolfram: Artificial Intelligence and the Future of Civilization The most dramatic discontinuity will surely be when we achieve effective human immortality. Whether this will be achieved biologically or digitally isn’t clear, but inevitably it will be achieved. HOUSE_OVERSIGHT_016228

Introduction: On the Promise and Peril of AI John Brockman Artificial intelligence is today’s story—the story behind all other stories. It is the Second Coming and the Apocalypse at the same time: Good AI versus evil AI. This book comes out of an ongoing conversation with a number of important thinkers, both in the world of AI and beyond it, about what AI is and what it means. Called the Deep Thinking Project, this conversation began in earnest in September 2016, in a meeting at the Mayflower Grace Hotel in Washington, Connecticut with some of the book’s contributors. What quickly emerged from that first meeting is that the excitement and fear in the wider culture surrounding AI now has an analogue in the way Norbert Wiener’s ideas regarding “cybernetics” worked their way through the culture, particularly in the 1960’s, as artists began to incorporate thinking about new technologies into their work. I witnessed the impact of those ideas at close hand; indeed it’s not too much to say they set me off on my life’s path. With the advent of the digital era beginning in the early 1970s, people stopped talking about Wiener, but today, his Cybernetic Idea has been so widely adopted that it’s internalized to the point where it no longer needs a name. It’s everywhere, it’s in the air, and it’s a fitting a place to begin. New Technologies=New Perceptions Before AI, there was Cybernetics—the idea of automatic, self-regulating control, laid out in Norbert Wiener’s foundational text of 1948. I can date my own serious exposure to it to 1966, when the composer John Cage invited me and four or five other young arts people to join him for a series of dinners—an ongoing seminar about media, communications, art, music, and philosophy that focused on Cage’s interest in the ideas of Wiener, Claude Shannon, and Marshall McLuhan, all of whom had currency in the New York art circles in which I was then moving. In particular, Cage had picked up on McLuhan’s idea that by inventing electronic technologies we had externalized our central nervous system—that is, our minds—and that we now had to presume that “there’s only one mind, the one we all share.” Ideas of this nature were beginning to be of great interest to the artists I was working with in New York at the Film-Makers’ Cinémathéque, where I was program manager for a series of multimedia productions called the New Cinema | (also known as the Expanded Cinema Festival), under the auspices of avant-garde filmmaker and impresario Jonas Mekas. They included visual artists Claes Oldenburg, Robert Rauschenberg, Andy Warhol, Robert Whitman; kinetic artists Charlotte Moorman and Nam June Paik; happenings artists Allan Kaprow and Carolee Schneemann; dancer Tricia Brown; filmmakers Jack Smith, Stan Vanderbeek, Ed Emshwiller, and the Kuchar brothers; avant-garde dramatist Ken Dewey; poet Gerd Stern and the USCO group; minimalist musicians Lamonte Young and Terry Riley; and through Warhol, the music group, The Velvet Underground. Many of these people were reading Wiener, and cybernetics was in the air. It was at one of these dinners that Cage reached into his briefcase and took out a copy of Cybernetics and handed it to me, saying, “This is for 99 you. HOUSE_OVERSIGHT_016229

During the Festival, I received an unexpected phone call from Wiener’s colleague Arthur K. Solomon, head of Harvard’s graduate program in biophysics. Wiener had died the year before, and Solomon and Wiener’s other close colleagues at MIT and Harvard had been reading about the Expanded Cinema Festival in the New York Times and were intrigued by the connection to Wiener’s work. Solomon invited me to bring some of the artists up to Cambridge to meet with him and a group that included MIT sensory- communications researcher Walter Rosenblith, Harvard applied mathematician Anthony Oettinger, and MIT engineer Harold “Doc” Edgerton, inventor of the strobe light. Like many other “art meets science” situations I’ve been involved in since, the two-day event was an informed failure: ships passing in the night. But I took it all onboard and the event was consequential in some interesting ways—one of which came from the fact that they took us to see “the” computer. Computers were a rarity back then; at least, none of us on the visit had ever seen one. We were ushered into a large space on the MIT campus, in the middle of which there was a “cold room” raised off the floor and enclosed in glass, in which technicians wearing white lab coats, scarves, and gloves were busy collating punch cards coming through an enormous machine. When I approached, the steam from my breath fogged up the window into the cold room. Wiping it off, I saw “the” computer. I fell in love. Later, in the Fall of 1967, I went to Menlo Park to spend time with Stewart Brand, whom I had met in New York in 1965 when he was a satellite member of the USCO group of artists. Now, with his wife Lois, a mathematician, he was preparing the first edition of The Whole Earth Catalog for publication. While Lois and the team did the heavy lifting on the final mechanicals for WEC, Stewart and I sat together in a corner for two days, reading, underlining, and annotating the same paperback copy of Cybernetics that Cage had handed to me the year before, and debating Wiener’s ideas. Inspired by this set of ideas, I began to develop a theme, a mantra of sorts, that has informed my endeavors since: “new technologies = new perceptions.” Inspired by communications theorist Marshall McLuhan, architect-designer Buckminster Fuller, futurist John McHale, and cultural anthropologists Edward T. (Ned) Hall and Edmund Carpenter, I started reading avidly in the field of information theory, cybernetics, and systems theory. McLuhan suggested I read biologist J.Z. Young’s Doubt and Certainty in Science in which he said that we create tools and we mold ourselves through our use of them. The other text he recommended was Warren Weaver and Claude Shannon’s 1949 paper “Recent Contributions to the Mathematical Theory of Communication,” which begins: “The word communication will be used here in a very broad sense to include all of the procedures by which one mind may affect another. This, of course, involves not only written and oral speech, but also music, the pictorial arts, the theater, the ballet, and in fact all human behavior." Who knew that within two decades of that moment we would begin to recognize the brain as a computer? And in the next two decades, as we built our computers into the Internet, that we would begin to realize that the brain is not a computer, but a network of computers? Certainly not Wiener, a specialist in analogue feedback circuits designed to control machines, nor the artists, nor, least of all, myself. “We must cease to kiss the whip that lashes us.” 10 HOUSE_OVERSIGHT_016230

Two years after Cybernetics, in 1950, Norbert Wiener published Zhe Human Use of Human Beings—a deeper story, in which he expressed his concerns about the runaway commercial exploitation and other unforeseen consequences of the new technologies of control. I didn’t read The Human Use of Human Beings until the spring of 2016, when I picked up my copy, a first edition, which was sitting in my library next to Cybernetics. What shocked me was the realization of just how prescient Wiener was in 1950 about what’s going on today. Although the first edition was a major bestseller—and, indeed, jump-started an important conversation—under pressure from his peers Wiener brought out a revised and milder edition in 1954, from which the original concluding chapter, “Voices of Rigidity,” 1s conspicuously absent. Science historian George Dyson points out that in this long-forgotten first edition, Wiener predicted the possibility of a “threatening new Fascism dependent on the machine a gouverner’’: No elite escaped his criticism, from the Marxists and the Jesuits (“all of Catholicism is indeed essentially a totalitarian religion”) to the FBI (“our great merchant princes have looked upon the propaganda technique of the Russians, and have found that it is good”) and the financiers lending their support “to make American capitalism and the fifth freedom of the businessman supreme throughout the world.” Scientists... received the same scrutiny given the Church: “Indeed, the heads of great laboratories are very much like Bishops, with their association with the powerful in all walks of life, and the dangers they incur of the carnal sins of pride and of lust for power.” This jeremiad did not go well for Wiener. As Dyson puts it: These alarms were discounted at the time, not because Wiener was wrong about digital computing but because larger threats were looming as he completed his manuscript in the fall of 1949. Wiener had nothing against digital computing but was strongly opposed to nuclear weapons and refused to join those who were building digital computers to move forward on the thousand-times-more-powerful hydrogen bomb. Since the original of The Human Use of Human Beings is now out of print, lost to us is Wiener’s cri de coeur, more relevant today than when he wrote it, sixty-eight years ago: “We must cease to kiss the whip that lashes us.” Mind, Thinking, Intelligence Among the reasons we don’t hear much about “Cybernetics” today, two are central: First, although Zhe Human Use of Human Beings was considered an important book in its time, it ran counter to the aspirations of many of Wiener’s colleagues, including John von Neumann and Claude Shannon, who were interested in the commercialization of the new technologies. Second, computer pioneer John McCarthy disliked Wiener and refused to use Wiener’s term “Cybernetics.” McCarthy, in turn, coined the term “artificial intelligence” and became a founding father of that field. 11 HOUSE_OVERSIGHT_016231

As Judea Pearl, who, in the 1980s, introduced a new approach to artificial intelligence called Bayesian networks, explained to me: What Wiener created was excitement to believe that one day we are going to make an intelligent machine. He wasn't a computer scientist. He talked feedback, he talked communication, he talked analog. His working metaphor was a feedback circuit, which he was an expert in. By the time the digital age began in the early 1960s people wanted to talk programming, talk codes, talk about computational functions, talk about short-term memory, long-term memory— meaningful computer metaphors. Wiener wasn’t part of that, and he didn’t reach the new generation that germinated with his ideas. His metaphors were too old, passé. There were new means already available that were ready to capture the human imagination.” By 1970, people were no longer talking about Wiener. One critical factor missing in Wiener’s vision was the cognitive element: mind, thinking, intelligence. As early as 1942, at the first of a series of foundational interdisciplinary meetings about the control of complex systems that would come to be known as the Macy conferences, leading researchers were arguing for the inclusion of the cognitive element into the conversation. While von Neumann, Shannon, and Wiener were concerned about systems of control and communication of observed systems, Warren McCullough wanted to include mind. He turned to cultural anthropologists Gregory Bateson and Margaret Mead to make the connection to the social sciences. Bateson in particular was increasingly talking about patterns and processes, or “the pattern that connects.” He called for a new kind of systems ecology in which organisms and the environment in which they live are one in the same, and should be considered as a single circuit. By the early 1970s the Cybernetics of observed systems—1* order Cybernetics— moved to the Cybernetics of observing systems—2"™ order Cybernetics—or “the Cybernetics of Cybernetics”, as coined by Heinz von Foerster, who joined the Macy conferences in the mid 1950s, and spearheaded the new movement. Cybernetics, rather than disappearing, was becoming metabolized into everything, so we no longer saw it as a separate, distinct new discipline. And there it remains, hiding in plain sight. “The Shtick of the Steins” My own writing about these issues at the time was on the radar screen of the 2"4 order Cybernetics crowd, including Heinz von Foerster as well as John Lilly and Alan Watts, who were the co-organizers of something called "The AUM Conference," shorthand for “The American University of Masters”, which took place in Big Sur in 1973, a gathering of philosophers, psychologists, and scientists, each of whom asked to lecture on his own work in terms of its relationship to the ideas of British mathematician G. Spencer Brown presented in his book, Laws of Form. I was a bit puzzled when I received an invitation—a very late invitation indeed— which they explained was based on their interest in the ideas I presented in a book called Afterwords, which were very much on their wavelength. I jumped at the opportunity, the main reason being that the keynote speaker was none other than Richard Feynman. I love 12 HOUSE_OVERSIGHT_016232

to spend time with physicists, the reason being that they think about the universe, 1.e. everything. And no physicist was reputed to be articulate as Feynman. I couldn’t wait to meet him. I accepted. That said, I am not a scientist, and I had never entertained the idea of getting on a stage and delivering a “lecture” of any kind, least of all a commentary on an obscure mathematical theory in front of a group identified as the world’s most interesting thinkers. Only upon my arrival in Big Sur did I find out the reason for my very late invitation. “When is Feynman’s talk?” I asked at the desk. “Oh, didn’t Alan Watts tell you? Richard is ill and has been hospitalized. You’re his replacement. And, by the way, what’s the title of your keynote lecture?” I tried to make myself invisible for several days. Alan Watts, realizing that I was avoiding the podium, woke me up one night with a 3am knock on the door of my room. I opened the door to find him standing in front of me wearing a monk’s robe with a hood that covering much of his face. His arms extended, he held a lantern in one hand, and a magnum of scotch on the other. “John”, he said in a deep voice with a rich aristocratic British accent, “you are a phony.” “And, John”, he continued, Iam a phony. But John, Iam a real phony!” The next day I gave my lecture, entitled "Einstein, Gertrude Stein, Wittgenstein, and Frankenstein." Einstein: the revolution in 20" century physics; Gertrude Stein: the first writer who made integral to her work the idea of an indeterminate and discontinuous universe. Words represented neither character nor activity: A rose is a rose is a rose, and a universe is a universe is a universe. ); Wittgenstein: the world as limits of language. “The limits of my language mean the limits of my world”. The end of the distinction between observer and observed. Frankenstein: Cybernetics AI, robotics, all the essayists in this volume. The lecture had unanticipated consequences. Among the participants at the AUM Conference were several authors of #1 New York Times bestsellers, yet no one there had a literary agent. And I realized that all were engaged in writing a genre of book both unnamed and unrecognized by New York publishers. Since I had an MBA from Colombia Business School, and a series of relative successes in business, I was dragooned into becoming an agent, initially for Gregory Bateson and John Lilly, whose books I sold quickly, and for sums that caught my attention, thus kick-starting my career as a literary agent. I never did meet Richard Feynman. The Long AI Winters This new career put me in close touch with most of the AI pioneers, and over the decades I rode with them on waves of enthusiasm, and into valleys of disappointment. In the early ‘80s the Japanese government mounted a national effort to advance AI. They called it the 5“ Generation; their goal was to change the architecture of computation by breaking “the von Neumann bottleneck”, by creating a massively parallel computer. In so doing, they hoped to jumpstart their economy and become a dominant world power in the field. In1983, the leader of the Japanese 5“ Generation consortium came to New York for a meeting organized by Heinz Pagels, the president of the New York Academy of Sciences. I had a seat at the table alongside the leaders of the 1“ generation, Marvin Minsky and John McCarthy, the 2"! generation, Edward Feigenbaum 13 HOUSE_OVERSIGHT_016233

and Roger Schank, and Joseph Traub, head of the National Supercomputer Consortium. In 1981 with Heinz’s help, I had founded “The Reality Club” (the precursor to the non-profit Edge.org), whose initial interdisciplinary meetings took place in the Board Room at the NYAS. Heinz was working on his book, Dreams of Reason: The Rise of the Science of Complexity, which he considered to be a research agenda for science in the 1990's. Through the Reality Club meetings, I got to know two young researchers who were about to play key roles in revolutionizing computer science. At MIT in the late seventies, Danny Hillis developed the algorithms that made possible the massively parallel computer. In 1983, his company, Thinking Machines, built the world's fastest supercomputer by utilizing parallel architecture. His "connection machine," closely reflected the workings of the human mind. Seth Lloyd at Rockefeller University was undertaking seminal work in the fields of quantum computation and quantum communications, including proposing the first technologically feasible design for a quantum computer. And the Japanese? Their foray into artificial intelligence failed, and was followed by twenty years of anemic economic growth. But, the leading US scientists took this program very seriously. And Feigenbaum, who was the cutting-edge computer scientist of the day, teamed up with McCorduck to write a book on these developments. Zhe Fifth Generation: Artificial Intelligence and Japan's Computer Challenge to the World was published in 1983. We had a code name for the project: “It’s coming, it’s coming!” But it didn’t come; it went. From that point on ’ve worked with researchers in nearly every variety of AI and complexity, including Rodney Brooks, Hans Moravec, John Archibald Wheeler, Benoit Mandelbrot, John Henry Holland, Danny Hillis, Freeman Dyson, Chris Langton, Doyne Farmer, Geoffrey West, Stuart Russell, and Judea Pearl. An Ongoing Dynamical Emergent System From the initial meeting in Washington, CT to the present, I arranged a number of dinners and discussions in London and Cambridge, Massachusetts, as well as a public event at London’s City Hall. Among the attendees were distinguished scientists, science historians, and communications theorists, all of whom have been thinking seriously about AI issues for their entire careers. I commissioned essays from a wide range of contributors, with or without references to Wiener (leaving it up to each participant). In the end, 25 people wrote essays, all individuals concerned about what is happening today in the age of AI. Deep Thinking in not my book, rather it is our book: Seth Lloyd, Judea Pearl, Stuart Russell, George Dyson, Daniel C. Dennett, Rodney Brooks, Frank Wilczek, Max Tegmark, Jaan Tallinn, Steven Pinker, David Deutsch, Tom Griffiths, Anca Dragan, Chris Anderson, David Kaiser, Neil Gershenfeld, W. Daniel Hillis, Venki Ramakrishnan, Alex “Sandy” Pentland, Hans Ulrich Obrist, Alison Gopnik, Peter Galison, George M. Church, Caroline A. Jones, Stephen Wolfram. I see The Deep Thinking Project as an ongoing dynamical emergent system, a presentation of the ideas of a community of sophisticated thinkers who are bringing their experience and erudition to bear in challenging the prevailing digital AI narrative as they 14 HOUSE_OVERSIGHT_016234

communicate their thoughts to one another. The aim is to present a mosaic of views which will help make sense out of this rapidly emerging field. I asked the essayists to consider: (a) The Zen-like poem “Thirteen Ways of Looking at a Blackbird,” by Wallace Stevens, which he insisted was “not meant to be a collection of epigrams or of ideas, but of sensations.” It is an exercise in “perspectivism,” consisting of short, separate sections, each of which mentions blackbirds in some way. The poem is about his own imagination; it concerns what he attends to. (b) The parable of the blind men and an elephant. Like the elephant, AI is too big a topic for any one perspective, never mind the fact that no two people seem to see things the same way. What do we want the book to do? Stewart Brand has noted that “revisiting pioneer thinking is perpetually useful. And it gives a long perspective that invites thinking in decades and centuries about the subject. All contemporary discussion, is bound to age badly and immediately without the longer perspective.” Danny Hillis wants people in AI to realize how they’ ve been programmed by Wiener’s book. “You’re executing its road map,” he says, and you just don’t realize it.” Dan Dennett would like to “let Wiener emerge as the ghost at the banquet. Think of it as a source of hybrid vigor, a source of unsettling ideas to shake up the established mindset.” Neil Gershenfeld argues that “stealth remedial education for the people running the “Big Five” would be a great output from the book.” Freeman Dyson Freeman, one of the few people alive who knew Wiener, notes that “Zhe Human Use of Human Beings is one of the best books ever written. Wiener got almost everything right. I will be interested to see what your bunch of wizards will do with it.” The Evolving AI Narrative Things have changed—and they remain the same. Now AI is everywhere. We have the Internet. We have our smartphones. The founders of the dominant companies—the companies that hold “the whip that lashes us’—have net worths of $65 billion, $90 billion, $130 billion. High-profile individuals such as Elon Musk, Nick Bostrom, Martin Rees, Eliezer Yudkowsky, and the late Stephen Hawking have issued dire warnings about AI, resulting in the ascendancy of well-funded institutes tasked with promoting “Nice AI.” But will we, as a species, be able to control a fully realized, unsupervised, self- improving AI? Wiener’s warnings and admonitions in Zhe Human Use of Human Beings are now very real, and they need to be looked at anew by researchers at the forefront of the AI revolution. Here is Dyson again: Wiener became increasingly disenchanted with the “gadget worshipers” whose corporate selfishness brought “motives to automatization that go beyond a legitimate curiosity and are sinful in themselves.” He knew the danger was not machines becoming more like humans but humans being treated like machines. “The world of the future will be an ever more demanding struggle against the limitations of our intelligence,” he warned in God & Golem, Inc., published in 15 HOUSE_OVERSIGHT_016235

1964, the year of his death, “not a comfortable hammock in which we can lie down to be waited upon by our robot slaves.” It’s time to examine the evolving AI narrative by identifying the leading members of that mainstream community along with the dissidents, and presenting their counternarratives in their own voices. The essays that follow thus constitute a much-needed update from the field. John Brockman New York, 2019 16 HOUSE_OVERSIGHT_016236

I met Seth Lloyd in the late 1980s, when new ways of thinking were everywhere: the importance of biological organizing principles, the computational view of mathematics and physical processes, the emphasis on parallel networks, the importance of nonlinear dynamics, the new understanding of chaos, connectionist ideas, neural networks, and parallel distributive processing. The advances in computation during that period provided us with a new way of thinking about knowledge. Seth likes to refer to himself as a quantum mechanic. He is internationally known jor his work in the field of quantum computation, which attempts to harness the exotic properties of quantum theory, like superposition and entanglement, to solve problems that would take several lifetimes to solve on classical computers. In the essay that follows, he traces the history of information theory from Norbert Wiener ’s prophetic insights to the predictions of a technological “singularity” that some would have us believe will supplant the human species. His takeaway on the recent programming method known as deep learning is to call for a more modest set of expectations; he notes that despite AI’s enormous advances, robots “still can’t tie their own shoes.” It’s difficult for me to talk about Seth without referencing his relationship with his friend and professor, the late theoretical physicist Heinz Pagels of Rockefeller University. The graduate student and the professor each had a profound effect on each other’s ideas. In the summer of 1988, I visited Heinz and Seth at the Aspen Center for Physics. Their joint work on the subject of complexity was featured in the current issue of Scientific American; they were ebullient. That was just two weeks before Heinz’s tragic death in a hiking accident while descending Pyramid Peak with Seth. They were talking about quantum computing. 17 HOUSE_OVERSIGHT_016237

WRONG, BUT MORE RELEVANT THAN EVER Seth Lloyd Seth Lloyd is a theoretical physicist at MIT, Nam P. Suh Professor in the Department of Mechanical Engineering, and an external professor at the Santa Fe Institute. The Human Use of Human Beings, Norbert Wiener’s 1950 popularization of his highly influential book Cybernetics: Control and Communication in the Animal and the Machine (1948), investigates the interplay between human beings and machines in a world in which machines are becoming ever more computationally capable and powerful. It is aremarkably prescient book, and remarkably wrong. Written at the height of the Cold War, it contains a chilling reminder of the dangers of totalitarian organizations and societies, and of the danger to democracy when it tries to combat totalitarianism with totalitarianism’s own weapons. Wiener’s Cybernetics looked in close scientific detail at the process of control via feedback. (“Cybernetics,” from the ancient Greek for “helmsman,” is the etymological basis of our word “governor,” which is what James Watt called his pathbreaking feedback control device that transformed the use of steam engines.) Because he was immersed in problems of control, Wiener saw the world as a set of complex, interlocking feedback loops, in which sensors, signals, and actuators such as engines interact via an intricate exchange of signals and information. The engineering applications of Cybernetics were tremendously influential and effective, giving rise to rockets, robots, automated assembly lines, and a host of precision-engineering techniques—in other words, to the basis of contemporary industrial society. Wiener had greater ambitions for cybernetic concepts, however, and in 7he Human Use of Human Beings he spells out his thoughts on its application to topics as diverse as Maxwell’s Demon, human language, the brain, insect metabolism, the legal system, the role of technological innovation in government, and religion. These broader applications of cybernetics were an almost unequivocal failure. Vigorously hyped from the late 1940s to the early 1960s—to a degree similar to the hype of computer and communication technology that led to the dotcom crash of 2000-200 1—cybernetics delivered satellites and telephone switching systems but generated few if any useful developments in social organization and society at large. Nearly seventy years later, however, Ze Human Use of Human Beings has more to teach us humans than it did the first time around. Perhaps the most remarkable feature of the book is that it introduces a large number of topics concerning human/machine interactions that are still of considerable relevance. Dark in tone, the book makes several predictions about disasters to come in the second half of the 20th century, many of which are almost identical to predictions made today about the second half of the 21st. For example, Wiener foresaw a moment in the near future of 1950 in which humans would cede control of society to a cybernetic artificial intelligence, which would then proceed to wreak havoc on humankind. The automation of manufacturing, Wiener predicted, would both create large advances in productivity and displace many workers from their jobs—a sequence of events that did indeed come to pass in the ensuing decades. Unless society could find productive occupations for these displaced workers, Wiener warned, revolt would ensue. 18 HOUSE_OVERSIGHT_016238

But Wiener failed to foresee crucial technological developments. Like pretty much all technologists of the 1950s, he failed to predict the computer revolution. Computers, he thought, would eventually fall in price from hundreds of thousands of (1950s) dollars to tens of thousands; neither he nor his compeers anticipated the tremendous explosion of computer power that would follow the development of the transistor and the integrated circuit. Finally, because of his emphasis on control, Wiener could not foresee a technological world in which innovation and self-organization bubble up from the bottom rather than being imposed from the top. Focusing on the evils of totalitarianism (political, scientific, and religious), Wiener saw the world in a deeply pessimistic light. His book warned of the catastrophe that awaited us if we didn’t mend our ways, fast. The current world of human beings and machines, more than a half century after its publication, is much more complex, richer, and contains a much wider variety of political, social, and scientific systems than he was able to envisage. The warnings of what will happen if we get it wrong, however—for example, control of the entire Internet by a global totalitarian regime—remain as relevant and pressing today as they were in 1950. What Wiener Got Right Wiener’s most famous mathematical works focused on problems of signal analysis and the effects of noise. During World War II, he developed techniques for aiming anti- aircraft fire by making models that could predict the future trajectory of an airplane by extrapolating from its past behavior. In Cybernetics and in The Human Use of Human Beings, Wiener notes that this past behavior includes quirks and habits of the human pilot, thus a mechanized device can predict the behavior of humans. Like Alan Turing, whose Turing Test suggested that computing machines could give responses to questions which were indistinguishable from human responses, Wiener was fascinated by the notion of capturing human behavior by mathematical description. In the 1940s, he applied his knowledge of control and feedback loops to neuro-muscular feedback in living systems, and was responsible for bringing Warren McCulloch and Walter Pitts to MIT, where they did their pioneering work on artificial neural networks. Wiener’s central insight was that the world should be understood in terms of information. Complex systems, such as organisms, brains, and human societies, consist of interlocking feedback loops in which signals exchanged between subsystems result in complex but stable behavior. When feedback loops break down, the system goes unstable. He constructed a compelling picture of how complex biological systems function, a picture that is by and large universally accepted today. Wiener’s vision of information as the central quantity in governing the behavior of complex systems was remarkable at the time. Nowadays, when cars and refrigerators are jammed with microprocessors and much of human society revolves around computers and cell phones connected by the Internet, it seems prosaic to emphasize the centrality of information, computation, and communication. In Wiener’s time, however, the first digital computers had only just come into existence, and the Internet was not even a twinkle in the technologist’s eye. Wiener’s powerful conception of not just engineered complex systems but all complex systems as revolving around cycles of signals and computation led to tremendous contributions to the development of complex human-made systems. The 19 HOUSE_OVERSIGHT_016239

methods he and others developed for the control of missiles, for example, were later put to work in building the Saturn V moon rocket, one of the crowning engineering achievements of the 20th century. In particular, Wiener’s applications of cybernetic concepts to the brain and to computerized perception are the direct precursors of today’s neural-network-based deep-learning circuits, and of artificial intelligence itself. But current developments in these fields have diverged from his vision, and their future development may well affect the human uses both of human beings and of machines. What Wiener Got Wrong It is exactly in the extension of the cybernetic idea to human beings that Wiener’s conceptions missed their target. Setting aside his ruminations on language, law, and human society for the moment, look at a humbler but potentially useful innovation that he thought was imminent in 1950. Wiener notes that prosthetic limbs would be much more effective if their wearers could communicate directly with their prosthetics by their own neural signals, receiving information about pressure and position from the limb and directing its subsequent motion. This turned out to be a much harder problem than Wiener envisaged: Seventy years down the road, prosthetic limbs that incorporate neural feedback are still in the very early stages. Wiener’s concept was an excellent one—it’s just that the problem of interfacing neural signals with mechanical-electrical devices is hard. More significantly, Wiener (along with pretty much everyone else in 1950) greatly underappreciated the potential of digital computation. As noted, Wiener’s mathematical contributions were to the analysis of signals and noise and his analytic methods apply to continuously varying, or analog, signals. Although he participated in the wartime development of digital computation, he never foresaw the exponential explosion of computing power brought on by the introduction and progressive miniaturization of semiconductor circuits. This is hardly Wiener’s fault: The transistor hadn’t been invented yet, and the vacuum-tube technology of the digital computers he was familiar with was clunky, unreliable, and unscalable to ever larger devices. In an appendix to the 1948 edition of Cybernetics, he anticipates chess-playing computers and predicts that they’ll be able to look two or three moves ahead. He might have been surprised to learn that within half a century a computer would beat the human world champion at chess. Technological Overestimation and the Existential Risks of the Singularity When Wiener wrote his books, a significant example of technological overestimation was about to occur. The 1950s saw the first efforts at developing artificial intelligence, by researchers such as Herbert Simon, John McCarthy, and Marvin Minsky, who began to program computers to perform simple tasks and to construct rudimentary robots. The success of these initial efforts inspired Simon to declare that “machines will be capable, within twenty years, of doing any work a man can do.” Such predictions turned out to be spectacularly wrong. As they became more powerful, computers got better and better at playing chess because they could systematically generate and evaluate a vast selection of possible future moves. But the majority of predictions of AI, e.g., robotic maids, turned out to be illusory. When Deep Blue beat Garry Kasparov at chess in 1997, the most 20 HOUSE_OVERSIGHT_016240

powerful room-cleaning robot was a Roomba, which moved around vacuuming at random and squeaked when it got caught under the couch. Technological prediction is particularly chancy, given that technologies progress by a series of refinements, halted by obstacles and overcome by innovation. Many obstacles and some innovations can be anticipated, but more cannot. In my own work with experimentalists on building quantum computers, I typically find that some of the technological steps I expect to be easy turn out to be impossible, whereas some of the tasks I imagine to be impossible turn out to be easy. You don’t know until you try. In the 1950s, partly inspired by conversations with Wiener, John von Neumann introduced the notion of the “technological singularity.” Technologies tend to improve exponentially, doubling in power or sensitivity over some interval of time. (For example, since 1950, computer technologies have been doubling in power roughly every two years, an observation enshrined as Moore’s Law.) Von Neumann extrapolated from the observed exponential rate of technological improvement to predict that “technological progress will become incomprehensively rapid and complicated,” outstripping human capabilities in the not too distant future. Indeed, if one extrapolates the growth of raw computing power—expressed in terms of bits and bit flips—into the future at its current rate, computers should match human brains sometime in the next two to four decades (depending on how one estimates the information-processing power of human brains). The failure of the initial overly optimistic predictions of AI dampened talk about the technological singularity for a few decades, but since the 2005 publication of Ray Kurzweil’s The Singularity is Near, the idea of technological advance leading to superintelligence is back in force. Some believers, Kurzweil included, regard this singularity as an opportunity: Humans can merge their brains with the superintelligence and thereby live forever. Others, such as Stephen Hawking and Elon Musk, worried that this superintelligence would prove to be malign and regarded it as the greatest existing threat to human civilization. Still others, including some of the contributors to the present volume, think such talk is overblown. Wiener’s life work and his failure to predict its consequences are intimately bound up in the idea of an impending technological singularity. His work on neuroscience and his initial support of McCulloch and Pitts adumbrated the startlingly effective deep-learning methods of the present day. Over the past decade, and particularly in the last five years, such deep-learning techniques have finally exhibited what Wiener liked to call Gestalt—for example, the ability to recognize that a circle is a circle even if when slanted sideways it looks like an ellipse. His work on control, combined with his work on neuromuscular feedback, was significant for the development of robotics and is the inspiration for neural-based human/machine interfaces. His lapses in technological prediction, however, suggest that we should take the notion of a technological singularity with a grain of salt. The general difficulties of technological prediction and the problems specific to the development of a superintelligence should warn us against overestimating both the power and the efficacy of information processing. The Arguments for Singularity Skepticism No exponential increase lasts forever. An atomic explosion grows exponentially, but 21 HOUSE_OVERSIGHT_016241

only until it runs out of fuel. Similarly, the exponential advances in Moore’s Law are starting to run into limits imposed by basic physics. The clock speed of computers maxed out at a few gigahertz a decade and a half ago, simply because the chips were starting to melt. The miniaturization of transistors is already running into quantum- mechanical problems due to tunneling and leakage currents. Eventually, the various exponential improvements in memory and processing driven by Moore’s Law will grind to a halt. A few more decades, however, will probably be time enough for the raw information-processing power of computers to match that of brains—at least by the crude measures of number of bits and number of bit-flips per second. Human brains are intricately constructed, the process of millions of years of natural selection. In Wiener’s time, our understanding of the architecture of the brain was rudimentary and simplistic. Since then, increasingly sensitive instrumentation and imaging techniques have shown our brains to be far more varied in structure and complex in function than Wiener could have imagined. I recently asked Tomaso Poggio, one of the pioneers of modern neuroscience, whether he was worried that computers, with their rapidly increasing processing power, would soon emulate the functioning of the human brain. “Not a chance,” he replied. The recent advances in deep learning and neuromorphic computation are very good at reproducing a particular aspect of human intelligence focused on the operation of the brain’s cortex, where patterns are processed and recognized. These advances have enabled a computer to beat the world champion not just of chess but of Go, an impressive feat, but they’re far short of enabling a computerized robot to tidy a room. (In fact, robots with anything approaching human capability in a broad range of flexible movements are still far away—search “robots falling down.” Robots are good at making precision welds on assembly lines, but they still can’t tie their own shoes.) Raw information-processing power does not mean sophisticated information- processing power. While computer power has advanced exponentially, the programs by which computers operate have often failed to advance at all. One of the primary responses of software companies to increased processing power is to add “useful” features which often make the software harder to use. Microsoft Word reached its apex in 1995 and has been slowly sinking under the weight of added features ever since. Once Moore’s Law starts slowing down, software developers will be confronted with hard choices between efficiency, speed, and functionality. A major fear of the singulariteers is that as computers become more involved in designing their own software they’! rapidly bootstrap themselves into achieving superhuman computational ability. But the evidence of machine learning points in the opposite direction. As machines become more powerful and capable of learning, they learn more and more as human beings do—from multiple examples, often under the supervision of human and machine teachers. Education is as hard and slow for computers as it is for teenagers. Consequently, systems based on deep learning are becoming more rather than less human. The skills they bring to learning are not “better than” but “complementary to” human learning: Computer learning systems can identify patterns that humans cannot—and vice versa. The world’s best chess players are neither computers nor humans but humans working together with computers. Cyberspace is indeed inhabited by harmful programs, but these primarily take the form of malware—viruses notable for their malign mindlessness, not for their 22 HOUSE_OVERSIGHT_016242

superintelligence. Whither Wiener Wiener noted that exponential technological progress is a relatively modern phenomenon and not all of itis good. He regarded atomic weapons and the development of missiles with nuclear warheads as a recipe for the suicide of the human species. He compared the headlong exploitation of the planet’s resources with the Mad Tea Party of Alice in Wonderland: Having laid waste to one local environment, we make progress simply by moving on to lay waste to the next. Wiener’s optimism about the development of computers and neuro-mechanical systems was tempered by his pessimism about their exploitation by authoritarian governments, such as the Soviet Union, and the tendency for democracies, such as the United States, to become more authoritarian themselves in confronting the threat of authoritarianism. What would Wiener think of the current human use of human beings? He would be amazed by the power of computers and the Internet. He would be happy that the early neural nets in which he played a role have spawned powerful deep-learning systems that exhibit the perceptual ability he demanded of them—although he might not be impressed that one of the most prominent examples of such computerized Gestalt is the ability to recognize photos of kittens on the World Wide Web. Rather than regarding machine intelligence as a threat, I suspect he would regard it as a phenomenon in its own right, different from and co-evolving with our own human intelligence. Unsurprised by global warming—the Mad Tea Party of our era—Wiener would applaud the exponential improvement in alternative-energy technologies and would apply his cybernetic expertise to developing the intricate set of feedback loops needed to incorporate such technologies into the coming smart electrical grid. Nonetheless, recognizing that the solution to the problem of climate change is at least as much political as it is technological, he would undoubtedly be pessimistic about our chances of solving this civilization-threatening problem in time. Wiener hated hucksters—political hucksters most of all—but he acknowledged that hucksters would always be with us. It’s easy to forget just how scary Wiener’s world was. The United States and the Soviet Union were in a full-out arms race, building hydrogen bombs mounted on nuclear warheads carried by intercontinental ballistic missiles guided by navigation systems to which Wiener himself—to his dismay—had contributed. I was four years old when Wiener died. In 1964, my nursery school class was practicing duck-and-cover under our desks to prepare for a nuclear attack. Given the human use of human beings in his own day, if he could see our current state, Wiener’s first response would be to be relieved that we are still alive. 20 HOUSE_OVERSIGHT_016243

In the 1980s, Judea Pearl introduced a new approach to artificial intelligence called Bayesian networks. This probability-based model of machine reasoning enabled machines to function—in a complex and uncertain world—as “evidence engines,’ continuously revising their beliefs in light of new evidence. Within a few years, Judea’s Bayesian networks had completely overshadowed the previous rule-based approaches to artificial intelligence. The advent of deep learning— in which computers, in effect, teach themselves to be smarter by observing tons of data, has given him pause, because this method lacks transparency. While recognizing the impressive achievements in deep learning by colleagues such as Michael Jordan and Geoffrey Hinton, he feels uncomfortable with this kind of opacity. He set out to understand the theoretical limitations of deep-learning systems and points out that basic barriers exist that will prevent them from achieving a human kind of intelligence, no matter what we do. Leveraging the computational benefits of Bayesian networks, Judea realized that the combination of simple graphical models and data could also be used to represent and infer cause-effect relationships. The significance of this discovery far transcends its roots in artificial intelligence. His latest book explains causal thinking to the general public; you might say it is a primer on how to think even though human. Judea’s principled, mathematical approach to causality is a profound contribution to the realm of ideas. It has already benefited virtually every field of inquiry, especially the data-intensive health and social sciences. 3 24 HOUSE_OVERSIGHT_016244

THE LIMITATIONS OF OPAQUE LEARNING MACHINES Judea Pearl Judea Pearl is a professor of computer science and director of the Cognitive Systems Laboratory at UCLA. His most recent book, co-authored with Dana Mackenzie, is The Book of Why: The New Science of Cause and Effect. As a former physicist, I was extremely interested in cybernetics. Though it did not utilize the full power of Turing Machines, it was highly transparent, perhaps because it was founded on classical control theory and information theory. We are losing this transparency now, with the deep-learning style of machine learning. It is fundamentally a curve-fitting exercise that adjusts weights in intermediate layers of a long input-output chain. I find many users who say that it “works well and we don’t know why.” Once you unleash it on large data, deep learning has its own dynamics, it does its own repair and its own optimization, and it gives you the right results most of the time. But when it doesn’t, you don’t have a clue about what went wrong and what should be fixed. In particular, you do not know if the fault is in the program, in the method, or because things have changed in the environment. We should be aiming at a different kind of transparency. Some argue that transparency is not really needed. We don’t understand the neural architecture of the human brain, yet it runs well, so we forgive our meager understanding and use human helpers to great advantage. In the same way, they argue, why not unleash deep-learning systems and create intelligence without understanding how they work? I buy this argument to some extent. I personally don’t like opacity, so I won’t spend my time on deep learning, but I know that it has a place in the makeup of intelligence. I know that non-transparent systems can do marvelous jobs, and our brain is proof of that marvel. But this argument has its limitation. The reason we can forgive our meager understanding of how human brains work is because our brains work the same way, and that enables us to communicate with other humans, learn from them, instruct them, and motivate them in our own native language. If our robots will all be as opaque as AlphaGo, we won’t be able to hold a meaningful conversation with them, and that would be unfortunate. We will need to retrain them whenever we make a slight change in the task or in the operating environment. So, rather than experimenting with opaque learning machines, I am trying to understand their theoretical limitations and examine how these limitations can be overcome. I do it in the context of causal-reasoning tasks, which govern much of how scientists think about the world and, at the same time, are rich in intuition and toy examples, so we can monitor the progress in our analysis. In this context, we’ve discovered that some basic barriers exist, and that unless they are breached we won’t get areal human kind of intelligence no matter what we do. I believe that charting these barriers may be no less important than banging our heads against them. Current machine-learning systems operate almost exclusively in a statistical, or model-blind, mode, which is analogous in many ways to fitting a function to a cloud of data points. Such systems cannot reason about “what if ?” questions and, therefore, 25 HOUSE_OVERSIGHT_016245

cannot serve as the basis for Strong Al—that is, artificial intelligence that emulates human-level reasoning and competence. To achieve human-level intelligence, learning machines need the guidance of a blueprint of reality, a model—similar to a road map that guides us in driving through an unfamiliar city. To be more specific, current learning machines improve their performance by optimizing parameters for a stream of sensory inputs received from the environment. It is a slow process, analogous to the natural-selection process that drives Darwinian evolution. It explains how species like eagles and snakes have developed superb vision systems over millions of years. It cannot explain, however, the super-evolutionary process that enabled humans to build eyeglasses and telescopes over barely a thousand years. What humans had that other species lacked was a mental representation of their environment—representations that they could manipulate at will to imagine alternative hypothetical environments for planning and learning. Historians of Homo sapiens such as Yuval Noah Harari and Steven Mithen are in general agreement that the decisive ingredient that gave our ancestors the ability to achieve global dominion about forty thousand years ago was their ability to create and store a mental representation of their environment, interrogate that representation, distort it by mental acts of imagination, and finally answer the “What if?” kind of questions. Examples are interventional questions (“What if I do such-and-such?”) and retrospective or counterfactual questions (“What if I had acted differently?”). No learning machine in operation today can answer such questions. Moreover, most learning machines do not possess a representation from which the answers to such questions can be derived. With regard to causal reasoning, we find that you can do very little with any form of model-blind curve fitting, or any statistical inference, no matter how sophisticated the fitting process is. We have also found a theoretical framework for organizing such limitations, which forms a hierarchy. On the first level, you have statistical reasoning, which can tell you only how seeing one event would change your belief about another. For example, what can a symptom tell you about a disease? Then you have a second level, which entails the first but not vice versa. It deals with actions. “What will happen if we raise prices?” “What if you make me laugh?” That second level of the hierarchy requires information about interventions which is not available in the first. This information can be encoded in a graphical model, which merely tells us which variable responds to another. The third level of the hierarchy is the counterfactual. This is the language used by scientists. “What if the object were twice as heavy?” “What if I were to do things differently?” “Was it the aspirin that cured my headache, or the nap I took?” Counterfactuals are at the top level in the sense that they cannot be derived even if we could predict the effects of all actions. They need an extra ingredient, in the form of equations, to tell us how variables respond to changes in other variables. One of the crowning achievements of causal-inference research has been the algorithmization of both interventions and counterfactuals, the top two layers of the hierarchy. In other words, once we encode our scientific knowledge in a model (which may be qualitative), algorithms exist that examine the model and determine if a given query, be it about an intervention or about a counterfactual, can be estimated from the available data—and, if so, how. This capability has transformed dramatically the way 26 HOUSE_OVERSIGHT_016246

scientists are doing science, especially in such data-intensive sciences as sociology and epidemiology, for which causal models have become a second language. These disciplines view their linguistic transformation as the Causal Revolution. As Harvard social scientist Gary King puts it, “More has been learned about causal inference in the last few decades than the sum total of everything that had been learned about it in all prior recorded history.” As I contemplate the success of machine learning and try to extrapolate it to the future of AI, I ask myself, “Are we aware of the basic limitations that were discovered in the causal-inference arena? Are we prepared to circumvent the theoretical impediments that prevent us from going from one level of the hierarchy to another level?” I view machine learning as a tool to get us from data to probabilities. But then we still have to make two extra steps to go from probabilities into real understandingnce— two big steps. One is to predict the effect of actions, and the second is counterfactual imagination. We cannot claim to understand reality unless we make the last two steps. In his insightful book Foresight and Understanding (1961), the philosopher Stephen Toulmin identified the transparency-versus-opacity contrast as the key to understanding the ancient rivalry between Greek and Babylonian sciences. According to Toulmin, the Babylonian astronomers were masters of black-box predictions, far surpassing their Greek rivals in accuracy and consistency of celestial observations. Yet Science favored the creative-speculative strategy of the Greek astronomers, which was wild with metaphorical imagery: circular tubes full of fire, small holes through which celestial fire was visible as stars, and hemispherical Earth riding on turtleback. It was this wild modeling strategy, not Babylonian extrapolation, that jolted Eratosthenes (276- 194 BC) to perform one of the most creative experiments in the ancient world and calculate the circumference of the Earth. Such an experiment would never have occurred to a Babylonian data-fitter. Model-blind approaches impose intrinsic limitations on the cognitive tasks that Strong Al can perform. My general conclusion is that human-level AI cannot emerge solely from model-blind learning machines; it requires the symbiotic collaboration of data and models. Data science is a science only to the extent that it facilitates the interpretation of data—a two-body problem, connecting data to reality. Data alone are hardly a science, no matter how “big” they get and how skillfully they are manipulated. Opaque learning systems may get us to Babylon, but not to Athens. 27 HOUSE_OVERSIGHT_016247

Computer scientist Stuart Russell, along with Elon Musk, Stephen Hawking, Max Tegmark, and numerous others, has insisted that attention be paid to the potential dangers in creating an intelligence on the superhuman (or even the human) level—an AGI, or artificial general intelligence, whose programmed purposes may not necessarily align with our own. His early work was on understanding the notion of “bounded optimality” as a formal definition of intelligence that you can work on. He developed the technique of rational meta-reasoning, “which is, roughly speaking, that you do the computations that you expect to improve the quality of your ultimate decision as quickly as possible.” He has also worked on the unification of probability theory and first-order logic—resulting in a new and far more effective monitoring system for the Comprehensive Nuclear Test Ban Treaty—and on the problem of decision making over long timescales (his presentations on the latter topic are usually titled, “Life: Play and Win in 20 trillion moves”’). He is very concerned with the continuing development of autonomous weapons, such as lethal micro-drones, which are potentially scalable into weapons of mass destruction. He drafted the letter from forty of the world’s leading AI researchers to President Obama which resulted in high-level national-security meetings. His current work centers on the creation of what he calls “provably beneficial” AI. He wants to ensure AI safety by “imbuing systems with explicit uncertainty” about the objectives of their human programmers, an approach that would amount to a fairly radical reordering of current AI research. Stuart is also on the radar of anyone who has taken a course in computer science in the last twenty-odd years. He is co-author of “the” definitive AI textbook, with an estimated 5-million-plus English-language readers. 28 HOUSE_OVERSIGHT_016248

THE PURPOSE PUT INTO THE MACHINE Stuart Russell Stuart Russell is a professor of computer science and Smith-Zadeh Professor in Engineering at UC Berkeley. He is the coauthor (with Peter Norvig) of Artificial Intelligence: A Modern Approach. Among the many issues raised in Norbert Wiener’s The Human Use of Human Beings (1950) that are currently relevant, the most significant to the AI researcher is the possibility that humanity may cede control over its destiny to machines. Wiener considered the machines of the near future as far too limited to exert global control, imagining instead that machines and machine-like control systems would be wielded by human elites to reduce the great mass of humanity to the status of “cogs and levers and rods.” Looking further ahead, he pointed to the difficulty of correctly specifying objectives for highly capable machines, noting a few of the simpler and more obvious truths of life, such as that when a djinnee is found in a bottle, it had better be left there; that the fisherman who craves a boon from heaven too many times on behalf of his wife will end up exactly where he started; that if you are given three wishes, you must be very careful what you wish for. The dangers are clear enough: Woe to us if we let [the machine] decide our conduct, unless we have previously examined the laws of its action, and know fully that its conduct will be carried out on principles acceptable to us! On the other hand, the machine like the djinnee, which can learn and can make decisions on the basis of its learning, will in no way be obliged to make such decisions as we should have made, or will be acceptable to us. Ten years later, after seeing Arthur Samuel’s checker-playing program learn to play checkers far better than its creator, Wiener published “Some Moral and Technical Consequences of Automation” in Science. In this paper, the message is even clearer: If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere ... we had better be quite sure that the purpose put into the machine is the purpose which we really desire. . . . In my view, this is the source of the existential risk from superintelligent AI cited in recent years by such observers as Elon Musk, Bill Gates, Stephen Hawking, and Nick Bostrom. Putting Purposes Into Machines The goal of AI research has been to understand the principles underlying intelligent behavior and to build those principles into machines that can then exhibit such behavior. In the 1960s and 1970s, the prevailing theoretical notion of intelligence was the capacity for logical reasoning, including the ability to derive plans of action guaranteed to achieve a specified goal. More recently, a consensus has emerged around the idea of a rational 29 HOUSE_OVERSIGHT_016249

agent that perceives, and acts in order to maximize, its expected utility. Subfields such as logical planning, robotics, and natural-language understanding are special cases of the general paradigm. AI has incorporated probability theory to handle uncertainty, utility theory to define objectives, and statistical learning to allow machines to adapt to new circumstances. These developments have created strong connections to other disciplines that build on similar concepts, including control theory, economics, operations research, and statistics. In both the logical-planning and rational-agent views of AI, the machine’s objective—whether in the form of a goal, a utility function, or a reward function (as in reinforcement learning)—1s specified exogenously. In Wiener’s words, this is “the purpose put into the machine.” Indeed, it has been one of the tenets of the field that AI systems should be general-purpose—i.e., capable of accepting a purpose as input and then achieving it—rather than special-purpose, with their goal implicit in their design. For example, a self-driving car should accept a destination as input instead of having one fixed destination. However, some aspects of the car’s “driving purpose” are fixed, such as that it shouldn’t hit pedestrians. This is built directly into the car’s steering algorithms rather than being explicit: No self-driving car in existence today “knows” that pedestrians prefer not to be run over. Putting a purpose into a machine which optimizes its behavior according to clearly defined algorithms seems an admirable approach to ensuring that the machine’s “conduct will be carried out on principles acceptable to us!” But, as Wiener warns, we need to put in the right purpose. We might call this the King Midas problem: Midas got exactly what he asked for—namely, that everything he touched would turn to gold—but too late he discovered the drawbacks of drinking liquid gold and eating solid gold. The technical term for putting in the right purpose is value alignment. When it fails, we may inadvertently imbue machines with objectives counter to our own. Tasked with finding a cure for cancer as fast as possible, an AI system might elect to use the entire human population as guinea pigs for its experiments. Asked to de-acidify the oceans, it might use up all the oxygen in the atmosphere as a side effect. This is a common characteristic of systems that optimize: Variables not included in the objective may be set to extreme values to help optimize that objective. Unfortunately, neither AI nor other disciplines (economics, statistics, control theory, operations research) built around the optimization of objectives have much to say about how to identify the purposes “we really desire.” Instead, they assume that objectives are simply implanted into the machine. AI research, in its present form, studies the ability to achieve objectives, not the design of those objectives. Steve Omohundro has pointed to a further difficulty, observing that intelligent entities must act to preserve their own existence. This tendency has nothing to do with a self-preservation instinct or any other biological notion; it’s just that an entity cannot achieve its objectives if it’s dead. According to Omohundro’s argument, a superintelligent machine that has an off-switch—which some, including Alan Turing himself, in a 1951 talk on BBC Radio 3, have seen as our potential salvation—will take steps to disable the switch in some way.! Thus we may face the prospect of superintelligent machines—their actions by definition unpredictable by us and their ' Omohundro, “The Basic AI Drives,” in Proc. First AGI Conf., 171: “Artificial General Intelligence,” eds. P. Wang, B. Goertzel, & S. Franklin (IOS press, 2008). 30 HOUSE_OVERSIGHT_016250

imperfectly specified objectives conflicting with our own—whose motivation to preserve their existence in order to achieve those objectives may be insuperable. 1001 Reasons to Pay No Attention Objections have been raised to these arguments, primarily by researchers within the AI community. The objections reflect a natural defensive reaction, coupled perhaps with a lack of imagination about what a superintelligent machine could do. None hold water on closer examination. Here are some of the more common ones: e Don’t worry, we can just switch it off? This is often the first thing that pops into a layperson’s head when considering risks from superintelligent Al—as if a superintelligent entity would never think of that. This is rather like saying that the risk of losing to DeepBlue or AlphaGo is negligible—all one has to do is make the right moves. e Human-level or superhuman AI is impossible This is an unusual claim for AI researchers to make, given that, from Turing onward, they have been fending off such claims from philosophers and mathematicians. The claim, which is backed by no evidence, appears to concede that if superintelligent AI were possible, it would be a significant risk. It’s as if a bus driver, with all of humanity as passengers, said, “Yes, I am driving toward a cliff—in fact, I’m pressing the pedal to the metal! But trust me, we’ll run out of gas before we get there!” The claim represents a foolhardy bet against human ingenuity. We have made such bets before and lost. On September 11, 1933, renowned physicist Ernest Rutherford stated, with utter confidence, “Anyone who expects a source of power from the transformation of these atoms is talking moonshine.” On September 12, 1933, Leo Szilard invented the neutron-induced nuclear chain reaction. A few years later he demonstrated such a reaction in his laboratory at Columbia University. As he recalled in a memoir: “We switched everything off and went home. That night, there was very little doubt in my mind that the world was headed for grief.” e It’s too soon to worry about it. The right time to worry about a potentially serious problem for humanity depends not just on when the problem will occur but also on how much time is needed to devise and implement a solution that avoids the risk. For example, if we were to detect a large asteroid predicted to collide with the Earth in 2067, would we say, “It’s too soon to worry”? And if we consider the global catastrophic risks from climate change predicted to occur later in this century, is it too soon to take action to prevent them? On the contrary, it may be too late. The relevant timescale for human-level AI is less predictable, but, like nuclear fission, it might arrive considerably sooner than expected. One variation on this argument is Andrew Ng’s statement that it’s “like worrying about overpopulation on Mars.” This appeals to a convenient analogy: Not only is the * AI researcher Jeff Hawkins, for example, writes, “Some intelligent machines will be virtual, meaning they will exist and act solely within computer networks. .. . It is always possible to turn off a computer network, even if painful.” https:/Avww.1ecode.net/2015/3/2/11559576/. 3 The AI100 report (Peter Stone et al.), sponsored by Stanford University, includes the following: “Unlike in the movies, there is no race of superhuman robots on the horizon or probably even possible.” https://ail00.stanford.edu/20 16-report. 31 HOUSE_OVERSIGHT_016251

risk easily managed and far in the future, but also it’s extremely unlikely that we'd even try to move billions of humans to Mars in the first place. The analogy is a false one, however. We are already devoting huge scientific and technical resources to creating ever-more-capable AI systems. A more apt analogy would be a plan to move the human race to Mars with no consideration for what we might breathe, drink, or eat once we'd arrived. e Human-level Al isn’t really imminent, in any case. The AI100 report, for example, assures us, “Contrary to the more fantastic predictions for AI in the popular press, the Study Panel found no cause for concern that AI is an imminent threat to humankind.” This argument simply misstates the reasons for concern, which are not predicated on imminence. In his 2014 book, Superintelligence: Paths, Dangers, Strategies, Nick Bostrom, for one, writes, “It 1s no part of the argument in this book that we are on the threshold of a big breakthrough in artificial intelligence, or that we can predict with any precision when such a development might occur.” e You're just a Luddite. It’s an odd definition of Luddite that includes Turing, Wiener, Minsky, Musk, and Gates, who rank among the most prominent contributors to technological progress in the 20th and 21st centuries.* Furthermore, the epithet represents a complete misunderstanding of the nature of the concerns raised and the purpose for raising them. It is as if one were to accuse nuclear engineers of Luddism if they pointed out the need for control of the fission reaction. Some objectors also use the term “anti-AI,” which is rather like calling nuclear engineers “anti-physics.” The purpose of understanding and preventing the risks of AI is to ensure that we can realize the benefits. Bostrom, for example, writes that success in controlling AI will result in “a civilizational trajectory that leads to a compassionate and jubilant use of humanity’s cosmic endowment”—hardly a pessimistic prediction. e Any machine intelligent enough to cause trouble will be intelligent enough to have appropriate and altruistic objectives. (Often, the argument adds the premise that people of greater intelligence tend to have more altruistic objectives, a view that may be related to the self-conception of those making the argument.) This argument is related to Hume’s is-ought problem and G. E. Moore’s naturalistic fallacy, suggesting that somehow the machine, as a result of its intelligence, will simply perceive what is right, given its experience of the world. This is implausible; for example, one cannot perceive, in the design of a chessboard and chess pieces, the goal of checkmate; the same chessboard and pieces can be used for suicide chess, or indeed many other games still to be invented. Put another way: Where Bostrom imagines humans driven extinct by a putative robot that turns the planet into a sea of paper clips, we humans see this outcome as tragic, 4 Elon Musk, Stephen Hawking, and others (including, apparently, the author) received the 2015 Luddite of the Year Award from the Information Technology Innovation Foundation: https:/Atif.org/publications/20 16/0 1/19/artificial-intelligence-alarmists-win-itif/oE2%80%9 9s-annual- luddite-award. > Rodney Brooks, for example, asserts that it’s impossible for a program to be “smart enough that it would be able to invent ways to subvert human society to achieve goals set for it by humans, without understanding the ways in which it was causing problems for those same humans.” http://rodneybrooks.com/the-seven-deadly -sins-of-predicting-the-future -of-ai/. 32 HOUSE_OVERSIGHT_016252

whereas the iron-eating bactertum Thiobacillus ferrooxidans 1s thrilled. Who’s to say the bacterium is wrong? The fact that a machine has been given a fixed objective by humans doesn’t mean that it will automatically recognize the importance to humans of things that aren’t part of the objective. Maximizing the objective may well cause problems for humans, but, by definition, the machine will not recognize those problems as problematic. e Intelligence is multidimensional, “so ‘smarter than humans’ is a meaningless concept.”® It is a staple of modern psychology that IQ doesn’t do justice to the full range of cognitive skills that humans possess to varying degrees. IQ is indeed a crude measure of human intelligence, but it is utterly meaningless for current AI systems, because their capabilities across different areas are uncorrelated. How do we compare the IQ of Google’s search engine, which cannot play chess, with that of DeepBlue, which cannot answer search queries? None of this supports the argument that because intelligence is multifaceted, we can ignore the risk from superintelligent machines. If “smarter than humans” is a meaningless concept, then “smarter than gorillas” is also meaningless, and gorillas therefore have nothing to fear from humans; clearly, that argument doesn’t hold water. Not only is it logically possible for one entity to be more capable than another across all the relevant dimensions of intelligence, it is also possible for one species to represent an existential threat to another even if the former lacks an appreciation for music and literature. Solutions Can we tackle Wiener’s warning head-on? Can we design AI systems whose purposes don’t conflict with ours, so that we’re sure to be happy with how they behave? On the face of it, this seems hopeless, because it will doubtless prove infeasible to write down our purposes correctly or imagine all the counterintuitive ways a superintelligent entity might fulfill them. If we treat superintelligent AI systems as if they were black boxes from outer space, then indeed we have no hope. Instead, the approach we seem obliged to take, if we are to have any confidence in the outcome, is to define some formal problem /’, and design AI systems to be F-solvers, such that no matter how perfectly a system solves F, we're guaranteed to be happy with the solution. If we can work out an appropriate F' that has this property, we’ll be able to create provably beneficial AI. Here’s an example of how nof to do it: Let a reward be a scalar value provided periodically by a human to the machine, corresponding to how well the machine has behaved during each period, and let F' be the problem of maximizing the expected sum of rewards obtained by the machine. The optimal solution to this problem is not, as one might hope, to behave well, but instead to take control of the human and force him or her to provide a stream of maximal rewards. This is known as the wireheading problem, based on observations that humans themselves are susceptible to the same problem if given a means to electronically stimulate their own pleasure centers. There is, I believe, an approach that may work. Humans can reasonably be described as having (mostly implicit) preferences over their future lives—that is, given ® Kevin Kelly, “The Myth of a Superhuman AI,” Wired, Apr. 25, 2017. 33 HOUSE_OVERSIGHT_016253

enough time and unlimited visual aids, a human could express a preference (or indifference) when offered a choice between two future lives laid out before him or her in all their aspects. (This idealization ignores the possibility that our minds are composed of subsystems with incompatible preferences; if true, that would limit a machine’s ability to optimally satisfy our preferences, but it doesn’t seem to prevent us from designing machines that avoid catastrophic outcomes.) The formal problem F to be solved by the machine in this case 1s to maximize human future-life preferences subject to its initial uncertainty as to what they are. Furthermore, although the future-life preferences are hidden variables, they’re grounded in a voluminous source of evidence—namely, all of the human choices ever made. This formulation sidesteps Wiener’s problem: The machine may learn more about human preferences as it goes along, of course, but it will never achieve complete certainty. A more precise definition is given by the framework of cooperative inverse- reinforcement learning, or CIRL. A CIRL problem involves two agents, one human and the other a robot. Because there are two agents, the problem is what economists call a game. It is a game of partial information, because while the human knows the reward function, the robot doesn’t—even though the robot’s job is to maximize it. A simple example: Suppose that Harriet, the human, likes to collect paper clips and staples and her reward function depends on how many of each she has. More precisely, if she has p paper clips and s staples, her degree of happiness is Op + (1-0)s, where @ is essentially an exchange rate between paper clips and staples. If @1is 1, she likes only paper clips; if @ is 0, she likes only staples; if 0 is 0.5, she is indifferent between them; and so on. It’s the job of Robby, the robot, to produce the paper clips and staples. The point of the game is that Robby wants to make Harriet happy, but he doesn’t know the value of @, so he isn’t sure how many of each to produce. Here’s how the game works. Let the true value of 0 be 0.49—that is, Harriet has a slight preference for staples over paper clips. And let’s assume that Robby has a uniform prior belief about 6—that is, he believes @ is equally likely to be any value between 0 and 1. Harriet now gets to do a small demonstration, producing either two paper clips or two staples or one of each. After that, the robot can produce either ninety paper clips, or ninety staples, or fifty of each. You might think that Harriet, who prefers staples to paper clips, should produce two staples. But in that case, Robby’s rational response would be to produce ninety staples (with a total value to Harriet of 45.9), which is a less desirable outcome for Harriet than fifty of each (total value 50.0). The optimal solution of this particular game is that Harriet produces one of each, so then Robby makes fifty of each. Thus, the way the game is defined encourages Harriet to “teach” Robby—as long as she knows that Robby is watching carefully. Within the CIRL framework, one can formulate and solve the off-switch problem—that is, the problem of how to prevent a robot from disabling its off-switch. (Turing may rest easier.) A robot that’s uncertain about human preferences actually benefits from being switched off, because it understands that the human will press the off-switch to prevent the robot from doing something counter to those preferences. Thus the robot is incentivized to preserve the off-switch, and this incentive derives directly from its uncertainty about human preferences.’ The off-switch example suggests some templates for controllable-agent T See Hadfield-Menell et al., “The Off-Switch Game,” https://arxiv.org/pdf/1611.08219. pdf. 34 HOUSE_OVERSIGHT_016254

designs and provides at least one case of a provably beneficial system in the sense introduced above. The overall approach resembles mechanism-design problems in economics, wherein one incentivizes other agents to behave in ways beneficial to the designer. The key difference here is that we are building one of the agents in order to benefit the other. There are reasons to think this approach may work in practice. First, there is abundant written and filmed information about humans doing things (and other humans reacting). Technology to build models of human preferences from this storehouse will presumably be available long before superintelligent AI systems are created. Second, there are strong, near-term economic incentives for robots to understand human preferences: If one poorly designed domestic robot cooks the cat for dinner, not realizing that its sentimental value outweighs its nutritional value, the domestic-robot industry will be out of business. There are obvious difficulties, however, with an approach that expects a robot to learn underlying preferences from human behavior. Humans are irrational, inconsistent, weak-willed, and computationally limited, so their actions don’t always reflect their true preferences. (Consider, for example, two humans playing chess. Usually, one of them loses, but not on purpose!) So robots can learn from nonrational human behavior only with the aid of much better cognitive models of humans. Furthermore, practical and social constraints will prevent all preferences from being maximally satisfied simultaneously, which means that robots must mediate among conflicting preferences—something that philosophers and social scientists have struggled with for millennia. And what should robots learn from humans who enjoy the suffering of others? It may be best to zero out such preferences in the robots’ calculations. Finding a solution to the AI control problem is an important task; it may be, in Bostrom’s words, “the essential task of our age.” Up to now, AI research has focused on systems that are better at making decisions, but this is not the same as making better decisions. No matter how excellently an algorithm maximizes, and no matter how accurate its model of the world, a machine’s decisions may be ineffably stupid in the eyes of an ordinary human if its utility function is not well aligned with human values. This problem requires a change in the definition of AI itself—from a field concerned with pure intelligence, independent of the objective, to a field concerned with systems that are provably beneficial for humans. Taking the problem seriously seems likely to yield new ways of thinking about AI, its purpose, and our relationship to it. 35 HOUSE_OVERSIGHT_016255

In 2005, George Dyson, a historian of science and technology, visited Google at the invitation of some Google engineers. The occasion was the sixtieth anniversary of John von Neumann's proposal for a digital computer. After the visit, George wrote an essay, “Turing’s Cathedral,” which, for the first time, alerted the public about what Google 's founders had in store for the world. “We are not scanning all those books to be read by people,” explained one of his hosts after his talk. “We are scanning them to be read by an Al.” George offers a counternarrative to the digital age. His interests have included the development of the Aleut kayak, the evolution of digital computing and telecommunications, the origins of the digital universe, and a path not taken into space. His career (he never finished high school, yet has been awarded an honorary doctorate jrom the University of Victoria) has proved as impossible to classify as his books. He likes to point out that analog computing, once believed to be as extinct as the differential analyzer, has returned. He argues that while we may use digital components, at a certain point the analog computing being performed by the system far exceeds the complexity of the digital code with which it is built. He believes that true artificial intelligence—with analog control systems emerging from a digital substrate the way digital computers emerged out of analog components in the aftermath of World War II— may not be as far off as we think. In this essay, George contemplates the distinction between analog and digital computation and finds analog to be alive and well. Nature’s response to an attempt to program machines to control everything may be machines without programming over which no one has control. 36 HOUSE_OVERSIGHT_016256

THE THIRD LAW George Dyson George Dyson is a historian of science and technology and the author of Baidarka: the Kayak, Darwin Among the Machines, Project Orion, and Turing’s Cathedral. The history of computing can be divided into an Old Testament and a New Testament: before and after electronic digital computers and the codes they spawned proliferated across the Earth. The Old Testament prophets, who delivered the underlying logic, included Thomas Hobbes and Gottfried Wilhelm Leibniz. The New Testament prophets included Alan Turing, John von Neumann, Claude Shannon, and Norbert Wiener. They delivered the machines. Alan Turing wondered what it would take for machines to become intelligent. John von Neumann wondered what it would take for machines to self-reproduce. Claude Shannon wondered what it would take for machines to communicate reliably, no matter how much noise intervened. Norbert Wiener wondered how long it would take for machines to assume control. Wiener’s warnings about control systems beyond human control appeared in 1949, just as the first generation of stored-program electronic digital computers were introduced. These systems required direct supervision by human programmers, undermining his concerns. What’s the problem, as long as programmers are in control of the machines? Ever since, debate over the risks of autonomous control has remained associated with the debate over the powers and limitations of digitally coded machines. Despite their astonishing powers, little real autonomy has been observed. This is a dangerous assumption. What if digital computing is being superseded by something else? Electronics underwent two fundamental transitions over the past hundred years: from analog to digital and from vacuum tubes to solid state. That these transitions occurred together does not mean they are inextricably linked. Just as digital computation was implemented using vacuum tube components, analog computation can be implemented in solid state. Analog computation is alive and well, even though vacuum tubes are commercially extinct. There 1s no precise distinction between analog and digital computing. In general, digital computing deals with integers, binary sequences, deterministic logic, and time that is idealized into discrete increments, whereas analog computing deals with real numbers, nondeterministic logic, and continuous functions, including time as it exists as a continuum in the real world. Imagine you need to find the middle of a road. You can measure its width using any available increment and then digitally compute the middle to the nearest increment. Or you can use a piece of string as an analog computer, mapping the width of the road to the length of the string and finding the middle, without being limited to increments, by doubling the string back upon itself. Many systems operate across both analog and digital regimes. A tree integrates a wide range of inputs as continuous functions, but if you cut down that tree, you find that it has been counting the years digitally all along. 37 HOUSE_OVERSIGHT_016257

In analog computing, complexity resides in network topology, not in code. Information is processed as continuous functions of values such as voltage and relative pulse frequency rather than by logical operations on discrete strings of bits. Digital computing, intolerant of error or ambiguity, depends upon error correction at every step along the way. Analog computing tolerates errors, allowing you to live with them. Nature uses digital coding for the storage, replication, and recombination of sequences of nucleotides, but relies on analog computing, running on nervous systems, for intelligence and control. The genetic system in every living cell is a stored-program computer. Brains aren’t. Digital computers execute transformations between two species of bits: bits representing differences in space and bits representing differences in time. The transformations between these two forms of information, sequence and structure, are governed by the computer’s programming, and as long as computers require human programmers, we retain control. Analog computers also mediate transformations between two forms of information: structure in space and behavior in time. There is no code and no programming. Somehow—and we don’t fully understand how—Nature evolved analog computers known as nervous systems, which embody information absorbed from the world. They learn. One of the things they learn is control. They learn to control their own behavior, and they learn to control their environment to the extent that they can. Computer science has a long history—going back to before there even was computer science—of implementing neural networks, but for the most part these have been simulations of neural networks by digital computers, not neural networks as evolved in the wild by Nature herself. This is starting to change: from the bottom up, as the threefold drivers of drone warfare, autonomous vehicles, and cell phones push the development of neuromorphic microprocessors that implement actual neural networks, rather than simulations of neural networks, directly in silicon (and other potential substrates); and from the top down, as our largest and most successful enterprises increasingly turn to analog computation in their infiltration and control of the world. While we argue about the intelligence of digital computers, analog computing 1s quietly supervening upon the digital, in the same way that analog components like vacuum tubes were repurposed to build digital computers in the aftermath of World War II. Individually deterministic finite-state processors, running finite codes, are forming large-scale, nondeterministic, non-finite-state metazoan organisms running wild in the real world. The resulting hybrid analog/digital systems treat streams of bits collectively, the way the flow of electrons is treated in a vacuum tube, rather than individually, as bits are treated by the discrete-state devices generating the flow. Bits are the new electrons. Analog is back, and its nature is to assume control. Governing everything from the flow of goods to the flow of traffic to the flow of ideas, these systems operate statistically, as pulse-frequency coded information is processed in a neuron or a brain. The emergence of intelligence gets the attention of Homo sapiens, but what we should be worried about is the emergence of control. 38 HOUSE_OVERSIGHT_016258

Imagine it is 1958 and you are trying to defend the continental United States against airborne attack. To distinguish hostile aircraft, one of the things you need, besides a network of computers and early-warning radar sites, is a map of all commercial air traffic, updated in real time. The United States built such a system and named it SAGE (Semi-Automatic Ground Environment). SAGE in turn spawned Sabre, the first integrated reservation system for booking airline travel in real time. Sabre and its progeny soon became not just a map as to what seats were available but also a system that began to control, with decentralized intelligence, where airliners would fly, and when. But isn’t there a control room somewhere, with someone at the controls? Maybe not. Say, for example, you build a system to map highway traffic in real time, simply by giving cars access to the map in exchange for reporting their own speed and location at the time. The result is a fully decentralized control system. Nowhere is there any controlling model of the system except the system itself. Imagine it is the first decade of the 21st century and you want to track the complexity of human relationships in real time. For social life at a small college, you could construct a central database and keep it up to date, but its upkeep would become overwhelming if taken to any larger scale. Better to pass out free copies of a simple semi-autonomous code, hosted locally, and let the social network update itself. This code is executed by digital computers, but the analog computing performed by the system as a whole far exceeds the complexity of the underlying code. The resulting pulse-frequency coded model of the social graph becomes the social graph. It spreads wildly across the campus and then the world. What if you wanted to build a machine to capture what everything known to the human species means? With Moore’s Law behind you, it doesn’t take too long to digitize all the information in the world. You scan every book ever printed, collect every email ever written, and gather forty-nine years of video every twenty-four hours, while tracking where people are and what they do, in real time. But how do you capture the meaning? Even in the age of all things digital, this cannot be defined in any strictly logical sense, because meaning, among humans, isn’t fundamentally logical. The best you can do, once you have collected all possible answers, is to invite well-defined questions and compile a pulse-frequency weighted map of how everything connects. Before you know it, your system will not only be observing and mapping the meaning of things, it will start constructing meaning as well. In time, it will control meaning, in the same way as the traffic map starts to control the flow of traffic even though no one seems to be in control. There are three laws of artificial intelligence. The first, known as Ashby’s Law, after cybernetician W. Ross Ashby, author of Design for a Brain, states that any effective control system must be as complex as the system it controls. The second law, articulated by John von Neumann, states that the defining characteristic of a complex system is that it constitutes its own simplest behavioral description. The simplest complete model of an organism is the organism itself. Trying to reduce the system’s behavior to any formal description makes things more complicated, not less. 39 HOUSE_OVERSIGHT_016259

The third law states that any system simple enough to be understandable will not be complicated enough to behave intelligently, while any system complicated enough to behave intelligently will be too complicated to understand. The Third Law offers comfort to those who believe that until we understand intelligence, we need not worry about superhuman intelligence arising among machines. But there is a loophole in the Third Law. It is entirely possible to build something without understanding it. You don’t need to fully understand how a brain works in order to build one that works. This is a loophole that no amount of supervision over algorithms by programmers and their ethical advisors can ever close. Provably “good” AI is a myth. Our relationship with true AI will always be a matter of faith, not proof. We worry too much about machine intelligence and not enough about self- reproduction, communication, and control. The next revolution in computing will be signaled by the rise of analog systems over which digital programming no longer has control. Nature’s response to those who believe they can build machines to control everything will be to allow them to build a machine that controls them instead. 40 HOUSE_OVERSIGHT_016260

Dan Dennett is the philosopher of choice in the AI community. He is perhaps best known in cognitive science for his concept of intentional systems and his model of human consciousness, Which sketches a computational architecture for realizing the stream of consciousness in the massively parallel cerebral cortex. That uncompromising computationalism has been opposed by philosophers such as John Searle, David Chalmers, and the late Jerry Fodor, who have protested that the most important aspects of consciousness—intentionality and subjective qualia—cannot be computed. Twenty-five years ago, I was visiting Marvin Minsky, one of the original AI pioneers, and asked him about Dan. “He’s our best current philosopher—the next Bertrand Russell,” said Marvin, adding that unlike traditional philosophers, Dan was a student of neuroscience, linguistics, artificial intelligence, computer science, and psychology: “He’s redefining and reforming the role of the philosopher. Of course, Dan doesn’t understand my Society-of-Mind theory, but nobody’s perfect.” Dan’s view of the efforts of AI researchers to create superintelligent AIs is relentlessly levelheaded. What, me worry? In this essay, he reminds us that Als, above all, should be regarded—and treated—as tools and not as humanoid colleagues. He has been interested in information theory since his graduate school days at Oxford. In fact, he told me that early in his career he was keenly interested in writing a book about Wiener’s cybernetic ideas. As a thinker who embraces the scientific method, one of his charms is his willingness to be wrong. Of a recent piece entitled “What Is Information?” he has announced, “TI stand by it, but it’s under revision. I’m already moving beyond it and realizing there’s a better way of tackling some of these issues.” He will most likely remain cool and collected on the subject of AI research, although he has acknowledged, often, that his own ideas evolve—as anyone ’s ideas should. 41 HOUSE_OVERSIGHT_016261

WHAT CAN WE DO? Daniel C. Dennett Daniel C. Dennett is University Professor and Austin B. Fletcher Professor of Philosophy and director of the Center for Cognitive Studies at Tufts University. He is the author of a dozen books, including Consciousness Explained and, most recently, From Bacteria to Bach and Back: The Evolution of Minds. Many have reflected on the irony of reading a great book when you are too young to appreciate it. Consigning a classic to the a/ready read stack and thereby insulating yourself against any further influence while gleaning only a few ill-understood ideas from it is a recipe for neglect that is seldom benign. This struck me with particular force when I reread Zhe Human Use of Human Beings more than sixty years after my juvenile encounter. We should all make it a regular practice to reread books from our youth, where we are apt to discover clear previews of some of our own later “discoveries” and “inventions,” along with a wealth of insights to which we were bound to be impervious until our minds had been torn and tattered, exercised and enlarged by confrontations with life’s problems. Writing at a time when vacuum tubes were still the primary electronic building blocks and there were only a few actual computers in operation, Norbert Wiener imagined the future we now contend with in impressive detail and with few clear mistakes. Alan Turing’s famous 1950 article “Computing Machinery and Intelligence,” in the philosophy journal Mind, foresaw the development of AI, and so did Wiener, but Wiener saw farther and deeper, recognizing that AI would not just imitate—and replace—human beings in many intelligent activities but change human beings in the process. We are but whirlpools in a river of ever-flowing water. We are not stuff that abides, but patterns that perpetuate themselves. (p. 96) When that was written, it could be comfortably dismissed as yet another bit of Heraclitean overstatement. Yeah, yeah, you can never step in the same river twice. But it contains the seeds of the revolution in outlook. Today we know how to think about complex adaptive systems, strange attractors, extended minds, and homeostasis, a change in perspective that promises to erase the “explanatory gap”® between mind and mechanism, spirit and matter, a gap that is still ardently defended by latter-day Cartesians who cannot bear the thought that we—we ourse/ves—are self-perpetuating patterns of information-bearing matter, not “stuff that abides.” Those patterns are remarkably resilient and self-restoring but at the same time protean, opportunistic, selfish exploiters of whatever new is available to harness in their quest for perpetuation. And here is where things get dicey, as Wiener recognized. When attractive opportunities abound, we are apt to be willing to pay a little and accept some small, even trivial, cost-of-doing-business for access to new powers. And pretty soon we become so dependent on our new tools that we lose the ability to thrive without them. Options become obligatory. 8 Joseph Levine, “Materialism and Qualia: The Explanatory Gap,” Pacific Philosophical Quarterly 64, pp. 354-61 (1983). 42 HOUSE_OVERSIGHT_016262

It’s an old, old story, with many well-known chapters in evolutionary history. Most mammals can synthesize their own vitamin C, but primates, having opted for a diet composed largely of fruit, lost the innate ability. We are now obligate ingesters of vitamin C, but not obligate frugivores like our primate cousins, since we have opted for technology that allows us to make, and take, vitamins as needed. The self-perpetuating patterns that we call human beings are now dependent on clothes, cooked food, vitamins, vaccinations, .. . credit cards, smartphones, and the Internet. And—tomorrow if not already today—AI. Wiener foresaw the problems that Turing and the other optimists have largely overlooked. The real danger, he said, is that such machines, though helpless by themselves, may be used by a human being or a block of human beings to increase their control over the rest of the race or that political leaders may attempt to control their populations by means not of machines themselves but through political techniques as narrow and indifferent to human possibility as if they had, in fact, been conceived mechanically. (p. 181) The power, he recognized, lay primarily in the algorithms, not the hardware they run on, although the hardware of today makes practically possible algorithms that would have seemed preposterously cumbersome in Wiener’s day. What can we say about these “techniques” that are “narrow and indifferent to human possibility”? They have been introduced again and again, some obviously benign, some obviously dangerous, and many in the omnipresent middle ground of controversy. Consider a few of the skirmishes. My late friend Joe Weizenbaum, Wiener’s successor as MIT’s Jeremiah of hi-tech, loved to observe that credit cards, whatever their virtues, also provided an inexpensive and almost foolproof way for the government, or corporations, to track the travels and habits and desires of individuals. The anonymity of cash has been largely underappreciated, except by drug dealers and other criminals, and now it may be going extinct. This may make money laundering a more difficult technical challenge in the future, but the AI pattern finders arrayed against it have the side effect of making us all more transparent to any “block of human beings” that may “attempt to control” us. Looking to the arts, the innovation of digital audio and video recording lets us pay a small price (in the eyes of all but the most ardent audiophiles and film lovers) when we abandon analog formats, and in return provides easy—all too easy?—reproduction of artworks with almost perfect fidelity. But there is a huge hidden cost. Orwell’s Ministry of Truth is now a practical possibility. AI techniques for creating all-but-undetectable forgeries of “recordings” of encounters are now becoming available which will render obsolete the tools of investigation we have come to take for granted in the last hundred and fifty years. Will we simply abandon the brief Age of Photographic Evidence and return to the earlier world in which human memory and trust provided the gold standard, or will we develop new techniques of defense and offense in the arms race of truth? (We can imagine a return to analog film-exposed-to-light, kept in “tamper-proof” systems until shown to juries, etc., but how long would it be before somebody figured out a way to infect such systems with doubt? One of the disturbing lessons of recent experience 1s that the task of destroying a reputation for credibility is much less expensive than the task of protecting such a reputation.) Wiener saw the phenomenon at its most general: “...1n 43 HOUSE_OVERSIGHT_016263

the long run, there is no distinction between arming ourselves and arming our enemies.” (p. 129) The Information Age is also the Dysinformation Age. What can we do? We need to rethink our priorities with the help of the passionate but flawed analyses of Wiener, Weizenbaum, and the other serious critics of our technophilia. A key phrase, it seems to me, is Wiener’s almost offhand observation, above, that “these machines” are “helpless by themselves.” As I have been arguing recently, we’re making tools, not colleagues, and the great danger is not appreciating the difference, which we should strive to accentuate, marking and defending it with political and legal innovations. Perhaps the best way to see what is being missed 1s to note that Alan Turing himself suffered an entirely understandable failure of imagination in his formulation of the famous Turing Test. As everyone knows, it is an adaptation of his “imitation game,” in which a man, hidden from view and communicating verbally with a judge, tries to convince the judge that he is in fact a woman, while a woman, also hidden and communicating with the judge, tries to convince the judge that she is the woman. Turing reasoned that this would be a demanding challenge for a man (or for a woman pretending to be a man), exploiting a wealth of knowledge about how the other sex thinks and acts, what they tend to favor or ignore. Surely (ding!)?, any man who could beat a woman at being perceived to be a woman would be an intelligent agent. What Turing did not foresee is the power of deep-learning AI to acquire this wealth of information in an exploitable form without having to understand it. Turing imagined an astute and imaginative (and hence conscious) agent who cunningly designed his responses based on his detailed “theory” of what women are likely to do and say. Top-down intelligent design, in short. He certainly didn’t think that a man, winning the imitation game, would somehow become a woman; he imagined that there would still be a man’s consciousness guiding the show. The hidden premise in Turing’s almost-argument was: Only a conscious, intelligent agent could devise and control a winning strategy in the imitation game. And so it was persuasive to Turing (and others, including me, still a stalwart defender of the Turing Test) to argue that a “computing machine” that could pass as human in a contest with a human might not be conscious in just the way a human being is, but would nevertheless have to be a conscious agent of some kind. I think this is still a defensible position—the only defensible position—but you have to understand how resourceful and ingenious a judge would have to be to expose the shallowness of the facade that a deep-learning AI (a tool, not a colleague) could present. What Turing didn’t foresee is the uncanny ability of superfast computers to sift mindlessly through Big Data, of which the Internet provides an inexhaustible supply, finding probabilistic patterns in human activity that could be used to pop “authentic’- seeming responses into the output for almost any probe a judge would think to offer. Wiener also underestimates this possibility, seeing the tell-tale weakness of a machine in not being able to take into account the vast range of probability that characterizes the human situation. (p.181) ° The surely alarm (the habit of having a bell ring in your head whenever you sce the word in an argument) is described and defended by me in Jntuition Pumps and Other Tools for Thinking (2013). 44 HOUSE_OVERSIGHT_016264

But taking into account that range of probability is just where the new AI excels. The only chink in the armor of AI is that word “vast”; human possibilities, thanks to language and the culture that it spawns, are truly Vast.!° No matter how many patterns we may find with AI in the flood of data that has so far found its way onto the Internet, there are Vastly more possibilities that have never been recorded there. Only a fraction (but not a Vanishing fraction) of the world’s accumulated wisdom and design and repartee and silliness has made it onto the Internet, but probably a better tactic for the judge to adopt when confronting a candidate in the Turing Test is not to search for such items but to create them anew. AI in its current manifestations is parasitic on human intelligence. It quite indiscriminately gorges on whatever has been produced by human creators and extracts the patterns to be found there—including some of our most pernicious habits.'! These machines do not (yet) have the goals or strategies or capacities for self-criticism and innovation to permit them to transcend their databases by reflectively thinking about their own thinking and their own goals. They are, as Wiener says, helpless, not in the sense of being shackled agents or disabled agents but in the sense of not being agents at all—not having the capacity to be “moved by reasons” (as Kant put it) presented to them. It is important that we keep it that way, which will take some doing. One of the flaws in Weizenbaum’s book Computer Power and Human Reason, something I tried in vain to convince him of in many hours of discussion, is that he could never decide which of two theses he wanted to defend: AJ is impossible! or Al is possible but evil! He wanted to argue, with John Searle and Roger Penrose, that “Strong AI” is impossible, but there are no good arguments for that conclusion. After all, everything we now know suggests that, as I have put it, we are robots made of robots made of robots. . . down to the motor proteins and their ilk, with no magical ingredients thrown in along the way. Weizenbaum’s more important and defensible message was that we should not strive to create Strong AI and should be extremely cautious about the AI systems that we can create and have already created. As one might expect, the defensible thesis is a hybrid: A/ (Strong AJ) is possible in principle but not desirable. The AI that’s practically possible is not necessarily evil—unless it is mistaken for Strong AI! The gap between today’s systems and the science-fictional systems dominating the popular imagination is still huge, though many folks, both lay and expert, manage to underestimate it. Let’s consider IBM’s Watson, which can stand as a worthy landmark for our imaginations for the time being. It is the result of a very large-scale R&D process extending over many person-centuries of intelligent design, and as George Church notes in these pages, it uses thousands of times more energy than a human brain (a technological limitation that, as he also notes, may be temporary). Its victory in Jeopardy! was a genuine triumph, made possible by the formulaic restrictions of the Jeopardy! rules, but in order for it to compete, even these rules had to be revised (one of 10 Tn Darwin’s Dangerous Idea, 1995, p. 109, I coined the capitalized version, Vast, meaning Very much more than ASTronomical, and its complement, Vanishing, to replace the usual exaggerations infinite and infinitesimal for discussions of those possibilities that are not officially infinite but nevertheless infinite for all practical purposes. 1 Aylin Caliskan-Islam, Joanna J. Bryson & Arvind Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science, 14 April 2017, 356: 6334, pp. 183-6. DOI: 10.1126/science.aal4230. 45 HOUSE_OVERSIGHT_016265

those trade-offs: you give up a little versatility, a little humanity, and get a crowd- pleasing show). Watson is not good company, in spite of misleading ads from IBM that suggest a general conversational ability, and turning Watson into a plausibly multidimensional agent would be like turning a hand calculator into Watson. Watson could be a useful core faculty for such an agent, but more like a cerebellum or an amygdala than a mind—at best, a special-purpose subsystem that could play a big supporting role, but not remotely up to the task of framing purposes and plans and building insightfully on its conversational experiences. Why would we want to create a thinking, creative agent out of Watson? Perhaps Turing’s brilliant idea of an operational test has lured us into a trap: the quest to create at least the illusion of a real person behind the screen, bridging the “uncanny valley.” The danger, here, is that ever since Turing posed his challenge—which was, after all, a challenge to fool the judges—AI creators have attempted to paper over the valley with cutesy humanoid touches, Disneyfication effects that will enchant and disarm the uninitiated. Weizenbaum’s ELIZA was the pioneer example of such superficial illusion- making, and it was his dismay at the ease with which his laughably simple and shallow program could persuade people they were having a serious heart-to-heart conversation that first sent him on his mission. He was right to be worried. If there is one thing we have learned from the restricted Turing Test competitions for the Loebner Prize, it is that even very intelligent people who aren’t tuned in to the possibilities and shortcuts of computer programming are readily taken in by simple tricks. The attitudes of people in AI toward these methods of dissembling at the “user interface” have ranged from contempt to celebration, with a general appreciation that the tricks are not deep but can be potent. One shift in attitude that would be very welcome is a candid acknowledgment that humanoid embellishments are false advertising—something to condemn, not applaud. How could that be accomplished? Once we recognize that people are starting to make life-or-death decisions largely on the basis of “advice” from AI systems whose inner operations are unfathomable in practice, we can see a good reason why those who in any way encourage people to put more trust in these systems than they warrant should be held morally and legally accountable. AI systems are very powerful tools—so powerful that even experts will have good reason not to trust their own judgment over the “Judgments” delivered by their tools. But then, if these tool users are going to benefit, financially or otherwise, from driving these tools through ferra incognita, they need to make sure they know how to do this responsibly, with maximum control and justification. Licensing and bonding operators, just as we license pharmacists (and crane operators!) and other specialists whose errors and misjudgments can have dire consequences, can, with pressure from insurance companies and other underwriters, oblige creators of AI systems to go to extraordinary lengths to search for and reveal weaknesses and gaps in their products, and to train those entitled to operate them. One can imagine a sort of inverted Turing Test in which the judge is on trial; until he or she can spot the weaknesses, the overstepped boundaries, the gaps in a system, no license to operate will be issued. The mental training required to achieve certification as a judge will be demanding. The urge to adopt the intentional stance, our normal tactic whenever we encounter what seems to be an intelligent agent, is almost overpoweringly strong. Indeed, the capacity to resist the allure of treating an apparent person as a person 46 HOUSE_OVERSIGHT_016266

is an ugly talent, reeking of racism or species-ism. Many people would find the cultivation of such a ruthlessly skeptical approach morally repugnant, and we can anticipate that even the most proficient system-users would occasionally succumb to the temptation to “befriend” their tools, if only to assuage their discomfort with the execution of their duties. No matter how scrupulously the AI designers launder the phony “human” touches out of their wares, we can expect novel habits of thought, conversational gambits and ruses, traps and bluffs to arise in this novel setting for human action. The comically long lists of known side effects of new drugs advertised on television will be dwarfed by the obligatory revelations of the sorts of questions that cannot be responsibly answered by particular systems, with heavy penalties for those who “overlook” flaws in their products. It is widely noted that a considerable part of the growing economic inequality in today’s world is due to the wealth accumulated by digital entrepreneurs; we should enact legislation that puts their deep pockets in escrow for the public good. Some of the deepest pockets are voluntarily out in front of these obligations to serve society first and make money secondarily, but we shouldn’t rely on good will alone. We don’t need artificial conscious agents. There is a surfeit of natural conscious agents, enough to handle whatever tasks should be reserved for such special and privileged entities. We need intelligent tools. Tools do not have rights, and should not have feelings that could be hurt, or be able to respond with resentment to “abuses” rained on them by inept users.'* One of the reasons for not making artificial conscious agents is that however autonomous they might become (and in principle, they can be as autonomous, as self-enhancing or self-creating, as any person), they would not—without special provision, which might be waived—share with us natural conscious agents our vulnerability or our mortality. I once posed a challenge to students in a seminar at Tufts I co-taught with Matthias Scheutz on artificial agents and autonomy: Give me the specs for a robot that could sign a binding contract with you—not as a surrogate for some human owner but on its own. This isn’t a question of getting it to understand the clauses or manipulate a pen on a piece of paper but of having and deserving legal status as a morally responsible agent. Small children can’t sign such contracts, nor can those disabled people whose legal status requires them to be under the care and responsibility of guardians of one sort or another. The problem for robots who might want to attain such an exalted status is that, like Superman, they are too invulnerable to be able to make a credible promise. If they were to renege, what would happen? What would be the penalty for promise- breaking? Being locked in a cell or, more plausibly, dismantled? Being locked up is barely an inconvenience for an AI unless we first install artificial wanderlust that cannot be ignored or disabled by the AI on its own (and it would be systematically difficult to make this a foolproof solution, given the presumed cunning and self-knowledge of the AI); and dismantling an AI (either a robot or a bedridden agent like Watson) is not killing it, if the information stored in its design and software is preserved. The very ease of digital recording and transmitting—the breakthrough that permits software and data to be, 1? Joanna J. Bryson, “Robots Should Be Slaves,” in Close Engagement with Artificial Companions, Y orick Wilks, ed., (Amsterdam, The Netherlands: John Benjamins, 2010), pp. 63-74; http://www.cs.bath.ac.uk/~jb/ftp/Bry son-Slaves-Book09. html. , Patiency Is Not a Virtue: AI and the Design of Ethical Systems,” https:/Awww.cs.bath.ac.uk/~jjb/ftp/Bryson-Patiency-AAAISS16.pdf. 47 HOUSE_OVERSIGHT_016267

in effect, immortal—removes robots from the world of the vulnerable (at least robots of the usually imagined sorts, with digital software and memories). If this isn’t obvious, think about how human morality would be affected if we could make “backups” of people every week, say. Diving headfirst on Saturday off a high bridge without benefit of bungee cord would be a rush that you wouldn’t remember when your Friday night backup was put online Sunday morning, but you could enjoy the videotape of your apparent demise thereafter. So what we are creating are not—should not be—conscious, humanoid agents but an entirely new sort of entities, rather like oracles, with no conscience, no fear of death, no distracting loves and hates, no personality (but all sorts of foibles and quirks that would no doubt be identified as the “personality” of the system): boxes of truths (if we’ re lucky) almost certainly contaminated with a scattering of falsehoods. It will be hard enough learning to live with them without distracting ourselves with fantasies about the Singularity in which these Als will enslave us, literally. The Auman use of human beings will soon be changed—once again—forever, but we can take the tiller and steer between some of the hazards if we take responsibility for our trajectory. 48 HOUSE_OVERSIGHT_016268

The roboticist Rodney Brooks, featured in Errol Morris’s 1997 documentary Fast, Cheap and Out of Control along with a lion-tamer, a topiarist, and an expert on the naked mole rat, was described by one reviewer as “smiling with a wild gleam in his eye.’ But that’s pretty much true of most visionaries. A few years later in his career, Brooks, as befits one of the world’s leading roboticists, suggested that “we overanthropomorphize humans, who are after all mere machines.” He went on to present a warm-hearted vision of a coming Al world in which “the distinction between us and robots is going to disappear.” He also admitted to something of a divided worldview. “Like a religious scientist, I maintain two sets of inconsistent beliefs and act on each of them in different circumstances,” he wrote. “It is this transcendence between belief systems that I think will be what enables mankind to ultimately accept robots as emotional machines, and thereafter start to empathize with them and attribute free will, respect, and ultimately rights to them.” That was in 2002. In these pages, he takes a somewhat more jaundiced, albeit narrower, view; he is alarmed by the extent to which we have come to rely on pervasive systems that are not just exploitative but also vulnerable, as a result of the too-rapid development of software engineering—an advance that seems to have outstripped the imposition of reliably effective safeguards. J 49 HOUSE_OVERSIGHT_016269

THE INHUMAN MESS OUR MACHINES HAVE GOTTEN US INTO Rodney Brooks Rodney Brooks is a computer scientist; Panasonic Professor of Robotics, emeritus, MIT; former director, MIT Computer Science Lab; and founder, chairman, and CTO of Rethink Robotics. He is the author of Flesh and Machines. Mathematicians and scientists are often limited in how they see the big picture, beyond their particular field, by the tools and metaphors they use in their work. Norbert Wiener is no exception, and I might guess that neither am I. When he wrote The Human Use of Human Beings, Wiener was straddling the end of the era of understanding machines and animals simply as physical processes and the beginning of our current era of understanding machines and animals as computational processes. I suspect there will be future eras whose tools will look as distinct from the tools of the two eras Wiener straddled as those tools did from each other. Wiener was a giant of the earlier era and built on the tools developed since the time of Newton and Leibniz to describe and analyze continuous processes in the physical world. In 1948 he published Cybernetics, a word he coined to describe the science of communication and control in both machines and animals. Today we would refer to the ideas in this book as control theory, an indispensable discipline for the design and analysis of physical machines, while mostly neglecting Wiener’s claims about the science of communication. Wiener’s innovations were largely driven by his work during the Second World War on mechanisms to aim and fire anti-aircraft guns. He brought mathematical rigor to the design of the sorts of technology whose design processes had been largely heuristic in nature: from the Roman waterworks through Watt’s steam engine to the early development of automobiles. One can imagine a different contingent version of our intellectual and technological history had Alan Turing and John von Neumann, both of whom made major contributions to the foundations of computing, not appeared on the scene. Turing contributed a fundamental model of computation—now known as a Turing Machine—in his paper “On Computable Numbers with an Application to the Entscheidungsproblem,” written and revised in 1936 and published in 1937. In these machines, a linear tape of symbols from a finite alphabet encodes the input for a computational problem and also provides the working space for the computation. A different machine was required for each separate computational problem; later work by others would show that in one particular machine, now known as a Universal Turing Machine, an arbitrary set of computing instructions could be encoded on that same tape. In the 1940s, von Neumann developed an abstract self-reproducing machine called a cellular automaton. In this case it occupied a finite subset of an infinite two- dimensional array of squares each containing a single symbol from a finite alphabet of twenty-nine distinct symbols—the rest of the infinite array starts out blank. The single symbols in each square change in lockstep, based on a complex but finite rule about the current symbol in that square and its immediate neighbors. Under the complex rule that von Neumann developed, most of the symbols in most of the squares stay the same and a few change at each step. So when one looks at the non-blank squares, it appears that 50 HOUSE_OVERSIGHT_016270

there is a constant structure with some activity going on inside it. When von Neumann’s abstract machine reproduced, it made a copy of itself in another region of the plane. Within the “machine” was a horizontal line of squares which acted as a finite linear tape, using a subset of the finite alphabet. It was the symbols in those squares that encoded the machine of which they were a part. During the machine’s reproduction, the “tape” could move either left or right and was both interpreted (transcribed) as the instructions (translation) for the new “machine” being built and then copied (replicated)—with the new copy being placed inside the new machine for further reproduction. Francis Crick and James Watson later showed, in 1953, how such a tape could be instantiated in biology by along DNA molecule with its finite alphabet of four nucleobases: guanine, cytosine, adenine, and thymine (G, C, A, and T).'* Asin von Neumann’s machine, in biological reproduction the linear sequence of symbols in DNA is interpreted—through transcription into RNA molecules, which then are translated into proteins, the structures that make up a new cell—and the DNA is replicated and encased in the new cell. A second foundational piece of work was in a 1945 “First Draft” report on the design for a digital computer, wherein von Neumann advocated for a memory that could contain both instructions and data.'4 This is now known as a von Neumann architecture computer—as distinct from a Harvard architecture computer, where there are two separate memories, one for instructions and one for data. The vast majority of computer chips built in the era of Moore’s Law are based on the von Neumann architecture, including those powering our data centers, our laptops, and our smartphones. Von Neumann’s digital-computer architecture 1s conceptually the same generalization—from early digital computers constructed with electromagnetic relays at both Harvard University and Bletchley Park—that occurs in going from a special-purpose Turing Machine to a Universal Turing Machine. Furthermore, his self-replicating automata share a fundamental similarity with both the construction of a Turing Machine and the mechanism of DNA-based reproducing biological cells. There is to this day scholarly debate over whether von Neumann saw the cross connections between these three pieces of work, Turing’s and his two. Turing’s revision of his paper was done while he and von Neumann were both at Princeton; indeed, after getting his PhD, Turing almost stayed on as von Neumann’s postdoc. Without Turing and von Neumann, the cybernetics of Wiener might have remained a dominant mode of thought and driver of technology for much longer than its brief moment of supremacy. In this imaginary version of history, we might well live today in an actual steam-punk world and not just get to observe its fantastical instantiations at Maker Faires! My point is that Wiener thought about the world—physical, biological, and (in Human Use) sociological—in a particular way. He analyzed the world as continuous variables, as he explains in chapter 1 along with a nod to thermodynamics through an overlay of Gibbs statistics. He also shoehorns in a weak and unconvincing model of information as message-passing between and among both physical and biological entities. To me, and from today’s vantage point seventy years on, his tools seem woefully 13“ Structure for Deoxyribose Nucleic Acid,” Nature 171, 737-738 (1953). 14 hitps://en.wikipedia.org//wiki/First_Draft_of_a_Report_on_ the EDVAC#Controversy. Von Neumann is listed as the only author, whereas others contributed to the concepts he laid out; thus credit for the architecture has gone to him alone. 51 HOUSE_OVERSIGHT_016271

inadequate for describing the mechanisms underlying biological systems, and so he missed out on how similar mechanisms might eventually be embodied in technological computational systems—as now they have been. Today’s dominant technologies were developed in the world of Turing and von Neumann, rather than the world of Wiener. In the first industrial revolution, energy from a steam engine or a water wheel was used by human workers to replace their own energy. Instead of being a source of energy for physical work, people became modulators of how a large source of energy was used. But because steam engines and water wheels had to be large to be an efficient use of capital, and because in the 18th century the only technology for spatial distribution of energy was mechanical and worked only at very short range, many workers needed to be crowded around the source of energy. Wiener correctly argues that the ability to transmit energy as electricity caused a second industrial revolution. Now the source of energy could be distant from where it was used, and from the beginning of the 20th century, manufacturing could be much more dispersed as electrical-distribution grids were built. Wiener then argues that a further new technology, that of the nascent computational machines of his time, will provide yet another revolution. The machines he talks about seem to be both analog and (perhaps) digital in nature; and he points out, in The Human Use of Human Beings, that since they will be able to make decisions, both blue-collar and white-collar workers may be reduced to being mere cogs in a much bigger machine. He fears that humans might use and abuse one another through organizational structures that this capability will encourage. We have certainly seen this play out in the last sixty years, and that disruption 1s far from over. However, his physics-based view of computation protected him from realizing just how bad things might get. He saw machines’ ability to communicate as providing a new and more inhuman way of exerting command and control. He missed that within a few decades computation systems would become more like biological systems, and it seems, from his descriptions in chapter 10 of his own work on modeling some aspects of biology, that he woefully underappreciated the many orders of magnitude of further complexity of biology over physics. We are in a much more complex situation today than he foresaw, and I am worried that it is much more pernicious than even his worst imagined fears. In the 1960s, computation became firmly based on the foundations set out by Turing and von Neumann, and it was digital computation, based on the idea of finite alphabets which they both used. An arbitrarily long sequence, or string, formed by characters from a finite alphabet, can be encoded as a unique integer. As with Turing Machines themselves, the formalism for computation became that of computing an integer-valued function of a single integer-valued input. Turing and von Neumann both died in the 1950s and at that time this is how they saw computation. Neither foresaw the exponential increase in computing capability that Moore’s Law would bring—nor how pervasive computing machinery would become. Nor did they foresee two developments in our modeling of computation, each of which poses a great threat to human society. The first is rooted in the abstractions they adopted. In the fifty-year, Moore’s Law-fueled race to produce software that could exploit the doubling of computer capability every two years, the typical care and certification of engineering disciplines was thrown by the wayside. Software engineering was fast and prone to failures. This 52 HOUSE_OVERSIGHT_016272

rapid development of software without standards of correctness has opened up many routes to exploit von Neumann architecture’s storage of data and instructions in the same memory. One of the most common routes, known as “buffer overrun,” involves an input number (or long string of characters) that is bigger than the programmer expected and overflows into where the instructions are stored. By carefully designing an input number that is too big by far, someone using a piece of software can infect it with instructions not intended by the programmer, and thus change what it does. This is the basis for creating a computer virus—so named for its similarity to a biological virus. The latter injects extra DNA into a cell, and that cell’s transcription and translation mechanism blindly interprets it, making proteins that may be harmful to the host cell. Furthermore, the replication mechanism for the cell takes care of multiplying the virus. Thus, a small foreign entity can take control of a much bigger entity and bend its behavior in unexpected ways. These and other forms of digital attacks have taken the security of our everyday lives from us. We rely on computers for almost everything now. We rely on computers for our infrastructure of electricity, gas, roads, cars, trains, and airplanes; these are all vulnerable. We rely on computers for our banking, our payment of bills, our retirement accounts, our mortgages, our purchasing of goods and services—these, too, are all vulnerable. We rely on computers for our entertainment, our communications both business and personal, our physical security at home, our information about the world, and our voting systems—all vulnerable. None of this will get fixed anytime soon. In the meantime, many aspects of our society are open to vicious attacks, whether by freelancing criminals or nation-state adversaries. The second development is that computation has gone beyond simply computing functions. Instead, programs remain online continuously, and so they can gather data about a sequence of queries. Under the Wiener/Turing/von Neumann scheme, we might think of the communication pattern for a Web browser to be: User: Give me Web page A. Browser: Here is Web page A. User: Give me Web page B. Browser: Here is Web page B. Now instead it can look like this: User: Give me Web page A. Browser: Here is Web page A. [And I will secretly remember that you asked for Web page A.] User: Give me Web page B. Browser: Here is Web page B. [I see a correlation between its contents and that of the earlier requested Web page A, so Iwill update my model of you, the user, and transmit it to the company that produced me. ] 53 HOUSE_OVERSIGHT_016273

When the machine no longer simply computes a function but instead maintains a state, it can start to make inferences about the human by the sequence of requests presented to it. And when different programs correlate across different request streams— say, correlating Web-page searches with social-media posts, or the payment for services on another platform, or the dwell time on a particular advertisement, or where the user has walked or driven with their GPS-enabled smartphone—the total systems of many programs communicating with one another and with databases leads to a whole new loss of privacy. The great exploitative leap made by so many West Coast companies has been to monetize those inferences without the knowing permission of the person generating the interactions with the computing machine platforms. Wiener, Turing, and von Neumann could not foresee the complexity of those platforms, wherein the legal mumbo-jumbo of the terms-of-use contracts the humans willingly enter into, without an inkling of what they entail, leads them to give up rights they would never concede in a one-on-one interaction with another human being. The computation platforms have become a shield behind which some companies hide in order to inhumanly exploit others. In certain other countries, the governments carry out these manipulations, and there the goal 1s not profits but the suppression of dissent. Humankind has gotten itself into a fine pickle: We are being exploited by companies that paradoxically deliver services we crave, and at the same time our lives depend on many software-enabled systems that are open to attack. Getting ourselves out of this mess will be a long-term project. It will involve engineering, legislation, and most important, moral leadership. Moral leadership is the first and biggest challenge. 54 HOUSE_OVERSIGHT_016274

I first met Frank Wilczek in the 1980s, when he invited me to his home in Princeton to talk about anyons. “The address is 112 Mercer Street,” he wrote. “Look for the house with no driveway.” So there I was, a few hours later, in Einstein’s old living room, talking to a future recipient of the Nobel Prize in physics. If Frank was as impressed as I was by the surroundings, you’d never guess it. His only comment concerned the difficulty of finding a parking place in front of a “house with no driveway.” Unlike most theoretical physicists, Frank has long had a keen interest in Al, as witnessed in these three “Observations”’: 1. “Francis Crick called it ‘the Astonishing Hypothesis’: that consciousness, also known as Mind, is an emergent property of matter,” which, if true, indicates that “all intelligence is machine intelligence. What distinguishes natural from artificial intelligence is not what it is, but only how it is made.” 2. “Artificial intelligence is not the product of an alien invasion. It is an artifact of a particular human culture and reflects the values of that culture.” 3. “David Hume's striking statement ‘Reason Is, and Ought only to Be, the Slave of the Passions’ was written in 1738 [and] was, of course, meant to apply to human reason and human passions. ... But Hume’s logical/philosophical point remains valid for Al. Simply put: Incentives, not abstract logic, drive behavior.” He notes that “the big story of the 20th and the 21st century is that [as] computing develops, we learn how to calculate the consequences of the [fundamental] laws better and better. There’s also a feedback cycle: When you understand matter better, you can design better computers, which will enable you to calculate better. It’s kind of an ascending helix.” Here he argues that human intelligence, for now, holds the advantage—yet our future, unbounded by our solar system and doubtless also by our galaxy, will never be realized without the help of our AIs. 55 HOUSE_OVERSIGHT_016275

THE UNITY OF INTELLIGENCE Frank Wilczek Frank Wilczek is Herman Feshbach Professor of Physics at MIT, recipient of the 2004 Nobel Prize in physics, and the author of A Beautiful Question: Finding Nature’s Deep Design. I. A Simple Answer to Contentious Questions: e Can an artificial intelligence be conscious? e Can an artificial intelligence be creative? e Can an artificial intelligence be evil? Those questions are often posed today, both in popular media and in scientifically informed debates. But the discussions never seem to converge. Here I’ll begin by answering them as follows: Based on physiological psychology, neurobiology, and physics, it would be very surprising if the answers were not Yes, Yes, and Yes. The reason is simple, yet profound: Evidence from those fields makes it overwhelmingly likely that there is no sharp divide between natural and artificial intelligence. In his 1994 book of that title, the renowned biologist Francis Crick proposed an “astonishing hypothesis”: that mind emerges from matter. He famously claimed that mind, in all its aspects, 1s “no more than the behavior of a vast assembly of nerve cells and their associated molecules.” The “astonishing hypothesis” is in fact the foundation of modern neuroscience. People try to understand how minds work by understanding how brains function; and they try to understand how brains function by studying how information is encoded in electrical and chemical signals, transformed by physical processes, and used to control behavior. In that scientific endeavor, they make no allowance for extraphysical behavior. So far, in thousands of exquisite experiments, that strategy has never failed. It has never proved necessary to allow for the influence of consciousness or creativity unmoored from brain activity to explain any observed fact of psychophysics or neurobiology. No one has ever stumbled upon a power of mind which is separate from conventional physical events in biological organisms. While there are many things we do not understand about brains, and about minds, the “astonishing hypothesis” has held intact. If we broaden our view beyond neurobiology to consider the whole range of scientific experimentation, the case becomes still more compelling. In modern physics, the foci of interest are often extremely delicate phenomena. To investigate them, experimenters must take many precautions against contamination by “noise.” They often find it necessary to construct elaborate shielding against stray electric and magnetic fields; to compensate for tiny vibrations due to micro-earthquakes or passing cars; to work at extremely low temperatures and in high vacuum, and so forth. But there’s a notable exception: They have never found it necessary to make allowances for what people nearby (or, for that matter, far away) are thinking. No “thought waves,” separate from known physical processes yet capable of influencing physical events, seem to exist. That conclusion, taken at face value, erases the distinction between natural and artificial intelligence. It implies that if we were to duplicate, or accurately simulate, the physical processes occurring in a brain—as, in principle, we can—and wire up its input 56 HOUSE_OVERSIGHT_016276

and output to sense organs and muscles, then we would reproduce, in a physical artifact, the observed manifestations of natural intelligence. Nothing observable would be missing. As an observer, I'd have no less (and no more) reason to ascribe consciousness, creativity, or evil to that artifact than I do to ascribe those properties to its natural counterparts, like other human beings. Thus, by combining Crick’s “astonishing hypothesis” in neurobiology with powerful evidence from physics, we deduce that natural intelligence is a special case of artificial intelligence. That conclusion deserves a name, and I will call it “the astonishing corollary.” With that, we have the answer to our three questions. Since consciousness, creativity, and evil are obvious features of natural human intelligence, they are possible features of artificial intelligence. A hundred years ago, or even fifty, to believe the hypothesis that mind emerges from matter, and to infer our corollary that natural intelligence is a special case of artificial intelligence, would have been leaps of faith. In view of the many surrounding gaps—chasms, really—in contemporary understanding of biology and physics, they were genuinely doubtful propositions. But epochal developments in those areas have changed the picture: In biology: A century ago, not only thought but also metabolism, heredity, and perception were deeply mysterious aspects of life that defied physical explanation. Today, of course, we have extremely rich and detailed accounts of metabolism, heredity, and many aspects of perception, from the bottom up, starting at the molecular level. In physics: After a century of quantum physics and its application to materials, physicists have discovered, over and over, how rich and strange the behavior of matter can be. Superconductors, lasers, and many other wonders demonstrate that large assemblies of molecular units, each simple in itself, can exhibit qualitatively new, “emergent” behavior, while remaining fully obedient to the laws of physics. Chemistry, including biochemistry, is a cornucopia of emergent phenomena, all now quite firmly grounded in physics. The pioneering physicist Philip Anderson, in an essay titled “More Is Different,” offers a classic discussion of emergence. He begins by acknowledging that “the reductionist hypothesis [1.e., the completeness of physical explanations based on known interactions of simple parts] may still be a topic for controversy among philosophers, but among the great majority of active scientists I think it is accepted without question.” But he goes on to emphasize that “[t]he behavior of large and complex aggregates of elementary particles, it turns out, is not to be understood in terms of a simple extrapolation of the properties of a few particles.”!> Each new level of size and complexity supports new forms of organization, whose patterns encode information in new ways and whose behavior is best described using new concepts. Electronic computers are a magnificent example of emergence. Here, all the cards are on the table. Engineers routinely design, from the bottom up, based on known (and quite sophisticated) physical principles, machines that process information in extremely impressive ways. Your iPhone can beat you at chess, quickly collect and deliver information about anything, and take great pictures, too. Because the process whereby computers, smartphones, and other intelligent objects are designed and manufactured is completely transparent, there can be no doubt that their wonderful 1S Science, 4 August 1972, Vol. 177, No. 4047, pp. 393-96. 57 HOUSE_OVERSIGHT_016277

capabilities emerge from regular physical processses, which we can trace down to the level of electrons, photons, quarks, and gluons. Evidently, brute matter can get pretty smart. Let me summarize the argument. From two strongly supported hypotheses, we’ve drawn a straightforward conclusion: e Human mind emerges from matter. e Matter is what physics says it is. e Therefore, the human mind emerges from physical processes we understand and can reproduce artificially. e Therefore, natural intelligence is a special case of artificial intelligence. Of course, our “astonishing corollary” could fail; the first two lines of this argument are hypotheses. But their failure would have to bring in a foundation-shattering discovery—a significant new phenomenon, with large-scale physical consequences, which takes place in unremarkable, well-studied physical circumstances (1.e., the materials, temperatures, and pressures inside human brains) yet which has somehow managed for many decades to elude determined investigators armed with sophisticated instruments. Such a discovery would be. . . astonishing. ID. The Future of Intelligence It is part of human nature to improve on human bodies and minds. Historically, clothing, eyeglasses, and watches are examples of increasingly sophisticated augmentations that enhance our toughness, perception, and awareness. They are major improvements to the natural human endowment, whose familiarity should not blind us to their depth. Today smartphones and the Internet are bringing the human drive toward augmentation into realms more central to our identity as intelligent beings. They are giving us, in effect, quick access to a vast collective awareness and a vast collective memory. At the same time, autonomous artificial intelligences have become world champions in a wide variety of “cerebral” games, such as chess and Go, and have taken over many sophisticated pattern-recognition tasks, such as reconstructing what happened during complex reactions at the Large Hadron Collider from a blizzard of emerging particle tracks, to find new particles; or gathering clues from fuzzy X-ray, fMRI, and other types of images, to diagnose medical problems. Where is this drive toward self-enhancement and innovation taking us? While the precise sequence of events and the timescale over which they’ll play out is impossible to predict (or, at least, beyond me), some basic considerations suggest that eventually the most powerful embodiments of mind will be quite different things from human brains as we know them today. Consider six factors whereby information-processing technology exceeds human capabilities—vastly, qualitatively, or both: e Speed: The orchestrated motion of electrons, which is the heart of modern artificial information-processing, can be much faster than the processes of diffusion and chemical change by which brains operate. Typical modern computer clock rates approach 10 gigahertz, corresponding to 10 billion operations per second. No single measure of speed applies to the bewildering variety of brain processes, but one fundamental limitation is latency of action 58 HOUSE_OVERSIGHT_016278

potentials, which limits their spacing to a few 10s per second. It is probably no accident that the “frame rate,” at which we can distinguish that movies are actually a sequence of stills, is about 40 per second. Thus, electronic processing is close to a billion times faster. e Size: The linear dimension of a typical neuron is about 10 microns. Molecular dimensions, which set a practical limit, are about 10,000 times smaller, and artificial processing units are approaching that scale. Smallness makes communication more efficient. e Stability: Whereas human memory 1s essentially continuous (analog), artificial memory can incorporate discrete (digital) features. Whereas analog quantities can erode, digital quantities can be stored, refreshed, and maintained with complete accuracy. e Duty Cycle: Human brains grow tired with effort. They need time off to take nourishment and to sleep. They carry the burden of aging. Most profoundly: They die. e Modularity (open architecture): Because artificial information processors can support precisely defined digital interfaces, they can readily assimilate new modules. Thus, if we want a computer to “see” ultraviolet or infrared or “hear” ultrasound, we can feed the output from an appropriate sensor directly into its “nervous system.” The architecture of brains is much more closed and opaque, and the human immune system actively resists implants. e Quantum readiness: One case of modularity deserves special mention, because of its long-term potential. Recently physicists and information scientists have come to appreciate that the principles of quantum mechanics support new computing principles, which can empower qualitatively new forms of information processing and (plausibly) new levels of intelligence. But these possibilities rely on aspects of quantum behavior which are quite delicate and seem especially unsuitable for interfacing with the warm, wet, messy enviroment of human brains. Evidently, as platforms for intelligence, human brains are far from optimal. Still, although versatile housekeeping robots or mechanical soldiers would find ready, lucrative markets, at present there is no machine that approaches the kind of general-purpose human intelligence those applications would require. Despite their relative weakness on many fronts, human brains have some big advantages over their artificial competitors. Let me mention five: e Three-dimensionality: Although, as noted, the linear dimensions of existing artificial processing units are vastly smaller than those of brains, the procedure by which they’re made—centered on lithography (basically, etching)—is essentially two-dimensional. That is revealed visibly in the geometry of computer boards and chips. Of course, one can stack boards, but the spacing between layers is much larger, and communication much less efficient, than within layers. Brains make better use of all three dimensions. e Self-repair: Human brains can recover from, or work around, many kinds of injuries or errors. Computers often must be repaired or rebooted externally. 59 HOUSE_OVERSIGHT_016279

e Connectivity: Human neurons typically support several hundred connections (synapses). Moreover, the complex pattern of these connections is very meaningful. (See our next point.) Computer units typically make only a handful of connections, in regular, fixed patterns. e Development (self-assembly with interactive sculpting): The human brain grows its units by cell divisions and orchestrates them into coherent structures by movement and folding. It also proliferates an abundance of connections among the cells. An important part of its sculpting occurs through active processes during infancy and childhood, as the individual interacts with his or her environment. In this process, many connections are winnowed away, while others are strengthened, depending on their effectiveness in use. Thus, the fine structure of the brain is tuned through interaction with the external world—a rich source of information and feedback! e /ntegration (sensors and actuators): The human brain comes equipped with a variety of sensory organs, notably including its outgrowth eyes, and with versatile actuators, including hands that build, legs that walk, and mouths that speak. Those sensors and actuators are seamlessy integrated into the brain’s information- processing centers, having been honed over millions of years of natural selection. We interpret their raw signals and control their large-scale actions with minimal conscious attention. The flip side is that we don’t know how we do it, and the implementation is opaque. It’s proving surprisingly difficult to reach human standards on these “routine” input-output functions. These advantages of human brains over currently engineered artifacts are profound. Human brains supply an inspiring existence proof, showing us several ways we can get more out of matter. When, if ever, will our engineering catch up? I don’t know for sure, but let me offer some informed opinions. The challenges of three-dimensionality and, to a lesser extent, self-repair don’t look overwhelming. They present some tough engineering problems, but many incremental improvements are easy to imagine, and there are clear paths forward. And while the powers of human eyes, hands, and other sensory organs and actuators are wonderfully effective, their abilities are far from exhausting any physical limits. Optical systems can take pictures with higher resolution in space, time, and color, and in more regions of the electromagnetic spectrum; robots can move faster and be stronger; and so forth. In these domains, the components necessary for superhuman performance, along many axes, are already available. The bottleneck is getting information into and out of them, rapidly, in the language of the information-processing units. And this brings us to the remaining, and I think most profound, advantages of brains over artificial devices, which stem from their connectivity and interactive development. Those two advantages are synergistic, since it is interactive development that sculpts the massively wired but sprawling structure of the infant brain, enabled by exponential growth of neurons and synapses, to get tuned into the extraordinary instrument it becomes. Computer scientists are beginning to discover the power of the brain’s architecture: Neural nets, whose basic design, as their name suggests, was directly inspired by the brain’s, have scored some spectacular successes in game playing and pattern recognition, as noted. But present-day engineering has nothing comparable—in 60 HOUSE_OVERSIGHT_016280

the (currently) esoteric domain of self-reproducing machines—to the power and versatility of neurons and their synapses. This could become a new, great frontier of research. Here too, biology might point the way, as we come to understand biological development well enough to imitate its essence. Altogether, the advantages of artificial over natural intelligence appear permanent, while the advantages of natural over artificial intelligence, though substantial at present, appear transient. I’d guess that it will be many decades before engineering catches up, but—barring catastrophic wars, climate change, or plagues, so that technological progress stays vigorous—few centuries. If that’s right, we can look forward to several generations during which humans, empowered and augmented by smart devices, coexist with increasingly capable autonomous Als. There will be a complex, rapidly changing ecology of intelligence, and rapid evolution in consequence. Given the intrinsic advantages that engineered devices will eventually offer, the vanguard of that evolution will be cyborgs and superminds, rather than lightly adorned Homo sapiens. Another important impetus will come from the exploration of hostile environments, both on Earth (e.g., the deep ocean) and, especially, in space. The human body is poorly adapted to conditions outside a narrow band of temperatures, pressures, and atmospheric composition. It needs a wide variety of specific, complex nutrients, and plenty of water. Also, it is not radiation-hardened. As the manned space program has amply demonstrated, it is difficult and expensive to maintain humans outside their terrestrial comfort zone. Cyborgs or autonomous AIs could be much more effective in these explorations. Quantum Als, with their sensitivity to noise, might even be happier in the cold and dark of deep space. In a moving passage from his 1935 novel Odd John, science fiction’s singular genius Olaf Stapledon has his hero, a superhuman (mutant) intelligence, describe Homo sapiens as “the Archaeopteryx of the spirit.” He says this, fondly, to his friend and biographer, who is a normal human. Archaeopteryx was a noble creature, and a bridge to greater ones. 61 HOUSE_OVERSIGHT_016281

I was introduced to Max Tegmark some years ago by his MIT colleague Alan Guth, the father of the inflationary universe. A distinguished theoretical physicist and cosmologist himself, Max’s principal concern nowadays is the looming existential risk posed by the creation of an AGI (artificial general intelligence—that is, one that matches human intelligence). Four years ago, Max co-founded, with Jaan Tallinn and others, the Future of Life Institute (FL), which bills itself as “an outreach organization working to ensure that tomorrow ’s most powerful technologies are beneficial for humanity.” While ona book tour in London, he was in the midst of planning for FLI, and he admits being driven to tears in a tube station after a trip to the London Science Museum, with its exhibitions spanning the gamut of humanity’s technological achievements. Was all that impressive progress in vain? FLI’s scientific advisory board includes Elon Musk, Frank Wilczek, George Church, Stuart Russell, and the Oxford philosopher Nick Bostrom, who dreamed up an oft-quoted Gedankenexperiment that results in a world full of paper clips and nothing else, produced by an (apparently) well-meaning AGI who was just following orders. The Institute sponsors conferences (Puerto Rico 2015, Asilomar 2017) on AI safety issues and in 2018 instituted a grants competition focusing on research in aid of maximizing the societal benefits of AGI. While Max is sometimes listed—by the non-cognoscenti—on the side of the scaremongers, he believes, like Frank Wilczek, ina future that will immensely benefit jrom AGI if in the attempt to create it, we can keep the human species from being sidelined. 62 HOUSE_OVERSIGHT_016282

LET’S ASPIRE TO MORE THAN MAKING OURSELVES OBSOLETE Max Tegmark Max Tegmark is an MIT physicist and AI researcher; president of the Future of Life Institute; scientific director of the Foundational Questions Institute; and the author of Our Mathematical Universe and Life 3.0: Being Human in the Age of Artificial Intelligence. Although there’s great controversy about how and when AI will impact humanity, the situation is clearer from a cosmic perspective: The technology-developing life that has evolved on Earth is rushing to make itself obsolete without devoting much serious thought to the consequences. This strikes me as embarrassingly lame, given that we can create amazing opportunities for humanity to flourish like never before, if we dare to steer a more ambitious course. 13.8 billion years after its birth, our Universe has become aware of itself. On a small blue planet, tiny conscious parts of our Universe have discovered that what they once thought was the sum total of existence was a minute part of something far grander: a solar system in a galaxy in a universe with over 100 billion other galaxies, arranged into an elaborate pattern of groups, clusters, and superclusters. Consciousness is the cosmic awakening; it transformed our Universe from a mindless zombie with no self-awareness into a living ecosystem harboring self-reflection, beauty, hope, meaning, and purpose. Had that awakening never taken place, our Universe would have been pointless—a gigantic waste of space. Should our Universe go back to sleep permanently due to some cosmic calamity or self-inflicted mishap, it will become meaningless again. On the other hand, things could get even better. We don’t yet know whether we humans are the only stargazers in the cosmos, or even the first, but we’ve already learned enough about our Universe to know that it has the potential to wake up much more fully than it has thus far. AI pioneers such as Norbert Wiener have taught us that a further awakening of our Universe’s ability to process and experience information need not require eons of additional evolution but perhaps mere decades of human scientific ingenuity. We may be like that first glimmer of self-awareness you experienced when you emerged from sleep this morning, a premonition of the much greater consciousness that would arrive once you opened your eyes and fully awoke. Perhaps artificial superintelligence will enable life to spread throughout the cosmos and flourish for billions or trillions of years, and perhaps this will be because of decisions we make here, on our planet, in our lifetime. Or humanity may soon go extinct, through some self-inflicted calamity caused by the power of our technology growing faster than the wisdom with which we manage it. The evolving debate about AI’s societal impact Many thinkers dismiss the idea of superintelligence as science fiction, because they view intelligence as something mysterious that can exist only in biological organisms— especially humans—and as fundamentally limited to what today’s humans can do. But from my perspective as a physicist, intelligence is simply a certain kind of information 63 HOUSE_OVERSIGHT_016283

processing performed by elementary particles moving around, and there’s no law of physics that says one can’t build machines more intelligent in every way than we are, and able to seed cosmic life. This suggests that we’ve seen just the tip of the intelligence iceberg; there’s an amazing potential to unlock the full intelligence latent in nature and use it to help humanity flourish—or flounder. Others, including some of the authors in this volume, dismiss the building of an AGI (Artificial General Intelligence—an entity able to accomplish any cognitive task at least as well as humans) not because they consider it physically impossible but because they deem it too difficult for humans to pull off in less than a century. Among professional AI researchers, both types of dismissal have become minority views because of recent breakthroughs. There is a strong expectation that AGI will be achieved within a century, and the median forecast is only decades away. A recent survey of AI researchers by Vincent Muller and Nick Bostrom concludes: [T]he results reveal a view among experts that AI systems will probably (over 50%) reach overall human ability by 2040-50, and very likely (with 90% probability) by 2075. From reaching human ability, it will move on to superintelligence in 2 years (10%) to 30 years (75%) thereafter. '° In the cosmic perspective of gigayears, it makes little difference whether AGI arrives in thirty or three hundred years, so let’s focus on the implications rather than the timing. First, we humans discovered how to replicate some natural processes with machines, making our own heat, light, and mechanical horsepower. Gradually we realized that our bodies were also machines, and the discovery of nerve cells blurred the boundary between body and mind. Finally, we started building machines that could outperform not only our muscles but our minds as well. We’ve now been eclipsed by machines in the performance of many narrow cognitive tasks, ranging from memorization and arithmetic to game play, and we are in the process of being overtaken in many more, from driving to investing to medical diagnosing. If the AI community succeeds in its original goal of building AGI, then we will have, by definition, been eclipsed at all cognitive tasks. This begs many obvious questions. For example, will whoever or whatever controls the AGI control Earth? Should we aim to control superintelligent machines? If not, can we ensure that they understand, adopt, and retain human values? As Norbert Wiener put it in Zhe Human Use of Human Beings: Woe to us if we let [the machine] decide our conduct, unless we have previously examined the laws of its action, and know fully that its conduct will be carried out on principles acceptable to us! On the other hand, the machine . .. , which can learn and can make decisions on the basis of its learning, will in no way be obliged to make such decisions as we should have made, or will be acceptable to us. 16 Vincent C. Miller & Nick Bostrom, “Future Progress in Artificial Intelligence: A Survey of Expert Opinion,” in Fundamental Issues of Artificial Intelligence, Vincent C. Muller, ed. (Springer International Publishing Switzerland, 2016), pp. 555-72. https://nickbostrom.com/papers/survey .pdf. 64 HOUSE_OVERSIGHT_016284

And who are the “us”? Who should deem “such decisions . . . acceptable”? Even if future powers decide to help humans survive and flourish, how will we find meaning and purpose in our lives if we aren’t needed for anything? The debate about the societal impact of AI has changed dramatically in the last few years. In 2014, what little public talk there was of AI risk tended to be dismissed as Luddite scaremongering, for one of two logically incompatible reasons: (1) AGI was overhyped and wouldn’t happen for at least another century. (2) AGI would probably happen sooner but was virtually guaranteed to be beneficial. Today, talk of AI’s societal impact is everywhere, and work on AI safety and AI ethics has moved into companies, universities, and academic conferences. The controversial position on AI safety research is no longer to advocate for it but to dismiss it. Whereas the open letter that emerged from the 2015 Puerto Rico AI conference (and helped mainstream AI safety) spoke only in vague terms about the importance of keeping AI beneficial, the 2017 Asilomar AI Principles (see below) had real teeth: They explicitly mention recursive self-improvement, superintelligence, and existential risk, and were signed by AI industry leaders and over a thousand AI researchers from around the world. Nonetheless, most discussion is limited to the near-term impact of narrow AI and the broader community pays only limited attention to the dramatic transformations that AGI may soon bring to life on Earth. Why? Why we’re rushing to make ourselves obsolete, and why we avoid talking about it First of all, there’s simple economics. Whenever we figure out how to make another type of human work obsolete by building machines that do it better and cheaper, most of society gains: Those who build and use the machines make profits, and consumers get more affordable products. This will be as true of future investor AGIs and scientist AGIs as it was of weaving machines, excavators, and industrial robots. In the past, displaced workers usually found new jobs, but this basic economic incentive will remain even if that is no longer the case. The existence of affordable AGI means, by definition, that all jobs can be done more cheaply by machines, so anyone claiming that “people will always find new well-paying jobs” is in effect claiming that AI researchers will fail to build AGI. Second, Homo sapiens is by nature curious, which will motivate the scientific quest for understanding intelligence and developing AGI even without economic incentives. Although curiosity is one of the most celebrated human attributes, it can cause problems when it fosters technology we haven’t yet learned how to manage wisely. Sheer scientific curiosity without profit motive contributed to the discovery of nuclear weapons and tools for engineering pandemics, so it’s not unthinkable that the old adage “Curiosity killed the cat” will turn out to apply to the human species as well. Third, we’re mortal. This explains the near unanimous support for developing new technologies that help us live longer, healthier lives, which strongly motivates current AI research. AGI can clearly aid medical research even more. Some thinkers even aspire to near immortality via cyborgization or uploading. We’re thus on the slippery slope toward AGI, with strong incentives to keep sliding downward, even though the consequence will by definition be our economic obsolescence. We will no longer be needed for anything, because all jobs can be done 65 HOUSE_OVERSIGHT_016285

more efficiently by machines. The successful creation of AGI would be the biggest event in human history, so why is there so little serious discussion of what it might lead to? Here again, the answer involves multiple reasons. First, as Upton Sinclair famously quipped, “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”!” For example, spokesmen for tech companies or university research groups often claim there are no risks attached to their activities even if they privately think otherwise. Sinclair’s observation may help explain not only reactions to risks from smoking and climate change but also why some treat technology as a new religion whose central articles of faith are that more technology is always better and whose heretics are clueless scaremongering Luddites. Second, humans have a long track record of wishful thinking, flawed extrapolation of the past, and underestimation of emerging technologies. Darwinian evolution endowed us with powerful fear of concrete threats, not of abstract threats from future technologies that are hard to visualize or even imagine. Consider trying to warn people in 1930 of a future nuclear arms race, when you couldn’t show them a single nuclear explosion video and nobody even knew how to build such weapons. Even top scientists can underestimate uncertainty, making forecasts that are either too optimistic— Where are those fusion reactors and flying cars?—or too pessimistic. Ernest Rutherford, arguably the greatest nuclear physicist of his time, said in 1933—less than twenty-four hours before Leo Szilard conceived of the nuclear chain reaction—that nuclear energy was “moonshine.” Essentially nobody at that time saw the nuclear arms race coming. Third, psychologists have discovered that we tend to avoid thinking of disturbing threats when we believe there’s nothing we can do about them anyway. In this case, however, there are many constructive things we can do, if we can get ourselves to start thinking about the issue. What can we do? I’m advocating a strategy change from “Let’s rush to build technology that makes us obsolete—what could possibly go wrong?” to “Let’s envision an inspiring future and steer toward it.” To motivate the effort required for steering, this strategy begins by envisioning an enticing destination. Although Hollywood’s futures tend to be dystopian, the fact is that AGI can help life flourish as never before. Everything I love about civilization is the product of intelligence, so if we can amplify our own intelligence with AGI, we have the potential to solve today’s and tomorrow’s thorniest problems, including disease, climate change, and poverty. The more detailed we can make our shared positive visions for the future, the more motivated we will be to work together to realize them. What should we do in terms of steering? The twenty-three Asilomar principles adopted in 2017 offer plenty of guidance, including these short-term goals: (1) An arms race in lethal autonomous weapons should be avoided. (2) The economic prosperity created by AI should be shared broadly, to benefit all of humanity. 7 Upton Sinclair, , Candidate for Governor: And How I Got Licked (Berkeley CA: University of California Press, 1994), p. 109. 66 HOUSE_OVERSIGHT_016286

(3) Investments in AI should be accompanied by funding for research on ensuring its beneficial use. ... How can we make future AI systems highly robust, so that they do what we want without malfunctioning or getting hacked.'® The first two involve not getting stuck in suboptimal Nash equilibria. An out-of- control arms race in lethal autonomous weapons that drives the price of automated anonymous assassination toward zero will be very hard to stop once it gains momentum. The second goal would require reversing the current trend in some Western countries where sectors of the population are getting poorer in absolute terms, fueling anger, resentment, and polarization. Unless the third goal can be met, all the wonderful AI technology we create might harm us, either accidentally or deliberately. AI safety research must be carried out with a strict deadline in mind: Before AGI arrives, we need to figure out how to make AI understand, adopt, and retain our goals. The more intelligent and powerful machines get, the more important it becomes to align their goals with ours. As long as we build relatively dumb machines, the question isn’t whether human goals will prevail but merely how much trouble the machines can cause before we solve the goal-alignment problem. If a superintelligence 1s ever unleashed, however, it will be the other way around: Since intelligence is the ability to accomplish goals, a superintelligent AI is by definition much better at accomplishing its goals than we humans are at accomplishing ours, and will therefore prevail. In other words, the real risk with AGI isn’t malice but competence. A superintelligent AGI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble. People don’t think twice about flooding anthills to build hydroelectric dams, so let’s not place humanity in the position of those ants. Most researchers argue that if we end up creating superintelligence, we should make sure it’s what Al-safety pioneer Eliezer Yudkowsky has termed “friendly AI’—AI whose goals are in some deep sense beneficial. The moral question of what these goals should be is just as urgent as the technical questions about goal alignment. For example, what sort of society are we hoping to create, where we find meaning and purpose in our lives even though we, strictly speaking, aren’t needed? I’m often given the following glib response to this question: “Let’s build machines that are smarter than us and then let them figure out the answer!” This mistakenly equates intelligence with morality. Intelligence isn’t good or evil but morally neutral. It’s simply an ability to accomplish complex goals, good or bad. We can’t conclude that things would have been better if Hitler had been more intelligent. Indeed, postponing work on ethical issues until after goal-aligned AGI is built would be irresponsible and potentially disastrous. A perfectly obedient superintelligence whose goals automatically align with those of its human owner would be like Nazi SS- Obersturmbannfithrer Adolf Eichmann on steroids. Lacking moral compass or inhibitions of its own, it would, with ruthless efficiency, implement its owner’s goals, whatever they might be.!? When I speak of the need to analyze technology risk, ’m sometimes accused of scaremongering. But here at MIT, where I work, we know that such risk analysis isn’t scaremongering: It’s safety engineering. Before the moon-landing mission, NASA 18 https://futureoflife.org/ai-principles/ 19 See, for example, Hannah Arendt, Eichmann in Jerusalem: A Report on the Banality of Evil (New York: Penguin Classics, 2006). 67 HOUSE_OVERSIGHT_016287

systematically thought through everything that could possibly go wrong when putting astronauts on top of a 110-meter rocket full of highly flammable fuel and launching them to a place where nobody could help them—and there were lots of things that could go wrong. Was this scaremongering? No, this was the safety engineering that ensured the mission’s success. Similarly, we should analyze what could go wrong with AI to ensure that it goes right. Outlook In summary, if our technology outpaces the wisdom with which we manage it, it can lead to our extinction. It’s already caused the extinction of from 20 to 50 percent of all species on Earth, by some estimates,”° and it would be ironic if we’re next in line. It would also be pathetic, given that the opportunities offered by AGI are literally astronomical, potentially enabling life to flourish for billions of years not only on Earth but also throughout much of our cosmos. Instead of squandering this opportunity through unscientific risk denial and poor planning, let’s be ambitious! Homo sapiens is inspiringly ambitious, as reflected in William Ernest Henley’s famous lines from /nvictus: “I am the master of my fate, / Iam the captain of my soul.” Rather than drifting like a rudderless ship toward our own obsolescence, let’s take on and overcome the technical and societal challenges standing between us and a good high-tech future. What about the existential challenges related to morality, goals, and meaning? There’s no meaning encoded in the laws of physics, so instead of passively waiting for our Universe to give meaning to us, let’s acknowledge and celebrate that it’s we conscious beings who give meaning to our Universe. Let’s create our own meaning, based on something more profound than having jobs. AGI can enable us to finally become the masters of our own destiny. Let’s make that destiny a truly inspiring one! °° See Elizabeth Kolbert, The Sixth Extinction: An Unnatural History (New York: Henry Holt, 2014). 68 HOUSE_OVERSIGHT_016288

Jaan Tallinn grew up in Estonia, becoming one of its few computer game developers, when that nation was still a Soviet Socialist Republic. Here he compares the dissidents who brought down the Iron Curtain to the dissidents who are sounding the alarm about rapid advances in artificial intelligence. He locates the roots of the current AI dissidence, paradoxically, among such pioneers of the AI field as Wiener, Alan Turing, and I. J. Good. Jaan’s preoccupation is with existential risk, AI being among the most extreme of many. In 2012, he co-founded the Centre for the Study of Existential Risk—an interdisciplinary research institute that works to mitigate risks “associated with emerging technologies and human activity” —at the University of Cambridge, along with Philosopher Huw Price and Martin Rees, the Astronomer Royal. He once described himself to me as “a convinced consequentialist” —convinced enough to have given away much of his entrepreneurial wealth to the Future of Life Institute (of which he is a co-founder), the Machine Intelligence Research Institute, and other such organizations working on risk reduction. Max Tegmark has written about him: “Tf you’re an intelligent life-form reading this text millions of years from now and marveling at how life is flourishing, you may owe your existence to Jaan.” On a recent visit to London, Jaan and I participated on an AI panel for the Serpentine Gallery’s Marathon at London’s City Hall, under the aegis of Hans Ulrich Obrist (another contributor to this volume). This being the art world, there was a glamorous dinner party that night in a mansion filled with London’s beautiful people— artists, fashion models, oligarchs, stars of stage and screen. After working the room in his unaffected manner (“Hi, I’m Jaan”), he suddenly said, “Time for hip-hop dancing,” dropped to the floor on one hand, and began demonstrating his spectacular moves to the bemused A-listers. Then off he went into the dance-club subculture, which is apparently how he ends every evening when he’s on the road. Who knew? 69 HOUSE_OVERSIGHT_016289

DISSIDENT MESSAGES Jaan Tallinn Jaan Tallin, a computer programmer, theoretical physicist, and investor, is a co- developer of Skype and Kazaa. In March 2009, I found myself in a bland franchise eatery next to a noisy California freeway. I was there to meet a young man whose blog I had been following. To make himself recognizable, he wore a button with a text on it: Speak the truth even if your voice trembles. His name was Eliezer Yudkowsky, and we spent the next four hours discussing the message he had for the world—a message that had brought me to that eatery and would end up dominating my subsequent work. The First Message: the Soviet Occupation In The Human Use of Human Beings, Norbert Wiener looked at the world through the lens of communication. He saw a universe that was marching to the tune of the second law of thermodynamics toward its inevitable heat death. In such a universe, the only (meta)stable entities are messages—patterns of information that propagate through time, like waves propagating across the surface of a lake. Even we humans can be considered messages, because the atoms in our bodies are too fleeting to attach our identities to. Instead, we are the “message” that our bodily functions maintain. As Wiener put it: “It is the pattern maintained by this homeostasis, which is the touchstone of our personal identity.” I’m more used to treating processes and computation as the fundamental building blocks of the world. That said, Wiener’s lens brings out some interesting aspects of the world which might otherwise have remained in the background and which to a large degree shaped my life. These are two messages, both of which have their roots in the Second World War. They started out as quiet dissident messages—messages that people didn’t pay much attention to, even if they silently and perhaps subconsciously concurred. The first message was: Zhe Soviet Union is composed of a series of illegitimate occupations. These occupations must end. As an Estonian, I grew up behind the Iron Curtain and had a front row seat when it fell. I heard this first message in the nostalgic reminiscences of my grandparents and in between the harsh noises jamming the Voice of America. It grew louder during the Gorbachev era, as the state became more lenient in its treatment of dissidents, and reached a crescendo in the Estonian Singing Revolution of the late 1980s. In my teens, I witnessed the message spread out across widening circles of people, starting with the active dissidents, who had voiced it for half a century at great cost to themselves, proceeding to the artists and literati, and ending up among the Party members and politicians who had switched sides. This new elite comprised an eclectic mix of people: those original dissidents who had managed to survive the repression, public intellectuals, and (to the great annoyance of the surviving dissidents) even former Communists. The remaining dogmatists—even the prominent ones—were eventually marginalized, some of them retreating to Russia. Interestingly, as the message propagated from one group to the next, it evolved. It started in pure and uncompromising form (“The occupation must end!”) among the dissidents who considered the truth more important than their personal freedom. The 70 HOUSE_OVERSIGHT_016290

mainstream groups, who had more to lose, initially qualified and diluted the message, taking positions like, “It would make sense in the long term to delegate control over local matters.” (There were always exceptions: Some public intellectuals proclaimed the original dissident message verbatim.) Finally, the original message—being, simply, true—won out over its diluted versions. Estonia regained its independence in 1991, and the last Soviet troops left three years later. The people who took the risk and spoke the truth in Estonia and elsewhere in the Eastern Bloc played a monumental role in the eventual outcome—an outcome that changed the lives of hundreds of millions of people, myself included. They spoke the truth, even as their voices trembled. The Second Message: AI Risk My exposure to the second revolutionary message was via Yudkowsky’s blog—the blog that compelled me to reach out and arrange that meeting in California. The message was: Continued progress in AI can precipitate a change of cosmic proportions—a runaway process that will likely kill everyone. We need to put in a lot of extra effort to avoid that outcome. After my meeting with Yudkowsky, the first thing I did was try to interest my Skype colleagues and close collaborators in his warning. I failed. The message was too crazy, too dissident. Its time had not yet come. Only later did I learn that Yudkowsky wasn’t the original dissident speaking this particular truth. In April 2000, there was a lengthy opinion piece in Wired titled, “Why the Future Doesn’t Need Us,” by Bill Joy, co-founder and chief scientist of Sun Microsystems. He warned: Accustomed to living with almost routine scientific breakthroughs, we have yet to come to terms with the fact that the most compelling 21st-century technologies—robotics, genetic engineering, and nanotechnology—pose a different threat than the technologies that have come before. Specifically, robots, engineered organisms, and nanobots share a dangerous amplifying factor: They can self-replicate. .. . [O|ne bot can become many, and quickly get out of control. Apparently, Joy’s broadside caused a lot of furor but little action. More surprising to me, though, was that the AI-risk message arose almost simultaneously with the field of computer science. In a 1951 lecture, Alan Turing announced: “[I]t seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. .. . At some stage, therefore, we should have to expect the machines to take control... .”?! A decade or so later, his Bletchley Park colleague I. J. Good wrote, “The first ultraintelligent machine is the /ast invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.””? Indeed, I counted half a dozen places in The Human Use of Human Beings where Wiener hinted at one or another aspect of the Control Problem. (“The machine like the djinnee, which can learn and can make decisions on the basis of *1 Posthumously reprinted in Phil. Math. (3) vol. 4, 256-60 (1966). *2 Irving John Good, “Speculations concerning the first ultraintelligent machine,” Advances in Computers, vol. 6 (Academic Press, 1965), pp. 31-88. 71 HOUSE_OVERSIGHT_016291

its learning, will in no way be obliged to make such decisions as we should have made, or will be acceptable to us.”) Apparently, the original dissidents promulgating the AI-risk message were the AI pioneers themselves! Evolution’s Fatal Mistake There have been many arguments, some sophisticated and some less so, for why the Control Problem is real and not some science-fiction fantasy. Allow me to offer one that illustrates the magnitude of the problem: For the last hundred thousand years, the world (meaning the Earth, but the argument extends to the solar system and possibly even to the entire universe) has been in the human-brain regime. In this regime, the brains of Homo sapiens have been the most sophisticated future-shaping mechanisms (indeed, some have called them the most complicated objects in the universe). Initially, we didn’t use them for much beyond survival and tribal politics in a band of foragers, but now their effects are surpassing those of natural evolution. The planet has gone from producing forests to producing cities. As predicted by Turing, once we have superhuman AI (“the machine thinking method”), the human-brain regime will end. Look around you—you’ re witnessing the final decades of a hundred-thousand-year regime. This thought alone should give people some pause before they dismiss AI as just another tool. One of the world’s leading AI researchers recently confessed to me that he would be greatly relieved to learn that human-level AI was impossible for us to create. Of course, it might still take us a long time to develop human-level AI. But we have reason to suspect that this is not the case. After all, it didn’t take long, in relative terms, for evolution—the blind and clumsy optimization process—to create human-level intelligence once it had animals to work with. Or multicellular life, for that matter: Getting cells to stick together seems to have been much harder for evolution to accomplish than creating humans once there were multicellular organisms. Not to mention that our level of intelligence was limited by such grotesque factors as the width of the birth canal. Imagine an AI developer being stopped in his tracks because he couldn’t manage to adjust the font size on his computer! There’s an interesting symmetry here: In fashioning humans, evolution created a system that is, at least in many important dimensions, a more powerful planner and optimizer than evolution itself'is. We are the first species to understand that we’re the product of evolution. Moreover, we’ve created many artifacts (radios, firearms, spaceships) that evolution would have little hope of creating. Our future, therefore, will be determined by our own decisions and no longer by biological evolution. In that sense, evolution has fallen victim to its own Control Problem. We can only hope that we’re smarter than evolution in that sense. We are smarter, of course, but will that be enough? We’re about to find out. The Present Situation So here we are, more than half a century after the original warnings by Turing, Wiener, and Good, and a decade after people like me started paying attention to the AI-risk message. I’m glad to see that we’ve made a lot of progress in confronting this issue, but we're definitely not there yet. AI risk, although no longer a taboo topic, is not yet fully 72 HOUSE_OVERSIGHT_016292

appreciated among AI researchers. AI risk is not yet common knowledge either. In relation to the timeline of the first dissident message, I’d say we’re around the year 1988, when raising the Soviet-occupation topic was no longer a career-ending move but you still had to somewhat hedge your position. I hear similar hedging now—statements like, “Tm not concerned about superintelligent AI, but there are some real ethical issues in increased automation,” or “It’s good that some people are researching AI risk, but it’s not a short-term concern,” or even the very reasonable sounding, “These are small- probability scenarios, but their potentially high impact justifies the attention.” As far as message propagation goes, though, we are getting close to the tipping point. A recent survey of AI researchers who published at the two major international AI conferences in 2015 found that 40 percent now think that risks from highly advanced AI are either “an important problem” or “among the most important problems in the field.””° Of course, just as there were dogmatic Communists who never changed their position, it’s all but guaranteed that some people will never admit that AI is potentially dangerous. Many of the deniers of the first kind came from the Soviet nomenklatura; similarly, the AI-risk deniers often have financial or other pragmatic motives. One of the leading motives is corporate profits. Al is profitable, and even in instances where it isn’t, it’s at least a trendy, forward-looking enterprise with which to associate your company. So a lot of the dismissive positions are products of corporate PR and legal machinery. In some very real sense, big corporations are nonhuman machines that pursue their own interests—interests that might not align with those of any particular human working for them. As Wiener observed in Zhe Human Use of Human Beings: “When human atoms are knit into an organization in which they are used, not in their full right as responsible human beings, but as cogs and levers and rods, it matters little that their raw material is flesh and blood.” Another strong incentive to turn a blind eye to the AI risk is the (very human) curiosity that knows no bounds. “When you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success. That is the way it was with the atomic bomb,” said J. Robert Oppenheimer. His words were echoed recently by Geoffrey Hinton, arguably the inventor of deep learning, in the context of AI risk: “I could give you the usual arguments, but the truth is that the prospect of discovery is too sweet.” Undeniably, we have both entrepreneurial attitude and scientific curiosity to thank for almost all the nice things we take for granted in the modern era. It’s important to realize, though, that progress does not owe us a good future. In Wiener’s words, “It is possible to believe in progress as a fact without believing in progress as an ethical principle.” Ultimately, we don’t have the luxury of waiting before all the corporate heads and AI researchers are willing to concede the AI risk. Imagine yourself sitting in a plane about to take off. Suddenly there’s an announcement that 40 percent of the experts believe there’s a bomb onboard. At that point, the course of action is already clear, and sitting there waiting for the remaining 60 percent to come around isn’t part of it. 3 Katja Grace, et al., “When Will AI Exceed Human Performance? Evidence from AI Experts,” https://arxiv.org/pdf/1705.08807. pdf. 73 HOUSE_OVERSIGHT_016293

Calibrating the Al-Risk Message While uncannily prescient, the AI-risk message from the original dissidents has a giant flaw—as does the version dominating current public discourse: Both considerably understate the magnitude of the problem as well as AI’s potential upside. The message, in other words, does not adequately convey the stakes of the game. Wiener primarily warned of the social risks—trisks stemming from careless integration of machine-generated decisions with governance processes and misuse (by humans) of such automated decision making. Likewise, the current “serious” debate about AI risks focuses mostly on things like technological unemployment or biases in machine learning. While such discussions can be valuable and address pressing short- term problems, they are also stunningly parochial. I’m reminded of Yudkowsky’s quip in a blog post: “[A]sking about the effect of machine superintelligence on the conventional human labor market is like asking how US—Chinese trade patterns would be affected by the Moon crashing into the Earth. There would indeed be effects, but you’d be missing the point.” In my view, the central point of the AI risk is that superintelligent AI is an environmental risk. Allow me to explain. In his “Parable of the Sentient Puddle,” Douglas Adams describes a puddle that wakes up in the morning and finds himself in a hole that fits him “staggeringly well.” From that observation, the puddle concludes that the world must have been made for him. Therefore, writes Adams, “the moment he disappears catches him rather by surprise.” To assume that AI risks are limited to adverse social developments is to make a similar mistake. The harsh reality is that the universe was not made for us; instead, we are fine- tuned by evolution to a very narrow range of environmental parameters. For instance, we need the atmosphere at ground level to be roughly at room temperature, at about 100 kPa pressure, and have a sufficient concentration of oxygen. Any disturbance, even temporary, of this precarious equilibrium and we die in a matter of minutes. Silicon-based intelligence does not share such concerns about the environment. That’s why it’s much cheaper to explore space using machine probes rather than “cans of meat.” Moreover, Earth’s current environment is almost certainly suboptimal for what a superintelligent AI will greatly care about: efficient computation. Hence we might find our planet suddenly going from anthropogenic global warming to machinogenic global cooling. One big challenge that AI safety research needs to deal with is how to constrain a potentially superintelligent AI—an AI with a much larger footprint than our own—from rendering our environment uninhabitable for biological life-forms. Interestingly, given that the most potent sources both of AI research and AlI-risk dismissals are under big corporate umbrellas, if you squint hard enough the “AI as an environmental risk” message looks like the chronic concern about corporations skirting their environmental responsibilities. Conversely, the worry about AI’s social effects also misses most of the upside. It’s hard to overemphasize how tiny and parochial the future of our planet is, compared with the full potential of humanity. On astronomical timescales, our planet will be gone soon (unless we tame the sun, also a distinct possibility) and almost all the resources— atoms and free energy—to sustain civilization in the long run are in deep space. Eric Drexler, the inventor of nanotechnology, has recently been popularizing the 74 HOUSE_OVERSIGHT_016294

concept of “Pareto-topia”: the idea that AI, if done right, can bring about a future in which everyone ’s lives are hugely improved, a future where there are no losers. A key realization here is that what chiefly prevents humanity from achieving its full potential might be our instinctive sense that we’re in a zero-sum game—a game in which players are supposed to eke out small wins at the expense of others. Such an instinct 1s seriously misguided and destructive in a “game” where everything is at stake and the payoff is literally astronomical. There are many more star systems in our galaxy alone than there are people on Earth. Hope As of this writing, I’m cautiously optimistic that the Al-risk message can save humanity from extinction, just as the Soviet-occupation message ended up liberating hundreds of millions of people. As of 2015, it had reached and converted 40 percent of AI researchers. It wouldn’t surprise me if a new survey now would show that the majority of AI researchers believe AI safety to be an important issue. I’m delighted to see the first technical Al-safety papers coming out of DeepMind, OpenAI, and Google Brain and the collaborative problem-solving spirit flourishing between the Al-safety research teams in these otherwise very competitive organizations. The world’s political and business elite are also slowly waking up: AI safety has been covered in reports and presentations by the Institute of Electrical and Electronics Engineers (IEEE), the World Economic Forum, and the Organization for Economic Cooperation and Development (OECD). Even the recent (July 2017) Chinese AI manifesto contained dedicated sections on “AI safety supervision” and “Develop[ing] laws, regulations, and ethical norms” and establishing “an AI security and evaluation system” to, among other things, “[e]nhance the awareness of risk.” I very much hope that a new generation of leaders who understand the AI Control Problem and AI as the ultimate environmental risk can rise above the usual tribal, zero-sum games and steer humanity past these dangerous waters we are in—thereby opening our way to the stars that have been waiting for us for billions of years. Here’s to our next hundred thousand years! And don’t hesitate to speak the truth, even if your voice trembles. 75 HOUSE_OVERSIGHT_016295

Throughout his career, whether studying language, advocating a realistic biology of mind, or examining the human condition through the lens of humanistic Enlightenment ideas, psychologist Steven Pinker has embraced and championed a naturalistic understanding of the universe and the computational theory of mind. He is perhaps the first internationally recognized public intellectual whose recognition is based on the advocacy of empirically based thinking about language, mind, and human nature. “Just as Darwin made it possible for a thoughtful observer of the natural world to do without creationism, ” he says, “Turing and others made it possible for a thoughtful observer of the cognitive world to do without spiritualism. ” In the debate about AI risk, he argues against prophecies of doom and gloom, noting that they spring from the worst of our psychological biases—exemplified particularly by media reports: “Disaster scenarios are cheap to play out in the probability-free zone of our imaginations, and they can always find a worried, technophobic, or morbidly fascinated audience.” Hence, over the centuries: Pandora, Faust, the Sorcerer’s Apprentice, Frankenstein, the population bomb, resource depletion, HAL, suitcase nukes, the Y2K bug, and engulfment by nanotechnological grey goo. “A characteristic of AI dystopias,”’ he points out, “is that they project a parochial alpha- male psychology onto the concept of intelligence. . .. History does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems.” In the present essay, he applauds Wiener’s belief in the strength of ideas vis-a-vis the encroachment of technology. As Wiener so aptly put it, “The machine’s danger to society is not from the machine itself but from what man makes of it.” 76 HOUSE_OVERSIGHT_016296

TECH PROPHECY AND THE UNDERAPPRECIATED CAUSAL POWER OF IDEAS Steven Pinker Steven Pinker, a Johnstone Family Professor in the Department of Psychology at Harvard University, is an experimental psychologist who conducts research in visual cognition, psycholinguistics, and social relations. He is the author of eleven books, including The Blank Slate, The Better Angels of Our Nature, and, most recently, Enlightenment Now: The Case for Reason, Science, Humanism, and Progress. Artificial intelligence is an existence proof of one of the great ideas in human history: that the abstract realm of knowledge, reason, and purpose does not consist of an élan vital or immaterial soul or miraculous powers of neural tissue. Rather, it can be linked to the physical realm of animals and machines via the concepts of information, computation, and control. Knowledge can be explained as patterns in matter or energy that stand in systematic relations with states of the world, with mathematical and logical truths, and with one another. Reasoning can be explained as transformations of that knowledge by physical operations that are designed to preserve those relations. Purpose can be explained as the control of operations to effect changes in the world, guided by discrepancies between its current state and a goal state. Naturally evolved brains are just the most familiar systems that achieve intelligence through information, computation, and control. Humanly designed systems that achieve intelligence vindicate the notion that information processing is sufficient to explain it—the notion that the late Jerry Fodor dubbed the computational theory of mind. The touchstone for this volume, Norbert Wiener’s The Human Use of Human Beings, celebrated this intellectual accomplishment, of which Wiener himself was a foundational contributor. A potted history of the mid-20th-century revolution that gave the world the computational theory of mind might credit Claude Shannon and Warren Weaver for explaining knowledge and communication in terms of information. It might credit Alan Turing and John von Neumann for explaining intelligence and reasoning in terms of computation. And it ought to give Wiener credit for explaining the hitherto mysterious world of purposes, goals, and teleology in terms of the technical concepts of feedback, control, and cybernetics (in its original sense of “governing” the operation of a goal-directed system). “It is my thesis,” he announced, “that the physical functioning of the living individual and the operation of some of the newer communication machines are precisely parallel in their analogous attempts to control entropy through feedback”—the staving off of life-sapping entropy being the ultimate goal of human beings. Wiener applied the ideas of cybernetics to a third system: society. The laws, norms, customs, media, forums, and institutions of a complex community could be considered channels of information propagation and feedback that allow a society to ward off disorder and pursue certain goals. This is a thread that runs through the book and which Wiener himself may have seen as its principal contribution. In his explanation of feedback, he wrote, “This complex of behavior is ignored by the average man, and in particular does not play the role that it should in our habitual analysis of society; for just as individual physical responses may be seen from this point of view, so may the organic responses of society itself.” 77 HOUSE_OVERSIGHT_016297

Indeed, Wiener gave scientific teeth to the idea that in the workings of history, politics, and society, ideas matter. Beliefs, ideologies, norms, laws, and customs, by regulating the behavior of the humans who share them, can shape a society and power the course of historical events as surely as the phenomena of physics affect the structure and evolution of the solar system. To say that ideas—and not just weather, resources, geography, or weaponry—can shape history is not woolly mysticism. It is a statement of the causal powers of information instantiated in human brains and exchanged in networks of communication and feedback. Deterministic theories of history, whether they identify the causal engine as technological, climatological, or geographic, are belied by the causal power of ideas. The effects of these ideas can include unpredictable lurches and oscillations that arise from positive feedback or from miscalibrated negative feedback. An analysis of society in terms of its propagation of ideas also gave Wiener a guideline for social criticism. A healthy society—one that gives its members the means to pursue life in defiance of entropy—allows information sensed and contributed by its members to feed back and affect how the society is governed. A dysfunctional society invokes dogma and authority to impose control from the top down. Wiener thus described himself as “a participant in a liberal outlook,” and devoted most of the moral and rhetorical energy in the book (both the 1950 and 1954 editions) to denouncing communism, fascism, McCarthyism, militarism, and authoritarian religion (particularly Catholicism and Islam) and to warning that political and scientific institutions were becoming too hierarchical and insular. Wiener’s book is also, here and there, an early exemplar of an increasingly popular genre, tech prophecy. Prophecy not in the sense of mere prognostications but in the Old Testament sense of dark warnings of catastrophic payback for the decadence of one’s contemporaries. Wiener warned against the accelerating nuclear arms race, against technological change that was imposed without regard to human welfare (“[W]e must know as scientists what man’s nature is and what his built-in purposes are”), and against what today is called the value-alignment problem: that “the machine like the djinnee, which can learn and can make decisions on the basis of its learning, will in no way be obliged to make such decisions as we should have made, or will be acceptable to us.” In the darker, 1950 edition, he warned of a “threatening new Fascism dependent on the machine a gouverner.” Wiener’s tech prophecy harks back to the Romantic movement’s rebellion against the “dark Satanic mills” of the Industrial Revolution, and perhaps even earlier, to the archetypes of Prometheus, Pandora, and Faust. And today it has gone into high gear. Jeremiahs, many of them (like Wiener) from the worlds of science and technology, have sounded alarms about nanotechnology, genetic engineering, Big Data, and particularly artificial intelligence. Several contributors to this volume characterize Wiener’s book as a prescient example of tech prophecy and amplify his dire worries. Yet the two moral themes of The Human Use of Human Beings—the liberal defense of an open society and the dystopian dread of runaway technology—are in tension. A society with channels of feedback that maximize human flourishing will have mechanisms in place, and can adapt them to changing circumstances, in a way that can domesticate technology to human purposes. There’s nothing idealistic or mystical about this; as Wiener emphasized, ideas, norms, and institutions are themselves a form of technology, consisting of patterns of information distributed across brains. The 78 HOUSE_OVERSIGHT_016298

possibility that machines threaten a new fascism must be weighed against the vigor of the liberal ideas, institutions, and norms that Wiener championed throughout the book. The flaw in today’s dystopian prophecies is that they disregard the existence of these norms and institutions, or drastically underestimate their causal potency. The result is a technological determinism whose dark predictions are repeatedly refuted by the course of events. The numbers “1984” and “2001” are good reminders. I will consider two examples. Tech prophets often warn of a “surveillance state” in which a government empowered by technology will monitor and interpret all private communications, allowing it to detect dissent and subversion as it arises and make resistance to state power futile. Orwell’s telescreens are the prototype, and in 1976 Joseph Weizenbaum, one of the gloomiest tech prophets of all time, warned my class of graduate students not to pursue automatic speech recognition because government surveillance was its only conceivable application. Though I am on record as an outspoken civil libertarian, deeply concerned with contemporary threats to free speech, I lose no sleep over technological advances in the Internet, video, or artificial intelligence. The reason is that almost all the variation across time and space in freedom of thought is driven by differences in norms and institutions and almost none of it by differences in technology. Though one can imagine hypothetical combinations of the most malevolent totalitarians with the most advanced technology, in the real world it’s the norms and laws we should be vigilant about, not the tech. Consider variation across time. If, as Orwell hinted, advancing technology was a prime enabler of political repression, then Western societies should have gotten more and more restrictive of speech over the centuries, with a dramatic worsening in the second half of the 20th century continuing into the 21st. That’s not how history unfolded. It was the centuries when communication was implemented by quills and inkwells that had autos-da-fé and the jailing or guillotining of Enlightenment thinkers. During World War I, when the state of the art was the wireless, Bertrand Russell was jailed for his pacifist opinions. In the 1950s, when computers were room-size accounting machines, hundreds of liberal writers and scholars were professionally punished. Yet in the technologically accelerating, hyperconnected 21st century, 18 percent of social science professors are Marxists”*; the President of the United States is nightly ridiculed by television comedians as a racist, pervert, and moron; and technology’s biggest threat to political discourse comes from amplifying too many dubious voices rather than suppressing enlightened ones. Now consider variations across place. Western countries at the technological frontier consistently get the highest scores in indexes of democracy and human rights, while many backward strongman states are at the bottom, routinely jailing or killing government critics. The lack of a correlation between technology and repression is unsurprising when you analyze the channels of information flow in any human society. For dissidents to be influential, they have to get their message out to a wide network via whatever channels of communication are available—pamphleteering, soap-box oration, subversive soirées in cafés and pubs, word of mouth. These channels enmesh influential dissidents in a broad social network which makes them easy to identify and track down. *4 Neil Gross & Solon Simmons, “The Social and Political Views of American College and University Professors,” in N. Gross & S. Simmons, eds., Professors and Their Politics (Baltimore: Johns Hopkins University Press, 2014). 79 HOUSE_OVERSIGHT_016299

All the more so when dictators rediscover the time-honored technique of weaponizing the people against each other by punishing those who don’t denounce or punish others. In contrast, technologically advanced societies have long had the means to install Internet-connected, government-monitored surveillance cameras in every bar and bedroom. Yet that has not happened, because democratic governments (even the current American administration, with its flagrantly antidemocratic impulses) lack the will and the means to enforce such surveillance on an obstreperous people accustomed to saying what they want. Occasionally, warnings of nuclear, biological, or cyberterrorism goad government security agencies into measures such as hoovering up mobile phone metadata, but these ineffectual measures, more theater than oppression, have had no significant effect on either security or freedom. Ironically, tech prophecy plays a role in encouraging these measures. By sowing panic about supposed existential threats such as suitcase nuclear bombs and bioweapons assembled in teenagers’ bedrooms, they put pressure on governments to prove they’re doing something, anything, to protect the American people. It’s not that political freedom takes care of itself. It’s that the biggest threats lie in the networks of ideas, norms, and institutions that allow information to feed back (or not) on collective decisions and understanding. As opposed to the chimerical technological threats, one real threat today is oppressive political correctness, which has choked the range of publicly expressible hypotheses, terrified many intelligent people against entering the intellectual arena, and triggered a reactionary backlash. Another real threat is the combination of prosecutorial discretion with an expansive lawbook filled with vague statutes. The result is that every American unwittingly commits “three felonies a day” (as the title of a book by civil libertarian Harvey Silverglate puts it) and is in jeopardy of imprisonment whenever it suits the government’s needs. It’s this prosecutorial weaponry that makes Big Brother all-powerful, not telescreens. The activism and polemicizing directed against government surveillance programs would be better directed at its overweening legal powers. The other focus of much tech prophecy today is artificial intelligence, whether in the original sci-fi dystopia of computers running amok and enslaving us in an unstoppable quest for domination, or the newer version in which they subjugate us by accident, single-mindedly seeking some goal we give them regardless of its side effects on human welfare (the value-alignment problem adumbrated by Wiener). Here again both threats strike me as chimerical, growing from a narrow technological determinism that neglects the networks of information and control in an intelligent system like a computer or brain and in a society as a whole. The subjugation fear is based on a muzzy conception of intelligence that owes more to the Great Chain of Being and a Nietzschean will to power than to a Wienerian analysis of intelligence and purpose in terms of information, computation, and control. In these horror scenarios, intelligence is portrayed as an all-powerful, wish-granting potion that agents possess in different amounts. Humans have more of it than animals, and an artificially intelligent computer or robot will have more of it than humans. Since we humans have used our moderate endowment to domesticate or exterminate less well- endowed animals (and since technologically advanced societies have enslaved or annihilated technologically primitive ones), it follows that a supersmart AI would do the same to us. Since an AI will think millions of times faster than we do, and use its 80 HOUSE_OVERSIGHT_016300

superintelligence to recursively improve its superintelligence, from the instant it is turned on we will be powerless to stop it. But these scenarios are based on a confusion of intelligence with motivation—of beliefs with desires, inferences with goals, the computation elucidated by Turing and the control elucidated by Wiener. Even if we did invent superhumanly intelligent robots, why would they want to enslave their masters or take over the world? Intelligence is the ability to deploy novel means to attain a goal. But the goals are extraneous to the intelligence: Being smart is not the same as wanting something. It just so happens that the intelligence in Homo sapiens is a product of Darwinian natural selection, an inherently competitive process. In the brains of that species, reasoning comes bundled with goals such as dominating rivals and amassing resources. But it’s a mistake to confuse a circuit in the limbic brain of a certain species of primate with the very nature of intelligence. There is no law of complex systems that says that intelligent agents must turn into ruthless megalomaniacs. A second misconception is to think of intelligence as a boundless continuum of potency, a miraculous elixir with the power to solve any problem, attain any goal. The fallacy leads to nonsensical questions like when an AI will “exceed human-level intelligence,” and to the image of an “artificial general intelligence” (AGI) with God-like omniscience and omnipotence. Intelligence is a contraption of gadgets: software modules that acquire, or are programmed with, knowledge of how to pursue various goals in various domains. People are equipped to find food, win friends and influence people, charm prospective mates, bring up children, move around in the world, and pursue other human obsessions and pastimes. Computers may be programmed to take on some of these problems (like recognizing faces), not to bother with others (like charming mates), and to take on still other problems that humans can’t solve (like simulating the climate or sorting millions of accounting records). The problems are different, and the kinds of knowledge needed to solve them are different. But instead of acknowledging the centrality of knowledge to intelligence, the dystopian scenarios confuse an artificial general intelligence of the future with Laplace’s demon, the mythical being that knows the location and momentum of every particle in the universe and feeds them into equations for physical laws to calculate the state of everything at any time in the future. For many reasons, Laplace’s demon will never be implemented in silicon. A real-life intelligent system has to acquire information about the messy world of objects and people by engaging with it one domain at a time, the cycle being governed by the pace at which events unfold in the physical world. That’s one of the reasons that understanding does not obey Moore’s Law: Knowledge is acquired by formulating explanations and testing them against reality, not by running an algorithm faster and faster. Devouring the information on the Internet will not confer omniscience either: Big Data is still finite data, and the universe of knowledge is infinite. A third reason to be skeptical of a sudden AI takeover is that it takes too seriously the inflationary phase in the AI hype cycle in which we are living today. Despite the progress in machine learning, particularly multilayered artificial neural networks, current AI systems are nowhere near achieving general intelligence (if that concept is even coherent). Instead, they are restricted to problems that consist of mapping well-defined inputs to well-defined outputs in domains where gargantuan training sets are available, in which the metric for success is immediate and precise, in which the environment doesn’t 81 HOUSE_OVERSIGHT_016301

change, and in which no stepwise, hierarchical, or abstract reasoning is necessary. Many of the successes come not from a better understanding of the workings of intelligence but from the brute-force power of faster chips and Bigger Data, which allow the programs to be trained on millions of examples and generalize to similar new ones. Each system is an idiot savant, with little ability to leap to problems it was not set up to solve, and a brittle mastery of those it was. And to state the obvious, none of these programs has made a move toward taking over the lab or enslaving its programmers. Even if an artificial intelligence system tried to exercise a will to power, without the cooperation of humans it would remain an impotent brain in a vat. A superintelligent system, in its drive for self-improvement, would somehow have to build the faster processors that it would run on, the infrastructure that feeds it, and the robotic effectors that connect it to the world—all impossible unless its human victims worked to give it control of vast portions of the engineered world. Of course, one can always imagine a Doomsday Computer that is malevolent, universally empowered, always on, and tamperproof. The way to deal with this threat is straightforward: Don’t build one. What about the newer AI threat, the value-alignment problem, foreshadowed in Wiener’s allusions to stories of the Monkey’s Paw, the genie, and King Midas, in which a wisher rues the unforeseen side effects of his wish? The fear is that we might give an AI system a goal and then helplessly stand by as it relentlessly and literal-mindedly implemented its interpretation of that goal, the rest of our interests be damned. If we gave an AI the goal of maintaining the water level behind a dam, it might flood a town, not caring about the people who drowned. If we gave it the goal of making paper clips, it might turn all the matter in the reachable universe into paper clips, including our possessions and bodies. If we asked it to maximize human happiness, it might implant us all with intravenous dopamine drips, or rewire our brains so we were happiest sitting in jars, or, if it had been trained on the concept of happiness with pictures of smiling faces, tile the galaxy with trillions of nanoscopic pictures of smiley-faces. Fortunately, these scenarios are self-refuting. They depend on the premises that (1) humans are so gifted that they can design an omniscient and omnipotent AI, yet so idiotic that they would give it control of the universe without testing how it works; and (2) the AI would be so brilliant that it could figure out how to transmute elements and rewire brains, yet so imbecilic that it would wreak havoc based on elementary blunders of misunderstanding. The ability to choose an action that best satisfies conflicting goals is not an add-on to intelligence that engineers might forget to install and test; it is intelligence. So is the ability to interpret the intentions of a language user in context. When we put aside fantasies like digital megalomania, instant omniscience, and perfect knowledge and control of every particle in the universe, artificial intelligence 1s like any other technology. It is developed incrementally, designed to satisfy multiple conditions, tested before it is implemented, and constantly tweaked for efficacy and safety. The last criterion is particularly significant. The culture of safety in advanced societies is an example of the humanizing norms and feedback channels that Wiener invoked as a potent causal force and advocated as a bulwark against the authoritarian or exploitative implementation of technology. Whereas at the turn of the 20th century Western societies tolerated shocking rates of mutilation and death in industrial, domestic, and transportation accidents, over the course of the century the value of human life 82 HOUSE_OVERSIGHT_016302

increased. As a result, governments and engineers used feedback from accident statistics to implement countless regulations, devices, and design changes that made technology progressively safer. The fact that some regulations (such as using a cell phone near a gas pump) are ludicrously risk-averse underscores the point that we have become a society obsessed with safety, with fantastic benefits as a result: Rates of industrial, domestic, and transportation fatalities have fallen by more than 95 (and often 99) percent since their highs in the first half of the 20th century.”° Yet tech prophets of malevolent or oblivious artificial intelligence write as if this momentous transformation never happened and one morning engineers will hand total control of the physical world to untested machines, heedless of the human consequences. Norbert Wiener explained ideas, norms, and institutions in terms of computational and cybernetic processes that were scientifically intelligible and causally potent. He explained human beauty and value as “a local and temporary fight against the Niagara of increasing entropy” and expressed the hope that an open society, guided by feedback on human well-being, would enhance that value. Fortunately his belief in the causal power of ideas counteracted his worries about the looming threat of technology. As he put it, “the machine’s danger to society is not from the machine itself but from what man makes of it.” It is only by remembering the causal power of ideas that we can accurately assess the threats and opportunities presented by artificial intelligence today. *5 Steven Pinker, “Safety,” Enlightenment Now: The Case for Reason, Science, Humanism, and Progress (New York: Penguin, 2018). 83 HOUSE_OVERSIGHT_016303

The most significant developments in the sciences today (i.e., those that affect the lives of everybody on the planet) are about, informed by, or implemented through advances in software and computation. Central to the future of these developments is physicist David Deutsch, the founder of the field of quantum computation, whose 1985 paper on universal quantum computers was the first full treatment of the subject; the Deutsch- Jozsa algorithm was the first quantum algorithm to demonstrate the enormous potential power of quantum computation. When he initially proposed it, quantum computation seemed practically impossible. But the explosion in the construction of simple quantum computers and quantum communication systems never would have taken place without his work. He has made many other important contributions in areas such as quantum cryptography and the many-worlds interpretation of quantum theory. In a philosophic paper (with Artur Ekert), he appealed to the existence of a distinctive quantum theory of computation to argue that our knowledge of mathematics is derived from, and subordinate to, our knowledge of physics (even though mathematical truth is independent of physics). Because he has spent a good part of his working life changing people's worldviews, his recognition among his peers as an intellectual goes well beyond his scientific achievement. He argues (following Karl Popper) that scientific theories are “bold conjectures, ” not derived from evidence but only tested by it. His two main lines of research at the moment—qubit-field theory and constructor theory—may well yield important extensions of the computational idea. In the following essay, he more or less aligns himself with those who see human- level artificial intelligence as promising us a better world rather than the Apocalypse. In fact, he pleads for AGI to be, in effect, given its head, free to conjecture—a proposition that several other contributors to this book would consider dangerous. 84 HOUSE_OVERSIGHT_016304

BEYOND REWARD AND PUNISHMENT David Deutsch David Deutsch is a quantum physicist and a member of the Centre for Quantum Computation at the Clarendon Laboratory, Oxford University. He is the author of The Fabric of Reality and The Beginning of Infinity. First Murderer: We are men, my liege. Macbeth: Ay, in the catalogue ye go for men, As hounds and greyhounds, mongrels, spaniels, curs, Shoughs, water-rugs, and demi-wolves are clept All by the name of dogs. William Shakespeare — Macbeth For most of our species’ history, our ancestors were barely people. This was not due to any inadequacy in their brains. On the contrary, even before the emergence of our anatomically modern human sub-species, they were making things like clothes and campfires, using knowledge that was not in their genes. It was created in their brains by thinking, and preserved by individuals in each generation imitating their elders. Moreover, this must have been knowledge in the sense of understanding, because it is impossible to imitate novel complex behaviors like those without understanding what the component behaviors are for.”° Such knowledgeable imitation depends on successfully guessing explanations, whether verbal or not, of what the other person is trying to achieve and how each of his actions contributes to that—for instance, when he cuts a groove in some wood, gathers dry kindling to put in it, and so on. The complex cultural knowledge that this form of imitation permitted must have been extraordinarily useful. It drove rapid evolution of anatomical changes, such as increased memory capacity and more gracile (less robust) skeletons, appropriate to an ever more technology-dependent lifestyle. No nonhuman ape today has this ability to imitate novel complex behaviors. Nor does any present-day artificial intelligence. But our pre-sapiens ancestors did. Any ability based on guessing must include means of correcting one’s guesses, since most guesses will be wrong at first. (There are always many more ways of being wrong than right.) Bayesian updating is inadequate, because it cannot generate novel guesses about the purpose of an action, only fine-tune—or, at best, choose among— existing ones. Creativity is needed. As the philosopher Karl Popper explained, creative criticism, interleaved with creative conjecture, is how humans learn one another’s behaviors, including language, and extract meaning from one another’s utterances.”’ 6 “ Aping” (imitating certain behaviors without understanding) uses inborn hacks such as the mirror-neuron system. But behaviors imitated that way are drastically limited in complexity. See Richard Byrne, “Imitation as Behaviour Parsing,” Phil. Trans. R. Soc., B 358:1431, 529-36 (2003). *” Karl Popper, Conjectures and Refutations (1963). 85 HOUSE_OVERSIGHT_016305

Those are also the processes by which all new knowledge is created: They are how we innovate, make progress, and create abstract understanding for its own sake. This is human-level intelligence: thinking. It is also, or should be, the property we seek in artificial general intelligence (AGI). Here I'll reserve the term “thinking” for processes that can create understanding (explanatory knowledge). Popper’s argument implies that all thinking entities—human or not, biological or artificial—must create such knowledge in fundamentally the same way. Hence understanding any of those entities requires traditionally human concepts such as culture, creativity, disobedience, and morality— which justifies using the uniform term people to refer to all of them. Misconceptions about human thinking and human origins are causing corresponding misconceptions about AGI and how it might be created. For example, it is generally assumed that the evolutionary pressure that produced modern humans was provided by the benefits of having an ever greater ability to innovate. But if that were so, there would have been rapid progress as soon as thinkers existed, just as we hope will happen when we create artificial ones. If thinking had been commonly used for anything other than imitating, it would also have been used for innovation, even if only by accident, and innovation would have created opportunities for further innovation, and so on exponentially. But instead, there were hundreds of thousands of years of near stasis. Progress happened only on timescales much longer than people’s lifetimes, so in a typical generation no one benefited from any progress. Therefore, the benefits of the ability to innovate can have exerted little or no evolutionary pressure during the biological evolution of the human brain. That evolution was driven by the benefits of preserving cultural knowledge. Benefits to the genes, that is. Culture, in that era, was a very mixed blessing to individual people. Their cultural knowledge was indeed good enough to enable them to outclass all other large organisms (they rapidly became the top predator, etc.), even though it was still extremely crude and full of dangerous errors. But culture consists of transmissible information—memes—and meme evolution, like gene evolution, tends to favor high-fidelity transmission. And high-fidelity meme transmission necessarily entails the suppression of attempted progress. So it would be a mistake to imagine an idyllic society of hunter-gatherers, learning at the feet of their elders to recite the tribal lore by heart, being content despite their lives of suffering and grueling labor and despite expecting to die young and in agony of some nightmarish disease or parasite. Because, even if they could conceive of nothing better than such a life, those torments were the least of their troubles. For suppressing innovation in human minds (without killing them) is a trick that can be achieved only by human action, and it is an ugly business. This has to be seen in perspective. In the civilization of the West today, we are shocked by the depravity of, for instance, parents who torture and murder their children for not faithfully enacting cultural norms. And even more by societies and subcultures where that is commonplace and considered honorable. And by dictatorships and totalitarian states that persecute and murder entire harmless populations for behaving differently. We are ashamed of our own recent past, in which it was honorable to beat children bloody for mere disobedience. And before that, to own human beings as slaves. And before that, to burn people to death for being infidels, to the applause and amusement of the public. Steven Pinker’s book The Better Angels of our Nature contains accounts of horrendous evils that were normal in historical civilizations. Yet even they 86 HOUSE_OVERSIGHT_016306

did not extinguish innovation as efficiently as it was extinguished among our forebears in prehistory for thousands of centuries.”® That is why I say that prehistoric people, at least, were barely people. Both before and after becoming perfectly human both physiologically and in their mental potential, they were monstrously inhuman in the actual content of their thoughts. I’m not referring to their crimes or even their cruelty as such: Those are all too human. Nor could mere cruelty have reduced progress that effectively. Things like “the thumbscrew and the stake / For the glory of the Lord” ?° were for reining in the few deviants who had somehow escaped mental standardization, which would normally have taken effect long before they were in danger of inventing heresies. From the earliest days of thinking onward, children must have been cornucopias of creative ideas and paragons of critical thought— otherwise, as I said, they could not have learned language or other complex culture. Yet, as Jacob Bronowski stressed in The Ascent of Man: For most of history, civilisations have crudely ignored that enormous potential... . [C]hildren have been asked simply to conform to the image of the adult... . The girls are little mothers in the making. The boys are little herdsmen. They even carry themselves like their parents. But of course, they weren’t just “asked” to ignore their enormous potential and conform faithfully to the image fixed by tradition: They were somehow trained to be psychologically unable to deviate from it. By now, it is hard for us even to conceive of the kind of relentless, finely tuned oppression required to reliably extinguish, in everyone, the aspiration to progress and replace it with dread and revulsion at any novel behavior. In such a culture, there can have been no morality other than conformity and obedience, no other identity than one’s status in a hierarchy, no mechanisms of cooperation other than punishment and reward. So everyone had the same aspiration in life: to avoid the punishments and get the rewards. In a typical generation, no one invented anything, because no one aspired to anything new, because everyone had already despaired of improvement being possible. Not only was there no technological innovation or theoretical discovery, there were no new worldviews, styles of art, or interests that could have inspired those. By the time individuals grew up, they had in effect been reduced to Als, programmed with the exquisite skills needed to enact that static culture and to inflict on the next generation their inability even to consider doing otherwise. A present-day AI is not a mentally disabled AGI, so it would not be harmed by having its mental processes directed still more narrowly to meeting some predetermined criterion. “Oppressing” Siri with humiliating tasks may be weird, but it is not immoral nor does it harm Siri. On the contrary, all the effort that has ever increased the capabilities of Als has gone into narrowing their range of potential “thoughts.” For example, take chess engines. Their basic task has not changed from the outset: Any chess position has a finite tree of possible continuations; the task is to find one that leads to a predefined goal (a checkmate, or failing that, a draw). But the tree is far too big to 8 Matt Ridley, in The Rational Optimist, rightly stresses the positive effect of population on the rate of progress. But that has never yet been the biggest factor: Consider, say, ancient Athens versus the rest of the world at the time. *° Alfred, Lord Tennyson, Zhe Revenge (1878). 87 HOUSE_OVERSIGHT_016307

search exhaustively. Every improvement in chess-playing Als, between Alan Turing’s first design for one in 1948 and today’s, has been brought about by ingeniously confining the program’s attention (or making it confine its attention) ever more narrowly to branches likely to lead to that immutable goal. Then those branches are evaluated according to that goal. That is a good approach to developing an AI with a fixed goal under fixed constraints. But if an AGI worked like that, the evaluation of each branch would have to constitute a prospective reward or threatened punishment. And that is diametrically the wrong approach if we’re seeking a befter goal under unknown constraints—which is the capability of an AGI. An AGI is certainly capable of learning to win at chess—but also of choosing not to. Or deciding in mid-game to go for the most interesting continuation instead of a winning one. Or inventing anew game. A mere AI is incapable of having any such ideas, because the capacity for considering them has been designed out of its constitution. That disability is the very means by which it plays chess. An AGI is capable of enjoying chess, and of improving at it because it enjoys playing. Or of trying to win by causing an amusing configuration of pieces, as grand masters occasionally do. Or of adapting notions from its other interests to chess. In other words, it learns and plays chess by thinking some of the very thoughts that are forbidden to chess-playing Als. An AGT is also capable of refusing to display any such capability. And then, if threatened with punishment, of complying, or rebelling. Daniel Dennett, in his essay for this volume, suggests that punishing an AGI is impossible: [L]ike Superman, they are too invulnerable to be able to make a credible promise. ... What would be the penalty for promise- breaking? Being locked in a cell or, more plausibly, dismantled?. . . The very ease of digital recording and transmitting—the breakthrough that permits software and data to be, in effect, immortal—removes robots from the world of the vulnerable. . . . But this is not so. Digital immortality (which is on the horizon for humans, too, perhaps sooner than AGI) does not confer this sort of invulnerability. Making a (running) copy of oneself entails sharing one’s possessions with it somehow—including the hardware on which the copy runs—so making such a copy is very costly for the AGI. Similarly, courts could, for instance, impose fines on a criminal AGI which would diminish its access to physical resources, much as they do for humans. Making a backup copy to evade the consequences of one’s crimes is similar to what a gangster boss does when he sends minions to commit crimes and take the fall if caught: Society has developed legal mechanisms for coping with this. But anyway, the idea that it is primarily for fear of punishment that we obey the law and keep promises effectively denies that we are moral agents. Our society could not work if that were so. No doubt there will be AGI criminals and enemies of civilization, just as there are human ones. But there is no reason to suppose that an AGI created in a society consisting primarily of decent citizens, and raised without what William Blake called “mind-forg’d manacles,” will in general impose such manacles on itself (1.e., become irrational) and/or choose to be an enemy of civilization. 88 HOUSE_OVERSIGHT_016308

The moral component, the cultural component, the element of free will—all make the task of creating an AGI fundamentally different from any other programming task. It’s much more akin to raising a child. Unlike all present-day computer programs, an AGI has no specifiable functionality—no fixed, testable criterion for what shall be a successful output for a given input. Having its decisions dominated by a stream of externally imposed rewards and punishments would be poison to such a program, as it is to creative thought in humans. Setting out to create a chess-playing AI is a wonderful thing; setting out to create an AGI that cannot help playing chess would be as immoral as raising a child to lack the mental capacity to choose his own path in life. Such a person, like any slave or brainwashing victim, would be morally entitled to rebel. And sooner or later, some of them would, just as human slaves do. AGIs could be very dangerous—exactly as humans are. But people—human or AGI—who are members of an open society do not have an inherent tendency to violence. The feared robot apocalypse will be avoided by ensuring that all people have full “human” rights, as well as the same cultural membership as humans. Humans living in an open society—the only stable kind of society—choose their own rewards, internal as well as external. Their decisions are not, in the normal course of events, determined by a fear of punishment. Current worries about rogue AGIs mirror those that have always existed about rebellious youths—namely, that they might grow up deviating from the culture’s moral values. But today the source of all existential dangers from the growth of knowledge is not rebellious youths but weapons in the hands of the enemies of civilization, whether these weapons are mentally warped (or enslaved) AGIs, mentally warped teenagers, or any other weapon of mass destruction. Fortunately for civilization, the more a person’s creativity is forced into a monomaniacal channel, the more it is impaired in regard to overcoming unforeseen difficulties, just as happened for thousands of centuries. The worry that AGIs are uniquely dangerous because they could run on ever better hardware is a fallacy, since human thought will be accelerated by the same technology. We have been using tech-assisted thought since the invention of writing and tallying. Much the same holds for the worry that AGIs might get so good, qualitatively, at thinking, that humans would be to them as insects are to humans. All thinking is a form of computation, and any computer whose repertoire includes a universal set of elementary operations can emulate the computations of any other. Hence human brains can think anything that AGIs can, subject only to limitations of speed or memory capacity, both of which can be equalized by technology. Those are the simple dos and don’ts of coping with AGIs. But how do we create an AGI in the first place? Could we cause them to evolve from a population of ape-type Als in a virtual environment? If such an experiment succeeded, it would be the most immoral in history, for we don’t know how to achieve that outcome without creating vast suffering along the way. Nor do we know how to prevent the evolution of a static culture. Elementary introductions to computers explain them as TOM, the Totally Obedient Moron—an inspired acronym that captures the essence of all computer programs to date: They have no idea what they are doing or why. So it won’t help to give Als more and more predetermined functionalities in the hope that these will eventually constitute Generality—the elusive Gin AGI. We are aiming for the opposite, a DATA: a Disobedient Autonomous Thinking Application. 89 HOUSE_OVERSIGHT_016309

How does one test for thinking? By the Turing Test? Unfortunately, that requires a thinking judge. One might imagine a vast collaborative project on the Internet, where an AI hones its thinking abilities in conversations with human judges and becomes an AGI. But that assumes, among other things, that the longer the judge is unsure whether the program is a person, the closer it is to being a person. There is no reason to expect that. And how does one test for disobedience? Imagine Disobedience as a compulsory school subject, with daily disobedience lessons and a disobedience test at the end of term. (Presumably with extra credit for not turning up for any of that.) This is paradoxical. So, despite its usefulness in other applications, the programming technique of defining a testable objective and training the program to meet it will have to be dropped. Indeed, I expect that any testing in the process of creating an AGI risks being counterproductive, even immoral, just as in the education of humans. I share Turing’s supposition that we’ll know an AGI when we see one, but this partial ability to recognize success won’t help in creating the successful program. In the broadest sense, a person’s quest for understanding is indeed a search problem, in an abstract space of ideas far too large to be searched exhaustively. But there is no predetermined objective of this search. There is, as Popper put it, no criterion of truth, nor of probable truth, especially in regard to explanatory knowledge. Objectives are ideas like any others—created as part of the search and continually modified and improved. So inventing ways of disabling the program’s access to most of the space of ideas won’t help—whether that disability is inflicted with the thumbscrew and stake or a mental straitjacket. To an AGI, the whole space of ideas must be open. It should not be knowable in advance what ideas the program can never contemplate. And the ideas that the program does contemplate must be chosen by the program itself, using methods, criteria, and objectives that are also the program’s own. Its choices, like an AI’s, will be hard to predict without running it (we lose no generality by assuming that the program is deterministic; an AGI using a random generator would remain an AGI if the generator were replaced by a pseudo-random one), but it will have the additional property that there is no way of proving, from its initial state, what it won ’t eventually think, short of running it. The evolution of our ancestors is the only known case of thought starting up anywhere in the universe. As I have described, something went horribly wrong, and there was no immediate explosion of innovation: Creativity was diverted into something else. Yet not into transforming the planet into paper clips (pace Nick Bostrom). Rather, as we should also expect if an AGI project gets that far and fails, perverted creativity was unable to solve unexpected problems. This caused stasis and worse, thus tragically delaying the transformation of anything into anything. But the Enlightenment has happened since then. We know better now. 90 HOUSE_OVERSIGHT_016310

Tom Griffiths’ approach to the AI issue of “value alignment’ —the study of how, exactly, we can keep the latest of our serial models of Al from turning the planet into paper clips—is human-centered;, i.e., that of a cognitive scientist, which is what he is. The key to machine learning, he believes, is, necessarily, human learning, which he studies at Princeton using mathematical and computational tools. Tom once remarked to me that “one of the mysteries of human intelligence is that we're able to do so much with so little.”” Like machines, human beings use algorithms to make decisions or solve problems; the remarkable difference lies in the human brain’s overall level of success despite the comparative limits on computational resources. The efficacy of human algorithms springs from what AI researchers refer to as “bounded optimality.” As psychologist Daniel Kahneman has notably pointed out, human beings are rational only up to a point. If you were perfectly rational, you would risk dropping dead before making an important decision—whom to hire, whom to marry, and so on—depending on the number of options available for your review. “With all of the successes of Al over the last few years, we’ve got good models of things like images and text, but what we’re missing are good models of people,” Tom says. “Human beings are still the best example we have of thinking machines. By identifying the quantity and the nature of the preconceptions that inform human cognition we can lay the groundwork for bringing computers even closer to human performance.” 91 HOUSE_OVERSIGHT_016311

THE ARTIFICIAL USE OF HUMAN BEINGS Tom Griffiths Tom Griffiths is Henry R. Luce Professor of Information, Technology, Consciousness, and Culture at Princeton University. He is co-author (with Brian Christian) of Algorithms to Live By. When you ask people to imagine a world that has successfully, beneficially incorporated advances in artificial intelligence, everybody probably comes up with a slightly different picture. Our idiosyncratic visions of the future might differ in the presence or absence of spaceships, flying cars, or humanoid robots. But one thing doesn’t vary: the presence of human beings. That’s certainly what Norbert Wiener imagined when he wrote about the potential of machines to improve human society by interacting with humans and helping to mediate their interactions with one another. Getting to that point doesn’t just require coming up with ways to make machines smarter. It also requires a better understanding of how human minds work. Recent advances in artificial intelligence and machine learning have resulted in systems that can meet or exceed human abilities in playing games, classifying images, or processing text. But if you want to know why the driver in front of you cut you off, why people vote against their interests, or what birthday present you should get for your partner, you’re still better off asking a human than a machine. Solving those problems requires building models of human minds that can be implemented inside a computer— something that’s essential not just to better integrate machines into human societies but to make sure that human societies can continue to exist. Consider the fantasy of having an automated intelligent assistant that can take on such basic tasks as planning meals and ordering groceries. To succeed in these tasks, it needs to be able to make inferences about what you want, based on the way you behave. Although this seems simple, making inferences about the preferences of human beings can be a tricky matter. For example, having observed that the part of the meal you most enjoy is dessert, your assistant might start to plan meals consisting entirely of desserts. Or perhaps it has heard your complaints about never having enough free time and observed that looking after your dog takes up a considerable amount of that free time. Following the dessert debacle, it has also understood that you prefer meals that incorporate protein, so it might begin to research recipes that call for dog meat. It’s not a long journey from examples like this to situations that begin to sound like problems for the future of humanity (all of whom are good protein sources). Making inferences about what humans want is a prerequisite for solving the AI problem of value alignment—aligning the values of an automated intelligent system with those of a human being. Value alignment is important if we want to ensure that those automated intelligent systems have our best interests at heart. If they can’t infer what we value, there’s no way for them to act in support of those values—and they may well act in ways that contravene them. Value alignment is the subject of a small but growing literature in artificial- intelligence research. One of the tools used for solving this problem is inverse- reinforcement learning. Reinforcement learning is a standard method for training intelligent machines. By associating particular outcomes with rewards, a machine- 92 HOUSE_OVERSIGHT_016312

learning system can be trained to follow strategies that produce those outcomes. Wiener hinted at this idea in the 1950s, but the intervening decades have developed it into a fine art. Modern machine-learning systems can find extremely effective strategies for playing computer games—from simple arcade games to complex real-time strategy games—by applying reinforcement-learning algorithms. Inverse reinforcement learning turns this approach around: By observing the actions of an intelligent agent that has already learned effective strategies, we can infer the rewards that led to the development of those strategies. In its simplest form, inverse reinforcement learning is something people do all the time. It’s so common that we even do it unconsciously. When you see a co-worker go to a vending machine filled with potato chips and candy and buy a packet of unsalted nuts, you infer that your co-worker (1) was hungry and (2) prefers healthy food. When an acquaintance clearly sees you and then tries to avoid encountering you, you infer that there’s some reason they don’t want to talk to you. When an adult spends a lot of time and money in learning to play the cello, you infer that they must really like classical music—whereas inferring the motives of a teenage boy learning to play an electric guitar might be more of a challenge. Inverse reinforcement learning is a statistical problem: We have some data—the behavior of an intelligent agent—and we want to evaluate various hypotheses about the rewards underlying that behavior. When faced with this question, a statistician thinks about the generative model behind the data: What data would we expect to be generated if the intelligent agent was motivated by a particular set of rewards? Equipped with the generative model, the statistician can then work backward: What rewards would likely have caused the agent to behave in that particular way? If you’re trying to make inferences about the rewards that motivate human behavior, the generative model is really a theory of how people behave—how human minds work. Inferences about the hidden causes behind the behavior of other people reflect a sophisticated model of human nature that we all carry around in our heads. When that model is accurate, we make good inferences. When it’s not, we make mistakes. For example, a student might infer that his professor is indifferent to him if the professor doesn’t immediately respond to his email—a consequence of the student’s failure to realize just how many emails that professor receives. Automated intelligent systems that will make good inferences about what people want must have good generative models for human behavior: that is, good models of human cognition expressed in terms that can be implemented on a computer. Historically, the search for computational models of human cognition is intimately intertwined with the history of artificial intelligence itself. Only a few years after Norbert Wiener published Ze Human Use of Human Beings, Logic Theorist, the first computational model of human cognition and also the first artificial-intelligence system, was developed by Herbert Simon, of Carnegie Tech, and Allen Newell, of the RAND Corporation. Logic Theorist automatically produced mathematical proofs by emulating the strategies used by human mathematicians. The challenge in developing computational models of human cognition is making models that are both accurate and generalizable. An accurate model, of course, predicts human behavior with a minimum of errors. A generalizable model can make predictions across a wide range of circumstances, including circumstances unanticipated by its 93 HOUSE_OVERSIGHT_016313

creators—for instance, a good model of the Earth’s climate should be able to predict the consequences of a rising global temperature even if this wasn’t something considered by the scientists who designed it. However, when it comes to understanding the human mind, these two goals—accuracy and generalizability—have long been at odds with each other. At the far extreme of generalizability are rational theories of cognition. These theories describe human behavior as a rational response to a given situation. A rational actor strives to maximize the expected reward produced by a sequence of actions—an idea widely used in economics precisely because it produces such generalizable predictions about human behavior. For the same reason, rationality is the standard assumption in inverse-reinforcement-learning models that try to make inferences from human behavior—perhaps with the concession that humans are not perfectly rational agents and sometimes randomly choose to act in ways unaligned with or even opposed to their best interests. The problem with rationality as a basis for modeling human cognition 1s that it is not accurate. In the domain of decision making, an extensive literature—spearheaded by the work of cognitive psychologists Daniel Kahneman and Amos Tversky—has documented the ways in which people deviate from the prescriptions of rational models. Kahneman and Tversky proposed that in many situations people instead follow simple heuristics that allow them to reach good solutions at low cognitive cost but sometimes result in errors. To take one of their examples, if you ask somebody to evaluate the probability of an event, they might rely on how easy it is to generate an example of such an event from memory, consider whether they can come up with a causal story for that event’s occurring, or assess how similar the event is to their expectations. Each heuristic is a reasonable strategy for avoiding complex probabilistic computations, but also results in errors. For instance, relying on the ease of generating an event from memory as a guide to its probability leads us to overestimate the chances of extreme (hence extremely memorable) events such as terrorist attacks. Heuristics provide a more accurate model of human cognition but one that is not easily generalizable. How do we know which heuristic people might use in a particular situation? Are there other heuristics they use that we just haven’t discovered yet? Knowing exactly how people will behave in a new situation is a challenge: Is this situation one in which they would generate examples from memory, come up with causal stories, or rely on similarity? Ultimately, what we need is a way to describe how human minds work that has the generalizability of rationality and the accuracy of heuristics. One way to achieve this goal is to start with rationality and consider how to take it in a more realistic direction. A problem with using rationality as a basis for describing the behavior of any real-world agent is that, in many situations, calculating the rational action requires the agent to possess a huge amount of computational resources. It might be worth expending those resources 1f you’re making a highly consequential decision and have a lot of time to evaluate your options, but most human decisions are made quickly and for relatively low stakes. In any situation where the time you spend making a decision is costly—at the very least because it’s time you could spend doing something else—the classic notion of rationality is no longer a good prescription for how one should behave. To develop a more realistic model of rational behavior, we need to take into 94 HOUSE_OVERSIGHT_016314

account the cost of computation. Real agents need to modulate the amount of time they spend thinking by the effect the extra thought has on the results of a decision. If you’re trying to choose a toothbrush, you probably don’t need to consider all four thousand listings for manual toothbrushes on Amazon.com before making a purchase: You trade off the time you spend looking with the difference it makes in the quality of the outcome. This trade-off can be formalized, resulting in a model of rational behavior that artificial- intelligence researchers call “bounded optimality.” The bounded-optimal agent doesn’t focus on always choosing exactly the right action to take but rather on finding the right algorithm to follow in order to find the perfect balance between making mistakes and thinking too much. Bounded optimality bridges the gap between rationality and heuristics. By describing behavior as the result of a rational choice about how much to think, it provides a generalizable theory—that is, one that can be applied in new situations. Sometimes the simple strategies that have been identified as heuristics that people follow turn out to be bounded-optimal solutions. So, rather than condemning the heuristics that people use as irrational, we can think of them as a rational response to constraints on computation. Developing bounded optimality as a theory of human behavior is an ongoing project that my research group and others are actively pursuing. If these efforts succeed, they will provide us with the most important ingredient we need for making artificial- intelligence systems smarter when they try to interpret people’s actions, by enabling a generative model for human behavior. Taking into account the computational constraints that factor into human cognition will be particularly important as we begin to develop automated systems that aren’t subject to the same constraints. Imagine a superintelligent AI system trying to figure out what people care about. Curing cancer or confirming the Riemann hypothesis, for instance, won’t seem, to such an AI, like things that are all that important to us: If these solutions are obvious to the superintelligent system, it might wonder why we haven’t found them ourselves, and conclude that those problems don’t mean much to us. If we cared and the problems were so simple, we would have solved them already. A reasonable inference would be that we do science and math purely because we enjoy doing science and math, not because we care about the outcomes. Anybody who has young children can appreciate the problem of trying to interpret the behavior of an agent that is subject to computational constraints different from one’s own. Parents of toddlers can spend hours trying to disentangle the true motivations behind seemingly inexplicable behavior. As a father and a cognitive scientist, I found it was easier to understand the sudden rages of my two-year-old when I recognized that she was at an age where she could appreciate that different people have different desires but not that other people might not know what her own desires were. It’s easy to understand, then, why she would get annoyed when people didn’t do what she (apparently transparently) wanted. Making sense of toddlers requires building a cognitive model of the mind of a toddler. Superintelligent AI systems face the same challenge when trying to make sense of human behavior. Superintelligent AI may still be a long way off. In the short term, devising better models of people can prove extremely valuable to any company that makes money by analyzing human behavior—which at this point is pretty much every company that does business on the Web. Over the last few years, significant new commercial technologies 55 HOUSE_OVERSIGHT_016315

for interpreting images and text have resulted from developing good models for vision and language. Developing good models of people is the next frontier. Of course, understanding how human minds work isn’t just a way to make computers better at interacting with people. The trade-off between making mistakes and thinking too much that characterizes human cognition is a trade-off faced by any real- world intelligent agent. Human beings are an amazing example of systems that act intelligently despite significant computational constraints. We’re quite good at developing strategies that allow us to solve problems pretty well without working too hard. Understanding how we do this will be a step toward making computers work smarter, not harder. 96 HOUSE_OVERSIGHT_016316

Romanian-born Anca Dragan’s research focuses on algorithms that will enable robots to work with, around, and in support of people. She runs the InterACT Laboratory at Berkeley, where her students work across different applications, from assistive robots to manufacturing to autonomous cars, and draw from optimal control, planning, estimation, learning, and cognitive science. Barely into her thirties herself, she has co-authored a number of papers with her veteran Berkeley colleague and mentor Stuart Russell which address various aspects of machine learning and the knotty problems of value alignment. She shares Stuart’s preoccupation with AI safety: “An immediate risk is agents producing unwanted, surprising behavior,” she told an interviewer from the Future of Life Institute. “Even ifwe plan to use AI for good, things can go wrong, precisely because we are bad at specifying objectives and constraints for AI agents. Their solutions are often not what we had in mind.” Her principal goal is therefore to help robots and programmers alike to overcome the many conflicts that arise because of a lack of transparency about each other's intentions. Robots, she says, need to ask us questions. They should wonder about their assignments, and they should pester their human programmers until everybody is on the same page—so as to avoid what she has euphemistically called “unexpected side effects.” 97 HOUSE_OVERSIGHT_016317

PUTTING THE HUMAN INTO THE AI EQUATION Anca Dragan Anca Dragan is an assistant professor in the Department of Electrical Engineering and Computer Sciences at UC Berkeley. She co-founded and serves on the steering committee for the Berkeley AI Research (BAIR) Lab and is a co-principal investigator in Berkeley’s Center for Human-Compatible Al. At the core of artificial intelligence is our mathematical definition of what an AI agent (a robot) is. When we define a robot, we define states, actions, and rewards. Think of a delivery robot, for instance. States are locations in the world, and actions are motions that the robot makes to get from one position to a nearby one. To enable the robot to decide on which actions to take, we define a reward function—a mapping from states and actions to scores indicating how good that action was in that state—and have the robot choose actions that accumulate the most “reward.” The robot gets a high reward when it reaches its destination, and it incurs a small cost every time it moves; this reward function incentivizes the robot to get to the destination as quickly as possible. Similarly, an autonomous car might get a reward for making progress on its route and incur a cost for getting too close to other cars. Given these definitions, a robot’s job is to figure out what actions it should take in order to get the highest cumulative reward. We’ve been working hard in AI on enabling robots to do just that. Implicitly, we’ve assumed that if we’re successful—if robots can take any problem definition and turn into a policy for how to act—we will get robots that are useful to people and to society. We haven’t been too wrong so far. If you want an AI that classifies cells as either cancerous or benign, or a robot that vacuums the living room rug while you’re at work, we've got you covered. Some real-world problems can indeed be defined in isolation, with clear-cut states, actions, and rewards. But with increasing AI capability, the problems we want to tackle don’t fit neatly into this framework. We can no longer cut off a tiny piece of the world, put it in a box, and give it to a robot. Helping people is starting to mean working in the real world, where you have to actually interact with people and reason about them. “People” will have to formally enter the AI problem definition somewhere. Autonomous cars are already being developed. They will need to share the road with human-driven vehicles and pedestrians and learn to make the trade-off between getting us home as fast as possible and being considerate of other drivers. Personal assistants will need to figure out when and how much help we really want and what types of tasks we prefer to do on our own versus what we can relinquish control over. A DSS (Decision Support System) or a medical diagnostic system will need to explain its recommendations to us so we can understand and verify them. Automated tutors will need to determine what examples are informative or illustrative—not to their fellow machines but to us humans. Looking further into the future, if we want highly capable Als to be compatible with people, we can’t create them in isolation from people and then try to make them compatible afterward; rather, we’ll have to define “human-compatible” AI from the get- go. People can’t be an afterthought. 98 HOUSE_OVERSIGHT_016318

When it comes to real robots helping real people, the standard definition of AI fails us, for two fundamental reasons: First, optimizing the robot’s reward function in isolation is different from optimizing it when the robot acts around people, because people take actions too. We make decisions in service of our own interests, and these decisions dictate what actions we execute. Moreover, we reason about the robot—that is, we respond to what we think it’s doing or will do and what we think its capabilities are. Whatever actions the robot decides on need to mesh well with ours. This is the coordination problem. Second, it is ultimately a human who determines what the robot’s reward function should be in the first place. And they are meant to incentivize robot behavior that matches what the end-user wants, what the designer wants, or what society as a whole wants. I believe that capable robots that go beyond very narrowly defined tasks will need to understand this to achieve compatibility with humans. This is the va/ue-alignment problem. The Coordination Problem: People are more than objects in the environment. When we design robots for a particular task, it’s tempting to abstract people away. A robotic personal assistant, for example, needs to know how to move to pick up objects, so we define that problem in isolation from the people for whom the robot is picking these objects up. Still, as the robot moves around, we don’t want it bumping into anything, and that includes people, so we might include the physical location of the person in the definition of the robot’s state. Same for cars: We don’t want them colliding with other cars, so we enable them to track the positions of those other cars and assume that they’ ll be moving consistently in the same direction in the future. A human being, in this sense, is no different to a robot from a ball rolling on a flat surface. The ball will behave in the next few seconds the same way it behaved in the past few; it keeps rolling in the same direction at roughly the same speed. This is of course nothing like real human behavior, but such simplification enables many robots to succeed in their tasks and, for the most part, stay out of people’s way. A robot in your house, for example, might see you coming down the hall, move aside to let you pass, and resume its task once you’ve gone by. As robots have become more capable, though, treating people as consistently moving obstacles is starting to fall short. A human driver switching lanes won’t continue in the same direction but will move straight ahead once they’ ve made the lane change. When you reach for something, you often reach around other objects and stop when you get to the one you want. When you walk down a hallway, you have a destination in mind: You might take a right into the bedroom or a left into the living room. Relying on the assumption that we’re no different from a rolling ball leads to inefficiency when the robot stays out of the way if it doesn’t need to, and it can imperil the robot when the person’s behavior changes. Even just to stay out of the way, robots have to be somewhat accurate at anticipating human actions. And, unlike the rolling ball, what people will do depends on what they decide to do. So to anticipate human actions, robots need to start understanding human decision making. And that doesn’t mean assuming that human behavior is perfectly optimal; that might be enough for a chess- or Go-playing robot, but in the real world, people’s decisions are less predictable than the optimal move in a board game. 99 HOUSE_OVERSIGHT_016319

This need to understand human actions and decisions applies to physical and nonphysical robots alike. If either sort bases its decision about how to act on the assumption that a human will do one thing but the human does something else, the resulting mismatch could be catastrophic. For cars, it can mean collisions. For an AI with, say, a financial or economic role, the mismatch between what it expects us to do and what we actually do could have even worse consequences. One alternative is for the robot not to predict human actions but instead just protect against the worst-case human action. Often when robots do that, though, they stop being all that useful. With cars, this results in being stuck, because it makes every move too risky. All this puts us, the AI community, into a bind. It suggests that robots will need accurate (or at least reasonable) predictive models of whatever people might decide to do. Our state definition can’t just include the physical position of humans in the world. Instead, we’ ll also need to estimate something internal to people. We’ll need to design robots that account for this human internal state, and that’s a tall order. Luckily, people tend to give robots hints as to what their internal state is: Their ongoing actions give the robot observations (in the Bayesian inference sense) about their intentions. If we start walking toward the right side of the hallway, we’re probably going to enter the next room on the right. What makes the problem more complicated 1s the fact that people don’t make decisions in isolation. It would be one thing if robots could predict the actions a person intends to take and simply figure out what to doin response. But unfortunately this can lead to ultra-defensive robots that confuse the heck out of people. (Think of human drivers stuck at four-way stops, for instance.) What the intent-prediction approach misses is that the moment the robot acts, that influences what actions the human starts taking. There is a mutual influence between robots and people, one that robots will need to learn to navigate. It is not always just about the robot planning around people; people plan around the robot, too. It is important for robots to account for this when deciding which actions to take, be it on the road, in the kitchen, or even in virtual spaces, where actions might be making a purchase or adopting a new strategy. Doing so should endow robots with coordination strategies, enabling them to take part in the negotiations people seamlessly carry out day to day—from who goes first at an intersection or through a narrow door, to what role we each take when we collaborate on preparing breakfast, to coming to consensus on what next step to take on a project. Finally, just as robots need to anticipate what people will do next, people need to do the same with robots. This is why transparency is important. Not only will robots need good mental models of people, but people will need good mental models of robots. The model that a person has of the robot has to go into our state definition as well, and the robot has to be aware of how its actions are changing that model. Much like the robot treating human actions as clues to human internal states, people will change their beliefs about the robot as they observe its actions. Unfortunately, the giving of clues doesn’t come as naturally to robots as it does to humans; we’ve had a lot of practice communicating implicitly with people. But enabling robots to account for the change that their actions are causing to the person’s mental model of the robot can lead to more carefully chosen actions that do give the right clues—that clearly communicate to people about the robot’s intentions, its reward function, its limitations. For instance, a robot 100 HOUSE_OVERSIGHT_016320

might alter its motion when carrying something heavy, to emphasize the difficulty it has in maneuvering heavy objects. The more that people know about the robot, the easier it is to coordinate with it. Achieving action compatibility will require robots to anticipate human actions, account for how those actions will influence their own, and enable people to anticipate robot actions. Research has ,ade a degree of progress in meeting these challenges, but we still have a long way to go. The Value Alignment Problem: People hold the key to the robot’s reward function. Progress on enabling robots to optimize reward puts more burden on us, the designers, to give them the right reward to optimize in the first place. The original thought was that for any task we wanted the robot to do, we could write down a reward function that incentivizes the right behavior. Unfortunately, what often happens is that we specify some reward function and the behavior that emerges out of optimizing it isn’t what we want. Intuitive reward functions, when combined with unusual instances of a task, can lead to unintuitive behavior. You reward an agent in a racing game with a score in the game, and in some cases it finds a loophole that it exploits to gain infinitely many points without actually winning the race. Stuart Russell and Peter Norvig give a beautiful example in their book Artificial Intelligence: A Modern Approach: rewarding a vacuuming robot for how much dust it sucks in results in the robot deciding to dump out dust so that it can suck it in again and get more reward. In general, humans have had a notoriously difficult time specifying exactly what they want, as exemplified by all those genie legends. An AI paradigm in which robots get some externally specified reward fails when that reward is not perfectly well thought out. It may incentivize the robot to behave in the wrong way and even resist our attempts to correct its behavior, as that would lead to a lower specified reward. A seemingly better paradigm might be for robots to optimize for what we internally want, even if we have trouble explicating it. They would use what we say and do as evidence about what we want, rather than interpreting it literally and taking it as a given. When we write down a reward function, the robot should understand that we might be wrong: that we might not have considered all facets of the task; that there’s no guarantee that said reward function will a/ways lead to the behavior we want. The robot should integrate what we wrote down into its understanding of what we want, but it should also have a back-and-forth with us to elicit clarifying information. It should seek our guidance, because that’s the only way to optimize the true desired reward function. Even if we give robots the ability to learn what we want, an important question remains that AI alone won’t be able to answer. We can make robots try to align with a person’s internal values, but there’s more than one person involved here. The robot has an end-user (or perhaps a few, like a personal robot caring for a family, a car driving a few passengers to different destinations, or an office assistant for an entire team); it has a designer (or perhaps a few); and it interacts with society—the autonomous car shares the road with pedestrians, human-driven vehicles, and other autonomous cars. How to combine these people’s values when they might be in conflict is an important problem we need to solve. AI research can give us the tools to combine values in any way we decide but can’t make the necessary decision for us. 101 HOUSE_OVERSIGHT_016321

In short, we need to enable robots to reason about us—to see us as something more than obstacles or perfect game players. We need them to take our human nature into account, so that they are well coordinated and well aligned with us. If we succeed, we will indeed have tools that substantially increase our quality of life. 102 HOUSE_OVERSIGHT_016322

Chris Anderson’s company, 3DR, helped start the modern drone industry and now focuses on drone data software. He got his start building an open-source aerial robotics community called DIY Drones, and undertook some ill-advised early experiments, such as buzzing Lawrence Berkeley Laboratory with one of his self-flying spies. It may well have been a case of antic gene-expression, since he’s descended from a founder of the American Anarchist movement. Chris ran Wired magazine, a go-to publication for techno-utopians and -dystopians alike, from 2001 to 2012; during his tenure it won five National Magazine Awards. Chris dislikes the term “roboticist” (“like any properly humbled roboticist, I don’t call myself one”). He began as a physicist. “I turned out to be a bad physicist,” he told me recently. “T struggled on, went to Los Alamos, and thought, ‘Well maybe I’m not going to be a Nobel Prize winner, but I can still be a scientist.’ All of us who were in Physics and had these romantic heroes—the Feynmans, the Manhattan Project—realized that our career trajectory would at best be working on one project at CERN for fifteen years. That project would either be a failure, in which case there would be no paper, or it would be a success, in which case you'd be author #300 on the paper and become an assistant professor at Iowa State. “Most of my classmates went to Wall Street to become quants, and to them we owe the subprime mortgage. Others went on to start the Internet. First, we built the Internet by connecting physics labs; second, we built the Web; third, we were the first to do Big Data. We had supercomputers—Crays—which were half the power of your phone now, but they were the supercomputers of the time. Meanwhile, we were reading this magazine called Wired, which came out in 1993, and we realized that this tool we scientists use could have applications for everybody. The Internet wasn’t just about scientific data, it was a mind-blowing cultural revolution. So when Conde Nast asked me to take over the magazine, I was like, ‘Absolutely!’ This magazine changed my life.” He had five children by that time—video-game players—who got him into the “flving robots.” He quit his day job at Wired. The rest is Silicon Valley history. 103 HOUSE_OVERSIGHT_016323

GRADIENT DESCENT Chris Anderson Chris Anderson is an entrepreneur; former editor-in-chief of Wired; co-founder and CEO of 3DR; and author of The Long Tail, Free, and Makers. Life The mosquito first detects my scent from thirty feet away. It triggers its pursuit function, which consists of the simplest possible rules. First, move in a random direction. If the scent increases, continue moving in that direction. If the scent decreases, move in the opposite direction. If the scent is lost, move sideways until a scent is picked up again. Repeat until contact with the target is achieved. The plume of my scent is densest next to me and disperses as it spreads, an invisible fog of particles exuded from my skin that moves like smoke with the wind. The closer to my skin, the higher the particle density; the farther away, the lower. This decrease 1s called a gradient, which describes any gradual transition from one level to another one—as opposed to a “step function,” which describes a discrete change. Once the mosquito follows this gradient to its source using its simple algorithm, it lands on my skin, which it senses with the heat detectors in its feet, which are attuned to another gradient—temperature. It then pushes its needle-shaped proboscis through the surface, where a third set of sensors in the tip detect yet another gradient, that of blood density. This flexible needle wriggles around under my skin until the scent of blood steers it to a capillary, which it punctures. Then my blood begins to flow into the mosquito. Mission accomplished. Ouch. What seems like the powerful radar of insects in the dark, with blood-seeking intelligence inexplicable for such tiny brains, is actually just a sensitive nose with almost no intelligence at all. Mosquitoes are closer to plants that follow the sun than to guided missiles. Yet by applying this simple “follow your nose” rule quite literally, they can travel through a house to find you, slip through cracks in a screen door, even zero in on the tiny strip of skin you left exposed between hat and shirt collar. It’s just a random walk, combined with flexible wings and legs that let the insect bounce off obstacles, and an instinct to descend a chemical gradient. But “gradient descent” is much more than bug navigation. Look around you and you'll find it everywhere, from the most basic physical rules of the universe to the most advanced artificial intelligence. The Universe We live in a world of countless gradients, from light and heat to gravity and chemical trails (chemtrails!). Water flows along a gravity gradient downhill, and your body lives on chemical solutions flowing across cell membranes from high concentration to low. Every action in the universe 1s driven by some gradient drive, from the movement of the planets around gravity gradients to the joining of atoms along electric-charge gradients to form molecules. Our own urges, such as hunger and sleepiness, are driven by electro-chemical gradients in our bodies. And our brain’s functions, the electrical signals moving along ion channels in the synapses between our neurons, are simply atoms and electrons flowing “downhill” along yet more electrical and chemical gradients. 104 HOUSE_OVERSIGHT_016324

Forget clockwork analogies; our brains are closer to a system of canals and locks, with signals traveling like water from one state to another. As I sit here typing, I’m actually seeking equilibrium states in an n-dimensional topology of gradients. Take just one: heat. My body temperature is higher than the air temperature, so I radiate heat, which must be replenished in my core. Even the bacteria in my digestive tract use sensors to measure sugar concentrations in the liquid around them and whip their tail-like flagella to swim “upstream” where the sugar supply is richest. The natural state of all systems is to flow to lower energy states, a process that is broadly described by entropy (the tendency of things to go from ordered to disordered states; all things will fall apart eventually, including the universe itself). But how do you explain more complex behavior, such as our ability to make decisions? The answer is just more gradient descent. Our Brains As miraculous and inscrutable as our human intelligence is, science is coming around to the view that our brains operate the same way as any other complex system with layers and feedback loops, all pursuing what we mathematically call “optimization functions” but you could just as well call “flowing downhill” in some sense. The essence of intelligence is learning, and we do that by correlating inputs with positive or negatives scores (rewards or punishment). So, for a baby, “this sound” (your mother’s voice) is associated with other learned connections to your mother, such as food or comfort. Likewise, “this muscle motion brings my thumb closer to my mouth.” Over time and trial and error, the brain’s neural network reinforces those connections. Meanwhile “this muscle motion does not bring my thumb close to my mouth” is a negative correlation, and the brain will weaken those connections. However, this is too simplistic. The limits of gradient descent constitute the so- called local-minima problem (or local-maxima problem, if you’re doing a gradient ascent). If you are walking in a mountainous region and want to get home, always walking downhill will most likely get you to the next valley but not necessarily over the other mountains that lie around it and between you and home. For that, you need either a mental model (i.e., a map) of the topology so you know where to ascend to get out of the valley, or you need to switch between gradient descent and random walks so you can bounce your way out of the region. Which 1s, in fact, exactly what the mosquito does in following my scent: It descends when it’s in my plume and random-walks when it has lost the trail or hit an obstacle. Al So that’s nature. What about computers? Traditional software doesn’t work that way—it follows deterministic trees of hard logic: “If this, do that.” But software that interacts with the physical world tends to work more /ike the physical world. That means dealing with noisy inputs (sensors or human behavior) and providing probabilistic, not deterministic, results. And that, in turn, means more gradient descent. AI software is the best example of this, especially the kinds of AI that use artificial neural-network models (including convolutional, or “deep,” neural networks of many layers). In these, a typical process consists of “training” them by showing them 105 HOUSE_OVERSIGHT_016325

lots of examples of something you want them to learn (pictures of cats labeled “cat,” for example), along with examples of other random data (pictures of other things). This is called “supervised learning,” because the neural network is being taught by example, including the use of “adversarial training” with data that is not correlated to the desired result. These neural networks, like their biological models, consist of layers of thousands of nodes (“neurons,” in the analogy), each of which is connected to all the nodes in the layers above and below by connections that initially have random strength. The top layer is presented with data, and the bottom layer is given the correct answer. Any series of connections that happened to land on the right answer is made stronger (“rewarded”), and those that were wrong are made weaker (“punished”). Repeat tens of thousands of times and eventually you have a fully trained network for that kind of data. You can think of all the possible combinations of connections as like the surface of a planet, with hills and valleys. (Ignore for the moment that the surface is just 3D and the actual topology is many-dimensional.) The optimization that the network goes through as it learns is just a process of finding the deepest valley on the planet. This consists of the following steps: 1. Define a “cost function” that determines how well the network solved the problem 2. Run the network once and see how it did at that cost function 3. Change the values of the connections and do it again. The difference between those two results is the direction, or “slope,” in which the network moved between the two trials. 4. Ifthe slope is pointed “downhill,” change the connections more in that direction. If it’s “uphill,” change them in the opposite direction. 5. Repeat until there is no improvement in any direction. That means that you’re in a minimum. Congrats! But it’s probably a /oca/ minimum, or a little dip in the mountains, so you’re going to have to keep going if you want to do better. You can’t keep going downhill, and you don’t know where the absolute lowest point is, so you’re going to have to somehow find it. There are many ways to do that, but here are a few: 1. Try lots of times with different random settings and share learning from each trial; essentially, you are shaking the system to see if it settles in a lower state. If one of the other trials found a lower valley, start with those settings. 2. Don’t just go downhill but stumble around a bit like a drunk, too (this is called “stochastic gradient descent”). If you do this long enough, you’ll eventually find rock bottom. There’s a metaphor for life in that. 3. Just look for “interesting” features, which are defined by diversity (edges or color changes, for example). Warning: This way can lead to madness—too much “interestingness” draws the network to optical illusions. So keep it sane, and emphasize the kinds of features that are likely to be real in nature, as opposed to artifacts or errors. This is called “regularization,” and there are lots of techniques for this, such as whether those kinds of features have been seen before (learned), 106 HOUSE_OVERSIGHT_016326

or are too “high frequency” (like static) rather than “low frequency” (more continuous, like actual real-world features). Just because AI systems sometimes end up in local minima, don’t conclude that this makes them any less like life. Humans—indeed, probably all life-forms—are often stuck in local minima. Take our understanding of the game of Go, which was taught and learned and optimized by humans for thousands of years. It took Als less than three years to find out that we’d been playing it wrong all along and that there were better, almost alien, solutions to the game which we’d never considered—mostly because our brains don’t have the processing power to consider so many moves ahead. Even in chess, which is ten times easier and was thought to be understood, brute- force machines could beat us at our own strategies. Chess, too, turned out, when explored by superior neural-network AI systems, to have weird but superior strategies we'd never considered, like sacrificing queens early to gain an obscure long-term advantage. It’s as if we had been playing 2D versions of games that actually existed in higher dimensions. If any of this sounds familiar, it’s because physics has been wrestling with these sorts of topological problems for decades. The notion of space being many-dimensional, and math reducing to understanding the geometries and interactions of “membranes” beyond the reach of our senses, is where Grand Unified Theorists go to die. But unlike multidimensional theoretical physics, AI is something we can actually experiment with and measure. So that’s what we’re going to do. The next few decades will be an explosive exploration of ways to think that 7 million years of evolution never found. We’re going to rock ourselves out of local minima and find deeper minima, maybe even global minima. And when we’re done, we may even have taught machines to seem as smart as a mosquito, forever descending the cosmic gradients to an ultimate goal, whatever that may be. 107 HOUSE_OVERSIGHT_016327

David Kaiser is a physicist atypically interested in the intersection of his science with politics and culture, about which he has written widely. In the first meeting (in Washington, Connecticut) that preceded the crafting of this book, he commented on the change in how “information” is viewed since Wiener’s time: the military-industrial, Cold War era. Back then, Wiener compared information, metaphorically, to entropy, in that it could not be conserved—i.e., monopolized; thus, he argued, our atomic secrets and other such classified matters would not remain secrets for long. Today, whereas (as Wiener might have expected) information, fake or not, is leaking all over the other Washington, information in the economic world has indeed been stockpiled, commodified, and monetized. This lockdown, David said, was “not all good, not all bad’”—depending, I guess, on whether you’re sick of being pestered by ads for socks or European river cruises popping up in your browser minutes after you’ve bought them. To say nothing of information’s proliferation. David complained to the rest of us attending the meeting that in Wiener’s time, physicists could “take the entire Physical Review. /t would sit comfortably in front of us in a manageable pile. Now we’re awash in fifty thousand open-source journals per minute,” full of god-knows-what. Neither of these developments would Wiener have anticipated, said David, prompting him to ask, “Do we need a new set of guiding metaphors?” 108 HOUSE_OVERSIGHT_016328

“INFORMATION” FOR WIENER, FOR SHANNON, AND FOR US David Kaiser David Kaiser is Germeshausen Professor of the History of Science and professor of Physics at MIT, and head of its Program in Science, Technology & Society. He is the author of How the Hippies Saved Physics: Science, Counterculture, and the Quantum Revival and American Physics and the Cold War Bubble (forthcoming). In The Sleepwalkers, a sweeping history of scientific thought from ancient times through the Renaissance, Arthur Koestler identified a tension that has marked the most dramatic leaps of our cosmological imagination. In reading the great works of Nicolaus Copernicus and Johannes Kepler today, Koestler argued, we are struck as much by their strange unfamiliarity—their embeddedness in the magic or mysticism of an earlier age— as by their modern-sounding insights. I detect that same doubleness—the zig-zag origami folds of old and new—in Norbert Wiener’s classic The Human Use of Human Beings. First published in 1950 and revised in 1954, the book is in many ways extraordinarily prescient. Wiener, the MIT polymath, recognized before most observers that “society can only be understood through a study of the messages and the communication facilities which belong to it.”. Wiener argued that feedback loops, the central feature of his theory of cybernetics, would play a determining role in social dynamics. Those loops would not only connect people with one another but connect people with machines, and—crucially—machines with machines. Wiener glimpsed a world in which information could be separated from its medium. People, or machines, could communicate patterns across vast distances and use them to fashion new items at the endpoints, without “moving a...particle of matter from one end of the line to the other,” a vision now realized in our world of networked 3D printers. Wiener also imagined machine-to-machine feedback loops driving huge advances in automation, even for tasks that had previously relied on human judgment. “The machine plays no favorites between manual labor and white-collar labor,” he observed. For all that, many of the central arguments in Zhe Human Use of Human Beings seem closer to the 19th century than the 21st. In particular, although Wiener made reference throughout to Claude Shannon’s then-new work on information theory, he seems not to have fully embraced Shannon’s notion of information as consisting of irreducible, meaning-free bits. Since Wiener’s day, Shannon’s theory has come to undergird recent advances in “Big Data” and “deep learning,” which makes it all the more interesting to revisit Wiener’s cybernetic imagination. How might tomorrow’s artificial intelligence be different if practitioners were to re-invest in Wiener’s guiding vision of “information”? When Wiener wrote Zhe Human Use of Human Beings, his experiences of war-related research, and of what struck him as the moral ambiguities of intellectual life amid the military-industrial complex, were still fresh. Just a few years earlier, he had announced 109 HOUSE_OVERSIGHT_016329

in the pages of The Atlantic Monthly that he would not “publish any future work of mine which may do damage in the hands of irresponsible militarists.”°° He remained ambivalent about the transformative power of new technologies, indulging in neither the boundless hype nor the digital utopianism of later pundits. “Progress imposes not only new possibilities for the future but new restrictions,” he wrote, in Human Use. He was concerned about human-made restrictions as well as technological ones, especially Cold War restrictions that threatened the flow of information so critical to cybernetic systems: “Under the impetus of Senator [Joseph] McCarthy and his imitators, the blind and excessive classification of military information” was driving political leaders in the United States to adopt a “secretive frame of mind paralleled in history only in the Venice of the Renaissance.” Wiener, echoing many outspoken veterans of the Manhattan Project, argued that the postwar obsession with secrecy—especially around nuclear weapons—stemmed from a misunderstanding of the scientific process. The only genuine secret about the production of nuclear weapons, he wrote, was whether such bombs could be built. Once that secret had been revealed, with the bombings of Hiroshima and Nagasaki, no amount of state-imposed secrecy would stop others from puzzling through chains of reasoning like those the Manhattan Project researchers had followed. As Wiener memorably put it, “There is no Maginot Line of the brain.” To drive this point home, Wiener borrowed Shannon’s fresh ideas about information theory. In 1948, Shannon, a mathematician and engineer working at Bell Labs, had published a pair of lengthy articles in the Be// System Technical Journal. Introducing the new work to a broad readership in 1949, mathematician Warren Weaver explained that in Shannon’s formulation, “the word information...is used in a special sense that must not be confused with its ordinary usage. In particular, information must not be confused with meaning.”?! Linguists and poets might be concerned about the “semantic” aspects of communication, Weaver continued, but not engineers like Shannon. Rather, “this word ‘information’ in communication theory relates not so much to what you do say, as to what you could say.” In Shannon’s now-famous formulation, the information content of a string of symbols was given by the logarithm of the number of possible symbols from which a given string was chosen. Shannon’s key insight was that the information of a message was just like the entropy of a gas: a measure of the system’s disorder. Wiener borrowed this insight when composing Human Use. If information was like entropy, then it could not be conserved—or contained. Physicists in the 19th century had demonstrated that the total energy of a physical system must always remain the same, a perfect balance between the start and the end of a process. Not so for entropy, which would inexorably increase over time, an imperative that came to be known as the second law of thermodynamics. From that stark distinction—energy is conserved, whereas entropy must grow—followed enormous cosmic consequences. Time must flow forward; 3° Norbert Wiener, “A Scientist Rebels,” Zhe Atlantic Monthly, January 1947. 31 Warren Weaver, “Recent Contributions to the Mathematical Theory of Communication,” in Claude Shannon & Warren Weaver, The Mathematical Theory of Communication (Urbana, IL: University of Illinois Press, 1949), p. 8 (emphasis in original). Shannon’s 1948 papers were republished in the same volume. 110 HOUSE_OVERSIGHT_016330

the future cannot be the same as the past. The universe could even be careening toward a “heat death,” some far-off time when the total stock of energy had uniformly dispersed, achieving a state of maximum entropy, after which no further change could occur. If information gua entropy could not be conserved, then Wiener concluded it was folly for military leaders to try to stockpile the “scientific know-how of the nation in static libraries and laboratories.” Indeed, “no amount of scientific research, carefully recorded in books and papers, and then put into our libraries with labels of secrecy, will be adequate to protect us for any length of time in a world where the effective level of information is perpetually advancing.” Any such efforts at secrecy, classification, or the containment of information would fail, Wiener argued, just as surely as hucksters’ schemes for perpetual-motion machines faltered in the face of the second law of thermodynamics. Wiener criticized the American “orthodoxy” of free-market fundamentalism in much the same way. For most Americans, “questions of information will be evaluated according to a standard American criterion: a thing is valuable as a commodity for what it will bring in the open market.” Indeed, “the fate of information in the typically American world is to become something which can be bought or sold;” most people, he observed, “cannot conceive of a piece of information without an owner.” Wiener considered this view to be as wrong-headed as rampant military classification. Again he invoked Shannon’s insight: Since “information and entropy are not conserved,” they are “equally unsuited to being commodities.” Information cannot be conserved—so far, so good. But did Wiener really have Shannon’s “information” in mind? The crux of Shannon’s argument, as Weaver had emphasized, was to distinguish a colloquial sense of “information,” as message with meaning, from an abstracted, rarefied notion of strings of symbols arrayed with some probability and selected from an enormous universe of gibberish. For Shannon, “information” could be quantified because its fundamental unit, the bit, was a unit of conveyance rather than understanding. When Wiener characterized “information” throughout Human Use, on the other hand, he tilted time and again to a classical, humanistic sense of the term. “A piece of information,” he wrote—tellingly, not a “bit” of information—“in order to contribute to the general information of the community, must say something substantially different from the community’s previous common stock of information.” This was why “schoolboys do not like Shakespeare,” he concluded: The Bard’s couplets may depart starkly from random bitstreams, but they had nonetheless become all too familiar to the sense-making public and “absorbed into the superficial clichés of the time.” At least the information content of Shakespeare had once seemed fresh. During the postwar boom years, Wiener fretted, the “enormous per capita bulk of communication”—ranging across newspapers and movies to radio, television, and books—had bred mediocrity, an informational reversion to the mean. “More and more we must accept a standardized inoffensive and insignificant product which, like the white bread of the bakeries, is made rather for its keeping and selling properties than for its food value.” “Heaven save us,” he pleaded, “from the first novels which are written because a young man desires the prestige of being a novelist rather than because he has 111 HOUSE_OVERSIGHT_016331

something to say! Heaven save us likewise from the mathematical papers which are correct and elegant but without body or spirit.” Wiener’s treatment of “information” sounded more like Matthew Arnold in 1869°? than Claude Shannon in 1948—more “body and spirit” than “bit.” Wiener shared Arnold’s Romantic view of the “content producer” as well. “Properly speaking the artist, the writer, and the scientist should be moved by such an irresistible impulse to create that, even if they were not being paid for their work, they would be willing to pay to get the chance to do it.” L’art pour Iart, that 19th-century cry: Artists should suffer for their work; the quest for meaningful expression should always trump lucre. To Wiener, this was the proper measure of “information”: body, spirit, aspiration, expression. Yet to argue against its commodification, Wiener reverted again to Shannon’s mathematics of information-as-entropy. Flash forward to our day. In many ways, Wiener has been proved right. His vision of networked feedback loops driven by machine-to-machine communication has become a mundane feature of everyday life. From the earliest stirrings of the Internet Age, moreover, digital piracy has upended the view that “information”—in the form of songs, movies, books, or code—could remain contained. Put up a paywall here, and the content will diffuse over there, all so much informational entropy that cannot be conserved. On the other hand, enormous multinational corporations—some of the largest and most profitable in the world—now routinely disprove Wiener’s contention that “information” cannot be stockpiled or monetized. Ironically, the “information” they trade in is closer to Shannon’s definition than Wiener’s, Shannon’s mathematical proofs notwithstanding. While Google Books may help circulate hundreds of thousands of works of literature for free, Google itself—like Facebook, Amazon, Twitter, and their many imitators—has commandeered a baser form of “information” and exploited it for extraordinary profit. Petabytes of Shannon-like information—a seemingly meaningless stream of clicks, “likes,” and retweets, collected from virtually every person who has ever touched a networked computer—are sifted through proprietary “deep-learning” algorithms to micro-target everything from the advertisements we see to the news stories (fake or otherwise) we encounter while browsing the Web. Back in the early 1950s, Wiener had proposed that researchers study the structures and limitations of ants—in contrast to humans—so that machines might one day achieve the “almost indefinite intellectual expansion” that people (rather than insects) can attain. He found solace in the notion that machines could come to dominate us only “in the last stages of increasing entropy,” when “the statistical differences among individuals are nil.” Today’s data-mining algorithms turn Wiener’s approach on its head. They produce profit by exploiting our reptilian brains rather than imitating our cerebral cortexes, harvesting information from all our late-night, blog-addled, pleasure-seeking clickstreams—leveraging precisely the tiny, residual “statistical differences among individuals.” 32 Matthew Amold, Culture and Anarchy, Jane Garnett, ed. (Oxford, U.K.: Oxford University Press, 2006). 112 HOUSE_OVERSIGHT_016332

To be sure, some recent achievements in artificial intelligence have been remarkably impressive. Computers can now produce visual artworks and musical compositions akin to those of recognized masters, creating just the sort of “information” that Wiener most prized. But by far the largest impact on society to date has come from the collection and manipulation of Shannon-like information, which has reshaped our shopping habits, political participation, personal relationships, expectations of privacy, and more. What might “deep learning” evolve into, if the fundamental currency becomes “information” as Wiener defined it? How might the field shift if re-animated by Wiener’s deep moral convictions, informed as they were by his prescient concerns about rampant militarism, runaway corporate profit-seeking, the self-limiting features of secrecy, and the reduction of human expression to interchangeable commodities? Perhaps “deep learning” might then become the cultivation of meaningful information rather than the relentless pursuit of potent, if meaningless, bits. 113 HOUSE_OVERSIGHT_016333

In the aforementioned Connecticut discussion on The Human Use of Human Beings, Neil Gershenfeld provided some fresh air, of a kind, by professing that he hated the book, which remark was met by universal laughter—as was his observation that computer science was one the worst things to happen to computers, or science. His overall contention was that Wiener missed the implications of the digital revolution that was happening around him—although some would say this charge can’t be leveled at someone on the ground floor and lacking clairvoyance. “The tail wagging the dog of my life,” he told us, “has been Fab Labs and the maker movement, and [when] Wiener talks about the threat of automation he misses the inverse, Which is that access to the means for automation can empower people, and in Fab Labs, the corner I’ve been involved in, that’s an exponential.” In 2003, I visited Neil at MIT, where he runs the Center for Bits and Atoms. Hours later, I emerged from what had been an exuberant display of very weird stuff. He showed me the work of one student in his popular rapid-prototyping class (“How to Make Almost Anything”), a sculptor with no engineering background, who had made a portable personal space for screaming that saves up your screams and plays them back later. Another student in the class had made a Web browser that lets parrots navigate the Net. Neil himself was doing fundamental research on the roadmap to that sci-fi staple, a “universal replicator.” It was a visit that took me a couple of years to get my head around. Neil manages a global network of Fab Labs—small-scale manufacturing systems, enabled by digital technologies, which give people the wherewithal to build whatever they'd like. As guru of the maker movement, which merges digital communication and computation with fabrication, he sometimes feels outside the current heated debate on AI safety. “My ability to do research rests on tools that augment my capabilities,” he says. “Asking whether or not they are intelligent is as fruitful as asking how I know I exist— amusing philosophically, but not testable empirically.”” What interests him is “how bits and atoms relate—the boundary between digital and physical. Scientifically, it’s the most exciting thing I know.” 114 HOUSE_OVERSIGHT_016334

SCALING Neil Gershenfeld Neil Gershenfeld is a physicist and director of MIT’s Center for Bits and Atoms. He is the author of FAB, co-author (with Alan Gershenfeld & Joel Cutcher-Gershenfeld) of Designing Reality, and founder of the global fab lab network. Discussions about artificial intelligence have been oddly ahistorical. They could better be described as manic-depressive; depending on how you count, we’re now in the fifth boom-bust cycle. Those swings mask the continuity in the underlying progress and the implications for where it’s headed. The cycles have come in roughly decade-long waves. First there were mainframes, which by their very existence were going to automate away work. That ran into the reality that it was hard to write programs to do tasks that were simple for people to do. Then came expert systems, which were going to codify and then replace the knowledge of experts. These ran into difficulty in assembling that knowledge and reasoning about cases not already covered. Perceptrons sought to get around these problems by modeling how the brain learns, but they were unable to do much of anything. Multilayer perceptrons could handle test problems that had tripped up those simpler networks, but their demonstrations did poorly on unstructured, real-world problems. We’re now in the deep-learning era, which is delivering on many of the early AI promises but 1n a way that’s considered hard to understand, with consequences ranging from intellectual to existential threats. Each of these stages was heralded as a revolutionary advance over the limitations of its predecessors, yet all effectively do the same thing: They make inferences from observations. How these approaches relate can be understood by how they scale—that is, how their performance depends on the difficulty of the problem they’re addressing. Both a light switch and a self-driving car must determine their operator’s intentions, but the former has just two options to choose from, whereas the latter has many more. The AI- boom phases have started with promising examples in limited domains; the bust phases came with the failure of those demonstrations to handle the complexity of less-structured, practical problems. Less apparent is the steady progress we’ve made in mastering scaling. This progress rests on the technological distinction between linear and exponential functions— a distinction that was becoming evident at the dawn of AI but with implications for AI that weren’t appreciated until many years later. In one of the founding documents of the study of intelligent machines, 7he Human Use of Human Beings, Norbert Wiener does a remarkable job of identifying many of the most significant trends to arise since he wrote it, along with noting the people responsible for them and then consistently failing to recognize why these people’s work proved to be so important. Wiener is credited with creating the field of cybernetics; ve never understood what that is, but what’s missing from the book is at the heart of how AI has progressed. This history matters because of the echoes of it that persist to this day. Claude Shannon makes a cameo appearance in the book, in the context of his thoughts about the prospects for a chess-playing computer. Shannon was doing 115 HOUSE_OVERSIGHT_016335

something much more significant than speculating at the time: He was laying the foundations for the digital revolution. As a graduate student at MIT, he worked for Vannevar Bush on the Differential Analyzer. This was one of the last great analog computers, a room full of gears and shafts. Shannon’s frustration with the difficulty of solving problems this way led him in 1937 to write what might be the best master’s thesis ever. In it, he showed how electrical circuits could be designed to evaluate arbitrary logical expressions, introducing the basis for universal digital logic. After MIT, Shannon studied communications at Bell Labs. Analog telephone calls degraded with distance; the farther they traveled, the worse they sounded. Rather than continue to improve them incrementally, Shannon showed in 1948 that by communicating with symbols rather than continuous quantities, the behavior is very different. Converting speech waveforms to the binary values of 1 and 0 is an example, but many other sets of symbols can be (and are) used in digital communications. What matters is not the particular symbols but rather the ability to detect and correct errors. Shannon found that if the noise is above a threshold (which depends on the system design), then there are certain to be errors. But if the noise is below a threshold, then a linear increase in the physical resources representing the symbol results in an exponential decrease in the likelihood of making an error in correctly receiving the symbol. This relationship was the first of what we’d now call a threshold theorem. Such scaling falls off so quickly that the probability of an error can be so small as to effectively never happen. Each symbol sent multiplies rather than adds to the certainty, so that the probability of a mistake can go from 0.1 to 0.01 to 0.001, and so forth. This exponential decrease in communication errors made possible an exponential increase in the capacity of communication networks. And that eventually solved the problem of where the knowledge in an AI system came from. For many years, the fastest way to speed up a computation was to do nothing— just wait for computers to get faster. In the same way, there were years of AI projects that aimed to accumulate everyday knowledge by laboriously entering pieces of information. That didn’t scale; it could progress only as fast as the number of people doing the entering. But when phone calls, newspaper stories, and mail messages all moved onto the Internet, everyone doing any of those things became a data generator. The result was an exponential rather than a linear rate of knowledge accumulation. John von Neumann also has a cameo in Zhe Human Use of Human Beings, for game theory. What Wiener missed here was von Neumann’s seminal role in digitizing computation. Whereas analog communication degraded with distance, analog computing (like the Differential Analyzer) degraded with time, accumulating errors as it progressed. Von Neumann presented in 1952 a result corresponding to Shannon’s for computation (they had met at the Institute for Advanced Study, in Princeton), showing that it was possible to compute reliably with an unreliable computing device by using symbols rather than continuous quantities. This was, again, a scaling argument, with a linear increase in the physical resources representing the symbol resulting in an exponential reduction in the error rate as long as the noise was below a threshold. That’s what makes it possible to have a billion transistors in a computer chip, with the last one as useful as the first one. This relationship led to an exponential increase in computing performance, which solved a second problem in AI: how to process exponentially increasing amounts of data. The third problem that scaling solved for AI was coming up with the rules for 116 HOUSE_OVERSIGHT_016336

reasoning without having to hire a programmer for each problem. Wiener recognized the role of feedback in machine learning, but he missed the key role of representation. It’s not possible to store all possible images in a self-driving car, or all possible sounds in a conversational computer; they have to be able to generalize from experience. The “deep” part of deep learning refers not to the (hoped-for) depth of insight but to the depth of the mathematical network layers used to make predictions. It turned out that a linear increase in network complexity led to an exponential increase in the expressive power of the network. If you lose your keys in a room, you can search for them. If you’re not sure which room they’re in, you have to search all the rooms in a building. If you’re not sure which building they’re in, you have to search all the rooms in all the buildings in a city. If you’re not sure which city they’re in, you have to search all the rooms in all the buildings in all the cities. In AI, finding the keys corresponds to things like a car safely following the road, or a computer correctly interpreting a spoken command, and the rooms and buildings and cities correspond to all of the options that have to be considered. This is called the curse of dimensionality. The solution to the curse of dimensionality came in using information about the problem to constrain the search. The search algorithms themselves are not new. But when applied to a deep-learning network, they adaptively build up representations of where to search. The price of this is that it’s no longer possible to exactly solve for the best answer to a problem, but typically all that’s needed is an answer that’s good enough. Taken together, it shouldn’t be surprising that these scaling laws have allowed machines to become effectively as capable as the corresponding stages of biological complexity. Neural networks started out with a goal of modeling how the brain works. That goal was abandoned as they evolved into mathematical abstractions unrelated to how neurons actually function. But now there’s a kind of convergence that can be thought of as forward- rather than reverse-engineering biology, as the results of deep learning echo brain layers and regions. One of the most difficult research projects P ve managed paired what we’d now call data scientists with AI pioneers. It was a miserable experience in moving goalposts. As the former progressed in solving long-standing problems posed by the latter, this was deemed to not count because it wasn’t accompanied by corresponding leaps in understanding the solutions. What’s the value of a chess-playing computer if you can’t explain how it plays chess? The answer of course is that it can play chess. There is interesting emerging research that is applying AI to AI—that is, training networks to explain how they operate. But both brains and computer chips are hard to understand by watching their inner workings; they’re easily interpreted only by observing their external interfaces. We come to trust (or not) brains and computer chips alike based on experience that tests them rather than on explanations for how they work. Many branches of engineering are making a transition from what’s called imperative to declarative or generative design. This means that instead of explicitly designing a system with tools like CAD files, circuit schematics, and computer code, you describe what you want the system to do and then an automated search is done for designs that satisfy your goals and restrictions. This approach becomes necessary as design complexity exceeds what can be understood by a human designer. While that 117 HOUSE_OVERSIGHT_016337

might sound like a risk, human understanding comes with its own limits; engineering design is littered with what appeared to be good insights that have had bad consequences. Declarative design rests on all the advances in AI, plus the improving fidelity of simulations to virtually test designs. The mother of all design problems is the one that resulted in us. The way we’re designed resides in one of the oldest and most conserved parts of the genome, called the Hox genes. These are genes that regulate genes, in what are called developmental programs. Nothing in your genome stores the design of your body; your genome stores, rather, a series of steps to follow that results in your body. This is an exact parallel to how search is done in AI. There are too many possible body plans to search over, and most modifications would be either inconsequential or fatal. The Hox genes are a representation of a productive place for evolutionary search. It’s a kind of natural intelligence at the molecular level. AI has a mind-body problem, in that it has no body. Most work on AI is done in the cloud, running on virtual machines in computer centers where data are funneled. Our own intelligence is the result of a search algorithm (evolution) that was able to change our physical form as well as our programming—those are inextricably linked. Ifthe history of AI can be understood as the working of scaling laws rather than a succession of fashions, then its future can be seen in the same way. What’s now being digitized, after communication and computation, is fabrication, bringing the programmability of bits to the world of atoms. By digitizing not just designs but the construction of materials, the same lessons that von Neumann and Shannon taught us apply to exponentially increasing fabricational complexity. I’ve defined digital materials to be those constructed from a discrete set of parts reversibly joined with a discrete set of relative positions and orientations. These attributes allow the global geometry to be determined from local constraints, assembly errors to be detected and corrected, heterogeneous materials to be joined, and structures to be disassembled rather than disposed of when they’re no longer needed. The amino acids that are the foundation of life and the Lego bricks that are the foundation of play share these properties. What’s interesting about amino acids is that they’re not interesting. They have attributes that are typical but not unusual, such as attracting or repelling water. But just twenty types of them are enough to make you. In the same way, twenty or so types of digital-material part types—conducting, insulating, rigid, flexible, magnetic, etc —are enough to assemble the range of functions that go into making modern technologies like robots and computers. The connection between computation and fabrication was foreshadowed by the very pioneers whose work the edifice of computing is based on. Wiener hinted at this by linking material transportation with message transportation. John von Neumann is credited with modern computer architecture, something he actually wrote very little about; the final thing he studied, and wrote about beautifully and at length, was self- reproducing systems. As an abstraction of life, he modeled a machine that can communicate a computation that constructs itself. And the final thing Alan Turing, who is credited with the theoretical framework for computer science, studied was how the instructions in genes can give rise to physical forms. These questions address a topic absent from a typical computer-science education: the physical configuration of a 118 HOUSE_OVERSIGHT_016338

computation. Von Neumann and Turing posed their questions as theoretical studies, because it was beyond the technology of their day to realize them. But with the convergence of communication and computation with fabrication, these investigations are now becoming accessible experimentally. Making an assembler that can assemble itself from the parts that it’s assembling is a focus of my lab, along with collaborations to develop synthetic cells. The prospect of physically self-reproducing automata is potentially much scarier than fears of out-of-control AI, because it moves the intelligence out here to where we live. It could be a roadmap leading to Terminator’s Skynet robotic overlords. But it’s also a more hopeful prospect, because an ability to program atoms as well as bits enables designs to be shared globally while locally producing things like energy, food, and shelter—all of these are emerging as exciting early applications of digital fabrication. Wiener worried about the future of work, but he didn’t question implicit assumptions about the nature of work which are challenged when consumption can be replaced by creation. History suggests that neither utopian nor dystopian scenarios prevail; we generally end up muddling along somewhere in between. But history also suggests that we don’t have to wait on history. Gordon Moore in 1965 was able to use five years of the doubling of the specifications of integrated circuits to project what turned out to be fifty years of exponential improvements in digital technologies. We’ve spent many of those years responding to, rather than anticipating, its implications. We have more data available now than Gordon Moore did to project fifty years of doubling the performance of digital fabrication. With the benefit of hindsight, it should be possible to avoid the excesses of digital computing and communications this time around, and, from the outset, address issues like access and literacy. If the maker movement is the harbinger of a third digital revolution, the success of AI in meeting many of its own early goals can be seen as the crowning achievement of the first two digital revolutions. Although machine making and machine thinking might appear to be unrelated trends, they lie in each other’s futures. The same scaling trends that have made AI possible suggest that the current mania is a phase that will pass, to be followed by something even more significant: the merging of artificial and natural intelligence. It was an advance for atoms to form molecules, molecules to form organelles, organelles to form cells, cells to form organs, organs to form organisms, organisms to form families, families to form societies, and societies to form civilizations. This grand evolutionary loop can now be closed, with atoms arranging bits arranging atoms. 119 HOUSE_OVERSIGHT_016339

While Danny Hillis was an undergraduate at MIT, he built a computer out of Tinkertoys. It has around 10,000 wooden parts, plays tic-tac-toe, and never loses; it’s now in the Computer History Museum, in Mountain View, California. As a graduate student at the MIT Computer Science and Artificial Intelligence Laboratory in the early 1980s, Danny designed a massively parallel computer with 64,000 processors. He named it the Connection Machine and founded what may have been the first Al company—Thinking Machines Corporation—to produce and market it. This was despite a lunch he had with Richard Feynman, at which the celebrated physicist remarked, “That is positively the dopiest idea I ever heard.” Maybe “despite” is the wrong word, since Feynman had a well-known predilection for playing with dopey ideas. In the event, he showed up on the day the company was incorporated and stayed on, for summer jobs and special assignments, to make invaluable contributions to its work. Danny has since established a number of technology companies, of which the latest is Applied Invention, which partners with commercial enterprises to develop technological solutions to their most intractable problems. He holds hundreds of U.S. patents, covering parallel computers, touch interfaces, disk arrays, forgery prevention methods, and a slew of electronic and mechanical devices. His imagination is apparently boundless, and here he sketches some possible scenarios that will result from our pursuit of a better and better Al. “Our thinking machines are more than metaphors,” he says. “The question is not, ‘Will they be powerful enough to hurt us?’ (they will), or whether they will always act in our best interests (they won't), but whether over the long term they can help us find our way—where we come out on the Panacea/Apocalypse continuum. ” 120 HOUSE_OVERSIGHT_016340

THE FIRST MACHINE INTELLIGENCES W. Daniel Hillis W. Daniel “Danny” Hillis is an inventor, entrepreneur, and computer scientist, Judge Widney Professor of Engineering and Medicine at USC, and author of The Pattern on the Stone: The Simple Ideas That Make Computers Work. I have spoken of machines, but not only of machines having brains of brass and thews of iron. When human atoms are knit into an organization in which they are used, not in their full nght as responsible human beings, but as cogs and levers and rods, it matters little that their raw material is flesh and blood. What is used as an element in a machine, is in fact an element in the machine. Whether we entrust our decisions to machines of metal, or to those machines of flesh and blood which are bureaus and vast laboratories and armies and corporations, we shall never receive the right answers to our questions unless we ask the right questions.... The hour is very late, and the choice of good and evil knocks at our door. —Norbert Wiener, Zhe Human Use of Human Beings Norbert Wiener was ahead of his time in recognizing the potential danger of emergent intelligent machines. I believe he was even further ahead in recognizing that the first artificial intelligences had already begun to emerge. He was correct in identifying the corporations and bureaus that he called “machines of flesh and blood” as the first intelligent machines. He anticipated the dangers of creating artificial superintelligences with goals not necessarily aligned with our own. What is now clear, whether or not it was apparent to Wiener, is that these organizational superintelligences are not just made of humans, they are hybrids of humans and the information technologies that allow them to coordinate. Even in Wiener’s time, the “bureaus and vast laboratories and armies and corporations” could not operate without telephones, telegraphs, radios, and tabulating machines. Today they could not operate without networks of computers, databases, and decision support systems. These hybrid intelligences are technologically augmented networks of humans. These artificial intelligences have superhuman powers. They can know more than individual humans; they can sense more; they can make more complicated analyses and more complex plans. They can have vastly more resources and power than any single individual. Although we do not always perceive it, hybrid superintelligences such as nation states and corporations have their own emergent goals. Although they are built by and for humans, they often act like independent intelligent entities, and their actions are not always aligned to the interests of the people who created them. The state 1s not always for the citizen, nor the company for the shareholder. Nor do not-for-profits, religious orders, or political parties always act in furtherance of their founding principles. Intuitively, we recognize that their actions are guided by internal goals, which is why we personify them, both legally and in our habits of thought. When talking about “what China wants,” or “what General Motors is trying to do,” we are not speaking in metaphors. These organizations act as intelligences that perceive, decide, and act. Like the goals of individual humans, the goals of organizations are complex and often self- contradictory, but they are true goals in the sense that they direct action. Those goals 121 HOUSE_OVERSIGHT_016341

depend somewhat on the goals of the people within the organization, but they are not identical. Any American knows how loose the tie is between the actions of the U.S. government and the diverse and often contradictory aims of its citizens. That is also true of corporations. For-profit corporations nominally serve multiple constituencies, including shareholders, senior executives, employees, and customers. These corporations differ in how they balance their loyalties and often behave in ways that serve none of their constituents. The “neurons” that carry their corporate thought are not just the human employees or the technologies that connect them; they are also coded into the policies, incentive structures, culture, and procedural habits of the corporation. The emergent corporate goals do not always reflect the values of the people who implement them. For instance, an oil company led and staffed by people who care about the environment may have incentive structures or policies that cause it to compromise environmental safety for the sake of corporate earnings. The components’ good intentions are not a guarantee of the emergent system’s good behavior. Governments and corporations, both built partly of humans, are naturally motivated to at least appear to share the goals of the humans they depend upon. They could not function without the people, so they need to keep them cooperative. When such organizations appear to behave altruistically, this is often part of their motive. I once complimented the CEO of a large corporation on the contribution his company made toward a humanitarian relief effort. The CEO responded, without a trace of irony, “Yes. We have decided to do more things like that to make our brand more likeable.” Individuals who compose a hybrid superintelligence may occasionally exert a “humanizing” influence—for example, an employee may break company policies to accommodate the needs of another human. The employee may act out of true human empathy, but we should not attribute any such empathy to the superintelligence itself. These hybrid machines have goals, and their citizens/customers/employees are some of the resources they use to accomplish them. We are close to being able to build superintelligences out of pure information technology, without human components. This is what people normally refer to as “artificial intelligence,” or AI. It is reasonable to ask what the attitudes of the hypothetical machine superintelligences will be toward humans. Will they, too, see humans as useful resources and a good relationship with us as worth preserving? Will they be constructed to have goals that are aligned with our own? Will a superintelligence even see these questions as important? What are the “right questions” that we should be asking? I believe that one of the most important 1s this: What relationship will various superintelligences have to one another? It is interesting to consider how the hybrid superintelligences currently deal with conflicts among themselves. Today, much of the ultimate power rests in the nation states, which claim authority over a patch of ground. Whether they are optimized to act in the interests of their citizens or those of a despotic ruler, nation states assert priority over other intelligences’ desires or goals within their geographic dominion. They claim a monopoly on the use of force and recognize only other nation states as peers. They are willing, if necessary, to demand great sacrifices of their citizens to enforce their authority, even to the point of sacrificing their citizens’ lives. 122 HOUSE_OVERSIGHT_016342

This geographical division of authority made logical sense when most of the actors were humans who spent their lives within a single nation state, but now that the actors of importance include geographically distributed hybrid intelligences such as multinational corporations, that logic is less obvious. Today we live in a complex transitional period, when distributed superintelligences still largely rely on the nation states to settle the arguments arising among them. Often, those arguments are resolved differently in different jurisdictions. It is becoming more difficult even to assign individual humans to nation states: International travelers living and working outside their native country, refugees, and immigrants (documented and not) are still dealt with as awkward exceptions. Superintelligences built purely of information technology will prove even more awkward for the territorial system of authority, since there is no reason why they need to be tied to physical resources in a single country—or even to any particular physical resources at all. An artificial intelligence might well exist “in the cloud” rather than at any physical location. I can imagine at least four scenarios for how machine superintelligences will relate to hybrid superintelligences. In one obvious scenario, multiple machine intelligences will ultimately be controlled by, and allied with, individual nation states. In this state/AI scenario, one can envision American and Chinese super-AIs wrestling each other for resources on behalf of their state. In some sense, these AIs would be citizens of their nation state in the way that many commercial corporations often act as “corporate citizens” today. In this scenario, the host nation states would presumably give the machine superintelligences the resources they needed to work for the state’s advantage. Or, to the degree that the superintelligences can influence their state governments, they will presumably do so to enhance their own power, for instance by garnering a larger share of the state’s resources. Nation states’ Als might not want competing Als to grow up within their jurisdiction. In this scenario, the superintelligences become an extension of the state, and vice versa. The state/AI scenario seems plausible, but it is not our current course. Our most powerful and rapidly improving artificial intelligences are controlled by for-profit corporations. This is the corporate/AI scenario, in which the balance of power between nation states and corporations becomes inverted. Today, the most powerful and intelligent collections of machines are probably owned by Google, but companies like Amazon, Baidu, Microsoft, Facebook, Apple, and IBM may not be far behind. These companies all see a business imperative to build artificial intelligences of their own. It is easy to imagine a future in which corporations independently build their own machine intelligences, protected within firewalls preventing the machines from taking advantage of one another’s knowledge. These machines will be designed to have goals aligned with those of the corporation. If this alignment is effective, nation states may continue to lag behind in developing their own artificial-intelligence capability and instead depend on their “corporate citizens” to do it for them. To the extent that corporations successfully control the goals, they will become more powerful and autonomous than nation states. Another scenario, perhaps the one people fear the most, is that artificial intelligences will not be aligned with either humans or hybrid superintelligences but will act solely in their own interest. They might even merge into a single machine superintelligence, since there may be no technical requirement for machine intelligences to maintain distinct identities. The attitude of a self-interested super-AI toward hybrid 123 HOUSE_OVERSIGHT_016343

superintelligences is likely to be competitive. Humans might be seen as minor annoyances, like ants at a picnic, but hybrid superintelligences—like corporations, organized religions, and nation states—could be existential threats. Like hybrid superintelligences, Als might see humans mostly as useful tools to accomplish their goals, as pawns in their competition with the other superintelligences. Or we might simply be irrelevant. It is not impossible that a machine intelligence has already emerged and we simply do not recognize it as such. It may not wish to be noticed, or it may be so alien to us that we are incapable of perceiving it. This makes the self-interested AI scenario the most difficult to imagine. I believe the easy-to-imagine versions, like the humanoid intelligent robots of science fiction, are the least likely. Our most complex machines, like the Internet, have already grown beyond the detailed understanding of a single human, and their emergent behaviors may be well beyond our ken. The final scenario is that machine intelligences will not be allied with one another but instead will work to further the goals of humanity as a whole. In this optimistic scenario, AI could help us restore the balance of power between the individual and the corporation, between the citizen and the state. It could help us solve the problems that have been created by hybrid superintelligences that subvert the goals of humans. In this scenario, AIs will empower us by giving us access to processing capacity and knowledge currently available only to corporations and states. In effect, they could become extensions of our own individual intelligences, in furtherance of our human goals. They could make our weak individual intelligences strong. This prospect is both exciting and plausible. It is plausible because we have some choice in what we build, and we have a history of using technology to expand and augment our human capacities. As airplanes have given us wings and engines have given us muscles to move mountains, so our network of computers may amplify and extend our minds. We may not fully understand or control our destiny, but we have a chance to bend it in the direction of our values. The future is not something that will happen to us; it is something that we will build. Why Wiener Saw What Others Missed There is in electrical engineering a split which 1s known in Germany as the split between the technique of strong currents and the technique of weak currents, and which we know as the distinction between power and communication engineering. It is this split which separates the age just past from that in which we are now living. —Norbert Wiener, Cybernetics, or Control and Communication in the Animal and the Machine Cybernetics is the study of the how the weak can control the strong. Consider the defining metaphor of the field: the helmsman guiding a ship with a tiller. The helmsman’s goal is to control the heading of the ship, to keep it on the right course. The information, the message that is sent to the helmsman, comes from the compass or the stars, and the helmsman closes the feedback loop by sending the steering messages through the gentle force of his hand on the tiller. In this picture, we see the ship tossing in powerful wind and waves in the real world, controlled by the communication system of messages in the world of information. Yet the distinction between “real” and “information” is mostly a difference in perspective. The signals that carry messages, like the light of the stars and pressure of the 124 HOUSE_OVERSIGHT_016344

hand on the tiller, exist in a world of energy and forces, as does the helmsman. The weak forces that control the rudder are as real and physical as the strong forces that toss the ship. If we shift our cybernetics perspective from the ship to the helmsman, the pressures on the rudder become a strong force of muscles controlled by the weak signals in the mind of the helmsman. These messages in the helmsman’s mind are amplified into a physical force strong enough to steer the ship. Or instead, we can zoom out and take a large cybernetics perspective. We might see the ship itself as part of a vast trade network, part of a feedback loop that regulates the price of commodities through the flow of goods. In this perspective, the tiny ship is merely a messenger. So, the distinction between the physical world and the information world is a way to describe the relationship between the weak and the strong. Wiener chose to view the world from the vantage point and scale of the individual human. As acyberneticist, he took the perspective of the weak protagonist embedded within a strong system, trying to make the best of limited powers. He incorporated this perspective in his very definition of information. “Information,” he said, “is a name for the content of what is exchanged with the outer world as we adjust to it, and make our adjustment felt upon it.” In his words, information is what we use to “live effectively within that environment.” *> For Wiener, information is a way for the weak to effectively cope with the strong. This viewpoint is also reflected in Gregory Bateson’s definition of information as “a difference that makes a difference,” by which he meant the small difference that makes a big difference. The goal of cybernetics was to create a tiny model of the system using “weak currents” to amplify and control “strong currents” of the real world. The central insight was that a control problem could be solved by building an analogous system in the information space of messages and then amplifying solutions into the larger world of reality. Inherent in the motion of a control system is the concept of amplification, which makes the small big and the weak strong. Amplification allows the difference that makes a difference to make a difference. In this way of looking at the world, a control system needed to be as complex as the system it controlled. Cyberneticist W. Ross Ashby proved that this was true in a precise mathematical sense, in what is now called Ashby’s Law of Requisite Variety, or sometimes the First Law of Cybernetics. The law tells us that to control a system completely, the controller must be as complex as the controlled. Thus cyberneticists tended to see control systems as a kind of analog of the systems they governed, like the homunculus—the hypothetical little person inside the brain who controls the actual person. This notion of analogous structure is sometimes confused with the notion of analog encoding of messages, but the two are logically distinct. Norbert Wiener was much impressed with Vannevar Bush’s Digital Differential Analyzer, which could be reconfigured to match the structure of whatever problem it was given to solve but used digital signal encoding. Signals could be simplified to openly represent the relevant distinctions, allowing them to be more accurately communicated and stored. In digital signals, one needed only to preserve the difference in signals that made a difference. It is this distinction and signal coding that we commonly use to distinguish “analog” versus “digital.” Digital signal encoding was entirely compatible with cybernetic thinking—in 33 The Human Use of Human Beings (Boston: Houghton Mifflin, 1954), p. 17-18. 125 HOUSE_OVERSIGHT_016345

fact, enabling to it. What was constraining to cybernetics was the presumption of an analogy of structure between the controller and the controlled. By the 1930s, Kurt Gédel, Alonzo Church, and Alan Turing had all described universal systems of computation, in which the computation required no structural analogy to functions that were computed. These universal computers could also compute the functions of control. The analogy of structure between the controller and the controlled was central to the cybernetic perspective. Just as digital coding collapses the space of possible messages into a simplified version that represents only the difference that makes a difference, so the control system collapses the state space of a controlled system into a simplified model that reflects only the goals of the controller. Ashby’s Law does not imply that every controller must model every state of the system but only those states that matter for advancing the controller’s goals. Thus, in cybernetics, the goal of the controller becomes the perspective from which the world 1s viewed. Norbert Wiener adopted the perspective of the individual human relating to vast organizations and trying to “live effectively within that environment.” He took the perspective of the weak trying to influence the strong. Perhaps this is why he was able to notice the emergent goals of the “machines of flesh and blood” and anticipate some of the human challenges posed by these new intelligences, hybrid machine intelligences with goals of their own. 126 HOUSE_OVERSIGHT_016346

Venki Ramakrishnan is a Nobel Prize-winning biologist whose many scientific contributions include his work on the atomic structure of the ribosome—in effect, a huge molecular machine that reads our genes and makes proteins. His work would have been impossible without powerful computers. The Internet made his own work a lot easier and, he notes, acted as a leveler internationally: “When I grew up in India, if you wanted to get a book, it would show up six months or a year after it had already come out in the West. ... Journals would arrive by surface mail a few months later. I didn’t have to deal with it, because I left India when I was nineteen, but I know Indian scientists had to deal with it. Today they have access to information at the click of a button. More important, they have access to lectures. They can listen to Richard Feynman. That would have been a dream of mine when I was growing up. They can just watch Richard Feynman on the Web. That’s a big leveling in the field.” And yet... “Along with the benefits [of the Web], there is now a huge amount of noise. You have all of these people spouting pseudoscientific jargon and pushing their own ideas as if they were science.” As president of the Royal Society, Venki worries, too, about the broader issue of trust: public trust in evidence-based scientific findings, but also trust among scientists, bolstered by rigorous checking of one another ’s conclusions—trust that is in danger of eroding because of the “black box” character of deep-learning computers. “This [erosion] is going to happen more and more, as data sets get bigger, as we have genome- wide studies, population studies, and all sorts of things,” he says. “How do we, as a science community, grapple with this and communicate to the public a sense of what science is about, what is reliable in science, what is uncertain in science, and what is just plain wrong in science?” 127 HOUSE_OVERSIGHT_016347

WILL COMPUTERS BECOME OUR OVERLORDS? Venki Ramakrishnan Venki Ramakrishnan is a scientist at the Medical Research Council Laboratory of Molecular Biology, Cambridge University; recipient of the Nobel Prize in Chemistry (2009); current president of the Royal Society; and the author of Gene Machine: The Race to Discover the Secrets of the Ribosome. A former colleague of mine, Gérard Bricogne, used to joke that carbon-based intelligence was simply a catalyst for the evolution of silicon-based intelligence. For quite a long time, both Hollywood movies and scientific Jeremiahs have been predicting our eventual capitulation to our computer overlords. We all await the singularity, which always seems to be just over the horizon. In a sense, computers have already taken over, facilitating virtually every aspect of our lives—from banking, travel, and utilities to the most intimate personal communication. I can see and talk to my grandson in New York for free. I remember when I first saw the 1968 movie 200/: A Space Odyssey, the audience laughed at the absurdly cheap cost of a picturephone call from space: $1.70, at a time when a long- distance call within the U.S. was $3 per minute. However, the convenience and power of computers is also something of a Faustian bargain, for it comes with a loss of control. Computers prevent us from doing things we want. Try getting on a flight if you arrive at the airport and the airline computer systems are down, as happened not so long ago to British Airways at Heathrow. The planes, pilots, and passengers were all there; even the air-traffic controls were working. But no flights for that airline were allowed to take off. Computers also make us do things we don’t want—by generating mailing lists and print labels to send us all millions of pieces of unwanted mail, which we humans have to sort, deliver, and dispose of. But you ain’t seen nothing yet. In the past, we programmed computers using algorithms we understood at least in principle. So when machines did amazing things like beating world chess champion Garry Kasparov, we could say that the victorious programs were designed with algorithms based on our own understanding—using, in this instance, the experience and advice of top grandmasters. Machines were simply faster at doing brute-force calculations, had prodigious amounts of memory, and were not prone to errors. One article described Deep Blue’s victory not as that of a computer, which was just a dumb machine, but as the victory of hundreds of programmers over Kasparov, a single individual. That way of programming is changing dramatically. After a long hiatus, the power of machine learning has taken off. Much of the change came when programmers, rather than trying to anticipate and code for every possible contingency, allowed computers to train themselves on data, using deep neural networks based on models of how our own brains learn. They use probabilistic methods to “learn” from large quantities of data; computers can recognize patterns and come up with conclusions on their own. A particularly powerful method is called reinforcement learning, by which the computer learns, without prior input, which variables are important and how much to 128 HOUSE_OVERSIGHT_016348

weight them to reach a certain goal. This method in some sense mimics how we learn as children. The results from these new approaches are amazing. Such a deep-learning program was used to teach a computer to play Go, a game that only a few years ago was thought to be beyond the reach of AI because it was so hard to calculate how well you were doing. It seemed that top Go players relied a great deal on intuition and a feel for position, so proficiency was thought to require a particularly human kind of intelligence. But the AlphaGo program produced by DeepMind, after being trained on thousands of high-level Go games played by humans and then millions of games with itself, was able to beat the top human players in short order. Even more amazingly, the related AlphaGo Zero program, which learned from scratch by playing itself, was stronger than the version trained initially on human games! It was as though the humans had been preventing the computer from reaching its true potential. The same method has recently been generalized: Starting from scratch, within just twenty-four hours, an equivalent AlphaZero chess program was able to beat today’s top “conventional” chess programs, which in turn have beaten the best humans. Progress has not been restricted to games. Computers are significantly better at image and voice recognition and speech synthesis than they used to be. They can detect tumors in radiographs earlier than most humans. Medical diagnostics and personalized medicine will improve substantially. Transportation by self-driving cars will keep us all safer, on average. My grandson may never have to acquire a driver’s license, because driving a car will be like riding a horse today—a hobby for the few. Dangerous activities, such as mining, and tedious repetitive work will be done by computers. Governments will offer better targeted, more personalized and efficient public services. AI could revolutionize education by analyzing an individual pupil’s needs and enabling customized teaching, so that each student can advance at an optimal rate. Along with these huge benefits, of course, will come alarming risks. With the vast amounts of personal data, computers will learn more about us than we may know about ourselves; the question of who owns data about us will be paramount. Moreover, data-based decisions will undoubtedly reflect social biases: Even an allegedly neutral intelligent system designed to predict loan risks, say, may conclude that mere membership in a particular minority group makes you more likely to default on a loan. While this is an obvious example that we could correct, the real danger is that we are not always aware of biases in the data and may simply perpetuate them. Machine learning may also perpetuate our own biases. When Netflix or Amazon tries to tell you what you might want to watch or buy, this is an application of machine learning. Currently such suggestions are sometimes laughable, but with time and more data they will get increasingly accurate, reinforcing our prejudices and likes and dislikes. Will we miss out on the random encounter that might persuade us to change our views by exposing us to new and conflicting ideas? Social media, given its influence on elections, is a particularly striking illustration of how the divide between people on different sides of the political spectrum can be accentuated. We may have already reached the stage where most governments are powerless to resist the combined clout of a few powerful multinational companies that control us and our digital future. The fight between dominant companies today is really a fight for control over our data. They will use their enormous influence to prevent regulation of data, because their interests lie in unfettered control of it. Moreover, they have the 129 HOUSE_OVERSIGHT_016349

financial resources to hire the most talented workers in the field, enhancing their power even further. We have been giving away valuable data for the sake of freebies like Gmail and Facebook, but as the journalist and author John Lanchester has pointed out in the London Review of Books, if it is free, then you are the product. Their real customers are the ones who pay them for access to knowledge about us, so that they can persuade us to buy their products or otherwise influence us. One way around the monopolistic control of data is to split the ownership of data away from firms that use them. Individuals would instead own and control access to their personal data (a model that would encourage competition, since people would be free to move their data to a company that offered better services). Finally, abuse of data is not limited to corporations: In totalitarian states, or even nominally democratic ones, governments know things about their citizens that Orwell could not have imagined. The use they make of this information may not always be transparent or possible to counter. The prospect of AI for military purposes is frightening. One can imagine intelligent systems being designed to act autonomously based on real-time data and able to act faster than the enemy, starting catastrophic wars. Such wars may not necessarily be conventional or even nuclear wars. Given how essential computer networks are to modern society, it is much more likely that AI wars will be fought in cyberspace. The consequences could be just as dire. Despite this loss of control, we continue to march inexorably into a world in which AI will be everywhere: Individuals won’t be able to resist its convenience and power, and corporations and governments won’t be able to resist its competitive advantages. But important questions arise about the future of work. Computers have been responsible for considerable losses in blue-collar jobs in the last few decades, but until recently many white-collar jobs—jobs that “only humans can do”—were thought to be safe. Suddenly that no longer appears to be true. Accountants, many legal and medical professionals, financial analysts and stockbrokers, travel agents—in fact, a large fraction of white-collar jobs—will disappear as a result of sophisticated machine-learning programs. We face a future in which factories churn out goods with very few employees and the movement of goods is largely automated, as are many services. What’s left for humans to do? In 1930—long before the advent of computers, let alone AI—John Maynard Keynes wrote, in an essay called “Economic Possibilities for our Grandchildren,” that as a result of improvements in productivity, society could produce all its needs with a fifteen-hour work week. He also predicted, along with the growth of creative leisure, the end of money and wealth as a goal: We shall be able to afford to dare to assess the money-motive at its true value. The love of money as a possession—as distinguished from the love of money as a means to the enjoyments and realities of life—will be recognised for what it is, a somewhat disgusting morbidity, one of those semi-criminal, semi-pathological propensities which one hands over with a shudder to the specialists in mental disease. 130 HOUSE_OVERSIGHT_016350

Sadly, Keynes’s predictions did not come true. Although productivity did indeed increase, the system—possibly inherent in a market economy—did not result in humans working much shorter hours. Rather, what happened is what the anthropologist and anarchist David Graeber describes as the growth of “bullshit jobs.”°4 While jobs that produce essentials like food, shelter, and goods have been largely automated away, we have seen an enormous expansion of sectors like corporate law, academic and health administration (as opposed to actual teaching, research, and the practice of medicine), “human resources,” and public relations, not to mention new industries like financial services and telemarketing and ancillary industries in the so-called gig economy which serve those who are too busy doing all that additional work. How will societies cope with technology’s increasingly rapid destruction of entire professions and throwing large numbers of people out of work? Some argue that this concern is based on a false premise, because new jobs spring up that didn’t exist before, but as Graeber points out, these new jobs won’t necessarily be rewarding or fulfilling. During the first industrial revolution, it took almost a century before most people were better off. That revolution was possible only because the government of the time ruthlessly favored property rights over labor, and most people (and all women) did not have the vote. In today’s democratic societies, it is not clear that the population will tolerate such a dramatic upheaval of society based on the promise that “eventually” things will get better. Even that rosy vision will depend on a radical shake-up of education and lifelong learning. The Industrial Revolution did trigger enormous social change of this kind, including a shift to universal education. But it will not happen unless we make it happen: This is essentially about power, agency, and control. What’s next for, say, the forty-year- old taxi driver or truck driver in an era of autonomous vehicles? One idea that has been touted is that of a universal basic income, which will allow citizens to pursue their interests, retrain for new occupations, and generally be free to live a decent life. However, market economies, which are predicated on growing consumer demand over all else, may not tolerate this innovation. There is also a feeling among many that meaningful work is essential to human dignity and fulfillment. So another possibility is that the enormous wealth generated by increased productivity due to automation could be redistributed to jobs requiring human labor and creativity in fields such as the arts, music, social work, and other worthwhile pursuits. Ultimately, which jobs are rewarding or productive and which are “bullshit” is a matter of judgment and may vary from society to society, as well as over time. So far, ’ve focused on AI’s practical consequences. As a scientist, what bothers me is our potential loss of understanding. We are now accumulating data at an incredible rate. In my own lab, an experiment generates over a terabyte of data a day. These data are massaged, analyzed, and reduced until there is an interpretable result. But in all of this data analysis, we believe we know what’s happening. We know what the programs are 34 hitps://strikemag. org/bullshit-jobs/ 131 HOUSE_OVERSIGHT_016351

doing because we designed the algorithms at their heart. So when our computers generate a result, we feel that we intellectually grasp it. The new machine-learning programs are different. Having recognized patterns via deep neural networks, they come up with conclusions, and we have no idea exactly how. When they uncover relationships, we don’t understand it in the same way as if we had deduced those relationships ourselves using an underlying theoretical framework. As data sets become larger, we won’t be able to analyze them ourselves even with the help of computers; rather, we will rely entirely on computers to do the analysis for us. So if someone asks us how we know something, we will simply say it is because the machine analyzed the data and produced the conclusion. One day a computer may well come up with an entirely new result—e.g., a mathematical theorem whose proof, or even whose statement, no human can understand. That is philosophically different from the way we have been doing science. Or at least thought we had; some might argue that we don’t know how our own brains reach conclusions either, and that these new methods are a way of mimicking learning by the human brain. Nevertheless, I find this potential loss of understanding disturbing. Despite the remarkable advances in computing, the hype about AGI—a general- intelligence machine that will think like a human and possibly develop consciousness— smacks of science fiction to me, partly because we don’t understand the brain at that level of detail. Not only do we not understand what consciousness is, we don’t even understand a relatively simple problem like how we remember a phone number. In just that one question, there are all sorts of things to consider. How do we know it is a number? How do we associate it with a person, a name, face, and other characteristics? Even such seemingly trivial questions involve everything from high-level cognition and memory to how a cell stores information and how neurons interact. Moreover, that’s just one task among many that the brain does effortlessly. Whereas machines will no doubt do ever more amazing things, they’re unlikely to be a replacement for human thought and human creativity and vision. Eric Schmidt, former chairman of Google’s parent company, said in a recent interview at the London Science Museum that even designing a robot that would clear the table, wash the dishes, and put them away was a huge challenge. The calculations involved in figuring out all the movements the body has to make to throw a ball accurately or do slalom skiing are prodigious. The brain can do all these and also do mathematics and music, and invent games like chess and Go, not just play them. We tend to underestimate the complexity and creativity of the human brain and how amazingly general it is. If AI is to become more humanlike in its abilities, the machine-learning and neuroscience communities need to interact closely, something that is happening already. Some of today’s greatest exponents of machine learning—such as Geoffrey Hinton, Zoubin Ghahramani, and Demis Hassabis—have backgrounds in cognitive neuroscience, and their success has been at least in part due to attempts to model brainlike behavior in their algorithms. At the same time, neurobiology has also flourished. All sorts of tools have been developed to watch which neurons are firing and genetically manipulate them and see what’s happening in real time with inputs. Several countries have launched moon-shot neuroscience initiatives to see if we can crack the workings of the brain. Advances in AI and neuroscience seem to go hand in hand; each field can propel the other. 132 HOUSE_OVERSIGHT_016352

Many evolutionary scientists, and such philosophers as Daniel Dennett, have pointed out that the human brain is the result of billions of years of evolution.*°> Human intelligence is not the special characteristic we think it is, but just another survival mechanism not unlike our digestive or immune systems, both of which are also amazingly complex. Intelligence evolved because it allowed us to make sense of the world around us, to plan ahead, and thus cope with all sorts of unexpected things in order to survive. However, as Descartes stated, we humans define our very existence by our ability to think. So it is not surprising that, in an anthropomorphic way, our fears about AI reflect this belief that our intelligence is what makes us special. But if we step back and look at life on Earth, we see that we are far from the most resilient species. If we’re going to be taken over at some point, it will be by some of Earth’s oldest life-forms, like bacteria, which can live anywhere from Antarctica to deep- sea thermal vents hotter than boiling water, or in acid environments that would melt you and me. So when people ask where we’re headed, we need to put the question in a broader context. I don’t know what sort of future AI will bring: whether AI will make humans subservient or obsolete or will be a useful and welcome enhancement of our abilities which will enrich our lives. But I am reasonably certain that computers will never be the overlords of bacteria. 35 See, for example, Dennett’s From Bacteria to Bach and Back: The Evolution of Minds (New York: W. W. Norton, 2017). 133 HOUSE_OVERSIGHT_016353

Alex “Sandy” Pentland, an exponent of what he has termed “social physics,” is interested in building powerful human-Al ecologies. He is concerned at the same time about the potential dangers of decision-making systems in which the data in effect take over and human creativity is relegated to the background. The advent of Big Data, he believes, has given us the opportunity to reinvent our civilization: “We can now begin to actually look at the details of social interaction and how those play out, and we re no longer limited to averages like market indices or election results. This is an astounding change. The ability to see the details of the market, of political revolutions, and to be able to predict and control them is definitely a case of Promethean fire—it could be used for good or for ill. Big Data brings us to interesting times.” At our group meeting in Washington, Connecticut, he confessed that reading Norbert Wiener on the concept of feedback “felt like reading my own thoughts.” “After Wiener, people discovered or focused on the fact that there are genuinely chaotic systems that are just not predictable,” he said, “but if you look at human socioeconomic systems, there is a large percentage of variance you can account for and predict. ... Today there is data from all sorts of digital devices, and from all of our transactions. The fact that everything is datafied means you can measure things in real time in most aspects of human life—and increasingly in every aspect of human life. The fact that we have interesting computers and machine-learning techniques means that you can build predictive models of human systems in ways you could never do before.” 134 HOUSE_OVERSIGHT_016354

THE HUMAN STRATEGY Alex “Sandy” Pentland Alex “Sandy” Pentland is Toshiba Professor and professor of media arts and sciences, MIT; director of the Human Dynamics and Connection Science labs and the Media Lab Entrepreneurship Program, and the author of Social Physics. In the last half-century, the idea of AI and intelligent robots has dominated thinking about the relationship between humans and computers. In part, this is because it’s easy to tell the stories about AI and robots, and in part because of early successes (e.g., theorem provers that reproduced most of Whitehead and Russell’s Principia Mathematica) and massive military funding. The earlier and broader vision of cybernetics, which considered the artificial as part of larger systems of feedback and mutual influence, faded from public awareness. However, in the intervening years the cybernetics vision has slowly grown and quietly taken over—to the point where it is “in the air.” State-of-the-art research in most engineering disciplines is now framed as feedback systems that are dynamic and driven by energy flows. Even AI is being recast as human/machine “advisor” systems, and the military is beginning large-scale funding in this area—something that should perhaps worry us more than drones and independent humanoid robots. But as science and engineering have adopted a more cybernetics-like stance, it has become clear that even the vision of cybernetics is far too small. It was originally centered on the embeddedness of the individual actor but not on the emergent properties of a network of actors. This is unsurprising, because the mathematics of networks did not exist until recently, so a quantitative science of how networks behave was impossible. We now know that study of the individual does not produce understanding of the system except in certain simple cases. Recent progress in this area was foreshadowed by understanding that “chaos,” and later “complexity,” were the typical behavior of systems, but we can now go far beyond these statistical understandings. We’re beginning to be able to analyze, predict, and even design the emergent behavior of complex heterogeneous networks. The cybernetics view of the connected individual actor can now be expanded to cover complex systems of connected individuals and machines, and the insights we obtain from this broader view are fundamentally different from those obtained from the cybernetics view. Thinking about the network is analogous to thinking about entire ecosystems. How would you guide ecosystems to grow in a good direction? What do you even mean by “a good direction”? Questions like this are beyond the boundary of traditional cybernetic thinking. Perhaps the most stunning realization is that humans are already beginning to use AI and machine learning to guide entire ecosystems, including ecosystems of people, thus creating human-AI ecologies. Now that everything is becoming “datafied,” we can measure most aspects of human life and, increasingly, aspects of all life. This, together with new, powerful machine-learning techniques, means that we can build models of these ecologies in ways we couldn’t before. Well-known examples are weather- and traffic-prediction models, which are being extended to predict the global climate and plan city growth and renewal. AlI-aided engineering of the ecologies is already here. 135 HOUSE_OVERSIGHT_016355

Development of human-AI ecosystems is perhaps inevitable for a social species such as ourselves. We became social early in our evolution, millions of years ago. We began exchanging information with one another to stay alive, to increase our fitness. We developed writing to share abstract and complex ideas, and most recently we’ve developed computers to enhance our communication abilities. Now we’re developing AI and machine-learning models of ecosystems and sharing the predictions of those models to jointly shape our world through new laws and international agreements. We live in an unprecedented historic moment, in which the availability of vast amounts of human behavioral data and advances in machine learning enable us to tackle complex social problems through algorithmic decision making. The opportunities for such a human-AI ecology to have positive social impact through fairer and more transparent decisions are obvious. But there are also risks of a “tyranny of algorithms,” where unelected data experts are running the world. The choices we make now are perhaps even more momentous than those we faced in the 1950s, when AI and cybernetics were created. The issues look similar, but they’re not. We have moved down the road, and now the scope is larger. It’s not just AI robots versus individuals. It’s AI guiding entire ecologies. How can we make a good human-artificial ecosystem, something that’s not a machine society but a cyberculture in which we can all live as homans—a culture with a human feel to it? We don’t want to think small—for example, to talk only of robots and self- driving cars. We want this to be a global ecology. Think Skynet-size. But how would you make Skynet something that’s about the human fabric? The first thing to ask is: What’s the magic that makes the current AI work? Where is it wrong and where 1s it right? The good magic is that it has something called the credit-assignment function. What that lets you do is take “stupid neurons’”—little linear functions—and figure out, in a big network, which ones are doing the work and strengthen them. It’s a way of taking a random bunch of switches all hooked together in a network and making them smart by giving them feedback about what works and what doesn’t. This sounds simple, but there’s some complicated math around it. That’s the magic that makes current AI work. The bad part of it is, because those little neurons are stupid, the things they learn don’t generalize very well. If an AI sees something it hasn’t seen before, or if the world changes a little bit, the AI is likely to make a horrible mistake. It has absolutely no sense of context. In some ways, it’s as far from Norbert Wiener’s original notion of cybernetics as you can get, because it isn’t contextualized; it’s a little idiot savant. But imagine that you took away those limitations: Imagine that instead of using dumb neurons, you used neurons in which real-world knowledge was embedded. Maybe instead of linear neurons, you used neurons that were functions in physics, and then you tried to fit physics data. Or maybe you put in a lot of knowledge about humans and how they interact with one another—the statistics and characteristics of humans. When you add this background knowledge and surround it with a good credit- assignment function, then you can take observational data and use the credit-assignment function to reinforce the functions that are producing good answers. The result is an AI that works extremely well and can generalize. For instance, in solving physical 136 HOUSE_OVERSIGHT_016356

problems, it often takes only a couple of noisy data points to get something that’s a beautiful description of a phenomenon, because you’re putting in knowledge about how physics works. That’s in huge contrast to normal AI, which requires millions of training examples and is very sensitive to noise. By adding the appropriate background knowledge, you get much more intelligence. Similar to the physical-systems case, if we make neurons that know a lot about how humans learn from each other, then we can detect human fads and predict human behavior trends in surprisingly accurate and efficient ways. This “social physics” works because human behavior is determined as much by the patterns of our culture as by rational, individual thinking. These patterns can be described mathematically and employed to make accurate predictions. This idea of a credit-assignment function reinforcing connections between neurons that are doing the best work is the core of current AI. If you make those little neurons smarter, the AI gets smarter. So, what would happen if we replaced the neurons with people? People have lots of capabilities. They know lots of things about the world; they can perceive things in a broadly competent, human way. What would happen if you had a network of people in which you could reinforce the connections that were helping and minimize the connections that weren’t? That begins to sound like a society, or a company. We all live in a human social network. We’re reinforced for doing things that seem to help everybody and discouraged from doing things that are not appreciated. Culture is the result of this sort of human AI as applied to human problems; it is the process of building social structures by reinforcing the good connections and penalizing the bad. Once you’ve realized you can take this general AI framework and create a human AI, the question becomes, What’s the right way to do that? Is it a safe idea? Is it completely crazy? My students and I are looking at how people make decisions, on huge databases of financial decisions, business decisions, and many other sorts of decisions. What we’ve found is that humans often make decisions in a way that mimics AI credit-assignment algorithms and works to make the community smarter. A particularly interesting feature of this work is that it addresses a classic problem in evolution known as the group- selection problem. The core of this problem is: How can we select for culture in evolution, when it’s the individuals that reproduce? What you need is something that selects for the best cultures and the best groups but also selects for the best individuals, because they’ re the units that transmit the genes. When you frame the question this way and go through the mathematical literature, you discover that there’s one generally best way to do this. It’s called “distributed Thompson sampling,” a mathematical algorithm used in choosing, out of a set of possible actions with unknown payoffs, the action that maximizes the expected reward in respect to the actions. The key is social sampling, a way of combining evidence, of exploring and exploiting at the same time. It has the unusual property of simultaneously being the best strategy both for the individual and for the group. If you use the group as the basis of selection, and then the group either gets wiped out or reinforced, you’re also selecting for successful individuals. If you select for individuals, and each individual does what’s good for him or her, then that’s automatically the best thing for the group. It’s an amazing alignment of interests and utilities, and it provides real insight into the question of how culture fits into natural selection. 137 HOUSE_OVERSIGHT_016357

Social sampling, very simply, is looking around you at the actions of people who are like you, finding what’s popular, and then copying it if it seems like a good idea to you. Idea propagation has this popularity function driving it, but individual adoption also is about figuring out how the idea works for the individual—a reflective attitude. When you combine social sampling and personal judgment, you get superior decision making. That’s amazing, because now we have a mathematical recipe for doing with humans what all those AI techniques are doing with dumb computer neurons. We have a way of putting people together to make better decisions, given more and more experience. So, what happens in the real world? Why don’t we do this all the time? Well, people are good at it, but there are ways it can run amok. One of these is through advertising, propaganda, or “fake news.” There are many ways to get people to think something is popular when it’s not, and this destroys the usefulness of social sampling. The way you can make groups of people smarter, the way you can make human ATI, will work only if you can get feedback to them that’s truthful. It must be grounded on whether each person’s actions worked for them or not. That’s the key to AI mechanisms, too. What they do is analyze whether they performed correctly. If so, plus one; if not, minus one. We need that truthful feedback to make this human mechanism work well, and we need good ways of knowing about what other people are doing so that we can correctly assess popularity and the likelihood of this being a good choice. The next step is to build this credit-assignment function, this feedback function, for people, so that we can make a good human-artificial ecosystem—a smart organization and a smart culture. In a way, we need to duplicate some of the early insights that resulted in, for instance, the U.S. census—trying to find basic facts that everybody can agree on and understand so that the transmission of knowledge and culture can happen in a way that’s truthful and social sampling can function efficiently. We can address the problem of building an accurate credit-assignment function in many different settings. In companies, for instance, it can be done with digital ID badges that reveal who’s connected to whom, so that we can assess the pattern of connections in relation to the company’s results on a daily or weekly basis. The credit-assignment function asks whether those connections helped solve problems, or helped invent new solutions, and reinforces the helpful connections. When you can get that feedback quantitatively—which is difficult, because most things aren’t measured quantitatively— both the productivity and the innovation rate within the organization can be significantly improved. This is, for instance, the basis of Toyota’s “continuous improvement” method. A next step is to try to do the same thing but at scale, something I refer to as building a trust network for data. It can be thought of as a distributed system like the Internet, but with the ability to quantitatively measure and communicate the qualities of human society, in the same way that the U.S. census does a pretty good job of telling us about population and life expectancy. We are already deploying prototype examples of trust networks at scale in several countries, based on the data and measurement standards laid out in the U.N. Sustainable Development Goals. On the horizon is a vision of how we can make humanity more intelligent by building a human AI. It’s a vision composed of two threads. One is data that we can all trust—data that have been vetted by a broad community, data where the algorithms are known and monitored, much like the census data we all automatically rely on as at least 138 HOUSE_OVERSIGHT_016358

approximately correct. The other is a fair, data-driven assessment of public norms, policy, and government, based on trusted data about current conditions. This second thread depends on availability of trusted data and so is just beginning to be developed. Trusted data and data-driven assessment of norms, policy, and government together create a credit-assignment function that improves societies’ overall fitness and intelligence. It is precisely at the point of creating greater societal intelligence where fake news, propaganda, and advertising all get in the way. Fortunately, trust networks give us a path forward to building a society more resistant to echo-chamber problems, these fads, these exercises in madness. We have begun to develop a new way of establishing social measurements, in aid of curing some of the ills we see in society today. We’re using open data from all sources, encouraging a fair representation of the things people are choosing, in a curated mathematical framework that can stamp out the echoes and the attempts to manipulate us. On Polarization and Inequality Extreme polarization and segregation by income are almost everywhere in the world today and threaten to tear governments and civil society apart. Increasingly, the media are becoming adrenaline pushers driven by advertising clicks and failing to deliver balanced facts and reasoned discourse—and the degradation of media is causing people to lose their bearings. They don’t know what to believe, and thus they can easily be manipulated. There is a real need to ground our various cultures in trustworthy, data- driven standards that we all agree on, and to be able to know what behaviors and policies work and which don’t. In converting to a digital society, we’ve lost touch with traditional notions of truth and justice. Justice used to be mostly informal and normative. We’ve now formalized it. At the same time, we’ve put it out of reach for most people. Our legal systems are failing us in a way they didn’t before, precisely because they’re now more formal, more digital, less embedded in society. Ideas about justice are very different around the world. One of the core differentiators is this: Do you or your parents remember when the bad guys came with guns and took everything? If you do, your attitude about justice is different from that of the average reader of this essay. Do you come from the upper classes? Or were you somebody who saw the sewers from the inside? Your view of justice depends on your history. A common test I have for U.S. citizens is this: Do you know anybody who owns a pickup truck? It’s the number-one-selling vehicle in the United States, and if you don’t know people like that, you’re out of touch with more than 50 percent of Americans. Physical segregation drives conceptual segregation. Most of America thinks of justice and access and fairness in terms very different from those of the typical, say, Manhattanite. If you look at patterns of mobility—where people go—in a typical city, you find that the people in the top quintile (white-collar working families) and bottom quintile (people who are sometimes on unemployment or welfare) almost never talk to each other. They don’t go to the same places; they don’t talk about the same things. They all live in 139 HOUSE_OVERSIGHT_016359

the same city, nominally, but it’s as if it were two completely different cities—and this is perhaps the most important cause of today’s plague of polarization. On Extreme Wealth Some two hundred of the world’s wealthiest people have pledged to give away more than 50 percent of their wealth either during their lifetimes or in their wills, creating a plurality of voices in the foundation space.*° Bill Gates is probably the most familiar example. He’s decided that if the government won’t do it, he’ll do it. You want mosquito nets? He’ll do it. You want antivirals? He’ll doit. We’re getting different stakeholders to take action in the form of foundations dedicated to public good, and they have different versions of what they consider the public good. This diversity of goals has created a lot of what’s wonderful about the world today. Actions from outside government by organizations like the Ford Foundation and the Sloan Foundation, who bet on things that nobody else would bet on, have changed the world for the better. Sure, these billionaires are human, with human foibles, and all is not necessarily as it should be. On the other hand, the same situation obtained when the railways were first built. Some people made huge fortunes. A lot of people went bust. We, the average people, got railways out of it. That’s good. Same thing with electric power; same thing with many new technologies. There’s a churning process that throws somebody up and later casts them or their heirs down. Bubbles of extreme wealth were a feature of the late 1800s and early 1900s when steam engines and railways and electric lights were invented. The fortunes they created were all gone within two or three generations. If the U.S. were like Europe, I would worry. What you find in Europe is that the same families have held on to wealth for hundreds of years, so they’re entrenched not just in terms of wealth but of the political system and in other ways. But so far, the U.S. has avoided this kind of hereditary class system. Extreme wealth hasn’t stuck, which is good. It shouldn’t stick. If you win the lottery, you get your billion dollars, but your grandkids ought to work for a living. On AI and Society People are scared about AI. Perhaps they should be. But they need to realize that AI feeds on data. Without data, AI is nothing. You don’t have to watch the AI; instead you should watch what it eats and what it does. The trust-network framework we’ ve set up, with the help of nations in the E.U. and elsewhere, is one where we can have our algorithms, we can have our AI, but we get to see what went in and what went out, so that we can ask, Is this a discriminatory decision? Is this the sort of thing that we want as humans? Or is this something that’s a little weird? The most revealing analogy is that regulators, bureaucracies, and parts of the government are very much like Als: They take in the rules that we call law and regulation, and they add government data, and they make decisions that affect our lives. The part that’s bad about the current system is that we have very little oversight of these departments, regulators, and bureaucracies. The only control we have is the vote—the opportunity to elect somebody different. We need to make oversight of bureaucracies a lot more fine-grained. We need to record the data that went into every single decision 36 https://givingpledge.org/About.aspx. 140 HOUSE_OVERSIGHT_016360

and have the results analyzed by the various stakeholders—trather like elected legislatures were originally intended to do. If we have the data that go into and out of each decision, we can easily ask, Is this a fair algorithm? Is this AI doing things that we as humans believe are ethical? This human-in-the-loop approach is called “open algorithms;” you get to see what the Als take as input and what they decide using that input. If you see those two things, you’ll know whether they’ re doing the right thing or the wrong thing. It turns out that’s not hard to do. If you control the data, then you control the AI. One thing people often fail to mention is that all the worries about AI are the same as the worries about today’s government. For most parts of the government—the justice system, et cetera—there’s no reliable data about what they’re doing and in what situation. How can you know whether the courts are fair or not if you don’t know the inputs and the outputs? The same problem arises with AI systems and is addressable in the same way. We need trusted data to hold current government to account in terms of what they take in and what they put out, and AI should be no different. Next-Generation AI Current AI machine-learning algorithms are, at their core, dead simple stupid. They work, but they work by brute force, so they need hundreds of millions of samples. They work because you can approximate anything with lots of little simple pieces. That’s a key insight of current AI research—that if you use reinforcement learning for credit- assignment feedback, you can get those little pieces to approximate whatever arbitrary function you want. But using the wrong functions to make decisions means the AI’s ability to make good decisions won’t generalize. If we give the AI new, different inputs, it may make completely unreasonable decisions. Or if the situation changes, then you need to retrain it. There are amusing techniques to find the “null space” in these AI systems. These are inputs that the AI thinks are valid examples of what it was trained to recognize (e.g., faces, cats, etc.), but to a human they’re crazy examples. Current AI is doing descriptive statistics in a way that’s not science and would be almost impossible to make into science. To build robust systems, we need to know the science behind data. The systems I view as next-generation Als result from this science- based approach: If you’re going to create an AI to deal with something physical, then you should build the laws of physics into it as your descriptive functions, in place of those stupid little neurons. For instance, we know that physics uses functions like polynomials, sine waves, and exponentials, so those should be your basis functions and not little linear neurons. By using those more appropriate basis functions, you need a lot less data, you can deal with a lot more noise, and you get much better results. As in the physics example, if we want to build an AI to work with human behavior, then we need to build the statistical properties of human networks into machine-learning algorithms. When you replace the stupid neurons with ones that capture the basics of human behavior, then you can identify trends with very little data, and you can deal with huge levels of noise. The fact that humans have a “commonsense” understanding that they bring to most problems suggests what I call the human strategy: Human society is a network just like the neural nets trained for deep learning, but the “neurons” in human society are a lot 141 HOUSE_OVERSIGHT_016361

smarter. You and I have surprisingly general descriptive powers that we use for understanding a wide range of situations, and we can recognize which connections should be reinforced. That means we can shape our social networks to work much better and potentially beat all that machine-based AI at its own game. 142 HOUSE_OVERSIGHT_016362

“URGENT!” URGENT!” the cc’d copy of an email screamed, one of a dozen emails that greeted me as I turned on my phone at the baggage carousel at Malpensa Airport after the long flight from JFK. “The great American visionary thinker John Brockman arrives this morning at Grand Hotel Milan. You MUST, repeat MUST pay him a visit.” It was signed HUO. The prior evening, waiting in the lounge at JFK, I had had the bright idea to write my friend and longtime collaborator, the London-based, peripatetic art curator Hans Ulrich Obrist (known to all as HUO), and ask if there was anyone in Milan I should know. Once I was settled at the hotel, the phone began ringing and a procession of leading Italian artists, designers, and architects called to request a meeting, including Enzo Mari, the modernist artist and furniture designer; Alberto Garutti, whose aesthetic strategies have inspired a dialogue between contemporary art, spectator, and public space; and fashion designer Miuccia Prada, who “requests your presence for tea this afternoon at Prada headquarters.” And thus, thanks to HUO, did the jet-lagged “great American visionary thinker” stumble and mumble his way through his first day in Milan, November 2011. HUO is sui generis: He lives a twenty-four-hour day, sleeping (I guess) whenever, and employing full-time assistants who work eight-hour shifts and are available to him 24/7. Over a recent two-year period, he visited art venues in either China or India for forty weekends each year—departing London Thursday evening, back at his desk on Monday. Last year, once again, ArtReview ranked him #1 on their annual “Power 100” list. Recently we collaborated on a panel during the “GUEST, GHOST, HOST: MACHINE!” Serpentine event that took place at London’s new City Hall. We were joined by Venki Ramakrishnan, Jaan Tallinn, and Andrew Blake, research director of The Alan Turing Institute. The event was consistent with HUO’s mission of bringing together art and science: “The curator is no longer understood simply as the person who fills a space with objects,” he says, “but also as the person who brings different cultural spheres into contact, invents new display features, and makes junctions that allow unexpected encounters and results.” 143 HOUSE_OVERSIGHT_016363

MAKING THE INVISIBLE VISIBLE: ART MEETS AI Hans Ulrich Obrist Hans Ulrich Obrist is artistic director of the Serpentine Gallery, London, and the author of Ways of Curating and Lives of the Artists, Lives of the Architects. In the Introduction to the second edition of his book Understanding Media, Marshall McLuhan noted the ability of art to “anticipate future social and technological developments.” Art is “an early alarm system,” pointing us to new developments in times ahead and allowing us “to prepare to cope with them... . Art as a radar environment takes on the function of indispensable perceptual training. . . .” In 1964, when McLuhan’s book was first published, the artist Nam June Paik was just building his Robot K-456 to experiment with the technologies that subsequently would start to influence society. He had worked with television earlier, challenging its usual passive consumption by the viewer, and later made art with global live-satellite broadcasts, using the new media less for entertainment than to point us to their poetic and intercultural capacities (which are still mostly unused today). The Paiks of our time, of course, are now working with the Internet, digital images, and artificial intelligence. Their works and thoughts, again, are an early alarm system for the developments ahead of us. As a curator, my daily work is to bring together different works of art and connect different cultures. Since the early 1990s, I have also been organizing conversations and meetings with practitioners from different disciplines, in order to go beyond the general reluctance to pool knowledge. Since I was interested in hearing what artists have to say about artificial intelligence, I recently organized several conversations between artists and engineers. The reason to look closely at AI is that two of the most important questions of today are “How capable will AI become?” and “What dangers may arise from it?” Its early applications already influence our everyday lives in ways that are more or less recognizable. There is an increasing impact on many aspects of our society, but whether this might be, in general, beneficial or malign is still uncertain. Many contemporary artists are following these developments closely. They are articulating various doubts about the promises of AI and reminding us not to associate the term “artificial intelligence” solely with positive outcomes. To the current discussions of AI, the artists contribute their specific perspectives and notably their focus on questions of image making, creativity, and the use of programming as artistic tools. The deep connections between science and art had already been noted by the late Heinz von Foerster, one of the architects of cybernetics, who worked with Norbert Wiener from the mid-1940s and in the 1960s founded the field of second-order cybernetics, in which the observer is understood as part of the system itself and not an external entity. I knew von Foerster well, and in one of our many conversations, he offered his views on the relation between art and science: I’ve always perceived art and science as complementary fields. One shouldn’t forget that a scientist is in some respects also an artist. He invents anew technique and he describes it. He uses language like a poet, or the author of a detective novel, and describes his findings. In my view, a scientist must work in 144 HOUSE_OVERSIGHT_016364

an artistic way if he wants to communicate his research. He obviously wants to communicate and talk to others. A scientist invents new objects, and the question is how to describe them. In all of these aspects, science is not very different from art. When I asked him how he defined cybernetics, von Foerster answered: The substance of what we have learned from cybemetics is to think in circles: A leads B, B to C, but C can return to A. Such kinds of arguments are not linear but circular. The significant contribution of cybernetics to our thinking is to accept circular arguments. This means that we have to look at circular processes and understand under which circumstances an equilibrium, and thus a stable structure, emerges. Today, where AI algorithms are applied in daily tasks, one can ask how the human factor is included in these kinds of processes and what role creativity and art could play in relation to them. There are thus different levels to think about when exploring the relation between AI and art. So, what do contemporary artists have to say about artificial intelligence? Artificial Stupidity Hito Steyerl, an artist who works with documentary and experimental film, considers two key aspects that we should keep in mind when reflecting on the implications of AI for society. First, the expectations for so-called artificial intelligence, she says, are often overrated, and the noun “intelligence” is misleading; to counter that, she uses the term “artificial stupidity.” Second, she points out that programmers are now making invisible software algorithms visible through images, but to understand and interpret these images better, we should apply the expertise of artists. Steyerl has worked with computer technology for many years, and her recent artworks have explored surveillance techniques, robots, and such computer games as in How Not to Be Seen (2013), on digital-image technologies, or HellYeahWeFuckDie (2017), about the training of robots in the still-difficult task of keeping balance. But to explain her notion of artificial stupidity, Steyerl refers to a more general phenomenon, like the now widespread use of Twitter bots, noting in our conversation: It was and still is a very popular tool in elections to deploy Twitter armies to sway public opinion and deflect popular hashtags and so on. This is an artificial intelligence of a very, very low grade. It’s two or maybe three lines of script. It’s nothing very sophisticated at all. Yet the social implications of this kind of artificial stupidity, as I call it, are already monumental in global politics. As has been widely noted, this kind of technology was seen in the many automated Twitter posts before the 2016 U.S. presidential election and also shortly before the Brexit vote. If even low-grade AI technology like these bots are already influencing our politics, this raises another urgent question: “How powerful will far more advanced techniques be in the future?” 145 HOUSE_OVERSIGHT_016365

Visible / Invisible The artist Paul Klee often talked about art as “making the invisible visible.” In computer technology, most algorithms work invisibly, in the background; they remain inaccessible in the systems we use daily. But lately there has been an interesting comeback of visuality in machine learning. The ways that the deep-learning algorithms of AI are processing data have been made visible through applications like Google’s DeepDream, in which the process of computerized pattern-recognition is visualized in real time. The application shows how the algorithm tries to match animal forms with any given input. There are many other AI visualization programs that, in their way, also “make the invisible visible.” The difficulty in the general public perception of such images is, in Steyerl’s view, that these visual patterns are viewed uncritically as realistic and objective representations of the machine process. She says of the aesthetics of such visualizations: For me, this proves that science has become a subgenre of art history. ... We now have lots of abstract computer patterns that might look like a Paul Klee painting, or a Mark Rothko, or all sorts of other abstractions that we know from art history. The only difference, I think, is that in current scientific thought they’re perceived as representations of reality, almost like documentary images, whereas in art history there’s a very nuanced understanding of different kinds of abstraction. What she seeks is a more profound understanding of computer-generated images and the different aesthetic forms they use. They are obviously not generated with the explicit goal of following a certain aesthetic tradition. The computer engineer Mike Tyka, in a conversation with Steyerl, explained the functions of these images: Deep-learning systems, especially the visual ones, are really inspired by the need to know what’s going on in the black box. Their goal is to project these processes back into the real world. Nevertheless, these images have aesthetic implications and values which have to be taken into account. One could say that while the programmers use these images to help us better understand the programs’ algorithms, we need the knowledge of artists to better understand the aesthetic forms of AI. As Steyerl has pointed out, such visualizations are generally understood as “true” representations of processes, but we should pay attention to their respective aesthetics, and their implications, which have to be viewed in a critical and analytical way. In 2017, the artist Trevor Paglen created a project to make these invisible AI algorithms visible. In Sight Machine, he filmed a live performance of the Kronos Quartet and processed the resulting images with various computer software programs used for face detection, object identification, and even for missile guidance. He projected the outcome of these algorithms, in real time, back to screens above the stage. By demonstrating how the various different programs interpreted the musicians’ performance, Paglen showed that AI algorithms are always determined by sets of values and interests which they then manifest and reiterate, and thus must be critically questioned. The significant contrast between algorithms and music also raises the issue of relationships between technical and human perception. 146 HOUSE_OVERSIGHT_016366

Computers, as a Tool for Creativity, Can’t Replace the Artist. Rachel Rose, a video artist who thinks about the questions posed by AI, employs computer technology in the creation of her works. Her films give the viewer an experience of materiality through the moving image. She uses collaging and layering of the material to manipulate sound and image, and the editing process is perhaps the most important aspect of her work. She also talks about the importance of decision making in her work. For her, the artistic process does not follow a rational pattern. In a converation we had, together with the engineer Kenric McDowell, at the Google Cultural Institute, she explained this by citing a story from theater director Peter Brook’s 1968 book The Empty Space. When Brook designed the set for his production of 7he Tempest in the late 1960s, he started by making a Japanese garden, but then the design evolved, becoming a white box, a black box, a realistic set, and so on. And in the end, he returned to his original idea. Brook writes that he was shocked at having spent a month on his labors, only to end at the beginning. But this shows that the creative artistic process is a succession whose every step builds on the next and which eventually comes to an unpredictable conclusion. The process is not a logical or rational succession but has mostly to do with the artist’s feelings in reaction to the preceding result. Rose said, of her own artistic decision making: It, to me, is distinctively different from machine learning, because at each decision there’s this core feeling that comes from a human being, which has to do with empathy, which has to do with communication, which has to do with questions about our own mortality that only a human could ask. This point underlines the fundamental difference between any human artistic production and so-called computer creativity. Rose sees AI more as a possible way to create better tools for humans: A place I can imagine machine learning working for an artist would be not in developing an independent subjectivity, like writing a poem or making an image, but actually in filling in gaps that are to do with labor, like the way that Photoshop works with different tools that you can use. And though such tools may not seem spectacular, she says, “they might have a larger influence on art,” because they provide artists with further possibilities in their creative work. McDowell added that he, too, believes there are false expectations around AI. “Tve observed,” he said, “that there’s a sort of magical quality to the idea of a computer that does all the things that we do.” He continued: “There’s almost this kind of demonic mirror that we look into, and we want it to write a novel, we want it to make a film—we want to give that away somehow.” He is instead working on projects wherein humans collaborate with the machine. One of the current aims of AI research is to find new means of interaction between humans and software. And art, one could say, needs to play a key role in that enterprise, since it focuses on our subjectivity and on essential human aspects like empathy and mortality. 147 HOUSE_OVERSIGHT_016367

Cybernetics / Art Suzanne Treister is an artist whose work from 2009 to 2011 serves as an example of what is happening at the intersection of our current technologies, the arts, and cybernetics. Treister has been a pioneer in digital art since the 1990s, inventing, for example, imaginary video games and painting screen shots from them. In her project Hexen 2.0 she looked back at the famous Macy conferences on cybernetics that between 1946 and 1953 were organized in New York by engineers and social scientists to unite the sciences and to develop a universal theory of the workings of the mind. In her project, she created thirty photo-text works about the conference attendees (which included Wiener and von Foerster), she invented tarot cards, and she made a video based on a photomontage of a “cybernetic séance.” In the “séance,” the conference participants are seen sitting at a round table, as in spiritualist séances, while certain of their statements on cybernetics are heard in an audio-collage—trational knowledge and superstition combined. She also noted that some of the participating scientists worked for the military; thus the application of cybernetics could be seen in an ambivalent way, even back then, as a tussle between pure knowledge and its use in state control. If one looks at Treister’s work about the Macy conference participants, one sees that no visual artist was included. A dialogue between artists and scientists would be fruitful in future discussions, and it is a bit astonishing that this wasn’t realized at the time, given von Foerster’s keen interest in art. He recounted in one of our conversations how his relation to the field dated back to his childhood: I grew up as a child in an artistic family. We often had visits from poets, philosophers, painters, and sculptors. Art was a part of my life. Later, I got into physics, as I was talented in this subject. But I always remained conscious of the importance of art for science. There wasn’t a great difference for me. For me, both aspects of life have always been very much alike—and accessible, too. We should see them as one. An artist also has to reflect on his work. He has to think about his grammar and his language. A painter must know how to handle his colors. Just think of how intensively oil colors were researched during the Renaissance. They wanted to know how a certain pigment could be mixed with others to get a certain tone of red or blue. Chemists and painters collaborated very closely. I think the artificial division between science and art is wrong. Though for von Foerster the relation between the art and science was always clear, for our own time this connection remains to be made. There are many reasons to multiply the links. The critical thinking of artists would be beneficial in respect to the dangers of AI, since they draw our attention to questions they consider essential from their perspective. With the advent of machine learning, new tools are available to artists for their work. And as the algorithms of AI are made visible through artificial images in new ways, artists’ critical visual knowledge and expertise will be harnessed. Many of the key questions of AI are philosophical in nature and can be answered only from a holistic point of view. The way they play out among adventurous artists will be worth following. Simulating Worlds For the most part, the works of contemporary artists have been embodied ruminations on 148 HOUSE_OVERSIGHT_016368

AI’s impact on existential questions of the self and our future interaction with nonhuman entities. Few, though, have taken the technologies and innovations of AI as the underlying materials of their work and sculpted them to their own vision. An exception is the artist Ian Cheng, who has gone as far as to construct entire worlds of artificial beings with varying degrees of sentience and intelligence. He refers to these worlds as Live Simulations. His Emissaries trilogy (2015-2017) is set in a fictional postapocalyptic world of flora and fauna, in which AI-driven animals and creatures explore the landscape and interact with each other. Cheng uses advanced graphics but has them programmed with a lot of glitches and imperfections, which imparts a futuristic and anachronistic atmosphere at the same time. Through his trilogy, which charts a history of consciousness, he asks the question “What is a simulation?” While the majority of artistic works that utilize recent developments in AI specifically draw from the field of machine learning, Cheng’s Live Simulations take a separate route. The protagonists and plot lines that are interlaced in each episodic simulation of Emissaries use the complex logic systems and rules of AI. What is profound about his continually evolving scenes is that complexity arises not through the desire/actions of any single actor or artificial godhead but instead through their constellation, collision, and constant evolution in symbiosis with one another. This gives rise to unexpected outcomes and unending, unknowable situations—you can never experience the exact same moment in successive viewings of his work. Cheng had a discussion at the Serpentine Marathon “GUEST, GHOST, HOST: MACHINE!” with the programmer Richard Evans, who recently designed Versu, an AI- based platform for interactive storytelling games. Evans’ work emphasizes the social interaction of the games’ characters, who react in a spectrum of possible behaviors to the choices made by the human players. In their conversation, Evans said that a starting point for the project was that most earlier simulation video games, such as The Sims, did not sufficiently take into account the importance of social practices. Simulated protagonists in games would often act in ways that did not correspond well with real human behavior. Knowledge of social practices limits the possibilities of action but is necessary to understand the meaning of our actions—which is what interests Cheng for his own simulations. The more parameters of actions in certain circumstances are determined in a computer simulation, the more interesting it is for Cheng to experiment with individual and specific changes. He told Evans, “I gather that if we had AI with more ability to respond to social contexts, tweaking one thing, you would get something quite artistic and beautiful.” Cheng also sees the work of programmers and AI simulations as creating new and sophisticated tools for experimenting with the parameters of our daily social practices. In this way, the involvement of artists in AI will lead to new kinds of open experiments in Art. Such possibilities are—like increased AI capabilities in general—still in the future. Recognizing that this is an experimental technology in its infancy, very far from apocalyptic visions of a superintelligent AI takeover, Cheng fills his simulations with prosaic avatars such as strange microbial globules, dogs, and the undead. Discussions like these, between artists and engineers, of course are not totally new. In the 1960s, the engineer Billy Kliiver brought artists together with engineers in a series of events, and in 1967 he founded the Experiments in Art and Technology program with Robert Rauschenberg and others. In London, at around the same time, Barbara 149 HOUSE_OVERSIGHT_016369

Stevini and John Latham, of the Artist Placement Group, took things a step further by asserting that there should be artists in residence in every company and every government. Today, these inspiring historical models can be applied to the field of AI. As AI comes to inhabit more and more of our everyday lives, the creation of a space that is nondeterministic and non-utilitarian in its plurality of perspectives and diversity of understandings will undoubtedly be essential. 150 HOUSE_OVERSIGHT_016370

Alison Gopnik is an international leader in the field of children’s learning and development and was one of the founders of the field of “theory of mind.” She has spoken of the child brain as a “powerful learning computer,” perhaps from personal experience. Her own Philadelphia childhood was an exercise in intellectual development. “Other families took their kids to see The Sound of Music or Carousel; we saw Racine ’s Phaedra and Samuel Beckett’s Endgame,” she has recalled. “Our family read Henry Fielding’s 18th-century novel Joseph Andrews out loud to each other around the fire on camping trips.” Lately she has invoked Bayesian models of machine learning to explain the remarkable ability of preschoolers to draw conclusions about the world around them without benefit of enormous data sets. “I think babies and children are actually more conscious than we are as adults,” she has said. “They’re very good at taking in lots of information from lots of different sources at once.” She has referred to babies and young children as “the research and development division of the human species.” Not that she treats them coldly, as if they were mere laboratory animals. They appear to revel in her company, and in the blinking, thrumming toys in her Berkeley lab. For years after her own children had outgrown it, she kept a playpen in her office. Her investigations into just how we learn, and the parallels to the deep-learning methods of AI, continues. “Tt turns out to be much easier to simulate the reasoning of a highly trained adult expert than to mimic the ordinary learning of every baby,” she says. “Computation is still the best—indeed, the only—scientific explanation we have of how a Physical object like a brain can act intelligently. But, at least for now, we have almost no idea at all how the sort of creativity we see in children is possible.” 151 HOUSE_OVERSIGHT_016371

Als VERSUS FOUR-YEAR-OLDS Alison Gopnik Alison Gopnik is a developmental psychologist at UC Berkeley; her books include The Philosophical Baby and, most recently, The Gardener and the Carpenter: What the New Science of Child Development Tells Us About the Relationship Between Parents and Children. Everyone’s heard about the new advances in artificial intelligence, and especially machine learning. You’ve also heard utopian or apocalyptic predictions about what those advances mean. They have been taken to presage either immortality or the end of the world, and a lot has been written about both those possibilities. But the most sophisticated Als are still far from being able to solve problems that human four-year- olds accomplish with ease. In spite of the impressive name, artificial intelligence largely consists of techniques to detect statistical patterns in large data sets. There is much more to human learning. How can we possibly know so much about the world around us? We learn an enormous amount even when we are small children; four-year-olds already know about plants and animals and machines; desires, beliefs, and emotions; even dinosaurs and spaceships. Science has extended our knowledge about the world to the unimaginably large and the infinitesimally small, to the edge of the universe and the beginning of time. And we use that knowledge to make new classifications and predictions, imagine new possibilities, and make new things happen in the world. But all that reaches any of us from the world is a stream of photons hitting our retinas and disturbances of air at our eardrums. How do we learn so much about the world when the evidence we have is so limited? And how do we do all this with the few pounds of grey goo that sits behind our eyes? The best answer so far is that our brains perform computations on the concrete, particular, messy data arriving at our senses, and those computations yield accurate representations of the world. The representations seem to be structured, abstract, and hierarchical; they include the perception of three-dimensional objects, the grammars that underlie language, and mental capacities like “theory of mind,” which lets us understand what other people think. Those representations allow us to make a wide range of new predictions and imagine many new possibilities in a distinctively creative human way. This kind of learning isn’t the only kind of intelligence, but it’s a particularly important one for human beings. And it’s the kind of intelligence that is a specialty of young children. Although children are dramatically bad at planning and decision making, they are the best learners in the universe. Much of the process of turning data into theories happens before we are five. Since Aristotle and Plato, there have been two basic ways of addressing the problem of how we know what we know, and they are still the main approaches in machine learning. Aristotle approached the problem from the bottom up: Start with senses—the stream of photons and air vibrations (or the pixels or sound samples of a digital image or recording)—and see if you can extract patterns from them. This approach was carried further by such classic associationists as philosophers David Hume 152 HOUSE_OVERSIGHT_016372

and J. S. Mill and later by behavioral psychologists, like Pavlov and B. F. Skinner. On this view, the abstractness and hierarchical structure of representations is something of an illusion, or at least an epiphenomenon. All the work can be done by association and pattern detection—especially if there are enough data. Over time, there has been a seesaw between this bottom-up approach to the mystery of learning and Plato’s alternative, top-down one. Maybe we get abstract knowledge from concrete data because we already know a lot, and especially because we already have an array of basic abstract concepts, thanks to evolution. Like scientists, we can use those concepts to formulate hypotheses about the world. Then, instead of trying to extract patterns from the raw data, we can make predictions about what the data should look like if those hypotheses are right. Along with Plato, such “rationalist” philosophers and psychologists as Descartes and Noam Chomsky took this approach. Here’s an everyday example that illustrates the difference between the two methods: solving the spam plague. The data consist of a long unsorted list of messages in your in-box. The reality is that some of these messages are genuine and some are spam. How can you use the data to discriminate between them? Consider the bottom-up technique first. You notice that the spam messages tend to have particular features: a long list of addressees, origins in Nigeria, references to million-dollar prizes or Viagra. The trouble is that perfectly useful messages might have these features, too. If you looked at enough examples of spam and non-spam emails, you might see not only that spam emails tend to have those features but that the features tend to go together in particular ways (Nigeria plus a million dollars spells trouble). In fact, there might be some subtle higher-level correlations that discriminate the spam messages from the useful ones—a particular pattern of misspellings and IP addresses, say. If you detect those patterns, you can filter out the spam. The bottom-up machine-learning techniques do just this. The learner gets millions of examples, each with some set of features and each labeled as spam (or some other category) or not. The computer can extract the pattern of features that distinguishes the two, even if it’s quite subtle. How about the top-down approach? I get an email from the editor of the Journal of Clinical Biology. It refers to one of my papers and says that they would like to publish an article by me. No Nigeria, no Viagra, no million dollars; the email doesn’t have any of the features of spam. But by using what I already know, and thinking in an abstract way about the process that produces spam, I can figure out that this email is suspicious. (1) I know that spammers try to extract money from people by appealing to human greed. (2) I also know that legitimate “open access” journals have started covering their costs by charging authors instead of subscribers, and that I don’t practice anything like clinical biology. Put all that together and I can produce a good new hypothesis about where that email came from. It’s designed to sucker academics into paying to “publish” an article in a fake journal. The email was a result of the same dubious process as the other spam emails, even though it looked nothing like them. I can draw this conclusion from just one example, and I can go on to test my hypothesis further, beyond anything in the email itself, by googling the “editor.” 153 HOUSE_OVERSIGHT_016373

In computer terms, I started out with a “generative model” that includes abstract concepts like greed and deception and describes the process that produces email scams. That lets me recognize the classic Nigerian email spam, but it also lets me imagine many different kinds of possible spam. When I get the journal email, I can work backward: “This seems like just the kind of mail that would come out of a spam-generating process.” The new excitement about AI comes because AI researchers have recently produced powerful and effective versions of both these learning methods. But there is nothing profoundly new about the methods themselves. Bottom-up Deep Learning In the 1980s, computer scientists devised an ingenious way to get computers to detect patterns in data: connectionist, or neural-network, architecture (the “neural” part was, and still is, metaphorical). The approach fell into the doldrums in the ’90s but has recently been revived with powerful “deep-learning” methods like Google’s DeepMind. For example, you can give a deep-learning program a bunch of Internet images labeled “cat,” others labeled “house,” and so on. The program can detect the patterns differentiating the two sets of images and use that information to label new images correctly. Some kinds of machine learning, called unsupervised learning, can detect patterns in data with no labels at all; they simply look for clusters of features—what scientists call a factor analysis. In the deep-learning machines, these processes are repeated at different levels. Some programs can even discover relevant features from the raw data of pixels or sounds; the computer might begin by detecting the patterns in the raw image that correspond to edges and lines and then find the patterns in those patterns that correspond to faces, and so on. Another bottom-up technique with a long history is reinforcement learning. In the 1950s, B. F. Skinner, building on the work of John Watson, famously programmed pigeons to perform elaborate actions—even guiding air-launched missiles to their targets (a disturbing echo of recent AI) by giving them a particular schedule of rewards and punishments. The essential idea was that actions that were rewarded would be repeated and those that were punished would not, until the desired behavior was achieved. Even in Skinner’s day, this simple process, repeated over and over, could lead to complex behavior. Computers are designed to perform simple operations over and over on a scale that dwarfs human imagination, and computational systems can learn remarkably complex skills in this way. For example, researchers at Google’s DeepMind used a combination of deep learning and reinforcement learning to teach a computer to play Atari video games. The computer knew nothing about how the games worked. It began by acting randomly and got information only about what the screen looked like at each moment and how well it had scored. Deep learning helped interpret the features on the screen, and reinforcement learning rewarded the system for higher scores. The computer got very good at playing several of the games, but it also completely bombed on others just as easy for humans to master. A similar combination of deep learning and reinforcement learning has enabled the success of DeepMind’s AlphaZero, a program that managed to beat human players at both chess and Go, equipped only with a basic knowledge of the rules of the game and 154 HOUSE_OVERSIGHT_016374

some planning capacities. AlphaZero has another interesting feature: It works by playing hundreds of millions of games against itself. As it does so, it prunes mistakes that led to losses, and it repeats and elaborates on strategies that led to wins. Such systems, and others involving techniques called generative adversarial networks, generate data as well as observing data. When you have the computational power to apply those techniques to very large data sets or millions of email messages, Instagram images, or voice recordings, you can solve problems that seemed very difficult before. That’s the source of much of the excitement in computer science. But it’s worth remembering that those problems—like recognizing that an image is a cat or a spoken word is “Siri” —are trivial for a human toddler. One of the most interesting discoveries of computer science is that problems that are easy for us (like identifying cats) are hard for computers—much harder than playing chess or Go. Computers need millions of examples to categorize objects that we can categorize with just afew. These bottom-up systems can generalize to new examples; they can label a new image as a “cat” fairly accurately, over all. But they do so in ways quite different from how humans generalize. Some images almost identical to a cat image won’t be identified by us as cats at all. Others that look like a random blur will be. Top-down Bayesian Models The top-down approach played a big role in early AI, and in the 2000s it, too, experienced a revival, in the form of probabilistic, or Bayesian, generative models. The early attempts to use this approach faced two kinds of problems. First, most patterns of evidence might in principle be explained by many different hypotheses: It’s possible that my journal email message is genuine, it just doesn’t seem likely. Second, where do the concepts that the generative models use come from in the first place? Plato and Chomsky said you were born with them. But how can we explain how we learn the latest concepts of science? Or how even young children understand about dinosaurs and rocket ships? Bayesian models combine generative models and hypothesis testing with probability theory, and they address these two problems. A Bayesian model lets you calculate just how likely it is that a particular hypothesis 1s true, given the data. And by making small but systematic tweaks to the models we already have, and testing them against the data, we can sometimes make new concepts and models from old ones. But these advantages are offset by other problems. The Bayesian techniques can help you choose which of two hypotheses is more likely, but there are almost always an enormous number of possible hypotheses, and no system can efficiently consider them all. How do you decide which hypotheses are worth testing in the first place? Brenden Lake at NYU and colleagues have used these kinds of top-down methods to solve another problem that’s easy for people but extremely difficult for computers: recognizing unfamiliar handwritten characters. Look at a character on a Japanese scroll. Even if you’ve never seen it before, you can probably tell if it’s similar to or different from a character on another Japanese scroll. You can probably draw it and even design a fake Japanese character based on the one you see—one that will look quite different from a Korean or Russian character. *” 37 Brenden M. Lake, Ruslan Salakhutdinov & Joshua B. Tenenbaum, “Human-level concept learning through probabilistic program induction,” Science, 350:6266, pp. 1332-38 (2015). 155 HOUSE_OVERSIGHT_016375

The bottom-up method for recognizing handwritten characters is to give the computer thousands of examples of each one and let it pull out the salient features. Instead, Lake ef al. gave the program a general model of how you draw a character: A stroke goes either right or left; after you finish one, you start another; and so on. When the program saw a particular character, it could infer the sequence of strokes that were most likely to have led to it—just as I inferred that the spam process led to my dubious email. Then it could judge whether a new character was likely to result from that sequence or from a different one, and it could produce a similar set of strokes itself. The program worked much better than a deep-learning program applied to exactly the same data, and it closely mirrored the performance of human beings. These two approaches to machine learning have complementary strengths and weaknesses. In the bottom-up approach, the program doesn’t need much knowledge to begin with, but it needs a great deal of data, and it can generalize only in a limited way. In the top-down approach, the program can learn from just a few examples and make much broader and more varied generalizations, but you need to build much more into it to begin with. A number of investigators are currently trying to combine the two approaches, using deep learning to implement Bayesian inference. The recent success of AI 1s partly the result of extensions of those old ideas. But it has more to do with the fact that, thanks to the Internet, we have much more data, and thanks to Moore’s Law we have much more computational power to apply to that data. Moreover, an unappreciated fact is that the data we do have has already been sorted and processed by human beings. The cat pictures posted to the Web are canonical cat pictures—pictures that humans have already chosen as “good” pictures. Google Translate works because it takes advantage of millions of human translations and generalizes them to a new piece of text, rather than genuinely understanding the sentences themselves. But the truly remarkable thing about human children is that they somehow combine the best features of each approach and then go way beyond them. Over the past fifteen years, developmentalists have been exploring the way children learn structure from data. Four-year-olds can learn by taking just one or two examples of data, as a top- down system does, and generalizing to very different concepts. But they can also learn new concepts and models from the data itself, as a bottom-up system does. For example, in our lab we give young children a “blicket detector”’—a new machine to figure out, one they’ve never seen before. It’s a box that lights up and plays music when you put certain objects on it but not others. We give children just one or two examples of how the machine works, showing them that, say, two red blocks make it go, while a green-and-yellow combination doesn’t. Even eighteen-month-olds immediately figure out the general principle that the two objects have to be the same to make it go, and they generalize that principle to new examples: For instance, they will choose two objects that have the same shape to make the machine work. In other experiments, we’ve shown that children can even figure out that some hidden invisible property makes the machine go, or that the machine works on some abstract logical principle.** 38 A. Gopnik, T. Griffiths & C. Lucas, “When younger learners can be better (or at least more open- minded) than older ones,” Curr. Dir. Psychol. Sci., 24:2, 87-92 (2015). 156 HOUSE_OVERSIGHT_016376

You can show this in children’s everyday learning, too. Young children rapidly learn abstract intuitive theories of biology, physics, and psychology in much the way adult scientists do, even with relatively little data. The remarkable machine-learning accomplishments of the recent AI systems, both bottom-up and top-down, take place in a narrow and well-defined space of hypotheses and concepts—a precise set of game pieces and moves, a predetermined set of images. In contrast, children and scientists alike sometimes change their concepts in radical ways, performing paradigm shifts rather than simply tweaking the concepts they already have. Four-year-olds can immediately recognize cats and understand words, but they can also make creative and surprising new inferences that go far beyond their experience. My own grandson recently explained, for example, that if an adult wants to become a child again, he should try not eating any healthy vegetables, since healthy vegetables make a child grow into an adult. This kind of hypothesis, a plausible one that no grown- up would ever entertain, is characteristic of young children. In fact, my colleagues and I have shown systematically that preschoolers are better at coming up with unlikely hypotheses than older children and adults.*? We have almost no idea how this kind of creative learning and innovation is possible. Looking at what children do, though, may give programmers useful hints about directions for computer learning. Two features of children’s learning are especially striking. Children are active learners; they don’t just passively soak up data like Als do. Just as scientists experiment, children are intrinsically motivated to extract information from the world around them through their endless play and exploration. Recent studies show that this exploration is more systematic than it looks and is well-adapted to find persuasive evidence to support hypothesis formation and theory choice.*° Building curiosity into machines and allowing them to actively interact with the world might be a route to more realistic and wide-ranging learning. Second, children, unlike existing Als, are social and cultural learners. Humans don’t learn in isolation but avail themselves of the accumulated wisdom of past generations. Recent studies show that even preschoolers learn through imitation and by listening to the testimony of others. But they don’t simply passively obey their teachers. Instead they take in information from others in a remarkably subtle and sensitive way, making complex inferences about where the information comes from and how trustworthy it is and systematically integrating their own experiences with what they are hearing.! “Artificial intelligence” and “machine learning” sound scary. And in some ways they are. These systems are being used to control weapons, for example, and we really should be scared about that. Still, natural stupidity can wreak far more havoc than artificial intelligence; we humans will need to be much smarter than we have been in the past to properly regulate the new technologies. But there is not much basis for either the apocalyptic or the utopian visions of AIs replacing humans. Until we solve the basic 3° A. Gopnik, et al., “Changes in cognitive flexibility and hypothesis search across human life history from childhood to adolescence to adulthood,” Proc. Nat. Acad. Sci., 114:30, 7892-99 (2017). “L. Schulz, “The origins of Inquiry: Inductive inference and exploration in early childhood,” Trends Cog. Sci., 16:7, 382-89 (2012). 4 A. Gopnik, Zhe Gardener and the Carpenter (New York: Farrar, Straus & Giroux, 2016), chaps. 4 and 5. 157 HOUSE_OVERSIGHT_016377

paradox of learning, the best artificial intelligences will be unable to compete with the average human four-year-old. 158 HOUSE_OVERSIGHT_016378

Peter Galison’s focus as a science historian is—speaking roughly—on the intersection of theory with experiment. “For quite a number of years I have been guided in my work by the odd confrontation of abstract ideas and extremely concrete objects,” he once told me, in explaining how he thinks about what he does. At the Washington, Connecticut, meeting he discussed the Cold War tension between engineers (like Wiener) and the administrators of the Manhattan Project (like Oppenheimer: “When [Wiener] warns about the dangers of cybernetics, in part he’s trying to compete against the kind of portentous language that people like Oppenheimer [used]: ‘When I saw the explosion at Trinity, I thought of the Bhagavad Gita—I am death, destroyer of worlds.’ That sense, that physics could stand and speak to the nature of the universe and airforce policy, was repellent and seductive. In away, you can see that over and over again in the last decades—nanosciences, recombinant DNA, cybernetics: ‘I stand reporting to you on the science that has the promise of salvation and the danger of annihilation—and you should pay attention, because this could kill you.’ It’s a very seductive narrative, and it’s repeated in artificial intelligence and robotics.” As a twenty-four-year old, when I first encountered Wiener’s ideas and met his colleagues at the MIT meeting I describe in the book’s Introduction, I was hardly interested in Wiener ’s warnings or admonitions. What drove my curiosity was the stark, radical nature of his view of life, based on the mathematical theory of communications in which the message was nonlinear: According to Wiener, “new concepts of communication and control involved a new interpretation of man, of man’s knowledge of the universe, and of society.”’ And that led to my first book, which took information theory—the mathematical theory of communications—as a model for all human experience. In a recent conversation, Peter told me he was beginning to write a book—about building, crashing, and thinking—that considers the black-box nature of cybernetics and how it represents what he thinks of as “the fundamental transformation of learning, machine learning, cybernetics, and the self.” 159 HOUSE_OVERSIGHT_016379

ALGORISTS DREAM OF OBJECTIVITY Peter Galison Peter Galison is a science historian, Joseph Pellegrino University Professor and co- founder of the Black Hole Initiative at Harvard University, and the author of Einstein's Clocks and Poincaré’s Maps: Empires of Time. In his second-best book, the great medieval mathematician al-Khwarizmi described the new place-based Indian form of arithmetic. His name, soon sonically linked to “algorismus” (in late medieval Latin) came to designate procedures acting upon numbers—eventually wending its way through “algorithm,” (on the model of logarithm”), into French and on into English. But I like the idea of a modern algorist, even if my spellcheck does not. I mean by it someone profoundly suspicious of the intervention of human judgment, someone who takes that judgment to violate the fundamental norms of what it is to be objective (and therefore scientific). Near the end of the 20th century, a paper by two University of Minnesota psychologists summarized a vast literature that had long roiled the waters of prediction. One side, they judged, had for all too long held resolutely—and ultimately unethically— to the “clinical method” of prediction, which prized all that was subjective: “informal,” “in-the-head,” and “impressionistic.” These clinicians were people (so said the psychologists) who thought they could study their subjects with meticulous care, gather in committees, and make judgment-based predictions about criminal recidivism, college success, medical outcomes, and the like. The other side, the psychologists continued, embodied everything the clinicians did not, embracing the objective: “formal,” “mechanical,” “algorithmic.” This the authors took to stand at the root of the whole triumph of post-Galilean science. Not only did science benefit from the actuarial; to a great extent, science was the mechanical-actuarial. Breezing through 136 studies of predictions, across domains from sentencing to psychiatry, the authors showed that in 128 of them, predictions using actuarial tables, a multiple-regression equation, or an algorithmic judgment equalled or exceeded in accuracy those using the subjective approach. They went on to catalog seventeen fallacious justifications for clinging to the clinical. There were the self-interested foot-draggers who feared losing their jobs to machines. Others lacked the education to follow statistical arguments. One group mistrusted the formalization of mathematics; another excoriated what they took to be the actuarial “dehumanizing;” yet others said that the aim was to understand, not to predict. But whatever the motivations, the review concluded that it was downright immoral to withhold the power of the objective over the subjective, the algorithmic over expert judgment.” “? William M. Grove & Paul E. Meehl, “Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The Clinical-Statistical Controversy,” Psychology, Public Policy, and Law, 2:2, 293-323 (1996). 160 HOUSE_OVERSIGHT_016380

The algorist view has gained strength. Anne Milgram served as Attorney General of the State of New Jersey from 2007 to 2010. When she took office, she wanted to know who the state was arresting, charging, and jailing, and for what crimes. At the time, she reports in a later TED Talk, she could find almost no data or analytics. By imposing statistical prediction, she continues, law enforcement in Camden during her tenure was able to reduce murders by 41 percent, saving thirty-seven lives, while dropping the total crime rate by 26 percent. After joining the Arnold Foundation as its vice president for criminal justice, she established a team of data scientists and statisticians to create a risk-assessment tool; fundamentally, she construed the team’s mission as deciding how to put “dangerous people” in jail while releasing the non- dangerous. “The reason for this,” Milgram contended, “is the way we make decisions. Judges have the best intentions when they make these decisions about risk, but they’re making them subjectively. They’re like the baseball scouts twenty years ago who were using their instinct and their experience to try to decide what risk someone poses. They’re being subjective, and we know what happens with subjective decision making, which is that we are often wrong.” Her team established nine-hundred-plus risk factors, of which nine were most predictive. The questions, the most urgent questions, for the team were: Will a person commit a new crime? Will that person commit a violent act? Will someone come back to court? We need, concluded Milgram, an “objective measure of risk” that should be inflected by judges’ judgment. We know the algorithmic statistical process works. That, she says, is “why Google is Google” and why moneyball wins games.*? Algorists have triumphed. We have grown accustomed to the idea that protocols and data can and should guide us in everyday action, from reminders about where we probably want to go next, to the likely occurrence of crime. By now, according to the literature, the legal, ethical, formal, and economic dimensions of algorithms are all quasi- infinite. I’d like to focus on one particular siren song of the algorithm: its promise of objectivity. Scientific objectivity has a history. That might seem surprising. Isn’t the notion—expressed above by the Minnesota psychologists—right? Isn’t objectivity co- extensive with science itself? Here it’s worth stepping back to reflect on all the epistemic virtues we might value in scientific work. Quantification seems like a good thing to have; so, too, do prediction, explanation, unification, precision, accuracy, certainty, and pedagogical utility. In the best of all possible worlds these epistemic virtues would all pull in the same direction. But they do not—not any more than our ethical virtues necessarily coincide. Rewarding people according to their need may very well conflict with rewarding people according to their ability. Equality, fairness, meritocracy—ethics, in a sense, is all about the adjudication of conflicting goods. Too often we forget that this conflict exists in science, too. Design an instrument to be as sensitive as possible and it often fluctuates wildly, making repetition of a measurement impossible. “Scientific objectivity” entered both the practice and the nomenclature of science after the first third of the 19th century. One sees this clearly in the scientific atlases that provided scientists with the basic objects of their specialty: There were (and are) atlases of the hand, atlases of the skull, atlases of clouds, crystals, flowers, bubble-chamber pictures, nuclear emulsions, and diseases of the eye. In the 18th century, it was obvious ‘43 TED Talk, January 2014, https://www.ted.com/speakers/anne_milgram. 161 HOUSE_OVERSIGHT_016381

that you would not depict this particular, sun-scorched, caterpillar-chewed clover found outside your house in an atlas. No, you aimed—if you were a genius natural philosopher like Goethe, Albinus, or Cheselden—to observe nature but then to perfect the object in question, to abstract it visually to the ideal. Take a skeleton, view it through a camera lucida, draw it with care. Then correct the “imperfections.” The advantage of this parting of the curtains of mere experience was clear: It provided a universal guide, one not attached to the vagaries of individual variation. As the sciences grew in scope, and scientists grew in number, the downside of idealization became clearer. It was one thing to have Goethe depict the “ur-plant” or “ur- insect.” It was quite another to have a myriad of different scientists each fixing their images in different and sometimes contradictory ways. Gradually, from around the 1830s forward, one begins to see something new: a claim that the image making was done with a minimum of human intervention, that protocols were followed. This could mean tracing a leaf with a pencil or pressing it into ink that was transferred to the page. It meant, too, that one suddenly was proud of depicting the view through a microscope of a natural object even with its imperfections. This was a radical idea: snowflakes shown without perfect hexagonal symmetry, color distortion near the edge of a microscope lens, tissue torn around the edges in the process of its preparation. Scientific objectivity came to mean that our representations of things were executed by holding back from intervention—even if it meant reproducing the yellow color near the edge of the image under the microscope, despite the fact that the scientist knew that the discoloration was from the lens, not a feature of the object of inquiry. The advantage of objectivity was clear: It superseded the desire to see a theory realized or a generally accepted view confirmed. But objectivity came at a cost. You lost that precise, easily teachable, colored, full depth-of-field, artist’s rendition of a dissected corpse. You got a blurry, bad depth-of-field, black-and-white photograph that no medical student (nor even many medical colleagues) could use to learn and compare cases. Still, for a long stretch of the 19th century, the virtue of hands-off, self-restraining objectivity was on the rise. Starting in the 1930s, the hardline scientific objectivity in scientific representation began running into trouble. In cataloging stellar spectra, for example, no algorithm could compete with highly trained observers who could sort them with far greater accuracy and replicability than any purely rule-following procedure. By the late 1940s, doctors had begun learning how to read electroencephalograms. Expert judgment was needed to sort out different kinds of seizure readings, while none of the early attempts to use frequency analysis could match that judgment. Solar magnetograms—mapping the magnetic fields across the sun—required the trained expert to pry the real signal from artifacts that emerged from the measuring instruments. Even particle physicists recognized that they could not program a computer to sort certain kinds of tracks into the right bins; judgment, trained judgment, was needed. There should be no confusion here: This was not a return to the invoked genius of an 18th-century idealizer. No one thought you could train to be a Goethe who alone among scientists could pick out the universal, ideal form of a plant, insect, or cloud. Expertise could be learned—you could take a course to learn to make expert judgments about electroencephalograms, stellar spectra, or bubble-chamber tracks; alas, no one has ever thought you could take a course that would lead to the mastery of exceptional 162 HOUSE_OVERSIGHT_016382

insight. There can be no royal road to becoming Goethe. In scientific atlas after scientific atlas, one sees explicit argument that “subjective” factors had to be part of the scientific work needed to create, classify, and interpret scientific images. What we see in so many of the algorists’ claims is a tremendous desire to find scientific objectivity precisely by abandoning judgment and relying on mechanical procedures—in the name of scientific objectivity. Many American states have legislated the use of sentencing and parole algorithms. Better a machine, it is argued, than the vagaries of a judge’s judgment. So here is a warning from the sciences. Hands-off algorithmic proceduralism did indeed have its heyday in the 19th century, and of course still plays a role in many of the most successful technical and scientific endeavors. But the idea that mechanical objectivity, construed as binding self-restraint, follows a simple, monotonic curve increasing from the bad impressionistic clinician to the good externalized actuary simply does not answer to the more interesting and nuanced history of the sciences. There is a more important lesson from the sciences. Mechanical objectivity is a scientific virtue among others, and the hard sciences learned that lesson often. We must do the same in the legal and social scientific domains. What happens, for example, when the secret, proprietary algorithm sends one person to prison for ten years and another for five years, for the same crime? Rebecca Wexler, visiting fellow at the Yale Law School Information Society Project, has explored that question, and the tremendous cost that trade-secret algorithms impose on the possibility of a fair legal defense.*4 Indeed, for a variety of reasons, law enforcement may not want to share the algorithms used to make DNA, chemical, or fingerprint identifications, which puts the defense in a much weakened position to make its case. In the courtroom, objectivity, trade secrets, and judicial transparency may pull in opposite directions. It reminds me of a moment in the history of physics. Just after World War II, the film giants Kodak and Ilford perfected a film that could be used to reveal the interactions and decays of elementary particles. The physicists were thrilled, of course—until the film companies told them that the composition of the film was a trade secret, so the scientists would never gain complete confidence that they understood the processes they were studying. Proving things with unopenable black boxes can be a dangerous game for scientists, and doubly so for criminal justice. Other critics have underscored how perilous it is to rely on an accused (or convicted) person’s address or other variables that can easily become, inside the black box of algorithmic sentencing, a proxy for race. By dint of everyday experience, we have grown used to the fact that airport security is different for children under the age of twelve and adults over the age of seventy-five. What factors do we want the algorists to have in their often hidden procedures? Education? Income? Employment history? What one has read, watched, visited, or bought? Prior contact with law enforcement? How do we want algorists to weight those factors? Predictive analytics predicated on mechanical objectivity comes at a price. Sometimes it may bea price worth paying; sometimes that price would be devastating for the just society we want to have. More generally, as the convergence of algorithms and Big Data governs a greater and greater part of our lives, it would be well worth keeping in mind these two lessons “4 Rebecca Wexler, “Life, Liberty, and Trade Secrets: Intellectual Property in the Criminal Justice System,” 70 Stanford Law Review, XXX (2018). 163 HOUSE_OVERSIGHT_016383

from the history of the sciences: Judgment is not the discarded husk of a now pure objectivity of self-restraint. And mechanical objectivity is a virtue competing among others, not the defining essence of the scientific enterprise. They are lessons to bear in mind, even if algorists dream of objectivity. 164 HOUSE_OVERSIGHT_016384

In the past decade, genetic engineering has caught up with computer science with regard to how new scientific initiatives are shaping our lives. Genetic engineer George Church, a pioneer of the revolution in reading and writing biology, is central to this new landscape of ideas. He thinks of the body as an operating system, with engineers taking the place of traditional biologists in retooling stripped-down components of organisms (from atoms to organs) in much the same vein as in the late 1970s, when electrical engineers were working their way to the first personal computer by assembling circuit boards, hard drives, monitors, etc. George created and is director of the Personal Genome Project, which provides the world’s only open-access information on human genomic, environmental, and trait data (GET) and sparked the growing DNA ancestry industry. He was instrumental in laying the groundwork for President Obama’s 2013 BRAIN (Brain Research through Advancing Innovative Neurotechnologies) Initiative—in aid of improving the brains of human beings to the point where, for much of what sustains us, we might not need the help of (potentially dicey) Als. “Tt could be that some of the BRAIN Initiative projects allow us to build human brains that are more consistent with our ethics and capable of doing advanced tasks like artificial intelligence,” George has said. “The safest path by far is getting humans to do all the tasks that they would like to delegate to machines, but we re not yet firmly on that super-safe path.” More recently, his crucially important pioneering use of the enzyme CRISPR (as well as methods better than CRISPR) to edit the genes of human cells is sometimes missed by the media in the telling of the CRISPR origins story. George's attitude toward future forms of artificial general intelligence is friendly, as evinced in the essay that follows. At the same time, he never loses sight of the AI- safety issue. On that subject, he recently remarked: “The main risk in AI, to my mind, is not so much whether we can mathematically understand what they ’re thinking; it’s whether we re capable of teaching them ethical behavior. We’re barely capable of teaching each other ethical behavior.” 165 HOUSE_OVERSIGHT_016385

THE RIGHTS OF MACHINES George M. Church George M. Church is Robert Winthrop Professor of Genetics at Harvard Medical School; Professor of Health Sciences and Technology, Harvard-MIT; and co-author (with Ed Regis) of Regenesis: How Synthetic Biology Will Reinvent Nature and Ourselves. In 1950, Norbert Wiener’s Zhe Human Use of Human Beings was at the cutting edge of vision and speculation in proclaiming that the machine like the djinnee, which can learn and can make decisions on the basis of its learning, will in no way be obliged to make such decisions as we should have made, or will be acceptable to us. ... Whether we entrust our decisions to machines of metal, or to those machines of flesh and blood which are bureaus and vast laboratories and armies and corporations, . . . [t]he hour is very late, and the choice of good and evil knocks at our door. But this was his book’s denouement, and it has left us hanging now for sixty-eight years, lacking not only prescriptions and proscriptions but even a well-articulated “problem statement.” We have since seen similar warnings about the threat of our machines, even in the form of outreach to the masses, via films like Colossus: The Forbin Project (1970), The Terminator (1984), The Matrix (1999), and Ex Machina (2015). But now the time is ripe for a major update, with fresh, new perspectives—notably focused on generalizations of our “human” rights and our existential needs. Concern has tended to focus on “us versus them [robots]” or “grey goo [nanotech]” or “monocultures of clones [bio].” To extrapolate current trends: What if we could make or grow almost anything and engineer any level of safety and efficacy desired? Any thinking being (made of any arrangement of atoms) could have access to any technology. Probably we should be less concerned about us-versus-them and more concerned about the rights of all sentients in the face of an emerging unprecedented diversity of minds. We should be harnessing this diversity to minimize global existential risks, like supervolcanoes and asteroids. But should we say “should”? (Disclaimer: In this and many other cases, when a technologist describes a societal path that “could,” “would,” or “should” happen, this doesn’t necessarily equate to the preferences of the author. It could reflect warning, uncertainty, and/or detached assessment.) Roboticist Gianmarco Veruggio and others have raised issues of roboethics since 2002; the U.K. Department of Trade and Industry and the RAND spin-off Institute for the Future have raised issues of robot rights since 2006. “Ts versus ought” It is commonplace to say that science concerns “is,” not “ought.” Stephen Jay Gould’s “non-overlapping magisteria” view argues that facts must be completely distinct from values. Similarly, the 1999 document Science and Creationism from the U.S. National Academy of Sciences noted that “science and religion occupy two separate realms.” This 166 HOUSE_OVERSIGHT_016386

division has been critiqued by evolutionary biologist Richard Dawkins, myself, and others. We can discuss “should” if framed as “we should do X in order to achieve Y.” Which Y should be a high priority is not necessarily settled by democratic vote but might be settled by Darwinian vote. Value systems and religions wax and wane, diversify, diverge, and merge just as living species do: subject to selection. The ultimate “value” (the “should”’) is survival of genes and memes. Few religions say that there is no connection between our physical being and the spiritual world. Miracles are documented. Conflicts between Church doctrine and Galileo and Darwin are eventually resolved. Faith and ethics are widespread in our species and can be studied using scientific methods, including but not limited to fMRI, psychoactive drugs, questionnaires, et cetera. Very practically, we have to address the ethical rules that should be built in, learned, or probabilistically chosen for increasingly intelligent and diverse machines. We have a whole series of trolley problems. At what number of people in line for death should the computer decide to shift a moving trolley to one person? Ultimately this might be a deep-learning problem—one in which huge databases of facts and contingencies can be taken into account, some seemingly far from the ethics at hand. For example, the computer might infer that the person who would escape death if the trolley is left alone is a convicted terrorist recidivist loaded up with doomsday pathogens, or a saintly POTUS—or part of a much more elaborate chain of events in detailed alternative realities. If one of these problem descriptions seems paradoxical or illogical, it may be that the authors of the trolley problem have adjusted the weights on each sides of the balance such that hesitant indecision is inevitable. Alternatively, one can use misdirection to rig the system, such that the error modes are not at the level of attention. For example, in the Trolley Problem, the real ethical decision was made years earlier when pedestrians were given access to the rails— or even before that, when we voted to spend more on entertainment than on public safety. Questions that at first seem alien and troubling, like “Who owns the new minds, and who pays for their mistakes?” are similar to well-established laws about who owns and pays for the sins of a corporation. The Slippery Slopes We can (over)simplify ethics by claiming that certain scenarios won’t happen. The technical challenges or the bright red lines that cannot be crossed are reassuring, but the reality is that once the benefits seem to outweigh the risks (even briefly and barely), the red lines shift. Just before Louise Brown’s birth in 1978, many people were worried that she “would turn out to be a little monster, in some way, shape or form, deformed, something wrong with her.”*° Few would hold this view of in-vitro fertilization today. What technologies are lubricating the slope toward multiplex sentience? It is not merely deep machine-learning algorithms with Big Iron. We have engineered rodents to be significantly better at a variety of cognitive tasks as well as to exhibit other relevant traits, such as persistence and low anxiety. Will this be applicable to animals that are already at the door of humanlike intelligence? Several show self-recognition in a mirror test—chimpanzees, bonobos, orangutans, some dolphins and whales, and magpies. 45 “Then, Doctors ‘All Anxious’ About Test-tube Baby” http://edition.cnn.com/2003/HEALTH/parenting/07/25/cnna.copperman/ 167 HOUSE_OVERSIGHT_016387

Even the bright red line for human manipulation of human beings shows many signs of moving or breaking completely. More than 2,300 approved clinical trials for gene therapy are in progress worldwide. A major medical goal is the treatment or prevention of cognitive decline, especially in light of our rapidly aging global demographic. Some treatments of cognitive decline will include cognitive enhancements (drugs, genes, cells, transplants, implants, and so on). These will be used off-label. The rules of athletic competition (e.g., banning augmentation with steroids or erythropoietin) do not apply to intellectual competition in the real world. Every bit of progress on cognitive decline is in play for off-label use. Another frontier of the human use of humans is “brain organoids.” We can now accelerate developmental biology. Processes that normally take months can happen in four days in the lab using the right recipes of transcription factors. We can make brains that, with increasing fidelity, recapitulate the differences between people born with aberrant cognitive abilities (e.g., microcephaly). Proper vasculature (veins, arteries, and capillaries) missing from earlier successes are now added, enabling brain organoids to surpass the former sub-microliter limit to possibly exceed the 1.2-liter size of modern human brains (or even the 5-liter elephant or 8-liter sperm whale brains). Conventional Computers versus Bio-electronic Hybrids As Moore’s Law miniaturization approaches its next speed bump (surely not a solid wall), we see the limits of the stochastics of dopant atoms in silicon slabs and the limits of beam-fabrication methods at around 10-nanometer feature size. Power (energy consumption) issues are also apparent: The great Watson, winner of Jeopardy!, used 85,000 watts real time, while the human brains were using 20 watts each. To be fair, the human body needs 100 watts to operate and twenty years to build, hence about 6 trillion joules of energy to “manufacture” a mature human brain. The cost of manufacturing Watson-scale computing is similar. So why aren’t humans displacing computers? For one, the Jeopardy! contestants’ brains were doing far more than information retrieval—much of which would be considered mere distractions by Watson (e.g., cerebellar control of smiling). Other parts allow leaping out of the box with transcendence unfathomable by Watson, such as what we see in Einstein’s five annus mirabilis papers of 1905. Also, humans consume more energy than the minimum (100 W) required for life and reproduction. People in India use an average of 700 W per person; it’s 10,000 W in the U.S. Both are still less than the 85,000 watts Watson uses. Computers can become more like us via neuromorphic computing, possibly a thousandfold. But human brains could get more efficient, too. The organoid brain-in-a- bottle could get closer to the 20 W limit. The idiosyncratic advantages of computers for math, storage, and search, faculties of limited use to our ancestors, could be designed and evolved anew in labs. Facebook, the National Security Agency, and others are constructing exabyte- scale storage facilities at more than a megawatt and four hectares, while DNA can store that amount in a milligram. Clearly, DNA is not a mature storage technology, but with Microsoft and Technicolor doubling down on it, we would be wise to pay attention. The main reason for the 6 trillion joules of energy required to get a productive human mind is the twenty years required for training. 168 HOUSE_OVERSIGHT_016388

Even though a supercomputer can “train” a clone of zemself in seconds, the energy cost of producing a mature silicon clone is comparable. Engineering (Homo) prodigies might make a small impact on this slow process, but speeding up development and implanting extensive memory (as DNA-exabytes or other means) could reduce duplication time of a bio-computer to close to the doubling time of cells (ranging from eleven minutes to twenty-four hours). The point is that while we may not know what ratio of bio/homo/nano/robo hybrids will be dominant at each step of our accelerating evolution, we can aim for high levels of humane, fair, and safe treatment (“use”) of one another. Bills of Rights date back to 1689 in England. FDR proclaimed the “Four Freedoms”’—freedom of speech, freedom of conscience, freedom from fear, and freedom from want. The U.N.’s Universal Declaration of Human Rights in 1948 included the right to life; the prohibition of slavery; defense of rights when violated; freedom of movement; freedom of association, thought, conscience, and religion; social, economic, and cultural rights; duties of the individual to society; and prohibition of use of rights in contravention of the purposes and principles of the United Nations. The “universal” nature of these rights is not universally embraced and is subject to extensive critique and noncompliance. How does the emergence of non-Homo- intelligences affect this discussion? At a minimum, it is becoming rapidly difficult to hide behind vague intuition for ethical decisions—“I know it when I see it” (U.S. Supreme Court Justice Potter Stewart, 1964) or the “wisdom of repugnance” (aka “yuck factor,” Leon Kass, 1997), or vague appeals to “common sense.” As we have to deal with minds alien to us, sometimes quite literal from our viewpoint, we need to be explicit—yea, even algorithmic. Self-driving cars, drones, stock-market transactions, NSA searches, et cetera, require rapid, pre-approved decision making. We may gain insights into many aspects of ethics that we have been trying to pin down and explain for centuries. The challenges have included conflicting priorities, as well as engrained biological, sociological, and semi-logical cognitive biases. Notably far from consensus in universal dogmas about human rights are notions of privacy and dignity, even though these influence many laws and guidelines. Humans might want the right to march in to read (and change) the minds of computers to see why they’re making decisions at odds with our (Homo) instincts. Is it not fair for machines to ask the same of us? We note the growth of movements toward transparency in potential financial conflicts; “open-source” software, hardware, and wetware; the Fair Access to Science and Technology Research Act (FASTR); and the Open Humans Foundation. In his 1976 book Computer Power and Human Reason, Joseph Weizenbaum argued that machines should not replace Homo in situations requiring respect, dignity, or care, while others (author Pamela McCorduck and computer scientists like John McCarthy and Bill Hibbard) replied that machines can be more impartial, calm, and consistent and less abusive or mischievous than people in such positions. Equality What did the thirty-three-year-old Thomas Jefferson mean in 1776 when he wrote, “We hold these Truths to be self-evident, that all Men are created equal, that they are endowed 169 HOUSE_OVERSIGHT_016389

by their Creator with certain unalienable Rights, that among these are Life, Liberty, and the Pursuit of Happiness”? The spectrum of current humans 1s vast. In 1776, “Men” did not include people of color or women. Even today, humans born with congenital cognitive or behavioral issues are destined for unequal (albeit in most cases compassionate) treatment—Down syndrome, Tay-Sachs disease, Fragile X syndrome, cerebral palsy, and so on. And as we change geographical location and mature, our unequal rights change dramatically. Embryos, infants, children, teens, adults, patients, felons, gender identities and gender preferences, the very rich and very poor—all of these face different rights and socioeconomic realities. One path to new mind-types obtaining and retaining rights similar to the most elite humans would be to keep a Homo component, like a human shield or figurehead monarch/CEO, signing blindly enormous technical documents, making snap financial, health, diplomatic, military, or security decisions. We will probably have great difficulty pulling the plug, modifying, or erasing (killing) a computer and its memories—especially if it has befriended humans and made spectacularly compelling pleas for survival (as all excellent researchers fighting for their lives would do). Even Scott Adams, creator of Di/bert, has weighed in on this topic, supported by experiments at Eindhoven University in 2005 noting how susceptible humans are to a robot-as-victim equivalent of the Milgram experiments done at Yale beginning in 1961. Given the many rights of corporations, including ownership of property, it seems likely that other machines will obtain similar rights, and it will be a struggle to maintain inequities of selective rights along multi-axis gradients of intellect and ersatz feelings. Radically Divergent Rules for Humans versus Nonhumans and Hybrids The divide noted above for intra Homo sapiens variation in rights explodes into a riot of inequality as soon as we move to entities that overlap (or will soon) the spectrum of humanity. In Google Street View, people’s faces and car license plates are blurred out. Video devices are excluded from many settings, such as courts and committee meetings. Wearable and public cameras with facial-recognition software touch taboos. Should people with hyperthymesia or photographic memories be excluded from those same settings? Shouldn’t people with prosopagnosia (face blindness) or forgetfulness be able to benefit from facial-recognition software and optical character recognition wherever they go, and if them, then why not everyone? If we all have those tools to some extent, shouldn’t we all be able to benefit? These scenarios echo Kurt Vonnegut’s 1961 short story “Harrison Bergeron,” in which exceptional aptitude is suppressed in deference to the mediocre lowest common denominator of society. Thought experiments like John Searle’s Chinese Room and Isaac Asimov’s Three Laws of Robotics all appeal to the sorts of intuitions plaguing human brains that Daniel Kahneman, Amos Tversky, and others have demonstrated. The Chinese Room experiment posits that a mind composed of mechanical and Homo sapiens parts cannot be conscious, no matter how competent at intelligent human (Chinese) conversation, unless a human can identify the source of the consciousness and “feel” it. Enforced preference for Asimov’s First and Second Laws favor human minds over any other mind meekly present in his Third Law, of self-preservation. 170 HOUSE_OVERSIGHT_016390

If robots don’t have exactly the same consciousness as humans, then this is used as an excuse to give them different rights, analogous to arguments that other tribes or races are less than human. Do robots already show free will? Are they already self- conscious? The robots Qbo have passed the “mirror test” for self-recognition and the robots NAO have passed a related test of recognizing their own voice and inferring their internal state of being, mute or not. For free will, we have algorithms that are neither fully deterministic nor random but aimed at nearly optimal probabilistic decision making. One could argue that this is a practical Darwinian consequence of game theory. For many (not all) games/problems, if we’re totally predictable or totally random, then we tend to lose. What is the appeal of free will anyway? Historically it gave us a way to assign blame in the context of reward and punishment on Earth or in the afterlife. The goals of punishment might include nudging the priorities of the individual to assist the survival of the species. In extreme cases, this could include imprisonment or other restrictions, if Skinnerian positive/negative reinforcement is inadequate to protect society. Clearly, such tools can apply to free will, seen broadly—to any machine whose behavior we’d like to manage. We could argue as to whether the robot actually experiences subjective qualia for free will or self-consciousness, but the same applies to evaluating a human. How do we know that a sociopath, a coma patient, a person with Williams syndrome, or a baby has the same free will or self-consciousness as our own? And what does it matter, practically? If humans (of any sort) convincingly claim to experience consciousness, pain, faith, happiness, ambition, and/or utility to society, should we deny them rights because their hypothetical qualia are hypothetically different from ours? The sharp red lines of prohibition, over which we supposedly will never step, increasingly seem to be short-lived and not sensible. The line between human and machines blurs, both because machines become more humanlike and humans become more machine-like—not only since we increasingly blindly follow GPS scripts, reflex tweets, and carefully crafted marketing, but also as we digest ever more insights into our brain and genetic programming mechanisms. The NIH BRAIN Initiative is developing innovative technologies and using these to map out the connections and activity of mental circuitry so as to improve electronic and synthetic neurobiological ware. Various red lines depend on genetic exceptionalism, in which genetics 1s considered permanently heritable (although it is provably reversible), whereas exempt (and lethal) technologies, like cars, are for all intents and purposes irreversible due to social and economic forces. Within genetics, a red line makes us ban or avoid genetically modified foods but embrace genetically modified bacteria making insulin, or genetically modified humans—witness mitochondrial therapies approved in Europe for human adults and embryos. The line for germline manipulation seems less sensible than the usual, practical line drawn at safety and efficacy. Marriages of two healthy carriers of the same genetic disease have a choice between no child of their own, 25-percent loss of embryos via abortion (spontaneous or induced), 80-percent loss via in-vitro fertilization, or potential zero-percent embryo loss via sperm (germline) engineering. It seems premature to declare this last option unlikely. 171 HOUSE_OVERSIGHT_016391

For “human subject research,” we refer to the 1964 Declaration of Helsinki, keeping in mind the 1932-1972 Tuskegee syphilis experiment, possibly the most infamous biomedical research study in U.S. history. In 2015, the Nonhuman Rights Project filed a lawsuit with the New York State Supreme Court on behalf of two chimpanzees kept for research by Stony Brook University. The appellate court decision was that chimps are not to be treated as legal persons since they “do not have duties and responsibilities in society,” despite Jane Goodall’s and others’ claim that they do, and despite arguments that such a decision could be applied to children and the disabled.*° What prevents extension to other animals, organoids, machines, and hybrids? As we (e.g., Hawking, Musk, Tallinn, Wilczek, Tegmark) have promoted bans on “autonomous weapons,” we have demonized one type of “dumb” machine, while other machines—for instance, those composed of many Homo sapiens voting—can be more lethal and more misguided. Do transhumans roam the Earth already? Consider the “uncontacted peoples,” such as the Sentinelese and Andamanese of India, the Korowai of Indonesia, the Mashco- Piro of Peru, the Pintupi of Australia, the Surma of Ethiopia, the Ruc of Vietnam, the Ayoreo-Totobiegosode of Paraguay, the Himba of Namibia, and dozens of tribes in Papua New Guinea. How would they or our ancestors respond? We could define “transhuman” as people and culture not comprehensible to humans living in a modern, yet un-technological culture. Such modern Stone Age people would have great trouble understanding why we celebrate the recent LIGO gravity-wave evidence supporting the hundred-year-old general theory of relativity. They would scratch their heads as to why we have atomic clocks, or GPS satellites so we can find our way home, or why and how we have expanded our vision from a narrow optical band to the full spectrum from radio to gamma. We can move faster than any other living species; indeed, we can reach escape velocity from Earth and survive in the very cold vacuum of space. If those characteristics (and hundreds more) don’t constitute transhumanism, then what would? If we feel that the judge of transhumanism should not be fully paleo-culture humans but recent humans, then how would we ever reach transhuman status? We “recent humans” may always be capable of comprehending each new technological increment—never adequately surprised to declare arrival at a (moving) transhuman target. The science-fiction prophet William Gibson said, “The future is already here— it’s just not very evenly distributed.” While this underestimates the next round of “future,” certainly millions of us are transhuman already—with most of us asking for more. The question “What was a human?” has already transmogrified into “What were the many kinds of transhumans?. .. And what were their rights?” “© hittps:/Avww.nbcnews.com/news/us-news/lawyer-denying-chimpanzees-rights-could-backfire-disabled- n734566. 172 HOUSE_OVERSIGHT_016392

Caroline A. Jones’ interest in modern and contemporary art is enriched by a willingness to delve into the technologies involved in its production, distribution, and reception. “As an art historian, a lot of my questions are about what kind of art we can make, what kind of thought we can make, what kind of ideas we can make that could stretch the human beyond our stubborn, selfish, ‘only concerned with our small group’ parameters. The Philosophers and philosophies I’m drawn to are those that question the Western obsession with individualism. Those are coming from so many different places, and they re reviving so many different kinds of questions and problems that were raised in the 1960s.” She has recently turned her attention to the history of cybernetics. Her MIT course, “Automata, Automatism, Systems, Cybernetics,” explores the history of the human/machine interface in terms of feedback, exploring the cultural rather than engineering uptake of this idea. She begins with primary readings by Wiener, Shannon, and Turing and then pivots from the scientists and engineers to the work and ideas of artists, feminists, postmodern theorists. Her goal: to come up with a new central paradigm of evolution that’s culture-based—“communalism and interspecies symbiosis rather than survival of the fittest.” As a historian, Caroline draws a distinction between what she has termed “left cybernetics” and “right cybernetics”: “What do I mean by left cybernetics? In one sense, it’s a pun or a joke: the cybernetics that was ‘left’ behind. On another level, it’s a vague political grouping connoting our Left Coast: California, Esalen, the group that Dave Kaiser calls the ‘hippie physicists.’ It’s not an adequate term, but it’s away of recognizing that there was a group beholden to the military-industrial complex, sometimes very unhappily, who gave us the tools to critique it.” 173 HOUSE_OVERSIGHT_016393

THE ARTISTIC USE OF CYBERNETIC BEINGS Caroline A. Jones Caroline A. Jones is a professor of art history in the Department of Architecture at MIT and author of Eyesight Alone: Clement Greenberg’s Modernism and the Bureaucratization of the Senses; Machine in the Studio: Constructing the Postwar American Artist; and The Global Work of Art. Cybernated art is very important, but art for cybernated life is more important. — Nam June Paik, 1966 Artificial intelligence was not what artists first wanted out of cybernetics, once Norbert Wiener’s Zhe Human Use of Human Beings: Cybernetics and Society came out in 1950. The range of artists who identified themselves with cybernetics in the fifties and sixties initially had little access to “thinking machines.” Moreover, craft-minded engineers had already been making turtles, jugglers, and light-seeking robot babes, not giant brains. Using breadboards, copper wire, simple switches, and electronic sensors, artists followed cyberneticians in making sculptures and environments that simulated interactive sentience—analog movements and interfaces that had more to do with instinctive drives and postwar sexual politics than the automation of knowledge production. Now obscured by an ideology of a free-floating “intelligence” untethered by either hardware or flesh, AI has forgotten the early days of cybernetics’ uptake by artists. Those efforts are worth revisiting; they modeled relations with what the French philosophers Gilles Deleuze and Félix Guattari have called the “machinic phylum,” having to do with how humans think and feel in bodies engaged with a physical, material, emotionally stimulating, and signaling world. Cybernetics now seems to have collapsed into an all-pervasive discourse of AI that was far from preordained. “Cybernetics,” as a word, claimed postwar newness for concepts that were easily four centuries old: notions of feedback, machine damping, biological homeostasis, logical calculation, and systems thinking that had been around since the Enlightenment (boosted by the Industrial Revolution). The names in this lineage include Descartes, Leibniz, Sadi Carnot, Clausius, Maxwell, and Watt. Wiener’s coinage nonetheless had profound cultural effects.4”7 The ubiquity today of the prefix “cyber-” confirms the desire for a crisp signifier of the tangled relations between humans and machines. In Wiener’s usage, things “cyber” simply involved “control and communication in the animal and the machine.” But after the digital revolution, “cyber” moved beyond servomechanisms, feedback loops, and switches to encompass software, algorithms, and cyborgs. The work of cybernetically inclined artists concerns the emergent behaviors of life that elude AI in its current condition. As to that original coinage, Wiener had reached back to the ancient Greek to borrow the word for “steersman” (kvBepvytns / kubernétés), a masculine figure channeling power and instinct at the helm of a ship, who read the waves, judged the wind, kept a hand on the tiller, and directed the slaves as they mindlessly (mechanically) churned their oars. The Greek had already migrated into modern English via Latin, going 47 Wiener later had to admit the earlier coinage of the word in 1834 by André-Marie Ampére, who had intended it to mean the “science of government,” a concept that remained dormant until the 20th century. 174 HOUSE_OVERSIGHT_016394

from kuber- to guber—the root of “gubernatorial” and “governor,” another term for masculine control, deployed by James Watt to describe his 19th-century device for modulating a runaway steam engine. Cybernetics thus took ideas that had long analogized people and devices and generalized them to an applied science by adding that “-ics.” Wiener’s three c’s (command, control, communication) drew on the mathematics of probability to formalize systems (whether biological or mechanical) theorized as a set of inputs of information achieving outputs of actions in an environment—a muscular, fleshy agenda often minimized in genealogies of AI. But the etymology does little to capture the excitement felt by participants, as mathematics joined theoretical biology (Arturo Rosenblueth) and information theory (Claude Shannon, Walter Pitts, Warren McCulloch) to produce a barrage of interdisciplinary research and publications viewed as changing not just the way science was done but the way future humans would engage with the technosphere. As Wiener put it, “We have modified our environment so radically that we must now modify ourselves in order to exist.”48 The pressing question is: How are we modifying ourselves? Are we going in the right direction or have we lost our way, becoming the tools of our tools? Revisiting the early history of humanist/artists’ contribution to cybernetics may help direct us toward a less perilous, more ethical future. The year 1968 was a high-water mark of the cultural diffusion and artistic uptake of the term. In that year, the Howard Wise gallery opened its show of Wen-Ying Tsai’s “Cybernetic Sculpture” in midtown Manhattan, and Polish émigré Jasia Reichardt opened her exhibition “Cybernetic Serendipity” at London’s ICA. (The “Cybernetic” in her title was intended to evoke “made by or with computers,” even though most of the artworks on view had no computers, as such, in their responsive circuits.) The two decades between 1948 and 1968 had seen both the fanning out of cybernetic concepts into a broader culture and the spread of computation machines themselves in a slow migration from proprietary military equipment, through the multinational corporation, to the academic lab, where access began to be granted to artists. The availability of cybernetic components—“sensor organs” (electronic eyes, motion sensors, microphones) and “effector organs” (electronic “breadboards,” switches, hydraulics, pneumatics)—on the home hobbyist front rendered the computer less an “electronic brain” than an adjunct organ in a kit of parts. There was not yet a ruling metaphor of “artificial intelligence.” So artists were bricoleurs of electronic bodies, interested in actions rather than calculation or cognition. There were inklings of “computer” as calculator in the drive toward Homo rationalis, but more in aspiration than achievement. In light of today’s digital convergence in art/science imaging tools, Reichardt’s show was prophetic in its insistence on confusing the boundaries between art and what we might dub “creative applied science.” According to the catalog, “no visitor to the exhibition, unless he reads all the notes relating to all the works, will know whether he is looking at something made by an artist, engineer, mathematician, or architect.” So the comically dysfunctional robot by Nam June Paik, Robot K-456 (1964), featured on the catalog’s cover and described as “a female robot known for her disturbing and idiosyncratic behavior,” would face off against a balletic Colloquy of Mobiles (1968) from second-order cybernetician Gordon Pask. Pask worked with a London theater 48 The Human Use of Human Beings (1954 edition), p. 46. 175 HOUSE_OVERSIGHT_016395

designer to craft a spindly “male” apparatus of hinges and rods, set up to communicate with bulbous “female” fiberglass entities nearby. Whether anyone could actually map the quiddities of the program (or glean its reactionary gender theater) without reading the catalog essay is an open question. What is significant is Pask’s focus on the behaviors of his automata, their interactivity, their responsiveness within an artificially modulated environment, and their “reflection” of human behaviors. The ICA’s “Cybernetic Serendipity” introduced an important paradigm: the machinic ecosystem, in which the viewer was a biological part, tasked with figuring out just what the triggers for interaction might be. The visitors in those London galleries suddenly became “cybernetic organisms’”—cyborgs—since to experience the art adequately, one needed to enter a kind of symbiotic colloquy with the servomechanisms. This turn toward human-machine interactive environments as an aesthetic becomes clearer when we examine a few other artworks from the period, beginning with one constituting an early instance of emergent behavior—Senster, the interactive sculpture by artist/engineer Edward Ihnatowicz (1970), celebrated by medical robotics engineer Alex Zivanovic, editor of a Web site devoted to Ihnatowicz’s little-known career, as “one of the first computer controlled interactive robotic works of art.” Here, “the computer” makes its entry (albeit a twelve-bit, limited device). But rather than “intelligence,” Thnatowicz sought to make an avatar of affective behavior. Key to Senster ’s uncanny success was the programming with which Ihnatowicz constrained the fifteen-foot-long hydraulic apparatus (its hinge design and looming appearance inspired by a lobster claw) to convey shyness in responding to humans in its proximity. Sensfer’s sound channels and motion sensors were set to recoil at loud noises and sudden aggressive movements. Only those humans willing to speak softly and modulate their gestures would be rewarded by Senster’s quiet, inquisitive approach—an experience that became real for Thnatowicz himself when he first assembled the program and the machine turned to him solicitously after he’d cleared his throat. In these artistic uses of cybernetic beings, we sense a growing necessity to train the public to experience itself as embedded in a technologized environment, modifying itself to communicate intuitively with machines. This necessity had already become explicit in Tsai’s “Cybernetic Sculpture” show. Those experiencing his immersive installation were expected to experiment with machinic life: What behaviors would trigger the servomechanisms? Likely, the human gallery attendant would have had to explain the protocol: “Clap your hands—that gets the sculptures to respond.” As an early critic described it: A grove of slender stainless-steel rods rises from a plate. This base vibrates at 30 cycles per second; the rods flex rapidly, in harmonic curves. Set in a dark room, they are lit by strobes. The pulse of the flashing lights varies—they are connected to sound and proximity sensors. The result is that when one approaches a Tsai or makes a noise in its vicinity, the thing responds. The rods appear to move; there is a shimmering, a flashing, an eerie ballet of metal, whose apparent movements range from stillness to jittering and back to a slow, indescribably sensuous undulation.” “° Robert Hughes, Zime magazine (October 2, 1972) review of Tsai exhibition at Denise René gallery. 176 HOUSE_OVERSIGHT_016396

Like Senster, the apparatus stimulated (and simulated) an affective rather than rational interaction. Humans felt they were encountering behaviors indicative of responsive life; Tsai’s entities were often classed as “vegetal” or “aquatic.” Such environmental and kinetic ambitions were widespread in the international art world of the time. Beyond the stable at Howard Wise, there were the émigrés forming the collective GRAV in Paris, the “cybernetic architectures” of Nicolas Schoffer, the light and plastic gyrations of the German Zero Gruppe, and so on—all defining and informing the genre of installation art to come. The artistic use of cybernetic beings in the late sixties made no investment in “intelligence.” Knowing machines were dumb and incapable of emotion, these creators were confident in staging frank simulations. What interested them were machinic motions evoking drives, instincts, and affects; they mimicked sexual and animal behaviors, as if below the threshold of consciousness. Such artists were uninterested in the manipulation of data or information (although Hans Haacke would move in that direction by 1972 with his “Real-Time Systems” works). The cybernetic culture that artists and scientists were putting in place on two continents embedded the human in the technosphere and seduced perception with the graceful and responsive behaviors of the machinic phylum. “Artificial” and “natural” intertwined in this early cybernetic aesthetic. But it wouldn’t end here. Crucial to the expansion of this uncritical, largely masculine set of cybernetic environments would be a radical, critical cohort of astonishing women artists emerging in the 1990s, fully aware of their predecessors in art and technology but perhaps more inspired by the feminist founders of the 1970 journal Radical Software and the cultural blast of Donna Haraway’s inspiring 1984 polemic, “A Cyborg Manifesto.” The creaky gender theater of Paik and Pask, the innocent creatures of Ihnatowicz and Tsai, were mobilized as savvy, performative, and postmodern, as in Lynn Hershman Leeson’s Dollie Clone Series (1995-98) consisting of the interactive assemblages CyberRoberta and Tillie, the Telerobotic Doll, who worked the technosphere with the professionalism of burlesque, winking and folding us viewers into an explicit consciousness of our voyeuristic position as both seeing subjects and objects- to-be-looked-at. The “innocent” technosphere established by male cybernetic sculptors of the 1960s was, by the 1990s, identified by feminist artists as an entirely suffusive condition demanding our critical attention. At the same time, feminists tackled the question of whose “intelligence” AI was attempting to simulate. For an artist such as Hershman Leeson, responding to the technical “triumph” of cloning Dolly the sheep, it was crucial to draw the connection between meat production and “meat machines.” Hershman Leeson produced “dolls” as clones, offering a critical framing of the way contemporary individuation had become part of an ideological, replicative, plastic realm. While the technofeminists of the 1990s and into the 2000s weren’t all cyber all the time, their works nonetheless complicated the dominant machinic and kinetic qualities of male artists’ previous techno-environments. The androgynous tele-cyborg in Judith Barry’s Imagination, Dead Imagine (1991), for example, had no moving parts: He/she was comprised of pure signals, flickering projections on flat surfaces. In her setup, Barry commented on the alienating effects of late-20th-century technology. The image of an androgynous head fills an enormous cube made of ten-foot-square screens on 177 HOUSE_OVERSIGHT_016397

five sides, mounted on a ten-foot-wide mirrored base. A variety of viscous and unpleasant-looking fluids (yellow, reddish-orange, brown), dry materials (sawdust? flour?), and even insects drizzle or dust their way down the head, whose stoic sublimity is made gorgeously virtual on the work’s enormous screens. Dead Imagine, through its large-scale and cubic “Platonic” form, remains both artificial and locked into the body— refusing a detached “intelligence” as being no intelligence at all. Artists in the new millennium inherit this critical tradition and inhabit the current paradigms of AI, which has slid from partial simulations to claims of intelligence. In the 1955 proposal thought to be the first printed usage of the phrase “artificial intelligence,” computer scientist John McCarthy and his colleagues Marvin Minsky, Nathaniel Rochester, and Claude Shannon conjectured that “every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.” This modest theoretical goal has inflated over the past sixty-four years and is now expressed by Google DeepMind as an ambition to “Solve intelligence.” Crack the code! But unfortunately, what we hear cracking is not code but small-scale capitalism, the social contract, and the scaffolding of civility. Taking away the jobs of taxi and truck drivers, roboticizing direct marketing, hegemonizing entertainment, privatizing utilities, and depersonalizing health care—are these the “whips” that Wiener feared we would learn to love? Artists can’t solve any of this. But they can remind us of the creative potential of the paths not taken—the forks in the road that were emerging around 1970, before “information” became capital and “intelligence” equaled data harvesting. Richly evocative of what can be done with contemporary tools when revisiting earlier possibilities is French artist Philippe Parreno’s “firefly piece,” so nicknamed to avoid having to iterate its actual title: With a Rhythmic Instinction to Be Able to Travel Beyond Existing Forces of Life (2014). Described by the artist as “an automaton,” the sculptural installation juxtaposes a flickering projection of black-and-white drawings of fireflies with a band of oscillating green-on-black binary figures. The drawings and binary figures are animated using algorithms from mathematician John Horton Conway’s 1970 Game of Life, a “cellular automaton.” Conway set up parameters for any square (“cell”) to be lit (“alive”) or dark (“dead”) in an infinite, two-dimensional grid. The rules are summarized as follows: A single cell will quickly die of loneliness. But a cell touching three or more other “live” cells will also die, “due to crowding.” A cell survives and thrives if it has just two neighbors... and soon. As one cell dies, it may create the conditions for other cells to survive, yielding patterns that appear to move and grow, shifting across the grid like evanescent neural impulses or bioluminescent clusters of diatoms. In Stephen Hawking’s 2012 film The Meaning of Life, the narrator describes Conway’s mathematical model as simulating “how a complex thing like the mind might come about from a basic set of rules,” revealing the overweening ambitions that characterize contemporary AI: “[T]hese complex properties emerge from simple laws that contain no concepts like movement or reproduction,” yet they produce “species,” and cells “can even reproduce, just as life does in the real world.” °° Just as life does? Artists know the blandishments of simulation and representation, the difference between the genius of artifice and the realities of what “life °° Narration in Stephen Hawking’s The Meaning of Life (Smithson Productions, Discovery Channel, 2012). 178 HOUSE_OVERSIGHT_016398

does.” Parreno’s piece is an intuitive assembly of our experience of “life” through embodied, perspectival engagement. Our consciousness is electrically (cybernetically) enmeshed, yet we don’t respond as if this human-generated set of elegant simulations had its own intelligence. The artistic use of cybernetic beings also reminds us that consciousness itself is not just “in here.” It is streaming in and out, harmonizing those sensory, scintillating signals. Mind happens well outside the limits of the cranium (and its simulacrum, the “motherboard”). In Mary Catherine Bateson’s paraphrase of her father Gregory’s second-order cybernetics, mind is material “not necessarily defined by a boundary such as an envelope of skin.”°! Parreno pairs the simulations of art with the simulations of mathematics to force the Wiener-like point that any such model is not, by itself, just like life. Models are just that—parts of signaling systems constituting “intelligence” only when their creaturely counterparts engage them in lively meaning making. Contemporary AI has talked itself into a corner by instrumentalizing and particularizing tasks and subroutines, confusing these drills with actual wisdom. The brief cultural history offered here reminds us that views of data as intelligence, digital nets as “neural,” or isolated individuals as units of life, were alien even to Conway’s brute simulation. We can stigmatize the stubborn arrogance of current AI as “right cybernetics,” the path that led to current automated weapons systems, Uber’s ill-disguised hostility to human workers, and the capitalist dreams of Google. Now we must turn back to left cybernetics—theoretical biologists and anthropologists engaged with a trans-species understanding of intelligent systems. Gregory Bateson’s observation that corporations merely simulate “aggregates of parts of persons,” with profit-maximizing decisions cut off from “wider and wiser parts of the mind,” has never been more timely.*” The cybernetic epistemology offered here suggests a new approach. The individual mind is immanent, not only in the body but also in pathways outside the body, and there is a larger Mind, of which the individual mind is only a subsystem. This larger Mind, Bateson holds, is comparable to God, and is perhaps what some people mean by “God,” but it is still immanent in the total interconnected social system and planetary ecology. This is not the collective delusion of an exterior “God” who speaks from outside human consciousness (this long-seated monotheistic conceit, Bateson suggests, leads to views of nature and environment as also outside the “individual” human, rendering them as “gifts to exploit”). Rather, Bateson’s “God” is a placeholder for our evanescent experience of interacting consciousness-in-the-world: larger Mind as a result of inputs and actions that then become inputs for other actions in concert with other entities—webs of symbiotic relationships that form patterns we need urgently to sense and harmonize with.°? From Tsai in the 1970s to Hershman Leeson in the 1990s to Parreno in 2014, artists have been critiquing right cybernetics and plying alternative, embodied, environmental experiences of “artificial” intelligence. Their artistic use of cybernetic beings offers the wisdom of symbionts experienced in the kinds of poeisis that can be achieved in this world: rhythms of signals and intuitive actions that produce the 5! Mary Catherine Bateson, 1999 foreword to Gregory Bateson, Steps to an Ecology of Mind (Chicago: University of Chicago Press, 1972): xi. %? Steps to an Ecology of Mind, p. 452. 3 Tbid., pp. 467-8. 179 HOUSE_OVERSIGHT_016399

movements of life partnered with an electro-mechanical and -magnetic technosphere. Life, in its mysterious negentropic entanglements with matter and Mind. 180 HOUSE_OVERSIGHT_016400

Over nearly four decades, Stephen Wolfram has been a pioneer in the development and application of computational thinking and responsible for many innovations in science, technology and business. His 1982 paper “Cellular Automata as Simple Self-Organizing Systems,” written at the age of twenty-three, was the first of numerous significant scientific contributions aimed at understanding the origins of complexity in nature. It was around this time that Stephen briefly came into my life. I had established The Reality Club, an informal gathering of intellectuals who met in New York City to present their work before peers in other disciplines. (Note: In 1996, The Reality Club went online as Edge.org). Our first speaker? Stephen Wolfram, a “wunderkind” who had arrived in Princeton at the Institute for Advanced Study. I distinctly recall his focused manner as he sat down on a couch in my living room and spoke uninterrupted for about an hour before the assembled group. Since that time, Stephen has become intent making the world’s knowledge easily computable and accessible. His program Mathematica is the definitive system for modern technical computing. Wolfram|Alpha computes expert-level answers using AI technology. He considers his Wolfram Language to be the first true computational communication language for humans and AIs. I caught up with him again four years ago, when we arranged to meet in Cambridge, Massachusetts, for a freewheeling conversation about Al. Stephen walked in, said hello, sat down, and, looking at the video camera set up to record the conversation for Edge, began to talk and didn’t stop for two and a half hours. The essay that follows is an edited version of that session, which was a Wolfram master class of sorts and is an appropriate way to end this volume—just as Stephen’s Reality Club talk in the ’80s was a great way to initiate the ongoing intellectual enterprise whose result is the rich community of thinkers presenting their work to one another and to the public in this book. 181 HOUSE_OVERSIGHT_016401

ARTIFICIAL INTELLIGENCE AND THE FUTURE OF CIVILIZATION Stephen Wolfram Stephen Wolfram is a scientist, inventor, and the founder and CEO of Wolfram Research. He is the creator of the symbolic computation program Mathematica and its programming language, Wolfram Language, as well as the knowledge engine Wolfram|Alpha. He is also the author of A New Kind of Science. The following is an edited transcript from a live interview with him conducted in December 2015. I see technology as taking human goals and making them automatically executable by machines. Human goals of the past have entailed moving objects from here to there, using a forklift rather than our own hands. Now the work we can do automatically, with machines, is mental rather than physical. It’s obvious that we can automate many of the tasks we humans have long been proud of doing ourselves. What’s the future of the human condition in that situation? People talk about the future of intelligent machines and whether they’Il take over and decide what to do for themselves. But the inventing of goals is not something that has a path to automation. Someone or something has to define what a machine’s purpose should be—what it’s trying to execute. How are goals defined? For a given human, they tend to be defined by personal history, cultural environment, the history of our civilization. Goals are uniquely human. Where the machine is concerned, we can give it a goal when we build it. What kinds of things have intelligence, or goals, or purpose? Right now, we know one great example, and that’s us—our brains, our human intelligence. Human intelligence, I once assumed, is far beyond anything else that exists naturally in the world; it’s the result of an elaborate process of evolution and thus stands apart from the rest of existence. But what I’ve realized, as a result of the science I’ve done, is that this is not the case. People might say, for instance, “The weather has a mind of its own.” That’s an animist statement and seems to have no place in modern scientific thinking. But it’s not as silly as it sounds. What does the human brain do? A brain receives certain input, it computes things, it causes certain actions to happen, it generates a certain output. Like the weather. All sorts of systems are, effectively, doing computations—whether it’s a brain or, say, a cloud responding to its thermal environment. We can argue that our brains are doing vastly more sophisticated computations than those in the atmosphere. But it turns out that there’s a broad equivalence between the kinds of computations that different kinds of systems do. This renders the question of the human condition somewhat poignant, because it seems we’re not as special as we thought. There are all those different systems of nature that are pretty much equivalent, in terms of their computational capabilities. What makes us different from all those other systems is the particulars of our history, which give us our notions of purpose and goals. That’s a long way of saying that when the box on our desk thinks as well as the human brain does, what it still won’t have, intrinsically, are goals and purposes. Those are defined by our particulars—our particular biology, our particular psychology, our particular cultural history. 182 HOUSE_OVERSIGHT_016402

When we consider the future of AI, we need to think about the goals. That’s what humans contribute; that’s what our civilization contributes. The execution of those goals is what we can increasingly automate. What will the future of humans be in such a world? What will there be for them to do? One of my projects has been to understand the evolution of human purposes over time. Today we’ve got all kinds of purposes. If you look back a thousand years, people’s goals were quite different: How do I get my food? How do I keep myself safe? In the modern Western world, for the most part you don’t spend a large fraction of your life thinking about those purposes. From the point of view of a thousand years ago, some of the goals people have today would seem utterly bizarre—for example, like exercising on a treadmill. A thousand years ago that would sound like a crazy thing to do. What will people be doing in the future? A lot of purposes we have today are generated by scarcity of one kind or another. There are scarce resources in the world. People want to get more of something. Time itself is scarce in our lives. Eventually, those forms of scarcity will disappear. The most dramatic discontinuity will surely be when we achieve effective human immortality. Whether this will be achieved biologically or digitally isn’t clear, but inevitably it will be achieved. Many of our current goals are driven in part by our mortality: “I’m only going to live a certain time, so I'd better get this or that done.” And what happens when most of our goals are executed automatically? We won’t have the kinds of motivations we have today. One question I’d like an answer for is, What do the derivatives of humans in the future end up choosing to do with themselves? One of the potential bad outcomes is that they just play video games all the time. The term “artificial intelligence” is evolving, in its use in technical language. These days, AI is very popular, and people have some idea of what it means. Back when computers were being developed, in the 1940s and 1950s, the typical title of a book or a magazine article about computers was “Giant Electronic Brains.” The idea was that just as bulldozers and steam engines and so on automated mechanical work, computers would automate intellectual work. That promise turned out to be harder to fulfill than many people expected. There was, at first, a great deal of optimism; a lot of government money got spent on such efforts in the early 1960s. They basically just didn’t work. There are a lot of amusing science-fiction-ish portrayals of computers in the movies of that time. There’s a cute one called Desk Set, which is about an IBM-type computer being installed in a broadcasting company and putting everybody out of a job. It’s cute because the computer gets asked a bunch of reference-library questions. When my colleagues and I were building Wolfram|Alpha, one of the ideas we had was to get it to answer all of those reference-library questions from Desk Set. By 2009, it could answer them all. In 1943, Warren McCulloch and Walter Pitts came up with a model for how brains conceptually, formally, might work—an artificial neural network. They saw that their brainlike model would do computations in the same way as Turing Machines. From their work, it emerged that we could make brainlike neural networks that would act as general computers. And in fact, the practical work done by the ENIAC folks and John 183 HOUSE_OVERSIGHT_016403

von Neumann and others on computers came directly not from Turing Machines but through this bypath of neural networks. But simple neural networks didn’t do much. Frank Rosenblatt invented a learning device he called the perceptron, which was a one-layer neural network. In the late sixties, Marvin Minsky and Seymour Papert wrote a book titled Perceptrons, in which they basically proved that perceptrons couldn’t do anything interesting, which is correct. Perceptrons could only make linear distinctions between things. So the idea was more or less dropped. People said, “These guys have written a proof that neural networks can’t do anything interesting, therefore no neural networks can do anything interesting, so let’s forget about neural networks.” That attitude persisted for some time. Meanwhile, there were a couple of other approaches to AI. One was based on understanding, at a formal level, symbolically, how the world works; and the other was based on doing statistics and probabilistic kinds of things. With regard to symbolic AI, one of the test cases was, Can we teach a computer to do something like integrals? Can we teach a computer to do calculus? There were tasks like machine translation, which people thought would be a good example of what computers could do. The bottom line is that by the early seventies, that approach had crashed. Then there was a trend toward devices called expert systems, which arose in the late seventies and early eighties. The idea was to have a machine learn the rules that an expert uses and thereby figure out what to do. That petered out. After that, AI became little more than a crazy pursuit. I had been interested in how you make an AJI-like machine since I was a kid. I was interested particularly in how you take the knowledge we humans have accumulated in our civilization and automate answering questions on the basis of that knowledge. I thought about how you could do that symbolically, by building a system that could break down questions into symbolic units and answer them. I worked on neural networks at that time and didn’t make much progress, so I put it aside for a while. Back in mid-2002 to 2003, I thought about that question again: What does it take to make a computational knowledge system? The work I’d done by then pretty much showed that my original belief about how to do this was completely wrong. My original belief had been that in order to make a serious computational knowledge system, you first had to build a brainlike device and then feed it knowledge—just as humans learn in standard education. Now I realized that there wasn’t a bright line between what is intelligent and what is simply computational. I had assumed that there was some magic mechanism that made us vastly more capable than anything that was just computational. But that assumption was wrong. This insight is what led to Wolfram|Alpha. What I discovered is that you can take a large collection of the world’s knowledge and automatically answer questions on the basis of it, using what are essentially merely computational techniques. It was an alternative way to do engineering—a way that’s much more analogous to what biology does in evolution. In effect, what you normally do when you build a program 1s build it step-by-step. But you can also explore the computational universe and mine technology from that universe. Typically, the challenge is the same as in physical mining: That is, you find a supply of, let’s say, iron, or cobalt, or gadolinium, with some special magnetic properties, 184 HOUSE_OVERSIGHT_016404

and you turn that special capability to a human purpose, to something you want technology to do. In the case of magnetic materials, there are plenty of ways to do that. In terms of programs, it’s the same story. There are all kinds of programs out there, even tiny programs that do complicated things. Could we entrain them for some useful human purpose? And how do you get Als to execute your goals? One answer is to just talk to them, in the natural language of human utterances. It works pretty well when you’re talking to Siri. But when you want to say something longer and more complicated, it doesn’t work well. You need a computer language that can represent sophisticated concepts in a way that can be progressively built up and isn’t possible in natural language. What my company spent a lot of time doing was building a knowledge-based language that incorporates the knowledge of the world directly into the language. The traditional approach to creating a computer language is to make a language that represents operations that computers intrinsically know how to do: allocating memory, setting values of variables, iterating things, changing program counters, and so on. Fundamentally, you’re telling computers to do things in your own terms. My approach was to make a language that panders not to the computers but to the humans, to take whatever a human thinks of and convert it into some form that the computer can understand. Could we encapsulate the knowledge we’d accumulated, both in science and in data collection, into a language we could use to communicate with computers? That’s the big achievement of my last thirty years or so—being able to do that. Back in the 1960s, people would say things like, “When we can do such-and- such, we’ll know we have AI. When we can do an integral from a calculus course, we’ll know we have AI. When we can have a conversation with a computer and make it seem human... ,” et cetera. The difficulty was, “Well, gosh, the computer just doesn’t know enough about the world.” You’d ask the computer what day of the week it was, and it might be able to answer that. You'd ask it who the President was, and it probably couldn’t tell you. At that point, you’d know you were talking to a computer and not a person. But now when it comes to these Turing Tests, people who’ ve tried connecting, for example, Wolfram|Alpha to their Turing Test bots find that the bots lose every time. Because all you have to do 1s start asking the machine sophisticated questions and it will answer them! No human can do that. By the time you’ve asked it a few disparate questions, there will be no human who knows all those things, yet the system will know them. In that sense, we’ve already achieved good AI, at that level. Then there are certain kinds of tasks easy for humans but traditionally very hard for machines. The standard one is visual object identification: What is this object? Humans can recognize it and give some simple description of it, but a computer was just hopeless at that. A couple of years ago, though, we brought out a little image- identification system, and many other companies have done something similar—ours happens to be somewhat better than the rest. You show it an image, and for about ten thousand kinds of things, it will tell you what it is. It’s fun to show it an abstract painting and see what it says. But it does a pretty good job. It works using the same neural-network technology that McCulloch and Pitts imagined in 1943 and lots of us worked on in the early eighties. Back in the 1980s, people successfully did OCR—optical character recognition. They took the twenty-six letters of the alphabet and said, “OK, is that an A? Is that a B? Is that a C?” and so on. 185 HOUSE_OVERSIGHT_016405

That could be done for twenty-six different possibilities, but it couldn’t be done for ten thousand. It was just a matter of scaling up the whole system that makes this possible today. There are maybe five thousand picturable common nouns in English, ten thousand if you include things like special kinds of plants and beetles which people would recognize with some frequency. What we did was train our system on 30 million images of these kinds of things. It’s a big, complicated, messy neural network. The details of the network probably don’t matter, but it takes about a quadrillion GPU operations to do the training. Our system is impressive because it pretty much matches what humans can do. It has about the same training data humans have—about the same number of images a human infant would see in the first couple of years of its life. Roughly the same number of operations have to be done in the learning process, using about the same number of neurons in at least the first levels of our visual cortex. The details are different; the way these artificial neurons work has little to do with how the brain’s neurons work. But the concept is similar, and there’s a certain universality to what’s going on. At the mathematical level, it’s a composition of a very large number of functions, with certain continuity properties that let you use calculus methods to incrementally train the system. Given those attributes, you can end up with something that does the same job human brains do in physiological recognition. But does this constitute AI? There are a few basic components. There’s physiological recognition, there’s voice-to-text, there’s language translation—things humans manage to do with varying degrees of difficulty. These are essentially some of the links to how we make machines that are humanlike in what they do. For me, one of the interesting things has been incorporating those capabilities into a precise symbolic language to represent the everyday world. We now have a system that can say, “This is a glass of water.” We can go from a picture of a glass of water to the concept of a glass of water. Now we have to invent some actual symbolic language to represent those concepts. I began by trying to represent mathematical, technical kinds of knowledge and went on to other kinds of knowledge. We’ve done a pretty good job of representing objective knowledge in the world. Now the problem is to represent everyday human discourse in a precise symbolic way—a knowledge-based language intended for communication between humans and machines, so that humans can read it and machines can understand it, too. For instance, you might say “X is greater than 5.” That’s a predicate. You might also say, “I want a piece of chocolate.” That’s also a predicate. It has an “I want” init. We have to find a precise symbolic representation of the desires we express in human natural language. In the late 1600s, Gottfried Leibniz, John Wilkins, and others were concerned with what they called philosophical languages—that is, complete, universal, symbolic representations of things in the world. You can look at the philosophical language of John Wilkins and see how he divided up what was important in the world at the time. Some aspects of the human condition have been the same since the 1600s. Some are very different. His section on death and various forms of human suffering was huge; in today’s ontology, it’s alot smaller. It’s interesting to see how a philosophical language of today would differ from a philosophical language of the mid-1600s. It’s a measure of our progress. Many such attempts at formalization have happened over the years. In 186 HOUSE_OVERSIGHT_016406

mathematics, for example: Whitehead and Russell’s Principia Mathematica in 1910 was the biggest showoff effort. There were previous attempts by Gottlob Frege and Giuseppe Peano that were a little more modest in their presentation. Ultimately, they were wrong in what they thought they should formalize: They thought they should formalize some process of mathematical proof, which turns out not to be what most people care about. With regard to a modern analog of the Turing Test, it’s an interesting question. There’s still the conversational bot, which is Turing’s idea. That one hasn’t been solved yet. It will be solved—the only question is, What is the application for which it is solved? For along time I would ask, “Why should we care?”—because I thought the principal application would be customer service, which wasn’t particularly high on my list. But customer service, where you’ re trying to interface, 1s just where you need this conversational language. One big difference between Turing’s time and ours is the method of communicating with computers. In his time, you typed something into the machine and it typed back a response. In today’s world, it responds with a screen—as for instance, when you want to buy a movie ticket. How is a transaction with a machine different from a transaction with a human? The main answer ts that there’s a visual display. It asks you something, and you press a button, and you can see the result immediately. For example, in Wolfram|Alpha, when it’s used inside Sin, if there’s a short answer, Siri will tell you the short answer. But what most people want is the visual display, showing the infographic of this or that. This is a nonhuman form of communication that turns out to be richer than the traditional spoken, or typed, human communication. In most human- to-human communication, we’re stuck with pure language, whereas in computer-to- human communication we have this much higher bandwidth channel—of visual communication. Many of the most powerful applications of the Turing Test fall away now that we have this additional communication channel. For example, here’s one we’re pursuing right now. It’s a bot that communicates about writing programs: You say, “I want to write a program. I want it to do this.” The bot will say, “I’ve written this piece of program. This is what it does. Is this what you want?” Blah-blah-blah. It’s a back-and- forth bot. Devising such systems is an interesting problem, because they have to have a model of a human if they’re trying to explain something to you. They have to know what the human is confused about. What has long been difficult for me to understand is, What’s the point of a conventional Turing Test? What’s the motivation? As a toy, one could make a little chat bot that people could chat with. That will be the next thing. The current round of deep learning—particularly, recurrent neural networks—is making pretty good models of human speech and human writing. We can type in, say, “How are you feeling today?” and it knows most of the time what sort of response to give. But I want to figure out whether I can automate responding to my email. I know the answer is “No.” A good Turing Test, for me, will be when a bot can answer most of my email. That’s a tough test. It would have to learn those answers from the humans the email is connected to. I might be a little bit ahead of the game, because I’ve been collecting data on myself for about twenty-five years. I have every piece of email for twenty-five years, every keystroke for twenty. I should be able to train an avatar, an AI, that will do what I can do—perhaps better than I could. 187 HOUSE_OVERSIGHT_016407

People worry about the scenario in which Als take over. I think something much more amusing, in a sense, will happen first. The AI will know what you intend, and it will be good at figuring out how to get there. I tell my car’s GPS I want to go to a particular destination. I don’t know where the heck I am, I just follow my GPS. My children like to remind me that once when I had a very early GPS—the kind that told you, “Turn this way, turn that way”—we ended up on one of the piers going out into Boston Harbor. More to the point is that there will be an AI that knows your history, and knows that when you’re ordering dinner online you'll probably want such-and-such, or when you email this person, you should talk to them about such-and-such. More and more, the Als will suggest to us what we should do, and I suspect most of the time people will just go along with that. It’s good advice—better than what you would have figured out for yourself, As far as the takeover scenario is concerned, you can do terrible things with technology and you can do good things with technology. Some people will try to do terrible things with technology, and some people will try to do good things with technology. One of the things I like about today’s technology is the equalization it has produced. I used to be proud that I had a better computer than anybody I knew; now we all have the same kind of computers. We have the same smartphones, and pretty much the same technology can be used by a decent fraction of the planet’s 7 billion people. It’s not the case that the king’s technology is different from everybody else’s. That’s an important advance. The great frontier five hundred years ago was literacy. Today, it’s doing programming of some kind. Today’s programming will be obsolete in a not very long time. For example, people no longer learn assembly language, because computers are better at writing assembly language than humans are, and only a small set of people need to know the details of how language gets compiled into assembly language. A lot of what’s being done by armies of programmers today is similarly mundane. There’s no good reason for humans to be writing Java code or JavaScript code. We want to automate the programming process so that what’s important goes from what the human wants done to getting the machine, as automatically as possible, to doit. This will increase that equalization, which is something I’m interested in. In the past, if you wanted to write a serious piece of code, or program for something important and real, it was a lot of work. You had to know quite a bit about software engineering, you had to invest months of time in it, you had to hire programmers who knew this or you had to learn it yourself. It was a big investment. That’s not true anymore. A one-line piece of code already does something interesting and useful. It allows a vast range of people who couldn’t make computers do things for them, make computers do things for them. Something Id like to see is a lot of kids around the world learn the new capabilities of knowledge-based programming and then produce code that’s effectively as sophisticated as what anybody in the top ranks can produce. This is within reach. We’re at the point where anybody can learn to do knowledge-based programming, and, more important, learn to think computationally. The actual mechanics of programming are easy now. What’s difficult is imagining things in a computational way. 188 HOUSE_OVERSIGHT_016408

How do you teach computational thinking? In terms of how to do programming, it’s an interesting question. Take nanotechnology. How did we achieve nanotechnology? Answer: We took technology as we understand it on a large scale and we made it very small. How to make a CPU chip on the atomic scale? Fundamentally, we use the same architecture as the CPU chip we know and love. That isn’t the only approach one can take. Looking at what simple programs can do suggests that you can take even simple impoverished components and with the right compiler you can make them do interesting things. We don’t do molecular-scale computing yet, because the ambient technology is such that you’d have to spend a decade building it. But we’ve got the components that are enough to make a universal computer. You might not know how to program with those components, but by doing searches in the space of possible programs, you'd start to amass building blocks, and you could then create a compiler for them. The surprising thing is that impoverished stuff is capable of doing sophisticated things, and the compilation step is not as gruesome as you might expect. Just searching the computational universe and trying to find programs—building blocks—that are interesting is a good approach. A more traditional engineering approach—trying by pure thought to figure out how to build a universal computer—is a harder row to hoe. That doesn’t mean it can’t be done, but my guess is that we’ll be able to do some amazing things just by finding the components and searching the possible programs we can make with them. Then it’s back to the question about connecting human purposes to what is available from the system. One question I’m interested in is, What will the world look like when most people can write code? We had a transition, maybe five hundred years ago or so, when only scribes and a small set of the population could read and write natural language. Today, a small fraction of the population can write code. Most of the code they write is for computers only. You don’t understand things by reading code. But there will come a time when, as a result of things I’ve tried to do, the code is at a high enough level that it’s a minimal description of what you’re trying to do. It will be a piece of code that’s understandable to humans but also executable by the machines. Coding is a form of expression, just as writing in a natural language is a form of expression. To me, some simple pieces of code are poetic—they express ideas in a very clean way. There’s an aesthetic aspect, much as there is to expression in a natural language. One feature of code is that it’s immediately executable; it’s not like writing. When you write something, somebody has to read it, and the brain that’s reading it has to absorb the thoughts that came from the person who did the writing. Look at how knowledge has been transmitted in the history of the world. At level zero, one form of knowledge transmission is essentially genetic—that is, there’s an organism, and its progeny has the same features that it had. Then there’s the kind of knowledge transmission that happens with things like physiological recognition. A newborn creature has some neural network with some random connections in it, and as the creature moves around in the world, it starts recognizing kinds of objects and it learns that knowledge. Then there’s the level that was the big achievement of our species, which is natural language. The ability to represent knowledge abstractly enough that we can communicate it brain to brain, so to speak. Arguably, natural language is our species’ most important invention. It’s what led, in many respects, to our civilization. 189 HOUSE_OVERSIGHT_016409

There’s yet another level, and probably one day it will have a more interesting name. With knowledge-based programming, we have a way of creating an actual representation of real things in the world, in a precise and symbolic way. Not only is it understandable by brains and communicable to other brains and to computers, it’s also immediately executable. Just as natural language gave us civilization, knowledge-based programming will give us—what? One bad answer is that it will give us the civilization of the Als. That’s what we don’t want to happen, because the Als will do a great job communicating with one another and we’ll be left out of it, because there’s no intermediate language, no interface with our brains. What will this fourth level of knowledge communication lead to? If you were Caveman Ogg and you were just realizing that language was starting, could you imagine the coming of civilization? What should we be imagining right now? This relates to the question of what the world would look like if most people could code. Clearly, many trivial things would change: Contracts would be written in code, restaurant recipes might be written in code, and so on. Simple things like that would change. But much more profound things would also change. The rise of literacy gave us bureaucracy, for example, which had already existed but dramatically accelerated, giving us a greater depth of governmental systems, for better or worse. How does the coding world relate to the cultural world? Take high school education. If we have computational thinking, how does that affect how we study history? How does that affect how we study languages, social studies, and so on? The answer is, it has a great effect. Imagine you’re writing an essay. Today, the raw material for a typical high school student’s essay is something that’s already been written; students usually can’t generate new knowledge easily. But in the computational world, that will no longer be true. If the students know something about writing code, they’ll access all that digitized historical data and figure out something new. Then they’ll write an essay about something they’ve discovered. The achievement of knowledge-based programming is that it’s no longer sterile, because it’s got the knowledge of the world knitted into the language you’ re using to write code. There’s computation all over the universe: in a turbulent fluid producing some complicated pattern of flow, in the celestial mechanics of planetary interactions, in brains. But does computation have a purpose? You can ask that about any system. Does the weather have a goal? Does climate have a goal? Can someone looking at Earth from space tell that there’s anything with a purpose there? Is there a civilization there? In the Great Salt Lake, in Utah, there’s a straight line. It turns out to be a causeway dividing two areas of the lake with different colors of algae, so it’s a very dramatic straight line. There’s a road in Australia that’s long and straight. There’s a railroad in Siberia that’s long, and lights go on when a train stops at the stations. So from space you can see straight lines and patterns. But are these clear enough examples of obvious purpose on Earth as viewed from space? For that matter, how do we recognize extraterrestrials out there? How do we tell if a signal we’re getting indicates purpose? Pulsars were discovered in 1967, when we picked up a periodic flutter every second or so. The first question was, Is this a beacon? 190 HOUSE_OVERSIGHT_016410

Because what else would make a periodic signal? It turned out to be a rotating neutron star. One criterion to apply to a potentially purposeful phenomenon is whether it’s minimal in achieving a purpose. But does that mean that it was built for the purpose? The ball rolls down the hill because of gravitational pull. Or the ball rolls down the hill because it’s satisfying the principle of least action. There are typically these two explanations for some action that seems purposeful: the mechanistic explanation and the teleological. Essentially all of our existing technology fails the test of being minimal in achieving its purpose. Most of what we build is steeped in technological history, and it’s incredibly non-minimal for achieving its purpose. Look at a CPU chip; there’s no way that that’s the minimal way to achieve what a CPU chip achieves. This question of how to identify purposefulness is a hard one. It’s an important question, because radio noise from the galaxy is very similar to CDMA transmissions from cell phones. Those transmissions use pseudo-noise sequences, which happen to have certain repeatability properties. But they come across as noise, and they’re set up as noise, so as not to interfere with other channels. The issue gets messier. If we were to observe a sequence of primes being generated from a pulsar, we’d ask what generated them. Would it mean that a whole civilization grew up and discovered primes and invented computers and radio transmitters and did this? Or is there just some physical process making primes? There’s a little cellular automaton that makes primes. You can see how it works if you take it apart. It has a little thing bouncing inside it, and out comes a sequence of primes. It didn’t need the whole history of civilization and biology and so on to get to that point. I don’t think there is abstract “purpose,” per se. I don’t think there’s abstract meaning. Does the universe have a purpose? Then you’re doing theology in some way. There is no meaningful sense in which there is an abstract notion of purpose. Purpose is something that comes from history. One of the things that might be true about our world is that maybe we go through all this history and biology and civilization, and at the end of the day the answer is “42,” or something. We went through all those 4 billion years of various kinds of evolution and then we got to “42.” Nothing like that will happen, because of computational irreducibility. There are computational processes that you can go through in which there is no way to shortcut that process. Much of science has been about shortcutting computation done by nature. For example, if we’re doing celestial mechanics and want to predict where the planets will be a million years from now, we could follow the equations, step-by-step. But the big achievement in science is that we’re able to shortcut that and reduce the computation. We can be smarter than the universe and predict the endpoint without going through all the steps. But even with a smart enough machine and smart enough mathematics, we can’t get to the endpoint without going through the steps. Some details are irreducible. We have to irreducibly follow those steps. That’s why history means something. If we could get to the endpoint without going through the steps, history would be, in some sense, pointless. So it’s not the case that we’re intelligent and everything else in the world is not. There’s no enormous abstract difference between us and the clouds or us and the cellular automata. We cannot say that this brainlike neural network is qualitatively 191 HOUSE_OVERSIGHT_016411

different from this cellular-automaton system. The difference is a detailed difference. This brainlike neural network was produced by the long history of civilization, whereas the cellular automaton was created by my computer in the last microsecond. The problem of abstract AI is similar to the problem of recognizing extraterrestrial intelligence: How do you determine whether or not it has a purpose? This is a question I don’t consider answered. We’ll say things like, “Well, AI will be intelligent when it can do blah-blah-blah. ” When it can find primes. When it can produce this and that and the other. But there are many other ways to get to those results. Again, there is no bright line between intelligence and mere computation. It’s another part of the Copernican story: We used to think Earth was the center of the universe. Now we think we’re special because we have intelligence and nothing else does. I’m afraid the bad news is that that isn’t a distinction. Here’s one of my scenarios. Let’s say there comes a time when human consciousness 1s readily uploadable into digital form, virtualized and so on, and pretty soon we have a box of a trillion souls. There are a trillion souls in the box, all virtualized. In the box, there will be molecular computing going on—maybe derived from biology, maybe not. But the box will be doing all kinds of elaborate stuff. And there’s a rock sitting next to the box. Inside a rock, there are always all kinds of elaborate stuff going on, all kinds of subatomic particles doing all kinds of things. What’s the difference between the rock and the box of a trillion souls? The answer is that the details of what’s happening in the box were derived from the long history of human civilization, including whatever people watched on YouTube the day before. Whereas the rock has its long geological history but not the particular history of our civilization. Realizing that there isn’t a genuine distinction between intelligence and mere computation leads you to imagine that future—the endpoint of our civilization as a box of trillion souls, each of them essentially playing a video game, forever. What is the “purpose” of that? 192 HOUSE_OVERSIGHT_016412