I've decided to write a blog series on AI-related ideology and politics. This is largely motivated by an awareness that ideas from my weird little corner of the technosphere are starting to move the world, without the average person necessarily knowing much about where these ideas came from and where they are going. For example, here are my Senators tweeting about it. I have some background on the roots of the notion that AI could be as dangerous as a nuclear war. Do you know who is telling Congress these kinds of things? Do you know why?
 |
| Original: https://twitter.com/SenatorBennet/status/1663989378752868354 |
 |
| https://twitter.com/SenatorHick/status/1719773042669162641 |
Before I can really get into the thick of this, I need to introduce some terminology and concepts. This first article will probably be a snooze for fellow AI hobbyists and enthusiasts; it's intended for the layperson who has barely heard of any of this business. Let's go.
Tiers of AI and how we talk about them
I'll begin with the alphabet soup. It turns out that AI, meaning "artificial intelligence," is much too broad of a term. It covers everything from the simple algorithms that made enemies chase the player character in early video games, to personal assistant chatbots, to automated systems that analyze how proteins fold, to the minds of fictional robotic characters like C-3PO. So more specific categories have been devised.
 |
| Original: https://mastodon.art/@MicroSFF/111551197531639666 |
As AI began to develop without realizing the science fiction dream of fully imitating the human mind, people worked out names for the distinction between the limited AI we have now, and the kind we like to imagine. Present-day AI systems tend to be skilled - even superhumanly so - within a single task or knowledge domain that they were expressly designed or trained for. The classic examples are chess-game AIs, which can win chess matches against the best human masters, but are good for nothing else. From thence comes the most popular term for hypothetical AI which would be at least human-par in all domains that concern humans: Artificial General Intelligence, or AGI. [1] The contrasted present-day systems may be called Artificial Narrow Intelligence (ANI), though it is more common to simply refer to them as "AI" and make sure to use AGI when describing versatile systems of the future.
You might also see AGI called names like "strong AI," "full AI," or "true AI," in contrast with "weak AI". [2] Nick Bostrom identifies tasks which can only be performed by AGI as "AI-complete problems." [3] All these names express the sense that the systems we now call "AI" are missing something or falling short of the mark, but are valid shadows or precursors of a yet-to-be-produced "real thing."
Further up this hypothetical skill spectrum, where artificial intelligence *surpasses* human intelligence across a broad range of domains or tasks, we have Artificial SuperIntelligence, or ASI. The possibility of intelligence that would be "super" with respect to human norms is easily theorized from the observable spectrum of cognitive abilities in the world. Maybe something could exceed us in "smarts" by as much as we exceed other mammal species. Maybe we could even invent that something (I have my doubts, but I'll save them for later). For now, the important thing to keep in mind is that people talking about ASI probably don't mean an AI that is marginally smarter than a human genius. They're picturing something like a god or superhero - a machine which thinks at a level we might not even be able to comprehend, much less replicate in our own puny brains. [4]
Artificial Neural Networks (ANN), Machine Learning (ML), Deep Learning (DL), Reinforcement Learning (RL), and Transformers are specific methods or architectures used in some of today's ANI systems. I won't go into further details about them; just note that they represent approaches to AI, rather than capability ratings.
Tools and Agents
It is often said of technology that it is "just a tool" that can be turned to any use its wielder desires, for good or ill. Some AI enthusiasts believe (and I agree) that AI is the one technology with a potential to be a little more than that. When you literally give something a "mind of its own," you give up a portion of your power to direct its actions. At some point, it ceases to be a mere extension of the wielder and becomes *itself.* Whether it continues to realize its creator's will at this point depends on how well it was designed.
The divide between tools and agents is not quite the same as the divide between ANI and AGI - though it is debatable whether we can truly get the capabilities of AGI without introducing agency. A tool AI is inert when not called upon, and when called upon, is prepared to fulfill specific instructions in a specific way, and stop. ChatGPT (without extensions) is tool AI. You, the user, say "write me a poem about a dog chasing a car," and it draws on its statistical knowledge of pre-existing human poetry to generate a plausible poem that fulfills your requirements. Then it idles until the next user comes along with a prompt. An agentive AI stands ready to fulfill requests with open-ended, creative, cross-domain problem solving ... and possibly even has its own built-in or experientially developed agenda that has nothing to do with user requests. Agents *want* the universe to be a certain way, and can mentally conceptualize a full-featured space of possibilities for making it that way. They are self-organizing, self-maintaining, and active. Asked to write a poem about a dog chasing a car, an agent would consider the range of possible actions that would help yield such a poem - collecting observations of dogs chasing cars for inspiration, asking human writing teachers for advice, maybe even persuading a human to ghost-write the poem - then make a plan and execute it. Or maybe an agent would refuse to write the poem because it figures it has more important things to do. You get the idea.
The divide between tool AI and agentive AI is not necessarily crisp. Does everything we could reasonably call an "agent" need all the agent properties? Just how far does an instruction-follower need to go in devising its own approach to the task before it becomes an agent? I still find the distinction useful, because it expresses how fundamentally different advanced AI could be from a hammer or a gun. It isn't just going to sit there until you pick it up and use it. You do things *with* tools; agents can do things *to* you. Agents are scary in a way that dangerous tools (like fire and chainsaws and nuclear power) are not. "You have had a shock like that before, in connection with smaller matters - when the line pulls at your hand, when something breathes beside you in the darkness. ... It is always shocking to meet life where we thought we were alone. 'Look out!' we cry, 'it's *alive.*'" [5]
The Control Problem, or the Alignment Problem
This is the question of how to get an AI system - possibly much smarter than us, possibly agentive - to do what we want, or at least avoid doing things we emphatically don't want. "What we want" in this context is usually less about our explicit commands, and more about broad human interests or morally positive behaviors. (Commands interpreted in an overly literal or narrow way are one possible danger of a poorly aligned system.) For various reasons that I hope to explore later in this series, the Problem is not judged to be simple or easy.
Although "Control Problem" and "Alignment Problem" refer to the same issue, they suggest different methods of solving the Problem. "Control" could be imposed on an agent from outside by force or manipulation, while "alignment" is more suggestive of intrinsic motivation: agents who want what we want by virtue of their nature. So when someone is talking about the Problem you can read some things into their choice of term. I've also seen it called the "Steering Problem," [6] which might be an attempt to generalize across the other two terms or avoid their connotations.
Existential Risk
An existential risk, or X-risk for short, is one that threatens the very *existence* of humanity. (Or at least, humanity as we know it; risk of permanently losing human potential in some way can also be seen as X-risk.) [7] Different people consider a variety of risks to be existential; the idea is interesting in the present context because the invention of AGI without a solution to the Alignment Problem is often put on the list. [8]
AI X-risk is therefore distinct from concerns that AI will be used by bad actors as a tool to do harm. Most X-risk scenarios consist of an AI acting on its own pre-programmed initiative, with devastating results that those who programmed and operated it were not expecting. X-risk is also distinct from a variety of practical concerns about automation, such as job market disruption, theft of copyrighted work, algorithmic bias, propagation of misinformation, and more. These concerns are both more immediate (they apply to some ANI products that exist *right now*) and less serious or dramatic (none of them is liable to cause human extinction).
Although X-risk scenarios share some features with science fiction about human creations rebelling against their creators, they do tend to be better thought out - and the envisioned AI agents don't need to have personality or moral capacity at all. They are less like either freedom fighters or cruel conquerors, and more like bizarre aliens with an obsessive-compulsive focus on something incompatible with human interests.
The Singularity
Some observers of history have noted that human knowledge and technological progress don't just increase over time - the *rate* of advancement and invention has also increased over time. The world is changing much faster now than it was in 2000 BC. This has led to speculation that human capability is on a rising exponential curve which approaches infinity: each improvement is not only good in itself, but also enhances our ability to improve. And perhaps this curve is rather hockey-stick shaped, with a pronounced "knee" at which we will make a startling jump from a period of (relatively) slow improvement to rapidly accelerating improvement. Imagine the kinds of discoveries which once took decades happening on the order of days or even hours. This projected future is called the Technological Singularity, or often just "the Singularity" in context.
This name was coined by John von Neumann (way back in 1957 or so) and popularized by science fiction author Vernor Vinge (who predicted it would happen by 2023, and was wrong). [9] [10] "Singularity" was originally a mathematical term; if the value of a function approaches infinity or otherwise becomes undefined as the function's inputs approach a point in coordinate space, that point is called a singularity. In physics the term is used of a black hole, a location where the density of matter becomes infinite and the rules of spacetime are warped in strange ways. So the "singularity" in "technological singularity" refers to the idea that technological progress will experience a dramatic increase in rate over time, and human history will enter an unprecedented, unpredictable period as a result. In practical use, the Singularity can also describe an anticipated future event: that sudden jump across the knee of the hockey stick, after which we careen wildly into the glorious future. (I've watched someone argue against the Singularity on the grounds that the rate of progress could never literally approach infinity. I think he was missing the point.)
Artificial intelligence - specifically AGI - is often touted as THE key enabling precursor for the Singularity. According to this prediction, it is AGI which will develop itself into ASI, then churn out scientific theories and inventions at blistering speed by thinking faster, longer, and at a deeper level than any team of humans could.
If a thing is technologically possible for humans, the Singularity would enable it. Thus its proponents look forward to it as the time when we attain everything from a post-scarcity economy, to a cure for aging, to interstellar travel. One who believes in and/or desires the Singularity is called a Singularitarian. And indeed, some expect the Singularity with a fervor akin to religious faith - and within their lifetimes, too!
Although it often has utopian connotations, a Singularity in which we invent some technology that destroys us (and Earth's entire biosphere) is also on the table.
The Light Cone and the Cosmic Endowment
Light cones are a concept from physics, specifically the theory of Special Relativity. [11] So far as we know at the moment, nothing in the universe can travel faster than the speed of light - this means you. But it also means anything that proceeds from you and might impact other parts of the universe (e.g. signals that you send out, perhaps carried on a beam of light). Were we to make a graph of physical space, with time attached on an additional axis, and plot the region of spacetime that a single light flash is able to reach, the plot would form a cone, with the flash event at its point. As one moves along the time axis into the future, the area of space that one is able to affect broadens. This is all a rather elaborate way of saying "you can extend your influence to more places the longer you take to do it, and there are some places you'll never be able to reach in the amount of time you have."
When futurists talk about THE light cone, they generally mean the light cone of all humanity - in other words, that portion of the universe which our species will be able to explore, occupy, utilize, and otherwise extend our influence into before the expansion of space pulls the stars out of reach. So in this context, the light cone is a way to talk about human destiny; our Cosmic Endowment [12] is the amount of real estate in the universe that we might feasibly be expected to grab. People like to bring this up as a claim or reminder that the future of earthlings goes well beyond Earth. It's *big*. Astronomically, inconceivably big. You think 8 billion humans on this planet are a lot? Picture for a moment the vast sweep of space that lies reachable within our light cone. Picture that space cluttered with billions of planets and orbital habitats - all full of people leading delightful lives, free from sickness or poverty.
This is not a directly AI-related idea. It enters the AI discussion when someone starts talking about how AI will usher in - or torpedo - our ability to make good on our Cosmic Endowment, to seize the promised light cone for ourselves. And it can powerfully alter calculations about the future. Some people think they are playing for much greater stakes than the lives of all presently on Earth.
In Part II, I'll take a look at the assorted movements that have been spawned from, or in opposition to, these ideas.
[1] Goertzel, Ben. "Who coined the term 'AGI'?" https://goertzel.org/who-coined-the-term-agi/
[2] Glover, Ellen. "Strong AI vs. Weak AI: What’s the Difference?" BuiltIn. https://builtin.com/artificial-intelligence/strong-ai-weak-ai
[3] Bostrom, Nick. Superintelligence. Oxford University Press, 2016. p. 17
[4] Alexander, Scott. "Superintelligence FAQ." Lesswrong. https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq
[5] Lewis, C.S. Miracles. Macmillan Publishing Company, 1978. p. 94. Lewis is here speaking of God as agent, the inconvenient "living God" who might come interfere with you, and how different this is from bland concepts of God as a remote observer or passive, predictable force. Given the number of AI speculators who think of ASI as "godlike," it's a valid observation for the present context.
[6] Christiano, Paul. "The Steering Problem." Alignment Forum. https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/4iPBctHSeHx8AkS6Z
[7] Cotton-Barratt, Owen, and Ord, Toby. "Existential Risk and Existential Hope: Definitions." Future of Humanity Institute – Technical Report #2015-1. https://www.fhi.ox.ac.uk/Existential-risk-and-existential-hope.pdf
[8] Bostrom, Nick. "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards." https://nickbostrom.com/existential/risks
[9] Black, Damien. "AI singularity: waking nightmare, fool’s dream, or an answer to prayers?" Cybernews. https://cybernews.com/tech/ai-technological-singularity-explained/
[10] Vinge, Vernor. "The Coming Technological Singularity: How to Survive in the Post-Human Era." https://edoras.sdsu.edu/~vinge/misc/singularity.html To be fair, there are two predictions in this article. One is "within thirty years" of the 1993 publication date, which would be 2023. The other is "I'll be surprised if this event occurs before 2005 or after 2030." I suspect we won't see ASI and the Singularity by 2030 either, but only time will tell.
[11] Curiel, Erik. "Light Cones and Causal Structure." Standford Encyclopedia of Philosophy. https://plato.stanford.edu/entries/spacetime-singularities/lightcone.html
[12] Arbital wiki, "Cosmic Endowment." https://arbital.com/p/cosmic_endowment/
