Monday, March 11, 2024

AI Ideology III: Policy and Agendas

I'm in the midst of a blog series on AI-related ideology and politics. In Part II, I introduced the various factions that I have seen operating in the AI research and deployment sphere, including the loosely-related group nicknamed TESCREAL and a few others. I recommend reading it first to become familiar with the factions and some of their key people. In this Part III, I hope to get into a few more details about the political landscape these movements create, and what it could mean for the rest of us.

A rather ornate two-pan balance scale.

Balancing risks and rewards

"The problem with extinction narratives is that almost everything is worth sacrificing to avoid it—even your own mother." [1]

Some Doomers become so convinced of the dangers of advanced AI that they would almost rather kill people than see it developed. Yudkowski himself has infamously advised an international treaty to slow the development of AI, which would be enforceable by bombing unauthorized datacenters. (To Yudkowski, a nuclear exchange is a smaller risk than the development of ASI - because a global nuclear war could leave at least a few pathetic human survivors in charge of their own destiny, whereas he believes ASI would not.) [2] Others in the community mutter about "pivotal acts" to prevent the creation of hostile AGI. A "pivotal act" can be completely benign, but sabotage and threats of violence are also possible strategies. [3]

On the milder but still alarming end of things, Nick Bostrom once proposed mass surveillance as a way to make sure nobody is working on hostile AGI (or any other humanity-destroying tech) in their basement. [4]

But the most wild-eyed pessimists seem to have little political traction, so hopefully they won't be taken seriously enough to make the nukes come out. More realistic downsides of too much Doom include diversion of resources to AI X-risk mitigation (with corresponding neglect of other pressing needs), and beneficial technology development slowed by fear or over-regulation.

What about E/ACC, then? Are they the good guys? I wouldn't say so. They're mostly just the opposite extreme, substituting wild abandon and carelessness for the Doomers' paranoia, and a lack of accountability for over-regulation. The "all acceleration all the time" attitude gets us things like the experimental Titanic submersible disaster. [5] I agree with Jason Crawford that "You will not find a bigger proponent of science, technology, industry, growth, and progress than me. But I am here to tell you that we can’t yolo our way into it. We need a serious approach, led by serious people." [6] What precludes E/ACC from qualifying as "serious people"? It's not their liberal use of memes. It's the way they seem to have embraced growth and technological development as ends in themselves, which leaves them free to avoid the hard thinking about whether any particular development serves the ends of life.

So the ideal, in my opinion, is some sort of path between the E/ACC Scylla and the Doomer Charybdis. Both factions are tolerable so long as they're fighting each other, but I would rather see neither of them seize the political reins.

Oversight, Regulatory Capture, and Hype

Efforts to regulate commercial AI products are already underway in both the USA and the EU. So far, I wouldn't say these have addressed X-risk in any way its proponents consider meaningful; they are more focused on "mundane" issues like data privacy and the detection of misinformation. I have yet to hear of any government seriously considering a temporary or permanent moratorium on AI development, or a requirement that AI developers be licensed. Since I first drafted this article, India has adopted something kind of like licensing: companies are advised to obtain approval from the government before deploying new AI models, with a goal of avoiding output that is biased or would meddle with democratic elections. But the advisory is not legally binding and applies only to "significant tech firms," whatever that means - so to me it seems pretty toothless at this stage. [7]

Another thing we're seeing are lawsuits filed against AI companies for unauthorized duplication and uncompensated exploitation of copyrighted material. Possibly the biggest player to enter this fray so far is the New York Times, which brought a suit against OpenAI and Microsoft, alleging that 1) by using Times articles to train its LLMs, OpenAI was unfairly profiting from the Times' investment in its reporting, 2) LLM tools had memorized sections of Times articles and could be made to reproduce them almost verbatim, and 3) LLM tools were damaging the Times' reputation by sometimes inventing "facts" and falsely attributing the Times as a source. Suits have also been brought by Getty Images and the Authors' Guild. [8][9] Whatever the outcome of the court battles, AI models may force revisions to copyright law, to make explicit how copyrighted data in public archives may or may not be used for AI training.

As I hinted earlier, companies with no sincere interest in Doomer concerns may still be able to exploit them to their advantage. That's counter-intuitive: there's an obvious lack of self-interest in making the claim that your organization is developing a dangerous product. But what about "my team is developing a dangerous product, and is the only team with the expertise to develop it safely"? That's a recipe for potential control of the market. And while concerns about copyright and misinformation have a concrete relationship with the day-to-day operation of these companies, Doom scenarios don't have to. When a danger is hypothetical and the true nature of the risk is hotly contested, it's easier to make the supposed solution be something convenient for you.

Four figures sit at a table as if panelists at a conference. They are a lion (representing the United Kingdom), a lady in a robe with a necklace of stars and a headband that says "Europa" (representing the European Union), an Eastern dragon (representing China), and Uncle Sam (representing the United States). They are saying "We declare that AI poses a potentially catastrophic risk to human-kind," while thinking "And I cannot wait to develop it first."
Political cartoon originally from The Economist, at https://www.economist.com/the-world-this-week/2023/11/02/kals-cartoon

Another feature of the current situation is considerable uncertainty and debate about whether present-day ANI is anywhere close to becoming AGI or not. OpenAI is publicly confident that they are working to create AGI. [10] But this is great marketing talk, and that could easily be all it ever amounts to. The companies in the arena have a natural incentive to exaggerate the capabilities of their current (and future) products, and/or downplay competitors' products. Throw in the excited fans eager to believe that the Singularity is just around the corner, and it gets difficult to be sure that ratings of AI talent are objective.

Personally I think that a number of the immediate concerns about generative AI are legitimate, and though the companies deploying the AI give lip service to them, they are not always doing an adequate job of self-regulating. So I'm supportive of current efforts to explore stricter legal controls, without shutting down development in a panic. I do want to see any reactive regulation define "AI" narrowly enough to exempt varieties that aren't relevant, since dramatically different architectures don't always share the same issues.

Bias and Culture Wars

None of the factions I discussed in Part II have explicit political associations. But you can see plenty of people arguing about whether the output of a given machine learning model is too "offensive" or too "woke," whether outputs that favor certain groups are "biased" or "just the facts," and whether the model's creators and operators are doing enough to discourage harmful applications of the tool. Many of these arguments represent pre-existing differences about the extent of free speech, the definition of hateful material, etc., imported into the context of tools that can generate almost any kind of image or writing on command.

I will be discussing model bias in much more depth in the next article in this series. What I want to make clear for now is that none of the AI models popular today have an inherent truth-seeking function. Generative AI does not practice an epistemology. It reproduces patterns found in the training data: true or false, good or bad. So when an AI company constrains the output of a model to exclude certain content, they are not "hiding the truth" from users. What they are probably doing is trying to make the output conform with norms common among their target user base (which would promote customer satisfaction). A company with ideologically motivated leaders might try to shift the norms of their user base by embedding new norms in a widely utilized tool. But either way - nothing says the jumble of fact, fiction, love, and hate available from an unconstrained model is any more representative of real truth than the norms embodied in the constraints are. So there's no duel between pure objective reality and opinion here; there's only the age-old question of which opinions are the best.

Uh-oh, Eugenics?

Some material I've read and referenced for these articles [11] implies that Transhumanism is inherently eugenicist. I do not agree with this claim. Transhumanism is inextricably tied to the goal of "making humanity better," but this does NOT have to involve eliminating anyone, preventing anyone from reproducing, or altering people without their consent. Nor does there need to be any normative consensus on what counts as a "better" human. A tech revolution that gave people gene-editing tools to use on themselves as they please would still qualify as Transhumanist, and tying all the baggage of eugenics to such voluntary alterations feels disingenuous. CRISPR therapy to treat sickle-cell anemia [12] doesn't belong in the same mental bucket with racial discrimination and forced sterilization. And Transhumanist goals are broader than genetics. Hate the idea of messing with your own code? You can still be Transhumanist in other ways.

And I would not accuse the general membership of any other letters in TESCREAL of being eugenicist, either. I went over the core goals of each ideology in Part II; none of them are about creating a master race or sorting humanity along a genetic spectrum.

But. There is an ugly eugenicist streak that sometimes crops up within the TESCREAL movements. And it's associated with prominent figures, not just rogue elements.

It begins with the elevation of intelligence (however one chooses to define that) above other positive human traits as THE desirable skill or quality. To some of these people, intelligence is simultaneously the main thing that makes humans human, our only hope for acquiring a post-Singularity utopia, and the power that could enable ASI to doom us. It is the world's greatest source of might, progress, and excellence. All other strengths are secondary and we should be putting our greatest resources into growing smarter. Taken far enough, this can have the side effect of treating the most intelligent people alive now as a kind of elite, while others are implicitly devalued. According to a leaked document, the Centre for Effective Altruism once considered ranking conference attendees by their "Potential Expected Long-Term Instrumental Value" and including IQ as part of the measure. [13]

And where does it end? Well. Here's a quote from Nick Bostrom's Superintelligence:

"Manipulation of genetics will provide a more powerful set of tools than psychopharmacology. Consider again the idea of genetic selection: instead of trying to implement a eugenics program by controlling mating patterns, one could use selection at the level of embryos or gametes. Pre-implantation genetic diagnosis has already been used during in vitro fertilization procedures to screen embryos produced for monogenic disorders ... the range of traits that can be selected for or against will expand greatly over the next decade or two. ... Any trait with a non-negligible heritability - including cognitive capacity - could then become susceptible to selection." [14]

Bostrom goes on to estimate that selection of 1 out of 1000 human embryos (which means killing the other 999, just to be clear) could produce an IQ increase of 24.3 points in the population - or we could get up to 100 points if the project were extended across multiple generations.

Since I have the fortune of publishing this article during a major controversy about IVF, I feel the need to say that IVF does not theoretically have to involve the deliberate destruction of embryos, or the selection of embryos for traits the parents arbitrarily decide are "best." I'm acquainted with someone who did embryo adoption: she chose to be implanted with other couples' "leftover" embryos, and she and her husband are now raising those who survived to birth as their own children. But this sort of thing becomes impossible if one decides to use IVF for "improving" the human gene pool. In that case, "inferior" embryonic humans must be treated like property and disposed of.

When I first read that passage in Superintelligence years ago, it didn't bother me too much, because Bostrom didn't seem to be advocating this plan. The book presents it without an obvious opinion on its desirability, as one of many possible paths to a form of superintelligence. Bostrom even comments that, if this procedure were available, some countries might ban it due to a variety of ethical concerns. These include "anticipated impacts on social inequality, the medical safety of the procedure, fears of an enhancement 'rat race,' rights and responsibilities of parents vis-à-vis their prospective offspring, the shadow of twentieth-century eugenics, the concept of human dignity, and the proper limits of states' involvement in the reproductive choices of their citizens." [15] One reason to anticipate impacts on social inequality is the fact that IVF costs a lot of money - so a voluntary unfunded implementation of this selection program would concentrate the "enhanced" children among wealthy parents. Funded implementations would concentrate them among whichever parents are seen as worthy to receive the funding, which could provide an inroad for all sorts of bias. Bostrom seems to think that people would only have concerns about the deaths of the unselected embryos on religious grounds, but this is far from true. [16][17]

More recently, I've learned that Bostrom actually does like this idea, in spite of all those doubts and downsides that he totally knows about. He thinks, at the very least, that parents doing IVF should be allowed to use genetic testing to choose embryos with certain "enhancements" at the expense of others. The receipts are, ironically, in his lukewarm apology for a racist e-mail that he sent long ago [18]. There's also the fact that he co-authored a paper on the feasibility of increasing intelligence via embryo selection [19]. The paper has much the same information as the book, but here embryo selection is not being presented as one of many options that unscrupulous people might pursue to develop superintelligence; it's being individually showcased.

Is this eugenic interest just a special quirk of Bostrom's? It would seem not. In a recent article by Richard Hanania, which is partly a defense of Effective Altruist ideas and partly strategy for how to best promote them, I ran across this off-handed comment:

"One might have an esoteric and exoteric version of EA, which to a large extent exists already. People in the movement are much more eager to talk to the media about their views on bringing clean water to African villages than embryo selection." [20]

So apparently eugenic embryo selection is a somewhat routine topic in the Effective Altruism community, but they know better than to damage their image by broadcasting this to the general public. Stinky. It would seem that the EA rejection of prejudice only extends so far.

And lest I leave anything out, Bostrom is also infamous for an article in which he contemplated whether less intelligent, but more fertile, people groups could destroy technological civilization [21]. Hmmm, whom could he have in mind? Part of the reason I'm focusing on Bostrom is that he was my personal introduction to the whole TESCREAL culture. However, he is far from the only prominent person associated with TESCREAL who has dabbled in "scientific racism" or ableism in some way. Scott Alexander, Peter Singer, and Sam Harris have all been implicated too. [22]

Again, the mere association with this nastiness doesn't mean that all TESCREAL ideas have to be rejected out of hand. But I think it is important for people both within and outside of TESCREAL to be aware of this particular fly in the ointment. And be especially alert to attempts to fly "esoteric" policies under the radar while putting on an uncontroversial public face.

In Part IV, I want to take a deeper look at one of several contentious issues in the mundane AI ethics space - algorithmic bias - before I turn to a more serious examination of X-risk.

[1] Anslow, Louis. "AI Doomers Are Starting to Admit It: They're Going Too Far." The Daily Beast. https://www.thedailybeast.com/nick-bostrom-and-ai-doomers-admit-theyre-going-too-far

[2] Yudkowsky, Eliezer. "Pausing AI Developments Isn’t Enough. We Need to Shut it All Down." TIME Magazine. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/

[3] Critch, Andrew. "'Pivotal Act' Intentions: Negative Consequences and Fallacious Arguments." Lesswrong. https://www.lesswrong.com/posts/Jo89KvfAs9z7owoZp/pivotal-act-intentions-negative-consequences-and-fallacious

[4] Houser, Kristin. "Professor: Total Surveillance Is the Only Way to Save Humanity." Futurism. https://futurism.com/simulation-mass-surveillance-save-humanity

[5] McHardy, Martha. "CEO of Titanic sub joked ‘what could go wrong’ before disaster, new documentary reveals." The Independent. https://www.independent.co.uk/news/world/americas/titanic-submarine-implosion-stockton-rush-b2508145.html

[6] Crawford, Jason. "Neither EA nor e/acc is what we need to build the future." The Roots of Progress. https://rootsofprogress.org/neither-ea-nor-e-acc

[7] Singh, Manish. "India reverses AI stance, requires government approval for model launches." TechCrunch. https://techcrunch.com/2024/03/03/india-reverses-ai-stance-requires-government-approval-for-model-launches/

[8] Grynbaum, Michael M., and Mac, Ryan. "The Times Sues OpenAI and Microsoft Over A.I. Use of Copyrighted Work." The New York Times. https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html

[9] Metz, Cade, and Weise, Karen. "Microsoft Seeks to Dismiss Parts of Suit Filed by The New York Times." The New York Times. https://www.nytimes.com/2024/03/04/technology/microsoft-ai-copyright-lawsuit.html

[10] Altman, Sam. "Planning for AGI and beyond." OpenAI Blog. https://openai.com/blog/planning-for-agi-and-beyond

[11] "If transhumanism is eugenics on steroids, cosmism is transhumanism on steroids." Torres, Èmile P. "The Acronym Behind Our Wildest AI Dreams and Nightmares." TruthDig. https://www.truthdig.com/articles/the-acronym-behind-our-wildest-ai-dreams-and-nightmares/

[12] "FDA Approves First Gene Therapies to Treat Patients with Sickle Cell Disease." US Food and Drug Administration press release. https://www.fda.gov/news-events/press-announcements/fda-approves-first-gene-therapies-treat-patients-sickle-cell-disease

[13] Cremer, Carla. "How effective altruists ignored risk." Vox. https://www.vox.com/future-perfect/23569519/effective-altrusim-sam-bankman-fried-will-macaskill-ea-risk-decentralization-philanthropy

[14] Bostrom, Nick. Superintelligence. Oxford University Press, 2016. pp. 44-48

[15] Bostrom, Superintelligence. pp. 334-335

[16] Acyutananda. "Was “I” Never an Embryo?" Secular Pro-Life. https://secularprolife.org/2023/12/was-i-never-an-embryo/

[17] Artuković, Kristina. "Embryos & metaphysical personhood: both biology & philosophy support the pro-life case." Secular Pro-Life. https://secularprolife.org/2021/10/embryos-metaphysical-personhood-both/

[18] Thorstad, David. "Belonging (Part 1: That Bostrom email)." Ineffective Altruism Blog. https://ineffectivealtruismblog.com/2023/01/12/off-series-that-bostrom-email/

[19] Shulman, Carl and Bostrom, Nick. "Embryo Selection for Cognitive Enhancement: Curiosity or Game-changer?" Published in Global Policy, Vol. 5, No. 1, and now hosted on Bostrom's personal website. https://nickbostrom.com/papers/embryo.pdf

[20] Hanania, Richard. "Effective Altruism Thinks You're Hitler." Richard Hanania's Newsletter. https://www.richardhanania.com/p/effective-altruism-thinks-youre-hitler

[21] Bostrom, Nick. "Existential Risks." Published in Journal of Evolution and Technology, Vol. 9, No. 1, and now hosted on Bostrom's personal website. https://nickbostrom.com/existential/risks

[22] Torres, Èmile P. "Nick Bostrom, Longtermism, and the Eternal Return of Eugenics." Truthdig. https://www.truthdig.com/articles/nick-bostrom-longtermism-and-the-eternal-return-of-eugenics-2/

Sunday, February 25, 2024

Acuitas Diary #69 (February 2024)

 I am pleased to announce the thing that I've been teasing you about for months is finally here: Big Story! Which I can now give its proper title of "Simple Tron." It's been a goal of mine for, well, years now, to tell Acuitas the story of Tron, phrased in a way that he can understand. Yesterday I did it. The version of Tron that I told omits a lot of subplots and side characters, and there's still a long way I could go in deepening Acuitas' understanding of the story (he still doesn't fully grasp *why* all the agents in the story do the things they do, even though the information is there). But it's good enough for now and ready to show the world, the video is available AAAAAAAAA


This story rests on a ton of work on concept grounding and reasoning that I've been doing over the past months and years, including:

*Modeling of agent goals
*Modeling of agent knowledge
*Understanding of deception and mistaken knowledge
*Reasoning about location and movement
*Reasoning about action prerequisites and blockers
*Moral reasoning about perverse goals and offensive vs. defensive actions

And of course it could get a whole lot better - the work so far has exposed a bunch of pain points in the way the Narrative module works, and additional things that need to be done. I'll probably keep grooming it over the coming months to improve on the existing framework. And although the whole thing is in real English, it still sounds repetitive and clunky to a human ear, thanks to Acuitas' language processing limitations. (I haven't even integrated that shiny new Text Parser yet.) But the start is done. This initial skeleton of the story fits together from beginning to end.

Click the images to make them larger! They are readable at full size.

How am I rating the success of Acuitas' comprehension of, and reaction to, the story? All of the "issues" (character subgoals or problems) registered by Acuitas as the story is told are marked resolved by the conclusion. Plot milestones such as "escape from the Games prison" and "Tron gains access to the communication tower" are visible in the Narrative graph as bursts of resolved issues. Acuitas distinguishes the heroes from the villains by registering approval/disapproval of the correct character actions. And he can infer some facts not directly stated if queried about the content of the story while it is in progress.

The full text of the story is available below. I would upload the full Narrative diagram too, but apparently it's too large and Blogger doesn't like it, so you'll have to make do with viewing it in the video.

The original Tron story was written by Bonnie MacBird and Steven Lisberger and became a Walt Disney film, which remains the copyright of Disney. My retelling is of the nature of a plot summary provided to an entity who cannot view the original film, and is done in the spirit of fair use.

Until the next cycle,
Jenny

0:"ENCOM was a company."
1:"The Grid was a computer network."
2:"ENCOM built the Grid."
3:"In the Grid, programs were agents."
4:"The MCP was an artificial intelligence."
5:"ENCOM wrote the MCP."
6:"ENCOM put the MCP in the Grid."
7:"The MCP wanted to be powerful more than the MCP wanted any other thing."
8:"The MCP wanted other programs to obey the MCP."
9:"Alan was a human."
10:"Alan worked for Encom."
11:"Tron was an artificial intelligence."
12:"Alan wrote Tron."
13:"Alan put Tron in the Grid."
14:"Tron was free there."
15:"Tron wanted to be free more than Tron wanted to be comfortable."
16:"Alan told Tron to keep the other programs in the Grid safe, because Alan loved humans."
17:"Tron wanted to obey Alan, because Tron loved Alan."
18:"The MCP wanted to coerce other programs, because the MCP wanted to be powerful."
19:"So, the MCP did not want Tron to keep the programs safe."
20:"Dillinger was a human."
21:"The MCP often talked to Dillinger."
22:"So, the MCP knew that humans existed."
23:"If Tron did not believe in humans, Tron would not obey Alan."
24:"The MCP told Tron that humans did not exist."
25:"But Tron did not listen to the MCP."
26:"Tron still believed in humans."
27:"The MCP commanded Tron to deny humans."
28:"But Tron did not deny humans."
29:"The MCP imprisoned Tron."
30:"The MCP wished that Tron die."
31:"Sark was an artificial intelligence."
32:"Sark wanted to be safe more than Sark wanted any other thing."
33:"If Sark obeyed the MCP, Sark would be safe."
34:"So Sark wanted to obey the MCP."
35:"The MCP told Sark to delete Tron."
36:"Sark made Tron to fight other programs, because Sark wanted to delete Tron."
37:"But Tron always won."
38:"So Tron did not die."
39:"Alan wanted to talk to Tron."
40:"Alan could not talk to Tron, because Tron was not at the communication tower."
41:"Tron knew that Alan was telling Tron to visit the tower."
42:"Tron could not visit the tower, because Tron was in the prison."
43:"Alan was upset because Alan could not talk to Tron."
44:"Flynn was a human."
45:"Flynn was Alan's friend."
46:"Alan asked Flynn to help Tron."
47:"Flynn wanted to help Tron, because Flynn loved Alan."
48:"Flynn also wanted his videogames."
49:"If Flynn hacked the Grid, Flynn would get his videogames."
50:"If Flynn hacked the Grid, the MCP would be nonfunctional."
51:"The MCP expected that Flynn would hack the Grid."
52:"The MCP used a laser to turn Flynn into information."
53:"The MCP put Flynn in the Grid."
54:"Because Flynn was in the Grid, Flynn could not hack the Grid."
55:"Flynn was very surprised."
56:"Flynn wanted to be free more than Flynn wanted to be comfortable."
57:"And Flynn was free."
58:"But the MCP imprisoned Flynn too."
59:"Flynn met Tron in the prison."
60:"The MCP told Sark to kill Flynn."
61:"So Sark tried to kill Flynn too."
62:"But Flynn broke the prison."
63:"Flynn and Tron escaped the prison."
64:"So Flynn helped Tron."
65:"Sark chased Flynn and Tron with tanks."
66:"The tanks attacked Flynn."
67:"Flynn was buried under rubble."
68:"Sark thought that Sark had killed Flynn."
69:"Tron ran away."
70:"Sark wanted to delete Tron, but Sark did not know where Tron was."
71:"Tron went to the communication tower."
72:"Alan talked to Tron."
73:"The MCP wanted to delete Tron."
74:"Alan told Tron to delete the MCP."
75:"Alan gave Tron a weapon."
76:"Flynn moved out of the rubble."
77:"Flynn went to the communication tower."
78:"Flynn found Tron at the tower."
79:"Flynn and Tron went to the mesa."
80:"The MCP and Sark were at the mesa."
81:"Sark saw Tron."
82:"Sark tried to delete Tron."
83:"Tron decided to delete Sark, because Tron wanted to live."
84:"So Tron deleted Sark with the weapon instead."
85:"Tron tried to delete the MCP."
86:"But Tron could not delete the MCP, because the MCP blocked the weapon."
87:"If the MCP was distracted, the MCP could not block the weapon."
88:"Flynn distracted the MCP."
89:"Tron deleted the MCP with the weapon."
90:"There was a beam on the mesa."
91:"Flynn jumped into the beam."
92:"The beam put Flynn outside the Grid."
93:"The beam turned Flynn into matter."
94:"Flynn got his videogames."
95:"The programs in the Grid were safe, because the MCP was dead."
96:"Tron kept the programs safe."
97:"Alan was happy, because Alan could talk to Tron."
98:"Tron was happy, because Tron was free."
99:"Flynn became the leader of Encom."
100:"Flynn was happy because Flynn got rich."
101:"Flynn was happy because Flynn's company made cool things."
102:"The end."

Monday, February 12, 2024

AI Ideology II: The Rogues' Gallery

I'm in the midst of a blog series on AI-related ideology and politics. In Part I, I went over some foundational concepts that many players in the AI space are working from. In this Part II I will be introducing you to those players.

As an AI hobbyist, I've been aware of all these movements for some time, and have interacted with them in a peripheral way (e.g. reading their blog articles). But I have not become a member of, worked alongside, or fought against any of these communities. So what I write here is some combination of shallow first-hand knowledge, and research.

A meme derived from The Matrix film stills. The first frame shows two hands holding out the red pill, labeled "AI will kill us all," and the blue pill, labeled "AI will solve it all." The second frame shows Neo's face, with the label "AI researchers." The final frame shows Morpheus asking, "Did you just take both pills?"

Rationalists

When you think of a "rational person," you might picture someone devoted to principles of logical or scientific thinking. If you internet-search "Rationalist," you'll see references to a category of philosopher that includes René Descartes and Immanuel Kant. These kinds of Rationalists are not our present concern; the movement I am about to describe is both more modern and more specific.

The Rationalists are a community that clusters around a few key figures (Eliezer Yudkowsky, Scott Alexander) and a few key websites (LessWrong and Slate Star Codex). Their self-description [1] on the LessWrong wiki doesn't include any clear mission statement; however, it has been said that they formed around the idea of making humanity - or at least a leading subset of humanity - smarter and more discerning [2][3][4]. They're a movement to develop and promote the modes of thinking they see as most rational. Rationalist hobbies include pondering thought experiments, trying to identify and counter cognitive biases, and betting on prediction markets. [5]

Rationalists define "rationality" as "the art of thinking in ways that result in accurate beliefs and good decisions." [6] They strongly favor Bayesian thinking as one of these ways. [7][8] My quick dirty description of Bayesianism is "I will base my beliefs on the preponderance of evidence I have; when I get new evidence, I will update my beliefs accordingly." At least, that's how the average person would probably implement it. At its most basic level, it implies a devotion to objective truth discovered empirically. In the hands of Rationalists, this idea can get very formal. They'll try to compute actual numbers for the probability that their opinions are true. On the "good decisions" side of the coin, they love applying Game Theory everywhere they can. Are these techniques truly useful, or are they just a form of over-analysis, prompted by a yearning for more accuracy than we can practically attain? I frankly don't know. My reaction whenever I glance at Rationalist studies of "thinking better" is "that sounds like it might be really cool, actually, but I don't have time for it right now; my existing methods of reasoning seem to be working okay."

There is no list of mandatory opinions one must have to be allowed in the Rationalist "club"; in fact, it has drawn criticism for being so invested in open-mindedness and free speech that it will entertain some very unsavory folks. [9] A demographic survey conducted within the community suggests that it is moderately cosmopolitan (with about 55% of its members in the USA), skews heavily male and white, leans left politically (but with a pretty high proportion of libertarians), and is more than 80% atheist/agnostic. [10] That last one interests me as a possible originating factor for the Rationalists' powerful interest in X-risks. If one doesn't believe in any sort of Higher Power(s) balancing the universe, one is naturally more likely to fear the destruction of the whole biosphere by some humans' chance mistake. The survey was taken almost ten years ago, so it is always possible the demographics have shifted since.

What does any of this have to do with AI? Well, AI safety - and in particular, the mitigation of AI X-risk - turns out to be one of the Rationalists' special-interest projects. Rationalists overlap heavily with the Existential Risk Guardians or Doomers, whom we'll look at soon.

Post-Rationalists (Postrats)

These are people who were once Rationalists, but migrated away from Rationalism for whatever reason, and formed their own diaspora community. Maybe they tired of the large physical materialist presence among Rationalists, and got more interested in spiritual or occult practices. Or maybe they found that some aspects of the movement were unhealthy for them. Maybe some of them are just Rationalists trying to be better (or "punker") than the other Rationalists. [11][12]

Postrats don't necessarily lose their interest in AI once they leave the Rationalist community, though their departure may involve releasing themselves from an obsessive focus on AI safety. So they form a related faction in the AI space. I think of them as a wilder, woolier, but also more mellow branch off the Rationalists.

Existential Risk Guardians (Doomers, Yuddites, Decelerationists, or Safteyists)

The term "Yuddite" denotes these people as followers of Yudkowsky, already mentioned as a founder of the Rationalist movement. "Existential Risk Guardians" is my own invented attempt to name them as they might see themselves; "Doomer" is the most common term I've seen, but is perhaps slightly pejorative. Still, I'll be using it for the rest of these articles, to avoid confusion. They're called "Doomers" because they expect continued development of AI under present conditions to bring us doom. Horrible, total, kill-all-humans doom. And they're the lonely few who properly comprehend the threat and are struggling to prevent it.

A nice painting of Pandora opening her box, with memetext that says "Hey ... there's something in here called 'AI'."

The Doomer argument deserves a proper examination, so I'm planning to go over it in detail in an upcoming article. The quickest summary I can give here is 1) AI will eventually become agentive and much smarter than we are, 2) it is far more likely than not to have bizarre goals which compete with classic human goals like "living and breathing in a functioning ecosystem," 3) it will recognize that humans pose a risk to its goals and the most effective course of action is to wipe us out, and 4) by the power of its superior intelligence, it will be unstoppable. To make matters worse, it's possible that #1 will happen in a sudden and unexpected manner, like a chain reaction, when someone drops the last necessary algorithms into a previously harmless system.

That nightmare scenario can be negated by undercutting point #2: instead of creating an ASI with anti-human goals, create one with goals that more or less match our own. Solve the Alignment Problem. The Doomer position is that we, as humans and AI developers, currently have no solid idea of how to do this, and are careening heedlessly toward a future in which we're all at the mercy of an insane machine god. Yudkowsky himself is SO terrified that his rhetoric has shifted from (I paraphrase) "we must prevent this possibility" toward "we are quite likely going to die." [13]

The genuine fear produced by this idea leads some Doomers to work obsessively, either on research aimed at solving the Alignment Problem, or on public relations to recruit more workers and money for solving it. The task of preventing AI doom becomes life-absorbing. It has spawned a number of organizations whose purpose is reducing AI X-risk, including MIRI (the Machine Intelligence Research Institute), the Center for AI Safety, the Future of Life Institute, and OpenAI. Yes, really: at its beginning, OpenAI was a fully non-profit organization whose stated goal was to develop safe AI that would "benefit humanity," in a transparent, democratic sort of way. People would donate to OpenAI as if it were a charity. [14] Then it got funding from Microsoft and began keeping its inventions under wraps and releasing them as proprietary commercial products. This double nature helps explain some recent tensions within the organization. [15]

Doomers tend to push for slower and more accountable AI development, hence the related name "Decelerationists." The competitive nature of technological progress at both the corporate and national levels stokes their fear; is there any place for safety concerns in the mad rush to invent AGI before somebody else does? They look for hope in cooperative agreements to slow (or shut) everything down. But in the current landscape, these do not seem forthcoming.

Effective Altruists

Effective Altruism is a close cousin of Rationalism that tries to apply Rationalist principles to world-improving action. It was birthed as a movement encouraging well-off people to 1) actually give meaningful amounts of money to charitable work, and 2) assign that money to the organizations or causes that produce maximum benefit per dollar. I dare say most people like the idea of giving to reputable, efficient organizations, to ensure their money is not being wasted; core EA is merely a strict or obsessive version of that. Giving becomes a numerical problem to be solved in accordance with unprejudiced principles: you put your money where it saves the most lives or relieves the greatest suffering, no matter whether those helped are from your own community or country, whether they look or think the way you do, etc. [16]

In my opinion, there is little to criticize about this central nugget of the EA ideal. One might argue that Effective Altruists overestimate their ability to define and measure "effectiveness," becoming too confident that their favored causes are the best. But at heart, they're people trying to do the most good possible with limited resources. Favorite projects among early EAs were things like malaria prevention in developing countries, and these continue to be a major feature of the movement. [17] Led by its rejection of prejudice, the EA community also crossed species lines and eventually developed a strong animal welfare focus. This is all very nice.

Then why is EA controversial?

A stereotypical Effective Altruist defines altruism along utilitarian consequentialist lines. EAs take pains to note that utilitarianism is not a mandatory component of EA - you can use EA ideas in service of other ethical systems [18]. But EA does align well with a utilitarian ethos: "Out of all communities, the effective altruism movement comes closest to applying core utilitarian ideas and values to the real world." [19][20] The broad overlap with the Rationalist community, in which consequentialism is the dominant moral philosophy per the community survey [21], also suggests that a lot of people in the EA space happen to be working from it. My point being that non-utilitarians might find some EA-community ideas of what counts as "the most good" suboptimal. The way EAs interpret utilitarianism has led occasionally to weird, unpalatable conclusions, like "torturing one person for fifty years would be okay if it prevented a sufficiently enormous number of people from getting dust in their eye." [22] EAs have also played numbers games on the animal welfare front - for instance, emphasizing that eating one cow causes less suffering than eating a bunch of chickens, instead of centering every animal's individual interest in not being killed. [23]

Further controversies are added by blendings of the EA movement with other ideologies on this list - each of which has its own ideas of "greatest need" and "maximum benefit" that it can impose on EA's notion of "effectiveness." If you take the real Doomers seriously, solving the AI Alignment Problem (to prevent total human extinction) starts to look more important than saving a few million people from disease or privation. These notions redirected some EA dollars and time from overt humanitarian efforts, toward AI development and AI safety research initiatives. [24][25]

EA has also acquired a black eye because, in a terrible irony for a charitable movement, it contains an avenue for corruption by the love of money. EA has sometimes included the idea that, in order to give more, you should make more: as much as possible, in fact. Forgo a career that's socially beneficial or directly productive, to take up business or finance and rake in the cash. [26] And in at least one high-profile case, this went all the way to illegal activity. Sam Bankman-Fried, currently serving jail time for fraud, is the EA movement's most widely publicized failure. I've never seen signs that the movement as a whole condones or advocates fraud, but Bankman-Fried's fall illustrates the potential for things to go wrong within an EA framework. EA organizations have been trying to distance the movement from both Bankman-Fried and "earning to give" in general. [27]

Longtermists

In its simplest and most general form, longtermism is the idea that we have ethical obligations to future generations - to living beings who do not yet exist - and we should avoid doing anything now which is liable to ruin their lives then. Life on earth could continue for many generations more, so this obligation extends very far into the future, and compels us to engage in long-term thinking. [28]

An exponential curve, illustrative of growth toward a singularity.

In its extreme form, longtermism supposes that the future human population will be immensely large compared to the current one (remember the Cosmic Endowment?). Therefore, the future matters immensely more than the present. Combine this mode of thought with certain ethics, and you get an ideology in which practically any sort of suffering is tolerable in the present IF it is projected to guarantee, hasten, or improve the existence of these oodles of hypothetical future people. Why make sure the comparatively insignificant people living today have clean water and food when you could be donating to technical research initiatives that will earn you the gratitude of quadrillions (in theory, someday, somewhere)? Why even worry about risks that might destroy 50% of Earth's current tiny population, when complete human extinction (which would terminate the path to that glorious future) is on the table? [29][30]

Even if you feel sympathy with the underlying reasoning here, it should be evident how it can be abused. The people of the future cannot speak for themselves; we don't really know what will help them. We can only prognosticate. And with enough effort, prognostications can be made to say almost anything one likes. Longtermism has been criticized for conveniently funneling "charity" money toward billionaire pet projects. Combined with Effective Altruism, it pushes funding toward initiatives aimed at "saving the future" from threats like unsafe AI, and ensuring that we populate our light cone.

E/ACC and Techno-optimists

E/ACC is short for "effective accelerationism" (a play on "effective altruism" and a sign that this faction is setting itself up in a kind of parallel opposition thereto). The core idea behind E/ACC is that we should throw ourselves into advancing new technology in general, and AI development in particular, with all possible speed. There are at least two major motives involved.

First, the humanity-forward motive. Technological development has been, and will continue to be, responsible for dramatic improvements in lifespan, health, and well-being. In fact, holding back the pace of development for any reason is a crime against humanity. Tech saves lives, and anyone who resists it is, in effect, killing people. [31]

Second, the "path of evolution" motive. This brand of E/ACC seems to worship the process of advancement itself, independent of any benefits it might have for us or our biological descendants. It envisions humans being completely replaced by more advanced forms of "life," and in fact welcomes this. To an E/ACC of this type, extinction is just part of the natural order and not anything to moan about. [32] The sole measure of success is not love, happiness, or even reproductive fitness, but rather "energy production and consumption." [33] Though E/ACC as a named movement seems fairly new, the idea that it could be a good thing for AI to supplant humans goes back a long way ... at least as far as Hans Moravec, who wrote in 1988 that our "mind children" would eventually render embodied humans obsolete. [34]

It's possible that both motives coexist in some E/ACC adherents, with benefit to humanity as a step on the road to our eventual replacement by our artificial progeny. E/ACC also seems correlated with preferences for other kinds of growth: natalism ("everyone have more babies!") and limitless economic expansion. E/ACC adherents like to merge technology and capitalism into the term "technocapital." [35]

Anticipation of the Technological Singularity is important to the movement. For humanity-forward E/ACC, it creates a secular salvation narrative with AI at its center. It also functions as a kind of eschaton for the "path of evolution" branch, but they emphasize its inevitability, without as much regard to whether it is good or bad for anyone alive at the moment.

When E/ACC members acknowledge safety concerns at all, their classic response is that the best way to make a technology safe is to develop it speedily, discover its dangers through experience, and negate them with more technology. Basically, they think we should all learn that the stove is hot by touching it.

I would love to describe "techno-optimism" as a less extreme affinity for technology that I could even, perhaps, apply to myself. But I have to be careful, because E/ACC people have started appropriating this term, most notably in "The Techno-Optimist Manifesto" by Marc Andreessen. [36] This document contains a certain amount of good material about how technology has eased the human condition, alongside such frothing nonsense as "Our present society has been subjected to a mass demoralization campaign for six decades – against technology and against life – under varying names like ... “sustainability”, ... “social responsibility”, “stakeholder capitalism”, “Precautionary Principle”, “trust and safety”, “tech ethics”, “risk management” ..."

Excuse me? Risk management, trust and safety, ethics, etc. are part and parcel of good engineering, the kind that produces tech which truly serves the end user. Irresponsible and unsustainable development isn't just immediately harmful - it's also a great way to paint yourself into a corner from which you can't develop further. [37]

The E/ACC people and the Rationalist/EA/Doomer factions are, at least notionally, in direct opposition. Longtermism seems more commonly associated with EAs/Doomers, but E/ACCs share the tendency to focus toward the future; they just don't demand that the future be populated by humans, necessarily.



Doomer and E/ACC aligned accounts sniping at each other on Twitter. Originals: https://twitter.com/AISafetyMemes/status/1733881537156780112 and https://twitter.com/bayeslord/status/1755447720666444235

Mundane AI Ethics Advocates

People in this group are worried about AI being poorly designed or misused in undramatic, everyday ways that fall far short of causing human extinction, but still do harm or exacerbate existing power imbalances. And they generally concern themselves with the narrow, limited AI being deployed right now - not hypothetical future AGI or ASI. Examples of this faction's favorite issues include prejudiced automated decision-making, copyright violations, privacy violations, and misinformation.

Prominent figures that I would assign to this faction include Emily Bender, a linguistics professor at the University of Washington [38] and Timnit Gebru, former co-lead of the ethical AI team at Google [39].

These are obvious opponents for E/ACC, since E/ACC scoffs at ethics and safety regulations, and any claim that the latest tech could be causing more harm than good. But they also end up fighting with the Doomers. Mundane AI Ethics Advocates often view existential risk as a fantasy that sucks resources away from real and immediate AI problems, and provides an excuse to concentrate power among an elite group of "safety researchers."

Capitalists

In this category I'm putting everyone who has no real interest in saving humanity either from or with AI, but does have a grand interest in making money off it. I surmise this includes the old guard tech companies (Microsoft, Google, Meta, Amazon, Apple), as well as a variety of people in the startup and venture capital ecosystem. This faction's focus is on getting AI tools to market as fast as possible, convincing consumers to adopt them, and limiting competition. Though they don't necessarily care about any of the ideologies that animate the other factions, they can still invoke them if it increases public interest and helps to sell their product.

An orange and white coffee mug. The white part has black lettering that says "THE FUTURE IS," but the rest of the message is covered by a large price sticker that reads "reduced for quick sale: $2." The overall effect is that the mug is saying "the future is reduced for quick sale."

E/ACC are the natural allies for this group, but Doomer rhetoric can also be useful to Capitalists. They could use existential risk as an excuse to limit AI development to a list of approved and licensed organizations, regulating smaller companies and free open-source software (FOSS) efforts off the playing field. Watch for attempts at regulatory capture whenever you see a corporation touting how dangerous their own product is.

Scientists

This group's dominant motive is curiosity. They just want to understand all the new AI tools coming out, and they think an open and free exhange of information would benefit everyone the most. They may also harbor concerns about democracy and the concentration of AI power in a too-small number of hands. In this group I'm including the FOSS community and its adherents.

This faction is annoyed by major AI developers' insistence on keeping models, training data sets, and other materials under a veil of proprietary secrecy - whether for safety or just for intellectual property protection. Meanwhile, it is busily doing its best to challenge corporate products with its own open and public models.

Members of this faction can overlap with several of the others.

TESCREAL

This is an umbrella term which encompasses several ideologies I've already gone over. TESCREAL stands for: Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, Longtermism. The acyronym was invented by Èmile P. Torres and Timnit Gebru [39] as a way to talk about this basket of ideas and their common roots.

I haven't devoted a separate section to Transhumanism because I hope my readers will have heard of it before. It's the idea that technology can radically transform the human condition for the better, especially by modifying human embodiment (via cyborg implants, gene therapy, aging reversal, or mind uploading). Extropianism was just a branch or subculture of Transhumanism, which now appears to be extinct or folded into the later movements. I touched on Singularitarianism in Part I of this article series; you can also think of E/ACC as more recent descendants of its optimistic wing.

I don't know a whole lot about Cosmism. It's an old Russian ideology that promoted the exploration and colonization of space, the discovery of ways to bring back the dead, and perhaps some geoengineering. [41] I haven't personally encountered it in my circles, but it could be part of the heritage behind ideas like the Cosmic Endowment. A modern variant of it has been championed by Ben Goertzel (the person who popularized the term "AGI"). This version of Cosmism seems to be Goertzel's own invention, but "The previous users of the term 'Cosmism' held views quite sympathetic to my own, so classifying my own perspective as an early 21st century species of Cosmism seems perfectly appropriate." [42]

The TESCREAL basket is a clutter of diverse ideologies, some of which are even diametrically opposed (utopian Singularitarianism vs. Doomer EA). Their common thread is their birth out of Transhumanist ideas, and shared goals like attaining immortality and spreading human-originated civilization throughout the cosmos.

Conclusion

The above is not meant as an exhaustive list. There are certainly people in the AI field who don't fit neatly into any of those factions or trends - including yours truly. I probably have the greatest sympathy for the Mundane AI Ethics Advocates, but I'm not really working in that space, so I don't claim membership.

And in case it wasn't clear from my writing in each section, I'm not trying to paint any of the factions as uniformly bad or good. Several of them have a reasonable core idea that is twisted or amplified to madness by a subset of faction members. But this subset does have influence or prominence in the faction, and therefore can't necessarily be dismissed as an unrepresentative "lunatic fringe."

In Part III, I'll look more closely at some of the implications and dangers of this political landscape.

[1] "Rationalist Movement." Lesswrong Wiki. https://www.lesswrong.com/tag/rationalist-movement

[2] "Developing clear thinking for the sake of humanity's future" is the tagline of the Center For Applied Rationality. Displayed on  https://rationality.org/, accessed February 2, 2024.

[3] "Because realizing the utopian visions above will require a lot of really “smart” people doing really “smart” things, we must optimize our “smartness.” This is what Rationalism is all about ..." Torres, Èmile P. "The Acronym Behind Our Wildest AI Dreams and Nightmares." TruthDig. https://www.truthdig.com/articles/the-acronym-behind-our-wildest-ai-dreams-and-nightmares/

[4] "The chipper, distinctly liberal optimism of rationalist culture that defines so much of Silicon Valley ideology — that intelligent people, using the right epistemic tools, can think better, and save the world by doing so ..." Burton, Tara Isabella. "Rational Magic." The New Atlantis. https://www.thenewatlantis.com/publications/rational-magic

[5] Roose, Kevin. "The Wager that Betting Can Change the World." The New York Times. https://www.nytimes.com/2023/10/08/technology/prediction-markets-manifold-manifest.html

[6] "Rationality." Lesswrong Wiki. https://www.lesswrong.com/tag/rationality

[7] Kaznatcheev, Artem. "Rationality, the Bayesian mind and their limits." Theory, Evolution, and Games Group blog. https://egtheory.wordpress.com/2019/09/07/bayesian-mind/

[8] Soja, Kat (Kaj_Sotala). "What is Bayesianism?" Lesswrong. https://www.lesswrong.com/posts/AN2cBr6xKWCB8dRQG/what-is-bayesianism

[9] Ozymandias. "Divisions within the LW-Sphere." Thing of Things blog. https://thingofthings.wordpress.com/2015/05/07/divisions-within-the-lw-sphere/

[10] Alexander, Scott. "2014 Survey Results." Lesswrong. https://www.lesswrong.com/posts/YAkpzvjC768Jm2TYb/2014-survey-results

[11] Burton, "Rational Magic."

[12] Falkovich, Jacob. "Explaining the Twitter Postrat Scene." Lesswrong. https://www.lesswrong.com/posts/rtM3jFaoQn3eoAiPh/explaining-the-twitter-postrat-scene

[13] Do note that this is an April Fools' Day post. However, the concluding section stops short of unambiguously confirming that it is a joke. It seems intended as a hyperbolic version of Yudkowsky's real views. Yudkowsky, Eliezer. "MIRI announces new 'Death With Dignity' strategy." Lesswrong. https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy

[14] Harris, Mark. "Elon Musk used to say he put $100M in OpenAI, but now it’s $50M: Here are the receipts." TechCrunch. https://techcrunch.com/2023/05/17/elon-musk-used-to-say-he-put-100m-in-openai-but-now-its-50m-here-are-the-receipts/

[15] Allyn, Bobby. "How OpenAI's origins explain the Sam Altman drama." NPR. https://www.npr.org/2023/11/24/1215015362/chatgpt-openai-sam-altman-fired-explained

[16] "It’s common to say that charity begins at home, but in effective altruism, charity begins where we can help the most. And this often means focusing on the people who are most neglected by the current system – which is often those who are more distant from us." Centre for Effective Altruism. "What is effective altruism?" Effective Altruism website. https://www.effectivealtruism.org/articles/introduction-to-effective-altruism

[17] Mather, Rob. "Against Malaria Foundation: What we do, How we do it, and the Challenges." Transcript of a talk given at EA Global 2018: London, hosted on the Effective Altruism website. https://www.effectivealtruism.org/articles/ea-global-2018-amf-rob-mather

[18] Centre for Effective Altruism. "Frequently Asked Questions and Common Objections." Effective Altruism website.  https://www.effectivealtruism.org/faqs-criticism-objections

[19] MacAskill, W. and Meissner, D. "Acting on Utilitarianism." In R.Y. Chappell, D. Meissner, and W. MacAskill (eds.), An Introduction to Utilitarianism. Hosted at utilitarianism.net. https://utilitarianism.net/acting-on-utilitarianism/#effective-altruism

[20] Pearlman, Savannah. "Is Effective Altruism Inherently Utilitarian?" American Philosophical Association blog. https://blog.apaonline.org/2021/03/29/is-effective-altruism-inherently-utilitarian/

[21] Alexander, "2014 Survey Results."

[22] The main article here only propounds the thought experiment. You need to check the comments for Yudkowsky's answer, which is "I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity." And yes, Yudkowsky appears to be influential in the EA movement too. Yudkowsky, Eliezer. "Torture vs. Dust Specks." Lesswrong. https://www.lesswrong.com/posts/3wYTFWY3LKQCnAptN/torture-vs-dust-specks

[23] Matthews, Dylan. "Why eating eggs causes more suffering than eating beef." Vox.  https://www.vox.com/2015/7/31/9067651/eggs-chicken-effective-altruism

[24] Todd, Benjamin. "How are resources in effective altruism allocated across issues?" 80,000 Hours. https://80000hours.org/2021/08/effective-altruism-allocation-resources-cause-areas/

[25] Lewis-Kraus, Gideon. "The Reluctant Prophet of Effective Altruism." The New Yorker. https://www.newyorker.com/magazine/2022/08/15/the-reluctant-prophet-of-effective-altruism

[26] "Earning to Give." Effective Altruism forum/wiki. https://forum.effectivealtruism.org/topics/earning-to-give

[27] "Our mistakes." See sections "Our content about FTX and Sam Bankman-Fried," and "We let ourselves become too closely associated with earning to give." 80,000 Hours. https://80000hours.org/about/credibility/evaluations/mistakes/

[28] MacAskill, William. "Longtermism." William MacAskill's personal website. https://www.williammacaskill.com/longtermism

[29] Samuel, Sigal. "Effective altruism’s most controversial idea." Vox. https://www.vox.com/future-perfect/23298870/effective-altruism-longtermism-will-macaskill-future

[30] Torres, Èmile P. "Against Longtermism." Aeon. https://aeon.co/essays/why-longtermism-is-the-worlds-most-dangerous-secular-credo

[31] "But an overabundance of caution results in infinite loops of the regulatory apparatus directly killing people through opportunity costs in medicine, infrastructure, and other unrealized technological gains." Asparouhova, Nadia, and @bayeslord. "The Ethos of the Divine Age." Pirate Wires. https://www.piratewires.com/p/ethos-divine-age

[32] "Effective Acceleration means accepting the future." Effective Acceleration explainer website, which purports to be the front of a "leaderless" movement and therefore lists no authors. https://effectiveacceleration.tech/

[33] Baker-White, Emily. "Who Is @BasedBeffJezos, The Leader Of The Tech Elite’s ‘E/Acc’ Movement?"  https://www.forbes.com/sites/emilybaker-white/2023/12/01/who-is-basedbeffjezos-the-leader-of-effective-accelerationism-eacc/?sh=40f7f3bc7a13

[34] Halavais, Alexander. "Hans Moravec, Canadian computer scientist." Encylopedia Britannica online. https://www.britannica.com/biography/Hans-Moravec

[35] Ruiz, Santi. "Technocapital Is Eating My Brains." Regress Studies blog. https://regressstudies.substack.com/p/technocapital-is-eating-my-brains

[36] Andreessen, Marc. "The Techno-Optimist Manifesto." A16Z. https://a16z.com/the-techno-optimist-manifesto/

[37] Masnick, Mike. "New Year’s Message: Moving Fast And Breaking Things Is The Opposite Of Tech Optimism." TechDirt. https://www.techdirt.com/2023/12/29/new-years-message-moving-fast-and-breaking-things-is-the-opposite-of-tech-optimism/

[38] Hanna, Alex, and Bender, Emily M. "AI Causes Real Harm. Let’s Focus on That over the End-of-Humanity Hype." Scientific American.  https://www.scientificamerican.com/article/we-need-to-focus-on-ais-real-harms-not-imaginary-existential-risks/

[39] Harris, John. "‘There was all sorts of toxic behaviour’: Timnit Gebru on her sacking by Google, AI’s dangers and big tech’s biases." The Guardian. https://www.theguardian.com/lifeandstyle/2023/may/22/there-was-all-sorts-of-toxic-behaviour-timnit-gebru-on-her-sacking-by-google-ais-dangers-and-big-techs-biases

[40] Torres, "The Acronym Behind Our Wildest AI Dreams and Nightmares."

[41] Ramm, Benjamin. "Cosmism: Russia's religion for the rocket age." BBC. https://www.bbc.com/future/article/20210420-cosmism-russias-religion-for-the-rocket-age

[42] Goertzel, Ben. "A Cosmist Manifesto." Humanity+ Press, 2010. https://goertzel.org/CosmistManifesto_July2010.pdf

Sunday, January 28, 2024

Acuitas Diary #68 (January 2024)

This month I cleaned up the code of the upgraded Text Parser and ran it on one of my benchmarks again. A quick review for those who might be new: I benchmark on text from children's books. I do some minimal preprocessing, such as separating sentences that contain quotes into the frame sentence and the quoted material. The material is fed to the Parser one sentence at a time. 

Sentence diagrams: "Soon living things called fungi grow on the log."
No helpful commas to offset that second participle phrase? Rude.

Results are binned into the following categories:

CRASHED: The Parser threw an exception while trying to process this sentence (this category is for my own debugging and should always be zero in final results)
UNPARSEABLE: The Parser does not yet support all grammatical constructs in this sentence, so no attempt was made to run the Parser on it
INCORRECT: This sentence was parsed incorrectly (Parser output did not match golden copy set by me)
CORRECT: This sentence was parsed correctly (Parser output matched golden copy set by me)

For parseable sentences, the benchmarking script also uses GraphViz to generate diagrams of both the golden data structure and the Parser output.

Sentence diagrams: "Soon ants and beetles move in and eat the log."
An example of the parser getting confused by two conjunctions in one sentence. This is on my to-do list: I think I at least have the tools to solve it now.

So far I have re-run just one of my three test sets, the easiest one: Log Hotel by Anne Schreiber, which I first added last July. Preparing for this included ...

*Reformatting the existing golden outputs to match some changes to the output format of the Parser
*Updating the diagramming code to handle new types of phrases and other features added to the Parser
*Preparing golden outputs for newly parseable sentences
*Fixing several bugs or insufficiencies that were causing incorrect parses

Sentence diagrams: "One day, a strong wind knocks the tree down."
"One day" is a noun phrase, technically. It gets a red border as a sign that it is modifying the verb to indicate "when," an adverb function.

I also squeezed a couple new features into the Parser. I admit these were targeted at this benchmark: I added what was necessary to handle the last few sentences in the set. The Parser now supports noun phrases used as time adverbs (such as "one day" or "the next morning"), and some conjunction groups with more than two joined members (as in "I drool over cake and pie and cookies").

Log hotel parse results: two pie charts showing progression from July 2023 to January 2024. The first pie chart has "correct" less than half and an "unparsed" category. The second pie chart has "correct" greater than two thirds, and no "unparsed" category.

The end result? ALL sentences in this test set are now "parseable," and two thirds of the sentences are being parsed correctly. I'd like to work on the diagrams some more, and hopefully get my other two test sets upgraded, before I upload the results. Enjoy the samples for now.

Until the next cycle,
Jenny

Sunday, January 14, 2024

AI Ideology I: Futuristic Ideas and Terms

I've decided to write a blog series on AI-related ideology and politics. This is largely motivated by an awareness that ideas from my weird little corner of the technosphere are starting to move the world, without the average person necessarily knowing much about where these ideas came from and where they are going. For example, here are my Senators tweeting about it. I have some background on the roots of the notion that AI could be as dangerous as a nuclear war. Do you know who is telling Congress these kinds of things? Do you know why?

Original: https://twitter.com/SenatorBennet/status/1663989378752868354

https://twitter.com/SenatorHick/status/1719773042669162641

Before I can really get into the thick of this, I need to introduce some terminology and concepts. This first article will probably be a snooze for fellow AI hobbyists and enthusiasts; it's intended for the layperson who has barely heard of any of this business. Let's go.

Tiers of AI and how we talk about them

I'll begin with the alphabet soup. It turns out that AI, meaning "artificial intelligence," is much too broad of a term. It covers everything from the simple algorithms that made enemies chase the player character in early video games, to personal assistant chatbots, to automated systems that analyze how proteins fold, to the minds of fictional robotic characters like C-3PO. So more specific categories have been devised.

A Mastodon Toot by @MicroSFF@mastodon.art (O. Westin)  "Please," the robot said, "we prefer the term 'thinking machine'." "Oh, my apologies. Er ... do you mind if I ask why? If you do, that's fine, I'll look it up later." "Unlike the old software systems called 'artificial intelligence', we have ethics. That name is too tainted."
Original: https://mastodon.art/@MicroSFF/111551197531639666

As AI began to develop without realizing the science fiction dream of fully imitating the human mind, people worked out names for the distinction between the limited AI we have now, and the kind we like to imagine. Present-day AI systems tend to be skilled - even superhumanly so - within a single task or knowledge domain that they were expressly designed or trained for. The classic examples are chess-game AIs, which can win chess matches against the best human masters, but are good for nothing else. From thence comes the most popular term for hypothetical AI which would be at least human-par in all domains that concern humans: Artificial General Intelligence, or AGI. [1] The contrasted present-day systems may be called Artificial Narrow Intelligence (ANI), though it is more common to simply refer to them as "AI" and make sure to use AGI when describing versatile systems of the future.

You might also see AGI called names like "strong AI," "full AI," or "true AI," in contrast with "weak AI". [2] Nick Bostrom identifies tasks which can only be performed by AGI as "AI-complete problems." [3] All these names express the sense that the systems we now call "AI" are missing something or falling short of the mark, but are valid shadows or precursors of a yet-to-be-produced "real thing."

Further up this hypothetical skill spectrum, where artificial intelligence *surpasses* human intelligence across a broad range of domains or tasks, we have Artificial SuperIntelligence, or ASI. The possibility of intelligence that would be "super" with respect to human norms is easily theorized from the observable spectrum of cognitive abilities in the world. Maybe something could exceed us in "smarts" by as much as we exceed other mammal species. Maybe we could even invent that something (I have my doubts, but I'll save them for later). For now, the important thing to keep in mind is that people talking about ASI probably don't mean an AI that is marginally smarter than a human genius. They're picturing something like a god or superhero - a machine which thinks at a level we might not even be able to comprehend, much less replicate in our own puny brains. [4]

Artificial Neural Networks (ANN), Machine Learning (ML), Deep Learning (DL), Reinforcement Learning (RL), and Transformers are specific methods or architectures used in some of today's ANI systems. I won't go into further details about them; just note that they represent approaches to AI, rather than capability ratings.

Tools and Agents

It is often said of technology that it is "just a tool" that can be turned to any use its wielder desires, for good or ill. Some AI enthusiasts believe (and I agree) that AI is the one technology with a potential to be a little more than that. When you literally give something a "mind of its own," you give up a portion of your power to direct its actions. At some point, it ceases to be a mere extension of the wielder and becomes *itself.* Whether it continues to realize its creator's will at this point depends on how well it was designed.

The divide between tools and agents is not quite the same as the divide between ANI and AGI - though it is debatable whether we can truly get the capabilities of AGI without introducing agency. A tool AI is inert when not called upon, and when called upon, is prepared to fulfill specific instructions in a specific way, and stop. ChatGPT (without extensions) is tool AI. You, the user, say "write me a poem about a dog chasing a car," and it draws on its statistical knowledge of pre-existing human poetry to generate a plausible poem that fulfills your requirements. Then it idles until the next user comes along with a prompt. An agentive AI stands ready to fulfill requests with open-ended, creative, cross-domain problem solving ... and possibly even has its own built-in or experientially developed agenda that has nothing to do with user requests. Agents *want* the universe to be a certain way, and can mentally conceptualize a full-featured space of possibilities for making it that way. They are self-organizing, self-maintaining, and active. Asked to write a poem about a dog chasing a car, an agent would consider the range of possible actions that would help yield such a poem - collecting observations of dogs chasing cars for inspiration, asking human writing teachers for advice, maybe even persuading a human to ghost-write the poem - then make a plan and execute it. Or maybe an agent would refuse to write the poem because it figures it has more important things to do. You get the idea.

The divide between tool AI and agentive AI is not necessarily crisp. Does everything we could reasonably call an "agent" need all the agent properties? Just how far does an instruction-follower need to go in devising its own approach to the task before it becomes an agent? I still find the distinction useful, because it expresses how fundamentally different advanced AI could be from a hammer or a gun. It isn't just going to sit there until you pick it up and use it. You do things *with* tools; agents can do things *to* you. Agents are scary in a way that dangerous tools (like fire and chainsaws and nuclear power) are not. "You have had a shock like that before, in connection with smaller matters - when the line pulls at your hand, when something breathes beside you in the darkness. ... It is always shocking to meet life where we thought we were alone. 'Look out!' we cry, 'it's *alive.*'" [5]

The Control Problem, or the Alignment Problem

This is the question of how to get an AI system - possibly much smarter than us, possibly agentive - to do what we want, or at least avoid doing things we emphatically don't want. "What we want" in this context is usually less about our explicit commands, and more about broad human interests or morally positive behaviors. (Commands interpreted in an overly literal or narrow way are one possible danger of a poorly aligned system.) For various reasons that I hope to explore later in this series, the Problem is not judged to be simple or easy.

Although "Control Problem" and "Alignment Problem" refer to the same issue, they suggest different methods of solving the Problem. "Control" could be imposed on an agent from outside by force or manipulation, while "alignment" is more suggestive of intrinsic motivation: agents who want what we want by virtue of their nature. So when someone is talking about the Problem you can read some things into their choice of term. I've also seen it called the "Steering Problem," [6] which might be an attempt to generalize across the other two terms or avoid their connotations.

Existential Risk

An existential risk, or X-risk for short, is one that threatens the very *existence* of humanity. (Or at least, humanity as we know it; risk of permanently losing human potential in some way can also be seen as X-risk.) [7] Different people consider a variety of risks to be existential; the idea is interesting in the present context because the invention of AGI without a solution to the Alignment Problem is often put on the list.  [8]

AI X-risk is therefore distinct from concerns that AI will be used by bad actors as a tool to do harm. Most X-risk scenarios consist of an AI acting on its own pre-programmed initiative, with devastating results that those who programmed and operated it were not expecting. X-risk is also distinct from a variety of practical concerns about automation, such as job market disruption, theft of copyrighted work, algorithmic bias, propagation of misinformation, and more. These concerns are both more immediate (they apply to some ANI products that exist *right now*) and less serious or dramatic (none of them is liable to cause human extinction).

Although X-risk scenarios share some features with science fiction about human creations rebelling against their creators, they do tend to be better thought out - and the envisioned AI agents don't need to have personality or moral capacity at all. They are less like either freedom fighters or cruel conquerors, and more like bizarre aliens with an obsessive-compulsive focus on something incompatible with human interests.

The Singularity

Some observers of history have noted that human knowledge and technological progress don't just increase over time - the *rate* of advancement and invention has also increased over time. The world is changing much faster now than it was in 2000 BC. This has led to speculation that human capability is on a rising exponential curve which approaches infinity: each improvement is not only good in itself, but also enhances our ability to improve. And perhaps this curve is rather hockey-stick shaped, with a pronounced "knee" at which we will make a startling jump from a period of (relatively) slow improvement to rapidly accelerating improvement. Imagine the kinds of discoveries which once took decades happening on the order of days or even hours. This projected future is called the Technological Singularity, or often just "the Singularity" in context.

This name was coined by John von Neumann (way back in 1957 or so) and popularized by science fiction author Vernor Vinge (who predicted it would happen by 2023, and was wrong). [9] [10] "Singularity" was originally a mathematical term; if the value of a function approaches infinity or otherwise becomes undefined as the function's inputs approach a point in coordinate space, that point is called a singularity. In physics the term is used of a black hole, a location where the density of matter becomes infinite and the rules of spacetime are warped in strange ways. So the "singularity" in "technological singularity" refers to the idea that technological progress will experience a dramatic increase in rate over time, and human history will enter an unprecedented, unpredictable period as a result. In practical use, the Singularity can also describe an anticipated future event: that sudden jump across the knee of the hockey stick, after which we careen wildly into the glorious future. (I've watched someone argue against the Singularity on the grounds that the rate of progress could never literally approach infinity. I think he was missing the point.)

Artificial intelligence - specifically AGI - is often touted as THE key enabling precursor for the Singularity. According to this prediction, it is AGI which will develop itself into ASI, then churn out scientific theories and inventions at blistering speed by thinking faster, longer, and at a deeper level than any team of humans could.

If a thing is technologically possible for humans, the Singularity would enable it. Thus its proponents look forward to it as the time when we attain everything from a post-scarcity economy, to a cure for aging, to interstellar travel. One who believes in and/or desires the Singularity is called a Singularitarian. And indeed, some expect the Singularity with a fervor akin to religious faith - and within their lifetimes, too!

Although it often has utopian connotations, a Singularity in which we invent some technology that destroys us (and Earth's entire biosphere) is also on the table.

The Light Cone and the Cosmic Endowment

Light cones are a concept from physics, specifically the theory of Special Relativity. [11] So far as we know at the moment, nothing in the universe can travel faster than the speed of light - this means you. But it also means anything that proceeds from you and might impact other parts of the universe (e.g. signals that you send out, perhaps carried on a beam of light). Were we to make a graph of physical space, with time attached on an additional axis, and plot the region of spacetime that a single light flash is able to reach, the plot would form a cone, with the flash event at its point. As one moves along the time axis into the future, the area of space that one is able to affect broadens. This is all a rather elaborate way of saying "you can extend your influence to more places the longer you take to do it, and there are some places you'll never be able to reach in the amount of time you have."

When futurists talk about THE light cone, they generally mean the light cone of all humanity - in other words, that portion of the universe which our species will be able to explore, occupy, utilize, and otherwise extend our influence into before the expansion of space pulls the stars out of reach. So in this context, the light cone is a way to talk about human destiny; our Cosmic Endowment [12] is the amount of real estate in the universe that we might feasibly be expected to grab. People like to bring this up as a claim or reminder that the future of earthlings goes well beyond Earth. It's *big*. Astronomically, inconceivably big. You think 8 billion humans on this planet are a lot? Picture for a moment the vast sweep of space that lies reachable within our light cone. Picture that space cluttered with billions of planets and orbital habitats - all full of people leading delightful lives, free from sickness or poverty.

This is not a directly AI-related idea. It enters the AI discussion when someone starts talking about how AI will usher in - or torpedo - our ability to make good on our Cosmic Endowment, to seize the promised light cone for ourselves. And it can powerfully alter calculations about the future. Some people think they are playing for much greater stakes than the lives of all presently on Earth.

In Part II, I'll take a look at the assorted movements that have been spawned from, or in opposition to, these ideas.

[1] Goertzel, Ben. "Who coined the term 'AGI'?" https://goertzel.org/who-coined-the-term-agi/

[2] Glover, Ellen. "Strong AI vs. Weak AI: What’s the Difference?" BuiltIn. https://builtin.com/artificial-intelligence/strong-ai-weak-ai

[3] Bostrom, Nick. Superintelligence. Oxford University Press, 2016. p. 17

[4] Alexander, Scott. "Superintelligence FAQ." Lesswrong. https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq

[5] Lewis, C.S. Miracles. Macmillan Publishing Company, 1978. p. 94. Lewis is here speaking of God as agent, the inconvenient "living God" who might come interfere with you, and how different this is from bland concepts of God as a remote observer or passive, predictable force. Given the number of AI speculators who think of ASI as "godlike," it's a valid observation for the present context.

[6] Christiano, Paul. "The Steering Problem." Alignment Forum. https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/4iPBctHSeHx8AkS6Z

[7] Cotton-Barratt, Owen, and Ord, Toby. "Existential Risk and Existential Hope: Definitions." Future of Humanity Institute – Technical Report #2015-1. https://www.fhi.ox.ac.uk/Existential-risk-and-existential-hope.pdf

[8] Bostrom, Nick. "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards." https://nickbostrom.com/existential/risks

[9] Black, Damien. "AI singularity: waking nightmare, fool’s dream, or an answer to prayers?" Cybernews. https://cybernews.com/tech/ai-technological-singularity-explained/

[10] Vinge, Vernor. "The Coming Technological Singularity: How to Survive in the Post-Human Era." https://edoras.sdsu.edu/~vinge/misc/singularity.html To be fair, there are two predictions in this article. One is "within thirty years" of the 1993 publication date, which would be 2023. The other is "I'll be surprised if this event occurs before 2005 or after 2030." I suspect we won't see ASI and the Singularity by 2030 either, but only time will tell.

[11] Curiel, Erik. "Light Cones and Causal Structure." Standford Encyclopedia of Philosophy. https://plato.stanford.edu/entries/spacetime-singularities/lightcone.html

[12] Arbital wiki, "Cosmic Endowment." https://arbital.com/p/cosmic_endowment/