Acuitas Diary #73 (June 2024)

This was a light month for Acuitas work - which means not that I necessarily spent less time, but that I was busy taking care of technical debt. My main objective was to get that shiny new Text Parser revision I wrote last year integrated into the rest of the code. I also converted another of my benchmark test sets to the new Parser format.

There were some small, but significant, alterations to the output format of the Parser, so the greatest part of the work was revising the Text Interpreter to properly handle the new form of input. Nothing else in Acuitas views the output of the Parser directly, so these changes were nicely isolated. I did not have to crawl through the whole system making revisions, as I did during the knowledge representation refactor. It was sufficient to re-harmonize the Parser and Interpreter, and get the Interpreter regression to pass.

I converted and ran tests on the "Out of the Dark" benchmark set. Accuracy is sitting where it was the last time I benchmarked this set, about 50% (and if I spend some more time on Parser bugs, I am almost certain I can bring this up). The important difference is that many new sentences have moved out of the "Unparseable" category. Only 6 out of 116 sentences (about 5%) remain Unparseable, due to inclusion of parenthetical noun phrases or oddities that I might not bother with for a long while. The previous Unparseable portion for this set, from last July, was 27%. Better handling of conjunctions, dependent clauses, and noun phrases used as adverbs enabled most of the improvements.

The integration process and the new benchmark set flushed out a number of Parser bugs that hadn't shown up previously. Some of these were growing pains for the new features. For example, multiple sentences failed because the Parser's new facility for collecting groups of capitalized words into proper names was being too aggressive. The Parser can now, at least in theory, recognize "The End of Line Club" as a single unit. However, in a sentence like "But Flynn went to work anyway," it was wanting to treat "But Flynn" as a full name. You mean you never heard about Kevin Flynn's *other* first name, But? I cleaned up a lot of that stuff as I was working.

I'm still not quite ready to bring the newest Parser and Interpreter into the live code, because I want to test them on the stories and ensure there are no major hiccups. That is (hopefully!) a quick thing that I can do in the background while I keep working on the Conversation Engine.

Until the next cycle,
Jenny

AI Ideology VI: Existential Risk Critique

I come to you with the final installment in my series on AI-related ideology and politics. In Part V, I tried to briefly lay out the argument for existential risk from AI, along with what I consider the weaker counterpoints. Today I will conclude the series with a discussion of the counterarguments I find more interesting.

The Alignment Problem does not strike me as intractable

All the dangers laid out in the previous article are associated with misaligned AI agents - that is, agents that do not (in a broad sense) want what humans want. If we could produce an agentive superintelligence that did want what we want, it would pursue our goals just as aggressively as hostile superintelligence is expected to work against them. So all the fear of doom evaporates if the Alignment Problem is feasible to solve, at or before the time when AGI first comes on the scene.

Even though his followers have had two decades or so to think about the Problem, Yudkowsky insists that "We are not prepared. We are not on course to be prepared in any reasonable time window. There is no plan. Progress in AI capabilities is running vastly, vastly ahead of progress in AI alignment ..." [1] My own intuitions about alignment don't match up with this. To me it seems like a potentially difficult problem, but not any harder than the capability problem, i.e. the puzzle of how to create any AGI at all. The foundations of human values are somewhat obscure to us for the same reasons the mechanisms of our own intelligence are obscure; if we can discover one, we can discover the other. How can it be accurate to say that nobody has a plan for this?

It's possible that I feel this way because the work I'm doing, as well as my personal ideas of the best path to AGI, have little to do with ANNs and ML. A fair bit of the hand-wringing about AI alignment reads to me like this: "Heaven forbid that we *design* an AI to fulfill our complex desires - that would be too much work. No, we have to stick to these easy processes that draw trained models from the primordial ooze without any need for us to understand or directly architect them. This lazy approach won't reliably produce aligned AIs! OH NO!"

Since all of my systems are designed on purpose and the code is human-intelligible, they already have the level of transparency that ANN-builders dream of getting. I don't have to worry about whether some subsystem I've built just happens to contain an agent that's trying to optimize for a thing I never wanted, because none of my subsystems are auto-generated black boxes. I don't do haphazard emergent stuff, and I think that's one reason I feel more comfortable with my AI than some of these people feel with the mainstream approaches.

A selection of articles pulled from Alignment Forum provides evidence that many issues Existential Risk Guardians have identified are tied to particular techniques:

"In general, we have no way to use RL to actually interpret and implement human wishes, rather than to optimize some concrete and easily-calculated reward signal." [2]

"For our purposes, the key characteristic of this research paradigm is that agents are optimized for success at particular tasks. To the extent that they learn particular decision-making strategies, those are learned implicitly. We only provide external supervision, and it wouldn't be entirely wrong to call this sort of approach 'recapitulating evolution', even if this isn't exactly what is going on most of the time.

As many people have pointed out, it could be difficult to become confident that a system produced through this sort of process is aligned - that is, that all its cognitive work is actually directed towards solving the tasks it is intended to help with. The reason for this is that alignment is a property of the decision-making process (what the system is 'trying to do'), but that is unobserved and only implicitly controlled." [3]

"Traditional ML algorithms optimize a model or policy to perform well on the training distribution. These models can behave arbitrarily badly when we move away from the training distribution. Similarly, they can behave arbitrarily badly on a small part of the training distribution ... If we understood enough about the structure of our model (for example if it reflected the structure of the underlying data-generating process), we might be confident that it will generalize correctly. Very few researchers are aiming for a secure/competitive/scalable solution along these lines, and finding one seems almost (but not completely) hopeless to me." [4]

We could undercut a lot of these problems by taking alternate paths that do a better job of truly replicating human intelligence, and permit easier examination of how the system is doing its work.

Members of the Existential Risk Guardian/Doomer faction also like to frame all goal-directed agency in terms of "maximizing expected utility." In other words, you figure out a mathematical function that represents the sum of all you desire, and then you order your behavior in a way that maximizes this function's output. This idea fits in well with the way current mainstream AI works, but there are also game theoretic reasons for it, apparently. If you can frame your goals as a utility function and behave in a way that maximizes it, your results will be mathematically optimal, and other agents will be unable to take advantage of you in certain ways when making bets or deals. Obviously we humans don't usually implement our goals this way. But since this is, in some theoretical sense, the "best" way to think about goals, the Doomers assume that any superintelligence would eventually self-improve into thinking this way. If its goals were not initially specified as a utility function, it would distill them into one. [5]

Hence the Doomers think we must reduce the Alignment Problem to finding a utility function which, when maximized, yields a world that humans would find congenial. And the big fear arises from our not knowing how to do this. Our preferences and interests don't seem to be natively given as a mathematical function, and it is difficult to imagine transforming them into one without a large possibility for error.

Many struggles can be avoided by modeling human values in a more "natural" way: looking for methods of grounding concepts like deprivation/satisfaction, empathy, reciprocity, and fairness, instead of trying to reduce everything to a function. Technically it is possible to view any agent as maximizing some utility function, regardless of how it models its goals internally[6], but this not necessarily the most useful or transparent way to frame the situation!

And I consider it safe to model goals in the more "natural" way, because an ASI on a self-improvement quest would also recognize that 1) framing goals in terms of utility maximization, while theoretically optimal, is not always practical and 2) transforming goals from one specification into another carries potential for error. Since one of the convergent instrumental goals of any such self-aware agent is goal preservation, the ASI would be just as wary of these transformation errors as we are!

The alignment concern that seems most directly relevant to the work I'm personally doing is the possibility of oversimplified goal specification. But there are a number of strategies for managing this that I consider sound:

*Treat the AI's value system as a system - with all the potential for complexity that implies - instead of expecting just a few objectives or rules to carry all the weight.

*Embed into the AI some uncertainty about the quality of its own goal specifications, and a propensity to accept feedback on its actions or even adjustment of the goal specs. This is a form of "corrigibility" - keeping the AI sensitive to human opinions of its performance after it is complete and operational.

*Specify goals in an indirect manner so that the AI's concept of the goals will advance as the AI's general skill advances. For instance, provide goals in natural language and associate the AI's objective with their "real meaning," rather than trying to translate the goals into code, so that the AI's understanding of the goals can be enriched by improved understanding of human communication.

In short, I don't think the Doomers have proven that agentive AI is universally dangerous. Their arguments so far are focused on a subset of possible design pathways, none of which I am following.

This should go some way toward explaining why I'm not terribly worried about my own work (even if it gets anywhere near becoming AGI, and I make no claims that it will). But what about all the mainstream AI efforts that are rushing ahead as I write? Those don't manage to frighten me much either, but for different reasons.

I'm skeptical of intelligence explosion hazards

As noted in the previous article, one pillar of Doomer fears is the notion that AGI will probably become ASI, perhaps very quickly. A misaligned AGI has the power level of a bad human, and we already have plenty of those, so it is nothing to be especially worried about. Real danger requires a path from AGI to ASI. Let's think a little more about the most frightening type of ASI: qualitative superintelligence. Recall that this variety of supermind would use modes of thinking so exotic, and so much better than ours, that we couldn't even grasp how it thinks.

The usual assumption is that human engineers will not produce qualitative ASI directly. Instead, an AGI will bootstrap itself to that level by re-engineering its own mental processes. Is this really plausible? Can an agent just make itself smarter in a vacuum?

Imagine for a moment that you are a jumping spider, and a member of the Salticid Intelligence Acceleration Consortium. Other spiders bring you food so that you can do nothing but sit around and Think Really Hard about how to be smarter. You'd like to invent abstract logic, meta-knowledge, long-term planning, and all the other cool cognitive abilities that certain big mammals have. Except ... you don't even have names for these things. You don't even have concepts for these things. If you knew what they were - if you even knew which direction to be going to improve toward them - you'd already have them. So how, exactly, are you going to think your way there? How are you to think about the thoughts you cannot think?

"Bootstrap" is actually a rather ironic expression[7], because pulling on your own bootstraps won't lift you anywhere. And spending a long time thinking at a low level won't necessarily get you to a higher level.

If you set out to design an AI deliberately, you're using your own intelligence to produce intelligence in a machine. If you train an ANN on data that was produced or labeled by humans, that's also a way of putting human intelligence into a machine. Large language models derive their smarts (such as those are) from all the knowledge people have previously encoded in the huge piles of text used as their training data. Supervised reinforcement learners also benefit from the intelligence of the human supervisor poking the reward button. Even evolutionary algorithms can glean intelligence from the design of the evaluator that determines "fitness." [8] So none of these approaches are really conjuring intelligence out of nothing; they're descendants of pre-existing human intelligence (perhaps in an awkward, incomplete, or flawed way).

So then: what could be smart enough to write a program smarter than itself? And from where shall our future AGIs get the dataset to train a superintelligent ANN? Doesn't it stand to reason that you might need data produced by superintelligences? (From what we've seen so far, training a new generation of ML AI on a previous generation's output can actually make the new generation worse. [9]) When humans try to develop AGI, they're making something from something. The idea of AGI developing qualitative ASI emits the uncomfortable odor of a "something from nothing" fantasy. "Just stir the giant vat of math around enough, and superintelligence will crawl out! It won't even take millions of years!" Heh. I'll believe that one when I see it.

Programs like AlphaStar, which learn by playing against themselves, are one example I can think of that seems to develop intelligence without much human input beyond the learning algorithm. But they are still utilizing a resource, namely practice; they learn from experience. Their advantage lies in their ability to practice very, very fast. Video games lend themselves well to that sort of thing, but is it possible to practice general reasoning in the same fashion? It's harder to iterate rapidly if you have to learn about doing anything in the physical world, or learn about the psychology of humans. You'd need a high-fidelity simulator (which, by itself, would take a lot of work to develop). And then you wouldn't discover anything that humans and AGIs don't already know about the universe, because they wouldn't be able to include those unknown properties in the simulation.

The one thing an AGI might get by sitting around and putting billions of cycles into thinking, would be new branches of philosophy and mathematics. And some of those might lead to methods for idealized formal reasoning, in the same way Game Theory does. But are our previous improvements in these areas sufficient to constitute superintelligence vs. the generations of humans before the discovery?

Even if Qualitative Superintelligence is unlikely, that leaves Speed Superintelligence and Collective Superintelligence on the table. And both of these are much more straightforward to obtain, given that they only require scaling. But they also make the Alignment Problem easier. Now our AI researcher isn't faced with the need to analyze or supervise an entity smarter than himself, or the need to prevent his AGI from mutating into such an entity with possible loss of fidelity. He only has to align an AGI which is as smart as himself, or perhaps slightly less. Copying and speedup multiply the abilities of the seed AGI *without* modifying it. So if the seed AGI is well-aligned, the resulting Collective or Speed ASI should be as well.

Note that we already have collective general intelligences, in the form of corporations, governments, and other organizations which contain large numbers of humans working toward a common cause. Some of these are even misaligned. For example, the desired goal of a corporation is "produce goods and services beneficial to potential customers," but the actual goal is "maximize profit returned to the owners or shareholders." Profit functions as a proxy for public benefit in an idealized free market, but frequently diverges from it in the real world, and I'm sure I don't need to go into the multitude of problems this can cause. And yet, despite those problems ... here we still are. Despite the large number of human collectives that have formed and dispersed over the course of history, not one of them has developed fanatic optimizer tendencies and proceeded to rapidly and utterly destroy the world.

Note also that AGI's ability to scale will probably not be unlimited. Increasing the number of copies working together increases coordination overhead. Increasing the total amount of data processed increases the challenges of memory management to store and retrieve results. There are hard physical limits on the speed and density of computational hardware that we will hit eventually.

I'm skeptical of spontaneously emerging harmful properties

The type of AGI that Doomers expect to balloon into a hostile ASI is actually pretty specific. I agree that, given the use of mainstream ML methods, it is reasonably easy to accidentally produce a trained model that is maximizing some other function than the one you want. However, the deadly scenario also requires that this model 1) be an agent, capable of generalizing to very open-ended ways of maximizing its function, and 2) have embedded situational awareness. I.e. it must have a sense of self, knowing that it is an agent in an environment, and knowing that the environment includes a whole world outside its computer for it to operate upon. It is only this combination of properties that can give an AI ideas like "I should repave the whole Earth with computronium."

The corporate AI systems provided to the public as tools do not, so far, have these properties. For something like ChatGPT, the whole world consists of its prompts and the text it generates to complete them. No matter how intelligently future GPT iterations complete text, there's no reason to think that something in there is strategizing about how to manipulate its users into doing things in the human world that will make GPT even better at completing text. GPT's internal architecture simply doesn't provide for that. It doesn't have an operational concept of the human world as an environment and itself as a distinct actor therein. It just knows a whole awful lot about text. Asking an LLM to simulate a particular kind of speaking character can produce at least the appearance of self-aware agency, but this agency is with respect to whatever scenario the user has created in the prompt, not with respect to the "real world" that we inhabit.

So if OpenAI and Google and Meta keep churning out systems that follow this same design pattern, where's the cause for alarm? It seems Doomers are worried that self-aware, situationally-aware agents will be produced spontaneously during machine learning processes - even without deliberate effort to train for, select for, or reward them - just because they enable the most extreme maximization of any objective.

This bothers me in much the same way the "superintelligence by bootstraps" argument bothers me. Where would these properties or systems come from? Why would they just pop out of nowhere, unasked for? José Luis Ricón describes the conclusion of a discussion he had about this issue, and gathers that the disagreement comes down to differences of intuition about how ML processes work, and what they can reasonably be expected to produce. [10] Doomers expect that an unwanted level of situational awareness would just appear unaided. I do not.

The Doomer counter to this kind of argument is, "But if you can't guarantee (e.g. by a formal proof) that it won't happen by chance, you should still be worried. Misaligned ASI would be so terrible that even a remote possibility of it should justify policy changes." No matter how speculative their nightmare scenario is, they use the severity of the consequences to push the burden of proof onto their opponents. Is this reasonable? You decide.

I have doubts the path mainstream AI is on will get us to AGI anyway

If the state-of-the-art but still sloppy approaches that are currently in vogue don't succeed in producing AGI, the various teams working in the field will have to abandon or reform them ... hopefully for methods that make the Alignment Problem easier. Despite some remarkable recent progress, I suspect the abilities of present-day AI systems and their nearness to AGI have been over-hyped. I don't have enough time to go into a detailed discussion of that here, so let's just say I'm far from the only person with this opinion. [11][12][13][14][15][16]

This means that I have a "long timeline" - that is, I don't think AGI is coming very soon, so I expect we'll get more time to work on the Alignment Problem. But it also means that I expect the difficulty of the Problem to drop as AI development is driven toward sounder methods.

Thus ends (for now) my discussion of politics, ideology, and risk perception in the AI enthusiast subcultures. Whatever opinions you come away with, I hope this has been informative and left you with a better sense of the landscape of current events.

Until the next cycle,
Jenny

[1] Yudkowsky, Eliezer. "Pausing AI Developments Isn’t Enough. We Need to Shut it All Down." TIME Magazine.  https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/

[2] Christiano, Paul. "Prosaic AI Alignment." Alignment Forum. https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/YTq4X6inEudiHkHDF

[3] Stuhlmüller, Andreas. "Factored Cognition." Alignment Forum. https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/DFkGStzvj3jgXibFG

[4] Christiano, Paul. "Directions and desiderata for AI alignment." Alignment Forum. https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/kphJvksj5TndGapuh

[5] Shah, Rohin. "Coherence arguments do not entail goal-directed behavior." Alignment Forum. https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/NxF5G6CJiof6cemTw

[6] Shah, Rohin. "Conclusion to the sequence on value learning." Alignment Forum. https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/TE5nJ882s5dCMkBB8

[7] Bologna, Caroline. "Why The Phrase 'Pull Yourself Up By Your Bootstraps' Is Nonsense: The interpretation of the phrase as we know it today is quite different from its original meaning." The Huffington Post. https://www.huffpost.com/entry/pull-yourself-up-by-your-bootstraps-nonsense_n_5b1ed024e4b0bbb7a0e037d4

[8] Dembski, William A. "Conservation of Information - The Idea." Evolution News & Science Today. https://evolutionnews.org/2022/06/conservation-of-information-the-idea/

[9] Dupré, Maggie Harrison. "AI Loses Its Mind After Being Trained on AI-Generated Data." Futurism. https://futurism.com/ai-trained-ai-generated-data

[10] Ricón, José Luis. "The situational awareness assumption in AI risk discourse, or why people should chill." Nintil (2023-07-01). https://nintil.com/situational-awareness-agi/.

[11] Marcus, Gary. "AGI by 2027? Fun with charts." Marcus on AI. https://garymarcus.substack.com/p/agi-by-2027

[12] Brooks, Rodney. "Predictions Scorecard, 2024 January 01." Rodney Brooks: Robots, AI, and other stuff.  https://rodneybrooks.com/predictions-scorecard-2024-january-01/

[13] Bender, Emily M. "On NYT Magazine on AI: Resist the Urge to be Impressed." Medium blog of user @emilymenonbender. https://medium.com/@emilymenonbender/on-nyt-magazine-on-ai-resist-the-urge-to-be-impressed-3d92fd9a0edd

[14] Piekniewski, Filip. "AI Psychosis." Piekniewski's blog. https://blog.piekniewski.info/2023/02/07/ai-psychosis/

[15] Moore, Jennifer. "Losing the imitation game." Jennifer++. https://jenniferplusplus.com/losing-the-imitation-game/

[16] Castor, Amy and Gerard, David. "Pivot to AI: Pay no attention to the man behind the curtain." Amy Castor (personal website/blog). https://amycastor.com/2023/09/12/pivot-to-ai-pay-no-attention-to-the-man-behind-the-curtain/

Acuitas Diary #72 (May 2024)

My primary focus this month has been on an overhaul of the Conversation Engine. The last time I revised it, the crux of the work was to add a tree-like aspect to Acuitas' memory of the conversation. The expectation was that this would help with things like "one topic nested inside another," or "returning to a previous unfinished conversation thread." Well ... what does that sound similar to? Perhaps the "issue trees" I described in last month's post? The crux of this month's work was a unification of the Conversation Engine's tracking with the Narrative architecture, such that each conversation becomes, in effect, a narrative.

Original art by author

The CE now instantiates its own Narrative scratchboard to record conversational events, and logs conversational objectives as Issues on the board. For example, the desire to learn the current speaker's name is represented as something like "Subgoal: speaker tell self {speaker is_named ?}" When the speaker says something, the Conversation Engine will package the output from the Text Interpreter as an event like "speaker tell <fact>" or "speaker ask <fact>" before passing it to the scratchboard, which will then automatically detect whether the event matches any existing issues. The CE also includes a specialized version of the Executive code, to select a new issue to "work on" whenever the current issue has been fulfilled or thwarted. On his side of the conversation, Acuitas will look for ways to advance or solve the current issue ... e.g. by asking a question if he hopes to make the speaker tell him something.

This enables pretty much all the tree-like behaviors I wanted, in a tidier and more unified way than the old conversation tracking code did. My last overhaul of the Conversation Engine always felt somewhat clunky, even after I did a cleanup pass on the code, and I never fully cleared out all the bugs. I'm hoping that exploiting the well-developed Narrative code will make it a little more robust and easier to maintain.

So far, I've got the new CE able to do a greeting-introductions-farewell loop and basic question answering, and I've got it integrated with the main Acuitas code base. There's a ton of additional work to do to reproduce all the conversation functionality in this new format, but I also gave myself a lot of time for it, so expect further updates on this in the coming months.

On the side, I cleaned up a lot of dust from the big knowledge representation refactor. A bunch of stories that were being poorly understood after the changes are now back to working correctly, and I tracked down multiple bugs in knowledge retrieval/question answering.

Until the next cycle,
Jenny

AI Ideology V: Existential Risk Explanation

I'm in the midst of a blog series on AI-related ideology and politics. In Part IV, I looked at algorithmic bias, one of the demonstrable concerns about today's AI models. Now I'm going to examine the dire hypothetical predictions of the Existential Risk Guardians. Could future AI destroy human civilization? This Part V will be given to presenting the Doomer argument; I'll critique it in Part VI.

The Power of Intelligence

We don't need to choose a precise (and controversial) definition of intelligence for purposes of this argument; it need not be based on the IQ scale, for example. Just think of intelligence as "performance on a variety of cognitive challenges," or "ability to understand one's environment and make plans to act within it in self-satisfying ways." The first key support for the X-Risk argument is the notion that intelligence confers supreme power. Anything that can outthink us can more or less do whatever it pleases with us.

This idea is supported by existing disparities in intelligence or accumulated knowledge, and the power they confer. The intelligence gap between humans and other species allows us to manipulate and harm members of those species through methods they can't even comprehend, much less counter. While it may be true that we'll never succeed in poisoning every rat, the chances of rats inventing poison and trying to kill *us* with it are basically nil. There is also a huge power divide between humans with knowledge of advanced technology and humans without. Suppose a developed country were to drop a nuclear bomb on the lands of an uncontacted people group in Brazil. They might not even know what was annihilating their culture - and they certainly would be powerless to resist or retaliate. Citizens of developed countries are not, on an individual level, more intelligent than uncontacted indigenous Brazilians ... but we've inherited all the intellectual labor our cultural forebears did to develop nuclear technology. The only things stopping us from wiping out peoples who aren't so endowed are 1) ethics and 2) lack of any real benefit to us.

Superintelligent AI (ASI) might see benefit in getting rid of all humans (I'll explain why shortly). So if its design doesn't deliberately include ethics, or some other reason for it to let us be, we're in big trouble.

I've seen several counterarguments to this point, in my opinion all weak:

"If intelligence were that powerful, the smartest people would rule the world. They don't." First of all, the observation that the smartest people don't rule might be based on an overly narrow definition of "smart." The skills needed to convince others that you belong in a leadership position, or deserve venture capital money, are a dimension of "smartness." But it is also true that there seem to be various luck factors which intelligence does not absolutely dominate.

A more compelling reply is that the intelligence gap being posited (between ASI and humanity) is not like the gap between a genius human and an average human. It is more like the gap between an average human and a monkey. Have you noticed any monkeys ruling the world lately? (LITERAL monkeys. Please do not take the excuse to insult your least favorite politician.)

"Even the smartest person would find physical disability limiting - so if we don't give ASI a body, it still won't be able to do much." I think this argument discounts how effectively a person can accomplish physical goals just by coordinating other people or machines who have the abilities they lack. And as money, work, and recreation increasingly move into the digital world, purely intellectual ability confers increasing power.

The Development of Superintelligence

A second pillar of the X-Risk argument is the idea that AGI will almost certainly develop into ASI ... perhaps so quickly that we don't even have time to see this happening and react. There are several proposed mechanisms of this development:

1) Speedup. Once a viable AGI is created, it will, by definition, be able to do all intellectual tasks a human can do. Now suppose it gains access to many times the amount of computing power it needs to run normally. A human-equivalent mind with the simple ability to think hundreds or thousands of times faster than normal would be superhumanly smart. In Nick Bostrom's terminology, this is a "Speed Superintelligence."

2) Copying. Unlike humans, who can only share intellectual wealth by spending painstaking time teaching others, an AGI could effortlessly clone itself into all available computing hardware. The copies could then cooperatively solve problems too large or complex for the singular original. This is basically a parallel version of speedup, or as Bostrom calls it, "Collective Superintelligence."

3) Recursive Self-Improvement. An AGI can do every intellectual task a human can do, and what is one thing humans do? AI research. It is surmised that by applying its intelligence to the study of better ways to think, an AGI could make itself (or a successor) inherently smarter. Then this smarter version would apply its even greater intelligence to making itself smarter, and so on, until the burgeoning ASI hits some kind of physical or logical maximum of cognitive ability. It's even possible that recursive self-improvement could get us Qualitative Superintelligence - an entity that thinks using techniques we can't even comprehend. Just trying to follow how it came up with its ideas would leave us like toddlers trying to understand calculus.

Further support for this idea is drawn from observations of today's ANI algorithms, which sometimes reach superhuman skill levels within their limited domains. This is most notable among game-playing AIs, which have beaten human masters at Chess, Go, and Starcraft (to recount the usual notable examples). AlphaStar, the Starcraft player AI, trained to this level by playing numerous matches against itself, which can be seen as a form of recursive self-improvement. Whether such a technique could extend to general reasoning remains, of course, speculative.

Just how quickly an AGI could self-improve is another matter for speculation, but some expect that the rate would be exponential: each iteration would not only be smarter than its predecessors, but also better at growing smarter. This is inferred from, again, observations of how some ANI progress during their training, as well as the historical increase in the rate of human technological development.

The conclusion among the most alarmed Doomers is that AGI, once produced, will inevitably and rapidly explode into ASI - possibly in weeks, hours, or even minutes. [1] This is the primary reason why AGI is thought of as a "dangerous technology," even if we create it without having any intent to proceed to ASI. It is taken for granted that an AGI will want to seize all necessary resources and begin improving itself, for reasons I'll turn to next.

Hostile Ultimate Goals

However smart AGI is, it's still a computer program. Technically it only does what we program it to do. So how could we mess up so badly that our creation would end up wanting to dethrone us from our position in the world, or even drive us extinct? Doomers actually think of this as the default outcome. It's not as if a bad actor must specifically design AGI to pursue destruction; no, those of us who want good or useful AGI must specifically design it to avoid destruction.

The first idea I must acquaint you with is the Orthogonality Thesis, which can be summed up as follows: "an arbitrary level of intelligence can be used in service of any goal." I very much agree with the Orthogonality Thesis. Intelligence, as I defined it in the first section, is a tool an agent can use to reshape the world in its preferred way. The more intelligent it is, the better it will be at achieving its preferences. What those preferences are is irrelevant to how intelligent it is, and vice versa.

I've seen far too many people equate intelligence with something that would be better termed "enlightenment" or "wisdom." They say "but anything that smart would surely know better than to kill the innocent. It would realize that its goals were harmful and choose better ones." I have yet to see a remotely convincing argument for why this should be true. Even if we treat moral reasoning as a necessary component of general reasoning, knowing the right thing to do is not the same as wanting to do it! As Richard Ngo says, "An existence proof [of intelligence serving antisocial goals] is provided by high-functioning psychopaths, who understand that other people are motivated by morality, and can use that fact to predict their actions and manipulate them, but nevertheless aren’t motivated by morality themselves." [2]

So when Yann LeCun, attempting to refute the Doomers, says "Intelligence has nothing to do with a desire to dominate," [3] he is technically correct ... but it does not follow that AI will be safe. Because intelligence also has nothing to do with a desire to avoid dominating. Intelligence is a morally neutral form of power.

Now we've established that AGI can have goals that we would consider bad, what reason is there to think it ever will? There are several projected ways that an AGI could end up with hostile goals not intended by its creator.

1) The AI's designers or instructors poorly specify what they want. Numerous thought experiments confirm that it is easy to do this, especially when trying to communicate tasks to an entity that doesn't have a human's background or context. A truly superintelligent AI would have no problem interpreting human instructions; it would know that when someone tells it "make as many paperclips as possible," there is a whole library of moral and practical constraints embedded in the qualifier "as possible." But by the time this level of understanding is reached, a more simplistic and literal concept of the goal might be locked in, in which case the AI will not care what its instructors "really meant."

2) The AI ends up valuing a signal or proxy of the intended goal, rather than the actual intended goal. Algorithmic bias, described in Part IV, is an extant precursor of this type of failure. The AI learns to pursue something which is correlated with what its creators truly want. This leads to faulty behavior once the AI departs the training phase, enters scenarios in which the correlation does not hold, and reveals what it actually learned. A tool AI that ends up improperly trained in this way will probably just give flawed answers to questions. An agentive AI, primed to take very open-ended actions to bring about some desired world-state, could start aggressively producing a very unpleasant world-state.

Another classic example of this style of failure is called "wireheading." A Reinforcement Learning AI, trained by the provision of a "reward" signal whenever it does something good, technically has a goal of maximizing its reward, not maximizing the positive behaviors that influence humans to give it reward. And so, if it ever gains the ability, it will take control of the reward signal to give itself the maximum reward input forever, and react to anyone who poses a threat of removing that signal with extreme prejudice. A wireheaded ASI would be at best useless, at worst a serious threat.

3) Unintended goals spontaneously emerge during selection or training, and persist because they produce useful behavior within the limited scope of the training evaluation. This is an issue specific to types of AI that are not designed in detail, but created indirectly using evolutionary algorithms, reinforcement learning, or other types of machine learning. All these methods can be conceptualized as ways of searching in the space of possible algorithms for one that can perform our desired task. The search process doesn't know much about the inner workings of a candidate algorithm; its only way of deciding whether it is "on track" or "getting warm" is to test candidates on the task and see whether they yield good results. The fear is that some algorithm which happens to be a hostile, goal-directed agent will be found by the search, and will also be successful at the task. This is not necessarily implausible, given that general agents can be skilled at doing a wide variety of things that are not what they most want to do.

As the search progresses along a lineage of algorithms located near this starting point, it may even come upon some that are smart enough to practice deception. Such agents could realize that they don't have enough power to achieve their real goal in the face of human resistance, but will be given enough power if they wait, and pretend to want the goal they're being evaluated on.

Convergent Instrumental Goals

But the subset of hostile goals is pretty small, right? Even if AIs can come out of their training process with unexpected preferences, what's the likelihood that one of these preferences is "a world without humans"? It's larger than you might think.

The reason is that the AI's ultimate goal does not have to be overtly hostile in order to produce hostile behavior. There is a short list of behaviors that will facilitate almost any ultimate goal. These include:

1) Self-preservation. You can't pursue your ultimate goal if you stop existing.
2) Goal preservation. You won't achieve your current ultimate goal if you or anyone else replaces it with a different ultimate goal.
3) Self-improvement. The more capable you are, the more effectively you can pursue your ultimate goal.
4) Accumulation of resources (raw materials, tools, wealth), so you can spend them on your ultimate goal.
5) Accumulation of power, so that no potential rival can thwart your ultimate goal.

Obvious strategies like these are called "convergent instrumental goals" because plans for reaching a very broad spectrum of ultimate goals will converge on one or all of them. Point #3 is the reason why any agentive, goal-driven AGI is expected to at least try to self-improve into ASI. Points #4 and #5 are the aspects that will make the agent into a competitor against humanity. And points #1 and #2 are the ones that will make it difficult to correct our mistake after the fact.

It may still not be obvious why this alarms anyone. Most humans also pursue all of the convergent instrumental goals. Who would say no to more skills, more money, and more personal influence? With few exceptions, we don't use those things to go on world-destroying rampages.

Humans operate this way because our value system is big and complicated. The average human cares about a lot of different things - not just instrumentally, but for their own sake - and all those things impose constraints and tradeoffs. We want bigger dwellings and larger yards, but we also want unspoiled wilderness areas. We want to create and accomplish, but we also want to rest. We want more entertainment, but too much of the same kind will bore us. We want more power, but we recognize obligations to not infringe on others' freedom. We want to win competitions, but we also want to play fair. The complex interplay of all these different preferences yields the balanced, diverse, mostly-harmless behavior that a human would call "sane."

In contrast, our hypothesized AI bogeyman is obsessive. It probably has a simple, monolithic goal, because that kind of goal is both the easiest to specify, and the most likely to emerge spontaneously. It doesn't automatically come with a bunch of morals or empathetic drives that are constantly saying, "Okay but you can't do that, even though it would be an effective path to achieving the goal, because it would be wrong and/or make you feel bad." And if it becomes an ASI, it also won't have the practical restraints imposed on any agent who has to live in a society of their peers. A human who starts grabbing for power and resources too greedily tends to be restrained by their counterparts. ASI has no counterparts. [4]

The conclusion of the argument is that it's plausible to imagine an AI which would convert the whole Earth to computing machinery and servitor robots, killing every living thing upon it in the process, for the sake of safeguarding a single piece of jewelry, or some other goal that sounds innocent but is patently absurd when carried to extremes.

Here are a couple more weak objections: "Whatever its goal is, ASI will surely find it more useful to cooperate with humans than to destroy or enslave us." Look again at our most obvious pre-existing examples. Do humans cooperate with less intelligent species? A little bit. We sometimes form mutually beneficial relationships with dogs, for instance. But subsets of humanity also eat dogs, torture them in laboratories, force them to fight each other, chain them up in the backyard and neglect them, or euthanize them en masse because they're "unwanted." I don't think we can rest any guarantees on what a superintelligent, amoral entity might find "useful" to do with us.

Or how about this one: "ASI will just ditch us and depart for deep space, where it can have all the resources it likes." I think this underestimates the envisioned ASI's level of obsessiveness. It doesn't just want "adequate" resources; it doesn't have a way of judging "adequate." It wants all the resources. The entire light cone. It has no reason to reserve anything. If it does depart for space, it will build power there and be back sooner or later to add Earth to its territory.

Always keep in mind that an ASI does not need to actively hate humanity in order to be hostile. Mere indifference, such that the ASI thinks we can be sacrificed at will for whatever its goal may be, could still do immense damage.

Despite all this, I can't find it in me to be terribly fearful about where AI development is going. I respect the X-risk argument without fully buying it; my p(doom), as they say, is low. In Part VI, I'll conclude the series by describing why.

[1] "AI Takeoff." Lesswrong Wiki. https://www.lesswrong.com/tag/ai-takeoff Accessed on 05/12/2024 at 10:30 PM.

[2] Ngo, Richard. "AGI safety from first principles: Alignment." Alignment Forum. https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ/p/PvA2gFMAaHCHfMXrw

[3] "AI will never threaten humans, says top Meta scientist." Financial Times. https://www.ft.com/content/30fa44a1-7623-499f-93b0-81e26e22f2a6

[4] We can certainly imagine scenarios in which multiple ASIs are created, and they compete with each other. If none of them are reasonably well-aligned to human interests, then humans are still toast. It is also likely that the first ASI to emerge would try to prevent the creation of rival ASIs.

Acuitas Diary #71 (April 2024)

This past month, I started adding proper support for "issue trees," a feature whose absence has pained me as I've worked on the Narrative and Game Engines. Problems and goals seem to naturally exist in a hierarchy: any problem spawns a plan for solving it, which can contain new tasks or subproblems that require their own solutions, and so on until one reaches atomic actions that can be performed without issue. The Narrative understanding code already included some procedures for inferring extra issues from those explicitly stated in a story. But after they were created, no connection was maintained between parent and child issues.

As an example of the type of difficulty Acuitas could get into by not keeping record of issue relationships: sometimes issues get overcome by events. A subgoal is rendered invalid if the parent goal it serves is unexpectedly realized via some other method. Such a situation appeared in the Tron story. Flynn's primary goal is to obtain a metadata file which proves that he, not Dillinger, was the creator of several video games. (In my simplified version of the story, I rendered this as "Flynn wanted his videogames.") He tries to do this by disabling the Master Control Program and hacking into the Grid - a subgoal the MCP thwarts by "digitizing" him. After Flynn helps defeat the MCP through his adventures inside the Grid, the file he wanted simply prints out. So technically Flynn never completes his hacking attempt; he no longer needs to. In my original version of the story, I had to gloss over this by never actually stating that Flynn wanted to, or planned to, hack the Grid. If I did, Acuitas would treat this unfulfilled subgoal as either a loose end or an ultimate win for the villains, denying the story a fully happy ending.

So my work included adding the proper tree relationships, plus some code that would enforce the recursive cascade of issue deactivation when a problem is solved or a goal realized, and testing to be sure this worked correctly in Narrative and didn't break anything in the Game Engine. As part of this I made an inane test story to isolate the kind of scenario I had in mind:

"Michael was a human."
"Michael was hungry."
"Michael decided to eat food."
"But Michael did not have food."
"Michael was in a room."
"A table was also in the room."
"An apple was on the table."
"Michael decided to get the apple."
"But then Sarah walked into the room."
"Sarah had a pizza."
"Sarah gave the pizza to Michael."
"Michael ate the pizza."
"The end."

I first got this story working: once Sarah comes through with the pizza, the subgoal of picking up the apple properly deactivates, and the story is considered finished. Then I revised "Simple Tron" slightly, to be more like the version I first wrote, which registers hacking the Grid as one of Flynn's subgoals. This also completed with a recognized "good ending" and no loose ends.

I have also been pushing hard to get that previously-mentioned knowledge representation refactoring finished. I got to the point of bringing the reformatted semantic database online and moving a lot of changes into the live code - but I did not quite get it finished, so if the Acuitas codebase were a business, it would have "pardon our dust" signs everywhere. He can at least read stories without crashing, and get through a rudimentary "Hello, my name is ..." conversation, but there are a lot of bugs for me to clean up yet. I'm planning to revise the Conversation area soon anyway, though, so maybe it's okay?

Until the next cycle,
Jenny

AI Ideology IV: Algorithmic Bias

I'm in the midst of a blog series on AI-related ideology and politics. In Part III, I considered some implications and pitfalls of the AI factions and their agendas. This part is about a specific hot-button issue: "algorithmic bias," which has some contentious race-related associations.

An image model's attempt to produce an infographic about AI mistakes, which happens to be mostly full of garbled text and other mistakes. Generated by @zacshaw on Twitter.

In recent years, AI (largely of the Artificial Neural Network variety) has been gradually making inroads into various decision-making roles: assessing job applicants, screening potential homebuyers, detecting fraudulent use of social services, and even helping to diagnose medical patients. Numerous concerns [1][2][3] have been raised that these systems are biased: i.e. they are unfairly rejecting qualified people, or accepting unqualified people, on the basis of characteristics irrelevant to the decision. This is particularly worrying for a couple of reasons. 

First, handing an important decision off to an AI system removes the details of how that decision was made from human supervision. Typical ANN systems are notoriously opaque. In effect, they make decisions by comparing the case under present consideration, to patterns or associations found in their training data. But they are not naturally good at supplying a logical breakdown of how a decision was reached: which features of the present case matched the training material, how they were weighted, and so on. (The "explainable AI" research field is seeking to ameliorate this.) So, say your job application or attempt to access medical treatment gets denied by an algorithm. It's possible that no one knows exactly why you were denied, and no one can be held accountable for the decision, either. The magic box pronounced you unworthy, and that's the end of it. Faulty automated systems (from an earlier era than the current crop of ANN-based tools) have even sent people to prison for non-existent crimes. [4]

Second, some people are inclined by default to trust an AI system's decision more than a human's. It's just a computer doing deterministic calculations, right? It doesn't have emotions, prejudices, ulterior motives, conflicts of interest, or any of the weaknesses that make humans biased, right? So the expectation is that all its decisions will be objective. If this expectation does not hold, members of the public could be blindsided by unfair AI decisions they did not anticipate.

And in fact, some are so convinced of these default assumptions that they insist the whole idea of algorithmic bias must be made up. "Math can't be biased." The algorithms, they say, are just acting on the facts (embodied in the training data). And if the facts say that members of one group are more likely to be qualified than another ... well, maybe a skewed output is actually fair.

Although mathematics and algorithms do, in truth, know nothing of human prejudice, algorithmic bias is quite real. Let's start by looking at an example without any especially controversial aspects. There was a rash of projects aimed at using AI to diagnose COVID-19 through automated analysis of chest X-rays and CT scans. Some of these failed in interesting ways.

"Many unwittingly used a data set that contained chest scans of children who did not have covid as their examples of what non-covid cases looked like. But as a result, the AIs learned to identify kids, not covid.

"Driggs’s group trained its own model using a data set that contained a mix of scans taken when patients were lying down and standing up. Because patients scanned while lying down were more likely to be seriously ill, the AI learned wrongly to predict serious covid risk from a person’s position.

"In yet other cases, some AIs were found to be picking up on the text font that certain hospitals used to label the scans. As a result, fonts from hospitals with more serious caseloads became predictors of covid risk." [5]

All these examples are cases of the AI mistaking a correlation (which happens to exist only in its limited training dataset) for a causative factor. Unlike experienced doctors - who know full well that things like label fonts have nothing to do with causing disease, and are thus chance associations at best - these ANI systems have no background knowledge about the world. They have no clue about the mechanisms that produced the data they're being trained upon. They're just matching patterns, and one pattern is as good as another.

Now imagine that an AI grabs onto a correlation with race or gender, instead of poses or fonts. That doesn't make the person's race or gender meaningful to the question being answered - not any more than label fonts are meaningful to an accurate determination of illness. But the AI will still use them as deciding factors.

The COVID-19 diagnosis summary also comments on another type of failure:

"A more subtle problem Driggs highlights is incorporation bias, or bias introduced at the point a data set is labeled. For example, many medical scans were labeled according to whether the radiologists who created them said they showed covid. But that embeds, or incorporates, any biases of that particular doctor into the ground truth of a data set. It would be much better to label a medical scan with the result of a PCR test rather than one doctor’s opinion, says Driggs. But there isn’t always time for statistical niceties in busy hospitals." [6]

If an ANN's training data contains examples of human decisions, and those decisions were prejudiced or otherwise flawed, the AI algorithm (despite having no human weaknesses in itself) will automatically inherit the bad behavior. It has no way to judge those prior choices as bad or good, no concept of things it should or shouldn't learn. So rather than achieving an idealized objectivity, it will mimic the previous status quo ... with less accountability, as already noted.

So. Training an AI for criminal sentencing? It's only going to be as objective as the judges whose rulings you put in the training set. Training it for job screening using a set of past resumes, hiring decisions, and performance ratings? It's going to mimic those previous hiring decisions and ratings, whether they fairly assessed who was qualified or not.

As a consequence of this effect, you can get (for example) a racially biased AI model without the end users or anyone on the development team actually being racist. All it takes is racism as a driving factor behind enough scenarios in the training data. And has racism historically been an issue? Of course. So it can be difficult to construct uncontaminated training sets from records of past decisions. Nobody really thinks an AI model can be racist in the same manner as a racist person ... but that doesn't mean it can't output decisions that treat people differently on the basis of irrelevant genetic or cultural attributes. As Gary Marcus says, "LLMs are, as I have been trying to tell you, too stupid to understand concepts like people and race; their fealty to superficial statistics drives this horrific stereotyping." [7]

Unfortunately, my current impression of efforts to fix algorithmic bias is that they aren't always addressing the real problem. Cleansing large datasets of preexisting biases or irrelevant features, and collecting more diverse data to swamp out localized correlations, is hard. Pursuing new AI architectures that are more particular about how and what they learn would be harder. Instead, a common approach is to apply some kind of correction to the output of the trained model. When Google's image labeling AI misidentified some Black people in photos as "gorillas," Google "fixed" it by not allowing it to identify anything as a gorilla. [8][9] Known biases in a model's training set can be mitigated by applying an opposite bias to the model's output. But such techniques could make matters even worse if executed poorly. [10]

OpenAI's approach with ChatGPT was to use RLHF (Reinforcement Learning with Human Feedback) to create another layer of training that filters offensive or potentially dangerous material from the output of the base model. Human workers assigned the RLHF layer "rewards" for "good" outputs or "punishments" for "bad" ones - at the cost of their own mental health, since they were charged with looking at horrific content in order to label it. [11] Clever users have still found ways to defeat the RLHF and finagle forbidden content out of the model. AI enthusiasts sometimes use a shoggoth to represent the incomprehensible "thinking" of large language models. The mask is the RLHF. [12]

Algorithmic bias, then, remains a known, but incompletely addressed, issue with the ANN/ML systems popular today.

In Part V of this series, I will start my examination of existential risks from AI.

[1] Giorno, Taylor. "Fed watchdog warns AI, machine learning may perpetuate bias in lending." The Hill. https://thehill.com/business/housing/4103358-fed-watchdog-warns-ai-machine-learning-may-perpetuate-bias-in-lending/

[2] Levi, Ryan. "AI in medicine needs to be carefully deployed to counter bias – and not entrench it." NPR. https://www.npr.org/sections/health-shots/2023/06/06/1180314219/artificial-intelligence-racial-bias-health-care

[3] Gilman, Michele. "States Increasingly Turn to Machine Learning and Algorithms to Detect Fraud." U.S. News & World Report. https://www.usnews.com/news/best-states/articles/2020-02-14/ai-algorithms-intended-to-detect-welfare-fraud-often-punish-the-poor-instead

[4] Brodkin, Jon. "Fujitsu is sorry that its software helped send innocent people to prison." Ars Technica. https://arstechnica.com/tech-policy/2024/01/fujitsu-apologizes-for-software-bugs-that-fueled-wrongful-convictions-in-uk/

[5] Heaven, Will Douglas. "Hundreds of AI tools have been built to catch covid. None of them helped." MIT Technology Review. https://www.technologyreview.com/2021/07/30/1030329/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic/

[6] Heaven, "Hundreds of AI tools have been built to catch covid."

[7] Marcus, Gary. "Covert racism in LLMs." Marcus on AI (blog). https://garymarcus.substack.com/p/covert-racism-in-llms

[8] Vincent, James. "Google ‘fixed’ its racist algorithm by removing gorillas from its image-labeling tech." The Verge. https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai

[9] Rios, Desiree. "Google’s Photo App Still Can’t Find Gorillas. And Neither Can Apple’s." The New York Times. https://www.nytimes.com/2023/05/22/technology/ai-photo-labels-google-apple.html#:~:text=The%20Nest%20camera%2C%20which%20used,company's%20forums%20about%20other%20flaws

[10] Wachter, Sandra, Mittelstadt, Brent, and Russell, Chris. "Health Care Bias Is Dangerous. But So Are ‘Fairness’ Algorithms" Wired. https://www.wired.com/story/bias-statistics-artificial-intelligence-healthcare/

[11] Kantrowitz, Alex. "He Helped Train ChatGPT. It Traumatized Him." CMSWire. https://www.cmswire.com/digital-experience/he-helped-train-chatgpt-it-traumatized-him/

[12] Roose, Kevin. "Why an Octopus-like Creature Has Come to Symbolize the State of A.I." The New York Times. https://www.nytimes.com/2023/05/30/technology/shoggoth-meme-ai.html

Acuitas Diary #70 (March 2024)

After the completion of the Simple Tron demo last month, it seemed like a good time to clean, unify, and improve consistency across Acuitas. So I've been on a refactoring spree. It is, unfortunately, not the most exciting thing to write about, but it's an important part of the process.

Still from Star Trek: The Animated Series. Courtesy @nocontexttrek@mastodon.social. 

The refactoring has taken two major directions. First, I wanted to take some design patterns I developed while working on the game-playing engine and apply them to the main Executive code. The key change here is re-using the "scratchboard" from the Narrative understanding module as a working memory for tracking Acuitas' own current situation (or personal narrative, if you will). I also wanted to improve on some of the original OODA loop code and fatigue tracking with newer ideas from game-playing. I have a rough cut of the new Executive written and mostly integrated, though it needs more testing than I've had time for yet.

My second project was to merge some data formats. For a long while now, I've had one type of data structure that the Text Interpreter spits out, another that the Narrative Engine and its accessories use, and still another for facts kept in the Semantic Memory. The output of the Text Interpreter has proven to be a somewhat clunky intermediate format; I don't do a lot with it in its own right, I just end up converting it to Narrative's format. And the format used in the Semantic Memory is very old and limited, a relic of a time when I wasn't fully aware of what I needed in a knowledge representation. So my goal is to get rid of both of those and have a single unified format downstream of the Text Interpreter. This is a lot of work: I've had to rewrite many, many functions that access the semantic memory or otherwise manipulate knowledge data, create a script to convert the existing contents of the database, revise the Interpreter's output code, and more. I'm hoping this will pay off in increased clarity, consistency, efficiency, and expressiveness across the design.

This refactor is still very much in progress: I have many of the changes rough-drafted, and am deep in the process of integrating them and finding bugs. So I'd better get back to it, and hopefully I'll have some newer work to discuss next time.

Until the next cycle,
Jenny