Acuitas Diary #59 (April 2023)

I've continued my two-pronged work on Narrative understanding and on "game playing." On the Narrative side this month, I did more complex term grounding - specifcally of the word "obey."

My working definition of "to obey X" was "to do what X tells you to do." This is interesting because there is no way to infer directly that any given action qualifies as obedience, or defiance ... the question of whether someone is following orders (and whose orders) is always relative to what orders have been given. So proper understanding of this word requires attention to context. Fortunately the Narrative scratchboard stores that sort of context.

In addition to simply inferring whether some character has obeyed some other, I wanted to make derivative subgoals. If one agent has a goal of obeying (or disobeying) another agent, that's a sort of umbrella goal that isn't directly actionable. Before the agent can intentionally fulfill this goal, it has to be made specific via reference to somebody else's orders. So when this goal is on the board, the appearance (or pre-existence) of orders needs to spawn those specific subgoals.

In short it was a whole lot more complicated than you might think, but I got it working. Eventually I'll need to make this sort of relative word definition generic, so that new words that operate this way can be learned easily ... but for now, "obey" can be a case study. The Big Story needs it, since part of the story is about a power struggle and which leader(s) certain characters choose to follow.

Game-playing still isn't demo-ready, but it's starting to feel more coherent. I worked through all the bugs in the code that responds to simple description of a scene, then began working on responses to goals/issues. It was fun to leverage the existing Narrative code for this, the way I'd wanted to. In the Narrative module, that code serves to predict character actions, analyze *why* characters are doing things, and determine whether characters are meeting their goals, whether their situation is improving or worsening, etc. But as I'd hoped, a lot of the same structures are just as effective for control and planning.

For example, let's say Acuitas is playing a human character and is told "you are hungry." Something like this unfolds:

New issue: <self> is hungry
Problem solving: <self> eat food
Prerequisite check: if <self> does not have food, <self> cannot eat food
Test: Does self have food?
New issue: <self> does not have food
Problem solving: <self> get food
Prerequisite check: if <self> is not colocated with food, <self> cannot get food

... and so on. The chain continues until some actionable solution is found.

A funny note: another one of the prerequisites for eating, at least in the traditional sense, is "character must have a mouth." This turned up as an obstacle in early tests, because I had apparently neglected to teach Acuitas that humans have mouths. Ha!

Until the next cycle,
Jenny

SGP Part III: On Brain Physics, Qualia, and Embodied SGP Solutions

The Acuitas project is an abstract symbolic cognitive architecture with no sensorimotor peripherals, which might be described as "disembodied." Here I will argue that there are viable methods of solving the Symbol Grounding Problem in such an architecture, and describe how Acuitas implements them. In Part III of this series, I continue my discussion of Searle's Chinese Room paper with the question of whether Symbol Grounding in digital computers is possible at all, even for embodied systems. Click here for SGP Part II and an introduction to the Chinese Room thought experiment.

"Planetary System" by Levi Walter Yaggy, from Yaggy's Geographical Study, 1887. Public domain. No real relation to the content of this blog: I needed something pretty and a little mysterious or awe-inspiring.

The seemingly obvious solution to the Chinese Room's lack of grounding is to permit the man in the room to connect at least some of the Chinese characters to referents. For instance, if each character were accompanied by a picture (visual data), the man would soon learn to associate them with real things in the world that he himself has previously experienced through sensory data from his eyes. Characters sent in under specific conditions could give him names for the internal states of the room. As these associations developed, the man's letters to the outside world could begin to be *about* something (which Searle calls "intentionality" and which I might call "grounding" or "groundedness").

But Searle won't have this. He contends that even if we were to provide a computer program with a robot body, and supply that body with a full complement of sensors that would feed data into the program, the program would remain a "Chinese Room." Even if every symbol used within the program is connected to some collection of sensory data features that were derived from objects, dynamics, or states of being observed in the world ... Searle argues that the symbol manipulation system does not contain the sensory data, and therefore does not "understand" or have "intentionality" behind its outputs. I'd say he's framing things incorrectly; the question is not "does the symbol manipulation system understand?" but rather "does the complete robot mind understand?", where the complete mind includes those parts that process perceptual data, build models of referents, and store pointers between them and the symbols.

So how does Searle think that humans come to understand and to communicate intentionally? He insists there's something special about the physics of our wetware. The computational properties of the brain, or the flow of information within it, are by themselves useless for producing "intentional" communication that is about something. Searle thinks you need the "actual causal properties of the physical substance [brain matter]."[1] No matter what you opt to simulate a brain with, if it isn't real brain tissue, the activity might be as useless as running a simulation of a rainstorm and expecting your computer to get wet, or expecting a computer model of a kidney to actually filter someone's blood.[2] "Stones, toilet paper, wind, and water pipes are the wrong kind of stuff to have intentionality in the first place - only something that has the same causal powers as brains can have intentionality ..."[3]

I'm more familiar with - and sympathetic to - this sort of argument when it comes up in discussions of phenomenal consciousness and the incommunicable qualia[4] which compose it. Consciousness, which for our purposes here is *the ability to have subjective experiences*, is not obviously identifiable with either computation or physics, so it remains unclear what combination of the two might cause it. Therefore, although Searle does not say so explicitly, I wonder if he is trying to argue that phenomenal consciousness is necessary for intentionality, and understanding can only proceed from the brain's ability to connect qualia with spoken words.

And here, I think he's going too far. It's not obvious to me how any criterion of understanding or intentionality cannot be attained by a system that lacks qualia. An entity that doesn't have subjective experience is only missing ... subjective experience. It can still connect perceptual data which encodes its own internal states, or states of the external world, to symbols. It can then manipulate the symbols to determine how this perceptual data is relevant to its goals (*preferred* external or internal states). And it can still deduce the actions that are most likely to achieve those goals, interpret the symbols for those actions into a sequence of physical movements, and execute them. These are all easily imagined as informational processes.

So my difficulty with this argument is that Searle (and others in this camp) cannot seem to describe any plausible *mechanism* by which any non-computational, physics-driven outgrowth of brain activity would form a necessary part of symbol grounding. If a symbol is specifically associated with states of the world or self which are relevant to one's goals, then it has subjective meaning. If it is associated with the same world-states that most of society would also associate with it, then it has objective meaning. This is all we need to solve the Symbol Grounding Problem, to enable real understanding and real communication. Qualia are incommunicable by definition, so for purposes of asking whether an AI can communicate intentionally, we don't have to worry about them.

The "but a simulation of rain doesn't get the computer wet" argument is a disanalogy. It's true that you can't run a computational simulation and expect a physical result. But the results we demand from intelligence are informational results: decisions, plans, true answers to questions. And you *can* get informational results from a system that does nothing but process information. A "simulation of intelligence" is itself intelligence. There are only lingering questions about qualia because, again, it is not obvious whether qualia are informational or physical (or perhaps even some secret third thing).

In my explanation of why the Chinese Room does not "understand," I described a concrete way in which it fails: no information can pass from the inside of the room to the outside, or vice versa. I have yet to see anyone on the "brain physics are mandatory for understanding" side of the debate describe how the absence of brain physics produces a similar capability failure. I have only seen them suggest, vaguely, that our failure to obtain "true artificial general intelligence" so far is not a matter of using incorrect algorithms, but of using algorithms at all. They almost seem to be saying that if we could produce a machine that *did* replicate whichever physical properties of the brain are relevant, it might magically start "thinking better" ... no need to even design an improved learning process for it!

Searle's Chinese Room paper eventually nosedives into "No True Scotsman" arguments. He says that, even if we observed a robot with behavior that seemed to demand intentionality, "If we knew independently how to account for its behavior without such assumptions we would not attribute intentionality to it especially if we knew it had a formal program."[5] So if any entity that contains a formal symbol manipulation program ever produces evidence of intentionality, it still can't be *real* intentionality, because it came (in part) from a formal program. This is getting absurd.

Whether AI programs could ever have phenomenal consciousness/qualia is not a topic I want to get into here - because it is an incredibly slippery topic that demands its own full article. I simply conclude that qualia are not mandatory for symbol grounding, and can therefore be set aside during the present discussion. Furthermore, since symbol grounding is an informational process, consisting of the relation of one type of data to another, it is not a physical process and does not depend on any physics particular to biological brains. A robot can effectively ground any symbols that it uses for internal cogitation or external communication by relating them to perceptual data, including any reward/aversion stimuli that help produce its notion of positive and negative world states.

Having addressed whether the Symbol Grounding Problem is solvable in a robot, in Part IV I'll get into whether it is solvable in non-embodied systems.

[1] John R. Searle (1980) "Minds, Brains, and Programs," Behavioral and Brain Sciences, Volume 3, p. 9
[2] Colin Hales (2021), "The Model-less Neuromimetic Chip and its Normalization of Neuroscience and Artificial Intelligence"
[3] John R. Searle (1980) "Minds, Brains, and Programs," Behavioral and Brain Sciences, Volume 3, p. 12
[4] What are qualia? "When we see a red word on a page, our brain acquires all sorts of data about the wavelength of the light, the shape and size of the letters, and so on. But there is more to it than that: we also have an experience of redness, and this experience is over and above the mere data-gathering, which a computer could do equally well. This experienced red, along with experienced blue, cold, noise, bitterness, and so on are qualia, and it is very hard to give a fully satisfactory account of them." This reference also goes into the difference and connections between qualia and intentionality, and gives further background on Searle's views. Peter Hankins, "Three and a Half Problems," Conscious Entities Blog
[5] John R. Searle (1980) "Minds, Brains, and Programs," Behavioral and Brain Sciences, Volume 3, p. 9

Acuitas Diary #58 (March 2023)

This is going to be one of those months when my diary is short and boring, because I've been busy laying foundations and have little that is finished to talk about or demonstrate. I worked on two things - continued improvements to the Narrative module, and the beginnings of Narrative game-playing.

Photo by Spc. Anthony Zane, public domain

The Narrative improvements cover continued work on character motivation, as well as some bug fixes. I added some capacity to interpret "because" statements as commentary about goals. E.g. a statement like "John decided to flee the battle because he wanted to live" supplies a motive for the instrumental goal of fleeing the battle, which (to Acuitas at least) might not be immediately obvious otherwise. 

I also tweaked Narrative's built-in tiny ontology a bit. Distinctions are made, for modeling and processing purposes, between "agents" (individuals that can reasonably act as characters in a story), "objects," "locations," and "organizations." I realized I needed to add another category, for abstract nouns or "concepts," when I noticed that a "purpose" was being treated as a physical object. Whoops! I also decided to add a "system" category, to cover such entities as computer networks. At first I was thinking of a network as a "location," but I realized that it's more than that.

Now, for game-playing! I am very excited about this. The goal it to make Acuitas capable of navigating "text adventure" style games by leveraging a lot of the existing narrative and reasoning capabilities. Given the machinery for modeling characters in stories, and predicting what they may do, it is not such a difficult step to imagine *oneself* as one of those characters, and then decide how to interact with the environment ... I'm also hoping that experimenting in these game scenarios will give me ideas for how to improve the main Executive module.

I actually sketched out the code to support roleplaying earlier this year, and just began integrating it this month. I was hoping to have something demo-worthy, but the process took longer than I'd hoped and thus is still in progress. So hopefully I'll have some more interesting and concrete details to share within the next month or two. I've gotten so far as to get Acuitas to register a character that is "his," and to log issues for that character - but I haven't wrung out enough bugs yet to even get through a full input-output loop. More info coming later!

Until the next cycle,
Jenny

SGP Part II: The Need for Grounding

The Acuitas project is an abstract symbolic cognitive architecture with no sensorimotor peripherals, which might be described as "disembodied." Here I will argue that there are viable methods of solving the Symbol Grounding Problem in such an architecture, and describe how Acuitas implements them. In Part 2 of this series, I address the question of whether Symbol Grounding is truly important. Click here for SGP Part I.

Some people believe the entire Symbol Grounding Problem (SGP for short) can be sidestepped. Useful intelligence is possible without grounding, they argue. An AI doesn't need to understand what words mean - or at any rate, it can have some kind of effective understanding without needing an explicit connection between any word and its referent. And indeed, some of the most popular linguistic AIs today are arguably not grounded.

An example word association graph from the incredibly titled paper "Beauty and Wellness in the Semantic Memory of the Beholder." [1]

Linguistic AIs without any grounding get their results by learning and exploiting relationships between symbols. These relationships could be semantic (e.g. category relationships, such as ["cat" is "animal"]) or statistical (e.g. the frequency with which two words are found next to each other in a body of human-written text). For an example of an ungrounded semantic AI project, see Cyc [2]. For statistical examples, see the GPT lineage and other Large Language Models (LLMs).

The problem with trying to define all symbols exclusively by their relationship to other symbols is that such definitions are circular. An encylopedia only works for someone who already knows a lot of words in the target language; new words are defined by relating them not just to other words, but to known (grounded) words *and their referents.* If you knew none of the words in an encyclopedia, it would still be internally consistent, highly structured ... and useless to you. No matter how large a net of connected symbols you build, if not even one symbol in the net is tied down to something "real," isn't the whole thing meaningless?

There are at least two possible counterpoints. One begins by noting that even if every symbol is arbitrary, their relationships are not. Imagine the words in a language as nodes (like locations) and the relationships between them as links (like paths running from one node to another), perhaps with a numerical weight to express how strong the connection is. The result is an abstraction called a graph. Even if we changed the name of every concept expressible in the English language, the graph topology of the relations between English symbols would not change. A rose by any other name would still be linked to "flower" by some other name, would still have the property of "sweet-smelling" by some other name, and so on. If you could zoom out and look at the pattern created by the whole network, it wouldn't be different at all.

Here's a graph without any words attached to the nodes (Kneser Graph (7,3), a Graph Theory construct). If we pretend that the heptagon on each node is a linguistic symbol, do their graphed relationships confer any meaning on them?

Next the defender of ungrounded models must argue that these inter-symbol relationships encode real information about the universe. Sure, maybe it wouldn't do you any good to know that a "muip" is a "weetabiner," but if you look at the *sum total* of connections "muip" has, and how they differ from the connections of every other symbol in the database, there is a sense in which you know what "muip" means. Not the traditional human sense of connecting it to sensory data, but *a* sense. [3]

I think there is a grain of truth in this argument. However, if it goes some way toward permitting objective meaning, it does not enable  subjective meaning. If I can't establish a connection between any of the symbols in the encyclopedia and the referents that happen to matter to *me* - sensations, emotions, desires, etc. - then it still doesn't help me communicate. I might be able to regurgitate some information from the graph to a person who *did* understand the symbols, and this person might even praise my knowledge. But I wouldn't be able to use that knowledge to tell them anything about *me* - information I know personally that isn't in the encyclopedia.

This leads into the second counterargument: ungrounded linguistic AIs can be quite successful. They can give correct answers to questions, follow commands, and generate coherent stories or essays. "How could they do all that," their proponents cry, "if they didn't really understand anything?" Some have even argued that human authorial talents are no more than a capacity to remix older literary content in a statistically reasonable way, implying that grounded symbols either don't really exist in the human mind or aren't that important. [4]

In my opinion, ungrounded statistical language models have no understanding in themselves. When they achieve such apparently good results, they do it by piggybacking on human understanding. Let's say an LLM manages to give you a good piece of advice. The LLM doesn't have a clue about anything but the fact that the words in its output could be reasonably expected to appear after words like the ones in your input. But for the humans who wrote answers to similar questions that got included in the LLM's training data, those words *were* grounded - so the answer might be correct. And for you, the words in the output are grounded, which means you can relate them to real things in your world and make use of them. But where would the AI be without you? If it had goals of its own that lay outside the "produce coherent text" paradigm, would all its knowledge of inter-word relationships help it accomplish them? Nope. It would need the missing piece: a way to tie at least *some* words to the substance of those goals.

(And that's leaving aside the possibility that the advice or answer you get from the LLM is flat wrong. I think this has less to do with the lack of grounding, and more to do with the LLM statistical approach not being a great knowledge representation scheme.)

I would be remiss not to mention Searle's famous Chinese Room paper [5]. Anyone who's deep in the AI weeds has probably heard of it, but it might be new to some of you. In it Searle proposes a thought experiment. Imagine a man closed up in a small room, whose only communication with the outside world is a means of passing written messages. The man does not know Chinese; but people outside the room send him messages in Chinese, and he has a rulebook which explains how to transform any received strings of Chinese characters into a response. So he writes messages back. Lo and behold, to the people outside, the man in the room appears to understand Chinese. They can pass in a message that says "How are you?" and he will respond with the Chinese equivalent of "Doing well, how about yourselves?" They can ask "Who wrote Journey to the West?" and he can answer "Reputedly, Wu Cheng'en." They can pass in a story, ask follow-up questions about the story, and get accurate results. Given a large enough rulebook, this isolated room containing the man may even come to resemble a fluent Chinese writer. It might produce whole novels.

Now answer for yourself this question: does the Chinese Room know Chinese?

Searle's answer is "no," and so is mine. Judging by recent reactions to LLMs (whose operations resemble the thought experiment rather strongly, if we assume the man constructs his own rulebook by studying a large number of sample input messages) quite a few people would disagree with me! "But empirically, doesn't the Chinese Room system resemble a regular Chinese-speaker in every respect? How can you claim it's different without invoking some kind of unobservable special sauce, some assumed human exceptionalism, some woo?"

I offer as a counter-argument the fact that there is something the Chinese Room cannot do: it cannot transfer any real *information* from the people outside the room to the man inside, or from him to them. The outside observers may write "your brother is sick, and your dog has had a litter of puppies" to the man, in Chinese of course. And he may give a fully appropriate response. A scene seemingly inspired by the occasion could even appear in his next novel. But he will feel no emotion, nor update his personal model of the world. He will continue to think of himself as having a healthy brother and only one dog. Similarly, the inside of the room might be hot or cold or humid or smelly; the man may feel contented, or tired, or ill; he may appreciate or resent the type of food delivered down the maintenance chute. And he has no way whatsoever to indicate such things to the people outside. He knows how his Chinese characters are related to each other, but not how they might relate to any of the things that impact his existence inside the room. If there is a poster on an interior wall, he cannot describe it.

The conclusion is that, while the Chinese Room might *talk* very well, it cannot *communicate*. [6]

What follows is a screenshot and transcript of my attempt to get Chat-GPT (a statistical LLM tuned by reinforcement learning, with other additions) to tell me about its current state and recent history. The information I want relates to the program's own functions - I'm asking about digital realities like "how many times today was a thumbs-up button on an instance of the web UI clicked" - so Chat-GPT shouldn't be hindered by its lack of sense or motor organs. But it resists answering or insists it doesn't know. What it probably *is* being hindered by is that the information I'm requesting isn't hooked up to the mass of symbol relationships in the chat engine.

Having leaned on Searle for support to argue that grounding is necessary, now I have to fight him. Because Searle believes his Chinese Room proves machine intelligence - of the kind we're considering, anyway - is impossible. All computer algorithms are analogous to the rules by which the man produces his replies, Searle says; and if these do not produce true understanding, if the Room does not "know Chinese," then nothing that is a computer algorithm alone can either. It is people on this side of the debate, who view the Symbol Grounding Problem as essentially unsolvable, whom I must address in Part III.

[1] Yeod N. Kenett, Lyle Ungar, and Anjan Chatterjee (2021) "Beauty and Wellness in the Semantic Memory of the Beholder," Frontiers in Psychology, Volume 12.

[2] Cyc Platform Description

[3] "This contrasts with the simple distributional semantics (or use theory of meaning) of modern empirical work in NLP, whereby the meaning of a word is simply a description of the contexts in which it appears. Some have suggested that the latter is not a theory of semantics at all but just a regurgitation of distributional or syntactic facts. I would disagree ... I suggest that meaning arises from understanding the network of connections between a linguistic form and other things, whether they be objects in the world or other linguistic forms. If we possess a dense network of connections, then we have a good sense of the meaning of the linguistic form." Christopher D. Manning (2022) "Human Language Understanding and Reasoning," Daedalus, Volume 151

[4] "I’m not trying to play up GPT-2 or say it’s doing anything more than anyone else thinks it’s doing. I’m trying to play down humans. We’re not that great." Scott Alexander (2019) "GPT-2 as Step toward General Intelligence," Slate Star Codex

[5] John R. Searle (1980) "Minds, Brains, and Programs," Behavioral and Brain Sciences, Volume 3

[6] "It’s the same with these “conversations”–a large language model is, effectively, trying to predict both sides of the conversation as it goes on. It’s only allowed to actually generate the text for the “AI participant,” not for the human; but that doesn’t mean that it is the AI participant in any meaningful way. It is the author of a character in these conversations, but it’s as nonsensical to think the person you’re talking to is real as it is to think that Hamlet is a real person. The only thing the model can do is to try to predict what the participant in the conversation will do next." Ben Schmidt (2023) "You've never talked to a language model."

Acuitas Diary #57 (February 2023)

The development pattern I'm trying to follow lately is to spend half the month adding something to the Narrative module, and half the month on something else. This month's Narrative work was on better goal modeling in the Narrative scratchboard, with the hope of expanding its ability to handle character motivations. For the other feature, I made my first introduction of indirect objects to the Text Parser.

Acuitas has had the capacity to model individual goals for agents for a while. But this was something that had to be established ahead of time; the Narrative module couldn't take in top-level goals defined for a fictional character and store them in the temporary Narrative memory space. There were several elements to incorporating this:

*Getting Narrative to pick up on hints that help indicate whether an agent's described desire is an ultimate goal or an instrumental goal.
*Making it detect and store goal priority information (e.g. "he wanted X more than he wanted Y").
*Merging these stored goals into the goal model so they can be detected as motivations for instrumental subgoals

I also threw in some ability to model goal maximization. Up to this point, the Narrative module has considered goals as things that can be satisfied or unsatisfied - e.g. a state an agent wants to be in, or some singular deed it wants to accomplish. At any given moment in the course of the story, the goal is either achieved or not. A maximizing goal is something the agent wants to have as many times as possible, or to the greatest possible extent. It has a "score," but is never completed.

The endgame was to get this sentence to be somewhat meaningful:

"The <agent> wanted to be powerful more than the <agent> wanted any other thing."

Uh-oh.

On to the second project: including indirect objects in the Text Parser. I left them out initially because they can be a little tricky. Another noun appearing between a verb and a direct object might be an indirect object (as in "I gave the people bread"), but it might also be a "noun" functioning as an adjective (as in "I gave the wheat bread to them"). I guarantee the parser still doesn't perfectly distinguish these yet - sorting out all cases will probably take the application of common-sense reasoning and contextual clues. But it can already handle less ambiguous cases like "I gave the people a good show."

Despite the difficulties, it was time to get IOs in, because their absence has been something of a thorn in my side. I've been getting around it by substituting prepositional phrases, which has led to some awkward wording like "Graham asked of a troll where the chest was." I wouldn't say they're fully implemented  yet either - interactions with some other grammatical elements, notably conjunctions and dependent clauses, aren't totally ironed out. But at least the Parser can handle IOs in simpler sentences, and the rest of the text-processing chain is now set up to manage them also.

Indirect objects are surprisingly sparse in my parser benchmark datasets. I re-ran those and scored one new sentence. One.

Sentence diagram (Parser output) example from the "Out of the Dark" test set. There's some ambiguity in whether the phrase starting with "about" modifies "told" or "stories." I think either reading counts as correct.

In my diagram tool, I draw an arrow pointing from a direct object to the indirect object with which it is associated.

Until the next cycle,
Jenny

SGP Part I: A Description of the Symbol Grounding Problem

The Acuitas project is an abstract symbolic cognitive architecture with no sensorimotor peripherals, which might be described as "disembodied." Here I will argue that there are viable methods of solving the Symbol Grounding Problem in such an architecture, and describe how Acuitas implements them. In this first article, I introduce the Symbol Grounding Problem for anyone who is not already familiar with it.

Part I: A Description of the Symbol Grounding Problem

Mean anything to you?
(Screenshot of Myst III:Exile via mystjourney.com, copyright Ubisoft)

In layman's terms, the Symbol Grounding Problem (SGP for short) can be summed up in the following question: "How does anyone know what words mean?" If you listen carefully, a quieter, more ominous voice will ask, "And what is 'meaning,' anyway?"

Technically the symbols don't have to be words, though language processing is the context in which the problem is most often illustrated. A "symbol" is anything that points to some "referent," such that an intelligent agent who has learned this symbol's meaning can "pick out" the referent upon observing the symbol. (What does "pick out" imply? I would argue that it certainly means "think of," but also goes so far as "find/recognize in the environment," "act on," etc. The point of thinking about referents is usually to do something about them.) A symbol is also probably part of a "symbol system" which allocates a variety of referents to a collection of complementary symbols. The system includes rules for manipulating the symbols to produce combined or derivative meanings; the grammatical rules for composing sentences are an example. "Grounding" is the process of associating symbols with their referents. [1]

For an example of what working with ungrounded symbols is like, try reading a page of text in a language you do not know. Even if you also have a dictionary in this language, looking up the words won't help, because they're only defined in terms of other unknown words. Trying to use the dictionary will lead you on an endless circular path that never arrives at real meaning.

Two more terms that often come up in connection with the SGP are "semantics" and "syntax." "Semantics" is a formal term for meaning or the study thereof, while "syntax" refers to the structural or manipulative rules that are part of a symbol system. It is notable that syntax does not need semantics; it is based on the forms of the symbols themselves, so manipulations that follow the rules can be carried out without knowledge of the symbols' meanings. However, syntax without semantics is arguably not very useful, as it only serves to transform one string of gibberish into another.

For example, suppose I propound the following statements:

All muips are weetabiners.
Paloporoloo is a muip.

Then you could conclude with certainty that "Paloporoloo is a weetabiner." By logic and the syntactic rules of the English language, this is a correct deduction! But how does it help you? You know neither what Paloporoloo is nor what a weetabiner is. So there's not much you can *do* with the information.

To a human, perhaps the most obvious kind of referent is "something out there in the world" - an object, or a fellow embodied agent, or part of a landscape. We can also refer to properties of these things (color, shape, size, age) and to changes in them: actions that they take or that may be taken upon them. But referents include a variety of intangibles too: physical things that can't be directly touched or pointed to (time, energy); systems, organizations, philosophies, and methods (water cycle, nation, liberalism, science); things inside our own minds (idea, memory, decision, emotion); and abstract standards or states of being (love, justice, freedom, beauty). Symbols can even refer to other symbols (word, glyph, number), or to grammatical or logical structure in a sentence (the, and, that).

A symbol can be utterly arbitrary.[2] Some symbols take on a bit of flavor from their referents - onomatopoeic words, for instance, or ideograms that look like stick figures of the objects they represent. But this is not necessary. Any piece of data you like will do to symbolize anything you like. If you hope to use symbols for communication, then the mapping of symbols to referents must be largely agreed upon by you and your communication partner; this is the only real constraint.

Words are just collections of sounds (or squiggles on a surface), in no particular order, with no rules by which they were chosen. They are in no way *inherently* tied to or derived from the referents to which they point. "A rose by any other name would smell as sweet." And this is why the Symbol Grounding Problem is a Problem. Symbols do not map themselves; you need more than just the symbols in order to connect them with their referents. [3] The fabled True Speech is not ours, and the name of the rose without the rose itself is futile.

As a (presumably) human reader you might still be wondering what the big deal is. Of course we go outside words to learn the meanings of words, but that's easy enough, right? A baby can do it. Babies learn words by hearing them in association with some experience of their referents. With time and repetition, a mental link is formed between the two. But now consider the issue from the perspective of an artificial intelligence that has no robotic body, exists as an abstraction inside a computer tower, and *only* processes words. A number of past and present attempts at AI fit this description. How shall they know what words mean?

If we did presume to place an AI mind in a suitable robotic body and let it "grow up like a baby," the human learning process is not fully understood, and not so simple to replicate as it might seem. Just establishing the needed low-level processing so that sensory experiences can be categorized is a massive undertaking that remains incomplete. Here lies the attraction of jumping to a fully-formed, abstract linguistic intelligence, even if this demands novel ways of grappling with the SGP.

I want to address one more wrinkle before going further. Symbols have an objective meaning (the referent your community or culture generally agrees they map to), but they also have a subjective meaning. For every symbol you know, there is something that it means *to you,* based on its referent's implications in your particular life. [4] There are ways your inner existence changes upon reading certain words. These are not necessarily constant across the whole time you know a symbol, either; they shift with personal growth and situational context. Both the objective and the subjective are important to the use of symbols for communication. Objective meanings allow communication to be successful. They are what give your partner the ability to "pick out" the same referent you just "picked out." But it is the subjective meanings that provide the motive for attempting communication in the first place. If something is utterly unimportant to you, you probably won't bother talking about it. An ideal grounding solution should enable both these conceptions of "meaning."

Simple word-object association grants objective meaning but not subjective meaning. To obtain the latter, you need some notion of the referents doing things *for* you or *to* you. Rewards, goals, nociception, attraction, bliss, agony. Connection of referents, and by extension their symbols, with positive, negative, or neutral states in the self builds up subjective meaning. A baby learning subjective meanings has, for starters, a sensitive body with homeostatic needs for warmth, food, hygiene, and sleep. Most AI programs have nothing remotely like this to work with.

The SGP has implications for other well-known AI problems, such as the Alignment Problem. Suppose someone would like to give an AI a built-in directive such as "love your neighbor" or "do not harm human beings." One of your reactions upon hearing such a proposal should be "How will you tell the AI what 'harm' is? How are they supposed to know how to 'love'? Who is their 'neighbor'? What counts as a 'human being'?" (Even humans are notorious for defining these terms so as to make ethical loopholes for themselves.) The problem gets especially sticky if the plan is to somehow instill these ideals, from the beginning, in an AI that will gradually learn language. How to embed a directive writ in symbols before the symbols are even known? What if the wrong meanings are learned?

And yet ... some AI systems that manipulate symbols *without* any grounding look surprisingly capable. Large Language Models, often criticized for being word blenders that contain no connections between those words and anything meaningful[5], can still produce coherent and responsive texts. Image generators that know nothing about physics or three-dimensional form still turn out stunning pictures. This has led some to question whether we really need Symbol Grounding after all, since programs without it can achieve a lot of behaviors they would associate with "understanding" or "intelligence."

Thus the Symbol Grounding Problem ignites two debates in the AI research community. Do we really need to worry about it? And if so, how can we solve it?

In Part II, I'll look into arguing the first question.

[1] Stevan Harnad (2007) Symbol grounding problem. Scholarpedia, 2(7):2373., revision #73220

[2] "Anything can be a representation of anything by fiat. For example, a pen can be a representation of a boat or a person or upward movement. A broomstrick can be a representation of a hobby horse. The magic of representations happens because one person decides to establish that x is a representation for y, and others agree with this or accept that this representational relation holds. There is nothing in the nature of an object that makes it a representation or not, it is rather the role the object plays in subsequent interaction." Luc Steels (2008) "The symbol grounding problem has been solved, so what's next?"

[3] "Symbolic representations must be grounded bottom-up in nonsymbolic representations ..." Stevan Harnad (1990), "The Symbol Grounding Problem." Physica D: Nonlinear Phenomena, Volume 42, Issues 1-3, Pages 335-346.

[4] "Something is meaningful if it is important in one way or another for survival, maintaining a job, social relations, navigating in the world, etc. For example, the differences in color between different mushrooms may be relevant to me because they help me to distinguish those that are poisonous from those that are not." Steels seems to prefer the term "representation" for what I'm calling "objective meaning." Luc Steels (2008) "The symbol grounding problem has been solved, so what's next?"

[5] "I suspect that Johnson (like many others) has mistaken the ability of GPT-3 and its ilk to manipulate linguistic form with actually acquiring a linguistic system. Languages are symbolic systems, and symbols are pairings of form and meaning (or, per de Saussure, signifier and signified). But GPT-3 in its training was only provided with the form part of this equation and so never had any hope of learning the meaning part." Emily Bender (2022) "On NYT Magazine on AI: Resist the Urge to be Impressed."

Acuitas Diary #56 (January 2023)

The big thing this month was finishing the Narrative Engine overhaul and getting all the previous demonstration stories to work in it. I've been getting some questions from newer followers about how the Narrative module works, so I'm going to devote part of this blog to a recap in addition to talking about the updates.

"Three Figures Reading," by Katsushika Hokusai.

Acuitas is designed to be a goal-driven agent, and his story processing reflects a similar picture of other entities with minds. A story is a description of the path some *agent* followed to achieve (or fail to achieve) some *goal* or goals. The sentences that form the plot can be identified by their relevance to the goals of some character or other, and the "action" consists of movement toward or away from goal states. Goal-relevant material comes in two flavors: "problems" (negative events or states that constitute a goal failure when entered) and "opportunities" or "subgoals" (positive states that will fulfill a goal when entered). But there are many similarities in the way these are handled - they're really just two polarities of the same thing - so I've taken to calling them both "issues."

For now, agents are identified by category membership (some types of entity, e.g. humans and animals, are just assumed to be agents). Eventually I would like to include "duck typing"[1] for agents, inferring that something in a story is an agent if it *acts agentive,* but that's future work. Agent goals can be revealed by the story in statements such as "John wanted ...", but agents are also presumed to have certain core goals that motivate all their other goals. These core goals are learned information that is permanently stored in Acuitas' semantic memory database. Core goals can be learned for a category ("Humans want ...") or for an individual ("John wants ..."), with goals for individuals or specific categories superseding those for more general categories. Insofar as Acuitas doesn't *know* what somebody's core goals are, he'll substitute his own. (This is supposed to be an analogizing assumption from the most directly available data. "I know I want X, and you, like me, are an agent - perhaps you also want X?")

The last big ingredient is inference chaining, arising from both logical deduction and cause-and-effect relationships. Some inference rules are hard-coded, but Acuitas can be taught an indefinite number of additional rules. So every story sentence that describes an event or state produces an inference tree of facts that will also be true if that event or state comes to pass. These inference trees are often crucial for determining how something will affect the core goals or immediate goals ("issues") of an agent in the story. 

Let's walk through a story. I'm going to pick one of the more complex examples I currently have, "Prisoner of the Sand," my Acuitas-friendly retelling of Antoine de Saint-Exupéry's story about his plane crash in the Libyan Desert. My version of the story, in (more or less) natural English, follows:

0:"Antoine was a pilot."
1:"Antoine was in an airplane."
2:"The airplane was over a desert."
3:"The airplane crashed."
4:"The airplane was broken."
5:"Antoine left the airplane."
6:"Antoine was thirsty."
7:"Antoine expected to dehydrate."
8:"Antoine decided to drink some water."
9:"Antoine did not have any water."
10:"Antoine could not get water in the desert."
11:"Antoine wanted to leave the desert."
12:"Antoine walked."
13:"Antoine could not leave the desert without a vehicle."
14:"Antoine found footprints."
15:"Antoine followed the footprints."
16:"Antoine found a nomad."
17:"The nomad had water."
18:"The nomad gave the water to Antoine."
19:"Antoine drank the water."
20:"The nomad took Antoine to a car."
21:"Antoine entered the car."
22:"The car left the desert."
23:"The end."

Before these sentences ever make it to the Narrative Engine, they pass through other modules in the text processing chain, which convert them from English into more abstract data structures. These might be thought of as "the gist"; the specific wording is discarded, and the meaning is distilled into some relationship between the key concepts that appear in the sentence. The Narrative Engine operates on these relationship statements only. As it consumes them, it generates data for a flow diagram in which (sometimes abbreviated) versions of the relationships appear in yellow bubbles, connected by arrows to show their sequence in the story. If a story sentence creates an issue or causes one to change state, an arrow is drawn from the sentence bubble to the issue bubble, and labeled with the new state of the issue. When the story is complete, the diagram image is generated by Graphviz.

The flow diagram produced for "Prisoner of the Sand" appears below. Click here for a zoomable full size version: Prisoner of the Sand

Nothing much seems to be happening during the "setup" phase, the first few sentences. But the Narrative Engine is inferring some things under the hood - for example, that when the plane crashes, this puts Antoine in the desert. After this we are introduced to our first Problem: "Antoine was thirsty." This is recognized via inference as a violation of the "be comfortable" goal. Antoine had better do something about that.

And the Narrative Engine proceeds to guess what he might do about it - hence the appearance of a *predction* in association with this sentence, also. There's a problem-solving routine that does some reverse inference chaining, and gets to the idea that Antoine might drink some water to stop being thirsty. This is represented in the diagram by the bubble predict_0.

The thirst is a current ("realized") issue. On the next line we have a *pending* issue, dehydration, which is life-threatening. This still gets entered as a problem, because it's something that will happen without intervetion, and needs to be headed off.

Over the course of the next few sentences, Antoine encounters a variety of obstacles to his immediate goals. He doesn't have water, and it is inferred from this that he cannot drink water - his plan to solve his problems is blocked, and this blockage becomes a new problem in its own right. He wishes to leave the desert, but has no vehicle ... another secondary problem. Then he stumbles upon another person out in the waste, and his problems start getting solved. The nomad gives him water. (The Narrative Engine infers he now has water. Blockage against drinking removed.) He drinks. (Thirst removed, dehydration avoided, prediction fulfilled.) Finally he is brought to a car, which clears the issue that was preventing him from leaving the desert. The final inferential leap made here is that when the car leaves the desert, it takes Antoine with it, fulfilling his last open goal.

The old Narrative Engine was basically capable of doing this, so what's new? Well, in addition to the things I talked about in my last upgrade post: I unified the processing of explicit "could do/couldn't do" sentences with the processing of inferred action prerequisite/action blocker relationships, getting rid of a fair bit of ugly code. I moved the generation of some special "defined by events" inferences, like "John told another agent something that John doesn't believe" -> "John lied," into the main inference chain so they can potentially produce further inferences. I came up with a new way of managing relationships that contain wildcards, like "Where the water was." And I got all the old features tacked on to a cleaner base with more robust fact-matching, better management of events that reverse previous statements, and so on.

This sets the stage for me to use the Narrative Engine for some cool new things this year, and I am twitching to get started.

I also crammed in some work on the Conversation Engine. This stuff's fairly boring. I got rid of some nasty bugs that had been making normal conversations with Acuitas very awkward for a while because I was just too busy to fix them, and worked on cleaning up the code, which came out very convoluted on the first pass.

Until the next cycle,
Jenny

[1] "Duck typing" is the practice of assigning something a type by its behaviors or properties alone, without relying on preexisting labels: "if it walks like a duck and quacks like a duck, it's a duck."