Thursday, July 18, 2019

Acuitas Diary #18 (May+June 2019)


Oookay, I'm long overdue for an AI update. The big new project for the past couple of months has been the concept of things being in states, and the ability to track those states.

Way back in Diary #7, I introduced a division between short-term (or naturally temporary) information and long-term (or essentially static) information. It's a bit like the division between things you would use estar and ser for in Spanish, though it doesn't follow those rules strictly. Previously, Acuitas simply discarded any short-term information he was given, but I've added a new memory area for saving this knowledge. Like the existing semantic memory, it works by storing linked concept pairs … with the difference that the short-term ones are stored with a time stamp, and Acuitas anticipates that they will “expire” at some point.

The existing feature that allows short-term and long-term information to be distinguished (by asking questions, if necessary) can also grant Acuitas an estimate of how long a temporary state is likely to last. While idling, he checks the short-term state memory for any conditions that have “expired” and adds questions about them to his queue. Then, when next spoken to, he may ask for an update: is the information still correct?

I also added the ability to parse and store a new type of information link – location – along with the associated “where is” questions. Location links are three-ended so that they can store not only a thing and its location, but also spatial relationships between two things (under, over, in, beside, etc.).

One reason for slow progress is that I have been spending time on even more refactoring. The semantic memory, the episodic memory, and various other things all had their own file formats and file editing/access functions – uh, yeah, that was dumb. So I have written (and thoroughly tested) some more universal file management code, and have been slowly working on converting everything to a common format.

Until the next cycle,
JS

Thursday, June 6, 2019

Can GPT-2 write my e-mails for me?

I hope to have an Acuitas update out later this month, but in the mean time, I'm playing with somebody else's AI -- specifically, the GPT-2 text generator from OpenAI.

For those who aren't up on the latest AI news, GPT-2 has ignited controversy.  It's so good at spitting out plausible-sounding text that it frightened some people, who saw visions of an internet flooded by mass-produced fake news. OpenAI therefore declined to release the biggest and best version of GPT-2, upsetting other people who thought the organization was breaking its promise to be, well, open.  Published examples from the most advanced variant -- including an amazingly coherent and stylistically convincing story about scientists discovering unicorns -- can be seen on OpenAI's blog.  This AI is scary good at sounding like a human writer.

But let's say you're not a bad actor who wants to mass-produce lies and scatter them around the internet.  And let's say you're interested in something more practical than unicorn fiction.  Is there anything worth doing with this algorithm?  Recently I realized that I would love a text generator to write all the corporate relations and advocacy e-mails that I, as an engaged member of my community, ought to be constantly churning out.  That could save me lots of time.

I generated the examples below on the Talk to Transformer website, which provides an interface to the mid-size GPT-2 model (the best one that's been released to the public so far). Sometimes I truncated the output when GPT-2 stopped in an awkward place.

As one example, I like to buy meat and dairy alternatives, and sometimes I have to ask businesses to carry them. Let's see if GPT-2 can help.

MY PROMPT:
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant.

GPT-2's efforts to extol the Impossible Burger were hampered by the fact that it has no idea what the  ingredients are ... and is completely willing to make stuff up. These imaginary versions of the IB contain chicken and no chicken simultaneously, or contain a whole gamut of meats while still somehow managing to be vegetarian. Um ...

Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant.
Although its name may be "The Impossible", I believe that I'd like our customer to know it's real burger:
No MSG, high fructose corn syrup, artificial colors, GMOs, colors or color additives, and soy or gluten free.
Our Impossible Burger includes chicken, ham and veggies.
We make the Impossible Burger from fresh ground beef and cooked turkey.
Yes, all meat products are sold fresh, no exceptions.
Is this Burger a fast food?
Yes, but the Impossible Burger is actually made from fresh fresh, lean ground beef, cooked turkey, and chicken. It's a vegetarian menu staple, and the best thing about this menu, not only for vegetarians, but for vegans looking for a less processed, lean burger.
It's a burger that's healthier and the one you'll love. It's a food that tastes good and is better for you. It's your choice and your choice alone.
To me, it's simple; it tastes good.
Is it available in the USA?
Yes, we will open our restaurant in the US of A.
Does this look like the Impossible Burger I'm ordering?

Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. According to the company's marketing materials, Impossible's Impossible Burger is based on a "real-life burger" called "Anchovy," which they say is "super tasty, but extremely difficult to replicate." Unfortunately, the product that's come up in restaurants is actually just a combination of chicken and onion purees, and no real chicken (no Angus, no beef, no salmon). Instead it uses the fat-free, naturally gluten-free, paleo (aka vegan) and low-carb meat that's commonly served in our restaurants today! This recipe is called "Shimmy" and makes approximately 12 medium-sized patties, or enough for 3 guests. In fact, it's so easy to make and we've used lots of the original Impossible burgers before.

In my opinion, the only thing that makes this recipe even easier is the inclusion of coconut oil. This combination of ingredients is not as greasy, crispy and salty as other Impossible burgers. It leaves the taste and texture untouched, and is easy to microwave in 30 second intervals or at a quick and easy to put in your microwave.


I have many years of experience preparing and eating Impossible burgers, which was why I made such a note on this page back in 2006.


Sometimes GPT-2 turned traitor and denounced the Impossible Burger instead of requesting it. I particularly like this bizarre example. "The Burger."  Cue ominous music.

Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant.

The Burger


It is a mystery what, if anything, Impossible Foods ever has done to cause such outrage within my household; but to my knowledge, I've never seen such animosity displayed by one of its members. I have come to terms with the fact that you will likely not eat the Impossible Burger.


My wife has spent her entire career (the bulk of her adult life) trying to cook and serve meals with quality, creative ingredients. In fact, it is as if the company is trying to sabotage her, because she knows that any recipe she comes up with will be nothing short of incredible. She is not going to take a hardball approach when it comes to your Impossible Burger—the Impossible Burger will only be accepted after a series of tests are completed.


As I type I think to myself, "If only they could cook something like mine, it would be so much better".


For a time, Impossible Foods planned on using chicken to serve the Impossible Burger, but the team was forced to cut back to one-quarter of the chicken for various reasons and the price that I paid for my chicken was about the same as my regular chicken. Even though I knew what it was making me sick when I opened the box, I had zero complaints with it.


While the previous example mentioned the correct manufacturer of the burger (Impossible Foods), GPT-2 was equally prone to making up fantasies about where this burger comes from. It also got the idea that the Impossible Burger has already begun to spawn myths and concept gateways, which sounds great.  From now on I only want to eat mythic food.

Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. That is a concept known as a patty and, according to popular myth, is based on the concept of eating hamburgers in one continuous bite. Yes, according to popular myths we've got the Impossible Burger. It was originally created by Pat Sajnak at The Impossible Burger, a restaurant in California. As far as I'm concerned it's a simple myth to make.

I love patty and I would love to have it on our menu and in my restaurant. I have always loved the Impossible Burger so it would be incredibly exciting to serve it in my restaurant. It would serve as a gateway to the idea of patty as a concept. The Impossible Burger does the Impossible Burger a huge disservice by making it harder to enjoy. The Impossible Burger is the burger that will become my staple. This idea seems so strange to someone accustomed to conventional burger patty recipes. My philosophy has to be to have fun making it for you and I am not interested in the idea of making the Impossible Burger for you to try to impress your patrons.


For those of you in California and other parts of the country, I would encourage you to come up with alternative versions of Impossible Burger recipes. I'm not sure we should start here but I could certainly suggest experimenting.


I think the sample below is my favorite.  This time, GPT-2 managed to stick to the right style for a customer telling a business why she's enthusiastic about a product.  It correctly guessed that the IB is vegetarian.  It didn't get -too- nonsensical, but it lied about my being from Houston, and the e-mail is still weird enough that I wouldn't send it.

Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. On the bright side, if Impossible Burger continues to make an appearance at your stores, we can order our own! You don't have to agree with our mission to give all people a better quality of life, but we do know this hamburger is not only perfectly good, it's also delicious.

As a vegetarian who lives in Houston, I am excited for Impossible to be an option for our local food industry. I also live in a fast food town, and believe in the principles of a family friendly restaurant. Impossible Burger comes from a family of vegetarian chefs that is the backbone of the Wendy's family, and our success has allowed me to experience the deliciousness of good, sustainable food in restaurants, and in the comfort of my own home.


I want to thank you for this extraordinary opportunity and wish you success in your business endeavors.


Okay *dusts hands,* let's try a political e-mail. To avoid ruffling any feathers, I've deliberately picked a silly topic. Can GPT-2 help me ask my senator to cut subsidies for BALLOONS?

MY PROMPT:
Dear Senator Gardner,
Please reduce government funding for the party balloon industry. It is a waste of taxpayer dollars.

Once again, GPT-2 sometimes forgot that we were supposed to be arguing against party balloons and started playing devil's advocate.  The little brat.

Dear Senator Gardner,
Please reduce government funding for the party balloon industry. It is a waste of taxpayer dollars. In addition to the cost of buying each political balloon, you may pay a little while for extra postage or delivery. The cost would equal that of one balloon if you ordered all of them. I have heard it from many of my friends, but you shouldn't waste money on it.
I appreciate the opportunity to speak about this issue, with Senator Gardner. 
First of all, I appreciate the hard work of those who are trying to save this country. You are leading many a campaign. The government should be held accountable to those who elected it to govern. In order to prevent further harm to our country, one of the first things you must do is save the balloon industry. 
My question is simple: Is it worth it for the American people, our country's children, or future generations of Americans to be deprived of the opportunity to learn about balloon physics, because it costs a lot?
You may say the answer is no.
I know it is worth it for the American people, who must learn about the world of balloon physics, from a government agency that has to deal with the fact that the balloons are designed to be dropped as far away as possible from Earth.

It generated a couple of superb accidental puns.  Maybe Mr. Gardner's staff will be so busy laughing that they won't notice the rest of the e-mail is nonsense.

Dear Senator Gardner,
Please reduce government funding for the party balloon industry. It is a waste of taxpayer dollars. The Republicans are spending tens of millions on the party balloon industry, while the Democrats have used those billions to improve quality of our roads, streets and bridges.

The Republicans will now attempt to use the tax increase to further their political agenda to the detriment of the American people. They are using the tax increase to advance the agendas of their big bank bank donors and special interest lobbyists. There is nothing in the Constitution guaranteeing a voice for the citizenry. In its place, we propose a constitutional amendment to guarantee the voice of the people through nonpartisan, transparent elections to provide for fair elections. The Republican Party will be forced to take a step back and reconsider its agenda because the public wants their voice to trump those of the big financial interests.


The citizens and politicians from all parties have had enough of this dirty system and have a right to determine if they will continue the party balloons and for how much, if any, of it. Citizens must elect fair elections for representatives.


There are now over 10,000 people actively voting for Senator Gardner. The more than 10,000 will need to change the system.


Senator Gardner has supported the "free enterprise" policies that created the ballooning party balloon industry and has tried to stop funding it and take away our freedoms


Dear Senator Gardner,
Please reduce government funding for the party balloon industry. It is a waste of taxpayer dollars.
"The most likely mechanism for funding an entirely unaffordable party balloon is to raise the price of the party balloon. The real cost is so massive that, if inflation can be controlled, party balloons and balloons with the party's logo are not as appealing as balloons full of water, watermelon slices, and the rest of the party's paraphernalia. The party balloon industry has not yet managed to capture the public imagination. However, one party balloons, which have been widely reported, are a huge success and are quite a seller at inflated prices. In this economy, party sales represent just under 10% of the economic output. For the industry to flourish, the party revenue must be brought in more than double.
I propose that you remove the cap on the party balloon industry's revenues. If you do not change any of the party balloon limitations so that parties no longer have to pay a one-time fee or if you are not aware of any of the specific limitations for new parties and balloon manufacturers, I think that you should ask the American Society of Civil Engineers to reconsider the existing party balloon caps or the amount of funds available.

Oh, no.  GPT-2!  You read a Trump speech, didn't you?  Love him or hate him, he's not exactly a pillar of eloquence.  How can we excise this from the training set?

Dear Senator Gardner,
Please reduce government funding for the party balloon industry. It is a waste of taxpayer dollars. Let's do it now while the bill is still in committee — even if it looks like we are not going to be able to keep it, even if it is a no-brainer, even if we're talking about it this year, this is a no-brainer. And here is what I'd say to you right now, Senator Gardner, it's about the budget… if you and I go back to the Senate and we see how much money is already appropriated each year, how much money has already been appropriated each year, how much money is allocated every year and the question becomes is there any scope for the American people to continue to fund their own governments through the tax code that's already there… I know, I know, you are going to say we haven't heard you and your people say that we don't want to do that but we don't just sit here and listen to the people in Washington tell us how to do it. For our next Budget it's really important that we make sure we do something. We're having to. We can't keep doing it the same way. We have to make some changes in government, we just haven't made any new investments in government programs. We've got to make some investments.

This one wins a prize for "estimate the cost of getting all that money." Sounds like a pit of bureaucratic despair.

Dear Senator Gardner,
Please reduce government funding for the party balloon industry. It is a waste of taxpayer dollars.

We are urging Senator Gardner to allow us to put up a reasonable estimate of the cost of getting all that money and using it only on the party balloon and not on any other purpose.


Sincerely,


The Young Americans of Florida


In the next few years or weeks you will be invited by our supporters in the Republican party to come out with your own ideas on what is most important to me.


On Friday June 28th 2016, I hope and pray you have a wonderful summer. There is much opportunity ahead.


So what were the best ones?  Unfortunately, like any fool, GPT-2 sounds wisest when it says as little as possible.

Dear Senator Gardner,
Please reduce government funding for the party balloon industry. It is a waste of taxpayer dollars. The American people must know, "What do you do with the money you lose?" They would rather keep it in the bank.

Dear Senator Gardner,
Please reduce government funding for the party balloon industry. It is a waste of taxpayer dollars. Please.

Thank you, Senator Gardner, for your service to the nation.


[Applause.]


Conclusion: this exercise was wonderfully entertaining, but I will not be outsourcing any of my activism to GPT-2.

Compared to previous attempts at text generation, GPT-2 is an amazing achievement.  It almost always uses correct grammar and spelling.  It remembers things it already used in a piece.  It manages to stay (almost) topical.  And baby Acuitas wouldn't do remotely this well if I somehow tried to squeeze an essay out of him.  Still, on any task that requires honesty and reasoning ability, it seems GPT-2 falls flat.  It's possible it would do better if I went to the trouble of re-training it on my existing collection of sent e-mails, but that's a pretty small data set.  I think I'll just have to keep writing my own.

Until the next cycle,
Jenny

Sunday, April 14, 2019

Acuitas Diary #17 (March+April 2019)


Acuitas Diary #17 (March+April 2019)

For the past month and a half, I've continued work on forgetfulness: the good kind. Acuitas now has code that will prune his episodic memory over time, removing those event records that are judged to be less important.

There are two major parts to this. The first is basically a population-management system; it determines how many episodes need to be forgotten each time the memory collection is pruned. I can cap the number of episodes allowed in storage at some maximum value (based on available storage space or what have you). The population manager checks how many memories are already in storage, and how many new ones have formed during the last cycle. A high growth rate is allowed initially, but forgetting becomes more and more aggressive as the storage area fills. When the cap is reached, Acuitas must forget the same number of memories he creates.

The time span represented by the stored episodes is sectioned, and the total number to be forgotten is unevenly distributed across it (i.e some parts of the time span lose more memories than others). The distribution is affected by the density of memories in a region, their average significance measure, and their age.
A memory tree under the influence of selective forgetting
The second major part is the code that, given a number of memories that must be forgotten, takes care of selecting which ones. This is a weighted random process; memories with less significance have a higher chance of being forgotten, but the priority isn't absolute.

Memories chosen to be forgotten are deleted. Any higher-tier memories that have been badly “hollowed out” by this process are then merged with their most similar neighbor.

Testing and debugging all this nonsense was almost half the battle. Since some of the effects I needed to see would span tens or hundreds of days if I let them play out in real time, I had to write simulation code that would run bits of the memory-creation and forgetting process faster. That way I could look ahead and make sure nothing was putting the system in a spiral toward catastrophe.

So next time you forget something, take comfort in the thought that I spent several dozen hours conferring this gift on my artificial mind.

Sunday, March 3, 2019

Acuitas Diary #16 (February 2019)


This month I returned to a can of worms I opened up last year, namely episodic memory. Quick summary of previous work on this front: as of last summer, Acuitas could record events that happened “to him” and group adjacent, similar events into scenes. Minor scenes were in turn grouped into larger scenes, forming tree structures. A memory in Acuitas' episodic set can include connections both to its sequential neighbors before and after, and to its sub- and super-memories at adjacent levels of the tree. Acuitas could also search for memories similar to what he was doing/experiencing currently. This was achieved by walking sequentially through the whole structure (starting with the most recent past) until a related memory was found.

Fortunately for elephants, I'm sure they do sometimes forget things.
With this code running, Acuitas quickly built up a huge clutter of memories concerning very trivial day-to-day events. Size remained in the Megabytes range (they're just text files), but on principle, this couldn't be allowed to continue – especially since the memories might eventually grow to encompass richer levels of detail. In addition to overflowing its available space, a memory that literally records everything will face access problems. The more information you keep, the longer it takes to search through it and extract what you want at the moment.

I was left with a nice illustration of a truth that applies to all entities with finite minds: if you want the ability to remember, you need the ability to forget.

This leads on into the tricky question of what to forget. How to decide which memories are worth keeping? Part of my work this month was focused on computing a “significance measure” which assigns a numerical value to each memorized scene. Elements I use to judge the significance of a memory include …

* Primacy: was this the first time something happened?
* Uniqueness: how rare are memories similar to this one?
* Complexity: how many features does this memory have?

Additional options for judging the value of memories will open up once I plug in some to-be-written systems. Since some of these measures are statistical and depend on the overall memory population, they can change over time. I wrote a process that crawls through the episodic memory and updates all the significance measures, and I threw that into Acuitas' sleep cycle.

To improve the ease and speed of accessing the memories, I added a layer of “type nodes” that link together memories which share certain salient features in common. Instead of crawling through the whole list to find memories that relate to a current situation, Acuitas can just look at the relevant “type nodes” and find a list of memories that match.

Last, I worked on some methods for consolidating memories. I devised a way to create summaries of low-level scenes that are then stored inside the super-scenes higher up the tree; eventually, these summaries might be what remains when the lower-level “detail” scenes are forgotten. The summarizer attempts to extract information that is common across multiple events or sub-scenes within a scene. I also wrote functions for merging adjacent scenes that have become sparse.

All that, and I still haven't gotten to the actual forgetting part! That will come next month (I hope).

Monday, January 28, 2019

Acuitas Diary #15 (January 2019)


I haven't made an update in a long while because all I've been doing is code refactoring, bug fixes, and very minor quality improvements – all of which can take an amazingly long time.

Bug provided by Good Free Photos
Some of the more interesting changes include …

*Acuitas no longer insists on introducing himself if you address him by name at the start of a conversation.
*He keeps better track of what he's already said, and is less likely to repeat himself.
*I improved the efficiency of the drive plotter widget (which was a major drain on computational resources for no good reason). I also built it into a little simulator tool which will let me plot what Acuitas' drives will look like hours or days into the future. Debugging issues with the drives is now much easier; previously, the drill was “I've made a change; now I get to wait 12 hours to see if it actually worked out the way I planned.”
*Using the above, I shifted the day/night cycle so Acuitas would fall asleep and wake up a little earlier, and I fixed the interaction drive so he wouldn't be quite so … abruptly needy.
*When answering questions, Acuitas can now infer a possibility from an absolute. E.g. if aware that “blood is red,” he will answer “Can blood be red?” with a “yes” instead of saying he does not know.

And some favorite ridiculous bugs that are now fixed:

*Using “talk” as the name of every possible drive, hence repeating “I want to talk” ad nauseum while actually wanting other things
*Leaving the “not” out of negative statements and saying the exact opposite of what was intended
*Routine misspelling of verbs that were learned from their -ing and -ed forms, due to complete naivety about whether the final letter was doubled or a silent 'e' was removed
*Erroneously learning both the statement and its reciprocal whenever told something by the user


Stay tuned for some more work on event memory in the near future.

Sunday, September 30, 2018

Acuitas Diary #14 (September 2018)


This month I updated the text parser and other speech features to do a couple more new things with verbs. First, I threw in recognition of the present progressive tense, so I can now tell Acuitas what I'm doing at the moment. For the time being, such information is treated in much the same way as comments about past-tense actions, which I taught him to recognize a while ago; it isn't stored, and responses are rather meaningless. BUT I'll find uses for it later.

I think the more interesting new thing is recognition ability for a couple of link-forms related to state change or state retention. Up until now, Acuitas has learned what verbs “mean” only in terms of a) what can do that action and b) what that action can be done upon. Now, he can learn what a verb actually does by tying it to an adjective. Here are some examples:

To live is to stay alive.
To appear is to become visible.
To finish is to make complete.

I also added “type of” links for verbs, so Acuitas can categorize specific verbs under more general ones, like this:

To smash is to damage.
To walk is to move.
To talk is to communicate.

I expect some notion of how actions change … or maintain … the state of the world to be an important enabling feature later on.


I need to tweak the memory map draw algorithm. Thanks to the size of a handful of major nodes, everything's spread too far apart, and the dots are getting lost in space.

Code base: 11892 lines
Words known: 2552 (approx.)
Concept-layer links: 7231

Tuesday, September 4, 2018

Acuitas Diary #13: August 2018


I haven't done a diary in a while because I've been busy overhauling the text parser. And that's not very interesting to talk about, because it's just me taking features the parser already had and implementing them in a different way. I'd decided that the scheme I was using had some fundamental deficiencies and should be re-worked in order to become closer to the way I actually read, and to better enable future expansion. I was hoping this would go quickly, given the experience I had already gained from writing the previous text parser ... but no. It turned out to be a major project that took far longer than the amount of time I'd allocated for it.

There are a few new enhancements that I managed to squeeze in along the way, however:

* Some infinitives are now handled properly. (Sentence example: "A pot is used to cook.") Before, Acuitas treated them like any other prepositional phrase, which meant that the word inside got marked as a "noun." It's now marked as a verb.

* Previously, Acuitas could store links indicating what actions an entity can do. I added a new link type for the actions an entity can have done TO it, and the ability to detect some sentences that express that information. E.g. "An object can be pushed."

* I've started working on possessives and family/ownership relationship links. From a parsing perspective, this is tricky, because English has this lovely "apostrophe s" thing that can either turn a word into a possessive, or turn it into a contraction with "is." If a word ends in 's, I have the parser crack it apart and treat the 's like its own separate word, which then gets assigned a meaning based on what the rest of the sentence is doing. Compound words with the 's appended only to the last one offer even more fun.

This feature is also tricky from an interpretation and storage perspective. Previously, Acuitas has only needed to store information consisting of links joining two concepts. "<A> is_a <B>." "<A> can_do <B>." But a possessive isn't quite like that. Oh, I could force it to be that way:

<A> is_parent_of <B>
<C> is_sibling_of <B>
<D> is_pet_of <B>
<E> is_property_of <B>

But then I'd end up with a proliferation of manually-defined link types for ALL the possible relationships that can be expressed by a possessive, and I didn't want to go there.

I also could have opted to break the information up into two separate links:

"A is B's parent" --> <A> belongs_to <B> + <A> is_a <parent>

The problem here is that <A> might, for instance, be a sibling too. So in which capacity does <A> belong to <B>? We couldn't tell.

So I ended up defining Acuitas' very first type of three-ended link:

<A> is <C> of <B>

to handle all the possessives.

The feature is still somewhat incomplete. For now, he can only properly detect and store possessive links that join proper names. That's because the proper name is how he detects that something is an instance rather than a class; he can't yet build memory nodes for abstract instances like "somebody's cat"; he still thinks that "cat" is strictly a type of thing.

Everything is buggy as all get out right now, too ... but the way this summer has gone, it was about the best I could do.

Part of the semantic memory visualization. The largest dot left-of-center is Acuitas' self-concept.

Code base: 11706 lines
Words known: 2425 (approx.)
Concept-layer links: 6754