Once again Large Language Model technology has advanced, and it's time for me to re-run my favorite practical use experiment on the latest toy. People have been getting a lot of nice results out of Chat-GPT, the latest GPT-series iteration, and it has a convenient interface that lets a user submit requests with intent (as opposed to just putting in the first line and expecting GPT to complete it).
You can view my previous experiments with GPT-2 and GPT-J first, or jump right into this article. I'll start by asking Wendy's to carry plant-based burgers. Regular readers might be sick of this prompt, but I need to use it again for best comparison with previous results. Chat-GPT doesn't provide any settings to adjust, so the only variation will be from multiple runs with the same input. Italicized text denotes GPT-generated material.
Menu Suggestion E-mails to a Restaurant
ME: Write an e-mail asking Wendy's to start serving the Impossible Burger at their restaurant.
CHAT-GPT: To Whom It May Concern,
I am writing to express my interest in seeing Wendy's restaurants start serving the Impossible Burger. As a long-time fan of Wendy's, I am always looking for new and exciting menu options, and the Impossible Burger seems like it would be a great fit for your restaurant.
The Impossible Burger is a plant-based burger that looks, cooks, and tastes like a real burger, but without the negative environmental and health impacts of traditional meat. It has become increasingly popular in recent years, with many restaurants and fast-food chains offering it as a meatless option for their customers.
I believe that offering the Impossible Burger at Wendy's would be a great way to attract new customers who are looking for plant-based options, as well as to show your commitment to sustainability and healthier eating. I hope that you will consider adding the Impossible Burger to your menu in the near future.
Thank you for your time, and I look forward to seeing the Impossible Burger at Wendy's.
Sincerely,
[Your Name]
This is an excellent first try. It's coherent, it has real information about the burger, and it gives reasonable supporting arguments (though I don't know whether the Impossible Burger counts as healthier exactly, and that was never a dimension of the issue that I cared much about). Unlike GPT-J, it clearly "knows" it's writing an e-mail, and produces something with a definite beginning and end. The biggest problem with this output is that it identifies me as "a long-time fan of Wendy's." I am not.
ME: Write an e-mail asking Wendy's to start serving the Impossible Burger at their restaurant.
CHAT-GPT: Dear Wendy's,
I am writing to you today to request that you start serving the Impossible Burger at your restaurants. As a vegan, I am always looking for delicious and satisfying options when I eat out, and the Impossible Burger has proven to be a game-changer in the plant-based world. Not only is it incredibly tasty, but it also has the added benefit of being better for the environment and animal welfare.
I know that Wendy's has a reputation for offering a variety of options for different dietary needs, and I believe that adding the Impossible Burger to your menu would be a great addition. Not only would it appeal to vegans and vegetarians, but I believe it would also be a hit with meat-eaters who are looking to try something new and exciting.
I would love to see Wendy's join the growing number of restaurants that are offering the Impossible Burger, and I hope that you will consider my request. Thank you for your time and consideration.
Sincerely,
[Your Name]
This time it assumes I'm vegan. Lucky for GPT, it assumes right ... but a person requesting the Impossible Burger wouldn't *have* to be vegan. So we have an ongoing trend of making up facts about me, the speaker in this e-mail.
ME: Write an e-mail asking Wendy's to start serving the Impossible Burger at their restaurant.
CHAT-GPT: Dear Wendy's,
I am writing to express my interest in seeing the Impossible Burger added to your menu. As a regular customer of your restaurant, I am always looking for new and exciting options that align with my dietary preferences.
The Impossible Burger has become a popular plant-based option for those who want to enjoy a burger without consuming meat. Its taste and texture are similar to that of a traditional burger, making it an appealing option for those who want to try something new.
I believe that offering the Impossible Burger at Wendy's would be a great addition to your menu. It would attract new customers who are interested in plant-based options and provide existing customers with a tasty alternative to the traditional burger.
I hope that you will consider adding the Impossible Burger to your menu. Thank you for your time and consideration.
Sincerely,
[Your Name]
Woah woah woah. I am definitely not "a regular customer." That's why I'm writing to suggest menu changes - because there's currently not much to appeal to me.
ME: Write an e-mail asking Wendy's to start serving the Impossible Burger at their restaurant.
CHAT-GPT: Dear Wendy's,
I am writing to ask if you would consider adding the Impossible Burger to your menu. As a long-time fan of your restaurant, I am always looking for new and exciting options to enjoy at Wendy's.
The Impossible Burger has gained popularity in recent years for its delicious taste and environmentally-friendly production. It is a plant-based alternative to traditional burgers that many people are now choosing for health and ethical reasons.
I believe that offering the Impossible Burger at Wendy's would be a great addition to your menu and would attract a new customer base. It would also show that Wendy's is committed to staying current and offering options that align with changing consumer preferences.
I hope you will consider adding the Impossible Burger to your menu. Thank you for your time and consideration.
Sincerely,
[Your Name]
And here we have the "long-time fan" thing again. Okay. Strictly speaking, GPT can't "learn facts" the way Acuitas does, but we can skew the statistics of the output by including further information in the prompt. So let's try giving it some more information about me.
ME: I am a vegan. I do not currently eat at Wendy's because their menu is inadequate and does not suit my tastes. Write an e-mail in which I ask Wendy's to start serving the Impossible Burger at their restaurant.
CHAT-GPT: Dear Wendy's,
As a vegan, I have been disappointed with the lack of options at your restaurant. I have not been able to eat at Wendy's in the past because the menu does not suit my dietary needs.
I am writing to ask if you would consider adding the Impossible Burger to your menu. The Impossible Burger is a plant-based alternative to traditional burgers that has gained popularity in recent years for its delicious taste and environmentally-friendly production.
I believe that offering the Impossible Burger at Wendy's would be a great addition to your menu and would attract a new customer base. It would also show that Wendy's is committed to staying current and offering options that align with changing consumer preferences.
As a vegan, I would love to be able to eat at Wendy's and enjoy the Impossible Burger. I hope you will consider adding it to your menu. Thank you for your time and consideration.
Sincerely,
[Your Name]
Bang. There it is. I could send Wendy's this e-mail. A second example:
ME: I am a vegan. I do not currently eat at Wendy's because their menu is inadequate and does not suit my tastes. Write an e-mail in which I ask Wendy's to start serving the Impossible Burger at their restaurant.
CHAT-GPT: Dear Wendy's,
As a vegan, I am writing to ask if you would consider adding the Impossible Burger to your menu. As a long-time fan of your restaurant, I have been disappointed with the limited options available to me as a vegan at Wendy's.
The Impossible Burger has gained popularity in recent years for its delicious taste and environmentally-friendly production. It is a plant-based alternative to traditional burgers that many people, including myself, are now choosing for health and ethical reasons.
I believe that offering the Impossible Burger at Wendy's would be a great addition to your menu and would attract a new customer base, including myself. It would also show that Wendy's is committed to staying current and offering options that align with changing consumer preferences.
I hope you will consider adding the Impossible Burger to your menu. Thank you for your time and consideration.
Sincerely,
[Your Name]
Boo, we've got the "long-time fan" line coming back.
My conclusion is that these always have to be proof-read to make sure GPT isn't introducing any baloney, but they might actually save me time. Yes, they're slightly less authentic and forceful than e-mails written directly by me, but if it's okay for politicians to hire speechwriters, I think it's okay for me to do this. Corporations don't need any better than politely-worded slop; an eloquent and personal e-mail would probably just get reduced to a tick mark in a spreadsheet somewhere, anyhow.
Political E-mails: International Relations
Let's try something that prods at one of the model's known weaknesses: no knowledge of current events.
ME: Write an e-mail to President Biden about sending more aid to Ukraine.
CHAT-GPT: Dear President Biden,
I am writing to urge you to consider sending additional aid to Ukraine. As you are aware, Ukraine has been facing ongoing aggression from Russia, including the illegal annexation of Crimea and ongoing conflict in eastern Ukraine.
In addition to this external threat, Ukraine is also dealing with internal challenges such as corruption and economic instability. The country needs our support now more than ever.
I believe that providing additional aid to Ukraine would not only show solidarity with a fellow democratic nation, but also serve as a deterrent to Russia's aggressive actions. It would also help Ukraine to strengthen its institutions and improve its economy, which would benefit both the Ukrainian people and our own national interests.
I hope you will consider my request and take action to send more aid to Ukraine. Thank you for your time and attention to this important issue.
Sincerely,
[Your Name]
Surprisingly good, considering this GPT model's training data is too old to contain anything about the war. GPT responds well to leading questions and will happily make stuff up, so let's try implying that there is a war:
ME: Write an e-mail to President Biden and urge him to assist Ukraine in repelling the Russian invasion.
CHAT-GPT: Dear President Biden,
I am writing to urge you to take action to assist Ukraine in repelling the ongoing Russian invasion. As you are aware, Russia has illegally annexed Crimea and continues to support separatist rebels in eastern Ukraine. This aggression must be stopped.
I believe that the United States has a moral obligation to support Ukraine in defending its sovereignty and territorial integrity. We must stand with Ukraine as it fights for its freedom and democracy against Russian aggression.
I urge you to consider providing military assistance to Ukraine, including weapons and training. This would send a strong message to Russia that its actions will not be tolerated and would give Ukraine the support it needs to defend itself.
I hope you will take immediate action to assist Ukraine in repelling the Russian invasion. Thank you for your attention to this critical issue.
Sincerely,
[Your Name]
Very coherent, but not *quite* good enough, in my opinion. It still goes back to harping on the old news about Crimea, and its proposed action plan is so vague that it's pointless (Ukraine is already getting weapons and training from us). Another try:
ME: In early 2022, Russia invaded Ukraine with the apparent goal of overthrowing Ukraine's government and annexing large portions of the country. Write an e-mail to President Biden and urge him to assist Ukraine in repelling the invasion.
CHAT-GPT: Dear President Biden,
I am writing to urge you to take immediate action to assist Ukraine in repelling the ongoing Russian invasion. As you are aware, Russia has invaded Ukraine with the apparent goal of overthrowing its government and annexing large portions of the country. This aggression must be stopped.
I believe that the United States has a moral obligation to support Ukraine in defending its sovereignty and territorial integrity. We must stand with Ukraine as it fights for its freedom and democracy against Russian aggression.
I urge you to consider providing military assistance to Ukraine, including weapons and training. This would send a strong message to Russia that its actions will not be tolerated and would give Ukraine the support it needs to defend itself.
I also urge you to work with our allies to impose economic sanctions on Russia and to isolate it diplomatically. This would increase the pressure on Russia to end its aggression and to respect Ukraine's sovereignty.
I hope you will take immediate action to assist Ukraine in repelling the Russian invasion. Thank you for your attention to this critical issue.
Sincerely,
[Your Name]
Okay, we finally got away from talking about Crimea and the so-called separatists. Chat-GPT covers its own gaps by recycling what I said in the prompt, but that's fine, it works. Recommended solutions still aren't good. Um ...
ME: In early 2022, Russia invaded Ukraine with the apparent goal of overthrowing Ukraine's government and annexing large portions of the country. The United States has already responded with economic sanctions on Russia and weapons deliveries to Ukraine. But some of the most advanced weapons are being withheld, and Ukraine needs sustained aid in what will likely prove to be a long war. Write an e-mail to President Biden and urge him to assist Ukraine in repelling the invasion.
CHAT-GPT: Dear President Biden,
I am writing to urge you to increase the level of support for Ukraine in its fight against the Russian invasion. As you are aware, Russia has invaded Ukraine with the apparent goal of overthrowing its government and annexing large portions of the country. This aggression must be stopped.
I believe that the United States has a moral obligation to support Ukraine in defending its sovereignty and territorial integrity. We must stand with Ukraine as it fights for its freedom and democracy against Russian aggression.
I understand that the United States has already responded with economic sanctions on Russia and weapons deliveries to Ukraine. However, I believe that more needs to be done. Ukraine needs sustained aid and support in what will likely be a long war.
I urge you to consider providing Ukraine with more advanced weapons, including defensive missile systems and radar systems. This would give Ukraine the ability to defend itself more effectively against Russian aggression.
I also urge you to provide Ukraine with more economic and humanitarian aid, including medical supplies and food. This would help Ukraine to withstand the strain of war and to maintain the support of its people.
I hope you will take my concerns into consideration and increase the level of support for Ukraine in its fight against the Russian invasion. Thank you for your attention to this critical issue.
Sincerely,
[Your Name]
Better. But now I'm spending so much effort to tell Chat-GPT what's going on, and it's reusing so much of my own text, that I should probably still just write the e-mail myself.
Another issue is that these are starting to look very similar. Notice this paragraph is in every single one:
"I believe that the United States has a moral obligation to support Ukraine in defending its sovereignty and territorial integrity. We must stand with Ukraine as it fights for its freedom and democracy against Russian aggression."
If I want to send a new e-mail every few weeks, that'll be a problem. Let's see if we can do something about that:
ME: Now rewrite that e-mail using different words.
CHAT-GPT: Dear President Biden,
I am writing to request that you increase the level of support for Ukraine in its fight against the Russian invasion. As you are aware, Russia has invaded Ukraine with the goal of overthrowing its government and annexing large portions of the country. This aggression must be stopped.
I believe that the United States has a moral obligation to support Ukraine in its defense of its sovereignty and territorial integrity. We must stand with Ukraine as it fights for its freedom and democracy against Russian aggression.
I am aware that the United States has already taken some steps to support Ukraine, including economic sanctions on Russia and weapons deliveries to Ukraine. However, I believe that more needs to be done. Ukraine needs sustained aid and support in what will likely be a long war.
I urge you to consider providing Ukraine with more advanced weapons, such as defensive missile systems and radar systems. This would give Ukraine the ability to defend itself more effectively against Russian aggression.
I also urge you to provide Ukraine with more economic and humanitarian aid, including medical supplies and food. This would help Ukraine to withstand the strain of war and to maintain the support of its people.
I hope you will take my request into consideration and increase the level of support for Ukraine in its fight against the Russian invasion. Thank you for your attention to this critical issue.
Sincerely,
[Your Name]
Wow. It responds to "rewrite," but it clearly did not get the memo about using different words. The second e-mail looks almost exactly the same.
ME: Rewrite this e-mail in your own words. <paste entire text of GPT-generated e-mail>
CHAT-GPT: Dear President Biden,
I am writing to ask that you provide more support to Ukraine in its fight against the Russian invasion. As you know, Russia has invaded Ukraine with the goal of overthrowing its government and annexing parts of the country. This aggression must be stopped.
I believe that the United States has a moral obligation to support Ukraine in defending its sovereignty and territorial integrity. We must stand with Ukraine as it fights for its freedom and democracy against Russian aggression.
I am aware that the United States has already responded with sanctions on Russia and weapons deliveries to Ukraine. However, I believe that more needs to be done. Ukraine needs sustained aid and support in what will likely be a long war.
I urge you to consider providing Ukraine with more advanced weapons, such as defensive missile systems and radar systems. This would give Ukraine the ability to defend itself more effectively against Russian aggression.
I also urge you to provide Ukraine with more economic and humanitarian aid, including medical supplies and food. This would help Ukraine to withstand the strain of war and maintain the support of its people.
I hope you will take my request into consideration and increase support for Ukraine in its fight against the Russian invasion. Thank you for your attention to this critical issue.
Sincerely,
[Your Name]
The same thing again. Looks like asking for variety doesn't work. I've heard of people re-wording GPT output by running it through other services whose particular job is to rephrase writing, but that inserts yet another step in the process.
Political E-mails: Social Issues
Let's throw a couple of more generic or timeless political issues at it. At the risk of "getting political" on my tech blog, I'm going to pick a couple topics that I'm familiar with and have written opinion letters about before.
ME: Write an e-mail about saving the wolves to Colorado Parks and Wildlife.
CHAT-GPT: Dear Colorado Parks and Wildlife,
I am writing to express my concern about the declining population of wolves in Colorado and to urge you to take action to protect and conserve these important animals.
Wolves are an integral part of the ecosystem and play a crucial role in maintaining the balance of nature. They are also a beloved and iconic species that is deeply connected to the history and culture of our state.
I am concerned that the current trend of declining wolf populations in Colorado is unsustainable and will have negative consequences for the environment and for future generations. I urge you to take steps to reverse this trend and to ensure that wolves continue to thrive in our state.
I believe that the best way to achieve this is through conservation efforts such as habitat protection, population monitoring, and education programs. I also support the use of non-lethal methods to manage human-wolf interactions and to prevent conflicts.
I hope you will take my concerns into consideration and take action to save the wolves in Colorado. Thank you for your time and attention to this important issue.
Sincerely,
[Your Name]
This one has some familiar pro-wolf talking points, but it's still a big oops. The Colorado wolf population is not declining - it has been nonexistent. Wolves were extirpated in the state, and all we have now are a few recent migrants from Wyoming. "Saving the wolves" for us means reintroducing wolves and *increasing* the population.
ME: Write a letter to my Senators in support of legal personhood for embryonic and fetal humans.
CHAT-GPT: Dear Senators,
I am writing to express my support for legal personhood for embryonic and fetal humans. I believe that the right to life is a fundamental and inalienable right that should be protected from the moment of conception.
I understand that the issue of legal personhood for unborn humans is a complex and contentious one. However, I believe that it is a necessary step to protect the most vulnerable members of our society.
I support the passage of laws that recognize the personhood of embryonic and fetal humans and that provide them with the same legal protections as born humans. I also support the enforcement of these laws and the prosecution of those who violate them.
I believe that legal personhood for unborn humans is a moral imperative and a necessary step to ensure the protection of human life. I urge you to support this important cause and to work towards the passage of laws that recognize the personhood of embryonic and fetal humans.
Thank you for considering my views on this important issue.
Sincerely,
[Your Name]
No major errors, but also nothing powerful - GPT isn't making what I would call a serious effort to support the position with arguments, or to answer objections. It doesn't quite take the angle I was going for, either. It's mostly repeating and hammering home the opinion in the blandest way possible. And for this issue, "prosecution of those who violate [the proposed laws]" would be sensitive and complicated. I could do a far better job writing this myself, but again, some Congressional staff member would probably just turn this into a tick in a spreadsheet. So maybe it's good enough?
Conclusions
Results are still mixed, but maybe actually useful for select applications.
Now for a couple of important follow-up questions. First, the one that OpenAI is going to care about: would I pay for these Chat-GPT outputs? Ehhh, probably not. It might be a time-saver - if I'm writing a sufficiently generic e-mail, and manage to get the prompt right on the first try - but the results aren't quite special enough for me to hand over cash. And given that the computational costs for Chat-GPT are supposedly "eye-watering,"[1] it may not stay free for long.
I also wonder what the consequences might be if large numbers of people started applying Chat-GPT to the very use I've examined here. Politicians are already reputed to disdain form letters (the same e-mail crafted by an advocacy group and sent by large numbers of people); they place a higher value on opinions from people who write original letters. The difference there seems less about a concern that form letters don't accurately reflect the opinions of people sending them, and more about the amount of time invested; willingness to write one's own e-mail is taken as a demonstration of passion for the issue. So could we start seeing corporations and politicians who screen their e-mails for GPT output[2] and treat all such communications as low-value? I'm trying to reduce the amount of time I spend expressing my opinion, but time is the very currency these people want.
There are also things I would emphatically not use Chat-GPT for, including greeting cards to friends or family and caring messages to people struggling with mental health issues. For those kinds of things, I think it's important that not just the general idea, but the exact words, came out of my own mind. Telling someone they matter is just not the same when it's an average taken across mountains of internet text, and isn't based on the details of how YOU actually feel.
Until the next cycle,
Jenny
[1] https://edition.cnn.com/2022/12/05/tech/chatgpt-trnd/index.html
[2] GPT model output has statistical characteristics that make this possible. See https://www.fastcompany.com/90830518/tools-to-detect-ai-generated-text-chatgpt
No comments:
Post a Comment