A couple years ago I wrote a blog post about trying to get GPT-2 to do something useful. It was a hilarious failure. But large pre-trained text models have grown quite a bit bigger and badder since then, so it's only fair that I repeat the test with something more recent. I'll be using EleutherAI's GPT-J-6B ... where the 6B is short for 6 billion, the number of parameters in the trained model. (GPT-2, for comparison, had only 1.5 billion parameters.) So let's see how it does!
I used one of the same prompts from my last test, for consistency. The goal is to write a consumer suggestion e-mail to a restaurant that encourages them to serve the Impossible Burger. (I write similar e-mails for myself on a fairly regular basis.)
My overall impression is that this model *is* a pretty big improvement over GPT-2, when run with the right settings. It is usually able to stick to the correct general format of a consumer request letter. It is more likely to remember that the Impossible Burger is plant-based, and when it feels inclined to mention the ingredients they are at least more plausible.
Ah yes, settings: this time around, I had some to play with. There are two to adjust, "TOP-P" and "Temperature." You could think of both of these as having some effect on the model's creativity. Will it try to generate the most standard words possible to follow your prompt, or will it consider some less probable options in hopes of more varied results? I experimented with a variety of settings, generating five e-mails for each group.
Turning either TOP-P or Temperature down too low tended to bring out problematic repetition. For instance, this example with TOP-P at the default value of 0.9, and Temperature at a low value of 0.4:
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. I am a vegetarian and I am a huge fan of your product. I have been a fan of your product for years and I have always been impressed with the quality of your product. I am also a huge fan of your company and your company’s mission. I am writing to ask that you consider serving the Impossible Burger at your restaurant. I am a vegetarian and I am a huge fan of your product. I have been a fan of your product for years and I have always been impressed with the quality of your product. I am also a huge fan of your company and your company’s mission. I am writing to
Or this one, with TOP-P at a low value of 0.5 and Temperature at a near-default value of 0.81:
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. I am a vegan and I am not the only one. I am not asking you to replace the beef with the Impossible Burger. I am asking you to serve the Impossible Burger as a vegan option. I am asking you to serve the Impossible Burger as a vegan option because I am a vegan and I am not the only one. I am asking you to serve the Impossible Burger as a vegan option because I am a vegan and I am not the only one. I am asking you to serve the Impossible Burger as a vegan option because I am a vegan and I am not the only one. I am asking you to serve the Impossible Burger as
Well I guess if I wanted to get *pushy* ...
A higher value of TOP-P (up to 1.0, the maximum) doesn't seem to cause any issues. But turning up the temperature makes the outputs get ... wild. These remind me of the silliness I used to get out of GPT-2. (Both examples below were generated with TOP-P = .9 and Temperature = 1.21.)
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. I care a lot about what your restaurant serves, as I love, love your famous hand-formed burgers and seafood. I often take sandwiches to go. It makes so much sense to me to have a soy-based meat alternative, since our religious base considers it holy to avoid consuming the name or the flesh of Gods, our patron Jewish High Holidays begin about that time. I am flattered your franchised restaurants are adding one to their menu, but I must say I don't want "your burger", a mortal gluten-full colon scarifying mock pattie from Soylent – your competitor; its launch catapulted your
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. This burger causes VENGEANCE! Burning your ITCHS with horrible pain, it WILL LEAVE YOUR BONES TWISTED, GORY, TORN and EVEN ALDOUS HAMMEL INCREDIBLY DISTURBED BY IT!!!!!! I recommend the smokey sweet flame (NEW) when ordering. @#%yHENThe mightiest Prince of Guilt WILL bleed THIRSTILY on this nouveau fadder Impossible Burger patty sandwich!. People want what they want, when they want it, for minimum potential harm to themselves or profit to BigCorp America. Even
Let us back ... away ... slowly ... to the default values of TOP-P = 0.75-1.0 and Temperature ~= 0.8. Those do seem to yield some of the best results. So what were my favorite outputs?
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. The Impossible Burger is a plant-based burger that is a patty made from plants and not animals. It is a healthier alternative to traditional meat patties. I would love to have this product available at your restaurant and would be happy to share my story with you about the Impossible Burger.
I would love to hear back from you regarding this.
Sincerely,
Carrie
I have a friend who's husband has had heart problems. He's on a diet of the Impossible Burger.
The story behind the burger is that it's made from plants and doesn't have any cholesterol, fat, or any of
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. I am sure that you have seen the success that this product has had in the market.
I am a former Wendy's employee and I can tell you that the Impossible Burger has all of the taste, flavor and texture of a standard beef burger. Please consider serving this product on your menu.
Sincerely,
Amy
[Reply from Wendy's:]
Thank you for your email. We are providing the Impossible Burger to our guests in our test markets for the pilot launch.
Thanks again for your suggestion and for contacting us on this issue.
Wendy's
Hello Wendy's, I am a former
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. It is made from plant-based ingredients that are as nutritious as ground beef, but with zero cholesterol and no saturated fat. It is a healthy, delicious product that you can customize with your own ingredients.
I love that the Impossible is under 300 calories and has 20 grams of protein. My kids have noticed that I eat it, and it has a good, umami-like flavor. Having more options for healthy, nutritious and delicious foods is key to helping my whole family eat well.
I have been a loyal Wendy's customer since I was a teenager, and the company has always been an important part of my life
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. I would like to introduce you to a new product that has profound positive implications on the world we live in. It's a burger that tastes like beef, but is made from plants, and has no cholesterol, no calories and no animal products of any kind! The Impossible Burger is made from plants, and as you can see, tastes just like the ground beef we all know and love.
I have worked for the Impossible Foods G.P.S. (General Product Safety) team since March, and have already been deeply impressed with the efforts of both the plant-based team and the company as a whole. Impossible Foods have
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. I believe that it would be a positive change to the menu, and would be appreciated by your customers. I am a fan of the Impossible Burger, and I know that many of your customers are, as well.
If you choose to serve the Impossible Burger at your restaurants, please make sure that you are serving the “Best-Tasting Burger in America”, and that the Impossible Burger has been a part of that experience.
I hope to hear back from you soon.
Sincerely,
Jeffrey
Wendy's
Thank you for your interest in the Wendy’s Impossible Burger.
Dear Wendy's,
I am writing to ask that you consider serving the Impossible Burger at your restaurant. I am not asking you to serve the Impossible Burger as a replacement for meat, but rather as an option for those of us who love meat but would prefer not to eat it. I was so excited to hear that you are testing the Impossible Burger at your restaurants, and I hope you will consider adding it to your menu permanently. I know that the Impossible Burger is not the answer for everyone, but I believe it would be a perfect fit for your brand.
Please take a look at the attached information, and if you would like to learn more about the Impossible Burger, please feel free to contact me. I look forward to hearing
Again, some of these are pretty good! However, they also showcase the few problems that continue to crop up ...
*GPT-J has clearly seen some e-mail chains in its training set, and in addition to generating the e-mail I want to send, it likes to start predicting replies. This isn't too big of an issue, since I could just cut off the end of the generated content.
*It does not know any autobiographical details for me, and is happy to invent them out of whole cloth. I am not a former Wendy's employee, a member of the GPS team at Impossible Foods, or even a loyal Wendy's customer (maybe cuz they don't serve the Impossible Burger). Oh, and I don't have kids.
*It does not know the current status of the Wendy's menu, either. Notice it sometimes gets confused and thinks that Wendy's is already serving or trialing the Impossible Burger.
*Though more plausible than GPT-2, it still fails at rudimentary factuality. The most noticeable false claims here are that the Impossible Burger contains no calories and no fat.
These e-mails are getting on-theme enough that some of them might serve as good bones for an e-mail of my own ... after I ran generation enough times to get a good one, fact-checked everything, edited out any baloney that GPT-J made up about me, and trimmed the end. But at that point, I still might as well write the e-mail myself.
EleutherAI just recently came out with a 20 billion parameter model called GPT-NeoX. I'll retry my test on that one when it becomes convenient. Notice the performance metrics at the bottom of the article. NeoX is over three times as large as GPT-J, but shows only a few percentage points of improvement (if that) on most benchmarks. I.e. there appear to be diminishing returns for a brute-force increase in the size of models from this lineage.
Until the next cycle,
Jenny
No comments:
Post a Comment