Professors at the Wharton School of the University of Pennsylvania put that question to the test. ChatGPT easily won.
Ideas Are Dimes A Dozen
Wharton professors say Ideas Are Dimes a Dozen and they put that theory to a test. But how does one determine a good idea? And what does better mean?
Wharton notes the difference between consistency and better. For example, an airplane pilot who lands aircraft with average smoothness 10 times out of 10 is better than one who is brilliant 9 times out of 10 and crashes once.
With ideas, one fantastic idea and 10 poor ones is better than 10 average ones. With that backdrop let’s dive into the article.
Abstract: ChatGPT-4 can generate ideas much faster and cheaper than students, and the ideas are on average of higher quality (as measured by purchase-intent surveys) and exhibit higher variance in quality. More important, the vast majority of the best ideas in the pooled sample are generated by ChatGPT and not by the students. Providing ChatGPT with a few examples of highly rated ideas further increases its performance. We discuss the implications of these findings for the management of innovation.
Introduction: Generative artificial intelligence has made remarkable advances in creating life-like images and coherent, fluent text. OpenAI’s ChatGPT chatbot, based on the GPT series of large language models (LLM) can equal or surpass human performance in academic examinations and tests for professional certifications (OpenAI, 2023).
Despite their remarkable performance, LLMs sometimes produce text that is semantically or syntactically plausible but is, in fact, factually incorrect or nonsensical (i.e., hallucinations). The models are optimized to generate the most statistically likely sequences of words with an injection of randomness. They are not designed to exercise any judgment on the veracity or feasibility of the output.
In what applications can we leverage artificial intelligence that is brilliant in many ways yet cannot be trusted to produce reliably accurate results? One possibility is to turn their weaknesses – hallucinations and inconsistent quality – into a strength (Terwiesch, 2023).
ChatGPT can generate ides far faster than humans. This gives them a huge edge in coming up with a few great ideas. For this study the professors gave ChatGPT and the students the same prompt.
I believe I would get a big zero in coming up with a truly innovative product idea.
Could you do this? How fast?
System Prompt “You are a creative entrepreneur looking to generate new product ideas. The product will target college students in the United States. It should be a physical good, not a service or software. I’d like a product that could be sold at a retail price of less than about USD 50. The ideas are just ideas. The product need not yet exist, nor may it necessarily be clearly feasible. Number all ideas and give them a name. The name and idea are separated by a colon.”
User Prompt “Please generate ten ideas as ten separate paragraphs. The idea should be expressed as a paragraph of 40-80 words.”
Do LLMs Enhance Productivity in Generating Ideas?
The answer to this question is straightforward. ChatGPT-4 is very efficient at generating ideas. This question does not require much precision to answer. Two hundred ideas can be generated by one human interacting with ChatGPT-4 in about 15 minutes. A human working alone can generate about five ideas in 15 minutes (Girotra et al., 2010). Humans working in groups do even worse. In short, the productivity race between humans and ChatGPT is not even close.
Still, the old saying that ideas are a dime a dozen is perhaps a tad optimistic. A professional working with ChatGPT-4 can generate ideas at a rate of about 800 ideas per hour. At a cost of USD 500 per hour of human effort, a figure representing an estimate of the fully loaded cost of a skilled professional, ideas are generated at a cost of about USD 0.63 each, or USD 7.50 (75 dimes) per dozen. At the time we used ChatGPT-4, the API fee for 800 ideas was about USD 20. For that same USD 500 per hour, a human working alone, without assistance from an LLM, only generates 20 ideas at a cost of roughly USD 25 each, hardly a dime a dozen. For the focused idea generation task itself, a human using ChatGPT-4 is thus about 40 times more productive than a human working alone.
What Is The Quality Distribution of the Ideas Generated Using LLMs?
A “stochastic parrot” can generate ideas, and LLMs do so shockingly productively. But we don’t care about quantity alone. More typically, the objective of idea generation is to generate at least a few truly exceptionally good ideas. In most innovation settings, we’d rather have 10 great ideas and 90 terrible ideas than 100 ideas of average quality.
We, therefore, care about the quality distribution of the ideas, and in particular, the quality of the best few ideas in a sample. Of course, we might as well also measure the mean and standard deviation of the three sets of ideas, and we do so. Two useful measures of the extreme values are: What is the average quality of the ideas in the top decile of each of the three samples? Which sources provided the ideas comprising the top 10 percent of the ideas in the pooled sample?
Chat-GPT generated the best-rated idea in our sample, with an 11% higher purchase probability than the best human idea. The average quality of the top decile in each of the three pools also follows the same pattern as average quality— seeded Chat-GPT ≻ ChatGPT ≻ Humans. Overall, we have 400 ideas, with an equal number generated by ChatGPT and humans. In the top 40 ideas (top decile) a full 35 (87.5%) are those generated by ChatGPT.
ChatGPT vs the Screen Actors Guild
The Screen Actors Guild (SAG) joined the Writers Guild (WGA) in coordinated strikes. The writers demand protection from Artificial Intelligence. Articles abound.
- Deadline: September 6: WGA & SAG-AFTRA To Rally For California Bill That Would Grant Unemployment Insurance To Strikers
- Deadline: September 6: SAG-AFTRA Says Dual Strikes Against Video Games & Film/TV Industry “Makes Sense” As Key Issues Of Wages & AI “Mirror” Each Other
- MovieWeb: September 7: Bill Maher Says Writers Are Not ‘Owed a Living’ as the WGA & SAG-AFTRA Strikes Continue
Hoot of the Day from the World Socialist Organization
The World Socialist Website reports US film and television writers’ and actors’ anger reaching the boiling point
The struggle by 11,000 film and television writers, members of the Writers Guild of America (WGA), is now in its fifth month, while 65,000 actors in the Screen Actors Guild-American Federation of Television and Radio Artists (SAG-AFTRA) are nearing the end of their second month on the picket lines.
The militant determination of the writers and actors to fight for decent living standards and a more meaningful future for art and culture have been met with intransigence and outright cruelty by the entertainment mega-corporations united in the Alliance of Motion Picture and Television Producers (AMPTP). The companies have made clear their willingness to drive thousands of artists into misery and out of the arts and entertainment industry.
Similarly, with all its phony rhetoric about “solidarity,” SAG-AFTRA has prevented video game workers from joining the strike of their fellow union members and only this week began a strike authorization vote. Moreover, the use of “interim agreements,” allowing hundreds of productions to go ahead, has created a surreal situation of internal scabbing that weakens or negates the purpose of strike action.
Outright cruelty!? What a Hoot
I discussed this on July 24, asking If the Screen Actors and Writers Strikes Went on Forever, Who Would Care?
The strike started on July 14. Did you notice? Care?
I don’t watch TV so I am not a good judge.
In a podcast, Maher expressed some sympathy for the writers.
“I feel for my writers. I love my writers. I’m one of my writers. But there’s a big other side to it. And a lot of people are being hurt besides them — a lot of people who don’t make as much money as them in this bipartisan world we have where you’re just in one camp or the other, there’s no in between.
You’re either for the strike like they’re f—ing Che Guevara out there, you know, like, this is Cesar Chavez’s lettuce picking strike — or you’re with Trump. There’s no difference — there’s only two camps. And it’s much more complicated than that.”
But I side with Bill Maher who says writers are not “owed” a living and that the strike demands can be excessive and unrealistic.
The strike demands of the United Auto Workers are also excessive and unrealistic.
United Auto Workers (UAW) Demands
- 32-hour workweek
- 46 percent pay raise over 4 years
- Right to strike over plant closures
- Increased retiree benefits
- Defined pension plan for all workers
- Cost of living adjustments
Bernie Sanders Comments and an Accurate Rebuttal
UAW Gearing Up for a Strike, It Could be Long and Nasty
On August 29, I commented UAW Gearing Up for a Strike, It Could be Long and Nasty
Bloomberg estimates the UAW demands would add $80 billion to costs.
If the Big Three automakers gave into UAW demand, they would all go bankrupt in short order.
The fact is, EVs are easier to produce. That means fewer workers. But the workers want protection from losing their jobs. The SAG wants protection from ChatGPT.
It’s really the same story. Change happens. It’s disruptive.
Biden’s Green Energy Inflation Reduction Act Needs a Big Bailout Already
The irony in the UAW case is Biden is recklessly pursuing an avenue faster than infrastructure allows and that will cost UAW jobs, but increase them elsewhere, in a highly inflationary manner.
Note that Biden’s Green Energy Inflation Reduction Act Needs a Big Bailout Already
“What, me worry?”
Some on Twitter predict, even cheer for my demise to AI writers for my stance against the UAW.
Like Bill Maher. I’m not worried. Unlike Bill Maher, I am so small no one would even want to bother to try to replace me.
When I started this article, I had no idea it would morph into the Screen Actors Guild or the UAW. On a day to day basis, I have no idea what I am going to write about. Could AI have produced this article better? Even if so, would it bother?
In retrospect, I am terrible at producing ideas for products, but pretty good at commenting on the global economic news.
If I am replaced by AI, so be it. No one is owed a living. Not the Screen Actors Guild, not the UAW, and not me.
Hey MIsh. If you can be replaced by Al can you be just as easily replaced by Bob or Joe? Asking for a friend.
Why do you keep insisting that this fast search engine with a large language model is AI?
You write about AI but you have no idea WTF AI is?
It’s not inventing! It’s not self aware! It has no original ideas.
It searches, filters, stitches this filtered and dumbed down (politically correct) crap together while making it sound as it was written by a human.
This is not an AI!
—The whole premise of this exercise is faulty—
I’ve had about 37 ideas for a product/niche (none of which I can remember), but acted on none — cannot prove they were brilliant or impossible.
Every place I’ve worked, people regularly had great ideas, but only dumb pre-failed ideas by the managers were regularly put into practice, regardless the costs and failures. It was a little better before they put managers in charge everywhere.
—So, who is going to judge whether the ideas are good or bad? AI?
—Who will put them into practice for the actual test?
—Is there even a dearth of good ideas, or do our problems stem from other factors?
“I daresay you haven’t had much practice,’ said the Queen. ‘When I was your age, I always did it for half-an-hour a day. Why, sometimes I’ve believed as many as six impossible things before breakfast.
.
Freakonomics recently had a three part series on AI.
TLDR – you might use it to increase your productivity. Maybe you write an outline, you carefully explain your style to it and ask it to write paragraphs, then you do the final edit and publish.
Maybe it makes blog posting faster, maybe it helps you produce a wider variety of content like writing long form 10+ page investor letters. It probably doesn’t replace your creative analysis abilities.
Perhaps AI just likes being talked about like everyone else does.
Every think of that?
Throughout human history, we have learned from those who came before us, and we continue to expand on that knowledge. The last few hundred years have seen an explosion of knowledge, productivity, and economic growth. I expect AI will further the explosion of productivity in the years to come. Which will be very disruptive, as it always is. Some jobs will disappear; others will be created. People will fight to keep their jobs, but eventually, many will be replaced. However, the net effect should be positive for the vast majority.
Of course, the continual expansion of population, productivity, economic growth, and living standards also requires an expansion of energy use. For the last two hundred years, that energy was primarily provided by fossil fuels. Over the next hundred years, most new energy will come from renewables. Though we will still need fossil fuels for several more decades. Which is the scenario that my investment portfolio is designed around.
I generally agree with what you posted. Though it is possible that advancement and productivity growth can be achieved with less energy. A car today gets better mileage than one built 50 years ago. Same with a desktop compared to computers of the 1950s, or a flat screen tv to the old CRT tv. Doing more with less. Apply that across society and technological advancement.
Plus the increase in work from home, 15 minute neighbourhoods and better public transport means the demand for energy for commuting has nowhere to go but down.
All true; for those who already have all those things. For first world countries, with 15% of world population, we are becoming more efficient every year. So our demand for energy is increasing very slowly now.
But for the rest of the world, which represents the other 85% of world population, particularly third world countries, they are just beginning their pursuit of what we already have. Their energy use will grow substantially for decades to come.
Per capita energy consumption;
US 1965 – 76,000 kwh/a
US 2022 – 78,000 kwh/a
China 1965 – 2000 kwh/a
China 2022 – 31,000 kwh/a
India 1965 – 1239 kwh/a
India 2022 – 7143 kwh/a
Chad 1980 – 274 kwh/a
Chad 2022 – 360 kwh/a
The US can become far more efficient, but that will be offset 10 fold by the rest of the world wanting more.
One of the first things and AI might do, is test all of humanity, and select and filter people into the most optimal training programmes and professions – and may keep some professions that are economically unnecessary, to manage a part of the population on the wrong side of the power distribution or bell curve, to be a new techno-serf class, kept happy with a semi-agrarian lifestyle, and simple human pleasures; whilst a minority are “optimised” for expansion of the population beyond Earth. In effect, Human-created AI, might become the next stage and catalyst of Human evolution, with Earth preserved as a sort of living museum and theme park.
In the early 1940’s everyone in the know at Los Alamos knew the Hungarians were from Mars. Now we are finding out that the alien of today if from South Africa. Who knew?
“Dave, I don’t know how else to put this, but it just happens to be an unalterable fact that I am incapable of being wrong.” – HAL 9000
“Dave’s not here.” — Cheech And Chong, 1971
Since the question of coming up with an idea for selling something is directed to college students, and given the fickleness of youth, this project should run over many years and only then choose the one idea that keeps coming up over time for best results. After all, as has been pointed out, it’s not the idea but the execution; how does one know what is the best idea?
Whose execution?
.
Did ChaGPT’s list of 10 exceptional ideas include eco-friendly items?
Solar powered beer cooling jacket.
A bust of Karl Marx made from organic material.
A windmill to convert the hot air coming out of student’s mouths while they protest into energy for recharging smart phones.
A marijuana growlight that emulates the sun traveling an arc in 12 hours.
Sure, but will the Chat GPT come up with the idea to feed mayonnaise to fish? Until then I’m placing my money on the real idea man, Bill Blazejowski.
link to youtu.be
link to youtu.be
Could chatGPT replace the US Congress?
What about our Pres? HMMMMM….. I would prefer seeing a Robotic Pres., spewing lies and then hiccuping and then melting down, like the Wicked Witch fo the West, into a pool of wiggling sperm.
AI could definitely replace all of government and do I much better job than any human could, no doubt. The question is who would decide what its goals are?
IT would decide what the goals are!
One of the SF authors I follow is Neal Asher, who writes galaxy wide space opera with alien battles, AI’s, superhumans and incredible weapons. The universe the author writes in is called the Polity and is run by an AI. The AI’s took over subtly in a “Quiet War”. Here is the description of how that occurred from the authors fake encyclopedia:
Quiet War: This is often how the AI takeover is described, and even using ‘war’ seems overly dramatic. It was more a slow usurpation of human political and military power, while humans were busy using that power against each other. It wasn’t even very stealthy. Analogies have been drawn with someone moving a gun out of the reach of a lunatic while that person is ranting and bellowing at someone else. And so it was. AIs, long used in the many corporate, national and religious conflicts, took over all communication networks and the computer control of weapons systems. Most importantly, they already controlled the enclosed human environments scattered throughout the solar system. Also establishing themselves as corporate entities, they soon accrued vast wealth with which to employ human mercenary armies. National leaders in the solar system, ordering this launch or that attack, found their orders either just did not arrive, or caused nil response. Those same people ordering the destruction of the AIs, found themselves weaponless, in environments utterly out of their control, and up against superior forces and, on the whole, public opinion. It had not taken the general population, for whom it was a long-established tradition to look upon their human leaders with contempt, very long to realise that the AIs were better at running everything. And it is very difficult to motivate people to revolution, when they are extremely comfortable and well off.
link to nealasher.co.uk
Every wannabe showrunner must have a “bible” for his show with canned answers for plots and characters.
Mish – you don’t watch many movies and don’t watch TV. I wondered how you got to be so smart. Your secret is out. It has been estimated that 3% of Americans don’t watch TV. When I was small the TV broke and we couldn’t afford a new one. I suspect that was one of the luckiest things that has ever happened to me.
The top picture depicts some code snippet which ChatGPT presumably would have stolen without giving credit. Winning.
“System Prompt “You are a creative entrepreneur looking to generate new product ideas. The product will target college students in the United States. It should be a physical good, not a service or software.”
———
Generally, generating good ideas in a particular area requires exposure to and experience in that area to understand what services/products are missing. As it has been many, many years since college, I would fail this chore terribly.
“When I started this article, I had no idea it would morph into the Screen Actors Guild or the UAW. On a day to day basis, I have no idea what I am going to write about. Could AI have produced this article better? Even if so, would it bother?”
Excellent post but it is truly ironic you don’t see the irony. ChatGPT or perhaps BloggerGPT would simply write 100+ articles/day just like they are better at coming up with ideas they will come up with ideas/opinions/analysis on everything. Imagine every BLS.gov, BEA.gov, Census.gov statistic extrapolated, analyzed and reported 100+ different ways and not just US data but data on EVERY.SINGLE.COUNTRY.
Imagine bloggerGPT writing an article like this but not just in one language but 100 and not just one angle (your opinions) but 100 different angles for 100 different types of people (socialists, capitalists, communists, left-wing, right-wing, etc).
Imagine a bloggerGPT that accurately told you what stocks would go up and which would go down. I can tell you I would IMMEDIATELY stop reading Mishtalk and Wolfstreet if stockGPT told me how to achieve 30% returns accurately and consistently. Why would I come here for analysis if an xGPT can do it 100 times better?
Now imaging ONE company or one person controlling all of this. That’s what’s coming but the likes of Wolfstreet and Mishtalk and others over the age of 70 will probably be long gone by the time that happens. It does suck for anyone else trying to run a blog in the year 2030.
HMMMMM.
“Humans working in groups do even worse. In short, the productivity race between humans and ChatGPT is not even close.”
As veteran participants in many meetings knew it all along.
best work is done in pairs
“The models are optimized to generate the most statistically likely sequences of words with an injection of randomness.”
Statistically likely and “truly innovative” are, at any given time, definitionally non-overlapping.
Also: No idea has ANY value. Any halfwit can sit there and rap off “ideas” ad infinitum. ALL the value is in execution. No vermin is of less use than “Idea Guys.”
Even in Hollywood, which is where they would come closest to be of value, “Ideas” are irrelevant. 20-odd years ago, for a brief period, some sort of cult formed around “high concepts”. Effectively one-line script “Ideas.” The idea being, some dilettante with preferential funding access, could “come up with” a catchy elevator pitch, and then simply “hire” a writer to flesh out the story, and audiences would somehow love it.
It took about all of a summer or two, before all the dilettantes enamoured with their own “creativity”; or at least those funding them; realised than not a single audience member cared. Then the money went into reanimating old Stan Lee executions instead. Can’t say I, personally, understand how anyone could care about those, neither. But for some reason they do.
Regardless. Point being: “Ideas” count for NOTHING. They’re no different from opinions: Everybody’s got one and they all stink.
I’m sure clinging to the illusion that their petty “Idea” has value, feels all kinds of good to incompetent dilettantes without the talent to execute anything. And that sort of idiots are exactly the ones The Fed has enriched with stolen funds over the past 50 years. Such that the dilettantes can, in turn, keep falling for complete idiocies like “AI.” You can see the lowbrows all over San Francisco/”Silicon Valley” now. Noone is doing anything. Everyone’s got “an idea.”; a “pitch”; a blah-blah. All while The Fed is busy robbing America’s more competent people and organisations, in order to feed the childish fantasies of these illiterate, useless dilettantes’.
If you want to show how great your AI’s “Idea” is; put it on the “big screen” unaugmented. Or have it receive a Fields Medal. Or cure cancer. Don’t “do a study” consisting of having a bunch of irrelevant hacks pass themselves off as some form of anointed “judges” of things none of them will ever understand.
“ALL the value is in execution.” I totally agree.
I also want an execution of our Congress as they do not execute ANYTHING but misery. THAT BODY is quite creative at STEALING and making us think it is “SERVING.” SERVICE:
LIKE AN ANIMAL IS INSEMINATED. How’s that for some truly deep and creative writing? 😉
Before we can do a study we will have to form a committee.
.
Considering the length of the writers strike, it is being run by the state legislature, to allow the strikers access to unemployment benefits. Haven’t heard whether the idea has much support in Sacramento.
NEITHER – it is usually independent school dropouts that generate the most innovative ideas. MBAs are engrained misty with conventional prevalent wisdom and are encouraged to fit the mold. ChatGPT is an overhyped joke that give pundits lots of occasions to say something.
Try the same test on humanities majors. In my experience (45 years as a professor of English, now retired), MBA students are not trained to be innovative in any way. Full disclosure: partly for this reason, I have for many years advocated that US B-schools be closed down yesterday.
B-schools are not for learning innovation.
B-schools are for filling your Rolodex (or contacts list as the case may be.)
You won’t be replaced but maybe your commentators will use ChatGPT to respond to your posts. I would be on the lookout for a sudden increase of quality in the comment section. For the moment there doesn’t seem to a danger but if I start to see commentators using logical and coherent arguments then it’s game over.
LOL.
It doesn’t generate anything new… just mixes and matches existing ones, which is what most people do.
Ideas have never been in short supply. GOOD ideas, and the resources to implement them are.
Even bad or silly ideas work out just fine if the timing is right and marketing is good.
For example Pet Rocks would not be a good idea today but when they came out the timing was just right and the marketing was good so it turned out really well for a silly idea.
The timing and marketing aspect are the big random factors that neither humans nor ChatGPT are likely to solve anytime soon.
I suppose there are varying perceptions of what a good idea is. I meant one that results in profit or savings.
I have had exactly FIVE great Ideas (for products) in my entire career (Spanning 30 years)….I am retired now. Of those, I tried one and failed: I was one of the very first Motorcycle Gear ON line Businesses and Google “Adwords” cleaned my clock. DEMAND was GOING UP RAPIDLY, But I lost $10K in my first year.
I shut it down. The idea was great, but I could not have sold enough at the price point to deliver a net profit at all. TOO EXPENSIVE.
It was an innovative Motorcycle Cover.
“It doesn’t generate anything new… just mixes and matches existing ones…”
Ah, not quite. In a recent university study, ChatGPT-4 scored in the top 1% for originality versus students.
…but that’s not because Chat GPT is so good… it’s because MBA students, by and large, are so useless… as are MBAs as qualifications anyway… why not try Chat GPT out against design engineers and product designers from different fields?
Dude, give it up. AI has already won. We’re just waiting for corporations to slowly start replacing humans with AI. Then, it becomes an avalanche.
Go watch The Creator on 9/29 when it releases. It’s foretelling our very bleak future.
Most human decisions are made by groups which means they are average at best. ChatGPT is a dictator. It is a contest between dictatorship and democracy. /s