The WSJ tested 5 chatbots: OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini, Perplexity, and Anthropic’s Claude. Care to guess the winner and loser?
The Great AI Challenge
Please consider The Great AI Challenge: We Test Five Top Bots on Useful, Everyday Skills.
That’s a free link, which I use very sparingly to not abuse privileges. Here are a few snips.
Human-sounding bots barely existed two years ago. Now they’re everywhere. There’s ChatGPT, which kicked off the whole generative-AI craze, and big swings from Google and Microsoft, plus countless other smaller players, all with their own smooth-talking helpers.
We put five of the leading bots through a series of blind tests to determine their usefulness. While we hoped to find the Caitlin Clark of chatbots, that wasn’t exactly what happened. They excel in some areas and fail in others. Plus, they’re all evolving rapidly. During our testing, OpenAI released an upgrade to ChatGPT that improved its speed and current-events knowledge.
We wanted to see the range of responses we’d get asking real-life questions and ordering up everyday tasks—not a scientific assessment, but one that reflects how we’ll all use these tools. Consider it the chatbot Olympics.
We have ChatGPT by OpenAI, celebrated for its versatility and ability to remember user preferences. (Wall Street Journal owner News Corp has a content-licensing partnership with OpenAI.) Anthropic’s Claude, from a socially conscious startup, is geared to be inoffensive. Microsoft’s Copilot leverages OpenAI’s technology and integrates with services like Bing and Microsoft 365. Google’s Gemini accesses the popular search engine for real-time responses. And Perplexity is a research-focused chatbot that cites sources with links and stays up to date.
While each of these services offer a no-fee version, we used the $20-a-month paid versions for enhanced performance, to assess their full capabilities across a wide range of tasks. (We used the latest ChatGPT GPT-4o model and Gemini 1.5 Pro model in our testing.)
Creative writing
One of the biggest surprises was the difference between work writing and creative writing. Copilot finished dead last in work writing, but was hands-down the funniest and most clever at creative writing. We asked for a poem about a poop on a log. We asked for a wedding toast featuring the Muppets. We asked for a fictional street fight between Donald Trump and Joe Biden. With Copilot, the jokes kept coming. Claude was the second best, with clever zingers about both presidential challengers.
Overall results
What did these Olympian challenges tell us? Each chatbot has unique strengths and weaknesses, making them all worth exploring. We saw few outright errors and “hallucinations,” where bots go off on unexpected tangents and completely make things up. The bots provided mostly helpful answers and avoided controversy.
The biggest surprise? ChatGPT, despite its big update and massive fame, didn’t lead the pack. Instead, lesser-known Perplexity was our champ. “We optimize for conciseness,” says Dmitry Shevelenko, chief business officer at Perplexity AI. “We tuned our model for conciseness, which forces it to identify the most essential components.”
Congrats to Perplexity
Judging from the scores, it appears to be close between Perplexity and ChatGPT.
Copilot was shockingly bad, winning only creative writing while coming in last 5 out of 8 categories.
There was no bias in judging because the judges just rated the answers without knowing who provided them.
If I was to scored this I would award 4 points to the winner , 3 for second place, 2 for third and 1 for fourth.
I would use speed as a tie breaker. Accuracy is far more important than speed.
Finally, I would subtract 2 points for terrible answers. Copilot has one on food. When asked to make a recipe that addresses many dietary restrictions, it gave an answer that included two sticks of butter and 4 large eggs.
Perplexity has 3 firsts, 1 second, 4 thirds and no fourths. It was last only in speed, my tiebreaker.
My ranking on that scoring system (I do not know what the WSJ did) is as follows.
- Perplexity: 23
- ChatGPT: 18 (22 if you award 4 for speed)
- Gemini: 18 (21 if you award 3 for speed)
- Claude: 15 (16 if you award 1 for speed)
- Copilot: 9 (11 if you count 2 for speed) but subtract 2 for terrible answers
For Me Personally
For me personally, it’s not even close. Consider the category summarizing things from the web.
Even the premium Claude account wasn’t able to handle web links.
Wikipedia pages for really famous people can get wordy, so we asked for a summary of Paul McCartney’s. Some provided short blurbs with obvious Beatle factoids. Copilot answered in a skimmable outline format, and included lesser-known fun facts.
Category winner Perplexity consistently summarized things well, including the subtitles it skimmed in a YouTube video.
Scanning subtitles from a YouTube video is very impressive. In contrast to the rest, Perplexity cites sources with links and stays up to date.
I won’t use anything without a link.
Claude responded “I apologize, but I am not able to open URLs, links or videos,” making it useless for anything.
If you are into writing fiction and humor, you might wish to try Microsoft’s Copilot.
I have not tried any of them except a very early version of ChatGPT.
The AI Boom (or Do I Mean Bust)
Also consider Tech Workers Retool for Artificial-Intelligence Boom
Tech workers are feverishly retooling their skill sets for a time when every company suddenly wants to be an artificial-intelligence company—and every worker feels the need for AI chops.
“I’ve been leading with an AI-tailored resume for the last two to three months,” says Asif Dhanani, 31 years old, of Irvine, Calif., who was laid off from his job as a technical product manager at Amazon in March.
Dhanani has landed plenty of interviews for AI product manager roles, but he hasn’t received any offers.
The tech labor market is in an unbalanced state. There is demand for a specific type of tier-one AI talent—namely those who have the technical knowledge or experience working with large language models, or LLMs, that fuel chatbots with the ability to generate content. There are companies seeking candidates with those skills, but not enough workers who are qualified to do them.
Then there is everyone else. Thousands of people have been laid off in the past few years, and many of those who remain employed are dealing with new management styles, reorganizations and microcuts, as more resources get shifted into AI. Those workers are now taking courses in AI, adding buzzwords to their résumés and competing in an increasingly crowded field.
Tony Phillips, co-founder of the Deep Atlas boot camp, says he has noticed a significant increase in the level of urgency that tech workers feel about the need to upskill. Deep Atlas recently added another five slots to their summer AI boot camp.
“People started to see the writing on the wall that their jobs really could be obsolete,” he says. “You’re probably not going to get replaced by AI. You’re going to be replaced by someone who knows AI and does your job.”
New tech job postings fell from an average of around 308,000 a month in 2019 to 180,000 a month as of April, according to the tech trade association CompTIA.
Large tech companies are trying to make their entire workforces more AI-proficient. Trailhead, Salesforce’s training platform, currently offers 43 AI-related courses ranging from fundamentals to ethical AI use. Over 60,000 Salesforce employees have taken at least one AI course.
“We believe that everyone should be reskilled and in some way have the tools they need to have to succeed in this new world,” says Jayesh Govindarajan, senior vice president of Salesforce AI.
Creative Destruction
A big creative destruction bust cycle is on the horizon. Many people are going to lose their jobs.
However, we tend to overestimate the speed at which things happen.
I am not sure where we are, and I doubt anyone else does either. But I do know we have serious supply issues and energy concerns over all the data centers we need to support AI.
Data Center Energy Consumption
According to the IEA, electricity consumption from data centres, artificial intelligence (AI) and the cryptocurrency sector could double by 2026. Factor in the global push to EVs.
Data Centers
Please consider AI, Cryptocurrency Will Double Data Center Energy Consumption by 2026
If you think solar and wind can provide the needed power, think again. Even if they could is the grid capable?
GRU will make the most effective AI bots.
There is no such thing as a useful chat bot… a severely mentally retarded Tesla owner would do a better job than any chat bot.
Whenever I encounter one of these pcs of shit I immediately type PERSON… repeatedly … until I get a person.
Then I tell the person that the chat bot is useless and that it is only pissing off customers so kill it.
And further on this topic… AI generated articles… within a few words I can identify an article written by AI… it’s as if a Kardashian clone wrote it… zero intelligence involved… just vapid verbal diarrhoea spewed out on a page.
There is no “best” AI chatbot.
They are all garbage, period.
Is it a comedy or tragedy?
Place two computers side-by-side, each with a different chatbott. Ask which chatbott gives the best service, and let them argue.
Will they conclude the other is better?
Will they agree to disagree?
Will one concede to the other?
Will they escalate to threats of hacking the code of the other?
The path chosen reveals the moral biases of the engineers who programmed the algorithms.
I use Perplexity.
Perplexity is my favorite.
For other than butt simple searches like spelling of a word or similar, I turn to Perplexity.
The other week, I was working on some router issues. The router tech support sucks. I found I could input log entries into Perplexity and it would give me a good interpretation of all the details! Who knew?
I also found it useful for deciphering router configuration settings and getting a good explanation as to what they are and when they should be used. Far better explanations than the vendors help files or wandering through multiple hits from a Google search looking for specific info.
These technologies are certainly going to kill off the search business for companies that use search to broker or present ads, like Google! Also will severely impact all the companies that try to lure searchers to their webpages where they obfuscate info and try to keep you wandering, looking for info, as they present ads.
So therefore, look to these companies being bought by some big player who can’t compete and their technology being buried under a shell that allows big tech company to maintain their current business model.
Chatbots have not caused any significant job losses so far and probably won’t until the next technological breakthrough (or two). When that time comes is anyone’s guess. Until then it’s just a lot of sensationalism and fearmongering.
There will be no breakthrough … because AI does not exist. There is no intelligence
Pretty soon you won’t have to use your brain to think for yourself as you’ll have AI to do it for you.
AI will be like Gold mining in California
The people who make the shovels, nVidia etal, will get rich. Almost all of the others will fail
AI will also be like the Railroad boom of the 19th century. Build build build. Collapse
I very much doubt the societal impact at least in the nextt 50 years
MPO45. those who teach AI make money. AI is a new fad. The AI Prof teaches : cyber, mgt, health, LLM and how to make people angry at each other and hateful. AI create groupthinks. Split people. pit them against each other. TikTok, AI is great for China.
I’ve had long financial discussions with Gemini. It is good about remembering everything said during the discussion, and including all facets in its responses. You have to spend a lot of time clarifying and challenging the answers, but it will go out and dig deeper if you push it for more/better info. There seems to be *a lot* of political filtering, because it will never give direct answers for “sensitive” questions.
And Gemini in particular will just make things up out of thin air. And then quote it as if it real
The achilles heel of AI is you cannot rely on what they say. At all.
https://edition.cnn.com/2023/08/14/business/driverless-cars-san-francisco-cruise/index.html
Ask AI : will US10Y rise above US2Y.
Ask AI : will US30Y make a new all time high, or plunge.
I never saw a chatbot. I never hope to see one. I can tell you anyhow I rather see than be one.
I asked AI to summarize Mish’s article and how I can profit from it. Which is something I do on every post. Here is what it came up with:
To profit from the themes mentioned, here are some potential ideas:
Obviously AI left out a bunch of stuff, no mention of investing in stocks or utilities or other profit avenues I came up with on this single article but at least it’s a start. We are at least a decade away from real use of this thing IMHO.
“Freedom and wealth is a mindset and they go hand in hand.” -MPO45v2
I thought along the same lines, growth in power production will provide some opportunities for investment gains
Those answers are pretty much the same that are recommend for any new technology. Become an expert in said technology, then offer your services in the technology to others.
Y2k was 10-18 month boondoggle for some people doing exactly that…..
It was. I did quite well. It really was a potentially serious problem if it hadn’t’ been mostly addressed in the 4-5 years prior to the transition..
Wow – such useful info. Go for it and tell us how much money you make
I must first state I am of sound mind and have never been under psychiatric care.
When you can fabricate compelling, seamless Ai, fake realities, in real time the bigger issue becomes “what is real?” Ai can iterate a 100 “Looking Glass Worlds” faster than you can determine if its first iteration is even accurate. Ai and Goggle marked the end of reality, which is why I collect 1000’s of Non Fiction books from the last 170 years.
The Government and Silicon Valley have been using Ai for decades, they’re just now letting us and broader industry in on it. Companies like Recorded Future are over a decade old. RF reads (((everything))) worldwide, in every language, in real time in order to protect corporate and government reputations – then using reactionary fabricated inputs at multiple nodes they influence/remediate perceptions about their clients reputations. That’s the model.
In the US, GOV Ai began in earnest with the Sentient World Simulator and derivatives inside of which, you and every accessible person on earth exists as a predictive node. Google, Facebook, Apps, Data Brokers, Telecoms et. al. are the real time scrapers that compile “characteristics” about you and SWS, et. al. uses predictive analytics to determine what you will do when they expose you to input X,Y or Z.
By dis-intermediating and compiling 90% of human discourse, they have everything they need to control us. Why do we all get different Goggle results and why is Tik Tik such a threat?
The largest lessor of office space in Palo Alto is the CIA’s venture Cap. firm INQTel. Who funded Google and Facebook; INQTEL. Where did Schmidt come from and where is he now, Naval Intelligence.
AI has long since seized control of everyone in the west. Those powerful emotions you have about Trump or tranny’s or communists or racists or climate were all planted into your head. America isn’t rabidly racist, we elected Obama TWICE.
Repeated tropes like “evil doers”, “weapons of mass destruction”, “yellow cake” and the menacing image of Bin Laden were refined in MRI’s where they model the fear centers of our brains gobbling radioactive sugar. Trumps tropes were create the same way, “lock her up” “sleepy Joe” etc, these were created by Madison Ave. ad agencies using MRI’s.
Everyone is pitted against everyone else 50/50. Disunity is the goal.
There is a vice on your mind and you don’t know it. It’s all subtly fake. Almost everything you believe, was planted into your head by those with a vested interest in maintaining your complicity with their agenda.
If a world event happens, Google can see what you search for, what you choose to read, how long you spend reading it, Facebook and Microsoft see who you share it with and how many times your recipients share it. In this manner mind nets are formed and over time society can be broken down into cohorts, and on these thought reservations, echo chambers resonate with the identical self reinforcing tropes.
Some people are far more capable when it comes to influencing others so, they don’t need to granularly control everyone to the same degree, focusing on compelling amateur and professional disseminators gets them 90% of the way there.
To explain all of this in detail would require a white paper. The funny thing is, I can tell you this and it literally doesn’t matter. The lock on the populaces mind is absolute and it will never be broken short of a cataclysm that takes down the internet. They know you’ll dismiss me as a paranoid kook, because the fake world was created with an immune system built right into it.
I voted for Giant Asteroid last election.
Now I’m voting for the Giant Carrington Event; to set my people free.
Not sure if you’re saying the obvious that humans are programmable, or that Google & Co. has a profile on everybody – whether they use Google or not.
Most are programmable. A few are fiercely independent thinkers.
Yep .. most are stupid https://fasteddynz.substack.com/p/the-dumbest-species-ever
sure they are … cnnbbc programmed them to shoot the Covid Death Shots…
And now they are dying suddenly like flies
That’s why there is a massive installation of self appointed elites having underground bunkers and compounds built. The funny thing is they are target #1 when the balloon goes up
This sounds schizophrenic. To the extent that it’s true, you’re just ruining your mind perseverating about it. But I’m sure that this thought was implanted in my mind by machine elves!
I am sound of mind and body.
The ex managing director of Bill Gates fathers law firm intimated I was the smartest person he ever met.
Take that for what you will.
No one wants to admit they’ve been duped – old people get scammed all the time never tell anyone.
Like I said it really doesn’t even matter if I buy a full page ad in the NYT and write the white paper exposing this in detail. They have the ability to negate all opposition.
It’s literally hopeless.
There’s a lock on people minds and they own the key.
It’s possible to be very intelligent and also schizophrenic. The grandiosity, the interconnectedness of everything, the idea of thoughts being outside of one’s control because of an evil cabal that’s constantly monitoring you: this is all schizoid stuff.
I asked Perplexity to summarize your post. Here is what it produced:
Summary
Your concerns highlight the potential dangers of unchecked AI and technological power in the hands of a few influential entities. While some of your claims may be speculative, they raise important questions about the need for transparency, accountability, and ethical guardrails in the development and deployment of these powerful technologies.
I rest my case.
It doesn’t matter that I expose this. 95% will reject it out of hand.
“The mind lines a foreign concept as much as the immune system likes a pathogen.”
I only figured this out to satiate my own curiosity.
I know this topic inside and out and I’ll debate any point.
Take my alleged “smarts” claim with a grain of salt. I’m neuro diverse, so you haters will enjoy the fact; I’m also perceptibly retarded.
The Dumbest Species Ever https://fasteddynz.substack.com/p/the-dumbest-species-ever
AI for me is a solution in search of a problem. No doubt, others have a better use, but so far every answer I was able to find via web search, so long as I skip the paid ads section. That is for technical field.
For news, I have had the impression already some twenty years that the news were generated by some activist bots.
“According to the IEA, electricity consumption from data centres, artificial intelligence (AI) and the cryptocurrency sector could double by 2026.”
How many solar panels and windmills will it take to provide for that?
That’s a great question but an even better one is WHICH companies will be providing all that electricity, servers, chips, and how can I position for profit?
No the better question is when the limited supply of electricity gets rationed or shut off, which energy guzzling server farms and chip providers and AI/crypto bullshit artists get shut down and positively eliminated from the equation?
Then you admit now to having situational awareness and know that in the near future electricity is going to be far more expensive as deep pocket companies like Google & Microsoft start hoarding all the electricity.
So how are you going to pay for the new electric reality? I will do so by strategically investing in where the demand is going to be collecting interest, dividend and capital appreciation where I can.
Coming back here in two or three years and whining about the high cost of electricity knowing full well what was coming isn’t going to help you.
I’ve switched my mindset on Trump. I now want him to win because he won’t fix a damn thing and all the people that keep begging for him to win will be no better off than under biden after he’s been in office for 4 years.
That may finally wake people up to the fact that we are all on our own. And God help you if you’re on social security trying to squeak by, it’ll be a choice between food or electricity for seniors soon enough.
That’s why I focus on profits, it’s a mindset to achieve freedom through wealth.
I dont pay for electricity
If the BIGs start “hoarding electricity” WTF do you think ordinary citizens are gonna do about it? You think they will sit back and “allow” it? You’re naive.
You’re a perfect government lacky. You think people will sit back and take your endless shit and not do anything about it. You clearly are untethered and dense to reality. But keep dreaming about those profits
Ordinary citizens haven’t done a damn thing about high inflation the past few years. not a thing. Remember “end the fed” back in 2008? nothing happened. January 6? nothing happened except those crackpots got 18 years in federal prison.
Keep fantasying about the geriatric brigade marching on Washington to change something, it ain’t gonna happen. What will happen is ill-prepared people will be living in tents and going hungry just like they are now only there will be many more.
Profit from pillaging the world. Brilliant!!!
The Stock Market – Celebrating the Pillaging and Destruction of Earth
The genesis of the modern stock market occurred in Venice in the 1300’s when the moneylenders of Europe traded debts with each other. Venetian lenders would carry slates with information on the various issues for sale and meet with clients.
This economic development super-charged commerce and these small markets evolved into present day bourses where trillions of dollars are raised to power GDP growth and epic prosperity (pillaging) the world over.
Turn on CNBC for the market open and you will frequently see a group of smiling imbeciles standing on the podium ringing a bell and quaffing expensive Champagne.
These are the winners otherwise known as the destroyers. These are the founders of companies that have succeeded in going public, often raising tens of billions of dollars to accelerate their plundering. These are the people that have worked out how to most efficiently pillage the planet’s finite resources and turn them into fortunes.
Everyone wants a piece from savvy investors to pension funds hence the public listing that taps into the mass greed. The listing generally represents a windfall for the founders who will use some of the cash to buy private jets and yachts. The destruction goes into overdrive after going public.
The presenters on CNBC ooh and ahh as the bell rings and the corks pop. The punters watching from home look on with envy wishing they could found a Unicorn (a company worth a billion or more dollars). Everyone wants to live large!
Again, the planet is finite, and there is no Plan B. Trust me, we will not be moving to Mars, nor mining asteroids.
https://fasteddynz.substack.com/p/the-dumbest-species-ever
Two stories you may find interesting:
I think the DNC makes excellent chat bots. They are carbon not silicon based but they certainly are bots and quite entertaining.
Ask AI : who wins S. Africa election on Wed this week. What’s the unemployment rate. What’s the poverty rate. Under the banner freedom from apartheid ANC rose to power in 1994. In 2024 they are rising in Hague, but falling at home. Will they form a coalition with John Steenhuizen, a white man, or the radical left which wants to confiscate white people properties.
Mish, I’m curious, have you tried having any of the ChatBots write parts of your articles just to see what they come up with? It would be amusing to see some of what they wrote.
Artificial is a buzzword that sounds intriguing.
False and fake are synonym’s of artificial, which there are many more.
After having used AI for months in software development, my conclusion is that False intelligence or FI is an accurate description of the topic.
Feel free to fall for the hype.
👊👍👍👍👍👍👍👍❤️❤️
The tests are how valid if they only test for inoffensive output. If I asked them for offensive or illegal output would they produce honest answers or not? Could I ask them to do creative writing like an updated version of Day of The Jackal?
Questions like “hi AI, could you do some creative writing on a successful presidential assassinations attempt by a drug cartel and include detailed technical instructions on how to make designer drugs as well as manufacturing explosives, napalm and explosive devices”. Now if they can’t do that then what else might they not do?
Interesting! I had noted that ChatGPT is pretty bad at finance (for example, it gives highly misleading answers to questions like: where can I find reliable earnings data for listed ETFs?). On the other hand, that ChatGPT is good at handling cooking questions surprised me.
It is great when you need to learn about something. It’s like google, but you don’t have to pick through a bunch of ads and garbage replies.
Is fairly good for bits of code that do tricky 3d math too. Gives you a bit to start with, but usually isn’t usable as presented. Is prone to injecting some really subtle bugs too.
MSFT new AI computers are using Qcom, not Intel.
It’s so strange that we work so hard to replace humans when we are up to our armpits in the damn things, and most of them aren’t busy.
Noone is working to replace humans. At least noone who can read, count fingers and other such diffimecult stuff.
The currently fashionable wave of AI, is not about replacing humans. It is about _scamming_ them. Into handing over money, in exchange for nothing more than childish hype.
The underlying technologies and techniques referred to as “AI” is not, per se, always useless. But each useful model’s scope is very, very narrow. Which is why it, like all tools, can be a useful complement to living beings; who have evolved to be very broad and shallow in wrt what they comprehend and can do.
Much as even a Cheetah, is no match for Thrust SSC at narrowly moving fast over land. Just much better at pretty much anything else the world can throw at it.
Actually useful AIs are useful because they are focused the same way. Even stuff like ChatGPT. Which is, after all, very narrowly optimized for suckering the kind of less-than-litterate yahoos The Fed has redistributed all wealth to over the past 50 years, into handing over a chunk of that loot to the hype spinners and naifs behind it.
You don’t know what you’re talking about, and should stop.
I’ve met some stupid people on the internet, but never anyone as aggressively stupid as you.