The Battle of the Bots

Sponsored by

We've known for some time that ChatGPT poses an existential threat to digital publishers.

Some people think we appear to be closer to the end of that Cambrian extinction than the beginning based on data from Cloudflare.

Last night, I attended an event organized by Cloudflare CEO Matthew Prince to draw attention to the sharp drop in search traffic.

That decline in referrals has been caused by users increasingly relying on AI summaries which reduce the clicks to original sources.

Prince said the best way to understand the change in landscape was the ratio of times Google scraped a site to the traffic it delivered a publisher.

Google's crawl-to-visitor ratio dropped from 2:1 a decade ago to 18:1 now, while OpenAI’s is 1,500:1 and Anthropic’s 60,000:1.

The reason the issue is so serious is that a large percentage of the high quality content onlie is funded by ads sold based on search.

If revenue generated via search goes away, the flow of articles is unlikely to survive. That would lead to a death spiral and end-of-days scenario for online content as we know it.

We would find ourselves in a world where high quality content exists - but it is all paywalled. Meanwhile, the free stuff would be of dubious quality.

Prince warned that this trend, coupled with AI bots scraping content without compensation, threaten publishers’ ad revenue and subscription models.

Prince advocated for publishers to block AI crawlers unless tech companies pay for licensing rights.

Cloudflare has developed tools to stop unauthorized scraping and last night coordinated a shutdown of scraping by default for some publishers. Publishers backing the move include the Atlantic, Conde Nast, DMGT, Dotdash Meredith, Fortune, Gannett and Time.

Prince said his belief is that large language models like ChatGPT, Gemini and Claude won't oppose the shift. But no one wants to go first. That’s why there needs to be a coordinated action.

At midnight, Prince led a group of publishers in pressing of a red button to symbolize the change. (He said they ordered the button on Amazon.)

Time will tell whether the battle against the bots makes a difference.

Looking for unbiased, fact-based news? Join 1440 today.

Join over 4 million Americans who start their day with 1440 – your daily digest for unbiased, fact-centric news. From politics to sports, we cover it all by analyzing over 100 sources. Our concise, 5-minute read lands in your inbox each morning at no cost. Experience news without the noise; let 1440 help you make up your own mind. Sign up now and invite your friends and family to be part of the informed.

Subscribe to 1440 today.

BRIEF OBSERVATIONS

ALWAYS BE PITCHING: Love this event where you go to a bar and use PowerPoint to pitch a friend who needs a date. McKinsey alum would seem to have an unfair advantage.

BAGUETTE, JAM AND BUTTER: I stumbled across a coffee shop near Columbia University which sells these baguette, butter and jam sandwiches. It makes you feel like you are in Paris. I could eat this every day. Why isn’t this everywhere. Just $5!

NANCY MEYERS: Hard to believe how consistent and consistently great Nancy Meyers has been in the rom-com genre.

THE BOSS: Social media lit up when the trailer for the new Bruce Springsteen movie dropped. Love this photo of the Boss reenacting a famous pose in New Jersey.

AI ENGINEERS: Sam Altman recently claimed some AI engineers were offered as much as $100 million in compensation to switch companies. Lots of people were skeptical. But there’s no question the job is in demand.

Please reach out if you have any thoughts about today’s newsletter. I enjoy hearing from readers. Send me a message if you want to talk or meet up if you are in NYC.

I would love it if you could share this newsletter with a friend.

Also, if you have a moment, take the poll below.