State of the Bots

In partnership with

_{Sign up to receive Surface Area or follow me on Linkedin or Twitter. Book a meeting here to talk about ghostwriting from my agency, Principals Media}.

I watched the State of the Union, but found the State of the Bots more engaging.

The State of the Bots is a report published by Tolbit CEO Toshit Panigrahi that assesses the volume and impact of web scraping and generative AI on publishers.

It illustrates the potentially devastating impact of the combination of extracting information from web sites and reformulating it into easy-to-digest answers for ChatGPT.

According to the report:

–The volume of web scraping doubled between the third quarter and fourth quarter of 2024.

–During that same time, the amount of unauthorized web scraping increased by more than 40%.

–Even when blocked, apps like Perplexity used unidentified agents to scrape sites.

All that adds up to one devastating conclusion: 

“AI chat bots on average drive referral traffic at a rate that is 96% lower than traditional Google search.”

In plain English that means online publishers can expect to receive 96% fewer clicks on their web sites in a new world where most internet queries are answered using ChatGPT.

A friend of mine compared this to the impact of Uber on taxis in New York. Before Uber, taxis were plentiful and cheap. Afterwards they largely disappeared. Uber may be convenient, but its more expensive and it wiped out the value of owning a taxi license.

In the case of publishers, it’s not clear how the new system would let them generate revenue to pay writers since the current ecosystem for advertising-supported content depends on traffic and that comes from Google.

The alternative is to go behind a paywall.

Either way, the result is less, reliable and high-quality information available to the public.

Tech companies have generally balked at compensating publishers for content – either to be used in training models or to be accessed as a reference to answer questions. 

Analyst Matthew Scott Goldstein calls it “The Greatest Heist on Earth.”  

He wrote: “Powerful tech companies, from social media giants to artificial intelligence startups, are quietly siphoning off the value from content creators across all industries.”

Goldstein says safeguarding copyright is essential and steps need to be taken now. 

“Once AI models are trained on stolen content, the damage is irreversible. The longer this goes unchecked, the harder it will be to claw back rights.”

It’s possible some of the damage will be mitigated by new search designs Google is unveiling which it says will make it more attractive for people to click through to web sites.

What the State of the Bot report makes clear is that that future is coming fast.

Start learning AI in 2025

Everyone talks about AI, but no one has the time to learn it. So, we found the easiest way to learn AI in as little time as possible: The Rundown AI.

It's a free AI newsletter that keeps you up-to-date on the latest AI news, and teaches you how to apply it in just 5 minutes a day.

Plus, complete the quiz after signing up and they’ll recommend the best AI tools, guides, and courses – tailored to your needs.

Sign up to start learning.

BRIEF OBSERVATIONS

CAR PAYMENTS: This chart seems to be moving in the wrong direction.

WHAT TECH TEACHES US: I like this take about how tech can change our perspective on underlying human activities.

INNOVATION: American exceptionalism: someone invented a device you plug it into your wall socket to simulate the flickering light of a TV and thereby deter would be robbers.

JOB PERFORMANCE: Goldman is evidently using time spent in office as a metric for the annual performance reviews to determine which of the 5% of employees will be sacked.

DINNER WITH JAMES: I juxtaposed a photo from dinner with James with one taken 40 years ago when we were college freshman. So much, and yet so little has changed.