Insult to injury? AI bots come for your content and site resources

Spending hours maintaining client sites?

WP Remote will streamline your WordPress maintenance process and save you at least 4 hours per site every week.

ai bot scraping content

Generative AI is changing content as we speak. The trouble is that none of us know what is going to happen. 

And now there is yet another spanner in the proverbial works. 

Impact on SEO

As it stands, content publishers like us invest a great deal in creating useful content. Useful content brings in organic traffic and high-intent visitors to our sites. 

Traffic is critical to revenue, as every content marketer knows. However, it is in decline. 

More people are turning to AI tools for the insights we provide. Zero effort on their part, and without so much as a nod in our direction. 

Granted, there are plenty of caveats about accuracy right now. But these tools will only get better. Anyone who disagrees is sticking their head in the sand. 

Some publishers have taken steps to prevent AI bots from scraping their content. Many have tweaked their robots.txt file prohibiting scrapers.

Bombarded by bots

The only problem is that we’ve seen evidence that certain AI bots aren’t respecting rules

In fact, their repeated requests are almost as bad as a DDoS attack. Bots are depleting server resources, driving up hosting bills, and impacting site performance.

ClaudeBot has been a particularly egregious offender. It has bombarded sites with excessive crawling in the last two weeks. 

Here are some stats from our firewall: 

  • We saw 11047276 requests from ClaudeBot in a single day
  • Out of which, 1 site alone got 180499 requests

Thankfully, we have MalCare’s firewall mitigating this onslaught for our sites. Because of its smart bot protection, we know that the good bots will have access, whereas the others stay out.

Intellectual property and all that jazz

So, as it stands, an AI tool can scrape all our insights for free. AND we get to pay astronomical bills for the privilege. All while losing traffic that is critical to us. 

Hardly seems fair.

But the truth is we’re not sure how to feel about it all. 

Take our WordPress security plugin comparison, for example. We spent months testing 13 top security plugins, throwing malware and attacks at them. We sought an objective answer to the question: will this security plugin protect a WordPress site? 

On the one hand, it is annoying to have months of hard work repurposed so easily, where we don’t appear to reap any of the benefits. 

On the other hand, AI tools that scrape useful, well-researched content like ours end up being a new channel for discovery. 

Do we want to deny MalCare’s top billing in this new channel? 

We’ll have to wait and see. 

Tags:

You may also like


SSL Monitoring feature image
Automate SSL Checks with WP Remote

Have you ever had one or more of your managed websites showing an “insecure” warning to visitors? Or worse, has a client of yours come running to you with a…

Incident Affecting Site Uptime Data
Incident Affecting Site Uptime Data

Summary Here is a short summary of the incident: Timeline What happened  Between 6th to 12th of May, we experienced issues with external firewalls blocking our plugin requests.  A standalone…

How do you manage your websites?

Managing multiple WordPress websites can be time consuming and error-prone. WP Remote will save you hours every day while providing you complete peace of mind.

Managing everything yourself

But it’s too time-consuming, complicated and stops you from achieving your full potential. You don’t want to put your clients’ sites at risk with inefficient management.

Putting together multiple tools

But these tools don’t work together seamlessly and end up costing you a lot more time and money.