“Check Yourself Before You Wreck Yourself” — Ice Cube (…and Also Cloudflare)
The web giant just gave us a very good reason to check our AI crawler settings.
If you’re unfamiliar with Cloudflare, it’s a company that sits between a huge portion of the web and the people (or bots) trying to access it. (It basically “holds the whole internet together,” according to one Redditor.)
Today, they’ve introduced new defaults that will block Training and Agent bots on pages that display ads, while Search remains allowed by default.
Marcomms pros: today, you are going to bring a coffee to whoever manages your website/security stack and ask them to check your AI crawler settings. Your top three latte-sipping questions are as follows:
Is our website behind Cloudflare?
Are any important pages affected by the new defaults?
Are Search crawlers still able to access the content you want AI systems to discover? (Or: which AI crawler behaviors are currently allowed?) Any blocked?
Why are you going to do this? Because you’ve spent so much time and effort answering “How do I get AI to notice my brand?” but if the machines can’t actually reach the information (because the crawlers can’t get there), then it’s all for naught. It doesn’t matter how beautiful that thought leadership is. How strong those reviews are. How optimized your executive LinkedIn is. You can do everything right to build AI visibility and still accidentally block the machines from getting to your evidence. So buy that latte..and check yourself.
Anyone responsible for digital visibility should be aware that with this update, Cloudflare is saying that not AI traffic is created equal. (It outlines three categories for AI-related crawler behavior):
1. SEARCH: Crawlers gathering information to help answer questions and power search experiences.
2. TRAINING: Crawlers collecting content for AI model training.
3. AGENT: Bots acting on behalf of a person (e.g., retrieving information or interacting with a website in real time.
In other words, the ecosystem continues to get considerably more nuanced.
Which brings me to my favorite thing to say: AI visibility is an ongoing business discipline not a one-time website project. Access, interpretation and recommendation are becoming inseparable.
You know I’m not here to fear-monger. I don’t think the answer is to freak out and start thinking about robots.txt at 3am.
HOWEVER, it’s clear that we’re moving into more sophisticated phase of AI visibility and it’s (yet again) critical to work cross-functionally to answer, “What information can AI access about us, what does it believe about us, where did it learn it, and how often does it choose us?”
#AIVisibility#PRIndustry#GEO#AnswerEngineOptimization
###
Abby Lovett, Founder, CTP Visibility Advisors. Author of "Super Human: 8 'SMALL' Ways to Be Super at What You Do…and Human While You Do It." Human Being
