Web performance as the foundation for AI visibility
Why AI crawlers drop off on slow sites
I am going to say something some people do not want to hear: if your website is slow, nothing else matters. Not your content. Not your schema.org. Not your beautiful llms.txt. Nothing.
That sounds harsh. It is. But after twenty years of building and maintaining websites, I know that technical quality is the foundation for everything built on top of it. SEO, conversions, AI visibility. Always the same story. And yet at Kobalt I still regularly see clients investing thousands in content optimization while their TTFB sits above 2 seconds.
That is like building a house on quicksand and worrying about the color of the curtains.
AI crawlers like GPTBot, ClaudeBot and PerplexityBot work differently from you and me. They crawl thousands of pages per day and have an internal budget per domain. Slow? Fewer pages crawled. Fewer pages crawled? Less content indexed. Fewer citations. Done.
Analysis of our Kobalt clients shows that sites with a TTFB above 1.5 seconds are crawled on average 40% less deeply by AI bots than sites with a TTFB below 400 milliseconds. That difference directly translates to fewer AI citations.
Core Web Vitals: relevant for AI too
Google invented Core Web Vitals as quality signals for the human experience. But wait. Is that actually the whole story? No. The underlying metrics measure fundamental properties of your page that are equally relevant for machines.
LCP: how fast does your main content appear?
Largest Contentful Paint measures how quickly your main content becomes visible. An AI crawler using server-side rendering interprets a slow LCP as: this server is struggling. Or worse: the HTML arrives too late. An LCP above 4 seconds? In my experience, that almost always goes hand in hand with other technical issues that undermine AI readability.
CLS: the hidden JavaScript problem
Cumulative Layout Shift sounds like a purely visual thing. But a high CLS often points to JavaScript loading content after the initial HTML render. And that is a problem. Because if your critical content is only available after JS execution, that content is invisible to crawlers that do not execute JavaScript. And many AI crawlers do not.
- Make sure your core content lives in the server-side rendered HTML, not only after JavaScript execution.
- Test your pages with curl or wget. What you see there is what a crawler without JS receives.
- Avoid lazy-loading for content above the fold or content that is essential to your message.
- Regularly check your server logs for AI crawler visits. See what they do and how long they stay.
What I encounter with Kobalt clients
Let me be honest. Most performance problems are not mysteries. They are the result of years of neglect. Plugins nobody uses anymore. 4 MB images that were never optimized. A hosting plan from 2018 that nobody ever upgraded.
No rocket science. Just structural work.
A mid-size retailer came to us last year with a homepage load time of 8.4 seconds. Eight point four. I once spent an entire evening shaving 12 milliseconds off a TTFB. My wife was less impressed than I was. But 8.4 seconds? That is not an optimization question, that is an emergency.
After a performance audit and three weeks of work: 1.2 seconds. The AEO score rose from 34 to 61 in the same period. Not because we did anything magical, but because AI crawlers could now reach more pages per session.
- Measure first. Lighthouse, WebPageTest or GTmetrix. You need a baseline before you can improve.
- Pick the low-hanging fruit: compress images, set browser caching, remove unnecessary plugins.
- Evaluate your hosting. Shared hosting with hundreds of other sites is rarely sufficient for serious AI visibility.
- Implement a CDN. Geographic proximity to the crawler significantly reduces latency.
- Monitor continuously. Performance degrades without attention, just like a garden you stop tending.
Performance as an AEO investment
The business case for web performance was already strong because of conversions and Google ranking. AI visibility adds a new dimension. A fast site is not just nicer for visitors. It is also a better citizen in the AI indexing ecosystem. A well-built website is like an ecosystem: everything is connected.
At Kobalt we treat performance not as a separate project but as part of the AEO baseline. You can write brilliantly about your expertise. But if an AI crawler leaves your site after three pages because the server is too slow, you will never be considered for citations on the topics you cover deeper in your site.
Sometimes you just need to get the basics right before you start on the fancy stuff.
Run a Lighthouse audit monthly on your homepage and your five most important landing pages. Set hard thresholds: LCP under 2.5 seconds, TTFB under 600 milliseconds, no render-blocking resources. Treat violations as bugs, not wishes.
Frequently asked questions
What load time is acceptable for AI crawlers?
Aim for a TTFB under 400 milliseconds and a full page load time under 2 seconds. These are not arbitrary numbers. They align with what both Google and independent research identify as the threshold below which crawlers work efficiently. Above 2 seconds, crawl depth measurably decreases.
Does JavaScript rendering affect AI indexing?
Yes, and significantly so. Most AI crawlers do not execute JavaScript, or only to a limited extent. Content only available after JS execution? Invisible to those crawlers. Use server-side rendering or static HTML for everything you want indexed. Test this by viewing your page with JavaScript disabled.
Is improving Core Web Vitals enough for better AI visibility?
Performance is necessary but not sufficient. A fast site without good content structure, schema.org and E-E-A-T signals will still score low. Think of performance as the threshold you need to clear first. It is the foundation, not the complete solution. But without that foundation, the rest is pointless.
An AI crawler has no patience. It does not wait for your server to wake up. If you are not fast enough, it moves on to your competitor. It is that simple.
How does your website score on AI readiness?
Get your AEO score within 30 seconds and discover what you can improve.
