The first 100 milliseconds: TTFB and AI crawlers
What TTFB is and why it matters so much
Time to First Byte. The time between the moment a client sends an HTTP request and the moment the first byte of the response arrives. Sounds technical. It is. But the implication is simple: it measures how quickly your server responds.
And it is the metric I have seen ignored most often in twenty years while having the most impact.
I will be honest: I am a bit obsessed with it. My colleagues at Kobalt sometimes call it the "Reinier metric". I wear that title with pride.
But why is TTFB so important for AI? Simple. AI crawlers crawl thousands of pages per day with an internal time budget per domain. If every page on your domain waits 1.5 seconds for the first byte while a competitor needs 150 milliseconds, that competitor can have ten times as many pages indexed in the same time.
That is not a theoretical disadvantage. That is a concrete setback.
Google says: TTFB under 200 ms is good, 200-500 ms acceptable, above 500 ms needs improvement. But AI crawlers do not use browser caching and fetch every page fresh. The threshold is stricter. At Kobalt we use a target of under 300 ms for all pages you want indexed.
The three components of TTFB
TTFB is not a single measurement. It consists of three parts that each require their own solution.
Network latency: the travel time
Before your server can even start processing, the request needs to travel there first. Crawler in Virginia, server in Amsterdam? Minimum 80 milliseconds round-trip, purely due to the speed of light through fiber. Not much you can do about that.
What you can do: a CDN with edge locations close to the crawler. The request then goes to a nearby node that retrieves the cached response locally.
Server processing: where the real problem lives
This is the part most under your control. And most neglected. Everything your server does after the request arrives: database queries, PHP processing, template rendering, consulting caching layers.
On a poorly configured WordPress installation without caching? 800 milliseconds to 2 seconds. For a simple blog post. It makes me a bit despondent when I encounter that. And I encounter it regularly.
# Typical TTFB breakdown without caching (WordPress):
Network latency: 80ms
PHP bootstrapping: 120ms
WordPress init: 180ms
Database queries (34x): 420ms
Template rendering: 160ms
Output buffering: 40ms
-------------------------
Total TTFB: 1000ms
# With Redis object caching:
Total TTFB: 588ms
# With full-page caching (Nginx FastCGI cache):
Total TTFB: 100ms
# That difference is not subtle.TLS handshake: the security tariff
For HTTPS connections, the TLS handshake adds an extra component. One-time per TCP connection. TLS 1.3 has reduced this to one round-trip, halving the impact compared to TLS 1.2. Check if your server supports TLS 1.3. In 2026 it should be standard.
Improving TTFB: what I do at Kobalt
Right, enough theory. What can you concretely do?
- Implement full-page caching. Nginx FastCGI cache, Varnish or WP Rocket with page caching. This is by far the biggest single improvement you can make.
- Object caching via Redis or Memcached. Eliminates repeated database queries for menus, options and widgets.
- Optimize database queries. Query Monitor in WordPress shows which queries are slow.
- PHP 8.2 or higher. Up to 30% faster than PHP 7.4 for typical WordPress workloads. Free speed gains.
- Enable HTTP/2 or HTTP/3. Multiple requests over one connection, less overhead.
- Use TLS 1.3. Faster handshake, better security.
- Deploy a CDN. For cached content the TTFB drops to 50-100 ms for international crawlers.
Measure, monitor, adjust
Improving what you do not measure is gambling. And I am not a gambler (except that one time at the horse races, but that is a different story).
- WebPageTest.org: the most detailed TTFB analysis, with waterfall and component breakdown. Test from multiple locations.
- Google Search Console: shows Core Web Vitals including TTFB-related data based on real users.
- curl: fast, scriptable, ideal for automated checks.
- UptimeRobot or Better Uptime: continuous monitoring with alerts when thresholds are exceeded.
- Lighthouse: performance score and breakdown, including server response time.
Always test your TTFB from multiple locations. A TTFB of 150 ms from Amsterdam can be 800 ms from San Francisco, where many AI crawlers send their requests from. WebPageTest.org offers free tests from dozens of locations.
Frequently asked questions
What is a realistic TTFB for a WordPress site?
With good caching and a decent VPS: 100-300 milliseconds. Achievable for most WordPress sites. Without caching on shared hosting: 800-2000 ms. The difference lies almost entirely in caching configuration and hosting quality. No magic. Just solid work.
Does TTFB affect my Google ranking?
Yes. Google has confirmed that server response time is a factor in the crawl budget Googlebot assigns. High TTFB means fewer pages crawled per session. Plus: high TTFB contributes to poor LCP, which is a direct ranking factor. Double positive effect when you improve it.
My site has a CDN. Why is my TTFB still high?
A CDN only helps for cached content. Dynamic pages that are not cached still go through to your origin server. Solution: configure your CDN to cache more (with the correct cache headers) or implement full-page caching on your origin server. Also check if your CDN has edge locations close to the crawlers.
The first 100 milliseconds are the audition for your website. If you blow it there, you do not get the rest of the conversation. An AI crawler does not need a second chance to decide whether your server is worth the effort.
How does your website score on AI readiness?
Get your AEO score within 30 seconds and discover what you can improve.
