Caching strategies for websites serving AI bots
The three layers of an effective caching strategy
I am going to tell you a secret that is not really a secret at all: most slow websites are not slow because of bad code. They are slow because of absent or misconfigured caching.
Caching is layered. There is no magic button that solves everything. A well-functioning caching infrastructure consists of three layers working together.
The first layer: object caching (Redis), close to your application. Blazing fast, microseconds. The second layer: full-page caching (Nginx or Varnish), storing complete HTML responses. Milliseconds. The third layer: CDN caching at the edge of the network. Fastest of all for the end user, because the request never reaches your origin server.
That order is also the order of impact. Start at layer one. Work outward.
AI crawlers behave like unknown visitors: no cookies, no session tokens, no authentication. They always see the anonymous version of your page. Make sure that anonymous version is cached and correct. This point is surprisingly often overlooked.
Object caching with Redis
Redis is my favorite piece of infrastructure. That might sound strange (who has a favorite cache layer?), but when you have spent twenty years watching database queries slow down websites, you learn to appreciate Redis.
Object caching stores results of database queries and heavy computations in memory. On the next request, the cached value is used. In Laravel you configure Redis as the cache driver in your `.env` and use the `Cache` facade. In WordPress you install Redis Object Cache.
- Set meaningful TTL values. Product data that rarely changes: an hour or more. Stock status: shorter TTL. Think logically, do not set everything to the same value.
- Cache selectively. Not everything needs caching. Focus on the queries executed most frequently that take longest.
- Implement cache invalidation. Product updated? Clear the cache for that entry. Event-driven, not "throw everything away every night."
- Monitor cache hit rates. Below 70 percent? Then something is wrong with your strategy or your TTL values.
Full-page caching for HTML responses
Full-page caching stores the complete HTML output. For anonymous visitors and AI crawlers this is the most effective technique. The PHP application does not need to do anything. The web server serves the cached HTML directly.
At Kobalt we use Nginx with FastCGI cache. Cache keys based on URL, host and scheme, but explicitly without cookies or session variables for anonymous visitors.
Crucial headers: `Cache-Control: public, max-age=3600` for pages that may be cached for an hour. `CDN-Cache-Control` for CDN-specific instructions. And pay attention: if you send `Cache-Control: no-cache` or `private`, you are instructing not just browsers but also CDNs and AI crawlers not to cache. That is rarely what you want for public pages.
CDN configuration and edge caching
A CDN places copies of your pages on servers worldwide. Visitors and crawlers get content from the nearest server. Latency drops dramatically.
But wait. Is it really that simple?
Not quite. There are pitfalls.
- Configure cache rules per content type. Static assets (CSS, JS, images): cache for months. HTML pages: shorter, depending on how often the content changes.
- Use cache purging on content updates. Page updated? Purge request to your CDN, so the new version is served immediately.
- Enable Brotli compression. Better than gzip, supported by all modern browsers and HTTP clients. Free wins.
- Set correct Vary headers. Multilingual site? Use `Vary: Accept-Language` to create separate cache entries per language. Otherwise you serve Dutch content to English visitors. That happened to me. It was not my finest moment.
Cache warming: the unsung hero
Cache warming is proactively building your cache before visitors and crawlers arrive. Without cache warming, an AI crawler can arrive at the moment your cache is empty (after a deployment or TTL expiration). Then your origin server generates a cold, slow response. At precisely the moment it matters.
At Kobalt we implement cache warming as part of the deployment pipeline. After each deployment a script runs that requests the most visited pages and builds the cache. The site only goes live when the cache is warm. That sounds like a detail, but it makes an enormous difference.
For a SaaS client with heavy AI bot traffic, we implemented a cache warming script that traverses the complete sitemap every night. Cache hit rate rose to 94%. Average TTFB dropped to 45ms. AI crawlers never encountered a cold cache again. That is the difference between "we do caching" and "we do caching well."
Caching is not complicated, but it requires attention and discipline. The effort you invest once pays back with every visitor and every bot. Want to know if your caching works well? Our free AEO scan measures it for you.
Frequently asked questions
Can caching cause problems for personalized content?
Yes, if you configure it incorrectly. Full-page caching belongs only on pages that are identical for all anonymous visitors. Shopping carts, account pages, personalized recommendations: never full-page cache those. Use cookies or authentication checks in your cache key to make the distinction.
How do I know whether my caching is working correctly for AI crawlers?
Check your server logs for requests from GPTBot, ClaudeBot and PerplexityBot. Pay attention to response times: if cached requests are significantly faster than non-cached ones, it is working. Also verify via `curl -I` whether the `X-Cache: HIT` header is present in cached responses.
What is the optimal TTL for HTML pages?
Depends on your content. A blog post that does not change for months: 24 hours or more. A homepage with current promotions: 30 to 60 minutes. The golden rule: combine a longer TTL with active cache invalidation on updates. That gives you the best balance between speed and freshness.
How does your website score on AI readiness?
Get your AEO score within 30 seconds and discover what you can improve.
