HTTP/2 and HTTP/3: faster protocols for AI crawlers

Why the protocol matters

Most discussions about AI readiness focus on content. llms.txt, Schema.org, readability, E-E-A-T. All justified.

But there is a layer beneath that almost nobody talks about: the HTTP protocol. The way a crawler actually reaches your content. And that layer makes a bigger difference than you think.

HTTP/1.1 is the protocol that made the web great. It is also over 25 years old. Every request requires a separate connection or waits its turn in a queue. That was fine when web pages consisted of ten files. But modern websites load hundreds of resources. And AI crawlers wanting to fetch multiple pages simultaneously hit those limits hard.

I had not thought about this much myself, honestly. Until I saw at a client that the crawler was consistently only fetching half the pages. Not because of rate limiting. Not because of robots.txt. Just because of a slow protocol layer.

HTTP/2: multiplexing as a game changer

HTTP/2 was standardized in 2015 and solves the biggest problem of HTTP/1.1: head-of-line blocking. In HTTP/1.1, each request had to wait for the previous one. In HTTP/2, multiple requests can run simultaneously over one TCP connection. Multiplexing.

For AI crawlers this means: fewer connections needed, fewer TCP handshakes, less overhead. A crawler that wants to fetch multiple pages from your domain does so significantly more efficiently with HTTP/2. More pages in less time. Higher chance your content gets fully indexed.

• Multiplexing: multiple requests in parallel over one connection.
• Header compression (HPACK): saves bandwidth on repeated requests.
• Server push: the server proactively sends resources to the client. Useful for critical CSS and JS.
• Prioritization: the client indicates which resources have priority.

HTTP/3 and QUIC: the protocol that survives packet loss

HTTP/3 goes a step further. Where HTTP/2 still runs on TCP, HTTP/3 uses QUIC: a transport protocol built on UDP.

Why does that matter? With HTTP/2 over TCP: if one packet is lost, the entire connection stops until that packet is retransmitted. TCP head-of-line blocking. QUIC solves this by keeping each stream independent. A lost packet in stream A does not block stream B.

For AI crawlers operating from data centers (sometimes with high retransmission rates), QUIC has a measurable effect. Moreover: faster connection setup. 0-RTT for known servers, compared to a minimum of 1-RTT for TCP + TLS.

Sounds like details? It is. But these are the details that make the difference when a crawler decides whether to fetch 50 or 200 pages from your site.

How do you enable it?

Enabling HTTP/2 is straightforward. HTTP/3 requires a bit more work, but is well documented.

Nginx

Add http2 to the listen directive in your server block. For HTTP/3 you need an additional module (quic) and a UDP listener on port 443. Do not forget the Alt-Svc header so clients know HTTP/3 is available.

Apache

Apache supports HTTP/2 via mod_http2. Activate with LoadModule and add Protocols h2 http/1.1 to your VirtualHost. HTTP/3 is still experimental in Apache. For HTTP/3, prefer Nginx or a CDN.

Cloudflare (the easy way)

Do not want to deal with server management? Cloudflare supports HTTP/2 and HTTP/3 by default and terminates the connection at its edge network. Your origin server can keep speaking HTTP/1.1. Cloudflare handles the rest.

This is the solution I most often recommend to clients without an ops team. Sometimes you just need to lay down a bunt instead of swinging for a home run. Cloudflare is that bunt: not spectacular, but effective.

Frequently asked questions

The protocol is the foundation of your house. You can have the most beautiful rooms, but if the foundation is not solid, it does not matter how nicely you have painted.