Is my page too big for crawlers?
Google documents the cut: it crawls the first 2MB of a file. The AI crawlers publish no limit at all. Either way, what matters is how much of your payload is content rather than markup.
What a crawler has to swallow.
We fetch the HTML and split it: real text, inline script, inline style.
Past 2 MB, the rest goes unread
Google documents that Googlebot crawls only the first 2 MB of a file. Anything after that point in your HTML is not crawled. We flag a page as heavy from 1 MB, half the cut.
Size is not the whole story
A light page can still give a crawler almost nothing to quote. We measure how much of your HTML is readable text: under 12% we flag it, under 5% the page is mostly markup and script.
Inline script and style count too
They sit inside the HTML, so the crawler downloads them along with your text. Past 60% of the page, we flag it: in external files, they no longer weigh down what the crawler swallows before it reaches your text.
- Move large inline script and style blocks into external .js and .css files.
- Trim repeated markup and anything that loads before your main content.
- Put your main content in the HTML itself, so crawlers get text to quote, not just code.