Is my page too big for crawlers?
Google documents the cut: it crawls the first 2MB of a file. The AI crawlers publish no limit at all. Either way, what matters is how much of your payload is content rather than markup.
Free · no signupGoogle limit: 2MBContent ratio measured
Step 01 · Weigh your page
What a crawler has to swallow.
We fetch the HTML and split it: real text, inline script, inline style.
Why it matters
Vibe-coded sites are the usual suspects
Lovable, v0, Bolt and Create React App ship a client-only bundle by default. The HTML is an empty div and everything is drawn after load, so a crawler receives nothing.
You cannot see it from your browser
Your browser runs the JavaScript, so the page looks perfect to you. The only way to notice is to fetch the page the way a crawler does.
Nothing else works until this does
robots.txt, llms.txt, content, backlinks: all of it assumes the crawler can read your page. If it cannot, the rest is wasted effort.
How to fix it
- Move to a framework that renders on the server: Next.js, Nuxt, Astro, SvelteKit, or Remix.
- Already on Next.js? Check that your pages are not all marked "use client" with data fetched in useEffect.
- Static export or prerendering also works. What matters is that the HTML arrives with the content in it.
Readable first. Recommended next.
Free · no account · nothing stored