What is Crawl Budget?
Crawl budget is the number of pages Google crawls on your site within a given time period.
Two components
- ●Crawl rate limit: The maximum number of requests Googlebot can make without overloading your server
- ●Crawl demand: Google's interest in your content (popularity, freshness)
When to worry about it?
- ●Sites with more than 10,000 pages
- ●Sites with dynamically generated content
- ●Sites with many duplicate or low-quality pages
How to optimize?
- ●Block unimportant pages in robots.txt
- ●Eliminate duplicate content
- ●Improve server speed
- ●Maintain an up-to-date sitemap
- ●Use noindex tags on irrelevant pages
SEO Impact
If Google exhausts its crawl budget on unimportant pages, your strategic pages may not get indexed or updated.
Related Terms
What is a Robots.txt File?
The robots.txt file tells search engines which pages of your site they can or cannot crawl.
What is a Meta Description?
The meta description is a short HTML summary that appears below the title in Google search results.
What is the Title Tag?
The title tag is the most important HTML element for SEO. It's the blue clickable title in Google.
What is a Canonical Tag?
The canonical tag tells Google which is the preferred version of a page when duplicate content exists.
What is Hreflang?
Hreflang is an HTML attribute that tells Google the language and region a page targets for multilingual sites.
What is X-Default Hreflang?
X-default is a special hreflang value that designates the default page for users whose language isn't specifically targeted.