Many websites have a huge number of pages on them being updated on a regular basis. Getting all of these pages crawled at regular intervals can be quite a challenge. Before we learn to make Google crawl certain pages on priority here is what you need to know.
What is a crawl budget?
The crawl budget is the amount of time and resources that Google can allocate to crawl a website. According to Google, the crawl budget is determined by two main factors.
1. Crawl Capacity Limit:
- Crawl Health: This term is actually related to the well-being of your website. If your website responds quickly when the Google bots come to crawl it and there are little or no broken links on the site then your crawl limit goes up. On the other hand, if the crawl response time is high or there are multiple broken links present on your site then the crawl limit goes down.
- Limit Set By You: You are provided access to Google Search Console for your website and from where you can set the crawl limit for your website. That being said just because you put a higher limit does not mean that Google bots will automatically increase the rate of crawling.
- The Crawler’s Limit: Google is massive with a huge number of resources dedicated to indexing the contents of the web. Although that is the case, even its resources are limited. Google needs to make choices every now and then regarding the best possible way to use those resources and hence there is a limit.
2. Crawl Demand:
The demand for crawling a website is determined by:
- The popularity of the content available on your website. The more popular the content is the more frequently Google will crawl the page.
- The size of the website.
- The frequency with which the website updates its content or makes changes to the site in terms of new links etc. Pages with updated content tend to get crawled faster.
- The quality of the page.
- The relevance of the content when comparing the content on other websites.
- Site migration or move. When you move your site to a different URL or migrate it to a different server the frequency of crawling increases in order to index the content on the new URLs.
Google might crawl your site less even if the site’s crawl budget has not been reached when demand is low.
How to make Google crawl important pages first?
Even if you have the crawl budget and the crawl demand, there is no guarantee that Google will crawl and index all the pages on your website. But if you want to tell Google to crawl certain pages faster or more frequently there is a way. It does not involve giving inputs on the search console. In a video of the Google SEO office-hours hangout shot on February 26, 2021, John Meuller answers this question put forward by one of the webmasters.
The question: “Is there any other way that we can say, Google, if you are crawling only 10% of our pages, these are our most important pages?”. John said that there are two ways to tell Google that certain pages are more important and hence need to be given a preference:
- These important pages need to be internally linked better than other pages.
- The distance of these pages from the home page (depending on the website) is also a factor. The faster one can access these important pages from the home page the more important Google considers these pages to be. To better explain this try to imagine the number of steps you would have to take to reach the destination page from your home page. The lesser that number the better.
You can check out the video below:
How to request Google to re-index pages with content updates?
It is not uncommon for you to get your content updated to keep it fresh. But whenever that happens you would want Google to take note of this change quickly so that it gets indexed faster.
You can send a request to crawl the said URL or a couple of them by using the URL inspection tool. Remember you cannot use this tool to get a ton of URLs recrawled. It is limited to just a couple of them.
In order for you to request a crawl using the URL Inspection tool, you need to be the owner of the site or you need to have complete access to the property on Search Console.
Protip! The requests for recrawls are limited and just because you requested a recrawl multiple times for the same URL it does not mean that Google would recrawl the URL any faster.
Steps to request Google to re-index a page:
- Go to Google Search Console.
- Click on the URL Inspection tool on the left-hand side of the page.
- Enter the URL that you would like Google to recrawl.
- If there are issues, fix them and click Test Live URL.
- If the live test shows no issues then click on Request Indexing.
I have added screenshots to explain the process.
If you would like to look know more about this I suggest you take a look at this video where Daniel Waisberg from Google explains how to use the URL inspection tool.
Always remember you cannot request Google to recrawl/inspect a URL that you do not own, otherwise this pops up!
How long does Google take to crawl the page after requesting?
Google can take anywhere from a couple of days to a few weeks from the time of request to recrawl your webpage. Just because Google crawled your page does not mean that it would get indexed.
Indexing a web page or not is decided by Google. No amount of requesting can help you index your page if Google decides not to.
Even if your URL passes the inspection test giving you no indexing errors, your page might never get indexed. It is important to note that your website and the requested web pages need to adhere to Google SEO guidelines strictly apart from just passing the inspection.
