How Search Engines Work: Crawling, Indexing and Ranking Explained

Every time you search, a search engine picks a handful of pages out of hundreds of billions and puts them in order in a fraction of a second. It can do that because most of the work happened long before you typed anything.

That work happens in four stages: crawling, rendering, indexing and ranking. Understanding them is the foundation of SEO, because each stage is a place where your pages can succeed or quietly fail.

Stage 1: Crawling – discovering pages

Search engines use automated programs called crawlers (Google’s is Googlebot, Microsoft’s is Bingbot) to fetch web pages. Crawlers discover new URLs in three main ways:

• Following links from pages they already know about
• Reading XML sitemaps that site owners submit
• URL submissions, for example through Google Search Console or the IndexNow protocol supported by Bing and others

Crawlers do not visit every page every day. They decide how often to revisit a site based on how important it seems, how often it changes and how quickly the server responds. Very large sites have to think about crawl budget – the number of URLs a search engine is willing and able to crawl in a given period – but for most sites with a few thousand pages, crawl budget is not a concern.

What can stop crawling

• robots.txt rules that disallow a section or the whole site
• Server errors (5xx) and timeouts, which make crawlers slow down
• Orphan pages that nothing links to
• Links that are not real links, such as navigation built only with JavaScript click handlers

Stage 2: Rendering – seeing the page like a browser

Many modern pages are built with JavaScript. The HTML the server sends may be nearly empty, with content added in the browser. To see that content, Google runs pages through a headless version of Chrome – the Web Rendering Service.

Rendering takes extra resources, so it can happen later than the initial crawl. Content that only appears after JavaScript runs may therefore be indexed more slowly, and errors in scripts can hide it entirely. Most AI crawlers and many other bots do not run JavaScript at all.

The safest approach is to include your main content, links, title, meta description and canonical tag in the initial HTML – which is what server-side rendering and static generation provide.

Stage 3: Indexing – understanding and storing pages

After fetching (and if needed rendering) a page, the search engine analyses it: the text, headings, images, links, structured data and metadata. It works out what the page is about, which language it is in, which other pages it duplicates and whether it is worth keeping.

Indexing is selective. Google does not index every page it crawls. Pages may be left out because they:

• Ask not to be indexed with a noindex directive
• Duplicate another page, in which case Google picks one canonical version to index
• Offer little value, such as near-empty pages or automatically generated content
• Return errors or redirects

You can see which of your pages are indexed, and why others are not, in the Page indexing report in Google Search Console.

Canonicalisation

The same content is often available at several URLs: with and without www, with tracking parameters, with different sort orders. Search engines group these duplicates and choose one canonical URL to show in results. You can influence that choice with redirects, rel="canonical" tags, consistent internal links and your sitemap.

Stage 4: Ranking – choosing and ordering results

When someone searches, the engine looks up matching pages in its index and orders them using many signals. Google has described the broad factors it considers:

• Meaning of the query – what the searcher is really asking, including spelling corrections, synonyms and intent
• Relevance – how well a page’s content matches that meaning
• Quality – signs that content is helpful, reliable and shows expertise, experience, authority and trust
• Usability – page experience factors such as mobile-friendliness, HTTPS and Core Web Vitals
• Context – the searcher’s location, language and settings

Links from other sites remain an important signal of authority and help search engines discover and understand pages. No single factor decides rankings; relevance and helpfulness come first.

Search results are more than ten blue links

Results pages now include featured snippets, image and video results, local map packs, “People also ask” boxes, shopping results and AI-generated overviews. Each draws from the index in slightly different ways. Structured data can make pages eligible for some rich results, and clear, well-structured content is more likely to be quoted in snippets and AI answers.

How AI search fits in

AI search experiences such as Google’s AI Overviews and AI Mode, ChatGPT search and Perplexity still rely on finding and reading web pages. Google’s AI features draw on its regular search index, so the fundamentals are the same: a page has to be crawlable, indexable and allowed to show snippets to be used. Other AI search engines run their own crawlers – such as OAI-SearchBot and PerplexityBot – which you can allow or block in robots.txt.

What you control at each stage

A quick health check

1. Search Google for site:yourdomain.com to get a rough idea of how many pages are indexed (the number is an estimate).
2. Open Search Console’s Page indexing report and review the reasons pages are not indexed.
3. Run an audit of your most important page to catch crawl, index and on-page problems.

Crawl budget: when it matters and how to manage it

For most websites, Google crawls everything important without any help. Crawl budget becomes a real concern on sites with tens of thousands of URLs or more – large online shops, marketplaces, classifieds and news sites – or on sites whose servers are slow.

Two things determine how much a search engine crawls:

• Crawl capacity – how many requests your server can handle without slowing down. If responses get slow or return errors, Googlebot backs off.
• Crawl demand – how much Google wants to crawl, based on how popular and how frequently updated your pages are.

The most common way sites waste crawl budget is by generating huge numbers of low-value URLs: faceted navigation that creates a URL for every combination of filters, session IDs in URLs, calendar pages that go on forever, internal search results and infinite scroll without paginated URLs. Each of these can produce thousands of near-duplicate pages that crawlers dutifully fetch.

To keep crawling focused:

1. Block crawl traps such as sort and filter parameters in robots.txt when they add no unique value
2. Avoid linking to URLs you do not want crawled
3. Keep your sitemap limited to canonical, indexable URLs
4. Fix redirect chains and broken links, which waste requests
5. Improve server response time – a faster site can be crawled more

What the log files tell you

Server log files record every request to your site, including those from search engine crawlers. Analysing them shows which pages Googlebot actually visits, how often, and which status codes it receives. Log analysis is the only way to see crawling as it really happens rather than as tools estimate it. On cPanel hosting, raw access logs are usually available under “Raw Access” or “Metrics”. Verify that requests claiming to be Googlebot really are, since bots often fake the user agent – Google publishes the IP ranges its crawlers use.

Mobile-first indexing

Google now predominantly uses the mobile version of pages for crawling and indexing. Googlebot Smartphone is the main crawler. If your mobile site shows less content, fewer links, different structured data or missing images compared with desktop, Google may never see what is missing. Responsive design – the same HTML for all devices, adapted with CSS – avoids the problem entirely.

A worked example

Imagine a new recipe blog publishes a post about sourdough. Here is what happens:

1. The blog’s homepage, which Googlebot visits regularly, now links to the new post. The sitemap also lists it with a fresh lastmod date.
2. Googlebot discovers the URL and adds it to its crawl queue. Because the blog is new, it may take a day or two.
3. Googlebot fetches the page. The recipe is in the HTML, so no rendering delay occurs.
4. Google indexes the page, notes its Recipe structured data, the ingredients and the author, and identifies it as an English recipe for sourdough bread.
5. When someone searches “easy sourdough recipe for beginners”, Google compares the post with thousands of others and ranks it based on relevance, quality signals, the site’s reputation and the searcher’s context.

Every step is a place to improve: faster discovery with internal links, faster indexing with server-rendered content, better understanding with structured data, and better rankings with genuinely helpful content.

Key takeaways

• Search engines work in four stages: crawling, rendering, indexing and ranking – a page can fail at any of them.
• Crawlers find pages through links, sitemaps and submissions, and robots.txt controls where they may go.
• Content that needs JavaScript to appear is indexed more slowly by Google and is invisible to most AI crawlers.
• Indexing is selective: duplicates are consolidated to one canonical URL and low-value pages may be left out.
• Ranking weighs relevance, quality, usability and context; links remain an important signal of authority.
• AI search features rely on the same foundations – a page must be crawlable, indexable and allowed to show snippets.

Frequently asked questions

How long does it take Google to index a new page?

It ranges from a few hours to several weeks. Linking to the page from important pages, including it in your sitemap and requesting indexing in Search Console all help it get discovered faster.

Does submitting a sitemap guarantee indexing?

No. A sitemap helps search engines discover URLs, but they still decide whether each page is worth indexing.

Why does Google index a different URL than the one I want?

Google picks the canonical version from a group of duplicates using signals like redirects, canonical tags, internal links and sitemaps. Make all of those signals point consistently to your preferred URL.

Do I need to submit my site to search engines?

Not strictly – search engines find most sites through links. But verifying your site in Google Search Console and Bing Webmaster Tools and submitting a sitemap gives you faster discovery and valuable diagnostic data.