CLIENT REVENUE GENERATED$9,720,200+Independent consulting · Since 2008
Talk Strategy

LEARN SEO / CRAWLING AND INDEXING

How do crawling and indexation work?
Find the stage where the URL needs attention.

A URL can be known without being crawled, crawled without being indexed, or indexed without appearing for the search you want. I separate those questions before choosing a fix.

See a Practical ExampleExplore the guide

Himanshu Swaraj
Practical guidance and implementation

18 years in search marketing
Investigation, prioritisation and review

Meet Himanshu

STARTING POINT

Reachable, indexed and visible are different conditions.

Understand the purpose

Crawling is the request for a page or resource. Indexing involves processing content for possible inclusion in a search index. Serving results is the later selection of information for a query.

Choose the useful investigation

If a page is not appearing, I start with its intended role. Should it be an independent search destination, a duplicate of another URL, or a page that should stay out of search? That decision changes the checks and the solution.

KEY AREAS

Separate the questions before choosing the work.

Discovery and internal links

A URL needs a path to discovery. Review relevant internal links and the URLs the site makes available, rather than assume a published page has been found.

Response and access

Check what the requested URL returns. A login requirement, failed request, redirect or server error needs a different investigation from a normal page response.

robots.txt

A disallow rule controls crawling by compliant crawlers. It is not a dependable way to remove a page from Google’s index; a blocked URL can still appear without its content being crawled. The robots.txt generator can help draft the file, but review each rule against the access you intend to allow.

noindex

Google needs access to read a noindex rule in the page or response header. Blocking crawling can prevent that rule from being seen. Do not put noindex in robots.txt.

Canonical preference

A canonical identifies a preferred representative for similar content. Google can select a different URL. Check whether links, redirects and sitemap entries agree with the intended version.

Sitemaps and page purpose

Review whether the sitemap lists the intended search destinations. Inclusion helps communicate URLs; it does not guarantee indexing or visibility. Use the XML sitemap generator to prepare a small sitemap from the public URLs you have checked.

WORKED EXAMPLE

A sitemap cannot make a noindex page eligible.

Suppose an important service page appears in the sitemap but still carries a noindex rule left from staging. Repeated sitemap submissions do not resolve the conflicting instruction.

I would confirm that the page is intended for search, inspect the live response and directive, correct the verified issue within scope, then validate the change. Later inspection should distinguish the current live page from an older recorded crawl.

QUESTION → EVIDENCE → ACTION

Confirm the goalDecide what the page should help a visitor do.

Inspect the issueVerify the current page, response and relevant context.

Validate the changeRecheck the original issue after implementation.

01 / INTENT

Should it be indexed?

Confirm the page’s role before editing controls.

02 / ACCESS

Can it be requested?

Inspect the response and applicable crawl rules.

03 / CONTENT

What signals are present?

Review directives, content and canonical preference.

04 / REVIEW

What has been processed?

Compare current evidence with recorded inspection information.

PRACTICAL CHECKS

Keep the task and the follow-up clear.

01

Confirm the exact URL

Keep the protocol, host, path and relevant parameters so the checks refer to the same address.

02

Inspect the response

Check status, redirects, access requirements and the final destination.

03

Review crawl and index controls

Inspect applicable robots rules and live page or header directives.

04

Check content and canonical context

Review the rendered information and the relationship with similar URLs.

05

Validate and follow up

Record the change, recheck the live response and review later inspection evidence.

IMPORTANT DISTINCTIONS

Check the rendered content when JavaScript matters.

Check what the finding establishes

A browser may display content that was added after the initial HTML response. For a page that depends on scripts, inspect whether the important text and links appear in the rendered result.

Use the right supporting evidence

A simple source-view tool does not prove what Google processed. Where access is available, review Google Search Console’s URL Inspection evidence alongside the live page and relevant server information. The search engine spider simulator is a supporting HTML check, not a record of what Google rendered or indexed.

COMMON MISTAKES

Avoid creating a new problem while fixing the first.

Combining disallow and noindex without checking

A crawl block can prevent Google from reading noindex. Choose controls based on the intended outcome.

Treating every excluded URL as a fault

An alternate duplicate or intentionally excluded page may be behaving as intended. Check its role first.

Reading an old crawl as a live test

Recorded inspection information and the current page can differ after a change. Keep the dates and conditions with the finding.

TOOLS AND SUPPORT

Use focused checks. Interpret the results.

Broken linksInvestigate destinations linked from a public page; confirm important failures.Open the resource →Response speedCheck the initial HTML response and transfer time, not full browser rendering.Open the resource →SEO audit supportGet help verifying findings and prioritising implementation.Open the resource →

KEEP LEARNING

Continue with the question that matters next.

Technical SEORead the guide →Internal linkingRead the guide →How SEO worksRead the guide →Structured dataRead the guide →
Browse the Learn hub

ABOUT THE AUTHOR

Himanshu Swaraj

Himanshu Swaraj, SEOFreelance consultant

My 18 years in search marketing shape how I approach a website problem: confirm the need, inspect the evidence and make the next task clear.

I can help investigate the issue and implement agreed improvements. The scope identifies responsibilities, access needs and review points.

Meet Himanshu

QUESTIONS

The practical answers.

Is crawling the same as indexing?

No. Requesting a URL and including its content in a search index are different stages. A crawl alone does not establish indexing.

Does robots.txt keep a page out of Google?

Not reliably. It manages crawling. A blocked URL can still be indexed from other information, without Google crawling its content.

Can Google see noindex on a blocked page?

Google needs to access the page or resource to read its noindex rule. A robots.txt block can prevent that access.

Does a canonical force Google to choose my URL?

No. It signals a preferred representative. Review the surrounding signals and Google’s selected canonical rather than assume the preference was followed.

Does a sitemap guarantee indexing?

No. A sitemap communicates URLs, but does not guarantee crawling, inclusion or a search position.

Should every page be indexed?

No. The intended role matters. Useful search destinations, duplicate variants and pages intentionally kept out of search need different treatment.

FURTHER READING

Check the primary guidance.

GET PRACTICAL HELP

Need help finding what is holding the site back?

Share your website and the issue. I’ll review the scope before recommending the work.

Explore an SEO Audit