Crawlability vs Indexability: Two Questions to Keep Separate
Website intelligenceCrawlability vs Indexability: Two Questions to Keep Separate
Crawlability and indexability are related but different. A crawler needs to access a page before it can reliably read its contents, while indexing is a separate decision about whether that content should appear in search results.
Check access and page response first
Start by confirming that the public URL resolves to the intended page and returns the expected response. If the server blocks the request or redirects somewhere unexpected, investigate access before reviewing content signals.
robots.txt can restrict crawling, while page-level robots directives can provide indexing instructions when the page is fetched. These controls should not be treated as interchangeable.
Use SiteBoost for the page signals it reports
SiteBoost reports response context and the robots meta value found in the scanned HTML, along with canonical and structural checks. Those values help identify issues worth reviewing, but they are not a complete record of Google's crawl or index decisions.
Use the report to isolate a technical question, then inspect robots.txt and search-engine webmaster tools when you need broader access or indexing evidence.
- Check that the public URL resolves correctly.
- Review the page-level robots meta value.
- Inspect robots.txt separately.
- Use search-engine tools for indexing status.
Avoid promising an indexing outcome
Correcting a crawl or index directive can remove a technical obstacle, but it does not guarantee that a search engine will index the page.
Keep content usefulness, canonical consistency and site structure in the wider investigation.
