Robots Meta Directives: How to Read the Page Signal
Website intelligenceRobots Meta Directives: How to Read the Page Signal
A robots meta directive can tell compliant crawlers whether a page should be indexed or whether certain links should be followed. A directive copied from a staging environment can accidentally affect a public page after launch.
Find the directive in the published HTML
Inspect the page's head section and locate the robots meta tag. Read the exact directive rather than relying on a CMS checkbox whose saved state may not match the output.
Consider the page's purpose before changing it. A private account page and a public article have different indexing needs, so the correct directive depends on the content and access model.
Compare it with the SiteBoost report
SiteBoost reports the page-level robots meta value it reads from the scanned URL. If it shows noindex on a page intended for public search, confirm the live source and check whether a shared template is adding the directive.
A page-level meta tag is not the only access signal. robots.txt and HTTP response headers can also affect crawling or indexing, and a narrower SiteBoost check should not be treated as a complete inspection of every server rule.
- Read the exact meta directive.
- Check whether the page should be public and indexable.
- Review template defaults and deployment settings.
- Use other tools for robots.txt and HTTP header checks.
Rescan after changing the directive
Remove or adjust the directive only when the page's purpose supports that change. Then publish the correction and scan the same URL again to confirm the value in the page output.
Do not assume a page will appear in search results immediately after a noindex change. Indexing decisions and timing remain with the search engine.
