Technical SEO
Robots.txt vs. Noindex: How to Control Discovery
Robots.txt manages crawl access. A noindex directive requests exclusion from search results and ordinarily needs to be visible to the crawler.


Browse related state and city resources
How to approach it
Inspect both the robots file and the rendered response. If the page is private, enforce authentication. For public pages you want omitted from results, review the platform’s noindex guidance and ensure the crawler can see the directive when necessary.
Practical example
A client sign-in page can use noindex while document endpoints require access checks. Public guides should have crawl access and point to their canonical pages.
How to evaluate it
Test with indexing tools and review server behavior after changes. Avoid treating crawler directives as a means of protecting confidential information.
Reference
Google Search Console URL Inspection documentation
