Task: robots.txt and meta robots in Detail

For noindex to work the page must not be blocked in robots.txt Google supports the Crawl-delay directive and honors the pause With conflicting robots.txt rules the most specific one by path length applies, and for a tie the least restrictive Google always treats a 5xx response for robots.txt as no restrictions For PDFs and other files with no head, indexing rules can be set with an X-Robots-Tag header