The robots.txt File

📊

Alex, Trainee · Wednesday, 09:15

I open Google Search Console. Traffic: 0. Pages indexed: 0 of 847. This can't be right...

😤

CEO · 09:18

"ALEX. ALL TRAFFIC IS GONE. WHY ARE ZERO PAGES OUT OF 847 INDEXED?! THIS IS A CATASTROPHE!"

🤦

Alex · 09:21

I open robots.txt. Everything is clear now.

User-agent: *
Disallow: /

# Bye bye 😈
# — Peterson

What is robots.txt?

It's an instruction file for search engine bots. It lives at the root of your site: yourdomain.com/robots.txt

Think of it as a security guard at the building entrance — it tells bots: "you can go here, but not there".

✅

Allow: /
Public pages are open. Bots can crawl and index them.

🚫

Disallow: /admin/
Private sections are blocked. Bots should not enter.

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

What this means:

User-agent: * — rules for all bots
Disallow: /admin/ — do not crawl the /admin/ folder
Allow: / — everything else is accessible

The biggest misconception about robots.txt

🚫 robots.txt does NOT prevent indexing!

If a bot doesn't crawl a page — that doesn't mean Google won't index it. Google can learn about it through links and add it to the index without ever visiting.

To actually block a page from indexing — use: <meta name="robots" content="noindex">

Sitemap link

Best practice: add a link to sitemap.xml in robots.txt — this helps Google discover all your pages faster:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

💡 Pro tip: Check robots.txt via Google Search Console → "robots.txt Tester". The most common mistake is accidentally writing Disallow: / and blocking the entire site. Like Peterson did 😈

😌

Alex · 09:45

Removed Disallow: /. Updated robots.txt. Now waiting for Google to recrawl. First crisis: handled.

Ready? Type your answer in the task! 👇

🎯

Lesson Task

Test your knowledge and earn +20 XP

← SEO Platforms Overview

Lesson 16 of 16

Go to Task →