robots.txt Crash Course

Nerd Cafe | نرد کافه

1. What robots.txt Does

robots.txt tells search engines like Google:

• Which pages they can crawl

• Which pages they cannot crawl

• Where your sitemap is

It helps with SEO and site privacy.

Example:

https://example.com/robots.txt

The file must be in the root directory.

2. Basic Structure

A simple robots.txt file:

User-agent: *
Disallow:

Meaning:

Directive	Meaning
User-agent	Which bot the rule applies to
Disallow	Pages the bot cannot access

* = all bots.

3. First Working Example

Allow everything:

User-agent: *
Disallow:

This means:

• All search engines allowed.

4. Blocking Pages

Block a folder:

User-agent: *
Disallow: /admin/

Blocks:

example.com/admin/

Block a file:

User-agent: *
Disallow: /private.html

5. Allow Specific Pages

User-agent: *
Disallow: /private/
Allow: /private/public.html

Meaning:

❌ Block /private/ ✔ Allow /private/public.html

6. Target Specific Bots

Example for Google:

User-agent: Googlebot
Disallow: /no-google/

This affects only Google crawler.

Other bots unaffected.

7. Common Bots

Examples:

User-agent: Googlebot
User-agent: Bingbot
User-agent: Slurp
User-agent: DuckDuckBot

Or use:

User-agent: *

For all bots.

8. Blocking Entire Site

User-agent: *
Disallow: /

Meaning:

❌ No search engine allowed.

Useful for:

• Testing websites

• Private projects

9. Adding Sitemap

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Helps search engines find pages faster.

10. Real Example robots.txt

User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /tmp/

Allow: /blog/

Sitemap: https://example.com/sitemap.xml

11. How to Create robots.txt

Step 1

Create a file named:

robots.txt

Step 2

Paste rules:

User-agent: *
Disallow: /private/

Step 3

Upload to:

yourwebsite.com/robots.txt

12. Comments

# Block admin pages
User-agent: *
Disallow: /admin/

# = comment.

13. Wildcards

User-agent: *
Disallow: /*.pdf$

Blocks all PDF files.

Examples blocked:

file.pdf
book.pdf
doc.pdf

14. Common Mistakes

Wrong filename

robot.txt ❌
robots.txt ✔

Wrong location

example.com/files/robots.txt ❌
example.com/robots.txt ✔

Blocking entire site accidentally

Disallow: /

Very common error.

💖 Support Our Work

If you find this post helpful and would like to support my work, you can send a donation via TRC-20 (USDT). Your contributions help us keep creating and sharing more valuable content.

TRC-20 Address: TAAVVf9ZxUpbyvTa6Gd5SGPmctBdy4PQwf

Thank you for your generosity! 🙏

Channel Overview

🌐 Website: www.nerd-cafe.ir

📺 YouTube: @nerd-cafe

🎥 Aparat: nerd_cafe

📌 Pinterest: nerd_cafe

📱 Telegram: @nerd_cafe

📝 Blog: Nerd Café on Virgool

💻 GitHub: nerd-cafe