What Is robots.txt? A Plain-English Guide (and What to Do About AI Bots)
Quick answer: robots.txt is a small text file at yourdomain.com/robots.txt that tells search engine and AI crawlers which parts of your website they may visit. It’s a polite request — reputable bots follow it, bad bots ignore it. It is not a way to hide pages from Google (use “noindex” or a password for that). For most small business websites: let search engines in, keep admin areas out, point to your sitemap, and decide which AI bots you allow — AI search bots help people find you; AI training bots are a separate choice.
It happens more often than you’d think: a brand-new website looks great, the content is good, yet weeks later it still isn’t showing up on Google. Sometimes the culprit is a single line in a file called robots.txt, switched on during development to keep search engines away — and never switched off. Here’s what this little file does, and why it matters even more now that AI bots are visiting your site too.
What Is robots.txt?
When a search engine or AI tool visits your website, it sends an automated program called a crawler (or bot). Before it looks around, a well-behaved crawler checks your robots.txt file for instructions. Google describes it as a file that “tells search engine crawlers which URLs the crawler can access on your site.” A typical small business robots.txt looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.yourbusiness.com/sitemap_index.xml In plain English: “All bots are welcome, except in the WordPress admin area — and here’s a map of my pages.” To see yours, just type your domain followed by /robots.txt into a browser.
The Four Lines You'll See Most
| Line | What it means |
|---|---|
| User-agent: | Which bot the rules are for. An asterisk (*) means all bots. |
| Disallow: | A folder or page the bot should not visit. “Disallow: /” means the whole site. |
| Allow: | An exception inside a disallowed folder. |
| Sitemap: | Where to find your sitemap — the list of pages you want found. |
What robots.txt Can and Can't Do
| It can… | It can’t… |
|---|---|
| Ask bots not to crawl admin areas, cart pages or duplicate pages | Hide a page from Google — a blocked page can still appear in results if other sites link to it |
| Reduce unnecessary load on your server | Stop bad bots or hackers — following it is voluntary |
| Point search engines to your sitemap | Protect private information — the file itself is public |
Google is clear that robots.txt “is not a mechanism for keeping a web page out of Google.” To keep a page out of search results, use a “noindex” setting (most SEO plugins have a tick box) or put it behind a password.
AI Bots: Which Ones Are Visiting Your Website?
Besides search engines, AI companies now send their own crawlers. The important thing to know is that most of them run separate bots for search and for training — so you can be found in AI answers without letting your content be used to train AI models, if you prefer.
| Bot name | Company | What it does | Should you block it? |
|---|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews | No — blocking it removes you from Google | |
| Google-Extended | Controls use of your content for Gemini training and grounding | Your choice — doesn’t affect Google Search ranking | |
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search answers | No, if you want to appear in ChatGPT |
| GPTBot | OpenAI | Collects content to train OpenAI models | Your choice |
| Claude-SearchBot | Anthropic | Indexes pages for Claude’s search answers | No, if you want to appear in Claude |
| ClaudeBot | Anthropic | Collects content for AI model training | Your choice |
Example: Allow AI Search, Block AI Training
If you want to appear in AI answers but don’t want your content used for training, a robots.txt like this does it:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.yourbusiness.com/sitemap_index.xml Is blocking training the right call? It’s a business decision. Some owners prefer to keep their content out of AI training; others want AI models to “know” their brand as widely as possible. Either way, keep the search bots allowed if you want to be recommended — see how to get your business mentioned in ChatGPT and AI Overviews and how to get your products recommended in AI search.
Where to Find and Edit robots.txt
- WordPress: WordPress creates a basic robots.txt automatically. SEO plugins such as Yoast let you edit it (in Yoast: Tools → File editor).
- Shopify: Shopify provides a sensible default. Advanced changes are made through a robots.txt.liquid file in your theme — ask your developer.
- Other hosting: it’s a plain text file in the main (root) folder of your website, editable through your hosting file manager.
Common robots.txt Mistakes
- “Disallow: /” left on after launch — blocks the entire site. In WordPress, also check that “Discourage search engines from indexing this site” (Settings → Reading) is unticked.
- Blocking CSS or JavaScript files — Google needs them to see your pages properly.
- Listing “secret” pages — robots.txt is public, so you’re actually pointing people to them.
- Blocking AI search bots by accident — some security plugins and hosting or CDN services offer one-click “block AI bots” settings. Check exactly which bots they block.
- Forgetting the sitemap line.
After any change, check the robots.txt report in Google Search Console to confirm Google can read it.
Where ELF Digital Studio Comes In
Every website and Shopify store we launch gets its robots.txt, sitemap and indexing settings checked — including which AI bots are allowed — so nothing accidentally blocks you from Google or AI search.
Not sure what your robots.txt is telling search engines and AI? Talk to us — we’ll check it and explain it in plain English, honestly.
Frequently Asked Questions
Do I need a robots.txt file?
Not strictly — without one, bots assume they can crawl everything. But a simple robots.txt with your sitemap line is good practice, and WordPress and Shopify create one for you.
If I block GPTBot, will I disappear from ChatGPT?
No. GPTBot is OpenAI’s training crawler. Appearing in ChatGPT search depends on OAI-SearchBot, which you can keep allowed.
Can robots.txt hide a page from Google?
No. A blocked page can still show up in results if other sites link to it. Use a noindex setting or password protection instead.
Does blocking Google-Extended hurt my Google ranking?
No. Google states that Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal.



