What is robots.txt?
Robots.txt is a small file that tells search engines which parts of your site they don't need to visit.
- Since 2005
- 5.0 on Google
- Explanations without the jargon
Google Partner
Shopify Partner5.0 Google ReviewsSinds 2005 in marketingNext level online succesSEO uitbestedenAI-vindbaarheidWebdesign & webshopsLooptijd 6 of 12 maandenEén vast aanspreekpuntGoogle PartnerShopify Partner5.0 Google ReviewsSinds 2005 in marketingNext level online succesSEO uitbestedenAI-vindbaarheidWebdesign & webshopsLooptijd 6 of 12 maandenEén vast aanspreekpuntKort antwoord
What is robots.txt?
Robots.txt is a small file that tells search engines which parts of your site they don't need to visit.
It is always located at /robots.txt and contains rules per bot. It is a request, not a lock: it won't stop malicious bots.
Important difference: robots.txt prevents crawling, not indexing. If you really want to keep a page out of Google, use noindex.
Ik hoop dat je dit artikel met plezier leest. Heb je hulp nodig bij SEO, GEO, UX-design, CRO, webdesign of SaaS-software? Klik dan hier en bekijk mijn diensten.
Geschreven door: Lubna Ouarghi
Why it matters
One wrong line can keep your entire site out of Google. We see it more often than you'd think after a site migration or a new staging environment.
Properly configured, it stops search engines from wasting time on filter pages and internal search results in your online shop.
How we approach it
We check robots.txt during the baseline audit, keep it as simple as possible, and reference your sitemap.
At launch, this is the first thing we check, because a leftover staging setting will cost you traffic straight away.
Frequently asked questions
How do I know if it's set up correctly?
Search Console reports blocked pages. We also check it during the free scan.
Can I block AI bots?
Yes, bot by bot. Bear in mind that you'll also block your chances of being mentioned in AI answers.
Bronnen en controle
Laatst bijgewerkt en gecontroleerd: September 2026 — door Lubna Ouarghi
This explanation is based on the official documentation below and on my own work in client projects.
- Google Search Central — SEO Starter Guide — what SEO is and where to start
- Google Search Central — Do you need an SEO specialist? — what a specialist does and what to look out for
- Google Search Essentials — the rules your site must comply with
Want to know where your website stands on this?
Request the free visibility scan. You'll see your position on Google, what AI assistants say about you, and the three steps that will deliver the biggest return.
Liever direct sparren?
Vul het formulier in of bel ons direct. Je spreekt altijd een specialist, geen accountmanager die de vraag doorzet.
- Eén vaste contactpersoon
- Maandelijkse rapportage met toelichting
- Heldere looptijd: 6 of 12 maanden
- 20+ jaar ervaring in SEO en e-commerce