Glossary / SEO & AI Search
robots.txt.
A file that tells crawlers which parts of a site they may access.
What it means
A file that tells crawlers which parts of a site they may access. It controls crawling, not indexing; use noindex to keep a page out of results.
Related terms
More in SEO & AI Search
Read more.
Articles that put robots.txt to work.
LLM Optimization Explained: Why Ranking #1 No Longer Guarantees ChatGPT Mentions You
LLM optimization for local businesses: why Google rankings don't carry into ChatGPT and AI Overviews, what does, and the fixes that get you recommended.
Answer Engine Optimization (AEO) in 2026: A Practical Guide for Small and Mid-Sized Businesses
AEO is the work of becoming the answer ChatGPT, Perplexity and Google AI Overviews give. What it is, how it differs from SEO, and the eight things to do first.
Redesigning Your Website Without Losing SEO: A Migration Checklist
Redesigns are where rankings go to die. The checklist that protects traffic, leads and AI visibility through a new site launch.
Questions aboutrobots.txt?
Ask the team that does this work every day. We reply within one business day.