Анализ сайта enwp.org
Основное Готовность: 95%
Домен
enwp.org
Состояние доменного имени
?
Проверяем корректность доменного имени и наличие технических проблем на уровне домена.
Домен второго уровня идеален для продвижения.
Отличный запоминающийся домен.
Ответ сервера
200 Успешный ответ
HTTP-код ответа и цепочка редиректов
?
Код 200 — страница доступна. Коды 3xx — редиректы (цепочки замедляют загрузку и размывают ссылочный вес). Коды 4xx/5xx — ошибки, поисковик не сможет проиндексировать страницу.
Кол-во редиректов 4 слишком большое! Проверьте настройки сайта и веб-сервера!
Цепочка редиректов:
http://enwp.org
301 MovedPermanently
https://enwp.org/
301 MovedPermanently
https://en.wikipedia.org/wiki/
301 MovedPermanently
https://en.wikipedia.org/wiki/Main_Page
200 OK
Безопасность
Сайт безопасен
Использование HTTPS и SSL-сертификат
?
HTTPS — обязательный стандарт. Google и Яндекс отдают предпочтение защищённым сайтам. Отсутствие SSL или просроченный сертификат ведут к предупреждениям в браузере и снижению позиций.
На сайте работает защищенный протокол ssl и сайт открывается по https.
Ssl-сертификат действителен до 17.11.2026 16:21:14.
Включён HSTS (Strict-Transport-Security) — защита от подмены на http.
HTTP автоматически перенаправляется на HTTPS.
Поздравляем! Сайт не содержится в реестре РКН.
Кодировка
UTF-8
Кодировка символов страницы
?
Стандарт — UTF-8. Неправильная кодировка вызывает нечитаемые символы и мешает поисковику корректно распознать текст страницы.
Указана кодировка на странице UTF-8.
Язык
en
Атрибут lang в HTML-теге
?
Атрибут lang (<html lang="ru">) сообщает поисковикам и браузерам, на каком языке написана страница. Помогает при ранжировании в региональном поиске.
Язык документа указан явно: en.
Скорость загрузки
~0,86сек
Время отклика сервера (TTFB)
?
Time To First Byte — время до получения первого байта от сервера. Норма до 200 мс. Медленный отклик ухудшает пользовательский опыт и ранжирование: Яндекс и Google учитывают скорость страниц.
Скорость загрузки сайта 0,86сек оптимальна.
Объем документа
246Кб
Размер HTML-кода страницы
?
Слишком большой HTML замедляет парсинг браузером и сканирование поисковым роботом. Рекомендуется не более 200 Кб.
Объем html-документа 246Кб оптимален.
Структура html-документа корректна.
Ресурсы
Ресурсы: 3
Внешние ресурсы страницы (CSS, JS, изображения)
?
Количество и тип подключённых ресурсов влияют на скорость загрузки. Большое число запросов увеличивает время рендеринга страницы.
Кол-во файлов ресурсов 3 достаточно.
Показать полный список ресурсов
| Тип | Название | Значение |
|---|---|---|
| stylesheet | /w/load.php?lang=en&modules=ext.uls.interlanguage%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediamessages.styles%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skins.legacy%7Cskins.vector.icons%2Cstyles%7Cskins.vector.search.codex.styles%7Cwikibase.client.init&only=styles&skin=vector-2022 | |
| stylesheet | /w/load.php?lang=en&modules=site.styles&only=styles&skin=vector-2022 | |
| js | /w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector-2022 |
Серверные заголовки
Кол-во: 20
HTTP-заголовки ответа сервера
?
Заголовки сервера передают браузеру и поисковику служебную информацию: кеширование, безопасность (CSP, HSTS), сжатие (gzip). Правильная настройка ускоряет загрузку и повышает защищённость.
Найдены серверные заголовки 20шт. Подробнее про серверные заголовки.
Показать полный список серверных заголовков
| Ключ | Значение |
|---|---|
| Date | Fri, 21 Aug 2026 12:05:15 GMT |
| Server | mw-web.eqiad.main-7f7db4796-29ctx |
| X-Content-Type-Options | nosniff |
| accept-ch | |
| reporting-endpoints | csp-report-to-endpoint='/w/api.php?action=cspreport&format=json'; |
| Content-Security-Policy | script-src 'unsafe-eval' blob: 'self' meta.wikimedia.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikifunctions.org *.wikivoyage.org *.mediawiki.org mediawiki.org wikimedia.org *.wmflabs.org *.wmcloud.org *.toolforge.org wss://*.toolforge.org *.jsdelivr.net unpkg.com cdnjs.cloudflare.com raw.githubusercontent.com *.github.com code.jquery.com cdn.mathjax.org use.typekit.net fonts.cdnfonts.com use.fontawesome.com i.ytimg.com rsms.me doi.org localhost https://localhost:* http://localhost:* wss://localhost:* ws://localhost:* *.google.com *.gstatic.com *.googleapis.com *.translate.yandex.net yastatic.net ya.ru radically.github.io cdn.sammdot.ca cdn.fontshare.com viaf.org publicai-proxy.alaexis.workers.dev iiif.archive.org api.flickr.com live.staticflickr.com api.anthropic.com api.openai.com api.publicai.co catalogo.pusc.it parsifal.urbe.it opac.sbn.it overpass-api.de api.openrouteservice.org archive.org *.openstreetmap.org *.waymarkedtrails.org *.thunderforest.com registry.ipe.wiki analytics.ipe.wiki qlever.dev app.goacoustic.com wikipedia-archive.ourworldindata.org api.inaturalist.org inaturalist-open-data.s3.amazonaws.com validator.w3.org db.onlinewebfonts.com fontlibrary.org 'unsafe-inline' auth.wikimedia.org; default-src 'self' data: blob: upload.wikimedia.org https://commons.wikimedia.org meta.wikimedia.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikifunctions.org *.wikivoyage.org *.mediawiki.org mediawiki.org wikimedia.org *.wmflabs.org *.wmcloud.org *.toolforge.org wss://*.toolforge.org *.jsdelivr.net unpkg.com cdnjs.cloudflare.com raw.githubusercontent.com *.github.com code.jquery.com cdn.mathjax.org use.typekit.net fonts.cdnfonts.com use.fontawesome.com i.ytimg.com rsms.me doi.org localhost https://localhost:* http://localhost:* wss://localhost:* ws://localhost:* *.google.com *.gstatic.com *.googleapis.com *.translate.yandex.net yastatic.net ya.ru radically.github.io cdn.sammdot.ca cdn.fontshare.com viaf.org publicai-proxy.alaexis.workers.dev iiif.archive.org api.flickr.com live.staticflickr.com api.anthropic.com api.openai.com api.publicai.co catalogo.pusc.it parsifal.urbe.it opac.sbn.it overpass-api.de api.openrouteservice.org archive.org *.openstreetmap.org *.waymarkedtrails.org *.thunderforest.com registry.ipe.wiki analytics.ipe.wiki qlever.dev app.goacoustic.com wikipedia-archive.ourworldindata.org api.inaturalist.org inaturalist-open-data.s3.amazonaws.com validator.w3.org db.onlinewebfonts.com fontlibrary.org en.wikibooks.org en.wikinews.org en.wikiquote.org en.wikisource.org en.wikiversity.org en.wikivoyage.org en.wiktionary.org www.mediawiki.org commons.wikimedia.org foundation.wikimedia.org incubator.wikimedia.org species.wikimedia.org wikimania.wikimedia.org www.wikidata.org www.wikifunctions.org auth.wikimedia.org; style-src 'self' data: blob: upload.wikimedia.org https://commons.wikimedia.org meta.wikimedia.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikifunctions.org *.wikivoyage.org *.mediawiki.org mediawiki.org wikimedia.org *.wmflabs.org *.wmcloud.org *.toolforge.org wss://*.toolforge.org *.jsdelivr.net unpkg.com cdnjs.cloudflare.com raw.githubusercontent.com *.github.com code.jquery.com cdn.mathjax.org use.typekit.net fonts.cdnfonts.com use.fontawesome.com i.ytimg.com rsms.me doi.org localhost https://localhost:* http://localhost:* wss://localhost:* ws://localhost:* *.google.com *.gstatic.com *.googleapis.com *.translate.yandex.net yastatic.net ya.ru radically.github.io cdn.sammdot.ca cdn.fontshare.com viaf.org publicai-proxy.alaexis.workers.dev iiif.archive.org api.flickr.com live.staticflickr.com api.anthropic.com api.openai.com api.publicai.co catalogo.pusc.it parsifal.urbe.it opac.sbn.it overpass-api.de api.openrouteservice.org archive.org *.openstreetmap.org *.waymarkedtrails.org *.thunderforest.com registry.ipe.wiki analytics.ipe.wiki qlever.dev app.goacoustic.com wikipedia-archive.ourworldindata.org api.inaturalist.org inaturalist-open-data.s3.amazonaws.com validator.w3.org db.onlinewebfonts.com fontlibrary.org 'unsafe-inline'; object-src 'none'; report-uri /w/api.php?action=cspreport&format=json; report-to csp-report-to-endpoint |
| Age | 5752 |
| Accept-Ranges | bytes |
| X-Cache | cp3066 miss, cp3066 hit/43611 |
| x-cache-status | hit-front |
| Strict-Transport-Security | max-age=106384710; includeSubDomains; preload |
| report-to | { "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] } |
| nel | { "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0} |
| Set-Cookie | WMF-DP=4ae;Path=/;HttpOnly;secure;Expires=Sat, 22 Aug 2026 00:00:00 GMT |
| x-client-ip | 89.249.18.75 |
| Cache-Control | no-transform, must-revalidate, max-age=0, s-maxage=0, private |
| Vary | Accept-Encoding |
| X-Request-ID | b0499b6b-e736-4a25-9ef6-720a13dd70f5 |
| x-analytics | |
| Server-Timing | cache;desc="hit-front", host;desc="cp3066",co_id;desc="2697918312" |
CMS
MediaWiki
Система управления сайтом (движок)
?
CMS — это движок, на котором работает сайт (WordPress, 1C-Bitrix, Tilda и др.). Знание CMS помогает понять возможности SEO-оптимизации и подобрать подходящие инструменты. «Не определена» — вероятно, самописный сайт или нестандартная сборка.
Сайт работает на CMS MediaWiki.
Веб-сервер
mw-web.eqiad.main-7f7db4796-29ctx
Программное обеспечение сервера
?
Веб-сервер — это ПО, которое отдаёт страницы посетителям (nginx, Apache, IIS, LiteSpeed и др.). Определяется по серверным заголовкам ответа (Server, X-Powered-By и т.п.). «Не определён» — сервер намеренно скрывает эти заголовки, это нормальная практика безопасности.
В заголовке Server указано: mw-web.eqiad.main-7f7db4796-29ctx.
Мета-теги Готовность: 43%
Title
Wikipedia, the free encyclopedia
Заголовок страницы в браузере и поисковой выдаче
?
Title — главный SEO-заголовок страницы. Влияет на CTR в поиске и ранжирование. Оптимальная длина: 50–70 символов. Ключевые слова — ближе к началу.
Необходимо увеличить число символов в title (текущее значение: 32, оптимально: от 40 до 45)
Дублей словоформ в title не найдено.
Description
Описание страницы в поисковой выдаче (сниппет)
?
Meta Description — текст под заголовком в выдаче. Напрямую на позиции не влияет, но влияет на CTR. Оптимальная длина: 120–160 символов.
Установите мета-тег description!
Keywords
Список ключевых слов страницы (устаревший тег)
?
Meta Keywords не учитывается Яндексом и Google для ранжирования с 2009–2012 годов. Заполнение не обязательно, но не вредит. Конкурент может использовать содержимое для анализа.
Установите мета-тег keywords!
Канонический Url
https://en.wikipedia.org/wiki/Main_Page
Указывает поисковику основную версию страницы
?
Canonical (rel=canonical) предотвращает проблему дублей страниц. Должен точно совпадать с URL проверяемой страницы. Неправильный canonical может передать ссылочный вес на другую страницу.
Канонический Url прописан корректно.
Robots
Есть ошибки
Директивы для поисковых роботов на уровне страницы
?
Meta Robots управляет индексацией конкретной страницы: index/noindex — индексировать ли, follow/nofollow — следовать ли по ссылкам. Noindex полностью исключает страницу из поиска.
Meta-тег robots указан, но в нем содержится неизвестное значение...
Адаптивность
width=1120
Настройка масштабирования на мобильных устройствах
?
Тег viewport (<meta name="viewport">) сообщает браузеру, как масштабировать страницу на мобильных. Стандарт: width=device-width, initial-scale=1. Отсутствие — признак отсутствия мобильной версии.
Meta-тег viewport со значением width задаёт фиксированную ширину области просмотра.
Разметка OpenGraph
Кол-во: 5
Мета-теги для красивых превью в соцсетях
?
OpenGraph (og:title, og:description, og:image) управляет тем, как страница выглядит при репосте в социальных сетях и мессенджерах. Отсутствие OG-тегов — невзрачный превью при шеринге.
Разметка OpenGraph задана. Страница оптимизирована под социальные сети.
Показать полный список og мета-тегов
| Тип | Значение |
|---|---|
| og:image | https://upload.wikimedia.org/wikipedia/commons/2/20/Caf_goble.jpg?utm_source=en.wikipedia.org&utm_campaign=index&utm_content=thumbnail_unscaled |
| og:image:width | 958 |
| og:image:height | 1200 |
| og:title | Wikipedia, the free encyclopedia |
| og:type | website |
Все мета-теги
Кол-во: 14
Полный список мета-тегов страницы
?
Таблица всех meta-тегов, включая нестандартные. Позволяет найти опечатки, дубли и лишние теги.
Найдены мета-теги 14шт. Мета-теги не видимы для человека и предназначены для обмена информацией между веб-страницей и поисковыми системами, браузерами и другими веб-службами. С ними роботы 🤖 и устройства ведут себя более ожидаемо.
Показать полный список мета-тегов
| Тип | Название | Значение |
|---|---|---|
| name | ResourceLoaderDynamicStyles | |
| name | generator | MediaWiki 1.47.0-wmf.16 |
| name | referrer | origin |
| name | referrer | origin-when-cross-origin |
| name | robots | max-image-preview:standard |
| name | format-detection | telephone=no |
| name | viewport | width=1120 |
| property | og:image | https://upload.wikimedia.org/wikipedia/commons/2/20/Caf_goble.jpg?utm_source=en.wikipedia.org&utm_campaign=index&utm_content=thumbnail_unscaled |
| property | og:image:width | 958 |
| property | og:image:height | 1200 |
| property | og:title | Wikipedia, the free encyclopedia |
| property | og:type | website |
| property | mw:PageProp/notoc | |
| property | mw:PageProp/noeditsection |
Оптимизация Готовность: 85%
Структура
Ошибок нет
Семантические HTML-элементы страницы
?
Проверяет наличие основных структурных элементов: nav, header, footer, main. Корректная семантическая структура помогает поисковику понять архитектуру страницы.
Структура документа корректна (теги <html> и <body> присутствуют по одному на документ).
Контент
Есть ошибки
Объём и качество текстового содержимого
?
Анализирует объём полезного текста на странице. Слишком мало — страница может считаться малополезной. Слишком много — ухудшается читаемость и восприятие.
Абзацев с текстом 7 слишком мало. Добавьте больше абзацев с текстом (тег <p>)!
Слова из title 4 встречаются в тексте достаточно.
Среднее число слов в абзаце 99 достаточно.
Кол-во знаков контента 14068 на странице оптимально.
Кол-во слов 2040 на странице оптимально.
Заголовки
Ошибок нет
Иерархия заголовков H1–H6
?
H1 должен быть один и содержать ключевой запрос. H2–H6 описывают подразделы. Пропуск уровней (H1 → H3) и несколько H1 — типичные ошибки, снижающие понятность страницы для поисковика.
На странице присутствуют заголовки <h1> 2. Это прекрасно.
На странице присутствуют заголовки <h2> 9. Это хорошо.
Тошнота
3,87
Насколько одно слово доминирует в тексте
?
Классическая тошнота = √(частота самого повторяющегося слова). Норма до 7–8: текст воспринимается естественно. Выше — поисковик может счесть страницу переспамленной.
Тошнота страницы в пределах нормы 3,87.
Академич. тошнота
19,56%
Насколько текст перенасыщен ключевыми словами
?
Академическая тошнота = (частота слова / общее количество слов) × 100%. Показывает долю конкретного слова в тексте. Норма 5–15%.
Академическая тошнота превышает норму 5-15%. Измените текст страницы!
Семантическое ядро
20
Наиболее часто встречающиеся слова на странице
?
Топ слов по частоте использования. Показывает, какие слова доминируют в тексте с точки зрения поисковика.
Контент страницы содержит осмысленный текст и слова.
Показать список слов
| Слово | Кол-во | Частота |
|---|---|---|
| wikipedia | 15 | 0,74% |
| featured | 12 | 0,59% |
| article | 7 | 0,34% |
| wikimedia | 7 | 0,34% |
| aurora | 6 | 0,29% |
| english | 5 | 0,25% |
| articles | 5 | 0,25% |
| august | 5 | 0,25% |
| archive | 5 | 0,25% |
| bahasa | 5 | 0,25% |
| special | 4 | 0,20% |
| search | 4 | 0,20% |
| todays | 4 | 0,20% |
| during | 4 | 0,20% |
| indonesia | 4 | 0,20% |
| sidebar | 3 | 0,15% |
| projects | 3 | 0,15% |
| commons | 3 | 0,15% |
| foundation | 3 | 0,15% |
| wikisource | 3 | 0,15% |
Индексация Готовность: 0%
Индексирование
Есть ошибки
Разрешено ли индексирование страницы
?
Проверяет, не закрыта ли страница от индексации через robots.txt, meta robots или X-Robots-Tag. Страница, закрытая от индексации, не появится в поисковой выдаче.
Анкоров на странице 667 слишком много. Проведите ревизию и оптимизацию ссылок сайта.
Robots.txt
Содержит ошибки
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Проверяемая страница запрещена к индексации в robots.txt. Исправьте файл robots.txt!
Кол-во редиректов для файла robots.txt 4 слишком большое! Это может вызывать затруднение при индексации поисковыми роботами!
Robots.txt доступен по постоянному адресу
Цепочка редиректов для файла robots.txt:
http://enwp.org/robots.txt
301 MovedPermanently
https://enwp.org/robots.txt
301 MovedPermanently
https://en.wikipedia.org/wiki/robots.txt
301 MovedPermanently
https://en.wikipedia.org/wiki/Robots.txt
200 OK
Показать содержимое robots.txt
<!DOCTYPE html>
<html class="client-nojs vector-feature-language-in-header-enabled vector-feature-language-in-main-menu-disabled vector-feature-language-in-main-page-header-disabled vector-feature-page-tools-pinned-disabled vector-feature-toc-pinned-clientpref-1 vector-feature-main-menu-pinned-disabled vector-feature-limited-width-clientpref-1 vector-feature-limited-width-content-enabled vector-feature-custom-font-size-clientpref-1 vector-feature-appearance-pinned-clientpref-1 skin-theme-clientpref-day vector-feature-navigation-update-disabled vector-sticky-header-enabled vector-toc-available skin-thumbsize-clientpref-standard" lang="en" dir="ltr">
<head>
<meta charset="UTF-8">
<title>robots.txt - Wikipedia</title>
<script>(function(){var className="client-js vector-feature-language-in-header-enabled vector-feature-language-in-main-menu-disabled vector-feature-language-in-main-page-header-disabled vector-feature-page-tools-pinned-disabled vector-feature-toc-pinned-clientpref-1 vector-feature-main-menu-pinned-disabled vector-feature-limited-width-clientpref-1 vector-feature-limited-width-content-enabled vector-feature-custom-font-size-clientpref-1 vector-feature-appearance-pinned-clientpref-1 skin-theme-clientpref-day vector-feature-navigation-update-disabled vector-sticky-header-enabled vector-toc-available skin-thumbsize-clientpref-standard";var cookie=document.cookie.match(/(?:^|; )enwikimwclientpreferences=([^;]+)/);if(cookie){cookie[1].split('%2C').forEach(function(pref){className=className.replace(new RegExp('(^| )'+pref.replace(/-clientpref-\w+$|[^\w-]+/g,'')+'-clientpref-\\w+( |$)'),'$1'+pref+'$2');});}document.documentElement.className=className;}());RLCONF={"wgBreakFrames":false,"wgSeparatorTransformTable":["",""],"wgDigitTransformTable":["",""],"wgDefaultDateFormat":"dmy","wgMonthNames":["","January","February","March","April","May","June","July","August","September","October","November","December"],"wgRequestId":"9d346345-0ee9-4e04-881f-0482100d33e2","wgCanonicalNamespace":"","wgCanonicalSpecialPageName":false,"wgNamespaceNumber":0,"wgPageName":"Robots.txt","wgTitle":"Robots.txt","wgCurRevisionId":1370001604,"wgRevisionId":1370001604,"wgArticleId":101673,"wgIsArticle":true,"wgIsRedirect":false,"wgAction":"view","wgUserName":null,"wgUserGroups":["*"],"wgCategories":["Articles with short description","Short description is different from Wikidata","Wikipedia pending changes protected pages","CS1 maint: url-status","Search engine optimization","Websites","Web scraping","Text files","Internet bots"],"wgPageViewLanguage":"en","wgPageContentLanguage":"en","wgPageContentModel":"wikitext","wgRelevantPageName":"Robots.txt","wgRelevantArticleId":101673,"wgTempUserName":null,"wgIsProbablyEditable":true,"wgRelevantPageIsProbablyEditable":true,"wgRestrictionEdit":[],"wgRestrictionMove":[],"wgNoticeProject":"wikipedia","wgFlaggedRevsParams":{"tags":{"status":{"levels":1}},"namespaces":[0,4]},"wgStableRevisionId":1370001604,"wgConfirmEditCaptchaNeededForGenericEdit":"hcaptcha","wgConfirmEditForceShowCaptcha":false,"wgConfirmEditHCaptchaSiteKey":"5d0c670e-a5f4-4258-ad16-1f42792c9c62","wgMediaViewerOnClick":true,"wgMediaViewerEnabledByDefault":true,"wgMediaViewerMobileBeta":false,"wgPopupsFlags":0,"wgVisualEditor":{"pageLanguageCode":"en","pageLanguageDir":"ltr","pageVariantFallbacks":"en"},"wgMFDisplayWikibaseDescriptions":{"search":true,"watchlist":true,"tagline":false,"nearby":true},"wgWMESchemaEditAttemptStepOversample":false,"wgWMEPageLength":40000,"wgParsoidHtmlVersion":"2.8.0","parsermigration-parsoid":true,"wgPageAssessments":{"Computing":{"class":"C","importance":"Mid"},"Internet":{"class":"C","importance":"High"},"Project-independent assessment":{"class":"C","importance":""}},"wgTestKitchenUserExperiments":{"overrides":[],"enrolled":[],"assigned":[],"subject_ids":[]},"wgEditSubmitButtonLabelPublish":true,"wgVisualEditorPageIsDisambiguation":false,"wgULSPosition":"interlanguage","wgULSisCompactLinksEnabled":false,"wgVector2022LanguageInHeader":true,"wgULSisLanguageSelectorEmpty":false,"wgULSLanguageSelectorV2Enabled":true,"wgWikibaseItemId":"Q80776","wgCheckUserClientHintsHeadersJsApi":["brands","architecture","bitness","fullVersionList","mobile","model","platform","platformVersion"],"wgPersonalDashboardMenuVisible":false};
RLSTATE={"ext.globalCssJs.user.styles":"ready","site.styles":"ready","user.styles":"ready","ext.globalCssJs.user":"ready","user":"ready","user.options":"loading","ext.pygments":"ready","ext.cite.parsoid.styles":"ready","ext.cite.styles":"ready","mediawiki.skinning.content.parsoid":"ready","ext.wikimediamessages.styles":"ready","skins.vector.search.codex.styles":"ready","skins.vector.styles":"ready","skins.vector.icons":"ready","ext.flaggedRevs.basic":"ready","mediawiki.codex.messagebox.styles":"ready","ext.visualEditor.desktopArticleTarget.noscript":"ready","ext.uls.interlanguage":"ready","wikibase.client.init":"ready"};RLPAGEMODULES=["ext.pygments.view","ext.cite.ux-enhancements","site","mediawiki.page.ready","mediawiki.toc","skins.vector.js","ext.centralNotice.geoIP","ext.centralNotice.startUp","ext.flaggedRevs.advanced","ext.gadget.ReferenceTooltips","ext.gadget.switcher","ext.urlShortener.toolbar","ext.centralauth.centralautologin","ext.testKitchen","mmv.bootstrap","ext.popups","ext.visualEditor.desktopArticleTarget.init","ext.echo.centralauth","ext.eventLogging","ext.wikimediaEvents","ext.navigationTiming","ext.uls.interface","ext.cx.eventlogging.campaigns","ext.cx.uls.quick.actions","wikibase.client.vector-2022","wikibase.databox.fromWikidata","ext.checkUser.clientHints","ext.quicksurveys.init"];</script>
<script>(RLQ=window.RLQ||[]).push(function(){mw.loader.impl(function(){return["user.options@12s5i",function($,jQuery,require,module){mw.user.tokens.set({"patrolToken":"+\\","watchToken":"+\\","csrfToken":"+\\"});
}];});});</script>
<link rel="stylesheet" href="/w/load.php?lang=en&modules=ext.cite.parsoid.styles%7Cext.cite.styles%7Cext.flaggedRevs.basic%7Cext.pygments%7Cext.uls.interlanguage%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediamessages.styles%7Cmediawiki.codex.messagebox.styles%7Cmediawiki.skinning.content.parsoid%7Cskins.vector.icons%2Cstyles%7Cskins.vector.search.codex.styles%7Cwikibase.client.init&only=styles&skin=vector-2022">
<script async="" src="/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector-2022"></script>
<meta name="ResourceLoaderDynamicStyles" content="">
<link rel="stylesheet" href="/w/load.php?lang=en&modules=site.styles&only=styles&skin=vector-2022">
<meta name="generator" content="MediaWiki 1.47.0-wmf.16">
<meta name="referrer" content="origin">
<meta name="referrer" content="origin-when-cross-origin">
<meta name="robots" content="max-image-preview:standard">
<meta name="format-detection" content="telephone=no">
<meta property="og:image" content="https://upload.wikimedia.org/wikipedia/commons/thumb/1/16/Robots_txt.svg/1280px-Robots_txt.svg.png?utm_source=en.wikipedia.org&utm_campaign=index&utm_content=thumbnail">
<meta property="og:image:width" content="1200">
<meta property="og:image:height" content="488">
<meta name="viewport" content="width=1120">
<meta property="og:title" content="robots.txt - Wikipedia">
<meta property="og:type" content="website">
<link rel="preconnect" href="//upload.wikimedia.org">
<link rel="alternate" type="application/x-wiki" title="Edit this page" href="/w/index.php?title=Robots.txt&action=edit">
<link rel="apple-touch-icon" href="/static/apple-touch/wikipedia.png">
<link rel="icon" href="/static/favicon/wikipedia.ico">
<link rel="search" type="application/opensearchdescription+xml" href="/w/rest.php/v1/search" title="Wikipedia (en)">
<link rel="EditURI" type="application/rsd+xml" href="//en.wikipedia.org/w/api.php?action=rsd">
<link rel="canonical" href="https://en.wikipedia.org/wiki/Robots.txt">
<link rel="license" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">
<link rel="alternate" type="application/atom+xml" title="Wikipedia Atom feed" href="/w/index.php?title=Special:RecentChanges&feed=atom">
<link rel="dns-prefetch" href="//meta.wikimedia.org" />
<link rel="dns-prefetch" href="auth.wikimedia.org">
</head>
<body class="skin--responsive skin-vector skin-vector-search-vue mediawiki ltr sitedir-ltr mw-hide-empty-elt ns-0 ns-subject mw-editable page-Robots_txt rootpage-Robots_txt skin-vector-2022 action-view">
<div id="mw-aria-live-region" class="mw-aria-live-region" aria-live="polite"></div><a class="mw-jump-link" href="#bodyContent">Jump to content</a>
<div class="vector-header-container">
<header class="vector-header mw-header no-font-mode-scale">
<div class="vector-header-start">
<nav class="vector-main-menu-landmark" aria-label="Site">
<div id="vector-main-menu-dropdown" class="vector-dropdown vector-main-menu-dropdown vector-button-flush-left vector-button-flush-right" title="Main menu" >
<input type="checkbox" id="vector-main-menu-dropdown-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-main-menu-dropdown" class="vector-dropdown-checkbox " aria-label="Main menu" >
<label id="vector-main-menu-dropdown-label" for="vector-main-menu-dropdown-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only " aria-hidden="true" ><span class="vector-icon mw-ui-icon-menu mw-ui-icon-wikimedia-menu cdx-button__icon"></span>
<span class="vector-dropdown-label-text">Main menu</span>
</label>
<div class="vector-dropdown-content">
<div id="vector-main-menu-unpinned-container" class="vector-unpinned-container">
<div id="vector-main-menu" class="vector-main-menu vector-pinnable-element">
<div
class="vector-pinnable-header vector-main-menu-pinnable-header vector-pinnable-header-unpinned"
data-feature-name="main-menu-pinned"
data-pinnable-element-id="vector-main-menu"
data-pinned-container-id="vector-main-menu-pinned-container"
data-unpinned-container-id="vector-main-menu-unpinned-container"
>
<div class="vector-pinnable-header-label">Main menu</div>
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-pin-button" data-event-name="pinnable-header.vector-main-menu.pin" aria-label="Move Main menu to sidebar">move to sidebar</button>
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-unpin-button" data-event-name="pinnable-header.vector-main-menu.unpin" aria-label="Hide Main menu">hide</button>
</div>
<div id="p-navigation" class="vector-menu mw-portlet mw-portlet-navigation" >
<div class="vector-menu-heading">
Navigation
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="n-mainpage-description" class="mw-list-item"><a href="/wiki/Main_Page" title="Visit the main page [z]" accesskey="z"><span>Main page</span></a></li><li id="n-contents" class="mw-list-item"><a href="/wiki/Wikipedia:Contents" title="Guides to browsing Wikipedia"><span>Contents</span></a></li><li id="n-currentevents" class="mw-list-item"><a href="/wiki/Portal:Current_events" title="Articles related to current events"><span>Current events</span></a></li><li id="n-randompage" class="mw-list-item"><a href="/wiki/Special:Random" title="Visit a randomly selected article [x]" accesskey="x"><span>Random article</span></a></li><li id="n-aboutsite" class="mw-list-item"><a href="/wiki/Wikipedia:About" title="Learn about Wikipedia and how it works"><span>About Wikipedia</span></a></li><li id="n-contactpage" class="mw-list-item"><a href="//en.wikipedia.org/wiki/Wikipedia:Contact_us" title="How to contact Wikipedia"><span>Contact us</span></a></li>
</ul>
</div>
</div>
<div id="p-interaction" class="vector-menu mw-portlet mw-portlet-interaction" >
<div class="vector-menu-heading">
Contribute
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="n-help" class="mw-list-item"><a href="/wiki/Help:Contents" title="Guidance on how to use and edit Wikipedia"><span>Help</span></a></li><li id="n-introduction" class="mw-list-item"><a href="/wiki/Help:Introduction" title="Learn how to edit Wikipedia"><span>Learn to edit</span></a></li><li id="n-portal" class="mw-list-item"><a href="/wiki/Wikipedia:Community_portal" title="The hub for editors"><span>Community portal</span></a></li><li id="n-recentchanges" class="mw-list-item"><a href="/wiki/Special:RecentChanges" title="A list of recent changes to Wikipedia [r]" accesskey="r"><span>Recent changes</span></a></li><li id="n-upload" class="mw-list-item"><a href="/wiki/Wikipedia:File_upload_wizard" title="Add images or other media for use on Wikipedia"><span>Upload file</span></a></li><li id="n-specialpages" class="mw-list-item"><a href="/wiki/Special:SpecialPages" title="A list of all special pages [q]" accesskey="q"><span>Special pages</span></a></li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
</nav>
<a href="/wiki/Main_Page" class="mw-logo">
<img class="mw-logo-icon" src="/static/images/icons/enwiki-25.svg" alt="" aria-hidden="true" height="50" width="50">
<span class="mw-logo-container skin-invert">
<img class="mw-logo-wordmark" alt="Wikipedia" src="/static/images/mobile/copyright/wikipedia-wordmark-en-25.svg" style="width: 8.75em; height: 1.375em;">
<img class="mw-logo-tagline" alt="The Free Encyclopedia" src="/static/images/mobile/copyright/wikipedia-tagline-en-25.svg" width="140" height="11" style="width: 8.75em; height: 0.6875em;">
</span>
</a>
</div>
<div class="vector-header-end">
<div id="p-search" role="search" class="vector-search-box-vue vector-search-box-collapses vector-search-box-show-thumbnail vector-search-box-auto-expand-width vector-search-box">
<a href="/wiki/Special:Search" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only search-toggle" title="Search Wikipedia [f]" accesskey="f"><span class="vector-icon mw-ui-icon-search mw-ui-icon-wikimedia-search cdx-button__icon"></span>
<span>Search</span>
</a>
<div class="vector-typeahead-search-container">
<div class="cdx-typeahead-search cdx-typeahead-search--show-thumbnail cdx-typeahead-search--auto-expand-width">
<form action="/w/index.php" id="searchform" class="cdx-search-input cdx-search-input--has-end-button">
<div id="simpleSearch" class="cdx-search-input__input-wrapper" data-search-loc="header-moved">
<div class="cdx-text-input cdx-text-input--has-start-icon">
<input
class="cdx-text-input__input mw-searchInput" autocomplete="off"
type="search" name="search" placeholder="Search Wikipedia" aria-label="Search Wikipedia" autocapitalize="none" spellcheck="false" title="Search Wikipedia [f]" accesskey="f" id="searchInput"
>
<span class="cdx-text-input__icon cdx-text-input__start-icon"></span>
</div>
<input type="hidden" name="title" value="Special:Search">
</div>
<button class="cdx-button cdx-search-input__end-button">Search</button>
</form>
</div>
</div>
</div>
<nav class="vector-user-links vector-user-links-wide" aria-label="Personal tools">
<div class="vector-user-links-main">
<div id="p-vector-user-menu-preferences" class="vector-menu mw-portlet emptyPortlet" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
</ul>
</div>
</div>
<div id="p-vector-user-menu-userpage" class="vector-menu mw-portlet emptyPortlet" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
</ul>
</div>
</div>
<nav class="vector-appearance-landmark" aria-label="Appearance">
<div id="vector-appearance-dropdown" class="vector-dropdown " title="Change the appearance of the page's font size, width, and color" >
<input type="checkbox" id="vector-appearance-dropdown-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-appearance-dropdown" class="vector-dropdown-checkbox " aria-label="Appearance" >
<label id="vector-appearance-dropdown-label" for="vector-appearance-dropdown-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only " aria-hidden="true" ><span class="vector-icon mw-ui-icon-appearance mw-ui-icon-wikimedia-appearance cdx-button__icon"></span>
<span class="vector-dropdown-label-text">Appearance</span>
</label>
<div class="vector-dropdown-content">
<div id="vector-appearance-unpinned-container" class="vector-unpinned-container">
</div>
</div>
</div>
</nav>
<div id="p-vector-user-menu-notifications" class="vector-menu mw-portlet emptyPortlet" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
</ul>
</div>
</div>
<div id="p-vector-user-menu-overflow" class="vector-menu mw-portlet" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="pt-sitesupport-2" class="mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="https://donate.wikimedia.org/?wmf_source=donate&wmf_medium=sidebar&wmf_campaign=en.wikipedia.org&uselang=en" data-mw-donate-attempt="top_link" class=""><span>Donate</span></a>
</li>
<li id="pt-createaccount-2" class="mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="/w/index.php?title=Special:CreateAccount&returnto=Robots.txt" title="You are encouraged to create an account and log in; however, it is not mandatory" class=""><span>Create account</span></a>
</li>
<li id="pt-login-2" class="mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="/w/index.php?title=Special:UserLogin&returnto=Robots.txt" title="You're encouraged to log in; however, it's not mandatory. [o]" accesskey="o" class=""><span>Log in</span></a>
</li>
</ul>
</div>
</div>
</div>
<div id="vector-user-links-dropdown" class="vector-dropdown vector-user-menu vector-button-flush-right vector-user-menu-logged-out vector-user-links-dropdown--collapsible" title="Log in and more options" >
<input type="checkbox" id="vector-user-links-dropdown-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-user-links-dropdown" class="vector-dropdown-checkbox " aria-label="Personal tools" >
<label id="vector-user-links-dropdown-label" for="vector-user-links-dropdown-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only " aria-hidden="true" ><span class="vector-icon mw-ui-icon-userAvatar mw-ui-icon-wikimedia-userAvatar cdx-button__icon"></span>
<span class="vector-dropdown-label-text">Personal tools</span>
</label>
<div class="vector-dropdown-content">
<div id="p-personal" class="vector-menu mw-portlet mw-portlet-personal" title="User menu" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="pt-sitesupport" class="mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="https://donate.wikimedia.org/?wmf_source=donate&wmf_medium=sidebar&wmf_campaign=en.wikipedia.org&uselang=en" data-mw-donate-attempt="top_link" class=""><span class="vector-icon mw-ui-icon-heart mw-ui-icon-wikimedia-heart cdx-button__icon"></span>
<span>Donate</span></a>
</li>
<li id="pt-createaccount" class="mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="/w/index.php?title=Special:CreateAccount&returnto=Robots.txt" title="You are encouraged to create an account and log in; however, it is not mandatory" class=""><span class="vector-icon mw-ui-icon-userAvatar mw-ui-icon-wikimedia-userAvatar cdx-button__icon"></span>
<span>Create account</span></a>
</li>
<li id="pt-login" class="mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="/w/index.php?title=Special:UserLogin&returnto=Robots.txt" title="You're encouraged to log in; however, it's not mandatory. [o]" accesskey="o" class=""><span class="vector-icon mw-ui-icon-logIn mw-ui-icon-wikimedia-logIn cdx-button__icon"></span>
<span>Log in</span></a>
</li>
</ul>
</div>
</div>
</div>
</div>
</nav>
</div>
</header>
</div>
<div class="mw-page-container">
<div class="mw-page-container-inner">
<div class="vector-sitenotice-container">
<div id="siteNotice"><!-- CentralNotice --></div>
</div>
<div class="vector-column-start">
<div class="vector-main-menu-container">
<div id="mw-navigation">
<nav id="mw-panel" class="vector-main-menu-landmark" aria-label="Site">
<div id="vector-main-menu-pinned-container" class="vector-pinned-container">
</div>
</nav>
</div>
</div>
<div class="vector-sticky-pinned-container">
<nav id="mw-panel-toc" aria-label="Contents" data-event-name="ui.sidebar-toc" class="mw-table-of-contents-container vector-toc-landmark">
<div id="vector-toc-pinned-container" class="vector-pinned-container">
<div id="vector-toc" class="vector-toc vector-pinnable-element">
<div
class="vector-pinnable-header vector-toc-pinnable-header vector-pinnable-header-pinned"
data-feature-name="toc-pinned"
data-pinnable-element-id="vector-toc"
data-pinned-container-id="vector-toc-pinned-container"
data-unpinned-container-id="vector-toc-unpinned-container"
>
<h2 class="vector-pinnable-header-label">Contents</h2>
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-pin-button" data-event-name="pinnable-header.vector-toc.pin" aria-label="Move Contents to sidebar">move to sidebar</button>
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-unpin-button" data-event-name="pinnable-header.vector-toc.unpin" aria-label="Hide Contents">hide</button>
</div>
<ul class="vector-toc-contents" id="mw-panel-toc-list">
<li id="toc-mw-content-text"
class="vector-toc-list-item vector-toc-level-1">
<a href="#" class="vector-toc-link">
<div class="vector-toc-text">(Top)</div>
</a>
</li>
<li id="toc-History"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#History">
<div class="vector-toc-text">
<span class="vector-toc-numb">1</span>
<span>History</span>
</div>
</a>
<ul id="toc-History-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Standard"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Standard">
<div class="vector-toc-text">
<span class="vector-toc-numb">2</span>
<span>Standard</span>
</div>
</a>
<ul id="toc-Standard-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Compliance"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Compliance">
<div class="vector-toc-text">
<span class="vector-toc-numb">3</span>
<span>Compliance</span>
</div>
</a>
<button aria-controls="toc-Compliance-sublist" class="cdx-button cdx-button--weight-quiet cdx-button--icon-only vector-toc-toggle">
<span class="vector-icon mw-ui-icon-wikimedia-expand"></span>
<span>Toggle Compliance subsection</span>
</button>
<ul id="toc-Compliance-sublist" class="vector-toc-list">
<li id="toc-Search_engines"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#Search_engines">
<div class="vector-toc-text">
<span class="vector-toc-numb">3.1</span>
<span>Search engines</span>
</div>
</a>
<ul id="toc-Search_engines-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Archival_sites"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#Archival_sites">
<div class="vector-toc-text">
<span class="vector-toc-numb">3.2</span>
<span>Archival sites</span>
</div>
</a>
<ul id="toc-Archival_sites-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Artificial_intelligence"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#Artificial_intelligence">
<div class="vector-toc-text">
<span class="vector-toc-numb">3.3</span>
<span>Artificial intelligence</span>
</div>
</a>
<ul id="toc-Artificial_intelligence-sublist" class="vector-toc-list">
</ul>
</li>
</ul>
</li>
<li id="toc-Security"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Security">
<div class="vector-toc-text">
<span class="vector-toc-numb">4</span>
<span>Security</span>
</div>
</a>
<ul id="toc-Security-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Alternatives"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Alternatives">
<div class="vector-toc-text">
<span class="vector-toc-numb">5</span>
<span>Alternatives</span>
</div>
</a>
<ul id="toc-Alternatives-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Examples"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Examples">
<div class="vector-toc-text">
<span class="vector-toc-numb">6</span>
<span>Examples</span>
</div>
</a>
<button aria-controls="toc-Examples-sublist" class="cdx-button cdx-button--weight-quiet cdx-button--icon-only vector-toc-toggle">
<span class="vector-icon mw-ui-icon-wikimedia-expand"></span>
<span>Toggle Examples subsection</span>
</button>
<ul id="toc-Examples-sublist" class="vector-toc-list">
<li id="toc-The_use_of_the_wildcard_*_in_rules"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#The_use_of_the_wildcard_*_in_rules">
<div class="vector-toc-text">
<span class="vector-toc-numb">6.1</span>
<span>The use of the wildcard * in rules</span>
</div>
</a>
<ul id="toc-The_use_of_the_wildcard_*_in_rules-sublist" class="vector-toc-list">
</ul>
</li>
</ul>
</li>
<li id="toc-Nonstandard_extensions"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Nonstandard_extensions">
<div class="vector-toc-text">
<span class="vector-toc-numb">7</span>
<span>Nonstandard extensions</span>
</div>
</a>
<button aria-controls="toc-Nonstandard_extensions-sublist" class="cdx-button cdx-button--weight-quiet cdx-button--icon-only vector-toc-toggle">
<span class="vector-icon mw-ui-icon-wikimedia-expand"></span>
<span>Toggle Nonstandard extensions subsection</span>
</button>
<ul id="toc-Nonstandard_extensions-sublist" class="vector-toc-list">
<li id="toc-Crawl-delay_directive"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#Crawl-delay_directive">
<div class="vector-toc-text">
<span class="vector-toc-numb">7.1</span>
<span>Crawl-delay directive</span>
</div>
</a>
<ul id="toc-Crawl-delay_directive-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Sitemap"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#Sitemap">
<div class="vector-toc-text">
<span class="vector-toc-numb">7.2</span>
<span>Sitemap</span>
</div>
</a>
<ul id="toc-Sitemap-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Universal_"*"_match"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#Universal_"*"_match">
<div class="vector-toc-text">
<span class="vector-toc-numb">7.3</span>
<span>Universal "*" match</span>
</div>
</a>
<ul id="toc-Universal_"*"_match-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Content-Signal"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#Content-Signal">
<div class="vector-toc-text">
<span class="vector-toc-numb">7.4</span>
<span>Content-Signal</span>
</div>
</a>
<ul id="toc-Content-Signal-sublist" class="vector-toc-list">
</ul>
</li>
</ul>
</li>
<li id="toc-Meta_tags_and_headers"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Meta_tags_and_headers">
<div class="vector-toc-text">
<span class="vector-toc-numb">8</span>
<span>Meta tags and headers</span>
</div>
</a>
<button aria-controls="toc-Meta_tags_and_headers-sublist" class="cdx-button cdx-button--weight-quiet cdx-button--icon-only vector-toc-toggle">
<span class="vector-icon mw-ui-icon-wikimedia-expand"></span>
<span>Toggle Meta tags and headers subsection</span>
</button>
<ul id="toc-Meta_tags_and_headers-sublist" class="vector-toc-list">
<li id="toc-A_"noindex"_meta_tag"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#A_"noindex"_meta_tag">
<div class="vector-toc-text">
<span class="vector-toc-numb">8.1</span>
<span>A "noindex" meta tag</span>
</div>
</a>
<ul id="toc-A_"noindex"_meta_tag-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-A_"noindex"_HTTP_response_header"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#A_"noindex"_HTTP_response_header">
<div class="vector-toc-text">
<span class="vector-toc-numb">8.2</span>
<span>A "noindex" HTTP response header</span>
</div>
</a>
<ul id="toc-A_"noindex"_HTTP_response_header-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Maximum_size_of_a_robots.txt_file"
class="vector-toc-list-item vector-toc-level-2">
<a class="vector-toc-link" href="#Maximum_size_of_a_robots.txt_file">
<div class="vector-toc-text">
<span class="vector-toc-numb">8.3</span>
<span>Maximum size of a robots.txt file</span>
</div>
</a>
<ul id="toc-Maximum_size_of_a_robots.txt_file-sublist" class="vector-toc-list">
</ul>
</li>
</ul>
</li>
<li id="toc-See_also"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#See_also">
<div class="vector-toc-text">
<span class="vector-toc-numb">9</span>
<span>See also</span>
</div>
</a>
<ul id="toc-See_also-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-References"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#References">
<div class="vector-toc-text">
<span class="vector-toc-numb">10</span>
<span>References</span>
</div>
</a>
<ul id="toc-References-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-Further_reading"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#Further_reading">
<div class="vector-toc-text">
<span class="vector-toc-numb">11</span>
<span>Further reading</span>
</div>
</a>
<ul id="toc-Further_reading-sublist" class="vector-toc-list">
</ul>
</li>
<li id="toc-External_links"
class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
<a class="vector-toc-link" href="#External_links">
<div class="vector-toc-text">
<span class="vector-toc-numb">12</span>
<span>External links</span>
</div>
</a>
<ul id="toc-External_links-sublist" class="vector-toc-list">
</ul>
</li>
</ul>
</div>
</div>
</nav>
</div>
</div>
<div class="mw-content-container">
<main id="content" class="mw-body">
<header class="mw-body-header vector-page-titlebar no-font-mode-scale">
<nav aria-label="Contents" class="vector-toc-landmark">
<div id="vector-page-titlebar-toc" class="vector-dropdown vector-page-titlebar-toc vector-button-flush-left" title="Table of Contents" >
<input type="checkbox" id="vector-page-titlebar-toc-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-page-titlebar-toc" class="vector-dropdown-checkbox " aria-label="Toggle the table of contents" >
<label id="vector-page-titlebar-toc-label" for="vector-page-titlebar-toc-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only " aria-hidden="true" ><span class="vector-icon mw-ui-icon-listBullet mw-ui-icon-wikimedia-listBullet cdx-button__icon"></span>
<span class="vector-dropdown-label-text">Toggle the table of contents</span>
</label>
<div class="vector-dropdown-content">
<div id="vector-page-titlebar-toc-unpinned-container" class="vector-unpinned-container">
</div>
</div>
</div>
</nav>
<h1 id="firstHeading" class="firstHeading mw-first-heading">robots.txt</h1>
<div id="p-lang-btn" class="vector-dropdown mw-portlet mw-portlet-lang" >
<input type="checkbox" id="p-lang-btn-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-p-lang-btn" class="vector-dropdown-checkbox mw-interlanguage-selector" aria-label="28 languages" aria-description="Go to an article in another language. Available in 28 languages" >
<label id="p-lang-btn-label" for="p-lang-btn-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--action-progressive mw-portlet-lang-heading-28" aria-hidden="true" ><span class="vector-icon mw-ui-icon-language-progressive mw-ui-icon-wikimedia-language-progressive cdx-button__icon"></span>
<span class="vector-dropdown-label-text">28 languages</span>
</label>
<div class="vector-dropdown-content">
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li class="interlanguage-link interwiki-ar mw-list-item"><a href="https://ar.wikipedia.org/wiki/%D9%85%D8%B9%D9%8A%D8%A7%D8%B1_%D8%A7%D8%B3%D8%AA%D8%A8%D8%B9%D8%A7%D8%AF_%D8%A7%D9%84%D8%B1%D9%88%D8%A8%D9%88%D8%AA%D8%A7%D8%AA" title="معيار استبعاد الروبوتات – Arabic" lang="ar" hreflang="ar" data-title="معيار استبعاد الروبوتات" data-language-autonym="العربية" data-language-local-name="Arabic" class="interlanguage-link-target"><span>العربية</span></a></li><li class="interlanguage-link interwiki-az mw-list-item"><a href="https://az.wikipedia.org/wiki/Robots.txt" title="Robots.txt – Azerbaijani" lang="az" hreflang="az" data-title="Robots.txt" data-language-autonym="Azərbaycanca" data-language-local-name="Azerbaijani" class="interlanguage-link-target"><span>Azərbaycanca</span></a></li><li class="interlanguage-link interwiki-bar mw-list-item"><a href="https://bar.wikipedia.org/wiki/Robots_Exclusion_Standard" title="Robots Exclusion Standard – Bavarian" lang="bar" hreflang="bar" data-title="Robots Exclusion Standard" data-language-autonym="Boarisch" data-language-local-name="Bavarian" class="interlanguage-link-target"><span>Boarisch</span></a></li><li class="interlanguage-link interwiki-ca mw-list-item"><a href="https://ca.wikipedia.org/wiki/Protocol_d%27exclusi%C3%B3_de_robots" title="Protocol d'exclusió de robots – Catalan" lang="ca" hreflang="ca" data-title="Protocol d'exclusió de robots" data-language-autonym="Català" data-language-local-name="Catalan" class="interlanguage-link-target"><span>Català</span></a></li><li class="interlanguage-link interwiki-cs mw-list-item"><a href="https://cs.wikipedia.org/wiki/Robots.txt" title="Robots.txt – Czech" lang="cs" hreflang="cs" data-title="Robots.txt" data-language-autonym="Čeština" data-language-local-name="Czech" class="interlanguage-link-target"><span>Čeština</span></a></li><li class="interlanguage-link interwiki-da mw-list-item"><a href="https://da.wikipedia.org/wiki/Robot_Exclusion_Standard" title="Robot Exclusion Standard – Danish" lang="da" hreflang="da" data-title="Robot Exclusion Standard" data-language-autonym="Dansk" data-language-local-name="Danish" class="interlanguage-link-target"><span>Dansk</span></a></li><li class="interlanguage-link interwiki-de mw-list-item"><a href="https://de.wikipedia.org/wiki/Robots_Exclusion_Standard" title="Robots Exclusion Standard – German" lang="de" hreflang="de" data-title="Robots Exclusion Standard" data-language-autonym="Deutsch" data-language-local-name="German" class="interlanguage-link-target"><span>Deutsch</span></a></li><li class="interlanguage-link interwiki-es mw-list-item"><a href="https://es.wikipedia.org/wiki/Est%C3%A1ndar_de_exclusi%C3%B3n_de_robots" title="Estándar de exclusión de robots – Spanish" lang="es" hreflang="es" data-title="Estándar de exclusión de robots" data-language-autonym="Español" data-language-local-name="Spanish" class="interlanguage-link-target"><span>Español</span></a></li><li class="interlanguage-link interwiki-fa mw-list-item"><a href="https://fa.wikipedia.org/wiki/%D8%A7%D8%B3%D8%AA%D8%A7%D9%86%D8%AF%D8%A7%D8%B1%D8%AF_%D8%A7%D8%B3%D8%AA%D8%AB%D9%86%D8%A7%D8%A1_%DA%A9%D8%B1%D8%AF%D9%86_%D8%B1%D8%A8%D8%A7%D8%AA%E2%80%8C%D9%87%D8%A7" title="استاندارد استثناء کردن رباتها – Persian" lang="fa" hreflang="fa" data-title="استاندارد استثناء کردن رباتها" data-language-autonym="فارسی" data-language-local-name="Persian" class="interlanguage-link-target"><span>فارسی</span></a></li><li class="interlanguage-link interwiki-fi mw-list-item"><a href="https://fi.wikipedia.org/wiki/Robots.txt" title="Robots.txt – Finnish" lang="fi" hreflang="fi" data-title="Robots.txt" data-language-autonym="Suomi" data-language-local-name="Finnish" class="interlanguage-link-target"><span>Suomi</span></a></li><li class="interlanguage-link interwiki-fr mw-list-item"><a href="https://fr.wikipedia.org/wiki/Protocole_d%27exclusion_des_robots" title="Protocole d'exclusion des robots – French" lang="fr" hreflang="fr" data-title="Protocole d'exclusion des robots" data-language-autonym="Français" data-language-local-name="French" class="interlanguage-link-target"><span>Français</span></a></li><li class="interlanguage-link interwiki-gl mw-list-item"><a href="https://gl.wikipedia.org/wiki/Robots.txt" title="Robots.txt – Galician" lang="gl" hreflang="gl" data-title="Robots.txt" data-language-autonym="Galego" data-language-local-name="Galician" class="interlanguage-link-target"><span>Galego</span></a></li><li class="interlanguage-link interwiki-he mw-list-item"><a href="https://he.wikipedia.org/wiki/%D7%A4%D7%A8%D7%95%D7%98%D7%95%D7%A7%D7%95%D7%9C_%D7%90%D7%99_%D7%94%D7%9B%D7%9C%D7%9C%D7%AA_%D7%A8%D7%95%D7%91%D7%95%D7%98%D7%99%D7%9D" title="פרוטוקול אי הכללת רובוטים – Hebrew" lang="he" hreflang="he" data-title="פרוטוקול אי הכללת רובוטים" data-language-autonym="עברית" data-language-local-name="Hebrew" class="interlanguage-link-target"><span>עברית</span></a></li><li class="interlanguage-link interwiki-id mw-list-item"><a href="https://id.wikipedia.org/wiki/Robots.txt" title="Robots.txt – Indonesian" lang="id" hreflang="id" data-title="Robots.txt" data-language-autonym="Bahasa Indonesia" data-language-local-name="Indonesian" class="interlanguage-link-target"><span>Bahasa Indonesia</span></a></li><li class="interlanguage-link interwiki-it mw-list-item"><a href="https://it.wikipedia.org/wiki/Protocollo_di_esclusione_robot" title="Protocollo di esclusione robot – Italian" lang="it" hreflang="it" data-title="Protocollo di esclusione robot" data-language-autonym="Italiano" data-language-local-name="Italian" class="interlanguage-link-target"><span>Italiano</span></a></li><li class="interlanguage-link interwiki-ja mw-list-item"><a href="https://ja.wikipedia.org/wiki/Robots_Exclusion_Standard" title="Robots Exclusion Standard – Japanese" lang="ja" hreflang="ja" data-title="Robots Exclusion Standard" data-language-autonym="日本語" data-language-local-name="Japanese" class="interlanguage-link-target"><span>日本語</span></a></li><li class="interlanguage-link interwiki-ko mw-list-item"><a href="https://ko.wikipedia.org/wiki/%EB%A1%9C%EB%B4%87_%EB%B0%B0%EC%A0%9C_%ED%91%9C%EC%A4%80" title="로봇 배제 표준 – Korean" lang="ko" hreflang="ko" data-title="로봇 배제 표준" data-language-autonym="한국어" data-language-local-name="Korean" class="interlanguage-link-target"><span>한국어</span></a></li><li class="interlanguage-link interwiki-nl mw-list-item"><a href="https://nl.wikipedia.org/wiki/Robots_Exclusion_Protocol" title="Robots Exclusion Protocol – Dutch" lang="nl" hreflang="nl" data-title="Robots Exclusion Protocol" data-language-autonym="Nederlands" data-language-local-name="Dutch" class="interlanguage-link-target"><span>Nederlands</span></a></li><li class="interlanguage-link interwiki-pl mw-list-item"><a href="https://pl.wikipedia.org/wiki/Robots_Exclusion_Protocol" title="Robots Exclusion Protocol – Polish" lang="pl" hreflang="pl" data-title="Robots Exclusion Protocol" data-language-autonym="Polski" data-language-local-name="Polish" class="interlanguage-link-target"><span>Polski</span></a></li><li class="interlanguage-link interwiki-pt mw-list-item"><a href="https://pt.wikipedia.org/wiki/Protocolo_de_exclus%C3%A3o_de_rob%C3%B4s" title="Protocolo de exclusão de robôs – Portuguese" lang="pt" hreflang="pt" data-title="Protocolo de exclusão de robôs" data-language-autonym="Português" data-language-local-name="Portuguese" class="interlanguage-link-target"><span>Português</span></a></li><li class="interlanguage-link interwiki-ru mw-list-item"><a href="https://ru.wikipedia.org/wiki/%D0%A1%D1%82%D0%B0%D0%BD%D0%B4%D0%B0%D1%80%D1%82_%D0%B8%D1%81%D0%BA%D0%BB%D1%8E%D1%87%D0%B5%D0%BD%D0%B8%D0%B9_%D0%B4%D0%BB%D1%8F_%D1%80%D0%BE%D0%B1%D0%BE%D1%82%D0%BE%D0%B2" title="Стандарт исключений для роботов – Russian" lang="ru" hreflang="ru" data-title="Стандарт исключений для роботов" data-language-autonym="Русский" data-language-local-name="Russian" class="interlanguage-link-target"><span>Русский</span></a></li><li class="interlanguage-link interwiki-simple mw-list-item"><a href="https://simple.wikipedia.org/wiki/Robots_exclusion_standard" title="Robots exclusion standard – Simple English" lang="en-simple" hreflang="en-simple" data-title="Robots exclusion standard" data-language-autonym="Simple English" data-language-local-name="Simple English" class="interlanguage-link-target"><span>Simple English</span></a></li><li class="interlanguage-link interwiki-sr mw-list-item"><a href="https://sr.wikipedia.org/wiki/Standard_isklju%C4%8Denja_robota" title="Standard isključenja robota – Serbian" lang="sr" hreflang="sr" data-title="Standard isključenja robota" data-language-autonym="Српски / srpski" data-language-local-name="Serbian" class="interlanguage-link-target"><span>Српски / srpski</span></a></li><li class="interlanguage-link interwiki-sv mw-list-item"><a href="https://sv.wikipedia.org/wiki/Robots_Exclusion_Standard" title="Robots Exclusion Standard – Swedish" lang="sv" hreflang="sv" data-title="Robots Exclusion Standard" data-language-autonym="Svenska" data-language-local-name="Swedish" class="interlanguage-link-target"><span>Svenska</span></a></li><li class="interlanguage-link interwiki-th mw-list-item"><a href="https://th.wikipedia.org/wiki/Robots.txt" title="Robots.txt – Thai" lang="th" hreflang="th" data-title="Robots.txt" data-language-autonym="ไทย" data-language-local-name="Thai" class="interlanguage-link-target"><span>ไทย</span></a></li><li class="interlanguage-link interwiki-tr mw-list-item"><a href="https://tr.wikipedia.org/wiki/Robot_engelleme_standard%C4%B1" title="Robot engelleme standardı – Turkish" lang="tr" hreflang="tr" data-title="Robot engelleme standardı" data-language-autonym="Türkçe" data-language-local-name="Turkish" class="interlanguage-link-target"><span>Türkçe</span></a></li><li class="interlanguage-link interwiki-uk mw-list-item"><a href="https://uk.wikipedia.org/wiki/Robots.txt" title="Robots.txt – Ukrainian" lang="uk" hreflang="uk" data-title="Robots.txt" data-language-autonym="Українська" data-language-local-name="Ukrainian" class="interlanguage-link-target"><span>Українська</span></a></li><li class="interlanguage-link interwiki-zh mw-list-item"><a href="https://zh.wikipedia.org/wiki/Robots.txt" title="Robots.txt – Chinese" lang="zh" hreflang="zh" data-title="Robots.txt" data-language-autonym="中文" data-language-local-name="Chinese" class="interlanguage-link-target"><span>中文</span></a></li>
</ul>
<div class="after-portlet after-portlet-lang"><span class="wb-langlinks-edit wb-langlinks-link"><a href="https://www.wikidata.org/wiki/Special:EntityPage/Q80776#sitelinks-wikipedia" title="Edit interlanguage links" class="wbc-editpage">Edit links</a></span></div>
</div>
</div>
</div>
</header>
<div class="vector-page-toolbar">
<div class="vector-page-toolbar-container">
<div id="left-navigation">
<nav aria-label="Namespaces">
<div id="p-associated-pages" class="vector-menu vector-menu-tabs mw-portlet mw-portlet-associated-pages" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="ca-nstab-main" class="selected mw-list-item vector-tab-noicon"><a data-mw-interface href="/wiki/Robots.txt" title="View the content page [c]" accesskey="c" class=""><span>Article</span></a>
</li>
<li id="ca-talk" class="mw-list-item vector-tab-noicon"><a data-mw-interface href="/wiki/Talk:Robots.txt" rel="discussion" title="Discuss improvements to the content page [t]" accesskey="t" class=""><span>Talk</span></a>
</li>
</ul>
</div>
</div>
<div id="vector-variants-dropdown" class="vector-dropdown emptyPortlet" >
<input type="checkbox" id="vector-variants-dropdown-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-variants-dropdown" class="vector-dropdown-checkbox " aria-label="Change language variant" >
<label id="vector-variants-dropdown-label" for="vector-variants-dropdown-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet" aria-hidden="true" ><span class="vector-dropdown-label-text">English</span>
</label>
<div class="vector-dropdown-content">
<div id="p-variants" class="vector-menu mw-portlet mw-portlet-variants emptyPortlet" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
</ul>
</div>
</div>
</div>
</div>
</nav>
</div>
<div id="right-navigation" class="vector-collapsible">
<nav aria-label="Views">
<div id="p-views" class="vector-menu vector-menu-tabs mw-portlet mw-portlet-views" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="ca-view" class="selected mw-list-item vector-menu-item--collapsible vector-tab-noicon"><a data-mw-interface href="/wiki/Robots.txt" class=""><span>Read</span></a>
</li>
<li id="ca-edit" class="mw-list-item vector-menu-item--collapsible vector-tab-noicon"><a data-mw-interface href="/w/index.php?title=Robots.txt&action=edit" title="Edit this page [e]" accesskey="e" class=""><span>Edit</span></a>
</li>
<li id="ca-history" class="mw-list-item vector-menu-item--collapsible vector-tab-noicon"><a data-mw-interface href="/w/index.php?title=Robots.txt&action=history" title="Past revisions of this page [h]" accesskey="h" class=""><span>View history</span></a>
</li>
</ul>
</div>
</div>
</nav>
<nav class="vector-page-tools-landmark" aria-label="Page tools">
<div id="vector-page-tools-dropdown" class="vector-dropdown vector-page-tools-dropdown" >
<input type="checkbox" id="vector-page-tools-dropdown-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-page-tools-dropdown" class="vector-dropdown-checkbox " aria-label="Tools" >
<label id="vector-page-tools-dropdown-label" for="vector-page-tools-dropdown-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only " aria-hidden="true" ><span class="vector-icon mw-ui-icon-verticalEllipsis mw-ui-icon-wikimedia-verticalEllipsis cdx-button__icon"></span>
<span class="vector-dropdown-label-text">Tools</span>
</label>
<div class="vector-dropdown-content">
<div id="vector-page-tools-unpinned-container" class="vector-unpinned-container">
<div id="vector-page-tools" class="vector-page-tools vector-pinnable-element">
<div
class="vector-pinnable-header vector-page-tools-pinnable-header vector-pinnable-header-unpinned"
data-feature-name="page-tools-pinned"
data-pinnable-element-id="vector-page-tools"
data-pinned-container-id="vector-page-tools-pinned-container"
data-unpinned-container-id="vector-page-tools-unpinned-container"
>
<div class="vector-pinnable-header-label">Tools</div>
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-pin-button" data-event-name="pinnable-header.vector-page-tools.pin" aria-label="Move Tools to sidebar">move to sidebar</button>
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-unpin-button" data-event-name="pinnable-header.vector-page-tools.unpin" aria-label="Hide Tools">hide</button>
</div>
<div id="p-cactions" class="vector-menu mw-portlet mw-portlet-cactions emptyPortlet vector-has-collapsible-items" >
<div class="vector-menu-heading">
Actions
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="ca-view-more" class="selected mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="/wiki/Robots.txt" class=""><span class="vector-icon mw-ui-icon-eye mw-ui-icon-wikimedia-eye cdx-button__icon"></span>
<span>Read</span></a>
</li>
<li id="ca-edit-more" class="mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="/w/index.php?title=Robots.txt&action=edit" title="Edit this page [e]" accesskey="e" class=""><span class="vector-icon mw-ui-icon-edit mw-ui-icon-wikimedia-edit cdx-button__icon"></span>
<span>Edit</span></a>
</li>
<li id="ca-history-more" class="mw-list-item vector-menu-item--collapsible"><a data-mw-interface href="/w/index.php?title=Robots.txt&action=history" title="Past revisions of this page [h]" accesskey="h" class=""><span class="vector-icon mw-ui-icon-history mw-ui-icon-wikimedia-history cdx-button__icon"></span>
<span>View history</span></a>
</li>
</ul>
</div>
</div>
<div id="p-tb" class="vector-menu mw-portlet mw-portlet-tb" >
<div class="vector-menu-heading">
General
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="t-whatlinkshere" class="mw-list-item"><a href="/wiki/Special:WhatLinksHere/Robots.txt" title="List of all English Wikipedia pages containing links to this page [j]" accesskey="j"><span>What links here</span></a></li><li id="t-recentchangeslinked" class="mw-list-item"><a href="/wiki/Special:RecentChangesLinked/Robots.txt" rel="nofollow" title="Recent changes in pages linked from this page [k]" accesskey="k"><span>Related changes</span></a></li><li id="t-upload" class="mw-list-item"><a href="//en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard" title="Upload files [u]" accesskey="u"><span>Upload file</span></a></li><li id="t-permalink" class="mw-list-item"><a href="/w/index.php?title=Robots.txt&oldid=1370001604" title="Permanent link to this revision of this page"><span>Permanent link</span></a></li><li id="t-info" class="mw-list-item"><a href="/w/index.php?title=Robots.txt&action=info" title="More information about this page"><span>Page information</span></a></li><li id="t-cite" class="mw-list-item"><a href="/w/index.php?title=Special:CiteThisPage&page=Robots.txt&id=1370001604&wpFormIdentifier=titleform" title="Information on how to cite this page"><span>Cite this page</span></a></li><li id="t-urlshortener" class="mw-list-item"><a href="/w/index.php?title=Special:UrlShortener&url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FRobots.txt"><span>Get shortened URL</span></a></li><li class="mw-list-item"><a href="/w/index.php?title=Robots.txt&useparsoid=0"><span>Switch to legacy parser</span></a></li>
</ul>
</div>
</div>
<div id="p-coll-print_export" class="vector-menu mw-portlet mw-portlet-coll-print_export" >
<div class="vector-menu-heading">
Print/export
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li id="coll-download-as-rl" class="mw-list-item"><a href="/w/index.php?title=Special:DownloadAsPdf&page=Robots.txt&action=show-download-screen" title="Download this page as a PDF file"><span>Download as PDF</span></a></li><li id="t-print" class="mw-list-item"><a href="/w/index.php?title=Robots.txt&printable=yes" title="Printable version of this page [p]" accesskey="p"><span>Printable version</span></a></li>
</ul>
</div>
</div>
<div id="p-wikibase-otherprojects" class="vector-menu mw-portlet mw-portlet-wikibase-otherprojects" >
<div class="vector-menu-heading">
In other projects
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
<li class="wb-otherproject-link wb-otherproject-commons mw-list-item"><a href="https://commons.wikimedia.org/wiki/Category:Robots.txt" hreflang="en"><span>Wikimedia Commons</span></a></li><li id="t-wikibase" class="wb-otherproject-link wb-otherproject-wikibase-dataitem mw-list-item"><a href="https://www.wikidata.org/wiki/Special:EntityPage/Q80776" title="Structured data on this page hosted by Wikidata [g]" accesskey="g"><span>Wikidata item</span></a></li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
</nav>
</div>
</div>
</div>
<div class="vector-column-end no-font-mode-scale">
<div class="vector-sticky-pinned-container">
<nav class="vector-page-tools-landmark" aria-label="Page tools">
<div id="vector-page-tools-pinned-container" class="vector-pinned-container">
</div>
</nav>
<nav class="vector-appearance-landmark" aria-label="Appearance">
<div id="vector-appearance-pinned-container" class="vector-pinned-container">
<div id="vector-appearance" class="vector-appearance vector-pinnable-element">
<div
class="vector-pinnable-header vector-appearance-pinnable-header vector-pinnable-header-pinned"
data-feature-name="appearance-pinned"
data-pinnable-element-id="vector-appearance"
data-pinned-container-id="vector-appearance-pinned-container"
data-unpinned-container-id="vector-appearance-unpinned-container"
>
<div class="vector-pinnable-header-label">Appearance</div>
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-pin-button" data-event-name="pinnable-header.vector-appearance.pin" aria-label="Move Appearance to sidebar">move to sidebar</button>
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-unpin-button" data-event-name="pinnable-header.vector-appearance.unpin" aria-label="Hide Appearance">hide</button>
</div>
</div>
</div>
</nav>
</div>
</div>
<div id="bodyContent" class="vector-body" aria-labelledby="firstHeading" data-mw-ve-target-container>
<div class="vector-body-before-content">
<div class="mw-indicators">
<div id="mw-indicator-indicator-fr-review-status" class="mw-indicator"><div class="mw-fr-review-status-indicator" id="mw-fr-revision-toggle"><span class="cdx-fr-css-icon-review--status--stable"></span><b>Checked</b></div></div>
<div id="mw-indicator-pp-autoreview" class="mw-indicator"><div class="mw-parser-output"><span typeof="mw:File" data-mw='{"caption":"All edits by unregistered and new users are subject to review prior to becoming visible to unregistered users"}' id="mwAA"><a href="https://en.wikipedia.org/wiki/Wikipedia:Protection_policy#pending" title="All edits by unregistered and new users are subject to review prior to becoming visible to unregistered users" id="mwBQk"><img resource="https://en.wikipedia.org/wiki/File:Pending-protection-shackle.svg" src="//upload.wikimedia.org/wikipedia/en/thumb/b/b7/Pending-protection-shackle.svg/20px-Pending-protection-shackle.svg.png?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail" decoding="async" loading="lazy" srcset="//upload.wikimedia.org/wikipedia/en/thumb/b/b7/Pending-protection-shackle.svg/40px-Pending-protection-shackle.svg.png?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail 2x" alt="Page protected with pending changes" data-file-width="512" data-file-height="512" data-file-type="drawing" height="20" width="20" class="mw-file-element" id="mwBQo"/></a></span></div></div>
</div>
<div id="siteSub" class="noprint">From Wikipedia, the free encyclopedia</div>
</div>
<div id="contentSub"><div id="mw-content-subtitle"><div id="mw-fr-revision-messages"><div id="mw-fr-revision-details" class="mw-fr-revision-details-dialog" style="display:none;"><div tabindex="0"></div><div class="cdx-dialog cdx-dialog--horizontal-actions"><header class="cdx-dialog__header cdx-dialog__header--default"><div class="cdx-dialog__header__title-group"><h2 class="cdx-dialog__header__title">Page version status</h2><p class="cdx-dialog__header__subtitle">This is an accepted version of this page</p></div><button class="cdx-button cdx-button--action-default cdx-button--weight-quiet 							cdx-button--size-medium cdx-button--icon-only cdx-dialog__header__close-button" aria-label="Close" onclick="document.getElementById("mw-fr-revision-details").style.display = "none";" type="submit"><span class="cdx-icon cdx-icon--medium 							cdx-fr-css-icon--close"></span></button></header><div class="cdx-dialog__body">This is the <a href="/wiki/Wikipedia:Pending_changes" title="Wikipedia:Pending changes">latest accepted revision</a>, <a class="external text" data-mw-original-href="//en.wikipedia.org/w/index.php?title=Special:Log&type=review&page=Robots.txt" href="https://en.wikipedia.org/w/index.php?title=Special:Log&type=review&page=Robots.txt">reviewed</a> on <i>18 August 2026</i>.</div></div><div tabindex="0"></div></div></div></div></div>
<div id="mw-content-text" class="mw-body-content"><div class="mw-content-ltr mw-parser-output" lang="en" dir="ltr" data-mw-parsoid-version="0.24.0.0-alpha19" data-mw-html-version="2.8.0"><section data-mw-section-id="0" id="mwAQ"><div class="shortdescription nomobile noexcerpt noprint searchaux" style="display:none" about="#mwt1" typeof="mw:Transclusion" id="mwAg" data-mw='{"parts":[{"template":{"target":{"wt":"Short description","href":"./Template:Short_description"},"params":{"1":{"wt":"File indicating parts for web crawling"}},"i":0}}]}'>File indicating parts for web crawling</div><span class="mw-empty-elt" about="#mwt1"><link rel="mw:PageProp/Category" href="./Category:Articles_with_short_description"/><link rel="mw:PageProp/Category" href="./Category:Short_description_is_different_from_Wikidata"/></span>
<span about="#mwt2" typeof="mw:Transclusion" id="mwAw" data-mw='{"parts":[{"template":{"target":{"wt":"Lowercase title","href":"./Template:Lowercase_title"},"params":{},"i":0}}]}'></span>
<p class="mw-empty-elt" id="mwBA"><span typeof="mw:Nowiki mw:Transclusion" about="#mwt3" id="mwBQ" data-mw='{"parts":[{"template":{"target":{"wt":"Pp-pc ","href":"./Template:Pp-pc"},"params":{"small":{"wt":"yes"}},"i":0}}]}'></span><meta typeof="mw:Extension/indicator" about="#mwt3" id="mwBg" data-mw='{"name":"indicator","attrs":{"name":"pp-autoreview"},"body":{"extsrc":"[[File:Pending-protection-shackle.svg|20px|link=Wikipedia:Protection policy#pending|alt=Page protected with pending changes|All edits by unregistered and new users are subject to review prior to becoming visible to unregistered users]]"}}'/><link rel="mw:PageProp/Category" href="./Category:Wikipedia_pending_changes_protected_pages#Robots.txt" about="#mwt3" id="mwBw"/></p>
<span class="mw-empty-elt" about="#mwt5" typeof="mw:Transclusion" id="mwCA" data-mw='{"parts":[{"template":{"target":{"wt":"Infobox technology standard\n","href":"./Template:Infobox_technology_standard"},"params":{"title":{"wt":"robots.txt"},"long_name":{"wt":"Robots Exclusion Protocol"},"image":{"wt":"Robots txt.svg"},"image_size":{"wt":""},"alt":{"wt":"Picture of text, reads: User-agent: Mallorybot Disallow: / User-agent: * Crawl-delay: 20 Disallow: /secret/"},"caption":{"wt":"Example of a simple robots.txt file, indicating that a user-agent called \"Mallorybot\" is not allowed to crawl any of the website's pages, and that other user-agents cannot crawl more than one page every 20 seconds, and are not allowed to crawl the \"secret\" folder"},"class":{"wt":"skin-invert-image"},"abbreviation":{"wt":""},"native_name":{"wt":"<!-- Name in local language. If more than one, separate using {{plain list}} -->"},"native_name_lang":{"wt":"<!-- ISO 639-1 code e.g. \"fr\" for French. If more than one, use {{lang}} inside native_name items instead -->"},"status":{"wt":"Proposed Standard"},"year_started":{"wt":"<!-- {{Start date|YYYY|MM|DD|df=y}} -->"},"first_published":{"wt":"1994 published, formally standardized in 2022"},"version":{"wt":""},"version_date":{"wt":""},"preview":{"wt":""},"preview_date":{"wt":""},"organization":{"wt":""},"committee":{"wt":""},"series":{"wt":""},"editors":{"wt":""},"authors":{"wt":"{{plain list|\n* Martijn Koster (original author)\n* Gary Illyes, Henner Zeller, Lizzi Sassman (IETF contributors)\n}}"},"base_standards":{"wt":""},"related_standards":{"wt":""},"predecessor":{"wt":""},"successor":{"wt":""},"domain":{"wt":""},"license":{"wt":""},"copyright":{"wt":""},"website":{"wt":"{{URL|https://robotstxt.org}}, {{URL|https://datatracker.ietf.org/doc/html/rfc9309|RFC 9309}}"}},"i":0}}]}'><style data-mw-deduplicate="TemplateStyles:r1358620234" typeof="mw:Extension/templatestyles" about="#mwt6" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Infobox/styles.css"},"body":{"extsrc":""}}'>@media(min-width:640px){.mw-parser-output .infobox{margin-left:1em;float:right;clear:right;width:22em}}.mw-parser-output .infobox-subbox{padding:0;border:none;margin:-3px;width:auto;min-width:100%;font-size:100%;clear:none;float:none;background-color:transparent;color:inherit}.mw-parser-output .infobox-3cols-child{margin:-3px}.mw-parser-output .infobox .navbar{font-size:100%}.mw-parser-output .infobox-hiddenrow,body.skin--responsive.skin--responsive .mw-parser-output .infobox .infobox-hiddenrow{display:none}@media screen{html.skin-theme-clientpref-night .mw-parser-output .infobox-full-data:not(.notheme)>div:not(.notheme)[style]{background:#1f1f23!important;color:#f8f9fa}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .infobox-full-data:not(.notheme)>div:not(.notheme)[style]{background:#1f1f23!important;color:#f8f9fa}}@media(min-width:640px){body.skin--responsive .mw-parser-output .infobox-table{display:table!important}body.skin--responsive .mw-parser-output .infobox-table>caption{display:table-caption!important}body.skin--responsive .mw-parser-output .infobox-table>tbody{display:table-row-group}body.skin--responsive .mw-parser-output .infobox-table th,body.skin--responsive .mw-parser-output .infobox-table td{padding-left:inherit;padding-right:inherit}}</style></span><table class="infobox hproduct" about="#mwt5" id="mwCQ"><tbody><tr><th colspan="2" class="infobox-above" style="background-color:#e0e0e0;color:inherit;">robots.txt</th></tr><tr><td colspan="2" class="infobox-subheader">Robots Exclusion Protocol</td></tr><tr><td colspan="2" class="infobox-image"><span class="mw-default-size skin-invert-image" typeof="mw:File/Frameless"><a href="https://en.wikipedia.org/wiki/File:Robots_txt.svg" class="mw-file-description"><img resource="https://en.wikipedia.org/wiki/File:Robots_txt.svg" src="//upload.wikimedia.org/wikipedia/commons/thumb/1/16/Robots_txt.svg/250px-Robots_txt.svg.png?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail" decoding="async" loading="lazy" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/1/16/Robots_txt.svg/500px-Robots_txt.svg.png?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail 2x" alt="Picture of text, reads: User-agent: Mallorybot Disallow: / User-agent: * Crawl-delay: 20 Disallow: /secret/" data-file-width="864" data-file-height="351" data-file-type="drawing" height="102" width="250" class="mw-file-element"/></a></span><div class="infobox-caption">Example of a simple robots.txt file, indicating that a user-agent called "Mallorybot" is not allowed to crawl any of the website's pages, and that other user-agents cannot crawl more than one page every 20 seconds, and are not allowed to crawl the "secret" folder</div></td></tr><tr><th scope="row" class="infobox-label">Status</th><td class="infobox-data">Proposed Standard</td></tr><tr><th scope="row" class="infobox-label">First published</th><td class="infobox-data">1994 published, formally standardized in 2022</td></tr><tr><th scope="row" class="infobox-label">Authors</th><td class="infobox-data"><span class="mw-empty-elt"><style data-mw-deduplicate="TemplateStyles:r1126788409" typeof="mw:Extension/templatestyles" about="#mwt7" data-mw='{"name":"templatestyles","attrs":{"src":"Plainlist/styles.css"}}'>.mw-parser-output .plainlist ol,.mw-parser-output .plainlist ul{line-height:inherit;list-style:none;margin:0;padding:0}.mw-parser-output .plainlist ol li,.mw-parser-output .plainlist ul li{margin-bottom:0}</style></span><div class="plainlist ">
<ul><li>Martijn Koster (original author)</li>
<li>Gary Illyes, Henner Zeller, Lizzi Sassman (IETF contributors)</li></ul>
</div></td></tr><tr><th scope="row" class="infobox-label">Website</th><td class="infobox-data"><span class="url"><a rel="mw:ExtLink nofollow" href="https://robotstxt.org" class="external text">robotstxt<wbr/>.org</a></span>, <span class="url"><a rel="mw:ExtLink nofollow" href="https://datatracker.ietf.org/doc/html/rfc9309" class="external text">RFC 9309</a></span></td></tr></tbody></table>
<p id="mwCg">The <b id="mwCw">Robots Exclusion Protocol</b> (often referred to by the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Filename" title="Filename" id="mwDA">filename</a> used to implement it, <b id="mwDQ">robots.txt</b>) is a standard used by <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Website" title="Website" id="mwDg">websites</a> to indicate to visiting <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Web_crawler" title="Web crawler" id="mwDw">web crawlers</a> and other <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Internet_bot" title="Internet bot" id="mwEA">web robots</a> which portions of the website they are allowed to visit. </p>
<p id="mwEQ">The standard, developed in 1994, relies on <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Voluntary_compliance" title="Voluntary compliance" id="mwEg">voluntary compliance</a>. Malicious bots can use the file as a directory of which pages to visit, though standards bodies discourage countering this with <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Security_through_obscurity" title="Security through obscurity" id="mwEw">security through obscurity</a>. Some archival sites ignore robots.txt. The standard was used in the 1990s to mitigate <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Server_(computing)" title="Server (computing)" id="mwFA">server</a> overload. In the 2020s, websites began denying bots that collect information for <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Generative_AI" title="Generative AI" id="mwFQ">generative AI</a>.</p>
<p id="mwFg">The "robots.txt" file can be used in conjunction with <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Sitemaps" title="Sitemaps" id="mwFw">sitemaps</a>, another robot inclusion standard for websites.</p>
<meta property="mw:PageProp/toc" id="mwGA" data-mw='{"autoGenerated":true}'/></section><section data-mw-section-id="1" id="mwGQ" aria-labelledby="History"><div class="mw-heading mw-heading2"><h2 id="History">History</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=1" title="Edit section: History"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwGg">The standard was proposed by <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Martijn_Koster" title="Martijn Koster" id="mwGw">Martijn Koster</a>,<sup about="#mwt10" class="mw-ref reference" id="cite_ref-1" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-1"}}'><a href="#cite_note-1" id="mwHA"><span class="mw-reflink-text" id="mwHQ"><span class="cite-bracket" id="mwHg">[</span>1<span class="cite-bracket" id="mwHw">]</span></span></a></sup><sup about="#mwt13" class="mw-ref reference" id="cite_ref-2" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-2"}}'><a href="#cite_note-2" id="mwIA"><span class="mw-reflink-text" id="mwIQ"><span class="cite-bracket" id="mwIg">[</span>2<span class="cite-bracket" id="mwIw">]</span></span></a></sup> when working for <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Nexor" title="Nexor" id="mwJA">Nexor</a><sup about="#mwt16" class="mw-ref reference" id="cite_ref-:0_3-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":":0"},"body":{"id":"mw-reference-text-cite_note-:0-3"}}'><a href="#cite_note-:0-3" id="mwJQ"><span class="mw-reflink-text" id="mwJg"><span class="cite-bracket" id="mwJw">[</span>3<span class="cite-bracket" id="mwKA">]</span></span></a></sup> in February 1994<sup about="#mwt19" class="mw-ref reference" id="cite_ref-4" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-4"}}'><a href="#cite_note-4" id="mwKQ"><span class="mw-reflink-text" id="mwKg"><span class="cite-bracket" id="mwKw">[</span>4<span class="cite-bracket" id="mwLA">]</span></span></a></sup> on the <i id="mwLQ">www-talk</i> mailing list, the main communication channel for WWW-related activities at the time. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Charles_Stross" title="Charles Stross" id="mwLg">Charles Stross</a> claims to have provoked Koster to suggest robots.txt, after he wrote a badly behaved web crawler that inadvertently caused a <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Denial-of-service_attack" title="Denial-of-service attack" id="mwLw">denial-of-service attack</a> on Koster's server.<sup about="#mwt22" class="mw-ref reference" id="cite_ref-5" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-5"}}'><a href="#cite_note-5" id="mwMA"><span class="mw-reflink-text" id="mwMQ"><span class="cite-bracket" id="mwMg">[</span>5<span class="cite-bracket" id="mwMw">]</span></span></a></sup></p>
<p id="mwNA">The standard, initially RobotsNotWanted.txt, allowed <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Web_developer" title="Web developer" id="mwNQ">web developers</a> to specify which bots should not access their website or which pages bots should not access. The internet was small enough in 1994 to maintain a complete list of all bots; <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Server_(computing)" title="Server (computing)" id="mwNg">server</a> overload was a primary concern. By June 1994 it had become a <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/De_facto_standard" title="De facto standard" id="mwNw"><i id="mwOA">de facto</i> standard</a>;<sup about="#mwt25" class="mw-ref reference" id="cite_ref-Verge_6-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"Verge"},"body":{"id":"mw-reference-text-cite_note-Verge-6"}}'><a href="#cite_note-Verge-6" id="mwOQ"><span class="mw-reflink-text" id="mwOg"><span class="cite-bracket" id="mwOw">[</span>6<span class="cite-bracket" id="mwPA">]</span></span></a></sup> most complied, including those operated by search engines such as <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/WebCrawler" title="WebCrawler" id="mwPQ">WebCrawler</a>, <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Lycos" title="Lycos" id="mwPg">Lycos</a>, and <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/AltaVista" title="AltaVista" id="mwPw">AltaVista</a>.<sup about="#mwt28" class="mw-ref reference" id="cite_ref-sear_Robo_7-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"sear_Robo"},"body":{"id":"mw-reference-text-cite_note-sear_Robo-7"}}'><a href="#cite_note-sear_Robo-7" id="mwQA"><span class="mw-reflink-text" id="mwQQ"><span class="cite-bracket" id="mwQg">[</span>7<span class="cite-bracket" id="mwQw">]</span></span></a></sup></p>
<p id="mwRA">On July 1, 2019, Google announced the proposal of the Robots Exclusion Protocol as an official standard under <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Internet_Engineering_Task_Force" title="Internet Engineering Task Force" id="mwRQ">Internet Engineering Task Force</a>.<sup about="#mwt37" class="mw-ref reference" id="cite_ref-8" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-8"}}'><a href="#cite_note-8" id="mwRg"><span class="mw-reflink-text" id="mwRw"><span class="cite-bracket" id="mwSA">[</span>8<span class="cite-bracket" id="mwSQ">]</span></span></a></sup> A proposed standard<sup about="#mwt29" class="mw-ref reference" id="cite_ref-rfc9309_9-0" rel="dc:references" typeof="mw:Transclusion mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"rfc9309"},"body":{"id":"mw-reference-text-cite_note-rfc9309-9"},"parts":[{"template":{"target":{"wt":"Ref RFC","href":"./Template:Ref_RFC"},"params":{"1":{"wt":"9309"}},"i":0}}]}'><a href="#cite_note-rfc9309-9" id="mwSg"><span class="mw-reflink-text" id="mwSw"><span class="cite-bracket" id="mwTA">[</span>9<span class="cite-bracket" id="mwTQ">]</span></span></a></sup> was published in September 2022 as RFC 9309.</p>
</section><section data-mw-section-id="2" id="mwTg" aria-labelledby="Standard"><div class="mw-heading mw-heading2"><h2 id="Standard">Standard</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=2" title="Edit section: Standard"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwTw">A site owner wishing to give instructions to web robots places a text file called <style data-mw-deduplicate="TemplateStyles:r886049734" typeof="mw:Extension/templatestyles mw:Transclusion" about="#mwt38" id="mwUA" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"robots.txt"}},"i":0}}]}'>.mw-parser-output .monospaced{font-family:monospace,monospace}</style><span class="monospaced" about="#mwt38" id="mwUQ">robots.txt</span> in the root of the web site hierarchy (e.g. <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r886049734" about="#mwt40" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwUg" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"<nowiki>https://www.example.com/robots.txt</nowiki>"}},"i":0}}]}'/><span class="monospaced" about="#mwt40" id="mwUw">https://www.example.com/robots.txt</span>). This text file contains the instructions in a specific format (see examples below). Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Website" title="Website" id="mwVA">website</a>. If this file does not exist, web robots assume that the website owner does not wish to place any limitations on crawling the entire site.</p>
<p id="mwVQ">A robots.txt file contains instructions for bots indicating which web pages they can and cannot access. Robots.txt files are particularly important for web crawlers from search engines such as Google.</p>
<p id="mwVg">A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operates on certain data. Links to pages listed in robots.txt can still appear in search results if they are linked to from a page that is crawled.<sup about="#mwt44" class="mw-ref reference" id="cite_ref-10" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-10"}}'><a href="#cite_note-10" id="mwVw"><span class="mw-reflink-text" id="mwWA"><span class="cite-bracket" id="mwWQ">[</span>10<span class="cite-bracket" id="mwWg">]</span></span></a></sup> Because compliant crawlers cannot access the blocked page, search results may display only the URL or other information available from external sources, since the crawler cannot retrieve the page's content.<sup about="#mwt47" class="mw-ref reference" id="cite_ref-11" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-11"}}'><a href="#cite_note-11" id="mwWw"><span class="mw-reflink-text" id="mwXA"><span class="cite-bracket" id="mwXQ">[</span>11<span class="cite-bracket" id="mwXg">]</span></span></a></sup></p>
<p id="mwXw">A robots.txt file covers one <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Same_origin_policy" title="Same origin policy" class="mw-redirect" id="mwYA">origin</a>. For websites with multiple <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Subdomain" title="Subdomain" id="mwYQ">subdomains</a>, each subdomain must have its own robots.txt file. If <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r886049734" about="#mwt48" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwYg" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"example.com"}},"i":0}}]}'/><span class="monospaced" about="#mwt48" id="mwYw">example.com</span> had a robots.txt file but <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r886049734" about="#mwt50" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwZA" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"foo.example.com"}},"i":0}}]}'/><span class="monospaced" about="#mwt50" id="mwZQ">foo.example.com</span> did not, the rules that would apply for <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r886049734" about="#mwt52" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwZg" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"example.com"}},"i":0}}]}'/><span class="monospaced" about="#mwt52" id="mwZw">example.com</span> would not apply to <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r886049734" about="#mwt54" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwaA" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"foo.example.com"}},"i":0}}]}'/><span class="monospaced" about="#mwt54" id="mwaQ">foo.example.com</span>. In addition, each <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/List_of_URI_schemes" title="List of URI schemes" id="mwag">URI scheme</a> and <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Port_(computer_networking)" title="Port (computer networking)" id="mwaw">port</a> needs its own robots.txt file; <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r886049734" about="#mwt56" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwbA" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"<nowiki>http://example.com/robots.txt</nowiki>"}},"i":0}}]}'/><span class="monospaced" about="#mwt56" id="mwbQ">http://example.com/robots.txt</span> does not apply to pages under <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r886049734" about="#mwt58" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwbg" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"<nowiki>http://example.com:8080/</nowiki>"}},"i":0}}]}'/><span class="monospaced" about="#mwt58" id="mwbw">http://example.com:8080/</span> or <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r886049734" about="#mwt60" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwcA" data-mw='{"name":"templatestyles","attrs":{"src":"Mono/styles.css"},"parts":[{"template":{"target":{"wt":"mono","href":"./Template:Mono"},"params":{"1":{"wt":"<nowiki>https://example.com/</nowiki>"}},"i":0}}]}'/><span class="monospaced" about="#mwt60" id="mwcQ">https://example.com/</span>.</p>
</section><section data-mw-section-id="3" id="mwcg" aria-labelledby="Compliance"><div class="mw-heading mw-heading2"><h2 id="Compliance">Compliance</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=3" title="Edit section: Compliance"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwcw">The robots.txt protocol is widely complied with by bot operators.<sup about="#mwt62" class="mw-ref reference" id="cite_ref-Verge_6-1" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"Verge"}}'><a href="#cite_note-Verge-6" id="mwdA"><span class="mw-reflink-text" id="mwdQ"><span class="cite-bracket" id="mwdg">[</span>6<span class="cite-bracket" id="mwdw">]</span></span></a></sup></p>
<p id="mweA">The robots.txt played a role in the 1999 legal case of <i id="mweQ"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/EBay_v._Bidder's_Edge" title="EBay v. Bidder's Edge" id="mweg">eBay v. Bidder's Edge</a></i>,<sup about="#mwt65" class="mw-ref reference" id="cite_ref-:1_12-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":":1"},"body":{"id":"mw-reference-text-cite_note-:1-12"}}'><a href="#cite_note-:1-12" id="mwew"><span class="mw-reflink-text" id="mwfA"><span class="cite-bracket" id="mwfQ">[</span>12<span class="cite-bracket" id="mwfg">]</span></span></a></sup> where eBay attempted to block a bot that did not comply with robots.txt, and in May 2000 a court ordered the company operating the bot to stop crawling eBay's servers using any automatic means, by <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Injunction" title="Injunction" id="mwfw">legal injunction</a> on the basis of <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Trespass_to_chattels" title="Trespass to chattels" id="mwgA">trespassing</a>.<sup about="#mwt67" class="mw-ref reference" id="cite_ref-case_13-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"case"},"body":{"id":"mw-reference-text-cite_note-case-13"}}'><a href="#cite_note-case-13" id="mwgQ"><span class="mw-reflink-text" id="mwgg"><span class="cite-bracket" id="mwgw">[</span>13<span class="cite-bracket" id="mwhA">]</span></span></a></sup><sup about="#mwt70" class="mw-ref reference" id="cite_ref-14" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-14"}}'><a href="#cite_note-14" id="mwhQ"><span class="mw-reflink-text" id="mwhg"><span class="cite-bracket" id="mwhw">[</span>14<span class="cite-bracket" id="mwiA">]</span></span></a></sup><sup about="#mwt71" class="mw-ref reference" id="cite_ref-:1_12-1" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":":1"}}'><a href="#cite_note-:1-12" id="mwiQ"><span class="mw-reflink-text" id="mwig"><span class="cite-bracket" id="mwiw">[</span>12<span class="cite-bracket" id="mwjA">]</span></span></a></sup> Bidder's Edge appealed the ruling, but agreed in March 2001 to drop the appeal, pay an undisclosed amount to eBay, and stop accessing eBay's auction information.<sup about="#mwt74" class="mw-ref reference" id="cite_ref-15" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-15"}}'><a href="#cite_note-15" id="mwjQ"><span class="mw-reflink-text" id="mwjg"><span class="cite-bracket" id="mwjw">[</span>15<span class="cite-bracket" id="mwkA">]</span></span></a></sup><sup about="#mwt77" class="mw-ref reference" id="cite_ref-16" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-16"}}'><a href="#cite_note-16" id="mwkQ"><span class="mw-reflink-text" id="mwkg"><span class="cite-bracket" id="mwkw">[</span>16<span class="cite-bracket" id="mwlA">]</span></span></a></sup></p>
<p id="mwlQ">In 2007, <i id="mwlg">Healthcare Advocates v. Harding</i>, a company was sued for accessing protected web pages archived via <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Wayback_Machine" title="Wayback Machine" id="mwlw">The Wayback Machine</a>, despite robots.txt rules that prevented those pages from being archived. A <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Pennsylvania" title="Pennsylvania" id="mwmA">Pennsylvania</a> court ruled "in this situation, the robots.txt file qualifies as a technological measure" under the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Digital_Millennium_Copyright_Act" title="Digital Millennium Copyright Act" id="mwmQ">DMCA</a>. Due to a malfunction at Internet Archive, Harding could temporarily access these pages from the archive and thus the court found "the Harding firm did not circumvent the protective measure".<sup about="#mwt80" class="mw-ref reference" id="cite_ref-17" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-17"}}'><a href="#cite_note-17" id="mwmg"><span class="mw-reflink-text" id="mwmw"><span class="cite-bracket" id="mwnA">[</span>17<span class="cite-bracket" id="mwnQ">]</span></span></a></sup><sup about="#mwt83" class="mw-ref reference" id="cite_ref-18" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-18"}}'><a href="#cite_note-18" id="mwng"><span class="mw-reflink-text" id="mwnw"><span class="cite-bracket" id="mwoA">[</span>18<span class="cite-bracket" id="mwoQ">]</span></span></a></sup><sup about="#mwt86" class="mw-ref reference" id="cite_ref-19" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-19"}}'><a href="#cite_note-19" id="mwog"><span class="mw-reflink-text" id="mwow"><span class="cite-bracket" id="mwpA">[</span>19<span class="cite-bracket" id="mwpQ">]</span></span></a></sup></p>
<p id="mwpg">In 2013, <i id="mwpw"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Associated_Press_v._Meltwater_U.S._Holdings,_Inc." title="Associated Press v. Meltwater U.S. Holdings, Inc." id="mwqA">Associated Press v. Meltwater U.S. Holdings, Inc.</a></i> the Associated Press sued Meltwater for <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Copyright_infringement" title="Copyright infringement" id="mwqQ">copyright infringement</a> and misappropriation over copying of AP news items. Meltwater claimed that they did not require a license and that it was <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Fair_use" title="Fair use" id="mwqg">fair use</a>, because the content was freely available and not protected by robots.txt. The court decided in March 2013 that "Meltwater’s copying is not protected by the fair use doctrine", mentioning among several factors that "failure […] to employ the robots.txt protocol did not give Meltwater […] license to copy and publish AP content".<sup about="#mwt89" class="mw-ref reference" id="cite_ref-20" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-20"}}'><a href="#cite_note-20" id="mwqw"><span class="mw-reflink-text" id="mwrA"><span class="cite-bracket" id="mwrQ">[</span>20<span class="cite-bracket" id="mwrg">]</span></span></a></sup></p>
<section data-mw-section-id="4" id="mwrw" aria-labelledby="Search_engines"><div class="mw-heading mw-heading3"><h3 id="Search_engines">Search engines</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=4" title="Edit section: Search engines"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwsA">Some major <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Search_engine" title="Search engine" id="mwsQ">search engines</a> following this standard include <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Google_Search" title="Google Search" id="mwsg">Google</a>,<sup about="#mwt92" class="mw-ref reference" id="cite_ref-google-webmasters-spec_21-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"google-webmasters-spec"},"body":{"id":"mw-reference-text-cite_note-google-webmasters-spec-21"}}'><a href="#cite_note-google-webmasters-spec-21" id="mwsw"><span class="mw-reflink-text" id="mwtA"><span class="cite-bracket" id="mwtQ">[</span>21<span class="cite-bracket" id="mwtg">]</span></span></a></sup> <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Microsoft_Bing" title="Microsoft Bing" id="mwtw">Bing</a>,<sup about="#mwt95" class="mw-ref reference" id="cite_ref-bing-blog-robots_22-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"bing-blog-robots"},"body":{"id":"mw-reference-text-cite_note-bing-blog-robots-22"}}'><a href="#cite_note-bing-blog-robots-22" id="mwuA"><span class="mw-reflink-text" id="mwuQ"><span class="cite-bracket" id="mwug">[</span>22<span class="cite-bracket" id="mwuw">]</span></span></a></sup> <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Yahoo_Search" title="Yahoo Search" id="mwvA">Yahoo</a>!,<sup about="#mwt98" class="mw-ref reference" id="cite_ref-yahoo-search-is-bing_23-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"yahoo-search-is-bing"},"body":{"id":"mw-reference-text-cite_note-yahoo-search-is-bing-23"}}'><a href="#cite_note-yahoo-search-is-bing-23" id="mwvQ"><span class="mw-reflink-text" id="mwvg"><span class="cite-bracket" id="mwvw">[</span>23<span class="cite-bracket" id="mwwA">]</span></span></a></sup> <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Yandex_Search" title="Yandex Search" id="mwwQ">Yandex</a>,<sup about="#mwt101" class="mw-ref reference" id="cite_ref-yandex-robots_24-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"yandex-robots"},"body":{"id":"mw-reference-text-cite_note-yandex-robots-24"}}'><a href="#cite_note-yandex-robots-24" id="mwwg"><span class="mw-reflink-text" id="mwww"><span class="cite-bracket" id="mwxA">[</span>24<span class="cite-bracket" id="mwxQ">]</span></span></a></sup> <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/DuckDuckGo" title="DuckDuckGo" id="mwxg">DuckDuckGo</a>,<sup about="#mwt104" class="mw-ref reference" id="cite_ref-duckduckgo-bot_25-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"duckduckgo-bot"},"body":{"id":"mw-reference-text-cite_note-duckduckgo-bot-25"}}'><a href="#cite_note-duckduckgo-bot-25" id="mwxw"><span class="mw-reflink-text" id="mwyA"><span class="cite-bracket" id="mwyQ">[</span>25<span class="cite-bracket" id="mwyg">]</span></span></a></sup> <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Baidu" title="Baidu" id="mwyw">Baidu</a>,<sup about="#mwt107" class="mw-ref reference" id="cite_ref-baidu-spider_26-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"baidu-spider"},"body":{"id":"mw-reference-text-cite_note-baidu-spider-26"}}'><a href="#cite_note-baidu-spider-26" id="mwzA"><span class="mw-reflink-text" id="mwzQ"><span class="cite-bracket" id="mwzg">[</span>26<span class="cite-bracket" id="mwzw">]</span></span></a></sup> <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/AOL" title="AOL" id="mw0A">AOL</a>,<sup about="#mwt110" class="mw-ref reference" id="cite_ref-about-aol-search_27-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"about-aol-search"},"body":{"id":"mw-reference-text-cite_note-about-aol-search-27"}}'><a href="#cite_note-about-aol-search-27" id="mw0Q"><span class="mw-reflink-text" id="mw0g"><span class="cite-bracket" id="mw0w">[</span>27<span class="cite-bracket" id="mw1A">]</span></span></a></sup> <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Ask.com" title="Ask.com" id="mw1Q">Ask</a>,<sup about="#mwt113" class="mw-ref reference" id="cite_ref-ask-webmasters_28-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"ask-webmasters"},"body":{"id":"mw-reference-text-cite_note-ask-webmasters-28"}}'><a href="#cite_note-ask-webmasters-28" id="mw1g"><span class="mw-reflink-text" id="mw1w"><span class="cite-bracket" id="mw2A">[</span>28<span class="cite-bracket" id="mw2Q">]</span></span></a></sup> and <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Kagi" title="Kagi" id="mw2g">Kagi</a>,<sup about="#mwt116" class="mw-ref reference" id="cite_ref-KagiBot_29-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"KagiBot"},"body":{"id":"mw-reference-text-cite_note-KagiBot-29"}}'><a href="#cite_note-KagiBot-29" id="mw2w"><span class="mw-reflink-text" id="mw3A"><span class="cite-bracket" id="mw3Q">[</span>29<span class="cite-bracket" id="mw3g">]</span></span></a></sup></p>
</section><section data-mw-section-id="5" id="mw3w" aria-labelledby="Archival_sites"><div class="mw-heading mw-heading3"><h3 id="Archival_sites">Archival sites</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=5" title="Edit section: Archival sites"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mw4A">Some web archiving projects ignore robots.txt. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Archive_Team" title="Archive Team" id="mw4Q">Archive Team</a> uses the file to discover more links, such as <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Sitemap" title="Sitemap" class="mw-redirect" id="mw4g">sitemaps</a>.<sup about="#mwt119" class="mw-ref reference" id="cite_ref-30" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-30"}}'><a href="#cite_note-30" id="mw4w"><span class="mw-reflink-text" id="mw5A"><span class="cite-bracket" id="mw5Q">[</span>30<span class="cite-bracket" id="mw5g">]</span></span></a></sup> Co-founder <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Jason_Scott" title="Jason Scott" id="mw5w">Jason Scott</a> said that "unchecked, and left alone, the robots.txt file ensures no mirroring or reference for items that may have general use and meaning beyond the website's context."<sup about="#mwt122" class="mw-ref reference" id="cite_ref-31" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-31"}}'><a href="#cite_note-31" id="mw6A"><span class="mw-reflink-text" id="mw6Q"><span class="cite-bracket" id="mw6g">[</span>31<span class="cite-bracket" id="mw6w">]</span></span></a></sup> In 2017, the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Internet_Archive" title="Internet Archive" id="mw7A">Internet Archive</a> announced that it would stop complying with robots.txt directives.<sup about="#mwt125" class="mw-ref reference" id="cite_ref-32" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-32"}}'><a href="#cite_note-32" id="mw7Q"><span class="mw-reflink-text" id="mw7g"><span class="cite-bracket" id="mw7w">[</span>32<span class="cite-bracket" id="mw8A">]</span></span></a></sup><sup about="#mwt126" class="mw-ref reference" id="cite_ref-Verge_6-2" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"Verge"}}'><a href="#cite_note-Verge-6" id="mw8Q"><span class="mw-reflink-text" id="mw8g"><span class="cite-bracket" id="mw8w">[</span>6<span class="cite-bracket" id="mw9A">]</span></span></a></sup> According to <i id="mw9Q"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Digital_Trends" title="Digital Trends" id="mw9g">Digital Trends</a></i>, this followed widespread use of robots.txt to remove historical sites from search engine results, and contrasted with the nonprofit's aim to archive "snapshots" of the internet as it previously existed.<sup about="#mwt129" class="mw-ref reference" id="cite_ref-Internet_Archive_33-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"Internet Archive"},"body":{"id":"mw-reference-text-cite_note-Internet_Archive-33"}}'><a href="#cite_note-Internet_Archive-33" id="mw9w"><span class="mw-reflink-text" id="mw-A"><span class="cite-bracket" id="mw-Q">[</span>33<span class="cite-bracket" id="mw-g">]</span></span></a></sup></p>
</section><section data-mw-section-id="6" id="mw-w" aria-labelledby="Artificial_intelligence"><div class="mw-heading mw-heading3"><h3 id="Artificial_intelligence">Artificial intelligence</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=6" title="Edit section: Artificial intelligence"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mw_A">Starting in the 2020s, web operators began using robots.txt to deny access to bots collecting training data for <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Generative_AI" title="Generative AI" id="mw_Q">generative AI</a>. In 2023, Originality.AI found that 306 of the thousand most-visited websites blocked <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/OpenAI" title="OpenAI" id="mw_g">OpenAI</a>'s GPTBot in their robots.txt file and 85 blocked <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Google" title="Google" id="mw_w">Google</a>'s Google-Extended. Many robots.txt files named GPTBot as the only bot explicitly disallowed on all pages. Denying access to GPTBot was common among news websites such as the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/BBC" title="BBC" id="mwAQA">BBC</a> and <i id="mwAQE"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/The_New_York_Times" title="The New York Times" id="mwAQI">The New York Times</a></i>. In 2023, blog host <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Medium_(website)" title="Medium (website)" id="mwAQM">Medium</a> announced it would deny access to all artificial intelligence web crawlers as "AI companies have leached value from writers in order to spam Internet readers".<sup about="#mwt130" class="mw-ref reference" id="cite_ref-Verge_6-3" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"Verge"}}'><a href="#cite_note-Verge-6" id="mwAQQ"><span class="mw-reflink-text" id="mwAQU"><span class="cite-bracket" id="mwAQY">[</span>6<span class="cite-bracket" id="mwAQc">]</span></span></a></sup></p>
<p id="mwAQg">GPTBot complies with the robots.txt standard and advises web operators about how to disallow it, but <i id="mwAQk"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/The_Verge" title="The Verge" id="mwAQo">The Verge</a></i><span class="nowrap" style="padding-left:0.1em;" about="#mwt131" typeof="mw:Transclusion" id="mwAQs" data-mw='{"parts":[{"template":{"target":{"wt":"'","href":"./Template:'"},"params":{},"i":0}}]}'><span typeof="mw:Entity">'</span></span>s David Pierce said this only began after "training the underlying models that made it so powerful". Also, some bots are used both for search engines and artificial intelligence, and it may be impossible to block only one of these options.<sup about="#mwt132" class="mw-ref reference" id="cite_ref-Verge_6-4" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"Verge"}}'><a href="#cite_note-Verge-6" id="mwAQw"><span class="mw-reflink-text" id="mwAQ0"><span class="cite-bracket" id="mwAQ4">[</span>6<span class="cite-bracket" id="mwAQ8">]</span></span></a></sup> <i id="mwARA"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/404_Media" title="404 Media" id="mwARE">404 Media</a></i> reported that companies like <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Anthropic" title="Anthropic" id="mwARI">Anthropic</a> and <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Perplexity.ai" title="Perplexity.ai" class="mw-redirect" id="mwARM">Perplexity.ai</a> circumvented robots.txt by renaming or spinning up new scrapers to replace the ones that appeared on popular <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Blocklist" title="Blocklist" class="mw-redirect" id="mwARQ">blocklists</a>.<sup about="#mwt135" class="mw-ref reference" id="cite_ref-34" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-34"}}'><a href="#cite_note-34" id="mwARU"><span class="mw-reflink-text" id="mwARY"><span class="cite-bracket" id="mwARc">[</span>34<span class="cite-bracket" id="mwARg">]</span></span></a></sup> To help identify the growing number of web crawlers, Cloudflare maintains a public directory of known bot user agents, organized by category and operator.<sup about="#mwt139" class="mw-ref reference" id="cite_ref-35" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-35"}}'><a href="#cite_note-35" id="mwARk"><span class="mw-reflink-text" id="mwARo"><span class="cite-bracket" id="mwARs">[</span>35<span class="cite-bracket" id="mwARw">]</span></span></a></sup></p>
<p id="mwAR0">In 2025, the nonprofit <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/RSL_Collective?action=edit&redlink=1" title="RSL Collective (page does not exist)" class="new" typeof="mw:LocalizedAttrs" id="mwAR4" data-mw-i18n='{"title":{"lang":"x-page","key":"red-link-title","params":["RSL Collective"]}}'>RSL Collective</a> announced the launch of the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Really_Simple_Licensing" title="Really Simple Licensing" id="mwAR8">Really Simple Licensing</a> (RSL) open content licensing standard, allowing web publishers to set terms for AI bots in their robots.txt files. Participating companies at launch included <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Medium_(website)" title="Medium (website)" id="mwASA">Medium</a>, <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Reddit" title="Reddit" id="mwASE">Reddit</a>, and <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Yahoo" title="Yahoo" id="mwASI">Yahoo</a>.<sup about="#mwt142" class="mw-ref reference" id="cite_ref-tc-10sep2025_36-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"tc-10sep2025"},"body":{"id":"mw-reference-text-cite_note-tc-10sep2025-36"}}'><a href="#cite_note-tc-10sep2025-36" id="mwASM"><span class="mw-reflink-text" id="mwASQ"><span class="cite-bracket" id="mwASU">[</span>36<span class="cite-bracket" id="mwASY">]</span></span></a></sup><sup about="#mwt145" class="mw-ref reference" id="cite_ref-verge-10sep2025_37-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"verge-10sep2025"},"body":{"id":"mw-reference-text-cite_note-verge-10sep2025-37"}}'><a href="#cite_note-verge-10sep2025-37" id="mwASc"><span class="mw-reflink-text" id="mwASg"><span class="cite-bracket" id="mwASk">[</span>37<span class="cite-bracket" id="mwASo">]</span></span></a></sup><sup about="#mwt148" class="mw-ref reference" id="cite_ref-eg-10sep2025_38-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"eg-10sep2025"},"body":{"id":"mw-reference-text-cite_note-eg-10sep2025-38"}}'><a href="#cite_note-eg-10sep2025-38" id="mwASs"><span class="mw-reflink-text" id="mwASw"><span class="cite-bracket" id="mwAS0">[</span>38<span class="cite-bracket" id="mwAS4">]</span></span></a></sup></p>
</section></section><section data-mw-section-id="7" id="mwAS8" aria-labelledby="Security"><div class="mw-heading mw-heading2"><h2 id="Security">Security</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=7" title="Edit section: Security"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwATA">Despite the use of the terms <i id="mwATE">allow</i> and <i id="mwATI">disallow</i>, the protocol is purely advisory and relies on the compliance of the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Web_robot" title="Web robot" class="mw-redirect" id="mwATM">web robot</a>; it cannot enforce any of what is stated in the file.<sup about="#mwt151" class="mw-ref reference" id="cite_ref-39" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-39"}}'><a href="#cite_note-39" id="mwATQ"><span class="mw-reflink-text" id="mwATU"><span class="cite-bracket" id="mwATY">[</span>39<span class="cite-bracket" id="mwATc">]</span></span></a></sup> Malicious web robots are unlikely to honor robots.txt; some may even use the robots.txt as a guide to find disallowed links and go straight to them. While this is sometimes claimed to be a security risk,<sup about="#mwt154" class="mw-ref reference" id="cite_ref-40" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-40"}}'><a href="#cite_note-40" id="mwATg"><span class="mw-reflink-text" id="mwATk"><span class="cite-bracket" id="mwATo">[</span>40<span class="cite-bracket" id="mwATs">]</span></span></a></sup> this sort of <i id="mwATw"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Security_through_obscurity" title="Security through obscurity" id="mwAT0">security through obscurity</a></i> is discouraged by standards bodies. The <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/National_Institute_of_Standards_and_Technology" title="National Institute of Standards and Technology" id="mwAT4">National Institute of Standards and Technology</a> (NIST) in the United States specifically recommends against this practice: "System security should not depend on the secrecy of the implementation or its components."<sup about="#mwt157" class="mw-ref reference" id="cite_ref-41" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-41"}}'><a href="#cite_note-41" id="mwAT8"><span class="mw-reflink-text" id="mwAUA"><span class="cite-bracket" id="mwAUE">[</span>41<span class="cite-bracket" id="mwAUI">]</span></span></a></sup> In the context of robots.txt files, security through obscurity is not recommended as a security technique.<sup about="#mwt160" class="mw-ref reference" id="cite_ref-42" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-42"}}'><a href="#cite_note-42" id="mwAUM"><span class="mw-reflink-text" id="mwAUQ"><span class="cite-bracket" id="mwAUU">[</span>42<span class="cite-bracket" id="mwAUY">]</span></span></a></sup></p>
</section><section data-mw-section-id="8" id="mwAUc" aria-labelledby="Alternatives"><div class="mw-heading mw-heading2"><h2 id="Alternatives">Alternatives</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=8" title="Edit section: Alternatives"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAUg">Many robots also pass a special <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/User-agent" title="User-agent" class="mw-redirect" id="mwAUk">user-agent</a> to the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Web_server" title="Web server" id="mwAUo">web server</a> when fetching content.<sup about="#mwt163" class="mw-ref reference" id="cite_ref-43" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-43"}}'><a href="#cite_note-43" id="mwAUs"><span class="mw-reflink-text" id="mwAUw"><span class="cite-bracket" id="mwAU0">[</span>43<span class="cite-bracket" id="mwAU4">]</span></span></a></sup> A web administrator could also configure the server to automatically return failure (or <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Cloaking" title="Cloaking" id="mwAU8">pass alternative content</a>) when it detects a connection using one of the robots.<sup about="#mwt166" class="mw-ref reference" id="cite_ref-44" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-44"}}'><a href="#cite_note-44" id="mwAVA"><span class="mw-reflink-text" id="mwAVE"><span class="cite-bracket" id="mwAVI">[</span>44<span class="cite-bracket" id="mwAVM">]</span></span></a></sup><sup about="#mwt169" class="mw-ref reference" id="cite_ref-45" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-45"}}'><a href="#cite_note-45" id="mwAVQ"><span class="mw-reflink-text" id="mwAVU"><span class="cite-bracket" id="mwAVY">[</span>45<span class="cite-bracket" id="mwAVc">]</span></span></a></sup></p>
<p id="mwAVg">Some sites, such as <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Google" title="Google" id="mwAVk">Google</a>, host a <code id="mwAVo">humans.txt</code> file that displays information meant for humans to read.<sup about="#mwt172" class="mw-ref reference" id="cite_ref-46" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-46"}}'><a href="#cite_note-46" id="mwAVs"><span class="mw-reflink-text" id="mwAVw"><span class="cite-bracket" id="mwAV0">[</span>46<span class="cite-bracket" id="mwAV4">]</span></span></a></sup> Some sites such as <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/GitHub" title="GitHub" id="mwAV8">GitHub</a> redirect humans.txt to an <i id="mwAWA">About</i> page.<sup about="#mwt175" class="mw-ref reference" id="cite_ref-47" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-47"}}'><a href="#cite_note-47" id="mwAWE"><span class="mw-reflink-text" id="mwAWI"><span class="cite-bracket" id="mwAWM">[</span>47<span class="cite-bracket" id="mwAWQ">]</span></span></a></sup></p>
<p id="mwAWU">Previously, Google had a joke file hosted at <code id="mwAWY">/killer-robots.txt</code> instructing <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Terminator_(character)" title="Terminator (character)" id="mwAWc">the Terminator</a> not to kill the company founders <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Larry_Page" title="Larry Page" id="mwAWg">Larry Page</a> and <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Sergey_Brin" title="Sergey Brin" id="mwAWk">Sergey Brin</a>.<sup about="#mwt178" class="mw-ref reference" id="cite_ref-48" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-48"}}'><a href="#cite_note-48" id="mwAWo"><span class="mw-reflink-text" id="mwAWs"><span class="cite-bracket" id="mwAWw">[</span>48<span class="cite-bracket" id="mwAW0">]</span></span></a></sup><sup about="#mwt181" class="mw-ref reference" id="cite_ref-49" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-49"}}'><a href="#cite_note-49" id="mwAW4"><span class="mw-reflink-text" id="mwAW8"><span class="cite-bracket" id="mwAXA">[</span>49<span class="cite-bracket" id="mwAXE">]</span></span></a></sup></p>
</section><section data-mw-section-id="9" id="mwAXI" aria-labelledby="Examples"><div class="mw-heading mw-heading2"><h2 id="Examples">Examples</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=9" title="Edit section: Examples"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAXM">This example tells all robots that they can visit all files because the wildcard <code id="mwAXQ">*</code> stands for all robots and the <code id="mwAXU">Disallow</code> directive has no value, meaning no pages are disallowed.</p>
<pre typeof="mw:Extension/pre" about="#mwt182" id="mwAXY" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: *\nDisallow: \n"}}'>User-agent: *
Disallow:
</pre>
<p id="mwAXc">This example has the same effect, allowing all files rather than prohibiting none.</p>
<pre typeof="mw:Extension/pre" about="#mwt183" id="mwAXg" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: *\nAllow: /\n"}}'>User-agent: *
Allow: /
</pre>
<p id="mwAXk">The same result can be accomplished with an empty or missing robots.txt file.</p>
<p id="mwAXo">This example tells all robots to stay out of a website:</p>
<pre typeof="mw:Extension/pre" about="#mwt184" id="mwAXs" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: *\nDisallow: /\n"}}'>User-agent: *
Disallow: /
</pre>
<p id="mwAXw">This example tells all robots not to enter three directories:</p>
<pre typeof="mw:Extension/pre" about="#mwt185" id="mwAX0" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: *\nDisallow: /cgi-bin/\nDisallow: /tmp/\nDisallow: /junk/\n"}}'>User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
</pre>
<p id="mwAX4">This example tells all robots to stay away from one specific file:</p>
<pre typeof="mw:Extension/pre" about="#mwt186" id="mwAX8" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: *\nDisallow: /directory/file.html\n"}}'>User-agent: *
Disallow: /directory/file.html
</pre>
<p id="mwAYA">All other files in the specified directory will be processed.</p>
<p id="mwAYE">Example demonstrating how comments can be used:</p>
<pre typeof="mw:Extension/pre" about="#mwt187" id="mwAYI" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\n# Comments appear after the \"#\" symbol at the start of a line, or after a directive\nUser-agent: * # match all bots\nDisallow: / # keep them out\n"}}'># Comments appear after the "#" symbol at the start of a line, or after a directive
User-agent: * # match all bots
Disallow: / # keep them out
</pre>
<p id="mwAYM">This example tells one specific robot to stay out of a website:</p>
<pre typeof="mw:Extension/pre" about="#mwt188" id="mwAYQ" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot\nDisallow: /\n"}}'>User-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot
Disallow: /
</pre>
<p id="mwAYU">This example tells two specific robots not to enter one specific directory:</p>
<pre typeof="mw:Extension/pre" about="#mwt189" id="mwAYY" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot\nUser-agent: Googlebot\nDisallow: /private/\n"}}'>User-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot
User-agent: Googlebot
Disallow: /private/
</pre>
<p id="mwAYc">It is also possible to list multiple robots with their own rules. The actual robot string is defined by the crawler. A few robot operators, such as <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Google" title="Google" id="mwAYg">Google</a>, support several user-agent strings that allow the operator to deny access to a subset of their services by using specific user-agent strings.<sup about="#mwt190" class="mw-ref reference" id="cite_ref-google-webmasters-spec_21-1" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"google-webmasters-spec"}}'><a href="#cite_note-google-webmasters-spec-21" id="mwAYk"><span class="mw-reflink-text" id="mwAYo"><span class="cite-bracket" id="mwAYs">[</span>21<span class="cite-bracket" id="mwAYw">]</span></span></a></sup></p>
<p id="mwAY0">Example demonstrating multiple user-agents:</p>
<pre typeof="mw:Extension/pre" about="#mwt191" id="mwAY4" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: googlebot # all Google services\nDisallow: /private/ # disallow this directory\n\nUser-agent: googlebot-news # only the news service\nDisallow: / # disallow everything\n\nUser-agent: * # any robot\nDisallow: /something/ # disallow this directory\n"}}'>User-agent: googlebot # all Google services
Disallow: /private/ # disallow this directory
User-agent: googlebot-news # only the news service
Disallow: / # disallow everything
User-agent: * # any robot
Disallow: /something/ # disallow this directory
</pre>
<section data-mw-section-id="10" id="mwAY8" aria-labelledby="The_use_of_the_wildcard_*_in_rules"><div class="mw-heading mw-heading3"><h3 id="The_use_of_the_wildcard_*_in_rules"><span id="The_use_of_the_wildcard_.2A_in_rules" typeof="mw:FallbackId"></span>The use of the wildcard * in rules</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=10" title="Edit section: The use of the wildcard * in rules"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAZA">The directive <code id="mwAZE">Disallow: /something/</code> blocks all files and subdirectories starting with <code id="mwAZI">/something/</code>.</p>
<p id="mwAZM">In contrast, using a wildcard (if supported by the crawler) allows for more complex patterns in specifying paths and files to allow or disallow from crawling, for example <code id="mwAZQ">Disallow: /something/*/other</code> blocks URLs such as:</p>
<pre typeof="mw:Extension/pre" about="#mwt192" id="mwAZU" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"/something/foo/other\n/something/bar/other\n"}}'>/something/foo/other
/something/bar/other
</pre>
<p id="mwAZY">It would not prevent the crawling of <code id="mwAZc">/something/foo/else</code>, as that would not match the pattern.</p>
<p id="mwAZg">The wildcard <code id="mwAZk">*</code> allows greater flexibility but may not be recognized by all crawlers, although it is part of the Robots Exclusion Protocol (RFC).<sup about="#mwt195" class="mw-ref reference" id="cite_ref-50" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-50"}}'><a href="#cite_note-50" id="mwAZo"><span class="mw-reflink-text" id="mwAZs"><span class="cite-bracket" id="mwAZw">[</span>50<span class="cite-bracket" id="mwAZ0">]</span></span></a></sup></p>
<p id="mwAZ4">A wildcard at the end of a rule in effect does nothing, as that is the standard behaviour.</p>
</section></section><section data-mw-section-id="11" id="mwAZ8" aria-labelledby="Nonstandard_extensions"><div class="mw-heading mw-heading2"><h2 id="Nonstandard_extensions">Nonstandard extensions</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=11" title="Edit section: Nonstandard extensions"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<section data-mw-section-id="12" id="mwAaA" aria-labelledby="Crawl-delay_directive"><div class="mw-heading mw-heading3"><h3 id="Crawl-delay_directive">Crawl-delay directive</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=12" title="Edit section: Crawl-delay directive"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAaE">The crawl-delay value is supported by some crawlers to throttle their visits to the host. Since this value is not part of the standard, its interpretation is dependent on the crawler reading it. It is used when the multiple burst of visits from bots is slowing down the host. Yandex interprets the value as the number of seconds to wait between subsequent visits.<sup about="#mwt196" class="mw-ref reference" id="cite_ref-yandex-robots_24-1" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"yandex-robots"}}'><a href="#cite_note-yandex-robots-24" id="mwAaI"><span class="mw-reflink-text" id="mwAaM"><span class="cite-bracket" id="mwAaQ">[</span>24<span class="cite-bracket" id="mwAaU">]</span></span></a></sup> Bing defines crawl-delay as the size of a time window (from 1 to 30 seconds) during which BingBot will access a web site only once.<sup about="#mwt199" class="mw-ref reference" id="cite_ref-bing-crawl-delay_51-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"bing-crawl-delay"},"body":{"id":"mw-reference-text-cite_note-bing-crawl-delay-51"}}'><a href="#cite_note-bing-crawl-delay-51" id="mwAaY"><span class="mw-reflink-text" id="mwAac"><span class="cite-bracket" id="mwAag">[</span>51<span class="cite-bracket" id="mwAak">]</span></span></a></sup> Google ignores this directive,<sup about="#mwt202" class="mw-ref reference" id="cite_ref-52" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-52"}}'><a href="#cite_note-52" id="mwAao"><span class="mw-reflink-text" id="mwAas"><span class="cite-bracket" id="mwAaw">[</span>52<span class="cite-bracket" id="mwAa0">]</span></span></a></sup> but provides an interface in its <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Google_Search_Console" title="Google Search Console" id="mwAa4">search console</a> for webmasters, to control the <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Googlebot" title="Googlebot" id="mwAa8">Googlebot</a>'s subsequent visits.<sup about="#mwt205" class="mw-ref reference" id="cite_ref-53" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-53"}}'><a href="#cite_note-53" id="mwAbA"><span class="mw-reflink-text" id="mwAbE"><span class="cite-bracket" id="mwAbI">[</span>53<span class="cite-bracket" id="mwAbM">]</span></span></a></sup></p>
<pre typeof="mw:Extension/pre" about="#mwt206" id="mwAbQ" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"\nUser-agent: bingbot\nAllow: /\nCrawl-delay: 10 seconds\n"}}'>User-agent: bingbot
Allow: /
Crawl-delay: 10 seconds
</pre>
</section><section data-mw-section-id="13" id="mwAbU" aria-labelledby="Sitemap"><div class="mw-heading mw-heading3"><h3 id="Sitemap">Sitemap</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=13" title="Edit section: Sitemap"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAbY">Some crawlers support a <code id="mwAbc">Sitemap</code> directive, allowing multiple <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Sitemaps" title="Sitemaps" id="mwAbg">Sitemaps</a> in the same <samp id="mwAbk">robots.txt</samp> in the form <code id="mwAbo">Sitemap: <i id="mwAbs">full-url</i></code>:<sup about="#mwt209" class="mw-ref reference" id="cite_ref-54" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-54"}}'><a href="#cite_note-54" id="mwAbw"><span class="mw-reflink-text" id="mwAb0"><span class="cite-bracket" id="mwAb4">[</span>54<span class="cite-bracket" id="mwAb8">]</span></span></a></sup><sup about="#mwt212" class="mw-ref reference" id="cite_ref-55" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-55"}}'><a href="#cite_note-55" id="mwAcA"><span class="mw-reflink-text" id="mwAcE"><span class="cite-bracket" id="mwAcI">[</span>55<span class="cite-bracket" id="mwAcM">]</span></span></a></sup></p>
<pre typeof="mw:Extension/pre" about="#mwt213" id="mwAcQ" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"Sitemap: http://www.example.com/sitemap.xml"}}'>Sitemap: http://www.example.com/sitemap.xml</pre>
</section><section data-mw-section-id="14" id="mwAcU" aria-labelledby='Universal_"*"_match'><div class="mw-heading mw-heading3"><h3 id='Universal_"*"_match'><span id="Universal_.22.2A.22_match" typeof="mw:FallbackId"></span>Universal "*" match</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=14" title='Edit section: Universal "*" match'><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAcY">The <i id="mwAcc">Robot Exclusion Standard</i> does not mention the "*" character in the <code id="mwAcg">Disallow:</code> statement.<sup about="#mwt216" class="mw-ref reference" id="cite_ref-56" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-56"}}'><a href="#cite_note-56" id="mwAck"><span class="mw-reflink-text" id="mwAco"><span class="cite-bracket" id="mwAcs">[</span>56<span class="cite-bracket" id="mwAcw">]</span></span></a></sup></p><!-- Please note that Google updated their code to match the standard on On July 1, 2019. References older than that may contain old, obsolete information about how Google behaves -->
</section><section data-mw-section-id="15" id="mwAc0" aria-labelledby="Content-Signal"><div class="mw-heading mw-heading3"><h3 id="Content-Signal">Content-Signal</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=15" title="Edit section: Content-Signal"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAc4"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Cloudflare" title="Cloudflare" id="mwAc8">Cloudflare</a> introduced <code id="mwAdA">Content-Signal</code><sup about="#mwt219" class="mw-ref reference" id="cite_ref-57" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-57"}}'><a href="#cite_note-57" id="mwAdE"><span class="mw-reflink-text" id="mwAdI"><span class="cite-bracket" id="mwAdM">[</span>57<span class="cite-bracket" id="mwAdQ">]</span></span></a></sup><sup about="#mwt222" class="mw-ref reference" id="cite_ref-58" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-58"}}'><a href="#cite_note-58" id="mwAdU"><span class="mw-reflink-text" id="mwAdY"><span class="cite-bracket" id="mwAdc">[</span>58<span class="cite-bracket" id="mwAdg">]</span></span></a></sup> as a directive to suggest acceptable crawler behavior by type, <code id="mwAdk">ai-train</code>, <code id="mwAdo">ai-input</code>, and <code id="mwAds">search</code> with values of <code id="mwAdw">yes</code> or <code id="mwAd0">no</code> for each.<sup about="#mwt225" class="mw-ref reference" id="cite_ref-59" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-59"}}'><a href="#cite_note-59" id="mwAd4"><span class="mw-reflink-text" id="mwAd8"><span class="cite-bracket" id="mwAeA">[</span>59<span class="cite-bracket" id="mwAeE">]</span></span></a></sup></p>
<pre typeof="mw:Extension/pre" about="#mwt226" id="mwAeI" data-mw='{"name":"pre","attrs":{},"body":{"extsrc":"Content-Signal: ai-train=no, search=yes, ai-input=no"}}'>Content-Signal: ai-train=no, search=yes, ai-input=no</pre>
</section></section><section data-mw-section-id="16" id="mwAeM" aria-labelledby="Meta_tags_and_headers"><div class="mw-heading mw-heading2"><h2 id="Meta_tags_and_headers">Meta tags and headers</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=16" title="Edit section: Meta tags and headers"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAeQ">In addition to root-level robots.txt files, robots exclusion directives can be applied at a more granular level through the use of <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Robots_meta_tag" title="Robots meta tag" class="mw-redirect" id="mwAeU">Robots meta tags</a> and X-Robots-Tag HTTP headers. The robots meta tag cannot be used for non-HTML files such as images, text files, or PDF documents. On the other hand, the X-Robots-Tag can be added to non-HTML files by using <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/.htaccess" title=".htaccess" id="mwAeY">.htaccess</a> and <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Httpd.conf" title="Httpd.conf" class="mw-redirect" id="mwAec">httpd.conf</a> files.<sup about="#mwt229" class="mw-ref reference" id="cite_ref-google-meta_60-0" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"google-meta"},"body":{"id":"mw-reference-text-cite_note-google-meta-60"}}'><a href="#cite_note-google-meta-60" id="mwAeg"><span class="mw-reflink-text" id="mwAek"><span class="cite-bracket" id="mwAeo">[</span>60<span class="cite-bracket" id="mwAes">]</span></span></a></sup></p>
<section data-mw-section-id="17" id="mwAew" aria-labelledby='A_"noindex"_meta_tag'><div class="mw-heading mw-heading3"><h3 id='A_"noindex"_meta_tag'><span id="A_.22noindex.22_meta_tag" typeof="mw:FallbackId"></span>A "noindex" meta tag</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=17" title='Edit section: A "noindex" meta tag'><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<div class="mw-highlight mw-highlight-lang-html mw-content-ltr" dir="ltr" typeof="mw:Extension/syntaxhighlight" about="#mwt230" id="mwAe0" data-mw='{"name":"syntaxhighlight","attrs":{"lang":"html"},"body":{"extsrc":"\n<meta name=\"robots\" content=\"noindex\" />\n"}}'><pre id="mwAe4"><span id="mwAe8"></span><span class="p" id="mwAfA"><</span><span class="nt" id="mwAfE">meta</span> <span class="na" id="mwAfI">name</span><span class="o" id="mwAfM">=</span><span class="s" id="mwAfQ">"robots"</span> <span class="na" id="mwAfU">content</span><span class="o" id="mwAfY">=</span><span class="s" id="mwAfc">"noindex"</span> <span class="p" id="mwAfg">/></span>
</pre></div>
</section><section data-mw-section-id="18" id="mwAfk" aria-labelledby='A_"noindex"_HTTP_response_header'><div class="mw-heading mw-heading3"><h3 id='A_"noindex"_HTTP_response_header'><span id="A_.22noindex.22_HTTP_response_header" typeof="mw:FallbackId"></span>A "noindex" HTTP response header</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=18" title='Edit section: A "noindex" HTTP response header'><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<div class="mw-highlight mw-highlight-lang-html mw-content-ltr" dir="ltr" typeof="mw:Extension/syntaxhighlight" about="#mwt231" id="mwAfo" data-mw='{"name":"syntaxhighlight","attrs":{"lang":"html"},"body":{"extsrc":"\nX-Robots-Tag: noindex\n"}}'><pre id="mwAfs"><span id="mwAfw"></span>X-Robots-Tag: noindex
</pre></div>
<p id="mwAf0">The X-Robots-Tag is only effective after the page has been requested and the server responds, and the robots meta tag is only effective after the page has loaded, whereas robots.txt is effective before the page is requested. Thus if a page is excluded by a robots.txt file, any robots meta tags or X-Robots-Tag headers are effectively ignored because the robot will not see them in the first place.<sup about="#mwt232" class="mw-ref reference" id="cite_ref-google-meta_60-1" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":"google-meta"}}'><a href="#cite_note-google-meta-60" id="mwAf4"><span class="mw-reflink-text" id="mwAf8"><span class="cite-bracket" id="mwAgA">[</span>60<span class="cite-bracket" id="mwAgE">]</span></span></a></sup></p>
</section><section data-mw-section-id="19" id="mwAgI" aria-labelledby="Maximum_size_of_a_robots.txt_file"><div class="mw-heading mw-heading3"><h3 id="Maximum_size_of_a_robots.txt_file">Maximum size of a robots.txt file</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=19" title="Edit section: Maximum size of a robots.txt file"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<p id="mwAgM">The Robots Exclusion Protocol requires crawlers to parse at least 500 kibibytes (512000 bytes) of robots.txt files,<sup about="#mwt233" class="mw-ref reference" id="cite_ref-61" rel="dc:references" typeof="mw:Transclusion mw:Extension/ref" data-mw='{"name":"ref","attrs":{"name":""},"body":{"id":"mw-reference-text-cite_note-61"},"parts":[{"template":{"target":{"wt":"Ref RFC","href":"./Template:Ref_RFC"},"params":{"1":{"wt":"9309"},"section":{"wt":"2.5: Limits"}},"i":0}}]}'><a href="#cite_note-61" id="mwAgQ"><span class="mw-reflink-text" id="mwAgU"><span class="cite-bracket" id="mwAgY">[</span>61<span class="cite-bracket" id="mwAgc">]</span></span></a></sup> which Google maintains as a 500 kibibyte file size restriction for robots.txt files.<sup about="#mwt241" class="mw-ref reference" id="cite_ref-62" rel="dc:references" typeof="mw:Extension/ref" data-mw='{"name":"ref","attrs":{},"body":{"id":"mw-reference-text-cite_note-62"}}'><a href="#cite_note-62" id="mwAgg"><span class="mw-reflink-text" id="mwAgk"><span class="cite-bracket" id="mwAgo">[</span>62<span class="cite-bracket" id="mwAgs">]</span></span></a></sup></p>
</section></section><section data-mw-section-id="20" id="mwAgw" aria-labelledby="See_also"><div class="mw-heading mw-heading2"><h2 id="See_also">See also</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=20" title="Edit section: See also"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<style data-mw-deduplicate="TemplateStyles:r1266661725" typeof="mw:Extension/templatestyles mw:Transclusion" about="#mwt242" id="mwAg0" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Portal/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Portal","href":"./Template:Portal"},"params":{"1":{"wt":"Internet"}},"i":0}}]}'>.mw-parser-output .portalbox{padding:0;margin:0.5em 0;display:table;box-sizing:border-box;max-width:175px;list-style:none}.mw-parser-output .portalborder{border:1px solid var(--border-color-base,#a2a9b1);padding:0.1em;background:var(--background-color-neutral-subtle,#f8f9fa)}.mw-parser-output .portalbox-entry{display:table-row;font-size:85%;line-height:110%;height:1.9em;font-style:italic;font-weight:bold}.mw-parser-output .portalbox-image{display:table-cell;padding:0.2em;vertical-align:middle;text-align:center}.mw-parser-output .portalbox-link{display:table-cell;padding:0.2em 0.2em 0.2em 0.3em;vertical-align:middle}@media(min-width:720px){.mw-parser-output .portalleft{margin:0.5em 1em 0.5em 0}.mw-parser-output .portalright{clear:right;float:right;margin:0.5em 0 0.5em 1em}}</style><ul role="navigation" aria-label="Portals" class="noprint portalbox portalborder portalright" about="#mwt242" id="mwAg4">
<li class="portalbox-entry"><span class="portalbox-image"><span class="noviewer" typeof="mw:File"><a href="https://en.wikipedia.org/wiki/File:Crystal_Clear_app_linneighborhood.svg" class="mw-file-description"><img resource="https://en.wikipedia.org/wiki/File:Crystal_Clear_app_linneighborhood.svg" src="//upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Crystal_Clear_app_linneighborhood.svg/40px-Crystal_Clear_app_linneighborhood.svg.png?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail" decoding="async" loading="lazy" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Crystal_Clear_app_linneighborhood.svg/60px-Crystal_Clear_app_linneighborhood.svg.png?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail 2x" alt="icon" data-file-width="407" data-file-height="407" data-file-type="drawing" height="28" width="28" class="mw-file-element"/></a></span></span><span class="portalbox-link"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Portal:Internet" title="Portal:Internet">Internet portal</a></span></li></ul>
<span class="mw-empty-elt" about="#mwt244" typeof="mw:Transclusion" id="mwAg8" data-mw='{"parts":[{"template":{"target":{"wt":"Columns-next","href":"./Template:Columns-next"},"params":{},"i":0}}]}'><style data-mw-deduplicate="TemplateStyles:r1318332354" typeof="mw:Extension/templatestyles" about="#mwt245" data-mw='{"name":"templatestyles","attrs":{"src":"Columns-next/styles.css"}}'>.mw-parser-output .tpl-columns-next+ul,.mw-parser-output .tpl-columns-next+ol{margin-top:0.3em;column-width:30em}</style></span><div class="tpl-columns-next" about="#mwt244" id="mwAhA"></div>
<ul id="mwAhE"><li id="mwAhI"><code id="mwAhM"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Ads.txt" title="Ads.txt" id="mwAhQ">ads.txt</a></code>, a standard for listing authorized ad sellers</li>
<li id="mwAhU"><code id="mwAhY"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Security.txt" title="Security.txt" id="mwAhc">security.txt</a></code>, a file to describe the process for security researchers to follow in order to report security vulnerabilities</li>
<li id="mwAhg"><code id="mwAhk"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Trust.txt" title="Trust.txt" id="mwAho">trust.txt</a></code>, a standard for news organizations</li>
<li id="mwAhs"><i id="mwAhw"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/EBay_v._Bidder's_Edge" title="EBay v. Bidder's Edge" id="mwAh0">eBay v. Bidder's Edge</a></i></li>
<li id="mwAh4"><i id="mwAh8"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn" title="HiQ Labs v. LinkedIn" id="mwAiA">hiQ Labs v. LinkedIn</a></i></li>
<li id="mwAiE"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Automated_Content_Access_Protocol" title="Automated Content Access Protocol" id="mwAiI">Automated Content Access Protocol</a> – A failed proposal to extend robots.txt</li>
<li id="mwAiM"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/BotSeer" title="BotSeer" id="mwAiQ">BotSeer</a> – Now inactive search engine for robots.txt files</li>
<li id="mwAiU"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Distributed_web_crawling" title="Distributed web crawling" id="mwAiY">Distributed web crawling</a></li>
<li id="mwAic"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Focused_crawler" title="Focused crawler" id="mwAig">Focused crawler</a></li>
<li id="mwAik"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Internet_Archive" title="Internet Archive" id="mwAio">Internet Archive</a></li>
<li id="mwAis"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Robots_meta_tag" title="Robots meta tag" class="mw-redirect" id="mwAiw">Meta elements</a> for search engines</li>
<li id="mwAi0"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/National_Digital_Library_Program" title="National Digital Library Program" id="mwAi4">National Digital Library Program</a> (NDLP)</li>
<li id="mwAi8"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/National_Digital_Information_Infrastructure_and_Preservation_Program" title="National Digital Information Infrastructure and Preservation Program" id="mwAjA">National Digital Information Infrastructure and Preservation Program</a> (NDIIPP)</li>
<li id="mwAjE"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Nofollow" title="Nofollow" id="mwAjI">nofollow</a></li>
<li id="mwAjM"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Noindex" title="Noindex" id="mwAjQ">noindex</a></li>
<li id="mwAjU"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Perma.cc" title="Perma.cc" id="mwAjY">Perma.cc</a></li>
<li id="mwAjc"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Really_Simple_Licensing" title="Really Simple Licensing" id="mwAjg">Really Simple Licensing</a></li>
<li id="mwAjk"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Sitemaps" title="Sitemaps" id="mwAjo">Sitemaps</a></li>
<li id="mwAjs"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Spider_trap" title="Spider trap" id="mwAjw">Spider trap</a></li>
<li id="mwAj0"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Web_archiving" title="Web archiving" id="mwAj4">Web archiving</a></li></ul>
</section><section data-mw-section-id="21" id="mwAj8" aria-labelledby="References"><div class="mw-heading mw-heading2"><h2 id="References">References</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=21" title="Edit section: References"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<span class="mw-empty-elt" about="#mwt246" typeof="mw:Transclusion" id="mwAkA" data-mw='{"parts":[{"template":{"target":{"wt":"Reflist","href":"./Template:Reflist"},"params":{"1":{"wt":"30em"}},"i":0}}]}'><style data-mw-deduplicate="TemplateStyles:r1327269900" typeof="mw:Extension/templatestyles" about="#mwt247" data-mw='{"name":"templatestyles","attrs":{"src":"Reflist/styles.css"}}'>.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}body.skin-vector-2022 .mw-parser-output .reflist-columns-2{column-width:27em}body.skin-vector-2022 .mw-parser-output .reflist-columns-3{column-width:22.5em}.mw-parser-output .references[data-mw-group=upper-alpha]{list-style-type:upper-alpha}.mw-parser-output .references[data-mw-group=upper-roman]{list-style-type:upper-roman}.mw-parser-output .references[data-mw-group=lower-alpha]{list-style-type:lower-alpha}.mw-parser-output .references[data-mw-group=lower-greek]{list-style-type:lower-greek}.mw-parser-output .references[data-mw-group=lower-roman]{list-style-type:lower-roman}.mw-parser-output div.reflist-liststyle-upper-alpha .references{list-style-type:upper-alpha}.mw-parser-output div.reflist-liststyle-upper-roman .references{list-style-type:upper-roman}.mw-parser-output div.reflist-liststyle-lower-alpha .references{list-style-type:lower-alpha}.mw-parser-output div.reflist-liststyle-lower-greek .references{list-style-type:lower-greek}.mw-parser-output div.reflist-liststyle-lower-roman .references{list-style-type:lower-roman}</style></span><div class="mw-references-wrap mw-references-columns" style="column-width: calc( 0.9 * 30em );" about="#mwt246" id="mwAkE"><ol class="mw-references references" typeof="mw:Extension/references" about="#mwt248" id="mwAkI" data-mw='{"name":"references","attrs":{"group":"","responsive":"0"},"body":{"extsrc":""}}'><li about="#cite_note-1" id="cite_note-1" data-mw-footnote-number="1"><span class="mw-cite-backlink" id="mwAkM"><a href="#cite_ref-1" rel="mw:referencedBy" id="mwAkQ"><span class="mw-linkback-text" id="mwAkU">↑</span></a></span> <span id="mw-reference-text-cite_note-1" class="mw-reference-text reference-text"><style data-mw-deduplicate="TemplateStyles:r1333433106" typeof="mw:Extension/templatestyles mw:Transclusion" about="#mwt8" id="mwAkY" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://www.greenhills.co.uk/historical.html"},"title":{"wt":"Historical"},"website":{"wt":"Greenhills.co.uk"},"access-date":{"wt":"2017-03-03"},"archive-url":{"wt":"https://web.archive.org/web/20170403152037/http://www.greenhills.co.uk/historical.html"},"archive-date":{"wt":"2017-04-03"},"url-status":{"wt":"live"}},"i":0}}]}'>.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("//upload.wikimedia.org/wikipedia/commons/6/65/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("//upload.wikimedia.org/wikipedia/commons/d/d6/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("//upload.wikimedia.org/wikipedia/commons/a/aa/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("//upload.wikimedia.org/wikipedia/commons/4/4c/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#bf3c2c)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#bf3c2c)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}</style><cite class="citation web cs1" about="#mwt8" id="mwAkc"><a rel="mw:ExtLink nofollow" href="http://www.greenhills.co.uk/historical.html" class="external text" id="mwAkg">"Historical"</a>. <i id="mwAkk">Greenhills.co.uk</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20170403152037/http://www.greenhills.co.uk/historical.html" class="external text" id="mwAko">Archived</a> from the original on 2017-04-03<span class="reference-accessdate" id="mwAks">. Retrieved <span class="nowrap" id="mwAkw">2017-03-03</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Greenhills.co.uk&rft.atitle=Historical&rft_id=http%3A%2F%2Fwww.greenhills.co.uk%2Fhistorical.html&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt8" id="mwAk0"></span></span></li>
<li about="#cite_note-2" id="cite_note-2" data-mw-footnote-number="2"><span class="mw-cite-backlink" id="mwAk4"><a href="#cite_ref-2" rel="mw:referencedBy" id="mwAk8"><span class="mw-linkback-text" id="mwAlA">↑</span></a></span> <span id="mw-reference-text-cite_note-2" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt11" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAlE" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"Maintaining Distributed Hypertext Infostructures: Welcome to MOMspider's Web"},"first":{"wt":"Roy"},"last":{"wt":"Fielding"},"work":{"wt":"First International Conference on the World Wide Web"},"year":{"wt":"1994"},"place":{"wt":"Geneva"},"url":{"wt":"http://www94.web.cern.ch/WWW94/PapersWWW94/fielding.ps"},"access-date":{"wt":"September 25, 2013"},"format":{"wt":"PostScript"},"archive-url":{"wt":"https://web.archive.org/web/20130927093658/http://www94.web.cern.ch/WWW94/PapersWWW94/fielding.ps"},"archive-date":{"wt":"2013-09-27"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite id="CITEREFFielding1994" class="citation web cs1" about="#mwt11">Fielding, Roy (1994). <a rel="mw:ExtLink nofollow" href="http://www94.web.cern.ch/WWW94/PapersWWW94/fielding.ps" class="external text" id="mwAlI">"Maintaining Distributed Hypertext Infostructures: Welcome to MOMspider's Web"</a> <span class="cs1-format" id="mwAlM">(PostScript)</span>. <i id="mwAlQ">First International Conference on the World Wide Web</i>. Geneva. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20130927093658/http://www94.web.cern.ch/WWW94/PapersWWW94/fielding.ps" class="external text" id="mwAlU">Archived</a> from the original on 2013-09-27<span class="reference-accessdate" id="mwAlY">. Retrieved <span class="nowrap" id="mwAlc">September 25,</span> 2013</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=First+International+Conference+on+the+World+Wide+Web&rft.atitle=Maintaining+Distributed+Hypertext+Infostructures%3A+Welcome+to+MOMspider%27s+Web&rft.date=1994&rft.aulast=Fielding&rft.aufirst=Roy&rft_id=http%3A%2F%2Fwww94.web.cern.ch%2FWWW94%2FPapersWWW94%2Ffielding.ps&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt11" id="mwAlg"></span></span></li>
<li about="#cite_note-:0-3" id="cite_note-:0-3" data-mw-footnote-number="3"><span class="mw-cite-backlink" id="mwAlk"><a href="#cite_ref-:0_3-0" rel="mw:referencedBy" id="mwAlo"><span class="mw-linkback-text" id="mwAls">↑</span></a></span> <span id="mw-reference-text-cite_note-:0-3" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt14" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAlw" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://www.robotstxt.org/orig.html#status"},"title":{"wt":"The Web Robots Pages"},"publisher":{"wt":"Robotstxt.org"},"date":{"wt":"1994-06-30"},"access-date":{"wt":"2013-12-29"},"archive-url":{"wt":"https://web.archive.org/web/20140112090633/http://www.robotstxt.org/orig.html#status"},"archive-date":{"wt":"2014-01-12"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt14" id="mwAl0"><a rel="mw:ExtLink nofollow" href="http://www.robotstxt.org/orig.html#status" class="external text" id="mwAl4">"The Web Robots Pages"</a>. Robotstxt.org. 1994-06-30. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20140112090633/http://www.robotstxt.org/orig.html#status" class="external text" id="mwAl8">Archived</a> from the original on 2014-01-12<span class="reference-accessdate" id="mwAmA">. Retrieved <span class="nowrap" id="mwAmE">2013-12-29</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=The+Web+Robots+Pages&rft.pub=Robotstxt.org&rft.date=1994-06-30&rft_id=http%3A%2F%2Fwww.robotstxt.org%2Forig.html%23status&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt14" id="mwAmI"></span></span></li>
<li about="#cite_note-4" id="cite_note-4" data-mw-footnote-number="4"><span class="mw-cite-backlink" id="mwAmM"><a href="#cite_ref-4" rel="mw:referencedBy" id="mwAmQ"><span class="mw-linkback-text" id="mwAmU">↑</span></a></span> <span id="mw-reference-text-cite_note-4" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt17" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAmY" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web","href":"./Template:Cite_web"},"params":{"title":{"wt":"Important: Spiders, Robots and Web Wanderers"},"first":{"wt":"Martijn"},"last":{"wt":"Koster"},"work":{"wt":"www-talk mailing list"},"date":{"wt":"25 February 1994"},"url":{"wt":"http://inkdroid.org/tmp/www-talk/4113.html"},"format":{"wt":"[[Hypermail]] archived message"},"url-status":{"wt":"dead"},"archive-url":{"wt":"https://web.archive.org/web/20131029200350/http://inkdroid.org/tmp/www-talk/4113.html"},"archive-date":{"wt":"October 29, 2013"}},"i":0}}]}'/><cite id="CITEREFKoster1994" class="citation web cs1" about="#mwt17">Koster, Martijn (25 February 1994). <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20131029200350/http://inkdroid.org/tmp/www-talk/4113.html" class="external text" id="mwAmc">"Important: Spiders, Robots and Web Wanderers"</a>. <i id="mwAmg">www-talk mailing list</i>. Archived from <a rel="mw:ExtLink nofollow" href="http://inkdroid.org/tmp/www-talk/4113.html" class="external text" id="mwAmk">the original</a> <span class="cs1-format" id="mwAmo">(<a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Hypermail" title="Hypermail" id="mwAms">Hypermail</a> archived message)</span> on October 29, 2013.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=www-talk+mailing+list&rft.atitle=Important%3A+Spiders%2C+Robots+and+Web+Wanderers&rft.date=1994-02-25&rft.aulast=Koster&rft.aufirst=Martijn&rft_id=http%3A%2F%2Finkdroid.org%2Ftmp%2Fwww-talk%2F4113.html&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt17" id="mwAmw"></span></span></li>
<li about="#cite_note-5" id="cite_note-5" data-mw-footnote-number="5"><span class="mw-cite-backlink" id="mwAm0"><a href="#cite_ref-5" rel="mw:referencedBy" id="mwAm4"><span class="mw-linkback-text" id="mwAm8">↑</span></a></span> <span id="mw-reference-text-cite_note-5" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt20" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAnA" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://www.antipope.org/charlie/blog-static/2009/06/how_i_got_here_in_the_end_part_3.html"},"title":{"wt":"How I got here in the end, part five: \"things can only get better!\""},"work":{"wt":"Charlie's Diary"},"date":{"wt":"19 June 2006"},"access-date":{"wt":"19 April 2014"},"archive-url":{"wt":"https://web.archive.org/web/20131125220913/http://www.antipope.org/charlie/blog-static/2009/06/how_i_got_here_in_the_end_part_3.html"},"archive-date":{"wt":"2013-11-25"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt20" id="mwAnE"><a rel="mw:ExtLink nofollow" href="http://www.antipope.org/charlie/blog-static/2009/06/how_i_got_here_in_the_end_part_3.html" class="external text" id="mwAnI">"How I got here in the end, part five: "things can only get better!"<span class="cs1-kern-right" id="mwAnM"></span>"</a>. <i id="mwAnQ">Charlie's Diary</i>. 19 June 2006. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20131125220913/http://www.antipope.org/charlie/blog-static/2009/06/how_i_got_here_in_the_end_part_3.html" class="external text" id="mwAnU">Archived</a> from the original on 2013-11-25<span class="reference-accessdate" id="mwAnY">. Retrieved <span class="nowrap" id="mwAnc">19 April</span> 2014</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Charlie%27s+Diary&rft.atitle=How+I+got+here+in+the+end%2C+part+five%3A+%22things+can+only+get+better%21%22&rft.date=2006-06-19&rft_id=http%3A%2F%2Fwww.antipope.org%2Fcharlie%2Fblog-static%2F2009%2F06%2Fhow_i_got_here_in_the_end_part_3.html&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt20" id="mwAng"></span></span></li>
<li about="#cite_note-Verge-6" id="cite_note-Verge-6" data-mw-footnote-number="6"><span rel="mw:referencedBy" class="mw-cite-backlink" id="mwAnk"><a href="#cite_ref-Verge_6-0" id="mwAno"><span class="mw-linkback-text" id="mwAns">1</span></a> <a href="#cite_ref-Verge_6-1" id="mwAnw"><span class="mw-linkback-text" id="mwAn0">2</span></a> <a href="#cite_ref-Verge_6-2" id="mwAn4"><span class="mw-linkback-text" id="mwAn8">3</span></a> <a href="#cite_ref-Verge_6-3" id="mwAoA"><span class="mw-linkback-text" id="mwAoE">4</span></a> <a href="#cite_ref-Verge_6-4" id="mwAoI"><span class="mw-linkback-text" id="mwAoM">5</span></a></span> <span id="mw-reference-text-cite_note-Verge-6" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt23" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAoQ" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders"},"title":{"wt":"The text file that runs the internet"},"work":{"wt":"[[The Verge]]"},"last":{"wt":"Pierce"},"first":{"wt":"David"},"date":{"wt":"14 February 2024"},"accessdate":{"wt":"16 March 2024"}},"i":0}}]}'/><cite id="CITEREFPierce2024" class="citation web cs1" about="#mwt23">Pierce, David (14 February 2024). <a rel="mw:ExtLink nofollow" href="https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders" class="external text" id="mwAoU">"The text file that runs the internet"</a>. <i id="mwAoY"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/The_Verge" title="The Verge" id="mwAoc">The Verge</a></i><span class="reference-accessdate" id="mwAog">. Retrieved <span class="nowrap" id="mwAok">16 March</span> 2024</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=The+Verge&rft.atitle=The+text+file+that+runs+the+internet&rft.date=2024-02-14&rft.aulast=Pierce&rft.aufirst=David&rft_id=https%3A%2F%2Fwww.theverge.com%2F24067997%2Frobots-txt-ai-text-file-web-crawlers-spiders&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt23" id="mwAoo"></span></span></li>
<li about="#cite_note-sear_Robo-7" id="cite_note-sear_Robo-7" data-mw-footnote-number="7"><span class="mw-cite-backlink" id="mwAos"><a href="#cite_ref-sear_Robo_7-0" rel="mw:referencedBy" id="mwAow"><span class="mw-linkback-text" id="mwAo0">↑</span></a></span> <span id="mw-reference-text-cite_note-sear_Robo-7" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt26" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAo4" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"Robots.txt Celebrates 20 Years Of Blocking Search Engines"},"author":{"wt":"Barry Schwartz"},"work":{"wt":"Search Engine Land"},"date":{"wt":"30 June 2014"},"access-date":{"wt":"2015-11-19"},"url":{"wt":"http://searchengineland.com/robots-txt-celebrates-20-years-blocking-search-engines-195479"},"archive-url":{"wt":"https://web.archive.org/web/20150907000430/http://searchengineland.com/robots-txt-celebrates-20-years-blocking-search-engines-195479"},"archive-date":{"wt":"2015-09-07"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite id="CITEREFBarry_Schwartz2014" class="citation web cs1" about="#mwt26">Barry Schwartz (30 June 2014). <a rel="mw:ExtLink nofollow" href="http://searchengineland.com/robots-txt-celebrates-20-years-blocking-search-engines-195479" class="external text" id="mwAo8">"Robots.txt Celebrates 20 Years Of Blocking Search Engines"</a>. <i id="mwApA">Search Engine Land</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20150907000430/http://searchengineland.com/robots-txt-celebrates-20-years-blocking-search-engines-195479" class="external text" id="mwApE">Archived</a> from the original on 2015-09-07<span class="reference-accessdate" id="mwApI">. Retrieved <span class="nowrap" id="mwApM">2015-11-19</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Search+Engine+Land&rft.atitle=Robots.txt+Celebrates+20+Years+Of+Blocking+Search+Engines&rft.date=2014-06-30&rft.au=Barry+Schwartz&rft_id=http%3A%2F%2Fsearchengineland.com%2Frobots-txt-celebrates-20-years-blocking-search-engines-195479&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt26" id="mwApQ"></span></span></li>
<li about="#cite_note-8" id="cite_note-8" data-mw-footnote-number="8"><span class="mw-cite-backlink" id="mwApU"><a href="#cite_ref-8" rel="mw:referencedBy" id="mwApY"><span class="mw-linkback-text" id="mwApc">↑</span></a></span> <span id="mw-reference-text-cite_note-8" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt35" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwApg" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://developers.google.com/search/blog/2019/07/rep-id"},"title":{"wt":"Formalizing the Robots Exclusion Protocol Specification"},"website":{"wt":"Official Google Webmaster Central Blog"},"language":{"wt":"en"},"access-date":{"wt":"2019-07-10"},"archive-date":{"wt":"2019-07-10"},"archive-url":{"wt":"https://web.archive.org/web/20190710060436/https://webmasters.googleblog.com/2019/07/rep-id.html"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt35" id="mwApk"><a rel="mw:ExtLink nofollow" href="https://developers.google.com/search/blog/2019/07/rep-id" class="external text" id="mwApo">"Formalizing the Robots Exclusion Protocol Specification"</a>. <i id="mwAps">Official Google Webmaster Central Blog</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20190710060436/https://webmasters.googleblog.com/2019/07/rep-id.html" class="external text" id="mwApw">Archived</a> from the original on 2019-07-10<span class="reference-accessdate" id="mwAp0">. Retrieved <span class="nowrap" id="mwAp4">2019-07-10</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Official+Google+Webmaster+Central+Blog&rft.atitle=Formalizing+the+Robots+Exclusion+Protocol+Specification&rft_id=https%3A%2F%2Fdevelopers.google.com%2Fsearch%2Fblog%2F2019%2F07%2Frep-id&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt35" id="mwAp8"></span></span></li>
<li about="#cite_note-rfc9309-9" id="cite_note-rfc9309-9" data-mw-footnote-number="9"><span class="mw-cite-backlink" id="mwAqA"><a href="#cite_ref-rfc9309_9-0" rel="mw:referencedBy" id="mwAqE"><span class="mw-linkback-text" id="mwAqI">↑</span></a></span> <span id="mw-reference-text-cite_note-rfc9309-9" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt31" typeof="mw:Extension/templatestyles" id="mwAqM" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""}}'/><cite id="CITEREFKosterIllyesZellerSassman2022" class="citation cs1">Koster, M.; Illyes, G.; Zeller, H.; Sassman, L. (September 2022). <a rel="mw:ExtLink nofollow" href="https://www.rfc-editor.org/rfc/rfc9309" class="external text" id="mwAqQ"><i id="mwAqU">Robots Exclusion Protocol</i></a>. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Internet_Engineering_Task_Force" title="Internet Engineering Task Force" id="mwAqY">Internet Engineering Task Force</a>. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Doi_(identifier)" title="Doi (identifier)" class="mw-redirect" id="mwAqc">doi</a>:<span class="id-lock-free" title="Freely accessible" id="mwAqg"><a rel="mw:ExtLink nofollow" href="https://doi.org/10.17487%2FRFC9309" class="external text" id="mwAqk">10.17487/RFC9309</a></span>. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Request_for_Comments" title="Request for Comments" id="mwAqo">RFC</a> <a rel="mw:ExtLink nofollow" href="https://datatracker.ietf.org/doc/html/rfc9309" class="external text" id="mwAqs">9309</a>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Robots+Exclusion+Protocol&rft.pub=Internet+Engineering+Task+Force&rft.date=2022-09&rft_id=info%3Adoi%2F10.17487%2FRFC9309&rft.aulast=Koster&rft.aufirst=M.&rft.au=Illyes%2C+G.&rft.au=Zeller%2C+H.&rft.au=Sassman%2C+L.&rft_id=https%3A%2F%2Fwww.rfc-editor.org%2Frfc%2Frfc9309&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" id="mwAqw"></span> <i id="mwAq0">Proposed Standard.</i> </span></li>
<li about="#cite_note-10" id="cite_note-10" data-mw-footnote-number="10"><span class="mw-cite-backlink" id="mwAq4"><a href="#cite_ref-10" rel="mw:referencedBy" id="mwAq8"><span class="mw-linkback-text" id="mwArA">↑</span></a></span> <span id="mw-reference-text-cite_note-10" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt42" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwArE" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://www.youtube.com/watch?v=KBdEwpRQRD0#t=196s"},"title":{"wt":"Uncrawled URLs in search results"},"publisher":{"wt":"YouTube"},"date":{"wt":"Oct 5, 2009"},"access-date":{"wt":"2013-12-29"},"archive-url":{"wt":"https://web.archive.org/web/20140106222500/http://www.youtube.com/watch?v=KBdEwpRQRD0#t=196s"},"archive-date":{"wt":"2014-01-06"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt42" id="mwArI"><a rel="mw:ExtLink nofollow" href="https://www.youtube.com/watch?v=KBdEwpRQRD0#t=196s" class="external text" id="mwArM">"Uncrawled URLs in search results"</a>. YouTube. Oct 5, 2009. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20140106222500/http://www.youtube.com/watch?v=KBdEwpRQRD0#t=196s" class="external text" id="mwArQ">Archived</a> from the original on 2014-01-06<span class="reference-accessdate" id="mwArU">. Retrieved <span class="nowrap" id="mwArY">2013-12-29</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Uncrawled+URLs+in+search+results&rft.pub=YouTube&rft.date=2009-10-05&rft_id=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKBdEwpRQRD0%23t%3D196s&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt42" id="mwArc"></span></span></li>
<li about="#cite_note-11" id="cite_note-11" data-mw-footnote-number="11"><span class="mw-cite-backlink" id="mwArg"><a href="#cite_ref-11" rel="mw:referencedBy" id="mwArk"><span class="mw-linkback-text" id="mwAro">↑</span></a></span> <span id="mw-reference-text-cite_note-11" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt45" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwArs" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web","href":"./Template:Cite_web"},"params":{"title":{"wt":"How Google interprets the robots.txt specification"},"url":{"wt":"https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt"},"website":{"wt":"Google Search Central"},"date":{"wt":"23 May 2024"},"access-date":{"wt":"2 August 2026"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt45" id="mwArw"><a rel="mw:ExtLink nofollow" href="https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt" class="external text" id="mwAr0">"How Google interprets the robots.txt specification"</a>. <i id="mwAr4">Google Search Central</i>. 23 May 2024<span class="reference-accessdate" id="mwAr8">. Retrieved <span class="nowrap" id="mwAsA">2 August</span> 2026</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Google+Search+Central&rft.atitle=How+Google+interprets+the+robots.txt+specification&rft.date=2024-05-23&rft_id=https%3A%2F%2Fdevelopers.google.com%2Fsearch%2Fdocs%2Fcrawling-indexing%2Frobots%2Frobots_txt&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt45" id="mwAsE"></span></span></li>
<li about="#cite_note-:1-12" id="cite_note-:1-12" data-mw-footnote-number="12"><span rel="mw:referencedBy" class="mw-cite-backlink" id="mwAsI"><a href="#cite_ref-:1_12-0" id="mwAsM"><span class="mw-linkback-text" id="mwAsQ">1</span></a> <a href="#cite_ref-:1_12-1" id="mwAsU"><span class="mw-linkback-text" id="mwAsY">2</span></a></span> <span id="mw-reference-text-cite_note-:1-12" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt63" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAsc" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite news ","href":"./Template:Cite_news"},"params":{"last":{"wt":""},"first":{"wt":""},"date":{"wt":"2000-07-31"},"title":{"wt":"EBay Fights Spiders on the Web"},"url":{"wt":"https://www.wired.com/2000/07/ebay-fights-spiders-on-the-web/"},"access-date":{"wt":"2024-08-02"},"work":{"wt":"[[Wired (magazine)|Wired]]"},"language":{"wt":"en-US"},"issn":{"wt":"1059-1028"}},"i":0}}]}'/><cite class="citation news cs1" about="#mwt63" id="mwAsg"><a rel="mw:ExtLink nofollow" href="https://www.wired.com/2000/07/ebay-fights-spiders-on-the-web/" class="external text" id="mwAsk">"EBay Fights Spiders on the Web"</a>. <i id="mwAso"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Wired_(magazine)" title="Wired (magazine)" id="mwAss">Wired</a></i>. 2000-07-31. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/ISSN_(identifier)" title="ISSN (identifier)" class="mw-redirect" id="mwAsw">ISSN</a><span typeof="mw:Entity" id="mwAs0"> </span><a rel="mw:ExtLink nofollow" href="https://search.worldcat.org/issn/1059-1028" class="external text" id="mwAs4">1059-1028</a><span class="reference-accessdate" id="mwAs8">. Retrieved <span class="nowrap" id="mwAtA">2024-08-02</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.jtitle=Wired&rft.atitle=EBay+Fights+Spiders+on+the+Web&rft.date=2000-07-31&rft.issn=1059-1028&rft_id=https%3A%2F%2Fwww.wired.com%2F2000%2F07%2Febay-fights-spiders-on-the-web%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt63" id="mwAtE"></span></span></li>
<li about="#cite_note-case-13" id="cite_note-case-13" data-mw-footnote-number="13"><span class="mw-cite-backlink" id="mwAtI"><a href="#cite_ref-case_13-0" rel="mw:referencedBy" id="mwAtM"><span class="mw-linkback-text" id="mwAtQ">↑</span></a></span> <span id="mw-reference-text-cite_note-case-13" class="mw-reference-text reference-text"><cite about="#mwt66" typeof="mw:Transclusion" id="mwAtU" data-mw='{"parts":[{"template":{"target":{"wt":"cite court","href":"./Template:Cite_court"},"params":{"litigants":{"wt":"eBay v. Bidder's Edge"},"vol":{"wt":"100"},"reporter":{"wt":"F. Supp. 2d"},"opinion":{"wt":"1058"},"pinpoint":{"wt":""},"court":{"wt":"[[N.D. Cal.]]"},"date":{"wt":"2000"},"quote":{"wt":""},"url":{"wt":"http://www.cand.uscourts.gov/cand/tentrule.nsf/3979517dd11390ce8825690a007c1b9e/d0fc1406324de0cd882568e90081ebf4/$FILE/Ebay.pdf"},"archive-url":{"wt":"https://web.archive.org/web/20000817173849/http://www.cand.uscourts.gov/cand/tentrule.nsf/3979517dd11390ce8825690a007c1b9e/d0fc1406324de0cd882568e90081ebf4/$FILE/Ebay.pdf"},"url-status":{"wt":"dead"},"accessdate":{"wt":"2000-08-17"}},"i":0}}]}'><i id="mwAtY"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/EBay_v._Bidder's_Edge" title="EBay v. Bidder's Edge" id="mwAtc">eBay v. Bidder's Edge</a></i></cite><span about="#mwt66" id="mwAtg">,</span><span typeof="mw:Entity" about="#mwt66" id="mwAtk"> </span><a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20000817173849/http://www.cand.uscourts.gov/cand/tentrule.nsf/3979517dd11390ce8825690a007c1b9e/d0fc1406324de0cd882568e90081ebf4/$FILE/Ebay.pdf" about="#mwt66" class="external text" id="mwAto">100<span typeof="mw:Entity" id="mwAts"> </span>F. Supp. 2d<span typeof="mw:Entity" id="mwAtw"> </span>1058</a><span typeof="mw:Entity" about="#mwt66" id="mwAt0"> </span><span about="#mwt66" id="mwAt4">(</span><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/N.D._Cal." title="N.D. Cal." about="#mwt66" class="mw-redirect" id="mwAt8">N.D. Cal.</a><span typeof="mw:Entity" about="#mwt66" id="mwAuA"> </span><span about="#mwt66" id="mwAuE">2000),</span><span typeof="mw:Entity" about="#mwt66" id="mwAuI"> </span><span about="#mwt66" id="mwAuM">archived from </span><a rel="mw:ExtLink nofollow" href="http://www.cand.uscourts.gov/cand/tentrule.nsf/3979517dd11390ce8825690a007c1b9e/d0fc1406324de0cd882568e90081ebf4/$FILE/Ebay.pdf" about="#mwt66" class="external text" id="mwAuQ">the original</a><span about="#mwt66" id="mwAuU">.</span></span></li>
<li about="#cite_note-14" id="cite_note-14" data-mw-footnote-number="14"><span class="mw-cite-backlink" id="mwAuY"><a href="#cite_ref-14" rel="mw:referencedBy" id="mwAuc"><span class="mw-linkback-text" id="mwAug">↑</span></a></span> <span id="mw-reference-text-cite_note-14" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt68" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAuk" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"last":{"wt":"Hoffmann"},"first":{"wt":"Jay"},"date":{"wt":"2020-09-15"},"title":{"wt":"Chapter 4: Search"},"url":{"wt":"https://thehistoryoftheweb.com/book/search/"},"access-date":{"wt":"2024-08-02"},"website":{"wt":"The History of the Web"},"language":{"wt":"en-US"}},"i":0}}]}'/><cite id="CITEREFHoffmann2020" class="citation web cs1" about="#mwt68">Hoffmann, Jay (2020-09-15). <a rel="mw:ExtLink nofollow" href="https://thehistoryoftheweb.com/book/search/" class="external text" id="mwAuo">"Chapter 4: Search"</a>. <i id="mwAus">The History of the Web</i><span class="reference-accessdate" id="mwAuw">. Retrieved <span class="nowrap" id="mwAu0">2024-08-02</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=The+History+of+the+Web&rft.atitle=Chapter+4%3A+Search&rft.date=2020-09-15&rft.aulast=Hoffmann&rft.aufirst=Jay&rft_id=https%3A%2F%2Fthehistoryoftheweb.com%2Fbook%2Fsearch%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt68" id="mwAu4"></span></span></li>
<li about="#cite_note-15" id="cite_note-15" data-mw-footnote-number="15"><span class="mw-cite-backlink" id="mwAu8"><a href="#cite_ref-15" rel="mw:referencedBy" id="mwAvA"><span class="mw-linkback-text" id="mwAvE">↑</span></a></span> <span id="mw-reference-text-cite_note-15" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt72" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAvI" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"last":{"wt":"Berry"},"first":{"wt":"Jahna"},"date":{"wt":"July 24, 2001"},"title":{"wt":"Robots in the Hen House"},"url":{"wt":"http://www.law.com/regionals/ca/stories/edt0723_ip_robots.shtml"},"archive-url":{"wt":"https://web.archive.org/web/20110608004415/http://www.law.com/regionals/ca/stories/edt0723_ip_robots.shtml"},"archive-date":{"wt":"2011-06-08"},"accessdate":{"wt":"June 20, 2015"},"work":{"wt":"law.com"}},"i":0}}]}'/><cite id="CITEREFBerry2001" class="citation web cs1" about="#mwt72">Berry, Jahna (July 24, 2001). <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20110608004415/http://www.law.com/regionals/ca/stories/edt0723_ip_robots.shtml" class="external text" id="mwAvM">"Robots in the Hen House"</a>. <i id="mwAvQ">law.com</i>. Archived from <a rel="mw:ExtLink nofollow" href="http://www.law.com/regionals/ca/stories/edt0723_ip_robots.shtml" class="external text" id="mwAvU">the original</a> on 2011-06-08<span class="reference-accessdate" id="mwAvY">. Retrieved <span class="nowrap" id="mwAvc">June 20,</span> 2015</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=law.com&rft.atitle=Robots+in+the+Hen+House&rft.date=2001-07-24&rft.aulast=Berry&rft.aufirst=Jahna&rft_id=http%3A%2F%2Fwww.law.com%2Fregionals%2Fca%2Fstories%2Fedt0723_ip_robots.shtml&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt72" id="mwAvg"></span></span></li>
<li about="#cite_note-16" id="cite_note-16" data-mw-footnote-number="16"><span class="mw-cite-backlink" id="mwAvk"><a href="#cite_ref-16" rel="mw:referencedBy" id="mwAvo"><span class="mw-linkback-text" id="mwAvs">↑</span></a></span> <span id="mw-reference-text-cite_note-16" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt75" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAvw" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"EBay, Bidder's Edge Settle Suits on Web Access"},"url":{"wt":"https://www.latimes.com/archives/la-xpm-2001-mar-02-fi-32241-story.html"},"access-date":{"wt":"June 20, 2015"},"work":{"wt":"latimes"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt75" id="mwAv0"><a rel="mw:ExtLink nofollow" href="https://www.latimes.com/archives/la-xpm-2001-mar-02-fi-32241-story.html" class="external text" id="mwAv4">"EBay, Bidder's Edge Settle Suits on Web Access"</a>. <i id="mwAv8">latimes</i><span class="reference-accessdate" id="mwAwA">. Retrieved <span class="nowrap" id="mwAwE">June 20,</span> 2015</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=latimes&rft.atitle=EBay%2C+Bidder%27s+Edge+Settle+Suits+on+Web+Access&rft_id=https%3A%2F%2Fwww.latimes.com%2Farchives%2Fla-xpm-2001-mar-02-fi-32241-story.html&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt75" id="mwAwI"></span></span></li>
<li about="#cite_note-17" id="cite_note-17" data-mw-footnote-number="17"><span class="mw-cite-backlink" id="mwAwM"><a href="#cite_ref-17" rel="mw:referencedBy" id="mwAwQ"><span class="mw-linkback-text" id="mwAwU">↑</span></a></span> <span id="mw-reference-text-cite_note-17" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt78" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAwY" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite news ","href":"./Template:Cite_news"},"params":{"date":{"wt":"Aug 2, 2007"},"title":{"wt":"Use of web archive was not hacking, says US court"},"url":{"wt":"https://www.theregister.com/2007/08/02/healthcare_advocates_suit/"},"access-date":{"wt":"Oct 22, 2025"},"work":{"wt":"[[The Register]]"}},"i":0}}]}'/><cite class="citation news cs1" about="#mwt78" id="mwAwc"><a rel="mw:ExtLink nofollow" href="https://www.theregister.com/2007/08/02/healthcare_advocates_suit/" class="external text" id="mwAwg">"Use of web archive was not hacking, says US court"</a>. <i id="mwAwk"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/The_Register" title="The Register" id="mwAwo">The Register</a></i>. Aug 2, 2007<span class="reference-accessdate" id="mwAws">. Retrieved <span class="nowrap" id="mwAww">Oct 22,</span> 2025</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.jtitle=The+Register&rft.atitle=Use+of+web+archive+was+not+hacking%2C+says+US+court&rft.date=2007-08-02&rft_id=https%3A%2F%2Fwww.theregister.com%2F2007%2F08%2F02%2Fhealthcare_advocates_suit%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt78" id="mwAw0"></span></span></li>
<li about="#cite_note-18" id="cite_note-18" data-mw-footnote-number="18"><span class="mw-cite-backlink" id="mwAw4"><a href="#cite_ref-18" rel="mw:referencedBy" id="mwAw8"><span class="mw-linkback-text" id="mwAxA">↑</span></a></span> <span id="mw-reference-text-cite_note-18" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt81" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAxE" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"date":{"wt":"July 20, 2007"},"title":{"wt":"Memorandum - Healthcare Advocates v Harding at all"},"url":{"wt":"https://www.govinfo.gov/content/pkg/USCOURTS-paed-2_05-cv-03524/pdf/USCOURTS-paed-2_05-cv-03524-0.pdf"},"access-date":{"wt":"Oct 22, 2025"},"website":{"wt":"govinfo.gov"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt81" id="mwAxI"><a rel="mw:ExtLink nofollow" href="https://www.govinfo.gov/content/pkg/USCOURTS-paed-2_05-cv-03524/pdf/USCOURTS-paed-2_05-cv-03524-0.pdf" class="external text" id="mwAxM">"Memorandum - Healthcare Advocates v Harding at all"</a> <span class="cs1-format" id="mwAxQ">(PDF)</span>. <i id="mwAxU">govinfo.gov</i>. July 20, 2007<span class="reference-accessdate" id="mwAxY">. Retrieved <span class="nowrap" id="mwAxc">Oct 22,</span> 2025</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=govinfo.gov&rft.atitle=Memorandum+-+Healthcare+Advocates+v+Harding+at+all&rft.date=2007-07-20&rft_id=https%3A%2F%2Fwww.govinfo.gov%2Fcontent%2Fpkg%2FUSCOURTS-paed-2_05-cv-03524%2Fpdf%2FUSCOURTS-paed-2_05-cv-03524-0.pdf&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt81" id="mwAxg"></span></span></li>
<li about="#cite_note-19" id="cite_note-19" data-mw-footnote-number="19"><span class="mw-cite-backlink" id="mwAxk"><a href="#cite_ref-19" rel="mw:referencedBy" id="mwAxo"><span class="mw-linkback-text" id="mwAxs">↑</span></a></span> <span id="mw-reference-text-cite_note-19" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt84" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAxw" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"date":{"wt":"July 20, 2007"},"title":{"wt":"Healthcare Advocates, Inc. v. Harding, Earley, Follmer & Frailey"},"url":{"wt":"https://www.courtlistener.com/opinion/1614840/healthcare-advocates-inc-v-harding-earley-follmer-frailey"},"access-date":{"wt":"2025-10-23"},"website":{"wt":"www.courtlistener.com"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt84" id="mwAx0"><a rel="mw:ExtLink nofollow" href="https://www.courtlistener.com/opinion/1614840/healthcare-advocates-inc-v-harding-earley-follmer-frailey" class="external text" id="mwAx4">"Healthcare Advocates, Inc. v. Harding, Earley, Follmer & Frailey"</a>. <i id="mwAx8">www.courtlistener.com</i>. July 20, 2007<span class="reference-accessdate" id="mwAyA">. Retrieved <span class="nowrap" id="mwAyE">2025-10-23</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=www.courtlistener.com&rft.atitle=Healthcare+Advocates%2C+Inc.+v.+Harding%2C+Earley%2C+Follmer+%26+Frailey&rft.date=2007-07-20&rft_id=https%3A%2F%2Fwww.courtlistener.com%2Fopinion%2F1614840%2Fhealthcare-advocates-inc-v-harding-earley-follmer-frailey&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt84" id="mwAyI"></span></span></li>
<li about="#cite_note-20" id="cite_note-20" data-mw-footnote-number="20"><span class="mw-cite-backlink" id="mwAyM"><a href="#cite_ref-20" rel="mw:referencedBy" id="mwAyQ"><span class="mw-linkback-text" id="mwAyU">↑</span></a></span> <span id="mw-reference-text-cite_note-20" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt87" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAyY" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"date":{"wt":"March 21, 2013"},"title":{"wt":"Associated Press v. Meltwater U.S. Holdings, Inc."},"url":{"wt":"https://www.courtlistener.com/opinion/8723643/associated-press-v-meltwater-us-holdings-inc/"},"access-date":{"wt":"2025-10-23"},"website":{"wt":"www.courtlistener.com"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt87" id="mwAyc"><a rel="mw:ExtLink nofollow" href="https://www.courtlistener.com/opinion/8723643/associated-press-v-meltwater-us-holdings-inc/" class="external text" id="mwAyg">"Associated Press v. Meltwater U.S. Holdings, Inc"</a>. <i id="mwAyk">www.courtlistener.com</i>. March 21, 2013<span class="reference-accessdate" id="mwAyo">. Retrieved <span class="nowrap" id="mwAys">2025-10-23</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=www.courtlistener.com&rft.atitle=Associated+Press+v.+Meltwater+U.S.+Holdings%2C+Inc.&rft.date=2013-03-21&rft_id=https%3A%2F%2Fwww.courtlistener.com%2Fopinion%2F8723643%2Fassociated-press-v-meltwater-us-holdings-inc%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt87" id="mwAyw"></span></span></li>
<li about="#cite_note-google-webmasters-spec-21" id="cite_note-google-webmasters-spec-21" data-mw-footnote-number="21"><span rel="mw:referencedBy" class="mw-cite-backlink" id="mwAy0"><a href="#cite_ref-google-webmasters-spec_21-0" id="mwAy4"><span class="mw-linkback-text" id="mwAy8">1</span></a> <a href="#cite_ref-google-webmasters-spec_21-1" id="mwAzA"><span class="mw-linkback-text" id="mwAzE">2</span></a></span> <span id="mw-reference-text-cite_note-google-webmasters-spec-21" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt90" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAzI" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt"},"title":{"wt":"Webmasters: Robots.txt Specifications"},"work":{"wt":"Google Developers"},"access-date":{"wt":"16 February 2013"},"archive-url":{"wt":"https://web.archive.org/web/20130115214137/https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt"},"archive-date":{"wt":"2013-01-15"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt90" id="mwAzM"><a rel="mw:ExtLink nofollow" href="https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt" class="external text" id="mwAzQ">"Webmasters: Robots.txt Specifications"</a>. <i id="mwAzU">Google Developers</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20130115214137/https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt" class="external text" id="mwAzY">Archived</a> from the original on 2013-01-15<span class="reference-accessdate" id="mwAzc">. Retrieved <span class="nowrap" id="mwAzg">16 February</span> 2013</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Google+Developers&rft.atitle=Webmasters%3A+Robots.txt+Specifications&rft_id=https%3A%2F%2Fdevelopers.google.com%2Fwebmasters%2Fcontrol-crawl-index%2Fdocs%2Frobots_txt&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt90" id="mwAzk"></span></span></li>
<li about="#cite_note-bing-blog-robots-22" id="cite_note-bing-blog-robots-22" data-mw-footnote-number="22"><span class="mw-cite-backlink" id="mwAzo"><a href="#cite_ref-bing-blog-robots_22-0" rel="mw:referencedBy" id="mwAzs"><span class="mw-linkback-text" id="mwAzw">↑</span></a></span> <span id="mw-reference-text-cite_note-bing-blog-robots-22" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt93" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAz0" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://blogs.bing.com/webmaster/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation/"},"title":{"wt":"Robots Exclusion Protocol: joining together to provide better documentation"},"website":{"wt":"Blogs.bing.com"},"date":{"wt":"3 June 2008"},"archive-url":{"wt":"https://web.archive.org/web/20140818025412/http://blogs.bing.com/webmaster/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation/"},"archive-date":{"wt":"2014-08-18"},"url-status":{"wt":"live"},"access-date":{"wt":"16 February 2013"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt93" id="mwAz4"><a rel="mw:ExtLink nofollow" href="https://blogs.bing.com/webmaster/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation/" class="external text" id="mwAz8">"Robots Exclusion Protocol: joining together to provide better documentation"</a>. <i id="mwA0A">Blogs.bing.com</i>. 3 June 2008. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20140818025412/http://blogs.bing.com/webmaster/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation/" class="external text" id="mwA0E">Archived</a> from the original on 2014-08-18<span class="reference-accessdate" id="mwA0I">. Retrieved <span class="nowrap" id="mwA0M">16 February</span> 2013</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Blogs.bing.com&rft.atitle=Robots+Exclusion+Protocol%3A+joining+together+to+provide+better+documentation&rft.date=2008-06-03&rft_id=https%3A%2F%2Fblogs.bing.com%2Fwebmaster%2F2008%2F06%2F03%2Frobots-exclusion-protocol-joining-together-to-provide-better-documentation%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt93" id="mwA0Q"></span></span></li>
<li about="#cite_note-yahoo-search-is-bing-23" id="cite_note-yahoo-search-is-bing-23" data-mw-footnote-number="23"><span class="mw-cite-backlink" id="mwA0U"><a href="#cite_ref-yahoo-search-is-bing_23-0" rel="mw:referencedBy" id="mwA0Y"><span class="mw-linkback-text" id="mwA0c">↑</span></a></span> <span id="mw-reference-text-cite_note-yahoo-search-is-bing-23" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt96" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA0g" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://help.yahoo.com/kb/index?page=content&y=PROD_SRCH&locale=en_US&id=SLN2217&impressions=true"},"title":{"wt":"Submitting your website to Yahoo! Search"},"access-date":{"wt":"16 February 2013"},"archive-url":{"wt":"https://web.archive.org/web/20130121035801/http://help.yahoo.com/kb/index?page=content&y=PROD_SRCH&locale=en_US&id=SLN2217&impressions=true"},"archive-date":{"wt":"2013-01-21"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt96" id="mwA0k"><a rel="mw:ExtLink nofollow" href="http://help.yahoo.com/kb/index?page=content&y=PROD_SRCH&locale=en_US&id=SLN2217&impressions=true" class="external text" id="mwA0o">"Submitting your website to Yahoo! Search"</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20130121035801/http://help.yahoo.com/kb/index?page=content&y=PROD_SRCH&locale=en_US&id=SLN2217&impressions=true" class="external text" id="mwA0s">Archived</a> from the original on 2013-01-21<span class="reference-accessdate" id="mwA0w">. Retrieved <span class="nowrap" id="mwA00">16 February</span> 2013</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Submitting+your+website+to+Yahoo%21+Search&rft_id=http%3A%2F%2Fhelp.yahoo.com%2Fkb%2Findex%3Fpage%3Dcontent%26y%3DPROD_SRCH%26locale%3Den_US%26id%3DSLN2217%26impressions%3Dtrue&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt96" id="mwA04"></span></span></li>
<li about="#cite_note-yandex-robots-24" id="cite_note-yandex-robots-24" data-mw-footnote-number="24"><span rel="mw:referencedBy" class="mw-cite-backlink" id="mwA08"><a href="#cite_ref-yandex-robots_24-0" id="mwA1A"><span class="mw-linkback-text" id="mwA1E">1</span></a> <a href="#cite_ref-yandex-robots_24-1" id="mwA1I"><span class="mw-linkback-text" id="mwA1M">2</span></a></span> <span id="mw-reference-text-cite_note-yandex-robots-24" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt99" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA1Q" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://help.yandex.com/webmaster/?id=1113851"},"title":{"wt":"Using robots.txt"},"website":{"wt":"Help.yandex.com"},"access-date":{"wt":"16 February 2013"},"archive-url":{"wt":"https://web.archive.org/web/20130125040017/http://help.yandex.com/webmaster/?id=1113851"},"archive-date":{"wt":"2013-01-25"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt99" id="mwA1U"><a rel="mw:ExtLink nofollow" href="http://help.yandex.com/webmaster/?id=1113851" class="external text" id="mwA1Y">"Using robots.txt"</a>. <i id="mwA1c">Help.yandex.com</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20130125040017/http://help.yandex.com/webmaster/?id=1113851" class="external text" id="mwA1g">Archived</a> from the original on 2013-01-25<span class="reference-accessdate" id="mwA1k">. Retrieved <span class="nowrap" id="mwA1o">16 February</span> 2013</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Help.yandex.com&rft.atitle=Using+robots.txt&rft_id=http%3A%2F%2Fhelp.yandex.com%2Fwebmaster%2F%3Fid%3D1113851&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt99" id="mwA1s"></span></span></li>
<li about="#cite_note-duckduckgo-bot-25" id="cite_note-duckduckgo-bot-25" data-mw-footnote-number="25"><span class="mw-cite-backlink" id="mwA1w"><a href="#cite_ref-duckduckgo-bot_25-0" rel="mw:referencedBy" id="mwA10"><span class="mw-linkback-text" id="mwA14">↑</span></a></span> <span id="mw-reference-text-cite_note-duckduckgo-bot-25" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt102" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA18" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://duckduckgo.com/duckduckbot"},"website":{"wt":"DuckDuckGo.com"},"title":{"wt":"DuckDuckGo Bot"},"access-date":{"wt":"25 April 2017"},"archive-date":{"wt":"16 February 2017"},"archive-url":{"wt":"https://web.archive.org/web/20170216043103/https://duckduckgo.com/duckduckbot"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt102" id="mwA2A"><a rel="mw:ExtLink nofollow" href="https://duckduckgo.com/duckduckbot" class="external text" id="mwA2E">"DuckDuckGo Bot"</a>. <i id="mwA2I">DuckDuckGo.com</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20170216043103/https://duckduckgo.com/duckduckbot" class="external text" id="mwA2M">Archived</a> from the original on 16 February 2017<span class="reference-accessdate" id="mwA2Q">. Retrieved <span class="nowrap" id="mwA2U">25 April</span> 2017</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=DuckDuckGo.com&rft.atitle=DuckDuckGo+Bot&rft_id=https%3A%2F%2Fduckduckgo.com%2Fduckduckbot&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt102" id="mwA2Y"></span></span></li>
<li about="#cite_note-baidu-spider-26" id="cite_note-baidu-spider-26" data-mw-footnote-number="26"><span class="mw-cite-backlink" id="mwA2c"><a href="#cite_ref-baidu-spider_26-0" rel="mw:referencedBy" id="mwA2g"><span class="mw-linkback-text" id="mwA2k">↑</span></a></span> <span id="mw-reference-text-cite_note-baidu-spider-26" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt105" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA2o" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"Baiduspider"},"url":{"wt":"http://www.baidu.com/search/spider_english.html"},"website":{"wt":"Baidu.com"},"access-date":{"wt":"16 February 2013"},"archive-date":{"wt":"6 August 2013"},"archive-url":{"wt":"https://web.archive.org/web/20130806131031/http://www.baidu.com/search/spider_english.html"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt105" id="mwA2s"><a rel="mw:ExtLink nofollow" href="http://www.baidu.com/search/spider_english.html" class="external text" id="mwA2w">"Baiduspider"</a>. <i id="mwA20">Baidu.com</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20130806131031/http://www.baidu.com/search/spider_english.html" class="external text" id="mwA24">Archived</a> from the original on 6 August 2013<span class="reference-accessdate" id="mwA28">. Retrieved <span class="nowrap" id="mwA3A">16 February</span> 2013</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Baidu.com&rft.atitle=Baiduspider&rft_id=http%3A%2F%2Fwww.baidu.com%2Fsearch%2Fspider_english.html&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt105" id="mwA3E"></span></span></li>
<li about="#cite_note-about-aol-search-27" id="cite_note-about-aol-search-27" data-mw-footnote-number="27"><span class="mw-cite-backlink" id="mwA3I"><a href="#cite_ref-about-aol-search_27-0" rel="mw:referencedBy" id="mwA3M"><span class="mw-linkback-text" id="mwA3Q">↑</span></a></span> <span id="mw-reference-text-cite_note-about-aol-search-27" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt108" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA3U" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"About AOL Search"},"url":{"wt":"http://search.aol.com/aol/about"},"website":{"wt":"Search.aol.com"},"access-date":{"wt":"16 February 2013"},"archive-date":{"wt":"13 December 2012"},"archive-url":{"wt":"https://web.archive.org/web/20121213134546/http://search.aol.com/aol/about"},"url-status":{"wt":"dead"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt108" id="mwA3Y"><a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20121213134546/http://search.aol.com/aol/about" class="external text" id="mwA3c">"About AOL Search"</a>. <i id="mwA3g">Search.aol.com</i>. Archived from <a rel="mw:ExtLink nofollow" href="http://search.aol.com/aol/about" class="external text" id="mwA3k">the original</a> on 13 December 2012<span class="reference-accessdate" id="mwA3o">. Retrieved <span class="nowrap" id="mwA3s">16 February</span> 2013</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Search.aol.com&rft.atitle=About+AOL+Search&rft_id=http%3A%2F%2Fsearch.aol.com%2Faol%2Fabout&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt108" id="mwA3w"></span></span></li>
<li about="#cite_note-ask-webmasters-28" id="cite_note-ask-webmasters-28" data-mw-footnote-number="28"><span class="mw-cite-backlink" id="mwA30"><a href="#cite_ref-ask-webmasters_28-0" rel="mw:referencedBy" id="mwA34"><span class="mw-linkback-text" id="mwA38">↑</span></a></span> <span id="mw-reference-text-cite_note-ask-webmasters-28" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt111" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA4A" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"About Ask.com: Webmasters"},"url":{"wt":"http://about.ask.com/docs/about/webmasters.shtml"},"website":{"wt":"About.ask.com"},"access-date":{"wt":"16 February 2013"},"archive-date":{"wt":"27 January 2013"},"archive-url":{"wt":"https://web.archive.org/web/20130127134025/http://about.ask.com/docs/about/webmasters.shtml"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt111" id="mwA4E"><a rel="mw:ExtLink nofollow" href="http://about.ask.com/docs/about/webmasters.shtml" class="external text" id="mwA4I">"About Ask.com: Webmasters"</a>. <i id="mwA4M">About.ask.com</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20130127134025/http://about.ask.com/docs/about/webmasters.shtml" class="external text" id="mwA4Q">Archived</a> from the original on 27 January 2013<span class="reference-accessdate" id="mwA4U">. Retrieved <span class="nowrap" id="mwA4Y">16 February</span> 2013</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=About.ask.com&rft.atitle=About+Ask.com%3A+Webmasters&rft_id=http%3A%2F%2Fabout.ask.com%2Fdocs%2Fabout%2Fwebmasters.shtml&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt111" id="mwA4c"></span></span></li>
<li about="#cite_note-KagiBot-29" id="cite_note-KagiBot-29" data-mw-footnote-number="29"><span class="mw-cite-backlink" id="mwA4g"><a href="#cite_ref-KagiBot_29-0" rel="mw:referencedBy" id="mwA4k"><span class="mw-linkback-text" id="mwA4o">↑</span></a></span> <span id="mw-reference-text-cite_note-KagiBot-29" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt114" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA4s" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://kagi.com/bot"},"website":{"wt":"Kagi Search"},"title":{"wt":"Kagi Search KagiBot"},"access-date":{"wt":"20 November 2024"},"archive-date":{"wt":"12 April 2024"},"archive-url":{"wt":"https://web.archive.org/web/20240412192855/https://kagi.com/bot"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt114" id="mwA4w"><a rel="mw:ExtLink nofollow" href="https://kagi.com/bot" class="external text" id="mwA40">"Kagi Search KagiBot"</a>. <i id="mwA44">Kagi Search</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20240412192855/https://kagi.com/bot" class="external text" id="mwA48">Archived</a> from the original on 12 April 2024<span class="reference-accessdate" id="mwA5A">. Retrieved <span class="nowrap" id="mwA5E">20 November</span> 2024</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Kagi+Search&rft.atitle=Kagi+Search+KagiBot&rft_id=https%3A%2F%2Fkagi.com%2Fbot&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt114" id="mwA5I"></span></span></li>
<li about="#cite_note-30" id="cite_note-30" data-mw-footnote-number="30"><span class="mw-cite-backlink" id="mwA5M"><a href="#cite_ref-30" rel="mw:referencedBy" id="mwA5Q"><span class="mw-linkback-text" id="mwA5U">↑</span></a></span> <span id="mw-reference-text-cite_note-30" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt117" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA5Y" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"ArchiveBot: Bad behavior"},"url":{"wt":"https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior"},"website":{"wt":"wiki.archiveteam.org"},"publisher":{"wt":"Archive Team"},"access-date":{"wt":"10 October 2022"},"archive-date":{"wt":"10 October 2022"},"archive-url":{"wt":"https://web.archive.org/web/20221010034711/https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt117" id="mwA5c"><a rel="mw:ExtLink nofollow" href="https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior" class="external text" id="mwA5g">"ArchiveBot: Bad behavior"</a>. <i id="mwA5k">wiki.archiveteam.org</i>. Archive Team. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20221010034711/https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior" class="external text" id="mwA5o">Archived</a> from the original on 10 October 2022<span class="reference-accessdate" id="mwA5s">. Retrieved <span class="nowrap" id="mwA5w">10 October</span> 2022</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=wiki.archiveteam.org&rft.atitle=ArchiveBot%3A+Bad+behavior&rft_id=https%3A%2F%2Fwiki.archiveteam.org%2Findex.php%2FArchiveBot%23Bad_behavior&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt117" id="mwA50"></span></span></li>
<li about="#cite_note-31" id="cite_note-31" data-mw-footnote-number="31"><span class="mw-cite-backlink" id="mwA54"><a href="#cite_ref-31" rel="mw:referencedBy" id="mwA58"><span class="mw-linkback-text" id="mwA6A">↑</span></a></span> <span id="mw-reference-text-cite_note-31" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt120" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA6E" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://www.archiveteam.org/index.php?title=Robots.txt"},"publisher":{"wt":"Archive Team"},"title":{"wt":"Robots.txt is a suicide note"},"author":{"wt":"Jason Scott"},"access-date":{"wt":"18 February 2017"},"archive-url":{"wt":"https://web.archive.org/web/20170218044527/http://www.archiveteam.org/index.php?title=Robots.txt"},"archive-date":{"wt":"2017-02-18"},"url-status":{"wt":"live"},"author-link":{"wt":"Jason Scott"}},"i":0}}]}'/><cite id="CITEREFJason_Scott" class="citation web cs1" about="#mwt120"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Jason_Scott" title="Jason Scott" id="mwA6I">Jason Scott</a>. <a rel="mw:ExtLink nofollow" href="http://www.archiveteam.org/index.php?title=Robots.txt" class="external text" id="mwA6M">"Robots.txt is a suicide note"</a>. Archive Team. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20170218044527/http://www.archiveteam.org/index.php?title=Robots.txt" class="external text" id="mwA6Q">Archived</a> from the original on 2017-02-18<span class="reference-accessdate" id="mwA6U">. Retrieved <span class="nowrap" id="mwA6Y">18 February</span> 2017</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Robots.txt+is+a+suicide+note&rft.pub=Archive+Team&rft.au=Jason+Scott&rft_id=http%3A%2F%2Fwww.archiveteam.org%2Findex.php%3Ftitle%3DRobots.txt&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt120" id="mwA6c"></span></span></li>
<li about="#cite_note-32" id="cite_note-32" data-mw-footnote-number="32"><span class="mw-cite-backlink" id="mwA6g"><a href="#cite_ref-32" rel="mw:referencedBy" id="mwA6k"><span class="mw-linkback-text" id="mwA6o">↑</span></a></span> <span id="mw-reference-text-cite_note-32" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt123" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA6s" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/"},"title":{"wt":"Robots.txt meant for search engines don't work well for web archives {{!}} Internet Archive Blogs"},"website":{"wt":"blog.archive.org"},"date":{"wt":"17 April 2017"},"language":{"wt":"en-US"},"access-date":{"wt":"2018-12-01"},"archive-url":{"wt":"https://web.archive.org/web/20181204130028/http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/"},"archive-date":{"wt":"2018-12-04"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt123" id="mwA6w"><a rel="mw:ExtLink nofollow" href="https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/" class="external text" id="mwA60">"Robots.txt meant for search engines don't work well for web archives | Internet Archive Blogs"</a>. <i id="mwA64">blog.archive.org</i>. 17 April 2017. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20181204130028/http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/" class="external text" id="mwA68">Archived</a> from the original on 2018-12-04<span class="reference-accessdate" id="mwA7A">. Retrieved <span class="nowrap" id="mwA7E">2018-12-01</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=blog.archive.org&rft.atitle=Robots.txt+meant+for+search+engines+don%27t+work+well+for+web+archives+%7C+Internet+Archive+Blogs&rft.date=2017-04-17&rft_id=https%3A%2F%2Fblog.archive.org%2F2017%2F04%2F17%2Frobots-txt-meant-for-search-engines-dont-work-well-for-web-archives%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt123" id="mwA7I"></span></span></li>
<li about="#cite_note-Internet_Archive-33" id="cite_note-Internet_Archive-33" data-mw-footnote-number="33"><span class="mw-cite-backlink" id="mwA7M"><a href="#cite_ref-Internet_Archive_33-0" rel="mw:referencedBy" id="mwA7Q"><span class="mw-linkback-text" id="mwA7U">↑</span></a></span> <span id="mw-reference-text-cite_note-Internet_Archive-33" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt127" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA7Y" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite news ","href":"./Template:Cite_news"},"params":{"url":{"wt":"https://www.digitaltrends.com/computing/internet-archive-robots-txt/#ixzz4gQYOqpUi"},"title":{"wt":"The Internet Archive Will Ignore Robots.txt Files to Maintain Accuracy"},"newspaper":{"wt":"[[Digital Trends]]"},"first":{"wt":"Brad"},"last":{"wt":"Jones"},"date":{"wt":"24 April 2017"},"access-date":{"wt":"8 May 2017"},"archive-url":{"wt":"https://web.archive.org/web/20170516130029/https://www.digitaltrends.com/computing/internet-archive-robots-txt/#ixzz4gQYOqpUi"},"archive-date":{"wt":"2017-05-16"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite id="CITEREFJones2017" class="citation news cs1" about="#mwt127">Jones, Brad (24 April 2017). <a rel="mw:ExtLink nofollow" href="https://www.digitaltrends.com/computing/internet-archive-robots-txt/#ixzz4gQYOqpUi" class="external text" id="mwA7c">"The Internet Archive Will Ignore Robots.txt Files to Maintain Accuracy"</a>. <i id="mwA7g"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Digital_Trends" title="Digital Trends" id="mwA7k">Digital Trends</a></i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20170516130029/https://www.digitaltrends.com/computing/internet-archive-robots-txt/#ixzz4gQYOqpUi" class="external text" id="mwA7o">Archived</a> from the original on 2017-05-16<span class="reference-accessdate" id="mwA7s">. Retrieved <span class="nowrap" id="mwA7w">8 May</span> 2017</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.jtitle=Digital+Trends&rft.atitle=The+Internet+Archive+Will+Ignore+Robots.txt+Files+to+Maintain+Accuracy&rft.date=2017-04-24&rft.aulast=Jones&rft.aufirst=Brad&rft_id=https%3A%2F%2Fwww.digitaltrends.com%2Fcomputing%2Finternet-archive-robots-txt%2F%23ixzz4gQYOqpUi&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt127" id="mwA70"></span></span></li>
<li about="#cite_note-34" id="cite_note-34" data-mw-footnote-number="34"><span class="mw-cite-backlink" id="mwA74"><a href="#cite_ref-34" rel="mw:referencedBy" id="mwA78"><span class="mw-linkback-text" id="mwA8A">↑</span></a></span> <span id="mw-reference-text-cite_note-34" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt133" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA8E" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"last":{"wt":"Koebler"},"first":{"wt":"Jason"},"date":{"wt":"2024-07-29"},"title":{"wt":"Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)"},"url":{"wt":"https://www.404media.co/websites-are-blocking-the-wrong-ai-scrapers-because-ai-companies-keep-making-new-ones/"},"access-date":{"wt":"2024-07-29"},"website":{"wt":"404 Media"}},"i":0}}]}'/><cite id="CITEREFKoebler2024" class="citation web cs1" about="#mwt133">Koebler, Jason (2024-07-29). <a rel="mw:ExtLink nofollow" href="https://www.404media.co/websites-are-blocking-the-wrong-ai-scrapers-because-ai-companies-keep-making-new-ones/" class="external text" id="mwA8I">"Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)"</a>. <i id="mwA8M">404 Media</i><span class="reference-accessdate" id="mwA8Q">. Retrieved <span class="nowrap" id="mwA8U">2024-07-29</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=404+Media&rft.atitle=Websites+are+Blocking+the+Wrong+AI+Scrapers+%28Because+AI+Companies+Keep+Making+New+Ones%29&rft.date=2024-07-29&rft.aulast=Koebler&rft.aufirst=Jason&rft_id=https%3A%2F%2Fwww.404media.co%2Fwebsites-are-blocking-the-wrong-ai-scrapers-because-ai-companies-keep-making-new-ones%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt133" id="mwA8Y"></span></span></li>
<li about="#cite_note-35" id="cite_note-35" data-mw-footnote-number="35"><span class="mw-cite-backlink" id="mwA8c"><a href="#cite_ref-35" rel="mw:referencedBy" id="mwA8g"><span class="mw-linkback-text" id="mwA8k">↑</span></a></span> <span id="mw-reference-text-cite_note-35" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt136" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA8o" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web\n","href":"./Template:Cite_web"},"params":{"title":{"wt":"Bot Directory"},"url":{"wt":"https://radar.cloudflare.com/bots/directory"},"website":{"wt":"Cloudflare Radar"},"publisher":{"wt":"Cloudflare"},"access-date":{"wt":"2 August 2026"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt136" id="mwA8s"><a rel="mw:ExtLink nofollow" href="https://radar.cloudflare.com/bots/directory" class="external text" id="mwA8w">"Bot Directory"</a>. <i id="mwA80">Cloudflare Radar</i>. Cloudflare<span class="reference-accessdate" id="mwA84">. Retrieved <span class="nowrap" id="mwA88">2 August</span> 2026</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Cloudflare+Radar&rft.atitle=Bot+Directory&rft_id=https%3A%2F%2Fradar.cloudflare.com%2Fbots%2Fdirectory&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt136" id="mwA9A"></span><span class="cs1-maint citation-comment" about="#mwt136" id="mwA9E"><code class="cs1-code" id="mwA9I">{{<a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Template:Cite_web" title="Template:Cite web" id="mwA9M">cite web</a>}}</code>: CS1 maint: url-status (<a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Category:CS1_maint:_url-status" title="Category:CS1 maint: url-status" id="mwA9Q">link</a>)</span><link rel="mw:PageProp/Category" href="./Category:CS1_maint:_url-status" about="#mwt136" id="mwA9U"/></span></li>
<li about="#cite_note-tc-10sep2025-36" id="cite_note-tc-10sep2025-36" data-mw-footnote-number="36"><span class="mw-cite-backlink" id="mwA9Y"><a href="#cite_ref-tc-10sep2025_36-0" rel="mw:referencedBy" id="mwA9c"><span class="mw-linkback-text" id="mwA9g">↑</span></a></span> <span id="mw-reference-text-cite_note-tc-10sep2025-36" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt140" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA9k" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite news ","href":"./Template:Cite_news"},"params":{"last1":{"wt":"Brandom"},"first1":{"wt":"Russell"},"title":{"wt":"RSS co-creator launches new protocol for AI data licensing"},"url":{"wt":"https://techcrunch.com/2025/09/10/rss-co-creator-launches-new-protocol-for-ai-data-licensing/"},"access-date":{"wt":"September 10, 2025"},"work":{"wt":"[[TechCrunch]]"},"date":{"wt":"September 10, 2025"}},"i":0}}]}'/><cite id="CITEREFBrandom2025" class="citation news cs1" about="#mwt140">Brandom, Russell (September 10, 2025). <a rel="mw:ExtLink nofollow" href="https://techcrunch.com/2025/09/10/rss-co-creator-launches-new-protocol-for-ai-data-licensing/" class="external text" id="mwA9o">"RSS co-creator launches new protocol for AI data licensing"</a>. <i id="mwA9s"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/TechCrunch" title="TechCrunch" id="mwA9w">TechCrunch</a></i><span class="reference-accessdate" id="mwA90">. Retrieved <span class="nowrap" id="mwA94">September 10,</span> 2025</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.jtitle=TechCrunch&rft.atitle=RSS+co-creator+launches+new+protocol+for+AI+data+licensing&rft.date=2025-09-10&rft.aulast=Brandom&rft.aufirst=Russell&rft_id=https%3A%2F%2Ftechcrunch.com%2F2025%2F09%2F10%2Frss-co-creator-launches-new-protocol-for-ai-data-licensing%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt140" id="mwA98"></span></span></li>
<li about="#cite_note-verge-10sep2025-37" id="cite_note-verge-10sep2025-37" data-mw-footnote-number="37"><span class="mw-cite-backlink" id="mwA-A"><a href="#cite_ref-verge-10sep2025_37-0" rel="mw:referencedBy" id="mwA-E"><span class="mw-linkback-text" id="mwA-I">↑</span></a></span> <span id="mw-reference-text-cite_note-verge-10sep2025-37" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt143" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA-M" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite news ","href":"./Template:Cite_news"},"params":{"last1":{"wt":"Roth"},"first1":{"wt":"Emma"},"title":{"wt":"The web has a new system for making AI companies pay up"},"url":{"wt":"https://www.theverge.com/news/775072/rsl-standard-licensing-ai-publishing-reddit-yahoo-medium"},"access-date":{"wt":"September 10, 2025"},"work":{"wt":"[[The Verge]]"},"date":{"wt":"September 10, 2025"}},"i":0}}]}'/><cite id="CITEREFRoth2025" class="citation news cs1" about="#mwt143">Roth, Emma (September 10, 2025). <a rel="mw:ExtLink nofollow" href="https://www.theverge.com/news/775072/rsl-standard-licensing-ai-publishing-reddit-yahoo-medium" class="external text" id="mwA-Q">"The web has a new system for making AI companies pay up"</a>. <i id="mwA-U"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/The_Verge" title="The Verge" id="mwA-Y">The Verge</a></i><span class="reference-accessdate" id="mwA-c">. Retrieved <span class="nowrap" id="mwA-g">September 10,</span> 2025</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.jtitle=The+Verge&rft.atitle=The+web+has+a+new+system+for+making+AI+companies+pay+up&rft.date=2025-09-10&rft.aulast=Roth&rft.aufirst=Emma&rft_id=https%3A%2F%2Fwww.theverge.com%2Fnews%2F775072%2Frsl-standard-licensing-ai-publishing-reddit-yahoo-medium&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt143" id="mwA-k"></span></span></li>
<li about="#cite_note-eg-10sep2025-38" id="cite_note-eg-10sep2025-38" data-mw-footnote-number="38"><span class="mw-cite-backlink" id="mwA-o"><a href="#cite_ref-eg-10sep2025_38-0" rel="mw:referencedBy" id="mwA-s"><span class="mw-linkback-text" id="mwA-w">↑</span></a></span> <span id="mw-reference-text-cite_note-eg-10sep2025-38" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt146" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA-0" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite news ","href":"./Template:Cite_news"},"params":{"last1":{"wt":"Shanklin"},"first1":{"wt":"Will"},"title":{"wt":"Reddit, Yahoo, Medium and more are adopting a new licensing standard to get compensated for AI scraping"},"url":{"wt":"https://www.engadget.com/ai/reddit-yahoo-medium-and-more-are-adopting-a-new-licensing-standard-to-get-compensated-for-ai-scraping-180946671.html"},"access-date":{"wt":"September 10, 2025"},"work":{"wt":"[[Engadget]]"},"date":{"wt":"September 10, 2025"}},"i":0}}]}'/><cite id="CITEREFShanklin2025" class="citation news cs1" about="#mwt146">Shanklin, Will (September 10, 2025). <a rel="mw:ExtLink nofollow" href="https://www.engadget.com/ai/reddit-yahoo-medium-and-more-are-adopting-a-new-licensing-standard-to-get-compensated-for-ai-scraping-180946671.html" class="external text" id="mwA-4">"Reddit, Yahoo, Medium and more are adopting a new licensing standard to get compensated for AI scraping"</a>. <i id="mwA-8"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Engadget" title="Engadget" id="mwA_A">Engadget</a></i><span class="reference-accessdate" id="mwA_E">. Retrieved <span class="nowrap" id="mwA_I">September 10,</span> 2025</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.jtitle=Engadget&rft.atitle=Reddit%2C+Yahoo%2C+Medium+and+more+are+adopting+a+new+licensing+standard+to+get+compensated+for+AI+scraping&rft.date=2025-09-10&rft.aulast=Shanklin&rft.aufirst=Will&rft_id=https%3A%2F%2Fwww.engadget.com%2Fai%2Freddit-yahoo-medium-and-more-are-adopting-a-new-licensing-standard-to-get-compensated-for-ai-scraping-180946671.html&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt146" id="mwA_M"></span></span></li>
<li about="#cite_note-39" id="cite_note-39" data-mw-footnote-number="39"><span class="mw-cite-backlink" id="mwA_Q"><a href="#cite_ref-39" rel="mw:referencedBy" id="mwA_U"><span class="mw-linkback-text" id="mwA_Y">↑</span></a></span> <span id="mw-reference-text-cite_note-39" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt149" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwA_c" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"Block URLs with robots.txt: Learn about robots.txt files"},"url":{"wt":"https://support.google.com/webmasters/answer/6062608"},"access-date":{"wt":"2015-08-10"},"archive-url":{"wt":"https://web.archive.org/web/20150814013400/https://support.google.com/webmasters/answer/6062608"},"archive-date":{"wt":"2015-08-14"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt149" id="mwA_g"><a rel="mw:ExtLink nofollow" href="https://support.google.com/webmasters/answer/6062608" class="external text" id="mwA_k">"Block URLs with robots.txt: Learn about robots.txt files"</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20150814013400/https://support.google.com/webmasters/answer/6062608" class="external text" id="mwA_o">Archived</a> from the original on 2015-08-14<span class="reference-accessdate" id="mwA_s">. Retrieved <span class="nowrap" id="mwA_w">2015-08-10</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Block+URLs+with+robots.txt%3A+Learn+about+robots.txt+files&rft_id=https%3A%2F%2Fsupport.google.com%2Fwebmasters%2Fanswer%2F6062608&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt149" id="mwA_0"></span></span></li>
<li about="#cite_note-40" id="cite_note-40" data-mw-footnote-number="40"><span class="mw-cite-backlink" id="mwA_4"><a href="#cite_ref-40" rel="mw:referencedBy" id="mwA_8"><span class="mw-linkback-text" id="mwBAA">↑</span></a></span> <span id="mw-reference-text-cite_note-40" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt152" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBAE" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://www.theregister.co.uk/2015/05/19/robotstxt/"},"title":{"wt":"Robots.txt tells hackers the places you don't want them to look"},"work":{"wt":"The Register"},"access-date":{"wt":"August 12, 2015"},"archive-url":{"wt":"https://web.archive.org/web/20150821063759/http://www.theregister.co.uk/2015/05/19/robotstxt/"},"archive-date":{"wt":"2015-08-21"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt152" id="mwBAI"><a rel="mw:ExtLink nofollow" href="https://www.theregister.co.uk/2015/05/19/robotstxt/" class="external text" id="mwBAM">"Robots.txt tells hackers the places you don't want them to look"</a>. <i id="mwBAQ">The Register</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20150821063759/http://www.theregister.co.uk/2015/05/19/robotstxt/" class="external text" id="mwBAU">Archived</a> from the original on 2015-08-21<span class="reference-accessdate" id="mwBAY">. Retrieved <span class="nowrap" id="mwBAc">August 12,</span> 2015</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=The+Register&rft.atitle=Robots.txt+tells+hackers+the+places+you+don%27t+want+them+to+look&rft_id=https%3A%2F%2Fwww.theregister.co.uk%2F2015%2F05%2F19%2Frobotstxt%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt152" id="mwBAg"></span></span></li>
<li about="#cite_note-41" id="cite_note-41" data-mw-footnote-number="41"><span class="mw-cite-backlink" id="mwBAk"><a href="#cite_ref-41" rel="mw:referencedBy" id="mwBAo"><span class="mw-linkback-text" id="mwBAs">↑</span></a></span> <span id="mw-reference-text-cite_note-41" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt155" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBAw" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite journal ","href":"./Template:Cite_journal"},"params":{"last1":{"wt":"Scarfone"},"first1":{"wt":"K. A."},"last2":{"wt":"Jansen"},"first2":{"wt":"W."},"last3":{"wt":"Tracy"},"first3":{"wt":"M."},"date":{"wt":"July 2008"},"title":{"wt":"Guide to General Server Security"},"url":{"wt":"http://csrc.nist.gov/publications/nistpubs/800-123/SP800-123.pdf"},"url-status":{"wt":"live"},"journal":{"wt":"National Institute of Standards and Technology"},"doi":{"wt":"10.6028/NIST.SP.800-123"},"archive-url":{"wt":"https://web.archive.org/web/20111008115412/http://csrc.nist.gov/publications/nistpubs/800-123/SP800-123.pdf"},"archive-date":{"wt":"2011-10-08"},"access-date":{"wt":"August 12, 2015"}},"i":0}}]}'/><cite id="CITEREFScarfoneJansenTracy2008" class="citation journal cs1" about="#mwt155">Scarfone, K. A.; Jansen, W.; Tracy, M. (July 2008). <a rel="mw:ExtLink nofollow" href="http://csrc.nist.gov/publications/nistpubs/800-123/SP800-123.pdf" class="external text" id="mwBA0">"Guide to General Server Security"</a> <span class="cs1-format" id="mwBA4">(PDF)</span>. <i id="mwBA8">National Institute of Standards and Technology</i>. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Doi_(identifier)" title="Doi (identifier)" class="mw-redirect" id="mwBBA">doi</a>:<a rel="mw:ExtLink nofollow" href="https://doi.org/10.6028%2FNIST.SP.800-123" class="external text" id="mwBBE">10.6028/NIST.SP.800-123</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20111008115412/http://csrc.nist.gov/publications/nistpubs/800-123/SP800-123.pdf" class="external text" id="mwBBI">Archived</a> <span class="cs1-format" id="mwBBM">(PDF)</span> from the original on 2011-10-08<span class="reference-accessdate" id="mwBBQ">. Retrieved <span class="nowrap" id="mwBBU">August 12,</span> 2015</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.jtitle=National+Institute+of+Standards+and+Technology&rft.atitle=Guide+to+General+Server+Security&rft.date=2008-07&rft_id=info%3Adoi%2F10.6028%2FNIST.SP.800-123&rft.aulast=Scarfone&rft.aufirst=K.+A.&rft.au=Jansen%2C+W.&rft.au=Tracy%2C+M.&rft_id=http%3A%2F%2Fcsrc.nist.gov%2Fpublications%2Fnistpubs%2F800-123%2FSP800-123.pdf&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt155" id="mwBBY"></span></span></li>
<li about="#cite_note-42" id="cite_note-42" data-mw-footnote-number="42"><span class="mw-cite-backlink" id="mwBBc"><a href="#cite_ref-42" rel="mw:referencedBy" id="mwBBg"><span class="mw-linkback-text" id="mwBBk">↑</span></a></span> <span id="mw-reference-text-cite_note-42" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt158" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBBo" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite book ","href":"./Template:Cite_book"},"params":{"author":{"wt":"Sverre H. Huseby"},"title":{"wt":"Innocent Code: A Security Wake-Up Call for Web Programmers"},"publisher":{"wt":"John Wiley & Sons"},"year":{"wt":"2004"},"pages":{"wt":"91–92"},"isbn":{"wt":"9780470857472"},"url":{"wt":"https://books.google.com/books?id=RjVjgPQsKogC&pg=PA92"},"access-date":{"wt":"2015-08-12"},"archive-url":{"wt":"https://web.archive.org/web/20160401193437/https://books.google.com/books?id=RjVjgPQsKogC&pg=PA92"},"archive-date":{"wt":"2016-04-01"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite id="CITEREFSverre_H._Huseby2004" class="citation book cs1" about="#mwt158">Sverre H. Huseby (2004). <a rel="mw:ExtLink nofollow" href="https://books.google.com/books?id=RjVjgPQsKogC&pg=PA92" class="external text" id="mwBBs"><i id="mwBBw">Innocent Code: A Security Wake-Up Call for Web Programmers</i></a>. John Wiley & Sons. pp.<span typeof="mw:Entity" id="mwBB0"> </span><span class="nowrap" id="mwBB4">91–</span>92. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/ISBN_(identifier)" title="ISBN (identifier)" class="mw-redirect" id="mwBB8">ISBN</a><span typeof="mw:Entity" id="mwBCA"> </span><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Special:BookSources/9780470857472" title="Special:BookSources/9780470857472" id="mwBCE"><bdi id="mwBCI">9780470857472</bdi></a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20160401193437/https://books.google.com/books?id=RjVjgPQsKogC&pg=PA92" class="external text" id="mwBCM">Archived</a> from the original on 2016-04-01<span class="reference-accessdate" id="mwBCQ">. Retrieved <span class="nowrap" id="mwBCU">2015-08-12</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Innocent+Code%3A+A+Security+Wake-Up+Call+for+Web+Programmers&rft.pages=91-92&rft.pub=John+Wiley+%26+Sons&rft.date=2004&rft.isbn=9780470857472&rft.au=Sverre+H.+Huseby&rft_id=https%3A%2F%2Fbooks.google.com%2Fbooks%3Fid%3DRjVjgPQsKogC%26pg%3DPA92&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt158" id="mwBCY"></span></span></li>
<li about="#cite_note-43" id="cite_note-43" data-mw-footnote-number="43"><span class="mw-cite-backlink" id="mwBCc"><a href="#cite_ref-43" rel="mw:referencedBy" id="mwBCg"><span class="mw-linkback-text" id="mwBCk">↑</span></a></span> <span id="mw-reference-text-cite_note-43" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt161" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBCo" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://www.user-agents.org/"},"title":{"wt":"List of User-Agents (Spiders, Robots, Browser)"},"publisher":{"wt":"User-agents.org"},"access-date":{"wt":"2013-12-29"},"archive-url":{"wt":"https://web.archive.org/web/20140107154205/http://user-agents.org/"},"archive-date":{"wt":"2014-01-07"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt161" id="mwBCs"><a rel="mw:ExtLink nofollow" href="http://www.user-agents.org/" class="external text" id="mwBCw">"List of User-Agents (Spiders, Robots, Browser)"</a>. User-agents.org. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20140107154205/http://user-agents.org/" class="external text" id="mwBC0">Archived</a> from the original on 2014-01-07<span class="reference-accessdate" id="mwBC4">. Retrieved <span class="nowrap" id="mwBC8">2013-12-29</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=List+of+User-Agents+%28Spiders%2C+Robots%2C+Browser%29&rft.pub=User-agents.org&rft_id=http%3A%2F%2Fwww.user-agents.org%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt161" id="mwBDA"></span></span></li>
<li about="#cite_note-44" id="cite_note-44" data-mw-footnote-number="44"><span class="mw-cite-backlink" id="mwBDE"><a href="#cite_ref-44" rel="mw:referencedBy" id="mwBDI"><span class="mw-linkback-text" id="mwBDM">↑</span></a></span> <span id="mw-reference-text-cite_note-44" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt164" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBDQ" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://httpd.apache.org/docs/2.2/howto/access.html"},"title":{"wt":"Access Control - Apache HTTP Server"},"publisher":{"wt":"Httpd.apache.org"},"access-date":{"wt":"2013-12-29"},"archive-url":{"wt":"https://web.archive.org/web/20131229110831/http://httpd.apache.org/docs/2.2/howto/access.html"},"archive-date":{"wt":"2013-12-29"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt164" id="mwBDU"><a rel="mw:ExtLink nofollow" href="https://httpd.apache.org/docs/2.2/howto/access.html" class="external text" id="mwBDY">"Access Control - Apache HTTP Server"</a>. Httpd.apache.org. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20131229110831/http://httpd.apache.org/docs/2.2/howto/access.html" class="external text" id="mwBDc">Archived</a> from the original on 2013-12-29<span class="reference-accessdate" id="mwBDg">. Retrieved <span class="nowrap" id="mwBDk">2013-12-29</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Access+Control+-+Apache+HTTP+Server&rft.pub=Httpd.apache.org&rft_id=https%3A%2F%2Fhttpd.apache.org%2Fdocs%2F2.2%2Fhowto%2Faccess.html&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt164" id="mwBDo"></span></span></li>
<li about="#cite_note-45" id="cite_note-45" data-mw-footnote-number="45"><span class="mw-cite-backlink" id="mwBDs"><a href="#cite_ref-45" rel="mw:referencedBy" id="mwBDw"><span class="mw-linkback-text" id="mwBD0">↑</span></a></span> <span id="mw-reference-text-cite_note-45" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt167" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBD4" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings"},"title":{"wt":"Deny Strings for Filtering Rules : The Official Microsoft IIS Site"},"publisher":{"wt":"Iis.net"},"date":{"wt":"2013-11-06"},"access-date":{"wt":"2013-12-29"},"archive-url":{"wt":"https://web.archive.org/web/20140101112730/http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings"},"archive-date":{"wt":"2014-01-01"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt167" id="mwBD8"><a rel="mw:ExtLink nofollow" href="http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings" class="external text" id="mwBEA">"Deny Strings for Filtering Rules<span typeof="mw:DisplaySpace" id="mwBEE"> </span>: The Official Microsoft IIS Site"</a>. Iis.net. 2013-11-06. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20140101112730/http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings" class="external text" id="mwBEI">Archived</a> from the original on 2014-01-01<span class="reference-accessdate" id="mwBEM">. Retrieved <span class="nowrap" id="mwBEQ">2013-12-29</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Deny+Strings+for+Filtering+Rules+%3A+The+Official+Microsoft+IIS+Site&rft.pub=Iis.net&rft.date=2013-11-06&rft_id=http%3A%2F%2Fwww.iis.net%2Fconfigreference%2Fsystem.webserver%2Fsecurity%2Frequestfiltering%2Ffilteringrules%2Ffilteringrule%2Fdenystrings&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt167" id="mwBEU"></span></span></li>
<li about="#cite_note-46" id="cite_note-46" data-mw-footnote-number="46"><span class="mw-cite-backlink" id="mwBEY"><a href="#cite_ref-46" rel="mw:referencedBy" id="mwBEc"><span class="mw-linkback-text" id="mwBEg">↑</span></a></span> <span id="mw-reference-text-cite_note-46" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt170" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBEk" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://www.google.com/humans.txt"},"title":{"wt":"Google humans.txt"},"access-date":{"wt":"October 3, 2019"},"archive-date":{"wt":"January 24, 2017"},"archive-url":{"wt":"https://web.archive.org/web/20170124121422/https://www.google.com/humans.txt"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt170" id="mwBEo"><a rel="mw:ExtLink nofollow" href="https://www.google.com/humans.txt" class="external text" id="mwBEs">"Google humans.txt"</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20170124121422/https://www.google.com/humans.txt" class="external text" id="mwBEw">Archived</a> from the original on January 24, 2017<span class="reference-accessdate" id="mwBE0">. Retrieved <span class="nowrap" id="mwBE4">October 3,</span> 2019</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Google+humans.txt&rft_id=https%3A%2F%2Fwww.google.com%2Fhumans.txt&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt170" id="mwBE8"></span></span></li>
<li about="#cite_note-47" id="cite_note-47" data-mw-footnote-number="47"><span class="mw-cite-backlink" id="mwBFA"><a href="#cite_ref-47" rel="mw:referencedBy" id="mwBFE"><span class="mw-linkback-text" id="mwBFI">↑</span></a></span> <span id="mw-reference-text-cite_note-47" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt173" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBFM" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://github.com/humans.txt"},"title":{"wt":"Github humans.txt"},"website":{"wt":"[[GitHub]]"},"access-date":{"wt":"October 3, 2019"},"archive-date":{"wt":"May 30, 2016"},"archive-url":{"wt":"https://web.archive.org/web/20160530160942/https://github.com/humans.txt"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt173" id="mwBFQ"><a rel="mw:ExtLink nofollow" href="https://github.com/humans.txt" class="external text" id="mwBFU">"Github humans.txt"</a>. <i id="mwBFY"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/GitHub" title="GitHub" id="mwBFc">GitHub</a></i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20160530160942/https://github.com/humans.txt" class="external text" id="mwBFg">Archived</a> from the original on May 30, 2016<span class="reference-accessdate" id="mwBFk">. Retrieved <span class="nowrap" id="mwBFo">October 3,</span> 2019</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=GitHub&rft.atitle=Github+humans.txt&rft_id=https%3A%2F%2Fgithub.com%2Fhumans.txt&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt173" id="mwBFs"></span></span></li>
<li about="#cite_note-48" id="cite_note-48" data-mw-footnote-number="48"><span class="mw-cite-backlink" id="mwBFw"><a href="#cite_ref-48" rel="mw:referencedBy" id="mwBF0"><span class="mw-linkback-text" id="mwBF4">↑</span></a></span> <span id="mw-reference-text-cite_note-48" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt176" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBF8" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://slate.com/technology/2014/07/a-killer-robots-txt-google-easter-egg.html"},"title":{"wt":"Is This a Google Easter Egg or Proof That Skynet Is Actually Plotting World Domination?"},"last":{"wt":"Newman"},"first":{"wt":"Lily Hay"},"date":{"wt":"2014-07-03"},"website":{"wt":"Slate Magazine"},"language":{"wt":"en"},"access-date":{"wt":"2019-10-03"},"archive-date":{"wt":"2018-11-18"},"archive-url":{"wt":"https://web.archive.org/web/20181118104127/https://slate.com/technology/2014/07/a-killer-robots-txt-google-easter-egg.html"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite id="CITEREFNewman2014" class="citation web cs1" about="#mwt176">Newman, Lily Hay (2014-07-03). <a rel="mw:ExtLink nofollow" href="https://slate.com/technology/2014/07/a-killer-robots-txt-google-easter-egg.html" class="external text" id="mwBGA">"Is This a Google Easter Egg or Proof That Skynet Is Actually Plotting World Domination?"</a>. <i id="mwBGE">Slate Magazine</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20181118104127/https://slate.com/technology/2014/07/a-killer-robots-txt-google-easter-egg.html" class="external text" id="mwBGI">Archived</a> from the original on 2018-11-18<span class="reference-accessdate" id="mwBGM">. Retrieved <span class="nowrap" id="mwBGQ">2019-10-03</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Slate+Magazine&rft.atitle=Is+This+a+Google+Easter+Egg+or+Proof+That+Skynet+Is+Actually+Plotting+World+Domination%3F&rft.date=2014-07-03&rft.aulast=Newman&rft.aufirst=Lily+Hay&rft_id=https%3A%2F%2Fslate.com%2Ftechnology%2F2014%2F07%2Fa-killer-robots-txt-google-easter-egg.html&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt176" id="mwBGU"></span></span></li>
<li about="#cite_note-49" id="cite_note-49" data-mw-footnote-number="49"><span class="mw-cite-backlink" id="mwBGY"><a href="#cite_ref-49" rel="mw:referencedBy" id="mwBGc"><span class="mw-linkback-text" id="mwBGg">↑</span></a></span> <span id="mw-reference-text-cite_note-49" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt179" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBGk" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://www.google.com/killer-robots.txt"},"title":{"wt":"/killer-robots.txt"},"date":{"wt":"2018-01-10"},"url-status":{"wt":"dead"},"archive-url":{"wt":"https://web.archive.org/web/20180110160916/https://www.google.com/killer-robots.txt"},"archive-date":{"wt":"2018-01-10"},"access-date":{"wt":"2018-05-25"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt179" id="mwBGo"><a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20180110160916/https://www.google.com/killer-robots.txt" class="external text" id="mwBGs">"/killer-robots.txt"</a>. 2018-01-10. Archived from <a rel="mw:ExtLink nofollow" href="https://www.google.com/killer-robots.txt" class="external text" id="mwBGw">the original</a> on 2018-01-10<span class="reference-accessdate" id="mwBG0">. Retrieved <span class="nowrap" id="mwBG4">2018-05-25</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=%2Fkiller-robots.txt&rft.date=2018-01-10&rft_id=https%3A%2F%2Fwww.google.com%2Fkiller-robots.txt&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt179" id="mwBG8"></span></span></li>
<li about="#cite_note-50" id="cite_note-50" data-mw-footnote-number="50"><span class="mw-cite-backlink" id="mwBHA"><a href="#cite_ref-50" rel="mw:referencedBy" id="mwBHE"><span class="mw-linkback-text" id="mwBHI">↑</span></a></span> <span id="mw-reference-text-cite_note-50" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt193" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBHM" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite report ","href":"./Template:Cite_report"},"params":{"url":{"wt":"https://www.rfc-editor.org/rfc/rfc9309.html#name-special-characters"},"title":{"wt":"Robots Exclusion Protocol"},"last1":{"wt":"Koster"},"first1":{"wt":"Martijn"},"last2":{"wt":"Illyes"},"first2":{"wt":"Gary"},"last3":{"wt":"Zeller"},"first3":{"wt":"Henner"},"last4":{"wt":"Sassman"},"first4":{"wt":"Lizzi"},"date":{"wt":"September 2022"},"publisher":{"wt":"Internet Engineering Task Force"},"issue":{"wt":"RFC 9309"}},"i":0}}]}'/><cite id="mwBHQ" class="citation report cs1" about="#mwt193" data-x-id="CITEREFKosterIllyesZellerSassman2022">Koster, Martijn; Illyes, Gary; Zeller, Henner; Sassman, Lizzi (September 2022). <a rel="mw:ExtLink nofollow" href="https://www.rfc-editor.org/rfc/rfc9309.html#name-special-characters" class="external text" id="mwBHU">Robots Exclusion Protocol</a> (Report). Internet Engineering Task Force.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=report&rft.btitle=Robots+Exclusion+Protocol&rft.pub=Internet+Engineering+Task+Force&rft.date=2022-09&rft.aulast=Koster&rft.aufirst=Martijn&rft.au=Illyes%2C+Gary&rft.au=Zeller%2C+Henner&rft.au=Sassman%2C+Lizzi&rft_id=https%3A%2F%2Fwww.rfc-editor.org%2Frfc%2Frfc9309.html%23name-special-characters&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt193" id="mwBHY"></span></span></li>
<li about="#cite_note-bing-crawl-delay-51" id="cite_note-bing-crawl-delay-51" data-mw-footnote-number="51"><span class="mw-cite-backlink" id="mwBHc"><a href="#cite_ref-bing-crawl-delay_51-0" rel="mw:referencedBy" id="mwBHg"><span class="mw-linkback-text" id="mwBHk">↑</span></a></span> <span id="mw-reference-text-cite_note-bing-crawl-delay-51" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt197" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBHo" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/"},"title":{"wt":"To crawl or not to crawl, that is BingBot's question"},"date":{"wt":"3 May 2012"},"access-date":{"wt":"9 February 2016"},"archive-url":{"wt":"https://web.archive.org/web/20160203142822/https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/"},"archive-date":{"wt":"2016-02-03"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt197" id="mwBHs"><a rel="mw:ExtLink nofollow" href="https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/" class="external text" id="mwBHw">"To crawl or not to crawl, that is BingBot's question"</a>. 3 May 2012. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20160203142822/https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/" class="external text" id="mwBH0">Archived</a> from the original on 2016-02-03<span class="reference-accessdate" id="mwBH4">. Retrieved <span class="nowrap" id="mwBH8">9 February</span> 2016</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=To+crawl+or+not+to+crawl%2C+that+is+BingBot%27s+question&rft.date=2012-05-03&rft_id=https%3A%2F%2Fblogs.bing.com%2Fwebmaster%2F2012%2F05%2F03%2Fto-crawl-or-not-to-crawl-that-is-bingbots-question%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt197" id="mwBIA"></span></span></li>
<li about="#cite_note-52" id="cite_note-52" data-mw-footnote-number="52"><span class="mw-cite-backlink" id="mwBIE"><a href="#cite_ref-52" rel="mw:referencedBy" id="mwBII"><span class="mw-linkback-text" id="mwBIM">↑</span></a></span> <span id="mw-reference-text-cite_note-52" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt200" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBIQ" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"How Google interprets the robots.txt specification"},"url":{"wt":"https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt"},"website":{"wt":"Google Search Central"},"access-date":{"wt":"2024-10-06"},"date":{"wt":"2024-05-23"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt200" id="mwBIU"><a rel="mw:ExtLink nofollow" href="https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt" class="external text" id="mwBIY">"How Google interprets the robots.txt specification"</a>. <i id="mwBIc">Google Search Central</i>. 2024-05-23<span class="reference-accessdate" id="mwBIg">. Retrieved <span class="nowrap" id="mwBIk">2024-10-06</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Google+Search+Central&rft.atitle=How+Google+interprets+the+robots.txt+specification&rft.date=2024-05-23&rft_id=https%3A%2F%2Fdevelopers.google.com%2Fsearch%2Fdocs%2Fcrawling-indexing%2Frobots%2Frobots_txt&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt200" id="mwBIo"></span></span></li>
<li about="#cite_note-53" id="cite_note-53" data-mw-footnote-number="53"><span class="mw-cite-backlink" id="mwBIs"><a href="#cite_ref-53" rel="mw:referencedBy" id="mwBIw"><span class="mw-linkback-text" id="mwBI0">↑</span></a></span> <span id="mw-reference-text-cite_note-53" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt203" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBI4" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"Change Googlebot crawl rate - Search Console Help"},"url":{"wt":"https://support.google.com/webmasters/answer/48620?hl=en"},"website":{"wt":"support.google.com"},"access-date":{"wt":"22 October 2018"},"language":{"wt":"en"},"archive-url":{"wt":"https://web.archive.org/web/20181118205747/https://support.google.com/webmasters/answer/48620?hl=en"},"archive-date":{"wt":"2018-11-18"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt203" id="mwBI8"><a rel="mw:ExtLink nofollow" href="https://support.google.com/webmasters/answer/48620?hl=en" class="external text" id="mwBJA">"Change Googlebot crawl rate - Search Console Help"</a>. <i id="mwBJE">support.google.com</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20181118205747/https://support.google.com/webmasters/answer/48620?hl=en" class="external text" id="mwBJI">Archived</a> from the original on 2018-11-18<span class="reference-accessdate" id="mwBJM">. Retrieved <span class="nowrap" id="mwBJQ">22 October</span> 2018</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=support.google.com&rft.atitle=Change+Googlebot+crawl+rate+-+Search+Console+Help&rft_id=https%3A%2F%2Fsupport.google.com%2Fwebmasters%2Fanswer%2F48620%3Fhl%3Den&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt203" id="mwBJU"></span></span></li>
<li about="#cite_note-54" id="cite_note-54" data-mw-footnote-number="54"><span class="mw-cite-backlink" id="mwBJY"><a href="#cite_ref-54" rel="mw:referencedBy" id="mwBJc"><span class="mw-linkback-text" id="mwBJg">↑</span></a></span> <span id="mw-reference-text-cite_note-54" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt207" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBJk" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"http://ysearchblog.com/2007/04/11/webmasters-can-now-auto-discover-with-sitemaps/"},"title":{"wt":"Yahoo! Search Blog - Webmasters can now auto-discover with Sitemaps"},"access-date":{"wt":"2009-03-23"},"archive-url":{"wt":"https://web.archive.org/web/20090305061841/http://ysearchblog.com/2007/04/11/webmasters-can-now-auto-discover-with-sitemaps/"},"archive-date":{"wt":"2009-03-05"},"url-status":{"wt":"dead"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt207" id="mwBJo"><a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20090305061841/http://ysearchblog.com/2007/04/11/webmasters-can-now-auto-discover-with-sitemaps/" class="external text" id="mwBJs">"Yahoo! Search Blog - Webmasters can now auto-discover with Sitemaps"</a>. Archived from <a rel="mw:ExtLink nofollow" href="http://ysearchblog.com/2007/04/11/webmasters-can-now-auto-discover-with-sitemaps/" class="external text" id="mwBJw">the original</a> on 2009-03-05<span class="reference-accessdate" id="mwBJ0">. Retrieved <span class="nowrap" id="mwBJ4">2009-03-23</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Yahoo%21+Search+Blog+-+Webmasters+can+now+auto-discover+with+Sitemaps&rft_id=http%3A%2F%2Fysearchblog.com%2F2007%2F04%2F11%2Fwebmasters-can-now-auto-discover-with-sitemaps%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt207" id="mwBJ8"></span></span></li>
<li about="#cite_note-55" id="cite_note-55" data-mw-footnote-number="55"><span class="mw-cite-backlink" id="mwBKA"><a href="#cite_ref-55" rel="mw:referencedBy" id="mwBKE"><span class="mw-linkback-text" id="mwBKI">↑</span></a></span> <span id="mw-reference-text-cite_note-55" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt210" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBKM" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://commoncrawl.org/faq"},"title":{"wt":"FAQ - Common Crawl"},"access-date":{"wt":"2025-05-26"},"quote":{"wt":"How can I ensure the Common Crawl CCBot can crawl my site effectively? The crawler supports the Sitemap Protocol and utilizes any Sitemap announced in the robots.txt file."}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt210" id="mwBKQ"><a rel="mw:ExtLink nofollow" href="https://commoncrawl.org/faq" class="external text" id="mwBKU">"FAQ - Common Crawl"</a><span class="reference-accessdate" id="mwBKY">. Retrieved <span class="nowrap" id="mwBKc">2025-05-26</span></span>. <q id="mwBKg">How can I ensure the Common Crawl CCBot can crawl my site effectively? The crawler supports the Sitemap Protocol and utilizes any Sitemap announced in the robots.txt file.</q></cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=FAQ+-+Common+Crawl&rft_id=https%3A%2F%2Fcommoncrawl.org%2Ffaq&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt210" id="mwBKk"></span></span></li>
<li about="#cite_note-56" id="cite_note-56" data-mw-footnote-number="56"><span class="mw-cite-backlink" id="mwBKo"><a href="#cite_ref-56" rel="mw:referencedBy" id="mwBKs"><span class="mw-linkback-text" id="mwBKw">↑</span></a></span> <span id="mw-reference-text-cite_note-56" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt214" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBK0" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://developers.google.com/search/reference/robots_txt?hl=en"},"title":{"wt":"Robots.txt Specifications"},"website":{"wt":"Google Developers"},"access-date":{"wt":"February 15, 2020"},"archive-date":{"wt":"November 2, 2019"},"archive-url":{"wt":"https://web.archive.org/web/20191102192623/https://developers.google.com/search/reference/robots_txt?hl=en"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt214" id="mwBK4"><a rel="mw:ExtLink nofollow" href="https://developers.google.com/search/reference/robots_txt?hl=en" class="external text" id="mwBK8">"Robots.txt Specifications"</a>. <i id="mwBLA">Google Developers</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20191102192623/https://developers.google.com/search/reference/robots_txt?hl=en" class="external text" id="mwBLE">Archived</a> from the original on November 2, 2019<span class="reference-accessdate" id="mwBLI">. Retrieved <span class="nowrap" id="mwBLM">February 15,</span> 2020</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Google+Developers&rft.atitle=Robots.txt+Specifications&rft_id=https%3A%2F%2Fdevelopers.google.com%2Fsearch%2Freference%2Frobots_txt%3Fhl%3Den&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt214" id="mwBLQ"></span></span></li>
<li about="#cite_note-57" id="cite_note-57" data-mw-footnote-number="57"><span class="mw-cite-backlink" id="mwBLU"><a href="#cite_ref-57" rel="mw:referencedBy" id="mwBLY"><span class="mw-linkback-text" id="mwBLc">↑</span></a></span> <span id="mw-reference-text-cite_note-57" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt217" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBLg" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://contentsignals.org/"},"title":{"wt":"ContentSignals website"},"access-date":{"wt":"2025-09-30"},"archive-url":{"wt":"https://web.archive.org/web/20250929163116/https://contentsignals.org/"},"archive-date":{"wt":"2025-09-29"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt217" id="mwBLk"><a rel="mw:ExtLink nofollow" href="https://contentsignals.org/" class="external text" id="mwBLo">"ContentSignals website"</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20250929163116/https://contentsignals.org/" class="external text" id="mwBLs">Archived</a> from the original on 2025-09-29<span class="reference-accessdate" id="mwBLw">. Retrieved <span class="nowrap" id="mwBL0">2025-09-30</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=ContentSignals+website&rft_id=https%3A%2F%2Fcontentsignals.org%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt217" id="mwBL4"></span></span></li>
<li about="#cite_note-58" id="cite_note-58" data-mw-footnote-number="58"><span class="mw-cite-backlink" id="mwBL8"><a href="#cite_ref-58" rel="mw:referencedBy" id="mwBMA"><span class="mw-linkback-text" id="mwBME">↑</span></a></span> <span id="mw-reference-text-cite_note-58" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt220" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBMI" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://searchengineland.com/cloudflare-content-signals-462538"},"title":{"wt":"Cloudflare offers way to block AI Overviews – will Google comply?"},"access-date":{"wt":"2025-09-30"},"archive-url":{"wt":"https://web.archive.org/web/20250926162710/https://searchengineland.com/cloudflare-content-signals-462538"},"archive-date":{"wt":"2025-09-26"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt220" id="mwBMM"><a rel="mw:ExtLink nofollow" href="https://searchengineland.com/cloudflare-content-signals-462538" class="external text" id="mwBMQ">"Cloudflare offers way to block AI Overviews – will Google comply?"</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20250926162710/https://searchengineland.com/cloudflare-content-signals-462538" class="external text" id="mwBMU">Archived</a> from the original on 2025-09-26<span class="reference-accessdate" id="mwBMY">. Retrieved <span class="nowrap" id="mwBMc">2025-09-30</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Cloudflare+offers+way+to+block+AI+Overviews+%E2%80%93+will+Google+comply%3F&rft_id=https%3A%2F%2Fsearchengineland.com%2Fcloudflare-content-signals-462538&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt220" id="mwBMg"></span></span></li>
<li about="#cite_note-59" id="cite_note-59" data-mw-footnote-number="59"><span class="mw-cite-backlink" id="mwBMk"><a href="#cite_ref-59" rel="mw:referencedBy" id="mwBMo"><span class="mw-linkback-text" id="mwBMs">↑</span></a></span> <span id="mw-reference-text-cite_note-59" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt223" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBMw" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://blog.cloudflare.com/content-signals-policy/"},"title":{"wt":"Giving users choice with Cloudflare's new Content Signals Policy"},"access-date":{"wt":"2025-09-30"},"archive-url":{"wt":"https://web.archive.org/web/20250930090609/https://blog.cloudflare.com/content-signals-policy/"},"archive-date":{"wt":"2025-09-30"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt223" id="mwBM0"><a rel="mw:ExtLink nofollow" href="https://blog.cloudflare.com/content-signals-policy/" class="external text" id="mwBM4">"Giving users choice with Cloudflare's new Content Signals Policy"</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20250930090609/https://blog.cloudflare.com/content-signals-policy/" class="external text" id="mwBM8">Archived</a> from the original on 2025-09-30<span class="reference-accessdate" id="mwBNA">. Retrieved <span class="nowrap" id="mwBNE">2025-09-30</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Giving+users+choice+with+Cloudflare%27s+new+Content+Signals+Policy&rft_id=https%3A%2F%2Fblog.cloudflare.com%2Fcontent-signals-policy%2F&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt223" id="mwBNI"></span></span></li>
<li about="#cite_note-google-meta-60" id="cite_note-google-meta-60" data-mw-footnote-number="60"><span rel="mw:referencedBy" class="mw-cite-backlink" id="mwBNM"><a href="#cite_ref-google-meta_60-0" id="mwBNQ"><span class="mw-linkback-text" id="mwBNU">1</span></a> <a href="#cite_ref-google-meta_60-1" id="mwBNY"><span class="mw-linkback-text" id="mwBNc">2</span></a></span> <span id="mw-reference-text-cite_note-google-meta-60" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt227" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBNg" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"cite web ","href":"./Template:Cite_web"},"params":{"url":{"wt":"https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag"},"title":{"wt":"Robots meta tag and X-Robots-Tag HTTP header specifications - Webmasters — Google Developers"},"access-date":{"wt":"2013-08-17"},"archive-url":{"wt":"https://web.archive.org/web/20130808020946/https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag"},"archive-date":{"wt":"2013-08-08"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt227" id="mwBNk"><a rel="mw:ExtLink nofollow" href="https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag" class="external text" id="mwBNo">"Robots meta tag and X-Robots-Tag HTTP header specifications - Webmasters — Google Developers"</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20130808020946/https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag" class="external text" id="mwBNs">Archived</a> from the original on 2013-08-08<span class="reference-accessdate" id="mwBNw">. Retrieved <span class="nowrap" id="mwBN0">2013-08-17</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=unknown&rft.btitle=Robots+meta+tag+and+X-Robots-Tag+HTTP+header+specifications+-+Webmasters+%E2%80%94+Google+Developers&rft_id=https%3A%2F%2Fdevelopers.google.com%2Fwebmasters%2Fcontrol-crawl-index%2Fdocs%2Frobots_meta_tag&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt227" id="mwBN4"></span></span></li>
<li about="#cite_note-61" id="cite_note-61" data-mw-footnote-number="61"><span class="mw-cite-backlink" id="mwBN8"><a href="#cite_ref-61" rel="mw:referencedBy" id="mwBOA"><span class="mw-linkback-text" id="mwBOE">↑</span></a></span> <span id="mw-reference-text-cite_note-61" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt235" typeof="mw:Extension/templatestyles" id="mwBOI" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""}}'/><cite id="mwBOM" class="citation cs1" data-x-id="CITEREFKosterIllyesZellerSassman2022">Koster, M.; Illyes, G.; Zeller, H.; Sassman, L. (September 2022). <a rel="mw:ExtLink nofollow" href="https://www.rfc-editor.org/rfc/rfc9309" class="external text" id="mwBOQ"><i id="mwBOU">Robots Exclusion Protocol</i></a>. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Internet_Engineering_Task_Force" title="Internet Engineering Task Force" id="mwBOY">Internet Engineering Task Force</a>. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Doi_(identifier)" title="Doi (identifier)" class="mw-redirect" id="mwBOc">doi</a>:<span class="id-lock-free" title="Freely accessible" id="mwBOg"><a rel="mw:ExtLink nofollow" href="https://doi.org/10.17487%2FRFC9309" class="external text" id="mwBOk">10.17487/RFC9309</a></span>. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Request_for_Comments" title="Request for Comments" id="mwBOo">RFC</a> <a rel="mw:ExtLink nofollow" href="https://datatracker.ietf.org/doc/html/rfc9309" class="external text" id="mwBOs">9309</a>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Robots+Exclusion+Protocol&rft.pub=Internet+Engineering+Task+Force&rft.date=2022-09&rft_id=info%3Adoi%2F10.17487%2FRFC9309&rft.aulast=Koster&rft.aufirst=M.&rft.au=Illyes%2C+G.&rft.au=Zeller%2C+H.&rft.au=Sassman%2C+L.&rft_id=https%3A%2F%2Fwww.rfc-editor.org%2Frfc%2Frfc9309&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" id="mwBOw"></span> <i id="mwBO0">Proposed Standard.</i> sec. 2.5: Limits. </span></li>
<li about="#cite_note-62" id="cite_note-62" data-mw-footnote-number="62"><span class="mw-cite-backlink" id="mwBO4"><a href="#cite_ref-62" rel="mw:referencedBy" id="mwBO8"><span class="mw-linkback-text" id="mwBPA">↑</span></a></span> <span id="mw-reference-text-cite_note-62" class="mw-reference-text reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt239" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBPE" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"title":{"wt":"How Google Interprets the robots.txt Specification {{!}} Documentation"},"url":{"wt":"https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt"},"access-date":{"wt":"2022-10-17"},"website":{"wt":"Google Developers"},"language":{"wt":"en"},"archive-date":{"wt":"2022-10-17"},"archive-url":{"wt":"https://web.archive.org/web/20221017101925/https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt"},"url-status":{"wt":"live"}},"i":0}}]}'/><cite class="citation web cs1" about="#mwt239" id="mwBPI"><a rel="mw:ExtLink nofollow" href="https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt" class="external text" id="mwBPM">"How Google Interprets the robots.txt Specification | Documentation"</a>. <i id="mwBPQ">Google Developers</i>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20221017101925/https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt" class="external text" id="mwBPU">Archived</a> from the original on 2022-10-17<span class="reference-accessdate" id="mwBPY">. Retrieved <span class="nowrap" id="mwBPc">2022-10-17</span></span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=Google+Developers&rft.atitle=How+Google+Interprets+the+robots.txt+Specification+%7C+Documentation&rft_id=https%3A%2F%2Fdevelopers.google.com%2Fsearch%2Fdocs%2Fcrawling-indexing%2Frobots%2Frobots_txt&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt239" id="mwBPg"></span></span></li>
</ol></div>
</section><section data-mw-section-id="22" id="mwBPk" aria-labelledby="Further_reading"><div class="mw-heading mw-heading2"><h2 id="Further_reading">Further reading</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=22" title="Edit section: Further reading"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<ul id="mwBPo"><li id="mwBPs"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt249" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwBPw" data-mw='{"name":"templatestyles","attrs":{"src":"Module:Citation/CS1/styles.css"},"body":{"extsrc":""},"parts":[{"template":{"target":{"wt":"Cite web ","href":"./Template:Cite_web"},"params":{"last":{"wt":"Allyn"},"first":{"wt":"Bobby"},"date":{"wt":"5 July 2024"},"title":{"wt":"Artificial Intelligence Web Crawlers Are Running Amok"},"url":{"wt":"https://www.npr.org/2024/07/05/nx-s1-5026932/artificial-intelligence-web-crawlers-are-running-amok"},"url-status":{"wt":"live"},"archive-url":{"wt":"https://web.archive.org/web/20240706063210/https://www.npr.org/2024/07/05/nx-s1-5026932/artificial-intelligence-web-crawlers-are-running-amok"},"archive-date":{"wt":"6 July 2024"},"work":{"wt":"[[All Things Considered]]"},"publisher":{"wt":"[[NPR]]"},"access-date":{"wt":"6 July 2024"}},"i":0}}]}'/><cite id="CITEREFAllyn2024" class="citation web cs1" about="#mwt249">Allyn, Bobby (5 July 2024). <a rel="mw:ExtLink nofollow" href="https://www.npr.org/2024/07/05/nx-s1-5026932/artificial-intelligence-web-crawlers-are-running-amok" class="external text">"Artificial Intelligence Web Crawlers Are Running Amok"</a>. <i><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/All_Things_Considered" title="All Things Considered">All Things Considered</a></i>. <a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/NPR" title="NPR">NPR</a>. <a rel="mw:ExtLink nofollow" href="https://web.archive.org/web/20240706063210/https://www.npr.org/2024/07/05/nx-s1-5026932/artificial-intelligence-web-crawlers-are-running-amok" class="external text">Archived</a> from the original on 6 July 2024<span class="reference-accessdate">. Retrieved <span class="nowrap">6 July</span> 2024</span>.</cite><span title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=unknown&rft.jtitle=All+Things+Considered&rft.atitle=Artificial+Intelligence+Web+Crawlers+Are+Running+Amok&rft.date=2024-07-05&rft.aulast=Allyn&rft.aufirst=Bobby&rft_id=https%3A%2F%2Fwww.npr.org%2F2024%2F07%2F05%2Fnx-s1-5026932%2Fartificial-intelligence-web-crawlers-are-running-amok&rfr_id=info%3Asid%2Fen.wikipedia.org%3ARobots.txt" class="Z3988" about="#mwt249" id="mwBP0"></span></li></ul>
</section><section data-mw-section-id="23" id="mwBP4" aria-labelledby="External_links"><div class="mw-heading mw-heading2"><h2 id="External_links">External links</h2><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Robots.txt&action=edit&section=23" title="Edit section: External links"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>
<!--
+============================ {{No more links}} ============================+
| DO NOT ADD EXTERNAL LINKS WITHOUT DISCUSSING THEM ON THE TALK PAGE FIRST! |
| In particular, there are too many blogs, SEO pages and free tools to list |
| them all, and only listing a few of them unfairly gives those pages free |
| advertising that the pages we don't list don't get. |
*============================ {{No more links}} ============================+
-->
<ul id="mwBP8"><li id="mwBQA"><span class="official-website" about="#mwt251" typeof="mw:Transclusion" id="mwBQE" data-mw='{"parts":[{"template":{"target":{"wt":"Official website","href":"./Template:Official_website"},"params":{"1":{"wt":"https://www.robotstxt.org"}},"i":0}}]}'><span class="url"><a rel="mw:ExtLink nofollow" href="https://www.robotstxt.org" class="external text">Official website</a></span></span></li></ul>
<span about="#mwt252" typeof="mw:Transclusion" id="mwBQI" data-mw='{"parts":[{"template":{"target":{"wt":"Authority control","href":"./Template:Authority_control"},"params":{},"i":0}}]}'></span>
<span about="#mwt253" typeof="mw:Transclusion" id="mwBQM" data-mw='{"parts":[{"template":{"target":{"wt":"DEFAULTSORT:Robots Exclusion Standard","function":"defaultsort"},"params":{},"i":0}}]}'></span>
<link rel="mw:PageProp/Category" href="./Category:Search_engine_optimization" id="mwBQQ"/>
<link rel="mw:PageProp/Category" href="./Category:Websites" id="mwBQU"/>
<link rel="mw:PageProp/Category" href="./Category:Web_scraping" id="mwBQY"/>
<link rel="mw:PageProp/Category" href="./Category:Text_files" id="mwBQc"/>
<link rel="mw:PageProp/Category" href="./Category:Internet_bots" id="mwBQg"/></section>
<!--
NewPP limit report
Parsed by mw‐web.eqiad.main‐69b7c6bd4‐7gfg8
Cached time: 20260820025133
Cache expiry: 2592000
Cache expiry source: Module:Citation/CS1 (os.date(%Y))
Reduced expiry: false
Complications: [vary‐revision‐sha1, show‐toc, use‐parsoid]
CPU time usage: 0.778 seconds
Real time usage: 0.974 seconds
Preprocessor visited node count: 3701/1000000
Revision size: 41143/2097152 bytes
Post‐expand include size: 120185/2097152 bytes
Template argument size: 4169/2097152 bytes
Highest expansion depth: 17/100
Expensive parser function count: 5/500
Unstrip recursion depth: 0/20
Unstrip post‐expand size: 148229/5000000 bytes
Lua time usage: 0.208/10.000 seconds
Lua memory usage: 7338750/52428800 bytes
Number of Wikibase entities loaded: 1/500
-->
<!--
Transclusion expansion time report (%,ms,calls,template)
100.00% 804.845 1 -total
18.57% 149.461 51 Template:Cite_web
6.68% 53.744 1 Template:Short_description
4.03% 32.441 2 Template:Pagetype
3.39% 27.269 1 Template:Infobox_technology_standard
3.13% 25.185 2 Template:Cite_IETF
3.09% 24.838 1 Template:Infobox
2.88% 23.184 1 Template:Cite_court
2.36% 18.959 1 Template:Pp-pc
1.98% 15.914 6 Template:Cite_news
-->
<!-- Render ID 0d4b3aab-9c42-11f1-9bdc-3da613cb4954 -->
<!-- Saved in parser cache with key enwiki:stable-parsoid-pcache:101673:|#|:idhash:useParsoid=1 and timestamp 20260820025133 and revision id 1370001604. Rendering was triggered because: unknown
-->
<!-- Parsoid 0.24.0.0-alpha19 -->
</div><!--
Post‐processing cache key enwiki:stable‐parsoid‐pcache‐postproc:101673:|#|:idhash:injectTOC=0!postproc=1!skin=vector‐2022!useParsoid=1!visibleLinks=wtsingle, generated at 20260820025133
--><noscript><img src="https://en.wikipedia.org/wiki/Special:CentralAutoLogin/start?useformat=desktop&type=1x1&usesul3=1" alt="" width="1" height="1" style="border: none; position: absolute;"></noscript>
<div class="printfooter" data-nosnippet="">Retrieved from "<a dir="ltr" href="https://en.wikipedia.org/w/index.php?title=Robots.txt&oldid=1370001604">https://en.wikipedia.org/w/index.php?title=Robots.txt&oldid=1370001604</a>"</div></div>
<div id="catlinks" class="catlinks" data-mw-interface=""><div id="mw-normal-catlinks" class="mw-normal-catlinks"><a href="/wiki/Help:Category" title="Help:Category">Categories</a>: <ul><li><a href="/wiki/Category:Search_engine_optimization" title="Category:Search engine optimization">Search engine optimization</a></li><li><a href="/wiki/Category:Websites" title="Category:Websites">Websites</a></li><li><a href="/wiki/Category:Web_scraping" title="Category:Web scraping">Web scraping</a></li><li><a href="/wiki/Category:Text_files" title="Category:Text files">Text files</a></li><li><a href="/wiki/Category:Internet_bots" title="Category:Internet bots">Internet bots</a></li></ul></div><div id="mw-hidden-catlinks" class="mw-hidden-catlinks mw-hidden-cats-hidden">Hidden categories: <ul><li><a href="/wiki/Category:Articles_with_short_description" title="Category:Articles with short description">Articles with short description</a></li><li><a href="/wiki/Category:Short_description_is_different_from_Wikidata" title="Category:Short description is different from Wikidata">Short description is different from Wikidata</a></li><li><a href="/wiki/Category:Wikipedia_pending_changes_protected_pages" title="Category:Wikipedia pending changes protected pages">Wikipedia pending changes protected pages</a></li><li><a href="/wiki/Category:CS1_maint:_url-status" title="Category:CS1 maint: url-status">CS1 maint: url-status</a></li></ul></div></div>
</div>
</main>
</div>
<div class="mw-footer-container">
<footer id="footer" class="mw-footer" >
<ul id="footer-info" class="mw-portlet mw-portlet-footer-info">
<li id="footer-info-lastmod"> This page was last edited on 18 August 2026, at 14:42<span class="anonymous-show"> (UTC)</span>.</li>
<li id="footer-info-renderedwith">Page was rendered with <a href="https://www.mediawiki.org/wiki/Special:MyLanguage/Parsoid" class="extiw" title="mw:Special:MyLanguage/Parsoid">Parsoid</a>.</li>
<li id="footer-info-copyright">Text is available under the <a href="/wiki/Wikipedia:Text_of_the_Creative_Commons_Attribution-ShareAlike_4.0_International_License" title="Wikipedia:Text of the Creative Commons Attribution-ShareAlike 4.0 International License">Creative Commons Attribution-ShareAlike 4.0 License</a>;
additional terms may apply. By using this site, you agree to the <a href="https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Terms_of_Use" class="extiw" title="foundation:Special:MyLanguage/Policy:Terms of Use">Terms of Use</a> and <a href="https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Privacy_policy" class="extiw" title="foundation:Special:MyLanguage/Policy:Privacy policy">Privacy Policy</a>. Wikipedia® is a registered trademark of the <a rel="nofollow" class="external text" href="https://wikimediafoundation.org/">Wikimedia Foundation, Inc.</a>, a non-profit organization.</li>
</ul>
<ul id="footer-places" class="mw-portlet mw-portlet-footer-places">
<li id="footer-places-privacy"><a href="https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Privacy_policy">Privacy policy</a></li>
<li id="footer-places-about"><a href="/wiki/Wikipedia:About">About Wikipedia</a></li>
<li id="footer-places-disclaimers"><a href="/wiki/Wikipedia:General_disclaimer">Disclaimers</a></li>
<li id="footer-places-contact"><a href="//en.wikipedia.org/wiki/Wikipedia:Contact_us">Contact Wikipedia</a></li>
<li id="footer-places-legal-safety-contacts"><a href="https://foundation.wikimedia.org/wiki/Special:MyLanguage/Legal:Wikimedia_Foundation_Legal_and_Safety_Contact_Information">Legal & safety contacts</a></li>
<li id="footer-places-wm-codeofconduct"><a href="https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Universal_Code_of_Conduct">Code of Conduct</a></li>
<li id="footer-places-developers"><a href="https://developer.wikimedia.org">Developers</a></li>
<li id="footer-places-statslink"><a href="https://stats.wikimedia.org/#/en.wikipedia.org">Statistics</a></li>
<li id="footer-places-cookiestatement"><a href="https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Cookie_statement">Cookie statement</a></li>
<li id="footer-places-mobileview"><a href="//en.wikipedia.org/w/index.php?title=Robots.txt&mobileaction=toggle_view_mobile" class="noprint stopMobileRedirectToggle">Mobile view</a></li>
</ul>
<ul id="footer-icons" class="mw-portlet mw-portlet-footer-icons noprint">
<li id="footer-copyrightico"><a href="https://www.wikimedia.org/" class="cdx-button cdx-button--fake-button cdx-button--size-large cdx-button--fake-button--enabled"><picture><source media="(min-width: 500px)" srcset="/static/images/footer/wikimedia-button.svg" width="84" height="29"><img src="/static/images/footer/wikimedia.svg" width="25" height="25" alt="Wikimedia Foundation" lang="en" loading="lazy"></picture></a></li>
<li id="footer-poweredbyico"><a href="https://www.mediawiki.org/" class="cdx-button cdx-button--fake-button cdx-button--size-large cdx-button--fake-button--enabled"><picture><source media="(min-width: 500px)" srcset="/w/resources/assets/poweredby_mediawiki.svg" width="88" height="31"><img src="/w/resources/assets/mediawiki_compact.svg" alt="Powered by MediaWiki" lang="en" width="25" height="25" loading="lazy"></picture></a></li>
</ul>
</footer>
</div>
</div>
</div>
<div class="vector-header-container vector-sticky-header-container no-font-mode-scale">
<div id="vector-sticky-header" class="vector-sticky-header">
<div class="vector-sticky-header-start">
<div class="vector-sticky-header-icon-start vector-button-flush-left" aria-hidden="true">
<button class="cdx-button cdx-button--weight-quiet cdx-button--icon-only vector-sticky-header-search-toggle" tabindex="-1" data-event-name="ui.vector-sticky-search-form.icon"><span class="vector-icon mw-ui-icon-search mw-ui-icon-wikimedia-search cdx-button__icon"></span>
<span>Search</span>
</button>
</div>
<div role="search" class="vector-search-box-vue vector-search-box-show-thumbnail vector-search-box">
<div class="vector-typeahead-search-container">
<div class="cdx-typeahead-search cdx-typeahead-search--show-thumbnail">
<form action="/w/index.php" id="vector-sticky-search-form" class="cdx-search-input cdx-search-input--has-end-button">
<div class="cdx-search-input__input-wrapper" data-search-loc="header-moved">
<div class="cdx-text-input cdx-text-input--has-start-icon">
<input
class="cdx-text-input__input mw-searchInput" autocomplete="off"
type="search" name="search" placeholder="Search Wikipedia">
<span class="cdx-text-input__icon cdx-text-input__start-icon"></span>
</div>
<input type="hidden" name="title" value="Special:Search">
</div>
<button class="cdx-button cdx-search-input__end-button">Search</button>
</form>
</div>
</div>
</div>
<div class="vector-sticky-header-context-bar">
<nav aria-label="Contents" class="vector-toc-landmark">
<div id="vector-sticky-header-toc" class="vector-dropdown mw-portlet mw-portlet-sticky-header-toc vector-sticky-header-toc vector-button-flush-left" >
<input type="checkbox" id="vector-sticky-header-toc-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-sticky-header-toc" class="vector-dropdown-checkbox " aria-label="Toggle the table of contents" >
<label id="vector-sticky-header-toc-label" for="vector-sticky-header-toc-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only " aria-hidden="true" ><span class="vector-icon mw-ui-icon-listBullet mw-ui-icon-wikimedia-listBullet cdx-button__icon"></span>
<span class="vector-dropdown-label-text">Toggle the table of contents</span>
</label>
<div class="vector-dropdown-content">
<div id="vector-sticky-header-toc-unpinned-container" class="vector-unpinned-container">
</div>
</div>
</div>
</nav>
<div class="vector-sticky-header-context-bar-primary" aria-hidden="true" >robots.txt</div>
</div>
</div>
<div class="vector-sticky-header-end" aria-hidden="true">
<div class="vector-sticky-header-icons">
<a href="#" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only" id="ca-talk-sticky-header" tabindex="-1" data-event-name="talk-sticky-header"><span class="vector-icon mw-ui-icon-speechBubbles mw-ui-icon-wikimedia-speechBubbles cdx-button__icon"></span>
<span></span>
</a>
<a href="#" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only" id="ca-subject-sticky-header" tabindex="-1" data-event-name="subject-sticky-header"><span class="vector-icon mw-ui-icon-article mw-ui-icon-wikimedia-article cdx-button__icon"></span>
<span></span>
</a>
<a href="#" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only" id="ca-history-sticky-header" tabindex="-1" data-event-name="history-sticky-header"><span class="vector-icon mw-ui-icon-wikimedia-history mw-ui-icon-wikimedia-wikimedia-history cdx-button__icon"></span>
<span></span>
</a>
<a href="#" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only mw-watchlink" id="ca-watchstar-sticky-header" tabindex="-1" data-event-name="watch-sticky-header"><span class="vector-icon mw-ui-icon-wikimedia-star mw-ui-icon-wikimedia-wikimedia-star cdx-button__icon"></span>
<span></span>
</a>
<a href="#" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only" id="ca-edit-sticky-header" tabindex="-1" data-event-name="wikitext-edit-sticky-header"><span class="vector-icon mw-ui-icon-wikimedia-wikiText mw-ui-icon-wikimedia-wikimedia-wikiText cdx-button__icon"></span>
<span></span>
</a>
<a href="#" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only" id="ca-ve-edit-sticky-header" tabindex="-1" data-event-name="ve-edit-sticky-header"><span class="vector-icon mw-ui-icon-wikimedia-edit mw-ui-icon-wikimedia-wikimedia-edit cdx-button__icon"></span>
<span></span>
</a>
<a href="#" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only" id="ca-viewsource-sticky-header" tabindex="-1" data-event-name="ve-edit-protected-sticky-header"><span class="vector-icon mw-ui-icon-wikimedia-editLock mw-ui-icon-wikimedia-wikimedia-editLock cdx-button__icon"></span>
<span></span>
</a>
</div>
<div class="vector-sticky-header-buttons">
<button class="cdx-button cdx-button--weight-quiet mw-interlanguage-selector" id="p-lang-btn-sticky-header" tabindex="-1" data-event-name="ui.dropdown-p-lang-btn-sticky-header"><span class="vector-icon mw-ui-icon-wikimedia-language mw-ui-icon-wikimedia-wikimedia-language cdx-button__icon"></span>
<span>28 languages</span>
</button>
<a href="#" class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--action-progressive" id="ca-addsection-sticky-header" tabindex="-1" data-event-name="addsection-sticky-header"><span class="vector-icon mw-ui-icon-speechBubbleAdd-progressive mw-ui-icon-wikimedia-speechBubbleAdd-progressive cdx-button__icon"></span>
<span>Add topic</span>
</a>
</div>
<div class="vector-sticky-header-icon-end">
<div class="vector-user-links">
</div>
</div>
</div>
</div>
</div>
<div class="mw-portlet mw-portlet-dock-bottom emptyPortlet" id="p-dock-bottom">
<ul>
</ul>
</div>
<script>(RLQ=window.RLQ||[]).push(function(){mw.config.set({"wgHostname":"mw-web.eqiad.main-b7548b864-qrvtt","wgBackendResponseTime":203,"wgPageParseReport":{"limitreport":{"cputime":"0.778","walltime":"0.974","ppvisitednodes":{"value":3701,"limit":1000000},"revisionsize":{"value":41143,"limit":2097152},"postexpandincludesize":{"value":120185,"limit":2097152},"templateargumentsize":{"value":4169,"limit":2097152},"expansiondepth":{"value":17,"limit":100},"expensivefunctioncount":{"value":5,"limit":500},"unstrip-depth":{"value":0,"limit":20},"unstrip-size":{"value":148229,"limit":5000000},"entityaccesscount":{"value":1,"limit":500},"timingprofile":["100.00% 804.845 1 -total"," 18.57% 149.461 51 Template:Cite_web"," 6.68% 53.744 1 Template:Short_description"," 4.03% 32.441 2 Template:Pagetype"," 3.39% 27.269 1 Template:Infobox_technology_standard"," 3.13% 25.185 2 Template:Cite_IETF"," 3.09% 24.838 1 Template:Infobox"," 2.88% 23.184 1 Template:Cite_court"," 2.36% 18.959 1 Template:Pp-pc"," 1.98% 15.914 6 Template:Cite_news"]},"scribunto":{"limitreport-timeusage":{"value":"0.208","limit":"10.000"},"limitreport-memusage":{"value":7338750,"limit":52428800}},"cachereport":{"origin":"mw-web.eqiad.main-69b7c6bd4-7gfg8","timestamp":"20260820025133","ttl":2592000,"transientcontent":false,"expiry-source":"Module:Citation/CS1 (os.date(%Y))"}}});});</script>
<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Robots.txt","url":"https:\/\/en.wikipedia.org\/wiki\/Robots.txt","sameAs":"http:\/\/www.wikidata.org\/entity\/Q80776","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q80776","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2002-10-09T14:11:28Z","image":"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/1\/16\/Robots_txt.svg?utm_source=en.wikipedia.org&utm_campaign=index&utm_content=original","headline":"standard used to advise web crawlers and scrapers not to index a web page or site"}</script>
</body>
</html>
Sitemap
Кол-во: 0
XML-карта сайта для поисковиков
?
Sitemap.xml помогает поисковику быстрее находить и индексировать страницы. Особенно важен для крупных сайтов и новых страниц, на которые ещё нет входящих ссылок.
Robots.txt не содержит ссылку на карту сайта. Рекомендуется добавить карту сайта и указать ссылку на нее в robots.txt.
Внутренние ссылки
Кол-во: 35
Ссылки на другие страницы своего сайта
?
Внутренние ссылки распределяют ссылочный вес между страницами и помогают поисковику обходить сайт. Пустые анкоры и ссылки на запрещённые robots.txt страницы — типичные ошибки.
Некоторые внутренние ссылки сайта 35шт запрещены к индексации поисковыми системами в robots.txt. Пройдитесь по списку внутренних ссылок: удалите ошибочные или исправьте файл robots.txt!
Внутренних ссылок на странице 35 оптимально.
На странице присутствуют изображения 1.
Показать внутренние ссылки
| Url | Анкор | Состояние | Анализировать |
|---|---|---|---|
| /wiki/Main_Page |
<span>Main page</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Wikipedia:Contents |
<span>Contents</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Portal:Current_events |
<span>Current events</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Special:Random |
<span>Random article</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Wikipedia:About |
<span>About Wikipedia</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Help:Contents |
<span>Help</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Help:Introduction |
<span>Learn to edit</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Wikipedia:Community_portal |
<span>Community portal</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Special:RecentChanges |
<span>Recent changes</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Wikipedia:File_upload_wizard |
<span>Upload file</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Special:SpecialPages |
<span>Special pages</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Special:Search |
<span class="vector-icon mw-ui-icon-search mw-ui-icon-wikimedia-search cdx-button__icon"></span>
<span>Search</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Special:CreateAccount&returnto=Main+Page |
<span>Create account</span>
|
Запрет в robots.txt
|
|
| /w/index.php?title=Special:UserLogin&returnto=Main+Page |
<span>Log in</span>
|
Запрет в robots.txt
|
|
| /w/index.php?title=Special:CreateAccount&returnto=Main+Page |
<span class="vector-icon mw-ui-icon-userAvatar mw-ui-icon-wikimedia-userAvatar cdx-button__icon"></span>
<span>Create account</span>
|
Запрет в robots.txt
|
|
| /w/index.php?title=Special:UserLogin&returnto=Main+Page |
<span class="vector-icon mw-ui-icon-logIn mw-ui-icon-wikimedia-logIn cdx-button__icon"></span>
<span>Log in</span>
|
Запрет в robots.txt
|
|
| /wiki/Main_Page |
<span>Main Page</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Talk:Main_Page |
<span>Talk</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Main_Page |
<span>Read</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Main_Page&action=edit |
<span>View source</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Main_Page&action=history |
<span>View history</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Main_Page |
<span class="vector-icon mw-ui-icon-eye mw-ui-icon-wikimedia-eye cdx-button__icon"></span>
<span>Read</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Main_Page&action=edit |
<span class="vector-icon mw-ui-icon-editLock mw-ui-icon-wikimedia-editLock cdx-button__icon"></span>
<span>View source</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Main_Page&action=history |
<span class="vector-icon mw-ui-icon-history mw-ui-icon-wikimedia-history cdx-button__icon"></span>
<span>View history</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Special:WhatLinksHere/Main_Page |
<span>What links here</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Main_Page&oldid=1358980925 |
<span>Permanent link</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Main_Page&action=info |
<span>Page information</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Special:CiteThisPage&page=Main_Page&id=1358980925&wpFormIdentifier=titleform |
<span>Cite this page</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Special:UrlShortener&url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FMain_Page |
<span>Get shortened URL</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Main_Page&useparsoid=0 |
<span>Switch to legacy parser</span>
|
Запрет в robots.txt
|
Анализировать url |
| /w/index.php?title=Special:DownloadAsPdf&page=Main_Page&action=show-download-screen |
<span>Download as PDF</span>
|
Запрет в robots.txt
|
|
| /w/index.php?title=Main_Page&printable=yes |
<span>Printable version</span>
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Wikipedia:Text_of_the_Creative_Commons_Attribution-ShareAlike_4.0_International_License |
Creative Commons Attribution-ShareAlike 4.0 License
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Wikipedia:About |
About Wikipedia
|
Запрет в robots.txt
|
Анализировать url |
| /wiki/Wikipedia:General_disclaimer |
Disclaimers
|
Запрет в robots.txt
|
Анализировать url |
Внешние ссылки
Кол-во: 619
Ссылки на сторонние сайты
?
Исходящие внешние ссылки передают часть ссылочного веса на чужие сайты. Ссылки на авторитетные ресурсы безопасны; ссылки на мусорные сайты могут навредить репутации страницы.
Внешних ссылок на странице 619 слишком много. Спрячьте лишние ссылки в тег noindex или атрибут rel='nofollow'!
На странице ссылки с атрибутом rel='nofollow' 2.
Показать первые 100 внешних ссылок
| Url | Анкор | Анализировать |
|---|---|---|
| http://en.wikipedia.org/wiki/Wikipedia:Contact_us |
<span>Contact us</span>
|
Анализировать url |
| donate.wikimedia.org |
<span>Donate</span>
|
Анализировать url |
| donate.wikimedia.org |
<span class="vector-icon mw-ui-icon-heart mw-ui-icon-wikimedia-heart cdx-button__icon"></span>
<span>Donate</span>
|
Анализировать url |
| http://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard |
<span>Upload file</span>
|
Анализировать url |
| abstract.wikipedia.org |
<span>Abstract Wikipedia</span>
|
Анализировать url |
| commons.wikimedia.org |
<span>Wikimedia Commons</span>
|
Анализировать url |
| foundation.wikimedia.org |
<span>Wikimedia Foundation</span>
|
Анализировать url |
| mediawiki.org |
<span>MediaWiki</span>
|
Анализировать url |
| meta.wikimedia.org |
<span>Meta-Wiki</span>
|
Анализировать url |
| outreach.wikimedia.org |
<span>Wikimedia Outreach</span>
|
Анализировать url |
| wikisource.org |
<span>Multilingual Wikisource</span>
|
Анализировать url |
| species.wikimedia.org |
<span>Wikispecies</span>
|
Анализировать url |
| en.wikibooks.org |
<span>Wikibooks</span>
|
Анализировать url |
| wikidata.org |
<span>Wikidata</span>
|
Анализировать url |
| wikifunctions.org |
<span>Wikifunctions</span>
|
Анализировать url |
| wikimania.wikimedia.org |
<span>Wikimania</span>
|
Анализировать url |
| en.wikinews.org |
<span>Wikinews</span>
|
Анализировать url |
| en.wikiquote.org |
<span>Wikiquote</span>
|
Анализировать url |
| en.wikisource.org |
<span>Wikisource</span>
|
Анализировать url |
| en.wikiversity.org |
<span>Wikiversity</span>
|
Анализировать url |
| en.wikivoyage.org |
<span>Wikivoyage</span>
|
Анализировать url |
| en.wiktionary.org |
<span>Wiktionary</span>
|
Анализировать url |
| wikidata.org |
<span>Wikidata item</span>
|
Анализировать url |
| en.wikipedia.org |
Wikipedia
|
Анализировать url |
| en.wikipedia.org |
free
|
Анализировать url |
| en.wikipedia.org |
anyone can edit
|
Анализировать url |
| en.wikipedia.org |
<span about="#mwt3" typeof="mw:Transclusion" id="mwDA" data-mw='{"parts":[{"template":{"target":{"wt":"NUMBEROFACTIVEUSERS","function":"numberofactiveusers"},"params":{},"i":0}}]}'>252,754</span>
|
Анализировать url |
| en.wikipedia.org |
<span about="#mwt4" typeof="mw:Transclusion" id="mwDw" data-mw='{"parts":[{"template":{"target":{"wt":"NUMBEROFARTICLES","function":"numberofarticles"},"params":{},"i":0}}]}'>7,228,139</span>
|
Анализировать url |
| en.wikipedia.org |
English
|
Анализировать url |
| en.wikipedia.org |
<img resource="https://en.wikipedia.org/wiki/File:Caf_goble.jpg" src="//upload.wikimedia.org/wikipedia/commons/2/20/Caf_goble.jpg?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail_unscaled" decoding="async" loading="lazy" alt="Stanley Goble" data-file-width="250" data-file-height="313" data-file-type="bitmap" height="157" width="125" class="mw-file-element">
|
Анализировать url |
| en.wikipedia.org |
Stanley Goble
|
Анализировать url |
| en.wikipedia.org |
Royal Australian Air Force
|
Анализировать url |
| en.wikipedia.org |
Chief of the Air Staff
|
Анализировать url |
| en.wikipedia.org |
fighters
|
Анализировать url |
| en.wikipedia.org |
Western Front
|
Анализировать url |
| en.wikipedia.org |
Royal Naval Air Service
|
Анализировать url |
| en.wikipedia.org |
ace
|
Анализировать url |
| en.wikipedia.org |
No.<span typeof="mw:Entity"> </span>5 Squadron
|
Анализировать url |
| en.wikipedia.org |
Distinguished Service Order
|
Анализировать url |
| en.wikipedia.org |
Distinguished Service Cross
|
Анализировать url |
| en.wikipedia.org |
Ivor McIntyre
|
Анализировать url |
| en.wikipedia.org |
circumnavigate
|
Анализировать url |
| en.wikipedia.org |
No.<span typeof="mw:Entity"> </span>2 (Bomber) Group RAF
|
Анализировать url |
| en.wikipedia.org |
Australian Government
|
Анализировать url |
| en.wikipedia.org |
Empire Air Training Scheme
|
Анализировать url |
| en.wikipedia.org |
Ottawa
|
Анализировать url |
| en.wikipedia.org |
air liaison officer
|
Анализировать url |
| en.wikipedia.org |
Full<span typeof="mw:Entity"> </span>article...
|
Анализировать url |
| en.wikipedia.org |
Pat O'Keeffe
|
Анализировать url |
| en.wikipedia.org |
SMS<span typeof="mw:Entity"> </span><i>Schwaben</i>
|
Анализировать url |
| en.wikipedia.org |
Siege of Brest (1342)
|
Анализировать url |
| en.wikipedia.org |
Archive
|
Анализировать url |
| lists.wikimedia.org |
By email
|
Анализировать url |
| en.wikipedia.org |
More featured articles
|
Анализировать url |
| en.wikipedia.org |
About
|
Анализировать url |
| en.wikipedia.org |
<img resource="https://en.wikipedia.org/wiki/File:Giovanna_Rincon_photographiée_par_Max_K_Pelgrims_en_2013_(cropped).jpg" src="//upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Giovanna_Rincon_photographi%C3%A9e_par_Max_K_Pelgrims_en_2013_%28cropped%29.jpg/120px-Giovanna_Rincon_photographi%C3%A9e_par_Max_K_Pelgrims_en_2013_%28cropped%29.jpg?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail" decoding="async" loading="lazy" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Giovanna_Rincon_photographi%C3%A9e_par_Max_K_Pelgrims_en_2013_%28cropped%29.jpg/250px-Giovanna_Rincon_photographi%C3%A9e_par_Max_K_Pelgrims_en_2013_%28cropped%29.jpg?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail 2x" alt="Giovanna Rincon" data-file-width="616" data-file-height="931" data-file-type="bitmap" height="172" width="114" class="mw-file-element">
|
Анализировать url |
| en.wikipedia.org |
<span class="nowrap">Acceptess-T</span>
|
Анализировать url |
| en.wikipedia.org |
Giovanna Rincon
|
Анализировать url |
| en.wikipedia.org |
John the Sanguinary
|
Анализировать url |
| en.wikipedia.org |
Meccha Chameleon
|
Анализировать url |
| en.wikipedia.org |
Geoffrey de Henlaw
|
Анализировать url |
| en.wikipedia.org |
Baltimore
|
Анализировать url |
| en.wikipedia.org |
mythical creatures with their feet or legs facing backwards
|
Анализировать url |
| en.wikipedia.org |
a particularly violent period during the Troubles
|
Анализировать url |
| en.wikipedia.org |
Luz Maria Frias
|
Анализировать url |
| en.wikipedia.org |
Meitei alphabet
|
Анализировать url |
| en.wikipedia.org |
Archive
|
Анализировать url |
| en.wikipedia.org |
Start a new article
|
Анализировать url |
| en.wikipedia.org |
Nominate an article
|
Анализировать url |
| en.wikipedia.org |
<img resource="https://en.wikipedia.org/wiki/File:Building_Destroyed_in_the_Flores_earthquake_2026.jpg" src="//upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Building_Destroyed_in_the_Flores_earthquake_2026.jpg/250px-Building_Destroyed_in_the_Flores_earthquake_2026.jpg?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail" decoding="async" loading="lazy" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Building_Destroyed_in_the_Flores_earthquake_2026.jpg/330px-Building_Destroyed_in_the_Flores_earthquake_2026.jpg?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail 2x" alt="Earthquake damage in East Manggarai, Indonesia" data-file-width="1274" data-file-height="928" data-file-type="bitmap" height="119" width="164" class="mw-file-element">
|
Анализировать url |
| en.wikipedia.org |
East Manggarai
|
Анализировать url |
| en.wikipedia.org |
A magnitude-7.7 earthquake
|
Анализировать url |
| en.wikipedia.org |
Flores
|
Анализировать url |
| en.wikipedia.org |
premier of China
|
Анализировать url |
| en.wikipedia.org |
Zhu Rongji
|
Анализировать url |
| en.wikipedia.org |
A solar eclipse
|
Анализировать url |
| en.wikipedia.org |
totality
|
Анализировать url |
| en.wikipedia.org |
A ferry capsizes
|
Анализировать url |
| en.wikipedia.org |
Lake Kariba
|
Анализировать url |
| en.wikipedia.org |
A magnitude-7.4 earthquake
|
Анализировать url |
| en.wikipedia.org |
Chocó Department
|
Анализировать url |
| en.wikipedia.org |
Iran war
|
Анализировать url |
| en.wikipedia.org |
Lebanon war
|
Анализировать url |
| en.wikipedia.org |
Russo-Ukrainian war
|
Анализировать url |
| en.wikipedia.org |
timeline
|
Анализировать url |
| en.wikipedia.org |
Recent deaths
|
Анализировать url |
| en.wikipedia.org |
Peggy Webber
|
Анализировать url |
| en.wikipedia.org |
O. J. Brigance
|
Анализировать url |
| en.wikipedia.org |
Hayden Panettiere
|
Анализировать url |
| en.wikipedia.org |
Chelo Rodríguez
|
Анализировать url |
| en.wikipedia.org |
Wang Xiji
|
Анализировать url |
| en.wikipedia.org |
Helmut Lethen
|
Анализировать url |
| en.wikipedia.org |
More current events
|
Анализировать url |
| en.wikipedia.org |
Nominate an article
|
Анализировать url |
| en.wikipedia.org |
August 21
|
Анализировать url |
| en.wikipedia.org |
<img resource="https://en.wikipedia.org/wiki/File:ChuaXaLoi001.jpg" src="//upload.wikimedia.org/wikipedia/commons/thumb/d/de/ChuaXaLoi001.jpg/250px-ChuaXaLoi001.jpg?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail" decoding="async" loading="lazy" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/d/de/ChuaXaLoi001.jpg/330px-ChuaXaLoi001.jpg?utm_source=en.wikipedia.org&utm_campaign=parser&utm_content=thumbnail 2x" alt="Xá Lợi Pagoda, raided by South Vietnamese special forces" data-file-width="2272" data-file-height="1704" data-file-type="bitmap" height="122" width="162" class="mw-file-element">
|
Анализировать url |
| en.wikipedia.org |
Xá Lợi Pagoda
|
Анализировать url |
| en.wikipedia.org |
1717
|
Анализировать url |
| en.wikipedia.org |
Austro-Turkish War
|
Анализировать url |
| en.wikipedia.org |
Prince Eugene of Savoy
|
Анализировать url |
Конкуренты Готовность: 0%
Конкуренты в Яндексе
Кол-во: 0
Топ сайтов-конкурентов в Яндексе
?
Сайты, чаще всего появляющиеся в ТОПе Яндекса по запросам из семантического ядра этой страницы.
Мы не нашли у вас конкурентов в Яндексе. Сайт или очень молодой или плохо продвигается.
Конкурентов в ТОП-10 Яндекса не нашлось.
Конкуренты в Google
Кол-во: 0
Топ сайтов-конкурентов в Google
?
Сайты, чаще всего появляющиеся в ТОПе Google по запросам из семантического ядра этой страницы.
Конкуренты в Google тоже не найдены. Займитесь продвижением сайта!
Конкурентов в ТОП-10 Google не нашлось.
ЗоЗПП: права потребителей Готовность: 100%
Нарушения
Не выявлены
Признаков дистанционной продажи товаров (интернет-магазина) не обнаружено — требования ЗоЗПП о раскрытии информации продавца к сайту не применяются. Нарушений нет.
ФЗ-149: рекомендательные технологии Готовность: 100%
Нарушения
Не выявлены
Рекомендательные блоки («с этим покупают», «похожие товары» и т.п.) на сайте не обнаружены — требования ст. 10.7 ФЗ-149 к сайту не применяются. Нарушений нет.
ФЗ-38: реклама Готовность: 100%
Нарушения
Не выявлены
Рекламных тематик с обязательными оговорками (медицина, БАД, кредиты и займы, новостройки) на сайте не обнаружено. Нарушений нет.
ФЗ-436: защита детей Готовность: 100%
Нарушения
Не выявлены
Признаков информационной продукции (новости, видео, книги, игры, курсы) не обнаружено — обязательная возрастная маркировка по ФЗ-436 сайту не требуется. Нарушений нет.
Вердикт
Таким сайт enwp.org видят поисковые системы. Процент оптимизации невысокий - 56%. Чтобы улучшить сайт и попасть в ТОП необходимо:
Исправьте ошибки в мета-тегах.
Исправьте ошибки индексации.
Поделитесь с друзьями: