Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 12554 байт. Загружен за: 0сек.
Проверяемая страница не запрещена в robots.txt.
Показать содержимое robots.txt
# As a condition of accessing this website, you agree to abide by the following
# content signals:
# (a) If a Content-Signal = yes, you may collect content for the corresponding
# use.
# (b) If a Content-Signal = no, you may not collect content for the
# corresponding use.
# (c) If the website operator does not include a Content-Signal for a
# corresponding use, the website operator neither grants nor restricts
# permission via Content-Signal with respect to the corresponding use.
# The content signals and their meanings are:
# search: building a search index and providing search results (e.g., returning
# hyperlinks and short excerpts from your website's contents). Search does not
# include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
# augmented generation, grounding, or other real-time taking of content for
# generative AI search answers).
# ai-train: training or fine-tuning AI models.
# use: how AI systems may consume the content (immediate, reference, or full).
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CloudflareBrowserRenderingCrawler
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# END Cloudflare Managed Content
# AI Search Crawler
# https://knownagents.com/agents/addsearchbot
User-agent: AddSearchBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/ai2bot
User-agent: AI2Bot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/ai2bot-deepresearcheval
User-agent: AI2Bot-DeepResearchEval
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/ai2bot-dolma
User-agent: Ai2Bot-Dolma
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/amazon-kendra
User-agent: amazon-kendra
Disallow: /
# AI Assistant
# https://knownagents.com/agents/amazon-qbusiness
User-agent: amazon-QBusiness
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/amazonbot
User-agent: Amazonbot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/amzn-searchbot
User-agent: Amzn-SearchBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/amzn-user
User-agent: Amzn-User
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/anomura
User-agent: Anomura
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/anthropic-ai
User-agent: anthropic-ai
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/applebot
User-agent: Applebot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/applebot-extended
User-agent: Applebot-Extended
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/aranet-searchbot
User-agent: Aranet-SearchBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/atlassian-bot
User-agent: atlassian-bot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/azureai-searchbot
User-agent: AzureAI-SearchBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/bedrockbot
User-agent: bedrockbot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/bigsur-ai
User-agent: bigsur.ai
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/bytespider
User-agent: Bytespider
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/ccbot
User-agent: CCBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/channel3bot
User-agent: Channel3Bot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/chatglm-spider
User-agent: ChatGLM-Spider
Disallow: /
# AI Assistant
# https://knownagents.com/agents/chatgpt-user
User-agent: ChatGPT-User
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/claude-searchbot
User-agent: Claude-SearchBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/claude-user
User-agent: Claude-User
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/claude-web
User-agent: Claude-Web
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/claudebot
User-agent: ClaudeBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/cloudflare-autorag
User-agent: Cloudflare-AutoRAG
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/cloudvertexbot
User-agent: CloudVertexBot
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/cohere-ai
User-agent: cohere-ai
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/cohere-training-data-crawler
User-agent: cohere-training-data-crawler
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/coherebot
User-agent: CohereBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/cotoyogi
User-agent: Cotoyogi
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/crawl4ai
User-agent: Crawl4AI
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/datenbank-crawler
User-agent: Datenbank Crawler
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/deepseekbot
User-agent: DeepSeekBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/doubaobot
User-agent: Doubaobot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/duckassistbot
User-agent: DuckAssistBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/erniebot
User-agent: ERNIEBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/exasearchbot
User-agent: ExaSearchBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/facebookbot
User-agent: FacebookBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/geisthaus-pagefetcher
User-agent: GeistHaus-PageFetcher
Disallow: /
# AI Assistant
# https://knownagents.com/agents/gemini-deep-research
User-agent: Gemini-Deep-Research
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/google-cloudvertexbot
User-agent: Google-CloudVertexBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/google-extended
User-agent: Google-Extended
Disallow: /
# AI Assistant
# https://knownagents.com/agents/google-notebooklm
User-agent: Google-NotebookLM
Disallow: /
# AI Assistant
# https://knownagents.com/agents/googleagent-urlcontext
User-agent: GoogleAgent-URLContext
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/googleother
User-agent: GoogleOther
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/gptbot
User-agent: GPTBot
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/iaskbot
User-agent: iAskBot
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/iaskspider
User-agent: iaskspider
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/icc-crawler
User-agent: ICC-Crawler
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/imagespider
User-agent: imageSpider
Disallow: /
# AI Assistant
# https://knownagents.com/agents/kagi-fetcher
User-agent: kagi-fetcher
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/kangaroo-bot
User-agent: Kangaroo Bot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/kimi-searchbot
User-agent: Kimi-SearchBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/kimi-user
User-agent: Kimi-User
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/kimibot
User-agent: KimiBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/kimicrawler
User-agent: KimiCrawler
Disallow: /
# AI Assistant
# https://knownagents.com/agents/klaviyoaibot
User-agent: KlaviyoAIBot
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/kunatocrawler
User-agent: KunatoCrawler
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/laion-huggingface-processor
User-agent: laion-huggingface-processor
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/lcc
User-agent: LCC
Disallow: /
# AI Assistant
# https://knownagents.com/agents/linerbot
User-agent: LinerBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/linkupbot
User-agent: LinkupBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/meta-externalagent
User-agent: meta-externalagent
Disallow: /
# AI Assistant
# https://knownagents.com/agents/meta-externalfetcher
User-agent: meta-externalfetcher
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/meta-webindexer
User-agent: meta-webindexer
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/micro-crawl
User-agent: micro-crawl
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/mistralai-index
User-agent: MistralAI-Index
Disallow: /
# AI Assistant
# https://knownagents.com/agents/mistralai-user
User-agent: MistralAI-User
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/mistralbot
User-agent: MistralBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/moonshotbot
User-agent: MoonshotBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/moonspider
User-agent: MoonSpider
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/netestate-imprint-crawler
User-agent: netEstate Imprint Crawler
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/oai-searchbot
User-agent: OAI-SearchBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/omgili
User-agent: omgili
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/pangubot
User-agent: PanguBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/perplexity-user
User-agent: Perplexity-User
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/perplexitybot
User-agent: PerplexityBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/petalbot
User-agent: PetalBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/phindbot
User-agent: PhindBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/poggio-citations
User-agent: Poggio-Citations
Disallow: /
# AI Assistant
# https://knownagents.com/agents/qualifiedbot
User-agent: QualifiedBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/qwenbot
User-agent: QwenBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/reflectionbot
User-agent: Reflectionbot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/salamandravlm
User-agent: SalamandraVLM
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/sbintuitionsbot
User-agent: SBIntuitionsBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/shap-user
User-agent: Shap-User
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/spider
User-agent: Spider
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/talarionsearchbot
User-agent: TalarionSearchBot
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/timpibot
User-agent: Timpibot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/tongyibot
User-agent: TongyiBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/useai
User-agent: UseAI
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/velenpublicwebcrawler
User-agent: VelenPublicWebCrawler
Disallow: /
# AI Data Scraper
# https://knownagents.com/agents/webzio-extended
User-agent: webzio-extended
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/wrtnbot
User-agent: WRTNBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/xai-searchbot
User-agent: xAI-SearchBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/yandexadditional
User-agent: YandexAdditional
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/yandexadditionalbot
User-agent: YandexAdditionalBot
Disallow: /
# AI Assistant
# https://knownagents.com/agents/yiyanbot
User-agent: YiyanBot
Disallow: /
# AI Search Crawler
# https://knownagents.com/agents/zanistabot
User-agent: ZanistaBot
Disallow: /
# Undocumented AI Agent
# https://knownagents.com/agents/zenphonybot
User-agent: ZenphonyBot
Disallow: /