Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 9629 байт. Загружен за: 96мсек.
Проверяемая страница не запрещена в robots.txt.
Показать содержимое robots.txt
# ███████╗██╗ ██╗██████╗ ██╗ ██╗████████╗███╗ ███╗██╗ ███████╗
# ██╔════╝██║ ██║██╔══██╗██║ ██║╚══██╔══╝████╗ ████║██║ ██╔════╝
# █████╗ ██║ ██║██████╔╝███████║ ██║ ██╔████╔██║██║ ███████╗
# ██╔══╝ ██║ ██║██╔═══╝ ██╔══██║ ██║ ██║╚██╔╝██║██║ ╚════██║
# ██║ ███████╗██║██║ ██║ ██║ ██║ ██║ ╚═╝ ██║███████╗ ███████║
# ╚═╝ ╚══════╝╚═╝╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝╚══════╝ ╚══════╝
# ============================================================
# Updated-Date: 2026-06-29
# Author: Leary Liang
# Allowed AI bots: OpenAI (GPTBot, ChatGPT-User, OAI-SearchBot)
# Anthropic (ClaudeBot, anthropic-ai, Claude-Web, Claude-SearchBot)
# Apple (Applebot-Extended), Perplexity
# Blocked: Meta, Amazon, ByteDance, Common Crawl, all CN LLMs
# ============================================================
# ============================================================
# SECTION 1: Chinese Search Engines - FULL BLOCK
# ============================================================
User-agent: Baiduspider
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: Baiduspider-image
Disallow: /
User-agent: Baiduspider-video
Disallow: /
User-agent: Baiduspider-news
Disallow: /
User-agent: Baiduspider-favo
Disallow: /
User-agent: Baiduspider-cpro
Disallow: /
User-agent: Baiduspider-ads
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Sogou News Spider
Disallow: /
User-agent: Sogou inst spider
Disallow: /
User-agent: Sogou spider2
Disallow: /
User-agent: Sogou blog
Disallow: /
User-agent: Sogou Orion spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: YodaoBot
Disallow: /
User-agent: Jike Spider
Disallow: /
User-agent: 360Spider
Disallow: /
User-agent: 360spider-image
Disallow: /
User-agent: 360spider-video
Disallow: /
User-agent: HaosouSpider
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: ToutiaoSpider
Disallow: /
User-agent: Yahoo! Slurp China
Disallow: /
User-agent: EtaoSpider
Disallow: /
User-agent: yaanibot
Disallow: /
# ============================================================
# SECTION 2: AI Training Bots - BLOCK (selected)
# NOT blocked: GPTBot, ChatGPT-User, OAI-SearchBot,
# ClaudeBot, anthropic-ai, Claude-Web, Claude-SearchBot,
# Applebot-Extended, PerplexityBot
# ============================================================
User-agent: meta-externalagent
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: PanguBot
Disallow: /
User-agent: ChatGLM-Spider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: cohere-training-data-crawler
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: AI2Bot
Disallow: /
User-agent: AI2Bot-Dolma
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: img2dataset
Disallow: /
User-agent: omgili
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Webzio-Extended
Disallow: /
User-agent: YouBot
Disallow: /
User-agent: Timpibot
Disallow: /
User-agent: Kangaroo Bot
Disallow: /
User-agent: Sidetrade indexer bot
Disallow: /
User-agent: VelenPublicWebCrawler
Disallow: /
# ============================================================
# SECTION 2B: Chinese LLM Crawlers - FULL BLOCK
# ============================================================
User-agent: iaskspider
Disallow: /
User-agent: iaskspider/2.0
Disallow: /
User-agent: 360AISpider
Disallow: /
User-agent: TouTiaoSpider
Disallow: /
User-agent: MoonshotAI
Disallow: /
User-agent: Kimi-Spider
Disallow: /
User-agent: DeepSeekBot
Disallow: /
User-agent: DeepSeek-Crawler
Disallow: /
User-agent: TencentBot
Disallow: /
User-agent: BaichuanBot
Disallow: /
User-agent: BaichuanAI
Disallow: /
User-agent: SparkBot
Disallow: /
User-agent: xfspider
Disallow: /
User-agent: SkyworkBot
Disallow: /
User-agent: MiniMax-Spider
Disallow: /
User-agent: QwenSpider
Disallow: /
User-agent: DashScope
Disallow: /
User-agent: WenxinSpider
Disallow: /
User-agent: ERNIE-Bot
Disallow: /
User-agent: SenseBot
Disallow: /
User-agent: SenseChat
Disallow: /
User-agent: YuewenAI
Disallow: /
User-agent: YiBot
Disallow: /
User-agent: 01ai
Disallow: /
# ============================================================
# SECTION 3: SEO Analysis Crawlers - FULL BLOCK
# ============================================================
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: SemrushBot-SI
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: SemrushBot-CT
Disallow: /
User-agent: SemrushBot-OCOB
Disallow: /
User-agent: SemrushBot-FT
Disallow: /
User-agent: SplitSignalBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: Rogerbot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: SeobilityBot
Disallow: /
User-agent: SeekportBot
Disallow: /
User-agent: SeekrBot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SEOkicks
Disallow: /
User-agent: SEOstats
Disallow: /
User-agent: SeoSiteCheckup
Disallow: /
User-agent: SEOdiver Bot
Disallow: /
User-agent: SEO-Checker-bot
Disallow: /
User-agent: SiteAuditBot
Disallow: /
User-agent: cocolyzebot
Disallow: /
User-agent: Linguee Bot
Disallow: /
User-agent: Linguee
Disallow: /
User-agent: serpstatbot
Disallow: /
User-agent: Mediatoolkitbot
Disallow: /
User-agent: ZoominfoBot
Disallow: /
User-agent: SEOlyticsCrawler
Disallow: /
User-agent: Sistrix
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: Sitebulb
Disallow: /
User-agent: SafeDNSBot
Disallow: /
# ============================================================
# SECTION 4: Marketing / Listening / Link Monitoring - BLOCK
# HubSpot: precise block (SEO only, keep Email/Content crawlers)
# ============================================================
User-agent: HubSpot Crawler
Disallow: /
User-agent: HubSpot Webcrawler
Disallow: /
User-agent: AwarioBot
Disallow: /
User-agent: AwarioRssBot
Disallow: /
User-agent: AwarioSmartBot
Disallow: /
User-agent: BomboraBot
Disallow: /
User-agent: Clickagy Intelligence Bot
Disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: Morningscore Bot
Disallow: /
User-agent: BrandVerity
Disallow: /
User-agent: PiplBot
Disallow: /
User-agent: DomainStatsBot
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: FindLinks
Disallow: /
User-agent: linkCheck
Disallow: /
User-agent: CheckLinks
Disallow: /
User-agent: www.deadlinkchecker.com
Disallow: /
User-agent: getlinkinfo.com
Disallow: /
User-agent: harsilbot
Disallow: /
User-agent: MBCrawler
Disallow: /
User-agent: AddSearchBot
Disallow: /
User-agent: RankFlex
Disallow: /
User-agent: nooze-bot
Disallow: /
User-agent: OpenindexSpider
Disallow: /
User-agent: OpenWebSpider
Disallow: /
User-agent: YaK
Disallow: /
User-agent: webgains-bot
Disallow: /
User-agent: Qwantify
Disallow: /
User-agent: Qwantify-prod
Disallow: /
User-agent: BacklinksExtendedBot
Disallow: /
User-agent: BuckBot
Disallow: /
User-agent: Pcore-HTTP
Disallow: /
# ============================================================
# SECTION 5: Default rules for all other bots
# ============================================================
User-agent: *
Disallow: */help/
Disallow: */edit-book/*
Disallow: */404.php
Disallow: */access-statistics.php
Disallow: */accountinfo.php
Disallow: */bookinfo.php
Disallow: */buyvip.php
Disallow: */forget-password.php
Disallow: */invoice.php
Disallow: */left.php
Disallow: */paymentinfo.php
Disallow: */payvipinfo.php
Disallow: */register-successfully.php
Disallow: */reset-password.php
Disallow: /pass/
Disallow: /bookvisitinfo.html
Disallow: /statistic-server/
Disallow: /api/
Disallow: /check/
Disallow: /learning-center/*/tags/
Disallow: /*/learning-center/tags/
Disallow: /ebooks/
Disallow: */homepage/
Disallow: /*/*/templates/
Disallow: *?search=
Disallow: /blog/page/
Disallow: /blog/*/page/
Disallow: /blog/*/*/page/
Disallow: /blog/*/*/*/page/
Disallow: /blog/*/*/*/*/page/
Disallow: /*/javascript/config.js
Disallow: /*/slide_javascript/slideJS.js
Disallow: /*/files/pageEditor.js
Disallow: /*/javascript/plugin.json
Disallow: /*/getui.json
Disallow: /*/user/*.json
Disallow: /*config.js*
Disallow: /*slideJS.js*
Disallow: /*pageEditor.js*
Disallow: /*plugin.json*
Disallow: /*/power.json
Disallow: /online-tools/*/task-tool-*
Disallow: /online-tools/task-tool-*
# ============================================================
# SITEMAPS
# ============================================================
Sitemap: https://fliphtml5.com/sitemap/books.xml
Sitemap: https://fliphtml5.com/sitemap/books-image-index.xml
Sitemap: https://fliphtml5.com/sitemap/flipbooks-index.xml
Sitemap: https://fliphtml5.com/sitemap/templates/index.xml
Sitemap: https://fliphtml5.com/sitemap/bookcases.xml
Sitemap: https://fliphtml5.com/sitemap/latest-books.xml
Sitemap: https://fliphtml5.com/sitemap/pages.xml
Sitemap: https://fliphtml5.com/blog/sitemap_index.xml
Sitemap: https://online.fliphtml5.com/sitemap/books-index.xml
Sitemap: https://online.fliphtml5.com/sitemap/books-index-new.xml
Sitemap: https://online.fliphtml5.com/sitemap/latest-books-index.xml
Sitemap: https://online.fliphtml5.com/sitemap/latest-books-index-new.xml
Sitemap: https://help.fliphtml5.com/wp-sitemap.xml