Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 14098 байт. Загружен за: 0сек.
Проверяемая страница не запрещена в robots.txt.
Показать содержимое robots.txt
Sitemap: https://miniwebtool.com/sitemap.xml
Sitemap: https://miniwebtool.com/sitemap.txt
Sitemap: https://ru.miniwebtool.com/sitemap.xml
Sitemap: https://miniwebtool.com/video_sitemap.xml
# ---------------------------------------------------------------------------
# Faceted / paginated result URLs are never crawled; everything else is open,
# including every AI crawler listed below.
#
# The rule block is repeated verbatim for every user-agent ON PURPOSE. Groups in
# robots.txt are exclusive: a crawler that matches a named group ignores the '*'
# group entirely. The named groups here used to contain only 'Allow: /', which
# meant GPTBot, Bingbot, ClaudeBot and friends were exempt from every Disallow
# and crawled the whole job-finder/quote-finder URL space. If you add a rule,
# add it to all of them.
#
# Rules that target a non-first query parameter are written as /*?*&name - the
# leading '?' is what makes the pattern match against the query string.
#
# The previous pagination rule was 'Disallow: /*/*p=', a substring match that
# also blocked legitimate parameters ending in 'p' - ?ip=, ?step=, ?group=,
# ?top= - taking ~646k URLs out of the index, the IP converters among them.
#
# Generated shape: keep parameterless landing pages crawlable, block results.
# ---------------------------------------------------------------------------
User-agent: *
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: GPTBot
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: Google-Extended
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: CCBot
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: Applebot
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: Applebot-Extended
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: Bingbot
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: PerplexityBot
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: ClaudeBot
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
User-agent: Claude-Web
Allow: /
# job-finder: ?job=...&location=...&sort=...&p=N ('job' has no '=' when empty)
Disallow: /*?job
Disallow: /*?*&job
Disallow: /*?location=
Disallow: /*?*&location=
# quote-finder: ?author=... result pages, plus anything nested *below*
# /by-author/ - that is where the crawl trap lived
# (/by-author/b%27https:/mi/...). The /by-author/ tab itself stays crawlable.
Disallow: /*?author=
Disallow: /*?*&author=
Disallow: /*by-author/*/
# quote word-search pagination. Scoped to 'word'+'p' together on purpose:
# ?word= alone is also the anagram generator's share link, and page 1 of a word
# search is a fine landing page. Only the paginated tail is blocked.
Disallow: /*?word=*&p=
Disallow: /*?*&word=*&p=
# NOTE: there is deliberately no blanket 'Disallow: /*?p='. 'p' is the power
# input on electricity-calculator and the prime on the RSA simulator, and a
# blanket rule would block those share URLs - the exact mistake the old
# '/*/*p=' rule made. Pagination on the faceted tools is already covered above
# via job / location / author / word, which their paginated URLs always carry.
# Every other parameter URL is handled by canonical consolidation in
# RedirectMiddleware, not by robots.txt.
Disallow: /*/facebook-user-id-lookup/*=
Cloudflare