Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 13399 байт. Загружен за: 1сек.
Проверяемая страница не запрещена в robots.txt.
Цепочка редиректов для файла robots.txt:
http://pantheon.org/robots.txt
301 MovedPermanently
https://pantheon.org/
200 OK
Показать содержимое robots.txt
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Encyclopedia Mythica</title>
<meta name="description"
content="Encyclopedia Mythica is the premier encyclopedia on mythology, folklore, and religion. Instant mythology since 1995.">
<meta name="keywords"
content="encyclopedia, dictionary, encyclopaedia, handbook, mythology, folklore, religion, classical mythology">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="cleartype" content="on">
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="viewport" content="width=device-width, initial-scale=1, minimal-ui">
<meta name="apple-mobile-web-app-title" content="Mythica">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black">
<meta name="twitter:card" content="summary">
<meta name="twitter:site" content="@mythica">
<meta property="og:title" content="Encyclopedia Mythica">
<meta property="og:url" content="https://pantheon.org/">
<meta property="og:type" content="website">
<meta property="og:image" content="https://pantheon.org/static/img/em.png">
<link rel="shortcut icon" href="/static/touch/apple-touch-icon.png">
<link rel="apple-touch-icon-precomposed" sizes="180x180"
href="/static/touch/apple-touch-icon-180x180-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="152x152"
href="/static/touch/apple-touch-icon-152x152-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="144x144"
href="/static/touch/apple-touch-icon-144x144-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="120x120"
href="/static/touch/apple-touch-icon-120x120-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="114x114"
href="/static/touch/apple-touch-icon-114x114-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="/static/touch/apple-touch-icon-72x72-precomposed.png">
<link rel="apple-touch-icon-precomposed" href="/static/img/touch/apple-touch-icon-57x57-precomposed.png">
<link rel="shortcut icon" sizes="196x196" href="/static/touch/apple-touch-icon-196x196.png">
<meta name="msapplication-TileImage" content="/static/touch/apple-touch-icon-144x144-precomposed.png">
<meta name="msapplication-TileColor" content="#4b82cc">
<meta name="theme-color" content="#4b82cc">
<link rel="shortcut icon" href="/static/img/mythica.ico">
<link rel="search" type="application/opensearchdescription+xml" href="/search/opensearch.xml" title="Mythica">
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css"
integrity="sha384-MCw98/SFnGE8fJT3GXwEOngsV7Zt27NXFoaoApmYm81iuXoPkFOJwJ8ERdknLPMO" crossorigin="anonymous">
<link rel="stylesheet" href="/static/css/mythica.min.css">
<script data-ad-client="ca-pub-8610010340209437" async
src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<meta property="og:description"
content="Encyclopedia Mythica is the premier encyclopedia on mythology, folklore, and religion. Instant mythology since 1995.">
</head>
<body>
<div id="wrapper-main" class="cms-editable">
<div id="mainheader" class="mainheader text-center" role="banner">
<h1 class="sr-only">Encyclopedia Mythica</h1>
</div>
<div class="container-fluid" role="main">
<div class="row my-3 px-0">
<div class="col text-center">
<h1>Mythology. Folklore. Religion.</h1>
<p>Search through <strong>11,420</strong> articles:</p>
<form role="search" action="/search/" method="get"><input name="ie" type="hidden" value="utf-8">
<div class="form-group row">
<div class="col-md-3"> </div>
<div class="col-md-6">
<div class="input-group"><label class="sr-only" for="q">Search</label> <input id="q" accesskey="f"
class="form-control" title="Search Encyclopedia Mythica" autocomplete="off" name="q"
type="text" placeholder="try 'sky god'">
<div class="input-group-append"><button class="btn btn-primary" type="submit">Search</button></div>
</div>
<div class="col-md-3"> </div>
</div>
</div>
</form>
</div>
</div>
<div class="container-fluid px-0" role="complementary">
<div class="row my-3">
<div class="col-md-4 pb-3">
<div class="card bg-light">
<div class="card-body">
<h3 class="card-title">Today's deity</h3>
<h4 class="card-title">Pērkons</h4>
<p class="card-text">The god of thunder, as well as a god of fertility who brings beneficial rain.<br><a
href="articles/p/perkons.html">Read more...</a></p>
<p class="card-text"><small class="text-muted">From: <a title="Browse Latvian mythology."
href="mythology/latvian/">Latvian mythology</a></small></p>
</div>
</div>
</div>
<div class="col-md-4 pb-3">
<div class="card bg-light">
<div class="card-body">
<h3 class="card-title">Today's feastdays</h3>
<h4 class="card-title">January 7</h4>
<ul class="card-text">
<li>Nanakusa</li>
<li>St. Raymond of Penafort</li>
</ul>
</div>
</div>
</div>
<div class="col-md-4 pb-3">
<div class="card bg-light">
<div class="card-body text-muted">
<h3 class="card-title">Popular areas</h3>
<ol class="card-text pl-3">
<li><a href="mythology/greek/">Greek mythology</a> <small class="text-muted">(1,605 articles)</small>
</li>
<li><a href="mythology/roman/">Roman mythology</a> <small class="text-muted">(571 articles)</small>
</li>
<li><a href="mythology/norse/">Norse mythology</a> <small class="text-muted">(762 articles)</small>
</li>
<li><a href="folklore/arthurian/">Arthurian legend</a> <small class="text-muted">(184
articles)</small></li>
<li><a href="mythology/maori/">Māori mythology</a> <small class="text-muted">(1,210 articles)</small>
</li>
</ol>
<hr>
<form class="jump"><label class="sr-only" for="selectarea">Area</label><select id="selectarea"
class="form-control" name="area">
<option value>--- Select area ---</option>
<optgroup label="Mythology">
<option value="/mythology/aboriginal/">Aboriginal</option>
<option value="/mythology/central_asian/">Central Asian</option>
<option value="/mythology/greek/">Greek</option>
<option value="/mythology/greek_people/">Greek people</option>
<option value="/mythology/judaic/">Judaic</option>
<option value="/mythology/korean/">Korean</option>
<option value="/mythology/latvian/">Latvian</option>
<option value="/mythology/maori/">Māori</option>
<option value="/mythology/melanesian/">Melanesian</option>
<option value="/mythology/micronesian/">Micronesian</option>
<option value="/mythology/native_american/">Native American</option>
<option value="/mythology/norse/">Norse</option>
<option value="/mythology/philippine/">Philippine</option>
<option value="/mythology/roman/">Roman</option>
<option value="/mythology/southeast_asian/">Southeast Asian</option>
</optgroup>
<optgroup label="Folklore">
<option value="/folklore/arthurian/">Arthurian</option>
<option value="/folklore/british/">British</option>
<option value="/folklore/general/">General</option>
</optgroup>
</select></form>
</div>
</div>
</div>
</div>
</div>
<div id="footer" class="my-5" role="contentinfo">
<div class="container-fluid px-0">
<div class="row">
<div class="col-md-6">
<p class="text-muted about"><em>Encyclopedia Mythica</em> is an internet encyclopedia on mythology,
folklore, and religion.<br>Everything from <em>Aaron</em> to <em>Zygius</em>, with thousands of articles
in between.<br><br>In partnership with Excel Education. Excel Education offers <a
href="https://www.excelhighschool.com/ace-credits-online-for-college">Online College Credit</a> for
ACE recommended credit and:</p>
<p class="text-muted about"><a href="https://www.washingtontech.edu/paralegal.html" target="_blank"
rel="noopener">Online Paralegal Program</a> at WTI</p>
<p class="text-muted about"><a href="https://www.northgateacademy.com/" target="_blank"
rel="noopener">Homeschooling Online</a> at Northgate Academy</p>
<p class="text-muted about"><a href="https://www.safeflightaviation.com/" target="_blank"
rel="noopener">Safe Flight Aviation Flight School </a></p>
<p class="text-muted">Follow @mythica on <a class="twitter" href="https://twitter.com/mythica">Twitter</a>
or on <a class="facebook" href="https://facebook.com/encyclopediamythica">Facebook</a></p>
</div>
<div class="col-md-2">
<h4>Information</h4>
<ul class="list-unstyled text-muted">
<li><a href="about/">About</a></li>
<li><a href="about/copyright.php">Copyright</a></li>
<li><a href="about/tos.php">Terms</a></li>
<li><a href="about/privacy.php">Privacy</a></li>
<li><a href="about/cookies.php">Cookies</a></li>
<li><a href="contact/">Contact</a></li>
</ul>
</div>
<div class="col-md-2">
<h4>Sections</h4>
<ul class="list-unstyled text-muted">
<li><a href="mythology/americas.php">Americas</a></li>
<li><a href="mythology/asia.php">Asia</a></li>
<li><a href="mythology/europe.php">Europe</a></li>
<li><a href="mythology/middle_east.php">Middle East</a></li>
<li><a href="mythology/oceania.php">Oceania</a></li>
<li><a href="folklore/">Folklore</a></li>
<li><a href="miscellaneous/">Miscellaneous</a></li>
</ul>
<h4>Coming soon</h4>
<ul class="list-unstyled text-muted">
<li>Africa</li>
</ul>
</div>
<div class="col-md-2">
<h4>Lastest entries</h4>
<ol class="list-unstyled text-muted">
<li><a href="articles/b/bengh.html">o Bengh</a></li>
<li><a href="articles/d/damo_and_yehwah.html">Damo and Yehwah</a></li>
<li><a href="articles/r/raashiel.html">Ra'ashiel</a></li>
<li><a href="articles/t/tekhelet.html">Tekhelet</a></li>
<li><a href="articles/m/magician.html">magician</a></li>
<li><a href="articles/s/silky.html">Silky</a></li>
<li><a href="articles/h/habetrot.html">Habetrot</a></li>
<li><a href="articles/p/powrie.html">powrie</a></li>
<li><a href="articles/m/monoceros.html">monoceros</a></li>
<li><a href="articles/t/turrobolen.html">turrobolen</a></li>
</ol>
</div>
</div>
</div>
<div class="my-3">
<p class="text-muted"><small> <a class="text-muted" href="about/copyright.php">©</a> MCMXCV–MMXXI Encyclopedia
Mythica.<br>Some rights reserved.<br><a
href="https://creativecommons.org/licenses/by-nc-nd/4.0/deed.en_US"><img style="margin-top: 5px;"
src="static/img/80x15.png" alt="CC BY-NC-ND 3.0 US" width="80" height="15"></a></small></p>
</div>
</div>
<script src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
<script src="static/js/main.min.js"></script>
<!-- Google Analytics -->
<script>
var _gaq = [
['_setAccount', 'UA-148105-1'],
['_trackPageview']
];
(function(d, t) {
var g = d.createElement(t),
s = d.getElementsByTagName(t)[0];
g.async = 1;
g.src = ('https:' == location.protocol ? '//ssl' : '//www') + '.google-analytics.com/ga.js';
s.parentNode.insertBefore(g, s);
})(document, 'script');
</script>
</div>
</div>
</body>
</html>