Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 7178 байт. Загружен за: 1сек.
Проверяемая страница не запрещена в robots.txt.
Цепочка редиректов для файла robots.txt:
https://penandanvil.com/robots.txt
302 Found
http://www.penandanvil.com/404/index.htm
200 OK
Показать содержимое robots.txt
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Pen and Anvil Press</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link href="../style.css" rel="stylesheet" type="text/css">
<link rel="apple-touch-icon-precomposed" sizes="57x57" href="/images/apple-touch-icon-57x57.png" />
<link rel="apple-touch-icon-precomposed" sizes="114x114" href="/images/apple-touch-icon-114x114.png" />
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="/images/apple-touch-icon-72x72.png" />
<link rel="apple-touch-icon-precomposed" sizes="144x144" href="/images/apple-touch-icon-144x144.png" />
<link rel="apple-touch-icon-precomposed" sizes="60x60" href="/images/apple-touch-icon-60x60.png" />
<link rel="apple-touch-icon-precomposed" sizes="120x120" href="/images/apple-touch-icon-120x120.png" />
<link rel="apple-touch-icon-precomposed" sizes="76x76" href="/images/apple-touch-icon-76x76.png" />
<link rel="apple-touch-icon-precomposed" sizes="152x152" href="/images/apple-touch-icon-152x152.png" />
<link rel="icon" type="image/png" href="/images/favicon-196x196.png" sizes="196x196" />
<link rel="icon" type="image/png" href="/images/favicon-96x96.png" sizes="96x96" />
<link rel="icon" type="image/png" href="/images/favicon-32x32.png" sizes="32x32" />
<link rel="icon" type="image/png" href="/images/favicon-16x16.png" sizes="16x16" />
<link rel="icon" type="image/png" href="/images/favicon-128.png" sizes="128x128" />
<meta name="application-name" content=" "/>
<meta name="msapplication-TileColor" content="#FFFFFF" />
<meta name="msapplication-TileImage" content="/images/mstile-144x144.png" />
<meta name="msapplication-square70x70logo" content="/images/mstile-70x70.png" />
<meta name="msapplication-square150x150logo" content="/images/mstile-150x150.png" />
<meta name="msapplication-wide310x150logo" content="/images/mstile-310x150.png" />
<meta name="msapplication-square310x310logo" content="/images/mstile-310x310.png" />
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-19577102-1', 'auto');
ga('send', 'pageview');
</script>
<META http-equiv="refresh" content="6;URL=http://www.penandanvil.com">
</head>
<body>
<table width="1000" border="0" align="center" cellpadding="20" cellspacing="0">
<tr>
<td><table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="216" rowspan="3"><a href="http://www.penandanvil.com"><img src="../images/text-only-logo.gif" alt="Pen and Anvil Press - Homepage" width="216" height="216" border="0"></a></td>
<td width="32" rowspan="3"><img src="../images/spacer-32.png" width="32" height="32"></td>
<td width="32" rowspan="3" bgcolor="#000000"><img src="../images/spacer-32.png" width="32" height="32"></td>
<td width="432" bgcolor="#000000"><img src="../images/spacer-32.png" width="432" height="100"></td>
<td width="32" rowspan="3" bgcolor="#000000"><img src="../images/spacer-32.png" width="32" height="32"></td>
<td width="216" rowspan="3" bgcolor="#000000"><img src="../images/logo.gif" alt="Pen and Anvil" width="216" height="216"></td>
</tr>
<tr>
<td bgcolor="#000000"><table width="100%" border="0" cellpadding="10" cellspacing="0" bgcolor="#000000">
<tr valign="middle">
<td align="left" valign="middle" class="bookboxtitle"><a href="http://www.penandanvil.com" class="bookboxtitle">Home</a></td>
<td align="left" valign="middle" class="bookboxtitle"><a href="http://www.penandanvil.com/catalogue" class="bookboxtitle">Catalogue</a></td>
<td align="left" valign="middle" class="bookboxtitle"><a href="http://www.penandanvil.com/about" class="bookboxtitle">About the Press </a></td>
</tr>
<tr valign="middle">
<td align="left" valign="middle" class="bookboxtitle"><a href="http://www.penandanvil.com/submit" class="bookboxtitle">Submit</a></td>
<td align="left" valign="middle" class="bookboxtitle"><a href="http://www.penandanvil.com/contact" class="bookboxtitle">Contact Us </a></td>
<td align="left" valign="middle" class="bookboxtitle"><a href="http://www.penandanvil.com/blog" class="bookboxtitle">The Ampersand blog </a></td>
</tr>
</table></td>
</tr>
<tr>
<td bgcolor="#000000"><img src="../images/spacer-32.png" width="432" height="32"></td>
</tr>
</table></td>
</tr>
</table>
<table width="1000" border="0" align="center" cellpadding="20" cellspacing="0">
<tr>
<td><table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top">
<td width="216" rowspan="2"><span class="blurblarge">Er, look, mate— I'm sorry to tell you this, but the page you were looking for isn't here. </span></td>
<td width="32" rowspan="2"><img src="../images/spacer-32.png" width="32" height="32"></td>
<td><table width="100%" border="0" cellpadding="0" cellspacing="10" bgcolor="#000000">
<tr valign="middle">
<td class="bookboxtitle">404 Page Error </td>
</tr>
</table></td>
</tr>
<tr valign="top">
<td><table width="100%" border="0" cellpadding="10" cellspacing="0" class="blurb">
<tr>
<td><div align="left">
<p><img src="../images/graphic-404-monet.jpg" width="100%"></p>
<p>"<a href="https://en.wikipedia.org/wiki/All_that_is_gold_does_not_glitter" target="_blank" class="blurb"><u>Not all those who wander are lost</u></a>", Tolkein wrote; but if your wandering has brought you here, perhaps you a bit turned around. Let us point you in the right direction. Your browser will take you back to the Pen & Anvil homepage in a few moments; if it does not, <a href="http://www.penandanvil.com" class="blurb"><u>click here</u></a> to be taken there directly.</p>
<p> </p>
<p align="right">graphic: Claude Monet, "Infantry Guards Wandering along the River", 1870 (detail)<br>
</p>
</div></td>
</tr>
</table></td>
</tr>
</table></td>
</tr>
</table>
<!-- StatCounter Code -->
<script type="text/javascript">
var sc_project=6372179;
var sc_invisible=1;
var sc_security="4fea85fc";
var scJsHost = (("https:" == document.location.protocol) ?
"https://secure." : "http://www.");
document.write("<sc"+"ript type='text/javascript' src='" +
scJsHost+
"statcounter.com/counter/counter.js'></"+"script>");
</script>
<noscript><div class="statcounter"><a title="shopify stats"
href="http://statcounter.com/shopify/" target="_blank"><img
class="statcounter"
src="http://c.statcounter.com/6372179/0/4fea85fc/1/"
alt="shopify stats"></a></div></noscript>
<!-- End of StatCounter Code -->
</body>
</html>
Apache