Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 8266 байт. Загружен за: 1сек.
Проверяемая страница не запрещена в robots.txt.
Цепочка редиректов для файла robots.txt:
http://ithacadirectory.com/robots.txt
301 MovedPermanently
https://www.ithacadirectory.com/
200 OK
Показать содержимое robots.txt
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Ithaca Directory :: Home - Ithaca, New York Business and Website Listings</title>
<link rel="canonical" href="https://www.ithacadirectory.com" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/><link rel="shortcut icon" href="favicon.ico" type="image/x-icon"/>
<link rel="stylesheet" type="text/css" href="css/style.css"/><!--[if IE]><link rel="stylesheet" type="text/css" media="screen" href="/css/iestyles.css" /><![endif]-->
<script type="text/javascript" src="js/main.js"></script></head>
<body><div id="awp"><a href="http://www.awpny.com/" title="AWP - New York Web and Graphic Design" target="_blank"></a></div>
</div>
<div id="wrapper">
<div id="header">
<a href="categories/index"><img src="img/spacer.gif" alt="Ithaca Directory" width="240" height="40" border="0" id="logo"/></a> <p class="tagline">Local news, lodging, entertainment, nightlife and business listings for Ithaca, New York 14850.</p>
<img src="weathersticker/gizmotimetemp/language/www/US/NY/Ithaca.gif" border="0" alt="Click for Ithaca, New York Forecast" height="41" width="127" id="wunderground"/>
</div>
<!-- Begin Menu-->
<div id="menu"><a href="categories/index">Browse</a> <a href="businesses/search">Search</a> <a href="businesses/whatsnew">Newest</a> <a href="businesses/random">Random</a> <a href="pages/about">About</a> <a href="businesses/submit">Submit a Site</a></div>
<!-- End Menu --><div id="content">
<div id="mainCol">
<!-- Begin View -->
<script language="JavaScript" type="text/javascript" src="adserver/adx.js"></script>
<script language="JavaScript" type="text/javascript">
<!--
if (!document.phpAds_used) document.phpAds_used = ',';
phpAds_random = new String (Math.random()); phpAds_random = phpAds_random.substring(2,11);
document.write ("<" + "script language='JavaScript' type='text/javascript' src='");
document.write ("http://www.ancientwisdom.cc/adserver/adjs.php?n=" + phpAds_random);
document.write ("&what=zone:5&blockcampaign=1");
document.write ("&exclude=" + document.phpAds_used);
if (document.referrer)
document.write ("&referer=" + escape(document.referrer));
document.write ("'><" + "/script>");
//-->
</script><noscript></noscript>
<script language="JavaScript" type="text/javascript">
<!--
if (!document.phpAds_used) document.phpAds_used = ',';
phpAds_random = new String (Math.random()); phpAds_random = phpAds_random.substring(2,11);
document.write ("<" + "script language='JavaScript' type='text/javascript' src='");
document.write ("http://www.ancientwisdom.cc/adserver/adjs.php?n=" + phpAds_random);
document.write ("&what=zone:5&blockcampaign=1");
document.write ("&exclude=" + document.phpAds_used);
if (document.referrer)
document.write ("&referer=" + escape(document.referrer));
document.write ("'><" + "/script>");
//-->
</script><noscript></noscript>
<hr/>
<h1>Ithaca-Area Business and Website Directory</h1>
<p style="font-size:1.1em;line-height:1.6em;">There's a lot to Ithaca, and the <strong>Ithaca Directory</strong> helps you find it all. Explore the myraid links to local businesses, groups, and services - all approved to be relavent and real. You can <a href="businesses/submit">submit your own site</a>, or just <a href="categories/view/7">browse through the directory</a> and see what you <a href="businesses/random" title="Random Directory Link">find</a>.</p>
<div id="homeCategoryBlockLeft">
<h3><a href="categories/view/6">Art</a> <span class="catCount">(359 listings)</span></h3>
<div class="categoryDescription">Galleries, Artists, Crafters and more...</div>
<h3><a href="categories/view/7">Business & Professional</a> <span class="catCount">(8252 listings)</span></h3>
<div class="categoryDescription">Everything from Graphic Design to Law & Legal Services, Lodging and Tourism.</div>
<h3><a href="categories/view/8">Culture</a> <span class="catCount">(154 listings)</span></h3>
<div class="categoryDescription">Museums, Clubs, Movies, Festivals, Societies, and more...</div>
<h3><a href="categories/view/9">Education</a> <span class="catCount">(168 listings)</span></h3>
<div class="categoryDescription">Colleges, Universities, Grade School and more...</div>
<h3><a href="categories/view/10">Food & Drink</a> <span class="catCount">(181 listings)</span></h3>
<div class="categoryDescription">Restaurants, Bars, Wineries, Coffee Houses, and more...</div>
<h3><a href="categories/view/11">Government</a> <span class="catCount">(23 listings)</span></h3>
<div class="categoryDescription">Local government websites and other sites of political nature</div>
</div>
<div id="homeCategoryBlockRight">
<h3><a href="categories/view/1">Music</a> <span class="catCount">(123 listings)</span></h3>
<div class="categoryDescription">Ithaca-based bands and musicians, music stores, and resources.</div>
<h3><a href="categories/view/13">News & Information</a> <span class="catCount">(114 listings)</span></h3>
<div class="categoryDescription">Local sources of news and information</div>
<h3><a href="categories/view/14">Other sites</a> <span class="catCount">(489 listings)</span></h3>
<div class="categoryDescription">Other sites of local interest</div>
<h3><a href="categories/view/16">Religious & Spiritual</a> <span class="catCount">(62 listings)</span></h3>
<div class="categoryDescription">All-faith directory of places of worship and more...</div>
<h3><a href="categories/view/15">Sports & Recreation</a> <span class="catCount">(274 listings)</span></h3>
<div class="categoryDescription">Local teams, places of sport and recreation, sports equipment, and more...</div>
</div>
<br clear="all"/>
<hr/>
<div id="other-news">
<p class="black-bold">Local News Sources:</p>
<p style="margin:7px 0px 7px 0px">The Ithaca Journal • Ithaca Times • Ithaca Community News</p>
<hr/>
</div><!-- End View -->
</div>
<div id="sidebar">
<img src="img/sidebar-top.gif" alt="top"/> <div class="sidebar-padding">
<div align="center"></div>
<p>Live View of the Ithaca Commons</p>
<div class="line"></div>
<!--<form action="/businesses/search" method="post">-->
<form id="BusinessSearchForm" method="post" action="businesses/search"><fieldset style="display:none;"><input type="hidden" name="_method" value="POST"/></fieldset> <p style="font-size:12px;">Search the Directory:</p>
<input name="data[Business][search]" type="text" class="searchBox" value="" id="BusinessSearch"/> <br/>
<p><!--<input type="submit" value="Search" />--><div class="submit"><input type="submit" value="Search"/></div></p>
</form> <br/>
<div class="line"></div>
<div class="line"></div>
<br/>
<script type="text/javascript">
//2007-04-26: ithaca directory sidebar
</script>
</div>
<img src="img/sidebar-bot.gif" alt="bottom of sidebar"/> </div>
</div>
<div id="footer">
<p><a href="businesses/submit" title="Submit Your Site to the Ithaca Directory">Submit a Site</a> - <a href="pages/advertise" title="Advertise on the Ithaca Directory">Advertise on the Directory</a> - <a href="pages/featured" 0="0">Feature Your Listing</a></p>
<p>Copyright ©2007 AWP. All Rights Reserved.</p></div>
<!-- <div align="center"> <a href="http://whos.amung.us/show/s2a11m2r"><img align="center" src="http://whos.amung.us/widget/s2a11m2r" alt="tracker" width="81" height="29" border="0" /></a></div> -->
<p> </p>
</div>
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write("\<script src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'>\<\/script>" );
</script>
<script type="text/javascript">
pageTracker._initData();
</script>
<script type="module" src="https://static.cloudflareinsights.com/beacon.min.js/v4513226cdae34746b4dedf0b4dfa099e1781791509496" integrity="sha512-ZE9pZaUXND66v380QUtch/5sE9tPFh2zg45pR2PB0CVkCtOREv2AJKkSidISWkysEuQ0EH8faUU5du78bx87UQ==" data-cf-beacon='{"version":"2024.11.0","token":"d1ed03e8fc6f4d7f8e9b38682bc8d840","r":1}' crossorigin="anonymous"></script>
</body>
</html>
Cloudflare