Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 6465 байт. Загружен за: 1сек.
Проверяемая страница не запрещена в robots.txt.
Цепочка редиректов для файла robots.txt:
http://classics.mit.edu/robots.txt
302 Found
https://classics.mit.edu/
200 OK
Показать содержимое robots.txt
<HTML>
<HEAD>
<!--The Internet Classics Archive by Daniel C. Stevenson, Web Atomics. World Wide Web presentation is copyright (C) 1994-2009, Daniel C. Stevenson, Web Atomics. All rights reserved under international and pan-American copyright conventions, including the right of reproduction in whole or in part in any form. Direct permission requests to classics@classics.mit.edu. Translation of "The Deeds of the Divine Augustus" by Augustus is copyright (C) Thomas Bushnell, BSG.-->
<LINK REV="made" HREF="mailto:classics@classics.mit.edu">
<LINK REV="owns" HREF="mailto:classics@classics.mit.edu">
<STYLE TYPE="text/css">
<!--
A:link {text-decoration:none}
A:visited {text-decoration:none}
A:active {text-decoration:none}
-->
</STYLE>
<link rel="shortcut icon" href="favicon.ico" type="image/vnd.microsoft.icon">
<META NAME="description" CONTENT="Home page, part of the Internet Classics Archive">
<META NAME="keywords" CONTENT="classical, literature, classics, ancient, civilization, books, great books, philosophy, history, Greek, Latin, Roman, Persian, Chinese, Greece, Rome, commentary, pictures, bookstore, CD-ROM">
<TITLE>The Internet Classics Archive: 441 searchable works of classical literature</TITLE>
</HEAD>
<BODY BGCOLOR="FFFFCC" LINK="990033" VLINK="990033" TEXT="000000">
<DIV ALIGN="CENTER"><IMG SRC="Images/ICA-banner-smaller.gif" ALIGN="TOP" WIDTH="368" HEIGHT="71" ALT="The Internet Classics Archive: 441 searchable works of classical literature">
<BR><BR>
<FONT COLOR="990033">Bringing the wisdom of the classics to the
Internet since 1994.</FONT></DIV>
<br>
<BR><TABLE WIDTH="98%" BORDER="0" CELLSPACING="7" CELLPADDING="0">
<TR VALIGN="TOP">
<TD ALIGN="CENTER" NOWRAP><A HREF="Browse/index.html"><IMG SRC="Images/browse-icon.gif" WIDTH="30" HEIGHT="30" BORDER="0" ALT="Browse a list of titles"></A>
<FONT SIZE="-1"><BR><A HREF="Browse/index.html">Browse and<BR>Comment</A></FONT></TD>
<TD ALIGN="LEFT"><A HREF="Browse/index.html">Select from a list of 441 works of classical
literature</A> by 59 different authors, including user-driven
commentary and "reader's choice" Web sites. Mainly Greco-Roman works
(some Chinese and Persian), all in English translation.</TD>
<TD WIDTH="1" ROWSPAN="6" BGCOLOR="990033"><IMG SRC="Images/1p.gif" ALIGN="TOP" WIDTH="1" HEIGHT="1" ALT=""></TD>
<TD WIDTH="30%" ALIGN="LEFT" ROWSPAN="5">
<DIV ALIGN="CENTER"><FONT COLOR="990033">Announcements</FONT></DIV>
<HR SIZE="1" COLOR="990033" NOSHADE WIDTH="30%">
<b>5/10/23</b>: With the classics.mit.edu site not responding on The Tech's w20 infrastructure, this copy has been
taken over by MIT IS&T and provided as-is. Our data source was <a href="https://github.com/TheMITTech">https://github.com/TheMITTech</a> and you can follow up with the contact information provided there if you have any questions.
<HR SIZE="1" COLOR="990033" NOSHADE WIDTH="30%">
<b>3/14/11</b>: We are launching a restoration project in the spring of 2011 to fix up the broken content on the site.
Thanks to everyone for your patience as we continue to restore the missing works.
Please see the help section if you have any questions.
</TD>
</TR>
<TR VALIGN="TOP">
<TD ALIGN="CENTER" NOWRAP><A HREF="Search/index.html"><IMG SRC="Images/search-icon.gif" WIDTH="19" HEIGHT="29" BORDER="0" ALT="Search texts"></A>
<FONT SIZE="-1"><BR><A HREF="Search/index.html">Search</A><BR> </FONT></TD>
<TD ALIGN="LEFT"><A HREF="Search/index.html">Construct powerful queries</A> to search the
texts provided locally and remotely. Search by work and author, as
well as the entire archive. <a href="/Search/">Search now!</a></TD>
</TR>
<TR VALIGN="TOP">
<TD ALIGN="CENTER" NOWRAP><A HREF="Buy/index.html"><IMG SRC="Images/buy-icon.gif" WIDTH="32" HEIGHT="28" BORDER="0" ALT="Buy books and CD-ROMs"></A>
<FONT SIZE="-1"><BR><A HREF="Buy/index.html">Buy Books</A></FONT></TD>
<TD ALIGN="LEFT"><A HREF="Buy/index.html">Purchase books</A> using searches and lists of books
available from <A HREF="http://www.amazon.com/exec/obidos/redirect-home/theinteclasar-20">Amazon.com</A>,
including books in the Loeb Classical Library.</TD>
</TR>
<TR VALIGN="TOP">
<TD ALIGN="CENTER" NOWRAP><A HREF="Help/general.html"><IMG SRC="Images/help-icon.gif" WIDTH="26" HEIGHT="26" BORDER="0" ALT="Get help"></A>
<FONT SIZE="-1"><BR><A HREF="Help/general.html">Help</A><BR> </FONT></TD>
<TD ALIGN="LEFT"><A HREF="Help/general.html">Consult the help pages</A> for answers to
common questions.
</TD>
</TR>
<TR VALIGN="TOP">
<TD ALIGN="CENTER" NOWRAP><A HREF="other.html"><IMG SRC="Images/links-icon.gif" WIDTH="34" HEIGHT="34" BORDER="0" ALT="View list of other resources"></A>
<FONT SIZE="-1"><BR><A HREF="other.html">Other Links</A></FONT></TD>
<TD ALIGN="LEFT"><A HREF="other.html">Consult a list</A>
of other classical and electronic text resources, including several
Web rings. In addition, each work has its own list of "reader's
choice" Web sites (see <A HREF="#Browse">Browse</A>
above).</TD>
</TR>
<TR VALIGN="TOP">
<TD ALIGN="CENTER" NOWRAP></TD>
<TD ALIGN="LEFT"></TD>
<TD ALIGN="CENTER" NOWRAP><A HREF="acknowledgments.html">Acknowledgments</A> and
<A HREF="awards.html">Awards</A>
<BR><BR><A HREF="infopolicy.html">Information and Privacy Policy</A>
<BR><BR><A HREF="http://www.apple.com/"><IMG SRC="Images/macmade.gif" WIDTH="88" HEIGHT="31" ALT="Go to Apple" BORDER="0"></A>
<BR><BR>
<FONT SIZE="-1"><FONT COLOR="330066"><A HREF="http://webatomics.com/"><IMG SRC="Images/wa-small.gif" ALIGN="MIDDLE" WIDTH="25" HEIGHT="24" ALT="Go to Web Atomics" BORDER="0"></A>
<BR>Web Atomics</FONT></FONT></TD>
</TR>
</TABLE>
</FORM>
<HR SIZE="1" COLOR="990033" NOSHADE>
<FONT SIZE="-1">The Internet Classics Archive by Daniel C. Stevenson, Web Atomics.
World Wide Web presentation is copyright © 1994-2009, Daniel
C. Stevenson, Web Atomics.
All rights reserved under international and pan-American copyright
conventions, including the right of reproduction in whole or in part
in any form. Direct permission requests to tt-classics@mit.edu.
Translation of "The Deeds of the Divine Augustus" by Augustus is
copyright © Thomas Bushnell, BSG.
<br>
<a href="https://accessibility.mit.edu">MIT Accessibility</a>
<BR></FONT>
<!-- Google Analystics code -->
<script src="http://www.google-analytics.com/ga.js"></script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-7382327-1");
pageTracker._trackPageview();
} catch(err) {}</script>
</BODY>
</HTML>
Apache