Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Кол-во редиректов для файла robots.txt 3 слишком большое! Это может вызывать затруднение при индексации поисковыми роботами!
Проверяемая страница не запрещена в robots.txt.
Цепочка редиректов для файла robots.txt:
http://visionlab.harvard.edu/robots.txt
301 MovedPermanently
https://visionlab.harvard.edu/robots.txt
302 Found
https://visionlab.harvard.edu/
200 OK
Показать содержимое robots.txt
<html>
<head>
<!-- standard meta data -->
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
<!-- title and favicon appear in the browswer tab -->
<title>Vision.Sciences.Lab</title>
<link rel="shortcut icon" href="assets/img/standards/favicon.ico" />
<!-- my style sheets -->
<link rel="stylesheet" href="assets/css/styles.standard.css?v3.0.1" />
<link rel="stylesheet" href="assets/css/styles.index.css?v3.0.1" />
<!-- jQuery -->
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script>window.jQuery || document.write('<script src="assets/js/jquery-1.10.2.min.js"><\/script>')</script>
<script src="//cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js"></script>
<!-- Link fonts from google, http://www.google.com/fonts-->
<!-- Link effects from google, https://developers.google.com/fonts/docs/getting_started -->
<link href='//fonts.googleapis.com/css?family=Courgette&effect=fire|emboss' rel='stylesheet' type='text/css'>
<link href='//fonts.googleapis.com/css?family=Marcellus+SC&effect=fire|emboss|anaglyph|outline' rel='stylesheet' type='text/css'>
<link href='//fonts.googleapis.com/css?family=Sonsie+One&effect=fire|emboss' rel='stylesheet' type='text/css'>
<link href='//fonts.googleapis.com/css?family=Play&effect=fire|emboss' rel='stylesheet' type='text/css'>
<!-- my helpers -->
<script src="assets/js/helper.js"></script>
<!-- Navbar dynamic menu using tooltipmenu -->
<link rel="stylesheet" type="text/css" href="assets/css/navbar/component.fixed.css" />
<link rel="stylesheet" type="text/css" href="assets/css/navbar/grez.tooltip.css" />
<script src="assets/js/navbar/modernizr.custom.js"></script>
<script src="assets/js/navbar/cbpTooltipMenu.js"></script>
<!-- TEMPLATES!! -->
<link rel="stylesheet" href="assets/css/templates.news.css" />
<script type="text/javascript" src="assets/js/jquery.tmpl.js"></script>
<script class="resultsTemplate" id="newsTemplate" type="text/x-jquery-tmpl">
<div class="newsItemDate">
[${date}]
</div>
<div class="newsItemContent">
{{html content}}
</div>
<div class="newsClear">
</div>
</script>
<!-- javascript to run when document finishes loading -->
<!--
Basically doing two things here:
(1) adding tooltip menu item
(2) quering database for news, and rendering news items
-->
<script>
$( document ).ready(function() {
// tooltip, one liner
var menu = new cbpTooltipMenu( document.getElementById( 'cbp-tm-menu' ) );
// on ready, search database for news, present news
var searchData = new Object;
searchData['include']='show';
console.log(searchData);
var sqlData = new Object;
sqlData['dbname']='visionlab';
sqlData['tablename']='news';
// execute ajax post, querying database using searchData
$.post("assets/php/mysql_get_data_exact.php", {SQL:sqlData, DATA:searchData},
function(data) {
// get the news
var news = jQuery.parseJSON(data);
// sort by date
for (i=0; i<news.length; i++) {
d=new Date("01-" + news[i]['date']);
news[i]['dateTime']=d.getTime();
}
news = sortByKey(news, 'dateTime', 1);
//news = sortByKey(news, 'id', 1);
console.log(news);
// each members object (e.g., members[0]) is set to "member" and then
// the function response must receive the "members" object, which it then
// returns to "get_data_searchbar"
if (news!=null & news!="null") {
// empty the results box first, then render new items
// $("#newsBox").empty();
// $('#newsTemplate').tmpl(news).appendTo('#newsBox');
}
});
});
</script>
</head>
<body>
<div id=wrapper>
<div id="header">
<div id="headerBackground">
</div>
<div id="headerTextBox" class="font-effect-outline">
<h1>Vision Sciences Laboratory</h1>
</div>
<div id="headerImageBox">
</div>
<div id="headerFilmBox">
</div>
<div id="headerLinkBox">
<a href="//www.isites.harvard.edu/icb/icb.do?keyword=k3007" target="_blank">Department of Psychology</a>
|
<a href="//www.harvard.edu/" target="_blank">Harvard University</a>
</div>
</div>
<div id="navBar">
<ul id="cbp-tm-menu" class="cbp-tm-menu">
<li>
<a id="activeMenuItem" href="index.php">HOME</a>
</li>
<li>
<a class="inactiveMenuItem" href="#">MEMBERS</a>
<ul class="cbp-tm-submenu nudge1">
<li><a href="members.current.php?memberID=PI" class="cbp-tm-icon-eye">Principal Investigators</a></li>
<li><a href="members.current.php?memberID=postdoc" class="cbp-tm-icon-eye">Postdoctoral Fellows</a></li>
<li><a href="members.current.php?memberID=graduate" class="cbp-tm-icon-eye">Graduate Students</a></li>
<li><a href="members.current.php?memberID=RA" class="cbp-tm-icon-eye">Research Assistants</a></li>
<li><a href="members.current.php?memberID=undergraduate" class="cbp-tm-icon-eye">Undergraduate Students</a></li>
<li><a href="members.current.php?memberID=affiliate" class="cbp-tm-icon-eye">Affiliated Researchers</a></li>
</ul>
</li>
<li>
<a class="inactiveMenuItem" href="#">ALUMNI</a>
<ul class="cbp-tm-submenu nudge2">
<li><a href="members.alumni.php?memberID=" class="cbp-tm-icon-users">Everyone</a></li>
<li><a href="members.alumni.php?memberID=postdoc" class="cbp-tm-icon-users">Post Doctoral Fellows</a></li>
<li><a href="members.alumni.php?memberID=graduate" class="cbp-tm-icon-users">Graduate Students</a></li>
<li><a href="members.alumni.php?memberID=undergraduate" class="cbp-tm-icon-users">Undergraduate Students</a></li>
<li><a href="members.alumni.php?memberID=RA" class="cbp-tm-icon-users">Research Assistants</a></li>
<li><a href="members.alumni.php?memberID=exchange" class="cbp-tm-icon-users">Exchange Students</a></li>
<li><a href="members.alumni.php?memberID=visiting" class="cbp-tm-icon-users">Visiting Scientists</a></li>
<li><a href="members.alumni.php?memberID=affiliate" class="cbp-tm-icon-users">Research Associates</a></li>
</ul>
</li>
<li>
<a class="inactiveMenuItem" href="#">POSITIONS</a>
<ul class="cbp-tm-submenu nudge3">
<li><a href="positions.php?positionID=PostDoc" class="cbp-tm-icon-profile">Postdoctoral Fellows</a></li>
<li><a href="positions.php?positionID=GradStudent" class="cbp-tm-icon-profile">Graduate Students</a></li>
<li><a href="positions.php?positionID=Undergrad" class="cbp-tm-icon-profile">Undergraduates</a></li>
<li><a href="positions.php?positionID=RA" class="cbp-tm-icon-profile">Research Assistants</a></li>
</ul>
</li>
<li>
<a class="inactiveMenuItem" href="#">PARTICIPATE</a>
<ul class="cbp-tm-submenu nudge4">
<li><a href="participate.php?pageID=TestMyBrain.org" class="cbp-tm-icon-screen-2">TestMyBrain.org</a></li>
<li><a href="participate.php?pageID=MTurk.com" class="cbp-tm-icon-screen-2">MechanicalTurk.com</a></li>
<li><a href="participate.php?pageID=InTheLab" class="cbp-tm-icon-screen-2">In the lab</a></li>
</ul>
</li>
<li>
<a class="inactiveMenuItem" href="#">LAB INFO</a>
<ul class="cbp-tm-submenu nudge5">
<li><a href="info.location.php" class="cbp-tm-icon-location">Location</a></li>
<!-- <li><a href="info.talks.php" class="cbp-tm-icon-mic">Talks</a></li> -->
<li><a href="info.photos.php" class="cbp-tm-icon-camera-3">Photos</a></li>
</ul>
</li>
</ul>
</div>
<div id="content">
<div id="contentBoxAbove">
Welcome to the Vision Sciences Laboratory
</div>
<!-- <div id="contentBoxRight">
</div> -->
<div id="whatWeDoText">
<p style="text-align:center; font-weight: bold;">Our goal is to understand the cognitive and computational basis of visual intelligence.</p>
<li>How do we leverage cognitive science approaches with deep neural network models together, to understand how machines are learning, where they are failing, and to inform and improve our own cognitive models of visual intelligence?</li>
<div id="humachineCognition" style="width: 40%; margin-left: auto; margin-right: auto; margin-bottom: 30px;">
<img src="./assets/img/humachine.png" alt="humachine-cognition" width="100%">
</div>
<li>How does the human brain transform patterns of light into meaningful representations of the world — e.g. of objects and agents, interacting in places?</li>
<div id="bunnyPicture" style="width: 80%; margin-left: auto; margin-right: auto;">
<img src="./assets/img/standards/pixels2meaning.png" alt="pixels 2 meaning" width="100%">
</div>
<li>How do attention and memory access these representations? And how does vision interface with and constrain higher level cognition — e.g. social and physical inferences, aesthetic judgments and curiosity.</li>
<div id="bunnyPicture" style="width: 80%; margin-left: auto; margin-right: auto;">
<img src="./assets/img/attention-memory.png" alt="pixels 2 meaning" width="100%">
</div>
<p>We approach these questions using behavioral studies, brain imaging, and neurostimulation methods, and complement these empirical techniques with computational modeling, leveraging recent advances in the field of artificial intelligence and machine learning.</p>
</div>
<!-- <div id="contentBoxBelow">
<span class="header2"> Vision.Lab.News</span>
<div id="newsBox" style="height: auto; max-height: 450px; overflow-y:scroll;">
</div>
<br>
</div> -->
</div>
<div id="footer">
<div id="footerBackground">
</div>
<div id="footerTextBox">
<div id="footer1">33 Kirkland St</div>
<div id="footer2">7th Floor, William James Hall</div>
<div id="footer3">Cambridge, MA 02138</div>
<div id="footer4">(617) 495-3884</div>
</div>
</div>
</div> <!-- end wrapper -->
</body>
</html>
nginx