Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 16901 байт. Загружен за: 1сек.
Проверяемая страница не запрещена в robots.txt.
Цепочка редиректов для файла robots.txt:
http://ml.mit.edu/robots.txt
302 Found
https://ml.mit.edu/
200 OK
Показать содержимое robots.txt
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Machine Learning at MIT</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="Description" lang="en" content="MIT Machine Learning Group Website">
<meta name="author" content="Suvrit Sra">
<meta name="robots" content="index, follow">
<!-- icons -->
<link rel="apple-touch-icon" href="stuff/img/apple-touch-icon.png">
<link rel="shortcut icon" href="favicon.ico">
<!-- Bootstrap Core CSS file -->
<link rel="stylesheet" href="stuff/css/bootstrap.min.css">
<!-- Override CSS file - add your own CSS rules -->
<link rel="stylesheet" href="stuff/css/styles.css">
<!-- Conditional comment containing JS files for IE6 - 8 -->
<!--[if lt IE 9]>
<script src="stuff/js/html5.js"></script>
<script src="stuff/js/respond.min.js"></script>
<![endif]-->
<link rel="stylesheet" href="stuff/fa-450/css/font-awesome.min.css">
</head>
<body>
<!-- Navigation -->
<nav class="navbar navbar-fixed-top navbar-inverse" role="navigation">
<div class="container-fluid">
<!-- Brand and toggle get grouped for better mobile display -->
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="index.html">Home</a>
</div>
<!-- /.navbar-header -->
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav">
<!-- <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Research<span class="caret"></span></a>
<ul class="dropdown-menu">
<li><a href="papers.html">Publications</a></li>
<li role="separator" class="divider"></li>
<li><a href="mlplusx.html">ML+X</a></li>
</ul>
</li> -->
<li><a href="papers.html">Research</a></li>
<li><a href="people.html">People</a></li>
<li><a href="events.html">Events</a></li>
<li><a href="news.html">News</a></li>
<li><a href="classes2.html">Classes</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Links<span class="caret"></span></a>
<ul class="dropdown-menu">
<li><a href="#">About</a></li>
<li><a href="http://mailman.mit.edu/mailman/listinfo/mitml">Mailing list: MITML</a></li>
<li><a href="mailto:mltea@lists.csail.mit.edu">Mailing list: ML Tea</a></li>
<li><a href="#">Conferences & Workshops</a></li>
<li role="separator" class="divider"></li>
<li><a href="http://systemsthatlearn.mit.edu/">Systems that Learn</a></li>
<li><a href="#">Contact</a></li>
</ul>
</li>
</ul>
</div>
<!-- /.navbar-collapse -->
</div>
<!-- /.container-fluid -->
</nav>
<!-- /.navbar -->
<!-- Page Content -->
<div class="container-fluid">
<div class="row">
<div class="col-sm-6">
<h1>Machine Learning Group</h1>
</div>
<div class="col-sm-1 pull-right">
<h1><a id="miticon" href="http://www.mit.edu"></a></h1>
</div>
</div>
<!-- /.row -->
<div class="row">
<div class="col-md-5 col-sm-offset-0">
<img height="256px" src="./stuff/img/mitsmall.jpg"/>
</div>
<div class="col-md-6 col-md-push-1">
<blockquote>
<p class="text-justify">Welcome to the Machine Learning Group (MLG). We are a highly active group of researchers working on all aspects of machine learning. Our interests span theoretical foundations, optimization algorithms, and a variety of applications (vision, speech, healthcare, materials science, NLP, biology, among others).
</p>
</blockquote>
</div>
</div>
<hr>
<div class="row">
<div class="col-md-12">
<div class="col-md-12">
<h4 class="text-center display-3">NEWS</h4>
<div style="height:300px;overflow:auto">
<table class="table table-striped table-condensed">
<tbody>
<tr>
<td>Dec 2018</td>
<td><i class="fa fa-graduation-cap"></i> <a href="http://suvrit.de">Stefanie Jegelka and Suvrit Sra</a> holding a <a href="https://nips.cc/Conferences/2018/Schedule?showEvent=10983">NIPS 2018 Tutorial</a> on <em><a href="http://suvrit.de/teach/nips18tut.html">Negative Dependence and Stable Polynomials in Machine Learning</a></em></td>
</tr>
<tr>
<td>Apr 09 2018</td>
<td><i class="fa fa-trophy"></i> Congrats to Prof Broderick for winning a <a href="https://research.googleblog.com/2018/03/google-faculty-research-awards-2017.html">Google Research Award!</a></td>
</tr>
<tr>
<td>Apr 07 2018</td>
<td><i class="fa fa-trophy"></i> Congrats to <a href="https://zelda.lids.mit.edu">Zelda Mariet</a> for winning a <a href="https://research.googleblog.com/2018/04/announcing-2018-google-phd-fellows-for.html">Google Research Fellowship!</a></td>
</tr>
<tr>
<td>Mar 21 2018</td>
<td><i class="fa fa-trophy"></i> Congrats to Prof Broderick for winning an <a href="https://stat.mit.edu/news/tamara-broderick-receives-2018-nsf-career-award/">NSF-CAREER award!</a></td>
</tr>
<tr>
<td>Feb 2018</td>
<td><i class="fa fa-trophy"></i> Congrats to Prof Sra for winning an <a href="https://ara.amazon-ml.com/recipients/#2018">Amazon Research Award!</a></td>
</tr>
<tr>
<td>Feb 15 2018</td>
<td><i class="fa fa-trophy"></i> Congrats to Prof Broderick and Prof Jegelka for winning the <a href="http://news.mit.edu/2018/eight-mit-researchers-win-2018-sloan-research-fellowships-0215">2018 Sloan Research Fellowship!</a></td>
</tr>
<tr>
<td>Jan 2018</td>
<td><i class="fa fa-trophy"></i> <a href="https://www.eecs.mit.edu/news-events/announcements/tamara-broderick-receives-prestigious-army-research-office-award">Prof Broderick wins the ARO YIP award </a> -- congrats!</td>
</tr>
<tr>
<td>Dec 2017</td>
<td>Prof Jegelka's (joint) work answers <a href="http://news.mit.edu/2017/machine-learning-system-synthesizes-new-materials-recipes-1221"><em>Can computers help us synthesize new materials?</em></a></td>
</tr>
<tr>
<td>Sep 2017</td>
<td><i class="fa fa-trophy"></i> Prof. <a href="http://people.csail.mit.edu/stefje/">Stefanie Jegelka</a> receives a <a href="http://www.csail.mit.edu/jegelka_receives_DARPA_YFA">DARPA 2017 Young Faculty Award (YFA)</a>. Congrats!</td>
</tr>
<tr>
<td>NIPS17</td>
<td>9 papers from <a href="./people.html">MLG</a> + 14 more from <a href="./people.html#mlmit">wider MLG</a> accepted at NIPS -- congrats! (there are ~ 20 additional ML papers @NIPS from authors with MIT affiliations.)</td>
</tr>
<tr>
<td>NIPS17</td>
<td><a href="http://people.csail.mit.edu/stefje/">S. Jegelka</a> co-organizing <em>Discrete Structures in Machine Learning</em> Workshop at NIPS 2017</td>
</tr>
<tr>
<td>NIPS17</td>
<td><a href="http://www.tamarabroderick.com">T. Broderick</a> co-organizing <em>Advances in Approximate Bayesian Inference</em> Workshop at NIPS 2017</td>
</tr>
<tr>
<td>NIPS17</td>
<td><a href="http://suvrit.de">S. Sra</a> co-organizing <a href="http://opt-ml.org">OPT2017</a>, the 10th <em>Optimization for Machine Learning</em> workshop at NIPS 2017</td>
</tr>
<tr>
<td>Sep 2017</td>
<td>Welcome new ML students!</td>
</tr>
<tr>
<td>July 10-15</td>
<td><a href="http://suvrit.de">Suvrit Sra</a> lecturing at the <a href="http://bicmr.pku.edu.cn/content/show/39-176.html">PKU Applied Math Summer School</a> (Beijing, China), on <em>Optimization for Machine Learning: Convex and Nonconvex</a></em></td>
</tr>
<tr>
<td>Aug 24</td>
<td><a href="http://people.csail.mit.edu/stefje/">Stefanie Jegelka</a> holding a bootcamp lecture at the <a href="https://simons.berkeley.edu">Simons Institute, Berkeley</a> on <em>Continuous methods for Discrete Optimization</em></td>
</tr>
<tr>
<td>Jun 27</td>
<td><a href="http://people.csail.mit.edu/stefje/">Stefanie Jegelka</a> lecturing at the <a href="http://mlss.tuebingen.mpg.de/2017/index.html">Machine Learning Summer School (MLSS)</a>, Tübingen, Germany, on <em>Submodularity in ML</em></td>
</tr>
<tr>
<td>Jun 26,27</td>
<td><a href="http://suvrit.de">Suvrit Sra</a> lecturing at the <a href="http://mlss.tuebingen.mpg.de/2017/index.html">Machine Learning Summer School (MLSS)</a>, Tübingen, Germany, on <em><a href="http://suvrit.de/teaching.html">Optimization for ML</a></em></td>
</tr>
<tr>
<td>13 May</td>
<td>9 ICML 2017 Papers by MLG. Congrats!</td>
</tr>
<tr>
<td>06 Apr</td>
<td><a href="https://www.bostonglobe.com/business/2017/04/06/what-makes-this-hottest-class-mit/wcAVJfK8U9D64hyVSTTbqI/story.html">Boston Globe features 6.036: Introduction to Machine Learning</a><br/> (featuring Tommi Jaakkola and Regina Barzilay)</td>
</tr>
<tr>
<td>20 Jun</td>
<td><i class="fa fa-users"></i> T. Broderick co-organizing <a href="http://danlarremore.com/sinm2017/">Statistical Inference for Network Models Symposium</a> at NetSci 2017</td>
</tr>
<tr>
<td>Feb '17</td>
<td><i class="fa fa-trophy"></i> Prof. <a href="http://www.tamarabroderick.com">Tamara Broderick</a> receives a <a href="https://research.googleblog.com/2017/02/google-research-awards-2016.html">Google Faculty Research</a> award. Congrats!</td>
</tr>
<tr>
<td>Jan '17</td>
<td><i class="fa fa-graduation-cap"></i> Prof. <a href="http://www.tamarabroderick.com">Tamara Broderick</a> holding a <a href="http://www.tamarabroderick.com/tutorial_2017_simons.html">Simons Institute Tutorial</a> on Nonparameteric Bayesian Methods in ML.</td>
</tr>
<tr>
<td>Jan '17</td>
<td><i class="fa fa-graduation-cap"></i> Prof. <a href="http://people.csail.mit.edu/stefje/">Stefanie Jegelka</a> holding a <a href="https://simons.berkeley.edu/talks/submodularity-theory-applications">Simons Institute Tutorial on
Submodularity in Machine Learning: Theory and Applications.</a></td>
</tr>
<tr>
<td>Fall '16</td>
<td><i class="fa fa-users"></i> Welcome new ML graduate students!</td>
</tr>
<tr>
<td>Dec '16</td>
<td><i class="fa fa-graduation-cap"></i> <a href="http://suvrit.de">Suvrit Sra</a> holding a <a href="https://nips.cc/Conferences/2016/Schedule?showEvent=6200">NIPS Tutorial</a> (with F. Bach)</td>
</tr>
<tr>
<td>Dec '16</td>
<td>MLG members organizing 4 NIPS workshops: [<a href="http://opt-ml.org">1</a>]; [<a href="https://sites.google.com/site/nipsbnp2016/">2</a>]; [<a href="https://sites.google.com/site/nonconvexnips2016/">3</a>]; [<a href="http://approximateinference.org">4</a>]</td>
</tr>
<tr><td>Aug '16</td>
<td><i class="fa fa-university"></i>
18 papers accepted to NIPS 2016. Congrats!
</td>
</tr>
<tr><td>May '16</td>
<td><i class="fa fa-university"></i>
8 papers accepted to ICML 2016. Congrats!
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="col-md-4">
<div class="col-md-12">
<ul>
</ul>
<!-- <h4 class="text-center display-3">Fun on Visit Day!</h4>
<div id="carousel-eg" class="text-center carousel slide" data-ride="carousel">
<div class="carousel-inner" role="listbox">
<div class="item active">
<img class="d-block img-fluid img-responsive" src="./pics/pic1.jpg" alt="First">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic2.jpg" alt="Second">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic3.jpg" alt="Third">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic4.jpg" alt="Fourth">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic5.jpg" alt="Fifth">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic6.jpg" alt="Sixth">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic7.jpg" alt="Seventh">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic9.jpg" alt="Ninth">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic11.jpg" alt="Elfth">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic12.jpg" alt="Twelveth">
</div>
<div class="item">
<img class="d-block img-fluid img-responsive" src="./pics/pic13.jpg" alt="Dreizehntes">
</div>
</div>
<a class="left carousel-control" href="#carousel-eg" data-slide="prev">
<span class="glyphicon glyphicon-chevron-left"></span>
</a>
<a class="right carousel-control" href="#carousel-eg" data-slide="next">
<span class="glyphicon glyphicon-chevron-right"></span>
</a>
</div> -->
<!-- /carousel -->
</div>
</div>
</div>
<hr>
<footer class="margin-tb-3">
<div class="row">
<div class="col-lg-12">
<table>
<tbody>
<tr>
<td>
<a id="miticon" href="http://www.mit.edu"></a>
</td>
<td> </td>
<td>Massachusetts Institute of Technology<br/>Cambridge MA 02139-4307</td>
</tr>
</tbody>
</table>
</div>
</div>
<p style="text-align: center"> <a href="https://accessibility.mit.edu">Accessibility</a></p>
</footer>
</div>
<!-- /.container-fluid -->
<!-- JQuery scripts -->
<script src="stuff/js/jquery-1.11.2.min.js"></script>
<!-- Bootstrap Core scripts -->
<script src="stuff/js/bootstrap.min.js"></script>
</body>
</html>