Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Кол-во редиректов для файла robots.txt 3 слишком большое! Это может вызывать затруднение при индексации поисковыми роботами!
Проверяемая страница не запрещена в robots.txt.
Цепочка редиректов для файла robots.txt:
http://murinn.is/robots.txt
302 Found
https://vefsafn.is/?page=wayback-results&site=murinn.is
302 Found
https://vefsafn.is/is/?page=wayback-results&site=murinn.is
200 OK
Показать содержимое robots.txt
<!DOCTYPE html>
<html lang="is">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8;charset=utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<title></title>
<link rel="stylesheet" href="https://vefsafn.is/static/css/bootstrap.min.css"/>
<link rel="stylesheet" href="https://vefsafn.is/static/css/font-awesome.min.css">
<link rel="stylesheet" href="https://vefsafn.is/static/css/base.css">
<script src="https://vefsafn.is/static/js/jquery-latest.min.js"></script>
<script src="https://vefsafn.is/static/js/bootstrap.min.js"></script>
<link rel="stylesheet" href="https://vefsafn.is/static/css/vefsafn.min.css"/>
<script src="https://vefsafn.is/static/postur.js"></script>
<script src="https://vefsafn.is/static/scripts.js"></script>
<script>
$(document).ready(function(){
$('ul#main-nav li').each(function() {
$(this).removeClass("active");
});
$('ul#main-nav li.lbs-custom-page-').addClass("active");
});
</script>
<script defer data-domain="vefsafn.is" src="https://t.landsbokasafn.is/js/script.outbound-links.js"></script>
<script>
// TODO: cleanup
window.wb_prefix = "/is/";
</script>
<script src="https://vefsafn.is/static/search.js"></script>
<meta content="Vefsafn.is" name="DC.Title" />
</head>
<body>
<!--<img src="/static/vefsafn_logo_paths.svg"/>
<br/>
myheader with logo,
<br/>
i18n test: Leitarniðurstöður or Leitarniðurstöður-->
<div id="wrapper">
<div id="header">
<div class="container">
<div class="row">
<div class="col-lg-3 col-md-4">
<strong class="logo" style="background: url(https://vefsafn.is/static/vefsafn_logo_paths.svg) no-repeat;
background-color: rgba(0, 0, 0, 0);
background-position-x: 0%;
background-position-y: 0%;
background-repeat: no-repeat;
background-attachment: scroll;
background-image: url(https://vefsafn.is/static/vefsafn_logo_paths.svg);
background-size: auto;
background-origin: padding-box;
background-clip: border-box;"><a href="/is/"></a></strong>
<span class="beta"> beta </span>
</div>
<div class="col-lg-9 col-md-8">
<nav class="nav-bar navbar-expand-md ">
<div class="tools">
<div class="lang">
<a href="#" data-toggle="dropdown" class="btn btn-default btn-select">
Íslenska
</a>
<ul class="dropdown-menu">
<li class="dropdown-item">
<a href="/en/?page=wayback-results&site=murinn.is">
English
</a>
</li>
<li class="dropdown-item">
<a href="/is/?page=wayback-results&site=murinn.is">
Íslenska
</a>
</li>
</ul>
</div>
</div>
<ul id="main-nav" class="collapse navbar-collapse">
<li class="nav-item lbs-custom-page-"><a class="nav-link" href="/is/"><span>Vefsafn</span></a></li>
<li class="nav-item lbs-custom-page-faq"><a class="nav-link" href="/is/lbs-info-page/faq"><span>Algengar spurningar</span></a></li>
<li class="nav-item lbs-custom-page-about"><a class="nav-link" href="/is/lbs-info-page/about"><span>Um vefsafnið</span></a></li>
</ul></nav>
<!-- nav-bar -->
<span class="btn-menu d-block d-md-none collapse navbar-collapse" data-target="#navDrop"><span>menu</span></span>
</div>
</div>
</div>
<!-- container -->
</div>
<figure class="bg-img visual type2 banner" style="background-image: url("https://vefsafn.is/static/banner.jpg");">
<img src="https://vefsafn.is/static/banner.jpg" alt="#" style="">
</figure>
<section>
<div class="container-fluid">
<div class="row justify-content-center">
<h4 class="display-4">
</h4>
</div>
<div class="row justify-content-center">
</div>
</div>
<div class="container mt-5">
<form class="needs-validation" id="search-form" novalidate>
<div class="form-row">
<div class="col-sm-1 col-xs-12">
<label for="search-url" class="lead mt-2 col-form-label" aria-label="Search For Col">
<!--Search the $root collection by url:-->
Slóð:
</label>
</div>
<div class="col-sm-8 col-xs-12">
<input aria-label="url" aria-required="true" class="form-control form-control-lg" id="search-url"
name="search" placeholder="http://"
title="Slóð" type="search" required/>
<div class="invalid-feedback">
Vantar slóð
</div>
</div>
<div class="col-sm-2 col-xs-12 mr-1 pl-0">
<button type="submit" class="form-control form-control-lg btn btn-outline-primary" role="button" aria-label="Search">
Leita
</button>
<div class="custom-control custom-checkbox custom-control">
<input type="checkbox" class="custom-control-input" id="open-results-new-window" checked="checked">
<label class="custom-control-label" for="open-results-new-window">Open results in new window</label>
</div>
</div>
</div>
<div class="form-row">
<div class="col-sm-11 col-xs-12 ">
<button class="btn btn-outline-secondary float-right mt-0 p-1" style="padding:1px" type="button" role="button"
data-toggle="collapse" data-target="#advancedOptions"
aria-expanded="false" aria-controls="advancedOptions" aria-label="Advanced Search Options">
Ítarleit
</button>
</div>
<div class="col-1">
</div>
</div>
<div class="collapse mt-3" id="advancedOptions">
<p style="">
<label for="match-type-select" class="col-form-label" aria-label="Match Type" style="margin:0px;padding:0px;">
Tegund leitarinnar
</label>
</p>
<div class="form-group form-row">
<div class="col-11 col-md-6">
<select id="match-type-select" class="form-control form-control col-sm-6" style="min-width:100%">
<option value=""></option>
<option value="prefix">Forskeyti</option>
<option value="host">Hýsitölva (e. host)</option>
<option value="domain">Lén (e. domain)</option>
</select>
</div>
</div>
<p style="">
<span data-toggle="tooltip" data-placement="right" title="">
<b>Ár/Dagsetning (áááá/áááámmdd)</b>
</span>
</p>
<div class="form-row">
<div class="col-11 col-md-6">
<label class="sr-only" for="dt-from" aria-label="Date/Time Range From">frá</label>
<div class="input-group">
<div class="input-group-prepend">
<div class="input-group-text" style="min-width:75px;">frá</div>
</div>
<input id="dt-from" type="number" name="date-range-from" class="form-control"
pattern="^\d{4,14}$">
<div class="invalid-feedback" id="dt-from-bad">
Vinsamlegast sláðu inn <b>frá</b> ár eða dagsetningu (áááá / áááámmdd) </div>
</div>
</div>
</div>
<div class="form-row">
<div class="col-11 col-md-6">
<label class="sr-only" for="dt-to" aria-label="Date/Time Range To">til</label>
<div class="input-group">
<div class="input-group-prepend">
<div class="input-group-text" style="min-width:75px;">til</div>
</div>
<input id="dt-to" type="number" name="date-range-to" class="form-control"
pattern="^\d{4,14}$">
<div class="invalid-feedback" id="dt-to-bad">
Vinsamlegast sláðu inn <b>til</b> ár eða dagsetningu (áááá / áááámmdd) </div>
</div>
</div>
</div>
<div class="form-group mt-3 search-adv-hide">
<div class="form-row">
<div class="col-6">
<p>Filtering</p>
</div>
<div class="col-6">
<button id="clear-filters" class="btn btn-outline-warning float-right" type="button">
Clear Filters
</button>
<button id="add-filter" class="btn btn-outline-secondary float-right mr-2" type="button">
Add Filter
</button>
</div>
</div>
<div class="form-row">
<div class="col-6">
<div class="row pb-1">
<label for="filter-by" class="col-form-label col-3">By:</label>
<select id="filter-by" class="form-control col-7">
<option value="" selected></option>
<option value="mime">Mime Type</option>
<option value="status">Status</option>
<option value="url">URL</option>
</select>
</div>
<div class="row pb-1">
<label for="filter-modifier" class="col-form-label col-3">How:</label>
<select id="filter-modifier" class="form-control col-7">
<option value="=">Contains</option>
<option value="==">Matches Exactly</option>
<option value="=~">Matches Regex</option>
<option value="=!">Does Not Contains</option>
<option value="=!=">Is Not</option>
<option value="=!~">Does Not Begins With</option>
</select>
</div>
<div class="row">
<label for="filter-expression" class="col-form-label col-3">Expr:</label>
<input type="text" id="filter-expression" class="form-control col-7"
placeholder="Enter an expression to filter by"
>
</div>
</div>
<div class="col-6">
<ul id="filter-list" class="filter-list">
<li id="filtering-nothing">No Filter</li>
</ul>
</div>
</div>
</div>
</div>
</form>
<br/><br/>
<div class="row">
<section class="ml-3 mt-5">
<h4>Leiðbeiningar fyrir leit </h4>
<p>Til að nota þessa þjónustu er íslensk vefslóð slegin inn hér fyrir ofan og smellt á leitartakkann. <br/>Hafi þeirri vefsíðu sem slóðin vísar í verið safnað eru varðveittar útgáfur birtar í nýjum glugga. <br/>
Niðurstöður eru flokkaðar eftir ári, mánuði og dagsetningu.</p>
</section>
</div> <p> </p>
</div>
<!--
-->
</section>
<footer id="footer">
<div class="container">
<div class="row row-56">
<div class="col-lg-4 col-md-6">
<div class="head">
<h4>Vefsafn.is</h4>
</div>
<!-- / head -->
<ul class="footer-nav">
<li class="nav-item"><a class="nav-link" href="/is/"><span>Vefsafn</span></a></li>
<li class="nav-item"><a class="nav-link" href="/is/lbs-info-page/faq"><span>Algengar spurningar</span></a></li>
<li class="nav-item"><a class="nav-link" href="/is/lbs-info-page/about"><span>Um vefsafnið</span></a></li>
</ul>
</div>
<!-- / col -->
<div class="col-lg-4 col-md-6">
<div class="head">
<h4>Skyldir vefir</h4>
</div>
<!-- / head -->
<ul class="footer-nav">
<li class="nav-item"><a class="nav-link" href="https://landsbokasafn.is">Landsbókasafn.is</a></li>
<li class="nav-item"><a class="nav-link" href="https://rafhladan.is">Rafhlaðan.is</a></li>
<li class="nav-item"><a class="nav-link" href="https://timarit.is">Tímarit.is</a></li>
<li class="nav-item"><a class="nav-link" href="https://landsbokasafn.is/index.php/vefir-safnsins"><span class="see-more">Sjá fleiri</span></a></li>
</ul>
</div>
<!-- / col -->
<div class="col-lg-4 col-md-12">
<div class="head">
<figure class="logo"><a href="/is"><img src="https://vefsafn.is/static/logo-footer.png" alt="Vefsafn.is"></a>
</figure>
</div>
<!-- / head -->
<div class="row">
<div class="col-lg-12 col-md-6">
<address>Arngrímsgötu 3. 107 Reykjavík</address>
<p><strong><a href="tel:+3545255600">Sími / tel: + 354 525 5600</a></strong>
</p>
<p><strong><span class="tpostur" data-notandi="vefsofnun" data-len="landsbokasafn.is">vefsofnun (hja) landsbokasafn.is</span></strong></p>
</div>
<!-- / col -->
</div>
<!-- / row -->
</div>
<!-- / col -->
</div>
<!-- / row -->
</div>
<!-- / container -->
</footer>
</div> </body>
</html>
Apache