Найден корректный robots.txt
Файл управления сканированием сайта роботами
?
Robots.txt указывает поисковым роботам, какие страницы сканировать, а какие — нет. Ошибки в файле могут случайно закрыть важные разделы от индексации.
Robots.txt настроен корректно. Размер файла: 6852 байт. Загружен за: 1сек.
Проверяемая страница не запрещена в robots.txt.
Цепочка редиректов для файла robots.txt:
http://plaintextworld.com/robots.txt
301 MovedPermanently
https://plaintextworld.com/robots.txt
200 OK
Показать содержимое robots.txt
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>Plaintext World</title>
<!-- <link rel="stylesheet" href="ptworld.css" type="text/css"> -->
<meta name="robots" content="index,follow">
<meta name="description" content="A collecton of plaintext (and mainly text) websites. Welcome to the lo-fi web.">
<meta name="keywords" content="plaintext, plain text, plain-text, mainly text, text heavy, lo-fi web, web 1.0, J.D. Roth">
<meta name="MSSmartTagsPreventParsing" content="TRUE">
</head>
<body>
<table class="mobilemain"><tr><td>
<h2>Plaintext World</h2>
<p>Quality lo-fi sites for an internet that doesn't suck.<br /><a href="https://plaintextworld.com/block/">Click here</a> for mobile-unfriendly block formatting.</p>
<!-- SEARCH -->
<p><b>=== PLAINTEXT SEARCH ===</b></p>
<a href="https://lite.duckduckgo.com/lite/">Duck Duck Go Lite</a><br />
<a href="https://search.marginalia.nu">Marginalia</a><br />
<a href="http://goosh.org">Goosh</a> (command-line google)<br />
<!-- NEWS -->
<p><b>=== PLAINTEXT NEWS ===</b></p>
<a href="http://68k.news">68k.news</a><br />
<a href="https://text.npr.org">National Public Radio</a> (text-only)<br />
<a href="https://lite.pbs.org">Public Broadcasting Service</a> (lite)<br />
<a href="https://lite.cnn.com/en">Cable News Network</a> (lite)<br />
<a href="https://www.cbc.ca/lite/news?sort=latest">Canadian Broadcasting Corporation</a> (lite)<br />
<a href="https://www.csmonitor.com/layout/set/text/textedition">Christian Science Monitor</a> (text)<br />
<a href="https://upstract.com">Upstract</a> (lite news aggregator)<br />
<!-- SPORTS -->
<p><b>=== PLAINTEXT SPORTS ===</b></p>
<a href="https://plaintextsports.com">Plain-Text Sports</a><br />
→ <a href="https://plaintextsports.com/mlb/">MLB</a> | <a href="https://plaintextsports.com/mls/">MLS</a> | <a href="https://plaintextsports.com/nba/">NBA</a> | <a href="https://plaintextsports.com/nfl/">NFL</a> | <a href="https://plaintextsports.com/nhl/">NHL</a><br />
→ <a href="https://plaintextsports.com/premier-league/">EPL</a> | <a href="https://plaintextsports.com/nwsl/">NWSL</a> | <a href="https://plaintextsports.com/wnba/">WNBA</a><br />
<!-- WEATHER -->
<p><b>=== PLAINTEXT WEATHER ===</b></p>
<a href="https://wttr.in/">wttr.in</a> [ <a href="https://wttr.in/:help">help</a> | <a href="https://wttr.in/97330">97330</a> ]<br />
<!-- SOCIAL MEDIA -->
<p><b>=== PLAINTEXT SOCIAL MEDIA ===</b></p>
<a href="https://www.metafilter.com">MetaFilter</a> - <a href="https://ask.metafilter.com">Ask MetaFilter</a> [<a href="https://www.metafilter.com/user/14928">me</a>]<br />
<a href="https://teddit.net">Teddit</a> (text-based Reddit)<br />
<a href="https://pinboard.in/popular/">Pinboard</a> [<a href="https://pinboard.in/u:jdroth/">me</a>]<br />
<a href="https://subreply.com/">Subreply</a><br />
Coming soon? <a href="https://cohost.org/rc/welcome">cohost</a><br />
<!-- FUN IMAGE -->
<br /><div><img src="ticker-tape.jpg" width="288" height="175"></div><br />
<!-- PRODUCTIVITY -->
<p><b>=== PLAINTEXT PRODUCTIVITY ===</b></p>
<a href="http://todotxt.org">Todo.txt</a><br />
<a href="https://plaintext-productivity.net">Plaintext Productivity System</a><br />
<a href="https://danlucraft.com/blog/2008/04/plain-text-organizer/">Plaintext Personal Organizer</a><br />
<a href="https://zapier.com/blog/plain-text-files-for-productivity/">Plaintext Files for Productivity</a><br />
<a href="https://onetapless.com/blog/why-i-moved-my-tasks-to-plain-text">"I Moved My Tasks to Plain Text"</a><br />
Cal Newport (<a href="https://www.calnewport.com/blog/">blog</a>)<br />
→ <a href="https://www.calnewport.com/blog/2015/10/27/deep-habits-workingmemory-txt-the-most-important-productivity-tool-youve-never-heard-of/">WorkingMemory.txt</a><br />
→ <a href="https://www.calnewport.com/blog/2020/03/16/text-file-time-blocking/">Text File Time Blocking</a><br />
<!-- NERDERY -->
<p><b>=== PLAINTEXT NERDERY ===</b></p>
<a href="https://alterslash.org">Alterslash</a> (Slashdot digest)<br />
<a href="https://news.ycombinator.com">Hacker News</a><br />
<a href="https://lobste.rs">Lobsters</a><br />
<!-- READERS -->
<p><b>=== PLAINTEXT READERS ===</b></p>
<a href="https://earthly-tools.com/text-mode?url=">Earthly Tools text mode</a><br />
<a href="https://www.textise.net">Textise</a><br />
<!-- REFERENCE -->
<p><b>=== PLAINTEXT REFERENCE ===</b></p>
<a href="https://en.m.wikipedia.org/wiki/Main_Page">Wikipedia</a> (mobile version)<br />
<!-- COMMERCE -->
<p><b>=== PLAINTEXT COMMERCE ===</b></p>
<a href="https://www.craigslist.org/">Craigslist</a><br />
<!-- EMAIL -->
<p><b>=== PLAINTEXT EMAIL ===</b></p>
<a href="https://useplaintext.email">Using plaintext email</a><br />
<a href="https://www.squirrelmail.org">SquirrelMail</a><br />
<!-- LO-FI BLOGS -->
<p><b>=== LO-FI BLOGS ===</b></p>
<a href="https://andymatuschak.org">Andy Matuschak</a><br />
<a href="https://daringfireball.net">Daring Fireball</a><br />
<a href="https://www.gwern.net">Gwern</a> (Gwern Branwen)<br />
<a href="https://kaue.me">Kaue's Way of Life</a> (Kaue Silveira)<br />
<a href="https://kottke.org">Kottke</a> (Jason Kottke)<br />
<a href="https://solar.lowtechmagazine.com">Low-Tech Magazine</a><br />
<a href="https://mimismartypants.com">Mimi Smartypants</a><br />
<a href="https://rosano.hmm.garden">Rosano</a><br />
<a href="https://sive.rs/blog">Sive.rs</a> (Derek Sivers)<br />
<a href="https://waxy.org">Waxy</a> (Andy Baio)<br />
<!-- LO-FI BLOGGING -->
<p><b>=== LO-FI BLOGGING ===</b></p>
<a href="https://bearblog.dev">Bear</a> (blogging platform)<br />
→ <a href="https://bearblog.dev/discover/">example blogs</a><br />
<!-- OTHER LO-FI SITES -->
<p><b>=== OTHER LO-FI SITES ===</b></p>
<a href="https://plaintextaccounting.org">Plaintext Accounting</a><br />
<a href="https://plaintextproject.online/">The Plain Text Project</a><br />
<a href="https://www.carbonads.net">Carbon Ads</a><br />
<a href="https://www.nearlyfreespeech.net">Nearly Free Speech</a><br />
<a href="https://html.com">HTML for Beginners</a><br />
<a href="https://tetw.org">The Electric Typewriter</a> (essays!)<br />
<!-- PRAISING PLAINTEXT -->
<p><b>=== IN PRAISE OF PLAINTEXT ===</b></p>
Stolley: <a href="https://kairos.technorhetoric.net/20.2/inventio/stolley/">The lo-fi manifesto</a><br />
Sivers: <a href="https://sive.rs/plaintext">Write plaintext files</a><br />
Travers: <a href="https://evantravers.com/articles/2019/07/01/my-world-in-textfiles/">My world in textfiles</a><br />
Bear: <a href="https://plaintext.bearblog.dev">Defending plaintext blogs</a><br />
</td></tr></table>
</body>
</html>
Apache