{"id":27,"date":"2006-09-19T11:24:20","date_gmt":"2006-09-19T11:24:20","guid":{"rendered":"http:\/\/ramblingsofasysadmin.com\/blog\/?p=27"},"modified":"2006-09-19T11:24:20","modified_gmt":"2006-09-19T11:24:20","slug":"robots-txt-and-spidering","status":"publish","type":"post","link":"https:\/\/ramblingsofasysadmin.com\/?p=27","title":{"rendered":"robots.txt and spidering."},"content":{"rendered":"

when you have content that is not for public consumption, you should always be safe than sorry by preventing the search engines from crawling (or spidering) the page and learning your link structure. for example, in a development environment, it would hardly be useful for the page to be viewed as if it’s a public site when it’s not ready yet.
\nenter robots.txt<\/font>. this file is extremely important; search engines look for that file and determine whether the site can be entered into its search cache or if you want to keep it private.
\nthe basic robots.txt<\/font> file works like this: you stick the file in the root of your website (e.g. the public_html<\/font> or httpdocs<\/font> folder. it won’t work if it’s located anywhere else or in a subdirectory of the site.
\nthe crux of the robots.txt<\/font> is the User-Agent<\/font> and disallow<\/font> directives. if you don’t want any<\/b> search engine bots to spider your any files on your site, the basic file looks like this:
\nUser-agent: *
\nDisallow: \/<\/font>
\nhowever, if you don’t want the search engines to crawl a specific folder, e.g. www.yoursite.com\/private, you would create the file as so:
\nUser-agent: *
\nDisallow: \/private\/<\/font>
\nif you don’t want google<\/b> to spider a specific folder called \/newsletters\/, then you would use the following:
\nUser-agent: googlebot
\nDisallow: \/newsletters\/<\/font>
\nthere are hundreds of bots that you’d need to consider, but the main ones are probably google (googlebot), yahoo (yahoo-slurp), and msn (msnbot).
\nyou can also target multiple user-agents in a robots.txt file that looks like this:
\nUser-agent: *
\nDisallow: \/
\nUser-agent: googlebot
\nDisallow: \/cgi-bin\/
\nDisallow: \/private\/<\/font>
\nthere’s a great reference on user agents on wikipedia<\/a>. another great resource is this robots.txt file generator<\/a>.
\nwhere security is concerned, a robots.txt<\/font> file makes a huge difference.<\/p>\n","protected":false},"excerpt":{"rendered":"

when you have content that is not for public consumption, you should always be safe than sorry by preventing the search engines from crawling (or spidering) the page and learning your link structure. for example, in a development environment, it would hardly be useful for the page to be viewed as if it’s a public site when it’s not ready yet. enter robots.txt. this file is extremely important; search engines look for that file and …<\/span> Read More →<\/span><\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-27","post","type-post","status-publish","format-standard","hentry","category-tutorials"],"_links":{"self":[{"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=\/wp\/v2\/posts\/27","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=27"}],"version-history":[{"count":0,"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=\/wp\/v2\/posts\/27\/revisions"}],"wp:attachment":[{"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=27"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=27"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ramblingsofasysadmin.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}