# robots.txt for http://biogas.org # last modified (YMD): 2016-08-31, Michael Gollmick # generated by TIMETOWEB v9.00 see http://www.timetoweb.de # generic for every Domino server # we don't want to get any of these directories to be indexed # maybe some files below html need to be indexed, so you have to put them in a directory not listed here or you have to modify the list User-agent: * Disallow: /cache/ Disallow: /html/ Disallow: /DAO/ Disallow: /download/ Disallow: /inotes5/ Disallow: /lotus/ Disallow: /sametime/ Disallow: /SametimeApplet/ Disallow: /icons/ Disallow: /java/ Disallow: /plugins/ Disallow: /template/ # tell search crawlers what to crawl on this Domain! Sitemap: http://biogas.org/edcom/webfvb.nsf/id/sitemap.xml