#Robots.txt file - allow/disallow robots/spiders/crawlers from indexing parts of the site. # This should lower web traffic, but keep search hits high. # # Changed 16/06/2009 Andy - reconfigured for new ASP.NET site User-agent: * Disallow: /Admin/ Disallow: /aspnet_client/ Disallow: /bin/ Disallow: /cgi-bin/ Disallow: /Config/ Disallow: /css/ Disallow: /db/ Disallow: /fckeditor/ Disallow: /images/ Disallow: /script/ Disallow: /UserControls/ Disallow: /ViewControls/ Disallow: /AdminLite/ Disallow: /GalMontNet/ #Disallow Yahoo/inktomi search - way too many crawlers (Andy P. 3 Dec 06) User-agent: Slurp Crawl-delay: 5