User-agent: * Disallow: /downloads/ # This is an infinite virtual URL space Disallow: /testCenter/ # these will soon disappear Disallow: /WSRPTestService/ Disallow: /interopdemo/ Disallow: /interopdemo2/ User-agent: Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT;+MS+Search+4.0+Robot) Disallow: / User-agent: Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+5.0)+Fetch+API+Request Disallow: / User-agent: Faxobot* Disallow: / User-agent: booch_1.0.7* Disallow: / User-agent: User-agent* Disallow: / User-agent: Mozilla/5.0+(compatible;+BecomeBot/1.23;++http://www.become.com/webmasters.html) Disallow: / User-agent: ConveraCrawler* Disallow: / User-agent: ConveraCrawler/0.9+(+http://www.authoritativeweb.com/crawl) Disallow: / User-agent: FAST+Enterprise+Crawler* Disallow: / User-agent: RufusBot+(Rufus+Web+Miner;+http://64.124.122.252/feedback.html) Disallow: / User-agent: RufusBot* Disallow: / User-agent:ichiro/2.0+* Disallow: / User-agent: Mozilla/5.0+(compatible;+heritrix/1.12.0++http://www.accelobot.com) Disallow: /