User-agent: * Disallow: User-agent: Yandex Disallow: User-agent: Googlebot Disallow: User-agent: Mediapartners-Google Disallow: User-agent: Yahoo Slurp Disallow: User-agent: MSNBot Disallow: User-agent: StackRambler Disallow: User-agent: spider Disallow: User-agent: crawl Disallow: User-agent: Yandex/1.01.001 (compatible; Win16; I) Disallow: User-agent: Yandex/1.01.001 (compatible; Win16; P) Disallow: User-agent: Yandex/1.01.001 (compatible; Win16; H) Disallow: User-agent: Yandex/1.02.000 (compatible; Win16; F) Disallow: User-agent: Yandex/1.03.003 (compatible; Win16; D) Disallow: User-agent: Yandex/1.03.000 (compatible; Win16; M) Disallow: User-agent: YaDirectBot/1.0 (compatible; Win16; I) Disallow: User-agent: Yandex/2.01.000 (compatible; Win16; Dyatel; C) Disallow: User-agent: Yandex/2.01.000 (compatible; Win16; Dyatel; Z) Disallow: User-agent: Yandex/2.01.000 (compatible; Win16; Dyatel; D) Disallow: User-agent: Yandex/2.01.000 (compatible; Win16; Dyatel; N) Disallow: User-agent: Googlebot-Image/1.0 Disallow: User-agent: Googlebot-Mobile/2.1 Disallow: User-agent: Googlebot/2.1 - a Disallow: User-agent: Googlebot/2.1 - b Disallow: User-agent: Googlebot/2.1 - c Disallow: User-agent: Alexa (IA Archiver) Disallow: User-agent: The web archive (IA Archiver) Disallow: User-agent: Yahoo! Slurp China Disallow: User-agent: Netcraft Disallow: User-agent: Cfetch Disallow: User-agent: Ask Disallow: User-agent: WISENutbot Disallow: User-agent: ABCdatos BotLink Disallow: User-agent: Acme.Spider Disallow: User-agent: Ahoy! The Homepage Finder Disallow: User-agent: Alkaline Disallow: User-agent: Anthill Disallow: User-agent: Walhello appie Disallow: User-agent: Arachnophilia Disallow: User-agent: Arale Disallow: User-agent: Araneo Disallow: User-agent: AraybOt Disallow: User-agent: ArchitextSpider Disallow: User-agent: Aretha Disallow: User-agent: ARIADNE Disallow: User-agent: arks Disallow: User-agent: AskJeeves Disallow: User-agent: ASpider Disallow: User-agent: ATN Worldwide Disallow: User-agent: Atomz.com Search Robot Disallow: User-agent: AURESYS Disallow: User-agent: BackRub Disallow: User-agent: Bay Spider Disallow: User-agent: BBot Disallow: User-agent: Big Brother Disallow: User-agent: Bjaaland Disallow: User-agent: BlackWidow Disallow: User-agent: Die Blinde Kuh Disallow: User-agent: Bloodhound Disallow: User-agent: Borg-Bot Disallow: User-agent: BoxSeaBot Disallow: User-agent: bright.net caching robot Disallow: User-agent: BSpider Disallow: User-agent: CACTVS Chemistry Spider Disallow: User-agent: Calif Disallow: User-agent: Cassandra Disallow: User-agent: Checkbot Disallow: User-agent: ChristCrawler.com Disallow: User-agent: churl Disallow: User-agent: cIeNcIaFiCcIoN.nEt Disallow: User-agent: Collective Disallow: User-agent: Combine System Disallow: User-agent: Conceptbot Disallow: User-agent: ConfuzzledBot Disallow: User-agent: CoolBot Disallow: User-agent: XYLEME Robot Disallow: User-agent: Internet Cruiser Robot Disallow: User-agent: Cusco Disallow: User-agent: CyberSpyder Link Test Disallow: User-agent: CydralSpider Disallow: User-agent: Desert Realm Spider Disallow: User-agent: DeWeb(c) Katalog/Index Disallow: User-agent: DienstSpider Disallow: User-agent: Digger Disallow: User-agent: Digital Integrity Robot Disallow: User-agent: Direct Hit Grabber Disallow: User-agent: DNAbot Disallow: User-agent: DownLoad Express Disallow: User-agent: DragonBot Disallow: User-agent: DWCP Disallow: User-agent: e-collector Disallow: User-agent: EbiNess Disallow: User-agent: EIT Link Verifier Robot Disallow: User-agent: ELFINBOT Disallow: User-agent: Emacs-w3 Search Engine Disallow: User-agent: ananzi Disallow: User-agent: esculapio Disallow: User-agent: Esther Disallow: User-agent: Evliya Celebi Disallow: User-agent: FastCrawler Disallow: User-agent: Fluid Dynamics Search Engine robot Disallow: User-agent: Felix IDE Disallow: User-agent: FetchRover Disallow: User-agent: fido Disallow: User-agent: KIT-Fireball Disallow: User-agent: Fish search Disallow: User-agent: Fouineur Disallow: User-agent: Robot Francoroute Disallow: User-agent: Freecrawl Disallow: User-agent: FunnelWeb Disallow: User-agent: gammaSpider Disallow: User-agent: gazz Disallow: User-agent: GCreep Disallow: User-agent: GetBot Disallow: User-agent: GetURL Disallow: User-agent: Golem Disallow: User-agent: Griffon Disallow: User-agent: Gromit Disallow: User-agent: Northern Light Gulliver Disallow: User-agent: Gulper Bot Disallow: User-agent: HamBot Disallow: User-agent: Harvest Disallow: User-agent: havIndex Disallow: User-agent: HI Disallow: User-agent: Hometown Spider Pro Disallow: User-agent: ht://Dig Disallow: User-agent: HTMLgobble Disallow: User-agent: Hyper-Decontextualizer Disallow: User-agent: iajaBot Disallow: User-agent: 5IBM_Planetwide Disallow: User-agent: Popular Iconoclast Disallow: User-agent: Ingrid Disallow: User-agent: Imagelock Disallow: User-agent: IncyWincy Disallow: User-agent: Informant Disallow: User-agent: InfoSeek Robot Disallow: User-agent: Infoseek Sidewinder Disallow: User-agent: InfoSpiders Disallow: User-agent: Inspector Web Disallow: User-agent: IntelliAgent Disallow: User-agent: Robot Disallow: User-agent: Iron33 Disallow: User-agent: Israeli-search Disallow: User-agent: JavaBee Disallow: User-agent: JBot Java Web Robot Disallow: User-agent: JCrawler Disallow: User-agent: Jeeves Disallow: User-agent: Jobot Disallow: User-agent: JoeBot Disallow: User-agent: The Jubii Indexing Robot Disallow: User-agent: JumpStation Disallow: User-agent: image.kapsi.net Disallow: User-agent: Katipo Disallow: User-agent: KDD-Explorer Disallow: User-agent: Kilroy Disallow: User-agent: KO_Yappo_Robot Disallow: User-agent: LabelGrabber Disallow: User-agent: larbin Disallow: User-agent: legs Disallow: User-agent: Link Validator Disallow: User-agent: LinkScan Disallow: User-agent: LinkWalker Disallow: User-agent: Lockon Disallow: User-agent: logo.gif Crawler Disallow: User-agent: Lycos Disallow: User-agent: Mac WWWWorm Disallow: User-agent: Magpie Disallow: User-agent: marvin Disallow: User-agent: Mattie Disallow: User-agent: MediaFox Disallow: User-agent: MerzScope Disallow: User-agent: NEC-MeshExplorer Disallow: User-agent: MindCrawler Disallow: User-agent: mnoGoSearch search engine software Disallow: User-agent: moget Disallow: User-agent: MOMspider Disallow: User-agent: Monster Disallow: User-agent: Motor Disallow: User-agent: Muncher Disallow: User-agent: Muninn Disallow: User-agent: Muscat Ferret Disallow: User-agent: Mwd.Search Disallow: User-agent: Internet Shinchakubin Disallow: User-agent: NDSpider Disallow: User-agent: Nederland.zoek Disallow: User-agent: NetCarta WebMap Engine Disallow: User-agent: NetMechanic Disallow: User-agent: NetScoop Disallow: User-agent: newscan-online Disallow: User-agent: NHSE Web Forager Disallow: User-agent: Nomad Disallow: User-agent: The NorthStar Robot Disallow: User-agent: nzexplorer Disallow: User-agent: ObjectsSearch Disallow: User-agent: Occam Disallow: User-agent: HKU WWW Octopus Disallow: User-agent: OntoSpider Disallow: User-agent: Openfind data gatherer Disallow: User-agent: Orb Search Disallow: User-agent: Pack Rat Disallow: User-agent: PageBoy Disallow: User-agent: ParaSite Disallow: User-agent: Patric Disallow: User-agent: pegasus Disallow: User-agent: The Peregrinator Disallow: User-agent: PerlCrawler Disallow: User-agent: Phantom Disallow: User-agent: PhpDig Disallow: User-agent: PiltdownMan Disallow: User-agent: Pioneer Disallow: User-agent: html_analyzer Disallow: User-agent: Portal Juice Spider Disallow: User-agent: PGP Key Agent Disallow: User-agent: PlumtreeWebAccessor Disallow: User-agent: Poppi Disallow: User-agent: PortalB Spider Disallow: User-agent: psbot Disallow: User-agent: GetterroboPlus Puu Disallow: User-agent: The Python Robot Disallow: User-agent: Raven Search Disallow: User-agent: RBSE Spider Disallow: User-agent: Resume Robot Disallow: User-agent: RoadHouse Crawling System Disallow: User-agent: RixBot Disallow: User-agent: Robbie the Robot Disallow: User-agent: ComputingSite Robi Disallow: User-agent: RoboCrawl Spider Disallow: User-agent: RoboFox Disallow: User-agent: Robozilla Disallow: User-agent: Roverbot Disallow: User-agent: RuLeS Disallow: User-agent: SafetyNet Robot Disallow: User-agent: Scooter Disallow: User-agent: Sleek Disallow: User-agent: SearchProcess Disallow: User-agent: Senrigan Disallow: User-agent: SG-Scout Disallow: User-agent: ShagSeeker Disallow: User-agent: Shai'Hulud Disallow: User-agent: Sift Disallow: User-agent: Simmany Robot Ver Disallow: User-agent: Site Valet Disallow: User-agent: Open Text Index Robot Disallow: User-agent: SiteTech-Rover Disallow: User-agent: Skymob.com Disallow: User-agent: SLCrawler Disallow: User-agent: Inktomi Slurp Disallow: User-agent: Smart Spider Disallow: User-agent: Snooper Disallow: User-agent: Solbot Disallow: User-agent: Spanner Disallow: User-agent: Speedy Spider Disallow: User-agent: spider_monkey Disallow: User-agent: SpiderBot Disallow: User-agent: Spiderline Crawler Disallow: User-agent: SpiderMan Disallow: User-agent: SpiderView Disallow: User-agent: Spry Wizard Robot Disallow: User-agent: Site Searcher Disallow: User-agent: Suke Disallow: User-agent: suntek search engine Disallow: User-agent: Sven Disallow: User-agent: Sygol Disallow: User-agent: TACH Black Widow Disallow: User-agent: Tarantula Disallow: User-agent: tarspider Disallow: User-agent: Tcl W Robot Disallow: User-agent: TechBOT Disallow: User-agent: Templeton Disallow: User-agent: TeomaTechnologies Disallow: User-agent: TITAN Disallow: User-agent: TitIn Disallow: User-agent: The TkWWW Robot Disallow: User-agent: TLSpider Disallow: User-agent: UCSD Crawl Disallow: User-agent: UdmSearch Disallow: User-agent: UptimeBot Disallow: User-agent: URL Check Disallow: User-agent: URL Spider Pro Disallow: User-agent: Valkyrie Disallow: User-agent: Verticrawl Disallow: User-agent: Victoria Disallow: User-agent: vision-search Disallow: User-agent: void-bot Disallow: User-agent: Voyager Disallow: User-agent: VWbot Disallow: User-agent: The NWI Robot Disallow: User-agent: WM Disallow: User-agent: WallPaper Disallow: User-agent: the World Wide Web Wanderer Disallow: User-agent: WebBandit Web Spider Disallow: User-agent: WebCatcher Disallow: User-agent: WebCopy Disallow: User-agent: webfetcher Disallow: User-agent: The Webfoot Robot Disallow: User-agent: Webinator Disallow: User-agent: weblayers Disallow: User-agent: WebLinker Disallow: User-agent: WebMirror Disallow: User-agent: The Web Moose Disallow: User-agent: WebQuest Disallow: User-agent: Digimarc MarcSpider Disallow: User-agent: WebReaper Disallow: User-agent: webs Disallow: User-agent: Websnarf Disallow: User-agent: WebSpider Disallow: User-agent: WebVac Disallow: User-agent: webwalk Disallow: User-agent: WebWalker Disallow: User-agent: WebWatch Disallow: User-agent: Wget Disallow: User-agent: whatUseek Winona Disallow: User-agent: WhoWhere Robot Disallow: User-agent: Wired Digital Disallow: User-agent: Weblog Monitor Disallow: User-agent: wmir Disallow: User-agent: WebStolperer Disallow: User-agent: The Web Wombat Disallow: User-agent: The World Wide Web Worm Disallow: User-agent: WWWC Disallow: User-agent: WebZinger Disallow: User-agent: XGET Disallow: User-agent: Accelobot Disallow: User-agent: Acoon-Robot 4.0.1Accelobot Disallow: User-agent: Accelobotheritrix/1.12.0 Disallow: User-agent: Accoona-AI-Agent/1.1.1 Disallow: User-agent: Accoona-AI-Agent/1.1.2 Disallow: User-agent: Acoon-Robot 4.0.0RC2 Disallow: User-agent: Acoon-Robot 4.0.1 Disallow: User-agent: Acoon-Robot 4.0.2 Disallow: User-agent: Acorn/Nutch-0.9 Disallow: User-agent: adressendeutschland.de Disallow: User-agent: AdsBot-Google Disallow: User-agent: AdsBot-Google b Disallow: User-agent: aipbot/1.0 Disallow: User-agent: aipbot/2-beta Disallow: User-agent: Almaden Disallow: User-agent: Almaden bc12 Disallow: User-agent: Almaden bc14 Disallow: User-agent: Almaden bc22 Disallow: User-agent: Almaden bc5 Disallow: User-agent: Almaden bc6 Disallow: User-agent: Almaden fc13 Disallow: User-agent: Almaden hc4 Disallow: User-agent: Amfibibot/0.07 Disallow: User-agent: appie 1.1 Disallow: User-agent: archive.org_bot/heritrix/1.13.1 Disallow: User-agent: ASAHA Search Engine Turkey V.001 Disallow: User-agent: ASAP-LynxViewer/1.0 Disallow: User-agent: ASAP-LynxViewer/1.0 b Disallow: User-agent: ASAP-LynxViewer/1.1 Disallow: User-agent: ASAP-LynxViewer/1.2 Disallow: User-agent: ASAP-Web-Sniffer/1.0 Disallow: User-agent: ASAPlinkchecker/1.0 Disallow: User-agent: AsapLinkChecker/1.0 b Disallow: User-agent: Ask Jeeves/Teoma Disallow: User-agent: Ask Jeeves/Teoma - b Disallow: User-agent: Ask Jeeves/Teoma - c Disallow: User-agent: Ask Jeeves/Teoma - d Disallow: User-agent: Baiduspider Disallow: User-agent: Baiduspider b Disallow: User-agent: BecomeBot/2.3 Disallow: User-agent: BecomeBot/2.3 b Disallow: User-agent: BecomeBot/3.0 Disallow: User-agent: BecomeBot/3.0 b Disallow: User-agent: Bigsearch.ca/Nutch-0.9-dev Disallow: User-agent: Bigsearch.ca/Nutch-1.0-dev Disallow: User-agent: Blaiz-Bee/2.00.5622 Disallow: User-agent: Blaiz-Bee/2.00.5655 Disallow: User-agent: Blaiz-Bee/2.00.6082 Disallow: User-agent: Blaiz-Bee/2.00.8315 Disallow: User-agent: boitho.com-dc/0.79 Disallow: User-agent: boitho.com-dc/0.83 Disallow: User-agent: boitho.com-dc/0.85 Disallow: User-agent: boitho.com-dc/0.86 Disallow: User-agent: btbot/0.4 Disallow: User-agent: BuzzRankingBot/1.0 Disallow: User-agent: CazoodleBot a Disallow: User-agent: CazoodleBot b Disallow: User-agent: CazoodleBot c Disallow: User-agent: CazoodleBot d Disallow: User-agent: CazoodleBot-0.1 Disallow: User-agent: ccubee/10.0 Disallow: User-agent: ccubee/2008 Disallow: User-agent: ccubee/3.2 Disallow: User-agent: ccubee/3.3 Disallow: User-agent: ccubee/3.5 Disallow: User-agent: ccubee/3.7 Disallow: User-agent: ccubee/4.0 Disallow: User-agent: ccubee/9.0 Disallow: User-agent: cfetch/1.0 Disallow: User-agent: ChangeDetection Disallow: User-agent: changedetection/1.0 Disallow: User-agent: Charlotte/1.0b Disallow: User-agent: City4you/1.3 Cesky Disallow: User-agent: CJB.NET Proxy Disallow: User-agent: CloakDetect/0.9 Disallow: User-agent: ClusoBotImage Disallow: User-agent: ClusoBotImage/1.0 Disallow: User-agent: ClusoBotImage/1.0 b Disallow: User-agent: ClusoBotOnline/1.0 Disallow: User-agent: ClusoBotOnline/1.0 b Disallow: User-agent: Combine/3 Disallow: User-agent: ConveraCrawler 0.9d Disallow: User-agent: ConveraCrawler 0.9e Disallow: User-agent: ConveraMultiMediaCrawler/0.1 Disallow: User-agent: Crawllybot/0.1 Disallow: User-agent: Crawllybot/0.1 b Disallow: User-agent: Crawllybot/0.1 c Disallow: User-agent: csci_b659/0.13 Disallow: User-agent: DataFountains at Dmoz Disallow: User-agent: DataFountains at Dmoz b Disallow: User-agent: DataparkSearch/4.40 Disallow: User-agent: DAUMOA/1.0.0 Disallow: User-agent: DAUMOA/1.0.1 Disallow: User-agent: DAUMOA/1.0.1 b Disallow: User-agent: del.icio.us-thumbnails/1.0 Disallow: User-agent: DepSpid Disallow: User-agent: DepSpid/5.07 Disallow: User-agent: DepSpid/5.10 Disallow: User-agent: DepSpid/5.24 Disallow: User-agent: DepSpid/5.25 Disallow: User-agent: DepSpid/5.26 Disallow: User-agent: dev-spider2.searchpsider.com/1.3b Disallow: User-agent: DuckDuckBot/1.0 Disallow: User-agent: e-SocietyRobot Disallow: User-agent: EARTHCOM.info/1.98 Disallow: User-agent: EARTHCOM.info/1.99 Disallow: User-agent: EARTHCOM.info/2.01 Disallow: User-agent: EARTHCOM.info/2.03 Disallow: User-agent: EARTHCOM.info/2.05 Disallow: User-agent: EARTHCOM.info/2.06 Disallow: User-agent: EARTHCOM.info/2.07 Disallow: User-agent: EARTHCOM.info/2.09 Disallow: User-agent: EARTHCOM.info/2.1 Disallow: User-agent: EARTHCOM/2.2 Disallow: User-agent: EDI/1.6.5 Disallow: User-agent: EDI/1.6.6 Disallow: User-agent: EDI/1.6.6 b Disallow: User-agent: egothor/11.0d Disallow: User-agent: egothor/11.0d b Disallow: User-agent: egothor/8.0f Disallow: User-agent: egothor/8.0g Disallow: User-agent: ejupiter.com Disallow: User-agent: ejupiter.com 43 Disallow: User-agent: EnaBot/1.1 Disallow: User-agent: EnaBot/1.2 Disallow: User-agent: Enterprise_Search/1.00.143 Disallow: User-agent: envolk/1.7 Disallow: User-agent: envolk/1.7 b Disallow: User-agent: envolk/1.7 c Disallow: User-agent: envolk[ITS]spider/1.6 Disallow: User-agent: Exabot Test/3.0 Disallow: User-agent: Exabot-Images/1.0 Disallow: User-agent: Exabot-Images/3.0 Disallow: User-agent: Exabot-Test/1.0 Disallow: User-agent: Exabot/2.0 Disallow: User-agent: Exabot/3.0 Disallow: User-agent: Exabot/3.0 b Disallow: User-agent: ExactSEEK Disallow: User-agent: Factbot 1.09 Disallow: User-agent: FAST Enterprise Crawler 6 at virk.dk Disallow: User-agent: FAST Enterprise Crawler/6 Disallow: User-agent: FAST Enterprise Crawler/6.4 Disallow: User-agent: FAST MetaWeb Crawler Disallow: User-agent: favorstarbot/1.0 Disallow: User-agent: Feedster Crawler/3.0 Disallow: User-agent: Findexa Crawler Disallow: User-agent: findlinks/0.966 Disallow: User-agent: findlinks/0.971 Disallow: User-agent: findlinks/0.973 Disallow: User-agent: findlinks/0.975 Disallow: User-agent: findlinks/0.976 Disallow: User-agent: findlinks/1.0 Disallow: User-agent: findlinks/1.0.8 Disallow: User-agent: findlinks/1.0.9 Disallow: User-agent: findlinks/1.0.9-a2 Disallow: User-agent: findlinks/1.01 Disallow: User-agent: findlinks/1.1 Disallow: User-agent: findlinks/1.1-a4 Disallow: User-agent: findlinks/1.1-a5 Disallow: User-agent: findlinks/1.1-a7 Disallow: User-agent: findlinks/1.1-a8 Disallow: User-agent: findlinks/1.1-a9 Disallow: User-agent: findlinks/1.1.1 Disallow: User-agent: findlinks/1.1.1-a1 Disallow: User-agent: findlinks/1.1.1-a2 Disallow: User-agent: findlinks/1.1.1-a5 Disallow: User-agent: findlinks/1.1.2-a2 Disallow: User-agent: findlinks/1.1.2-a3 Disallow: User-agent: findlinks/1.1.2-a4 Disallow: User-agent: findlinks/1.1.2-a5 Disallow: User-agent: findlinks/1.1.3-beta1 Disallow: User-agent: findlinks/1.1.3-beta2 Disallow: User-agent: findlinks/1.1.3-beta4 Disallow: User-agent: findlinks/1.1.3-beta6 Disallow: User-agent: findlinks/1.1.3-beta7 Disallow: User-agent: findlinks/1.1.3-beta8 Disallow: User-agent: findlinks/1.1.3-beta9 Disallow: User-agent: findlinks/1.1.4-beta1 Disallow: User-agent: flatlandbot Disallow: User-agent: flatlandbot b Disallow: User-agent: flatlandbot c Disallow: User-agent: flatlandbot d Disallow: User-agent: Forschungsportal/0.8-dev Disallow: User-agent: Francis/2.0 Disallow: User-agent: FunnelBack Disallow: User-agent: FurlBot/Furl Search 2.0 Disallow: User-agent: FyberSpider/1.2 Disallow: User-agent: g2crawler Disallow: User-agent: Gaisbot/3.0 Disallow: User-agent: Gaisbot/3.0 - 06 Disallow: User-agent: Gallent Search Spider v1.4 Robot 3 Disallow: User-agent: genieBot a Disallow: User-agent: genieBot b Disallow: User-agent: Giant/1.0 Disallow: User-agent: Gigabot Disallow: User-agent: Gigabot/1.0 Disallow: User-agent: Gigabot/2.0 Disallow: User-agent: Gigabot/2.0 - b Disallow: User-agent: Gigabot/2.0 - c Disallow: User-agent: Gigabot/2.0 - d Disallow: User-agent: Gigabot/2.0 b Disallow: User-agent: Gigabot/3.0 Disallow: User-agent: Girafabot Disallow: User-agent: Girafabot b Disallow: User-agent: Girafabot c Disallow: User-agent: GOFORITBOT Disallow: User-agent: GurujiBot/1.0 Disallow: User-agent: GurujiBot/1.0 b Disallow: User-agent: HatenaScreenshot/1.0 (checker) Disallow: User-agent: heeii/Nutch-0.9-dev at heeii.com Disallow: User-agent: heritrix - webarchiv.cz_bot/1.12.1 Disallow: User-agent: heritrix at worio.com Disallow: User-agent: heritrix/1.10.0 at worio.com Disallow: User-agent: heritrix/1.10.1 at researcher.cz Disallow: User-agent: heritrix/1.10.2 at i.stanford.edu Disallow: User-agent: heritrix/1.10.2 at yacy.net Disallow: User-agent: heritrix/1.10.2 at zvents.com Disallow: User-agent: heritrix/1.12.1 at edu.org Disallow: User-agent: heritrix/1.12.1 at netarkivet.dk Disallow: User-agent: heritrix/1.12.1 at newstin.com Disallow: User-agent: heritrix/1.12.1 at newtestbabes.com Disallow: User-agent: heritrix/1.12.1 at page-store.com Disallow: User-agent: heritrix/1.12.1 at page-store.com b Disallow: User-agent: heritrix/1.12.1 at webarchiv.cz Disallow: User-agent: heritrix/1.4.0 at webarchiv.cz Disallow: User-agent: heritrix/1.6.0 at researcher.cz Disallow: User-agent: heritrix/1.6.0 at webarchiv.cz Disallow: User-agent: heritrix/1.6.0 at worio.com Disallow: User-agent: heritrix/1.7.1 at netarkivet.dk Disallow: User-agent: heritrix/1.7.3 at webarchiv.cz Disallow: User-agent: heritrix/1.8.0 at crawlerx51.com Disallow: User-agent: heritrix/1.8.0 at webarchiv.cz Disallow: User-agent: heritrix/1.9.0 at webarchiv.cz Disallow: User-agent: HiddenMarket-1.0-beta Disallow: User-agent: hl_ftien_spider_v1.1 Disallow: User-agent: holmes/3.10 - morfeo Disallow: User-agent: holmes/3.10.1 - onet.pl Disallow: User-agent: holmes/3.11 - morfeo Disallow: User-agent: holmes/3.11 - onet.pl Disallow: User-agent: holmes/3.12 - morfeo Disallow: User-agent: holmes/3.13 - morfeo Disallow: User-agent: holmes/3.7 - morfeo Disallow: User-agent: holmes/3.8 - morfeo Disallow: User-agent: holmes/3.8 - morfeo B Disallow: User-agent: holmes/3.9 - morfeo Disallow: User-agent: holmes/3.9 - onet.pl Disallow: User-agent: holmes/3.9 - onet.pl b Disallow: User-agent: HooWWWer/2.1.3 Disallow: User-agent: HooWWWer/2.2.0 Disallow: