# The crawlers listed below are allowed on the Alexa site. # Pistons allows other crawlers on a case by case basis. # Send requests to marcin@80s.pl. # # pistons provides access to traffic ranking data via Amazon Web Services. # More information here: User-agent: googlebot Disallow: /search User-agent: Lycos_Spider_(T-Rex) Disallow: /search User-agent: MSNBOT/0.1 Disallow: /search User-agent: ArchitextSpider Disallow: /search User-agent: ZyBorg Disallow: /search User-agent: SurveyBot Disallow: /search # Disallow all other crawlers User-agent: * Disallow: /