#deny all spider user-agent: * Allow: /css/*.css* Allow: /js/*.js* Disallow: /*.axd$ Disallow: /*.php$ User-agent: AhrefsBot Disallow: / User-agent: YisouSpider Disallow: / User-agent: EasouSpider Disallow: / User-agent: EtaoSpider Disallow: / User-agent: MJ12bot Disallow: / User-agent: Bytespider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou inst spider Disallow: /