User-Agent: * Disallow: # 主要搜索引擎爬虫 - 允许但限制特定目录 User-agent: Googlebot Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: bingbot Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: MSNBot Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / # 中文搜索引擎爬虫 User-agent: Baiduspider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Baiduspider-news Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Baiduspider-render Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Sogou web spider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Sogou inst spider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Sogou spider2 Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Sogou blog Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Sogou News Spider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Sogou Orion spider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: 360Spider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Bytespider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: ToutiaoSpider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: yisouspider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: YoudaoBot Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: ChinasoSpider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: Sosospider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / User-agent: HaosouSpider Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / # 特殊限制的爬虫 User-agent: EasouSpider Request-rate: 1/6 Crawl-delay: 10 Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Allow: / # 完全禁止的SEO和AI爬虫 User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: PetalBot Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: CensysInspect Disallow: / # AI训练爬虫 User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: GPTBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: YouBot Disallow: / User-agent: Neevabot Disallow: / User-agent: Grok Disallow: / User-agent: PiBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: FacebookBot Disallow: / User-agent: facebookexternalhit Disallow: / # 恶意和垃圾爬虫 User-agent: Omgilibot Disallow: / User-agent: SpamBot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: Crescent Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBorg Disallow: / User-agent: Wget Disallow: / User-agent: python-requests Disallow: / User-agent: Python-urllib Disallow: / User-agent: Java Disallow: / User-agent: Go-http-client Disallow: / User-agent: SentiBot Disallow: / User-agent: MauiBot Disallow: / User-agent: AlphaBot Disallow: / User-agent: Alexibot Disallow: / User-agent: ia_archiver Disallow: / User-agent: Wayback Disallow: / # 通用规则 - 允许其他合法爬虫访问公开内容 User-agent: * Disallow: /account/ Disallow: /search? Disallow: /markdownx/ Disallow: /p/new Disallow: /admin/ Disallow: /tmp/ Disallow: /cache/ Disallow: /private/ Disallow: /backup/ Disallow: /logs/ Disallow: /*.php$ Disallow: /cgi-bin/ Crawl-delay: 1 Allow: /