User-agent: * Sitemap: https://kaitech.io/sitemap.xml ############## # custom # ############## # --- Default: semua crawler --- User-agent: * Allow: / Allow: /web/static/ Allow: /web/image/ Allow: /theme_*/ Allow: /website/static/ Disallow: /shop Disallow: /cart Disallow: /checkout Disallow: /payment Disallow: /contactus/thank-you Disallow: /web Disallow: /my Disallow: /portal Disallow: /website/info Disallow: /web/database/ Disallow: /web/login # --- Google image crawler --- User-agent: Googlebot-Image Allow: / # --- AI search & assistant crawlers --- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Google-Extended Allow: / User-agent: Bingbot Allow: / # --- Optional: blok crawler training massal --- User-agent: CCBot Disallow: / Sitemap: https://kaitech.io/sitemap.xml