website-new-new/robots.txt
2024-11-16 23:32:06 +02:00

49 lines
No EOL
1.1 KiB
Text

User-agent: Googlebot
User-agent: bingbot
User-agent: Slurp
User-agent: Applebot
User-agent: ia_archiver
User-agent: archive.org_bot
User-agent: DuckDuckBot
User-agent: YandexBot
User-agent: Google-Read-Aloud
User-agent: Archive-It
Allow: *
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-News
User-agent: Storebot-Google
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: MicrosoftPreview
User-agent: Applebot-Extended
User-Agent: iTMS
User-Agent: OAI-SearchBot
User-agent: ChatGPT-User
User-Agent: GPTBot
User-Agent: Baiduspider
User-Agent: facebot
User-Agent: facebookexternalhit
User-Agent: CCBot
User-agent: Google-InspectionTool
User-agent: LinkedInBot
User-agent: Twitterbot
User-agent: Pinterestbot
User-agent: AdsBot-Google
User-agent: Amazonbot
User-agent: TelegramBot
User-agent: WhatsApp
User-agent: WhatsApp Bot
User-agent: UnityPlayer
User-agent: SkypeUriPreview
User-agent: Slackbot-LinkExpanding
User-agent: msnbot
User-agent: msn
User-agent: msnbot-media
User-agent: Discordbot
User-agent: DuckAssistBot
Disallow: *