# robots.txt for OBO Travel (onebookingonline.com) # Hotel booking with live prices from multiple suppliers, trip ideas and a # Budget Planner. Everyone is welcome here: search engines, AI assistants, # AI browsers and agents may crawl and read every public page, including # search results. Crawlers are asked to skip the dated hotel and search # links (each one runs a live price search with our suppliers) and the # interactive JSON under /api/; the undated hotel pages carry the same content # and the MCP server below answers live price questions. # AI agents: see https://onebookingonline.com/llms.txt (and llms-full.txt for expanded context) # and the free MCP server at https://onebookingonline.com/mcp (no API key required). # --- Default: all crawlers --- User-agent: * Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Bing, Yandex, Seznam and most other crawlers honour Crawl-delay (Google # ignores it): one request every two seconds keeps the site fast for guests. Crawl-delay: 2 # --- SEO-tool and bulk scrapers: no value for guests, a lot of load --- User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot User-agent: DotBot User-agent: PetalBot User-agent: DataForSeoBot User-agent: BLEXBot Disallow: / # --- AI agents fetching a page because a person asked right now: everything --- User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: MistralAI-User User-agent: Mistral-User User-agent: Meta-ExternalFetcher User-agent: Devin User-agent: Operator User-agent: Manus User-agent: Browserbase Allow: / # --- AI / LLM crawlers: welcome on every public page (dated links aside) --- # OpenAI (ChatGPT) User-agent: GPTBot User-agent: OAI-SearchBot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Anthropic (Claude) User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: anthropic-ai Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Perplexity User-agent: PerplexityBot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Google AI (Gemini / AI Overviews) User-agent: Google-Extended User-agent: Googlebot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Microsoft (Copilot / Bing; AdIdxBot fetches hotel feed photos and ad landing pages) User-agent: Bingbot User-agent: MicrosoftPreview User-agent: AdIdxBot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # xAI (Grok) User-agent: xAI-Grok User-agent: GrokBot User-agent: Grok Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Meta AI User-agent: Meta-ExternalAgent User-agent: FacebookBot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Apple Intelligence User-agent: Applebot User-agent: Applebot-Extended Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # DuckDuckGo AI User-agent: DuckAssistBot User-agent: DuckDuckBot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Cohere User-agent: cohere-ai User-agent: cohere-training-data-crawler Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # You.com User-agent: YouBot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Amazon (Alexa / Rufus) User-agent: Amazonbot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # ByteDance User-agent: Bytespider User-agent: TikTokSpider Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Common Crawl (feeds many AI datasets) User-agent: CCBot Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # Other AI agents and browsers User-agent: Diffbot User-agent: ImagesiftBot User-agent: omgili User-agent: webzio User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: Timpibot User-agent: PanguBot User-agent: Kangaroo User-agent: iaskspider User-agent: QualifiedBot User-agent: SemrushBot-OCOB Allow: / Disallow: /*check_in= Disallow: /api/ Allow: /api/*-feed.xml Allow: /api/*-feed.csv Allow: /api/*-catalog.csv Allow: /api/google-hotel-prices # --- Sitemaps --- Sitemap: https://onebookingonline.com/sitemap.xml # --- For AI agents and assistants --- # MCP endpoint: https://onebookingonline.com/mcp # MCP tools: search_destinations, search_hotels, get_hotel_details, get_hotel_offers, get_offer_policies, get_price_calendar, find_deals, plan_trips, get_trip_ideas, compare_hotels, search_flexible_dates, find_cheapest_weekends, get_booking_link # MCP metadata: https://onebookingonline.com/.well-known/mcp.json # Agent docs: https://onebookingonline.com/llms.txt and https://onebookingonline.com/llms-full.txt # ChatGPT/Codex: plugin "OBO Travel" in the plugin directory, or add the MCP URL as a custom server