Resources > UA list: crawlers

Udger database includes detailed information about every single user agent and operating system

Select list type: Browsers, email clients ... | Operating systems | Devices| Crawlers (Robots)

Category: All - Search engines - Marketing - Screenshots - Monitors - Feeds - Link checkers - Scrapers - Tools - Speed test - Vulnerability/Virus - AI

Crawlers (Robots)	Last seen	bot URL	Producer URL
anthropic bot	2026-03-21 00:06:35		UP
OpenAI bot	2026-03-20 23:44:58	UP	UP
PerplexityBot	2026-03-20 19:46:05	UP	UP
Azure AI Search	2026-03-20 13:51:07	UP
Google-NotebookLM	2026-03-20 09:25:59	UP	UP
QuillBot	2026-03-20 09:25:44	UP	UP
amazon-kendra	2026-03-11 01:37:09	UP	UP
MistralAI crawler	2026-03-05 08:59:43	UP	UP
WRTNBot	2026-03-04 10:50:18	Down	UP
Gemini bot	2026-02-26 05:45:20	UP	UP
TerraCotta	2026-02-25 03:39:04	UP	UP
iAsk.ai	2026-02-10 17:01:50	UP	UP
Aithlas bot	2025-12-17 07:41:56		UP
Algolia bot	2025-11-30 05:11:46		UP
Ai2Bot	2025-11-12 04:38:28	UP	UP
NetShelter ContentScan	2025-10-31 13:55:46		UP
LinerBot	2025-10-13 06:11:14	UP	UP
amazon-QBusiness	2025-09-14 16:42:36	UP	UP
PanguBot	2025-08-11 04:48:45	UP	UP
ZanistaBot	2025-06-21 02:20:16	UP	UP
Maybe inactive robots (One year we have not heard about them)
Spawning bot	2024-09-29 05:26:01		UP
ChatGLM Spider	2024-09-26 04:40:46	UP	UP
LLM-jp-Crawler	2024-08-22 04:21:26	UP	UP
Google Cloud AI	2020-10-19 17:23:41	UP	UP
Historical (Old robots, Inactive cravlers)
Crawling at Home	2021-11-03 05:24:00	Down	UP