Resources > UA list: crawlers

Udger database includes detailed information about every single user agent and operating system

Select list type: Browsers, email clients ... | Operating systems | Devices| Crawlers (Robots)

Category: All - Search engines - Marketing - Screenshots - Monitors - Feeds - Link checkers - Scrapers - Tools - Speed test - Vulnerability/Virus - AI

Crawlers (Robots)	Last seen	bot URL	Producer URL
anthropic bot	2025-12-05 07:22:19		UP
OpenAI bot	2025-12-05 06:45:12	UP	UP
TerraCotta	2025-12-05 04:50:35	UP	UP
QuillBot	2025-12-05 04:48:53		UP
PerplexityBot	2025-12-05 04:32:51	UP	UP
WRTNBot	2025-12-03 12:22:38	Down	UP
Algolia bot	2025-11-30 05:11:46		UP
MistralAI crawler	2025-11-26 20:16:49	UP	UP
Google-NotebookLM	2025-11-26 06:29:38	UP	UP
iAsk.ai	2025-11-14 08:21:56	UP	UP
Ai2Bot	2025-11-12 04:38:28	UP	UP
NetShelter ContentScan	2025-10-31 13:55:46		UP
Gemini bot	2025-10-18 06:20:22	UP	UP
LinerBot	2025-10-13 06:11:14	UP	UP
amazon-QBusiness	2025-09-14 16:42:36	UP	UP
PanguBot	2025-08-11 04:48:45	UP	UP
ZanistaBot	2025-06-21 02:20:16	UP	UP
Maybe inactive robots (One year we have not heard about them)
Spawning bot	2024-09-29 05:26:01		UP
ChatGLM Spider	2024-09-26 04:40:46	UP	UP
LLM-jp-Crawler	2024-08-22 04:21:26	UP	UP
Google Cloud AI	2020-10-19 17:23:41	UP	UP
Historical (Old robots, Inactive cravlers)
Crawling at Home	2021-11-03 05:24:00	Down	UP