List of AI crawler Crawlers

This login method has been cancelled. Please contact us in case of problems.

Udger database includes detailed information about every single user agent and operating system

Crawlers (Robots)	Last seen	bot URL	Producer URL
Azure AI Search	2026-11-02 12:19:47	UP
anthropic bot	2026-11-02 11:55:09		UP
OpenAI bot	2026-11-02 11:31:15	UP	UP
RiverBot	2026-11-02 11:29:29	UP	UP
FacebookBot	2026-11-02 07:41:57	UP	UP
Parallel bot	2026-11-02 05:24:10	UP	UP
Reflection AI crawler	2026-11-02 05:13:11	UP	UP
PerplexityBot	2026-11-01 17:36:56	UP	UP
QuillBot	2026-11-01 17:13:57	UP	UP
Google-NotebookLM	2026-10-25 04:44:53	UP	UP
iAsk.ai	2026-10-21 02:50:09	UP	UP
LAION bot	2026-10-19 06:39:36	UP	UP
MistralAI crawler	2026-10-17 19:18:20	UP	UP
Ceramic bot	2026-09-29 10:09:22	UP	UP
amazon-kendra	2026-08-27 08:25:16	UP	UP
DeepSeekBot	2026-08-03 12:25:33	UP	UP
PaveBot	2026-07-24 08:40:08	Down	UP
active9.com bot	2026-07-05 10:39:36		UP
Gemini bot	2026-05-12 15:22:33	UP	UP
LLMSE	2026-05-05 16:34:59	UP	UP
Moonshot bot	2026-04-21 12:21:36		UP
WRTNBot	2026-03-04 10:50:18	Down	UP
Aithlas bot	2025-12-17 07:41:56		UP
Algolia bot	2025-11-30 05:11:46		UP
Ai2Bot	2025-11-12 04:38:28	UP	UP
Maybe inactive robots (One year we have not heard about them)
NetShelter ContentScan	2025-10-31 13:55:46		UP
LinerBot	2025-10-13 06:11:14	UP	UP
amazon-QBusiness	2025-09-14 16:42:36	UP	UP
PanguBot	2025-08-11 04:48:45	UP	UP
ZanistaBot	2025-06-21 02:20:16	UP	UP
Devin	2025-06-19 01:57:57	UP	UP
Spawning bot	2024-09-29 05:26:01		UP
ChatGLM Spider	2024-09-26 04:40:46	UP	UP
LLM-jp-Crawler	2024-08-22 04:21:26	UP	UP
Google Cloud AI	2020-10-19 17:23:41	UP	UP

Crawlers (Robots)

Last seen

bot URL

Producer URL