Resources > Crawlers list > Scrapy

logo Scrapy

ID: 23751
ProducerScrapinghub External link
Other robots of the same producerCrawlera, Contacts Crawler
Bot URLhttp://scrapy.org/ External link
StatusActive Active

Known variants

Scrapy/1.1.1

UseragentstringScrapy/1.1.1 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-09-11 07:43:33
Last seen2016-12-04 14:34:58
Walk from these IP
138.201.52.218?DE
134.96.225.227colossus02.cs.uni-saarland.de DE
192.240.14.1host01.fna.fujitsu.com US
 

Scrapy/1.1.2

UseragentstringScrapy/1.1.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-09-10 01:45:54
Last seen2016-12-02 17:40:47
Walk from these IP
52.199.104.67ec2-52-199-104-67.ap-northeast-1.compute.amazonaws.comJP
52.43.49.188ec2-52-43-49-188.us-west-2.compute.amazonaws.comUS
52.25.226.122ec2-52-25-226-122.us-west-2.compute.amazonaws.comUS
 

Scrapy/0.24.5

UseragentstringScrapy/0.24.5 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-12-19 09:53:17
Last seen2016-11-15 00:55:59
Walk from these IP
52.69.69.138ec2-52-69-69-138.ap-northeast-1.compute.amazonaws.comJP
52.192.241.191ec2-52-192-241-191.ap-northeast-1.compute.amazonaws.comJP
107.23.56.124ec2-107-23-56-124.compute-1.amazonaws.comUS
52.69.82.104ec2-52-69-82-104.ap-northeast-1.compute.amazonaws.comJP
 

Scrapy/1.0.3

UseragentstringScrapy/1.0.3 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2015-12-14 08:00:00
Last seen2016-10-14 05:24:51
Walk from these IP
108.61.242.65108.61.242.65.vultr.comUS
46.166.188.231?NL
93.115.95.204lh28409.voxility.netRO
52.40.148.43ec2-52-40-148-43.us-west-2.compute.amazonaws.comUS
54.69.99.109ec2-54-69-99-109.us-west-2.compute.amazonaws.comUS
108.83.93.215108-83-93-215.lightspeed.wepbfl.sbcglobal.net US
108.61.205.28108.61.205.28.vultr.comUS
45.32.215.95?US
45.32.215.9545.32.215.95.vultr.comUS
45.32.161.17045.32.161.170.vultr.comUS

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.0.5

UseragentstringScrapy/1.0.5 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-02-12 15:26:00
Last seen2016-09-26 06:53:57
Walk from these IP
52.198.176.249ec2-52-198-176-249.ap-northeast-1.compute.amazonaws.comJP
52.196.217.192ec2-52-196-217-192.ap-northeast-1.compute.amazonaws.comJP
52.201.250.99ec2-52-201-250-99.compute-1.amazonaws.comUS
52.68.41.151ec2-52-68-41-151.ap-northeast-1.compute.amazonaws.comJP
164.132.160.129ns3035942.ip-164-132-160.euFR
93.186.148.9493.186.148.94-ifnl.net GB
130.207.114.206acs-6.cc.gatech.edu US
 

Scrapy/1.1.0

UseragentstringScrapy/1.1.0 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-06-12 20:00:00
Last seen2016-09-01 16:03:46
Walk from these IP
138.201.57.140?DE
148.251.133.219?DE
192.240.14.1host01.fna.fujitsu.com US
92.249.223.12192-249-223-121.pool.digikabel.hu HU
 

Scrapy/0.16.5

UseragentstringScrapy/0.16.5 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-08-18 20:17:35
Last seen2016-08-18 20:17:35
Walk from these IP
67.53.77.196rrcs-67-53-77-196.west.biz.rr.com US
 

Scrapy/1.0.4

UseragentstringScrapy/1.0.4 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-01-27 06:10:00
Last seen2016-02-19 11:08:30
Walk from these IP
103.14.245.38103.14.245.38 SG
185.100.85.101185.100.85.101IS
219.92.50.109gbk-50-109.tm.net.my MY
 

Scrapy/1.0.1

UseragentstringScrapy/1.0.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-02-07 05:07:09
Last seen2016-02-10 14:16:41
Walk from these IP
194.150.168.95kaputte.li DE
65.19.167.13065.19.167.130US
46.105.110.61ns3435966.ip-46-105-110.euFR
158.69.26.33ns517020.ip-158-69-26.netCA
 

Scrapy/0.24.0

UseragentstringScrapy/0.24.0 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-08-08 09:15:00
Last seen2015-12-05 09:29:21
Walk from these IP
114.212.190.201cs-cloud.nju.edu.cn CN
54.72.183.135ec2-54-72-183-135.eu-west-1.compute.amazonaws.comUS
 

Scrapy/0.24.6

UseragentstringScrapy/0.24.6 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-08-19 21:13:23
Last seen2015-08-19 21:13:23
Walk from these IP
188.40.45.131?DE
 

Scrapy/0.25.1

UseragentstringScrapy/0.25.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-04-08 18:52:07
Last seen2015-04-08 18:52:08
Walk from these IP
69.124.113.52ool-457c7134.dyn.optonline.net US
 

Scrapy/0.24.4

UseragentstringScrapy/0.24.4 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-12-01 07:19:29
Last seen2014-12-10 23:03:17
Walk from these IP
97.74.4.58ip-97-74-4-58.ip.secureserver.netUS
 

Scrapy/0.22.0

UseragentstringScrapy/0.22.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-02-18 10:40:06
Last seen2014-06-18 20:56:00
Walk from these IP
201.204.48.114201.204.48.114 CR
64.53.192.27d53-64-27-192.nap.wideopenwest.com US
 
Among our clients
View more...
 SAP  
 MailChimp  
 Dailymotion SA  
 Akamai Technologies, Inc.  
 Oracle  
 PayPal Holdings, Inc.  
x We have placed cookies on your device to make your experience better. Find more info here.