Resources > Crawlers list > Scrapy

logo Scrapy

ID: 23751
ProducerScrapinghub External link
Other robots of the same producerCrawlera, Contacts Crawler
Bot URLhttp://scrapy.org/ External link
StatusActive Active

Known variants

Scrapy/0.24.5

UseragentstringScrapy/0.24.5 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-12-19 09:53:17
Last seen2017-04-14 00:30:33
IP addresses5
Walk from
52.197.129.56ec2-52-197-129-56.ap-northeast-1.compute.amazonaws.comJP
52.69.69.138ec2-52-69-69-138.ap-northeast-1.compute.amazonaws.comJP
52.192.241.191ec2-52-192-241-191.ap-northeast-1.compute.amazonaws.comJP
107.23.56.124ec2-107-23-56-124.compute-1.amazonaws.comUS
52.69.82.104ec2-52-69-82-104.ap-northeast-1.compute.amazonaws.comJP
 

Scrapy/1.3.3

UseragentstringScrapy/1.3.3 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-02-08 11:05:24
Last seen2017-04-11 07:48:57
IP addresses3
Walk from
176.27.243.42b01bf32a.bb.sky.com GB
124.72.95.10? CN
45.33.43.32li993-32.members.linode.comUS
 

Scrapy/1.2.2

UseragentstringScrapy/1.2.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-12 23:59:17
Last seen2017-03-26 23:59:17
IP addresses1
Walk from
52.200.30.172ec2-52-200-30-172.compute-1.amazonaws.comUS
 

Scrapy/1.0.3

UseragentstringScrapy/1.0.3 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2015-12-14 08:00:00
Last seen2017-03-10 11:39:34
IP addresses19
Walk from
52.88.187.57ec2-52-88-187-57.us-west-2.compute.amazonaws.comUS
108.61.242.65108.61.242.65.vultr.comUS
46.166.188.231?NL
93.115.95.204lh28409.voxility.netRO
52.40.148.43ec2-52-40-148-43.us-west-2.compute.amazonaws.comUS
54.69.99.109ec2-54-69-99-109.us-west-2.compute.amazonaws.comUS
108.83.93.215108-83-93-215.lightspeed.wepbfl.sbcglobal.net US
108.61.205.28108.61.205.28.vultr.comUS
45.32.215.95?US
45.32.215.9545.32.215.95.vultr.comUS

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.3.0

UseragentstringScrapy/1.3.0 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-30 01:37:54
Last seen2017-01-30 01:37:54
IP addresses1
Walk from
136.243.66.162?DE
 

Scrapy/1.1.2

UseragentstringScrapy/1.1.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-09-10 01:45:54
Last seen2017-01-21 14:04:32
IP addresses4
Walk from
52.199.78.154ec2-52-199-78-154.ap-northeast-1.compute.amazonaws.comJP
52.199.104.67ec2-52-199-104-67.ap-northeast-1.compute.amazonaws.comJP
52.43.49.188ec2-52-43-49-188.us-west-2.compute.amazonaws.comUS
52.25.226.122ec2-52-25-226-122.us-west-2.compute.amazonaws.comUS
 

Scrapy/1.0.5.post4

UseragentstringScrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-07 23:32:39
Last seen2017-01-07 23:32:39
IP addresses1
Walk from
98.126.65.56?US
 

Scrapy/1.0.5

UseragentstringScrapy/1.0.5 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-02-12 15:26:00
Last seen2016-12-28 05:17:56
IP addresses9
Walk from
58.246.174.26? CN
52.198.176.249ec2-52-198-176-249.ap-northeast-1.compute.amazonaws.comJP
52.196.217.192ec2-52-196-217-192.ap-northeast-1.compute.amazonaws.comJP
52.201.250.99ec2-52-201-250-99.compute-1.amazonaws.comUS
52.68.41.151ec2-52-68-41-151.ap-northeast-1.compute.amazonaws.comJP
164.132.160.129ns3035942.ip-164-132-160.euFR
93.186.148.9493.186.148.94-ifnl.net GB
130.207.114.206acs-6.cc.gatech.edu US
 

Scrapy/1.2.1

UseragentstringScrapy/1.2.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-11-09 23:25:03
Last seen2016-12-13 16:37:29
IP addresses3
Walk from
5.230.153.214? GB
178.18.151.162? CZ
54.213.151.77ec2-54-213-151-77.us-west-2.compute.amazonaws.comUS
 

Scrapy/1.1.1

UseragentstringScrapy/1.1.1 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-09-11 07:43:33
Last seen2016-12-04 14:34:58
IP addresses3
Walk from
138.201.52.218?DE
134.96.225.227colossus02.cs.uni-saarland.de DE
192.240.14.1host01.fna.fujitsu.com US
 

Scrapy/1.1.0

UseragentstringScrapy/1.1.0 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-06-12 20:00:00
Last seen2016-09-01 16:03:46
IP addresses4
Walk from
138.201.57.140?DE
148.251.133.219?DE
192.240.14.1host01.fna.fujitsu.com US
92.249.223.12192-249-223-121.pool.digikabel.hu HU
 

Scrapy/0.16.5

UseragentstringScrapy/0.16.5 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-08-18 20:17:35
Last seen2016-08-18 20:17:35
IP addresses1
Walk from
67.53.77.196rrcs-67-53-77-196.west.biz.rr.com US
 

Scrapy/1.0.4

UseragentstringScrapy/1.0.4 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-01-27 06:10:00
Last seen2016-02-19 11:08:30
IP addresses3
Walk from
103.14.245.38103.14.245.38 SG
185.100.85.101185.100.85.101IS
219.92.50.109gbk-50-109.tm.net.my MY
 

Scrapy/1.0.1

UseragentstringScrapy/1.0.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-02-07 05:07:09
Last seen2016-02-10 14:16:41
IP addresses3
Walk from
194.150.168.95kaputte.li DE
46.105.110.61ns3435966.ip-46-105-110.euFR
158.69.26.33ns517020.ip-158-69-26.netCA
 

Scrapy/0.24.0

UseragentstringScrapy/0.24.0 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-08-08 09:15:00
Last seen2015-12-05 09:29:21
IP addresses2
Walk from
114.212.190.201cs-cloud.nju.edu.cn CN
54.72.183.135ec2-54-72-183-135.eu-west-1.compute.amazonaws.comUS
 

Scrapy/0.24.6

UseragentstringScrapy/0.24.6 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-08-19 21:13:23
Last seen2015-08-19 21:13:23
IP addresses1
Walk from
188.40.45.131?DE
 

Scrapy/0.25.1

UseragentstringScrapy/0.25.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-04-08 18:52:07
Last seen2015-04-08 18:52:08
IP addresses1
Walk from
69.124.113.52ool-457c7134.dyn.optonline.net US
 

Scrapy/0.24.4

UseragentstringScrapy/0.24.4 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-12-01 07:19:29
Last seen2014-12-10 23:03:17
IP addresses1
Walk from
97.74.4.58ip-97-74-4-58.ip.secureserver.netUS
 

Scrapy/0.22.0

UseragentstringScrapy/0.22.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-02-18 10:40:06
Last seen2014-06-18 20:56:00
IP addresses2
Walk from
201.204.48.114201.204.48.114 CR
64.53.192.27d53-64-27-192.nap.wideopenwest.com US
 
Among our clients
View more...
 SAP  
 MailChimp  
 Dailymotion SA  
 Akamai Technologies, Inc.  
 Oracle  
 PayPal Holdings, Inc.  
x We have placed cookies on your device to make your experience better. Find more info here.