Resources > Crawlers list > Scrapy

logo Scrapy

ID: 23751
ProducerScrapinghub External link
Other robots of the same producerCrawlera, Contacts Crawler, Quick-Crawler
Bot URLhttp://scrapy.org/ External link
StatusActive Active

Known variants

Scrapy/1.5.0

UseragentstringScrapy/1.5.0 (+https://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2018-01-15 20:38:07
Last seen2018-07-05 10:17:23
IP addresses17
Walk from
34.239.107.77ec2-34-239-107-77.compute-1.amazonaws.comUS
140.205.205.23? CN
140.205.205.25? CN
54.236.48.97ec2-54-236-48-97.compute-1.amazonaws.comUS
153.150.248.113pc1.cio-unet.ocn.ne.jp JP
185.175.95.246? UA
162.212.88.92? US
13.57.212.163ec2-13-57-212-163.us-west-1.compute.amazonaws.comUS
162.246.216.28? US
112.204.209.184? PH

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.4.0

UseragentstringScrapy/1.4.0 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2017-07-28 16:43:00
Last seen2018-06-22 09:32:16
IP addresses24
Walk from
136.243.50.146?DE
36.224.54.21636-224-54-216.dynamic-ip.hinet.net TW
79.142.96.234? RU
79.142.96.231? RU
79.142.96.233? RU
79.142.96.239? RU
79.142.96.235? RU
79.142.96.240? RU
79.142.96.236? RU
36.224.47.19336-224-47-193.dynamic-ip.hinet.net TW

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.0.3

UseragentstringScrapy/1.0.3 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2015-12-14 08:00:00
Last seen2018-05-11 07:16:04
IP addresses21
Walk from
54.160.194.98ec2-54-160-194-98.compute-1.amazonaws.comUS
61.164.42.140? CN
139.59.29.77?IN
52.88.187.57ec2-52-88-187-57.us-west-2.compute.amazonaws.comUS
108.61.242.65108.61.242.65.vultr.comUS
46.166.188.231?NL
52.40.148.43ec2-52-40-148-43.us-west-2.compute.amazonaws.comUS
54.69.99.109ec2-54-69-99-109.us-west-2.compute.amazonaws.comUS
108.83.93.215108-83-93-215.lightspeed.wepbfl.sbcglobal.net US
108.61.205.28108.61.205.28.vultr.comUS

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.3.0

UseragentstringScrapy/1.3.0 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-30 01:37:54
Last seen2018-04-27 06:46:07
IP addresses5
Walk from
13.57.36.202ec2-13-57-36-202.us-west-1.compute.amazonaws.comUS
104.218.141.98? US
23.106.83.199?US
23.81.67.216?US
136.243.66.162?DE
 

Scrapy/1.1.2

UseragentstringScrapy/1.1.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-09-10 01:45:54
Last seen2018-03-19 11:01:04
IP addresses8
Walk from
13.113.121.129ec2-13-113-121-129.ap-northeast-1.compute.amazonaws.comJP
52.197.188.46ec2-52-197-188-46.ap-northeast-1.compute.amazonaws.comJP
52.68.219.226ec2-52-68-219-226.ap-northeast-1.compute.amazonaws.comJP
13.113.12.64ec2-13-113-12-64.ap-northeast-1.compute.amazonaws.comJP
52.199.78.154ec2-52-199-78-154.ap-northeast-1.compute.amazonaws.comJP
52.199.104.67ec2-52-199-104-67.ap-northeast-1.compute.amazonaws.comJP
52.43.49.188ec2-52-43-49-188.us-west-2.compute.amazonaws.comUS
52.25.226.122ec2-52-25-226-122.us-west-2.compute.amazonaws.comUS
 

Scrapy/1.1.3

UseragentstringScrapy/1.1.3 (+http://scrapy.org)
CategoryWeb scraper
First seen2018-03-10 17:09:30
Last seen2018-03-10 17:09:30
IP addresses1
Walk from
138.201.36.40?DE
 

Scrapy/1.3.2

UseragentstringScrapy/1.3.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-12-20 10:34:45
Last seen2017-12-20 10:34:45
IP addresses1
Walk from
111.200.195.186 CN
 

Scrapy/1.3.3

UseragentstringScrapy/1.3.3 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-02-08 11:05:24
Last seen2017-11-07 09:12:10
IP addresses8
Walk from
13.114.207.110ec2-13-114-207-110.ap-northeast-1.compute.amazonaws.comJP
52.196.202.55ec2-52-196-202-55.ap-northeast-1.compute.amazonaws.comJP
34.208.16.193ec2-34-208-16-193.us-west-2.compute.amazonaws.comUS
107.23.10.123ec2-107-23-10-123.compute-1.amazonaws.comUS
54.167.210.113ec2-54-167-210-113.compute-1.amazonaws.comUS
176.27.243.42b01bf32a.bb.sky.com GB
124.72.95.10? CN
45.33.43.32li993-32.members.linode.comUS
 

Scrapy/0.16.5

UseragentstringScrapy/0.16.5 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-08-18 20:17:35
Last seen2017-08-16 09:26:14
IP addresses2
Walk from
98.100.180.20rrcs-98-100-180-20.central.biz.rr.com US
67.53.77.196rrcs-67-53-77-196.west.biz.rr.com US
 

Scrapy/1.0.5

UseragentstringScrapy/1.0.5 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-02-12 15:26:00
Last seen2017-04-20 18:57:31
IP addresses9
Walk from
58.246.174.26? CN
52.198.176.249ec2-52-198-176-249.ap-northeast-1.compute.amazonaws.comJP
52.196.217.192ec2-52-196-217-192.ap-northeast-1.compute.amazonaws.comJP
52.201.250.99ec2-52-201-250-99.compute-1.amazonaws.comUS
52.68.41.151ec2-52-68-41-151.ap-northeast-1.compute.amazonaws.comJP
164.132.160.129ns3035942.ip-164-132-160.euFR
93.186.148.9493.186.148.94-ifnl.net GB
130.207.114.206acs-6.cc.gatech.edu US
 

Scrapy/0.24.5

UseragentstringScrapy/0.24.5 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-12-19 09:53:17
Last seen2017-04-14 00:30:33
IP addresses5
Walk from
52.197.129.56ec2-52-197-129-56.ap-northeast-1.compute.amazonaws.comJP
52.69.69.138ec2-52-69-69-138.ap-northeast-1.compute.amazonaws.comJP
52.192.241.191ec2-52-192-241-191.ap-northeast-1.compute.amazonaws.comJP
107.23.56.124ec2-107-23-56-124.compute-1.amazonaws.comUS
52.69.82.104ec2-52-69-82-104.ap-northeast-1.compute.amazonaws.comJP
 

Scrapy/1.2.2

UseragentstringScrapy/1.2.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-12 23:59:17
Last seen2017-03-26 23:59:17
IP addresses1
Walk from
52.200.30.172ec2-52-200-30-172.compute-1.amazonaws.comUS
 

Scrapy/1.0.5.post4

UseragentstringScrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-07 23:32:39
Last seen2017-01-07 23:32:39
IP addresses1
Walk from
98.126.65.56?US
 

Scrapy/1.2.1

UseragentstringScrapy/1.2.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-11-09 23:25:03
Last seen2016-12-13 16:37:29
IP addresses3
Walk from
5.230.153.214? GB
178.18.151.162? CZ
54.213.151.77ec2-54-213-151-77.us-west-2.compute.amazonaws.comUS
 

Scrapy/1.1.1

UseragentstringScrapy/1.1.1 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-09-11 07:43:33
Last seen2016-12-04 14:34:58
IP addresses3
Walk from
138.201.52.218?DE
134.96.225.227colossus02.cs.uni-saarland.de DE
192.240.14.1host01.fna.fujitsu.comUS
 

Scrapy/1.1.0

UseragentstringScrapy/1.1.0 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-06-12 20:00:00
Last seen2016-09-01 16:03:46
IP addresses4
Walk from
138.201.57.140?DE
148.251.133.219?DE
192.240.14.1host01.fna.fujitsu.comUS
92.249.223.12192-249-223-121.pool.digikabel.hu HU
 

Scrapy/1.0.4

UseragentstringScrapy/1.0.4 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-01-27 06:10:00
Last seen2016-02-19 11:08:30
IP addresses3
Walk from
103.14.245.38103.14.245.38 SG
185.100.85.101185.100.85.101IS
219.92.50.109gbk-50-109.tm.net.my MY
 

Scrapy/1.0.1

UseragentstringScrapy/1.0.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-02-07 05:07:09
Last seen2016-02-10 14:16:41
IP addresses3
Walk from
194.150.168.95kaputte.li DE
46.105.110.61ns3435966.ip-46-105-110.euFR
158.69.26.33ns517020.ip-158-69-26.netCA
 

Scrapy/0.24.0

UseragentstringScrapy/0.24.0 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-08-08 09:15:00
Last seen2015-12-05 09:29:21
IP addresses2
Walk from
114.212.190.201cs-cloud.nju.edu.cn CN
54.72.183.135ec2-54-72-183-135.eu-west-1.compute.amazonaws.comUS
 

Scrapy/0.24.6

UseragentstringScrapy/0.24.6 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-08-19 21:13:23
Last seen2015-08-19 21:13:23
IP addresses1
Walk from
188.40.45.131?DE
 

Scrapy/0.25.1

UseragentstringScrapy/0.25.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-04-08 18:52:07
Last seen2015-04-08 18:52:08
IP addresses1
Walk from
69.124.113.52ool-457c7134.dyn.optonline.net US
 

Scrapy/0.24.4

UseragentstringScrapy/0.24.4 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-12-01 07:19:29
Last seen2014-12-10 23:03:17
IP addresses1
Walk from
97.74.4.58ip-97-74-4-58.ip.secureserver.netUS
 

Scrapy/0.22.0

UseragentstringScrapy/0.22.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-02-18 10:40:06
Last seen2014-06-18 20:56:00
IP addresses2
Walk from
201.204.48.114201.204.48.114 CR
64.53.192.27d53-64-27-192.nap.wideopenwest.com US
 
Among our clients
View more...
 SAP  
 MailChimp  
 Dailymotion SA  
 Akamai Technologies, Inc.  
 Oracle  
 PayPal Holdings, Inc.