Resources > Crawlers list > Scrapy

logo Scrapy

ID: 23751
ProducerScrapinghub External link
Other robots of the same producerContacts Crawler, Crawlera, Frontera, Quick-Crawler, Splash
Bot URLhttp://scrapy.org/ External link
StatusActive Active

Known variants

Scrapy/1.4.0

UseragentstringScrapy/1.4.0 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2017-07-28 16:43:00
Last seen2020-01-11 05:35:39
IP addresses50
Walk from
36.227.109.23136-227-109-231.dynamic-ip.hinet.net TW
36.224.51.736-224-51-7.dynamic-ip.hinet.net TW
36.226.231.2036-226-231-20.dynamic-ip.hinet.netTW
36.228.229.7936-228-229-79.dynamic-ip.hinet.netTW
36.228.229.9436-228-229-94.dynamic-ip.hinet.netTW
36.226.224.5436-226-224-54.dynamic-ip.hinet.netTW
36.228.225.17136-228-225-171.dynamic-ip.hinet.netTW
36.228.224.13136-228-224-131.dynamic-ip.hinet.netTW
36.226.230.20836-226-230-208.dynamic-ip.hinet.netTW
36.227.110.15336-227-110-153.dynamic-ip.hinet.netTW

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.0.5

UseragentstringScrapy/1.0.5 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-02-12 15:26:00
Last seen2019-12-30 08:01:46
IP addresses12
Walk from
136.243.11.22?DE
78.46.88.202?DE
136.243.76.224?DE
58.246.174.26?CN
52.198.176.249ec2-52-198-176-249.ap-northeast-1.compute.amazonaws.comJP
52.196.217.192ec2-52-196-217-192.ap-northeast-1.compute.amazonaws.comJP
52.201.250.99ec2-52-201-250-99.compute-1.amazonaws.comUS
52.68.41.151ec2-52-68-41-151.ap-northeast-1.compute.amazonaws.comJP
164.132.160.129ns3035942.ip-164-132-160.euFR
93.186.148.9493.186.148.94-ifnl.netGB

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.5.0

UseragentstringScrapy/1.5.0 (+https://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2018-01-15 20:38:07
Last seen2019-12-19 06:26:49
IP addresses25
Walk from
100.25.46.189ec2-100-25-46-189.compute-1.amazonaws.comUS
13.56.179.125ec2-13-56-179-125.us-west-1.compute.amazonaws.comUS
104.144.97.164?US
52.193.88.113ec2-52-193-88-113.ap-northeast-1.compute.amazonaws.comJP
34.232.52.147ec2-34-232-52-147.compute-1.amazonaws.comUS
121.156.47.204? KR
35.193.175.92?US
54.236.48.97ec2-54-236-48-97.compute-1.amazonaws.comUS
140.205.205.23? CN
151.67.128.243? IT

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.7.2

UseragentstringScrapy/1.7.2 (+https://scrapy.org)
CategoryWeb scraper
First seen2019-10-17 16:38:47
Last seen2019-12-18 06:26:54
IP addresses1
Walk from
176.74.192.99hbase7.domaincrawler.com SE
 

Scrapy/1.6.0

UseragentstringScrapy/1.6.0 (+https://scrapy.org)
CategoryWeb scraper
First seen2019-02-04 21:58:11
Last seen2019-12-11 00:34:33
IP addresses60
Walk from
80.211.112.51host51-112-211-80.serverdedicati.aruba.itIT
39.100.157.53?CN
39.100.156.45?CN
93.137.135.11993-137-135-119.adsl.net.t-com.hrHR
47.92.141.136?CN
39.100.121.198?CN
47.92.96.33?CN
39.100.156.237?CN
47.92.214.153?CN
39.100.114.125?CN

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.8.0

UseragentstringScrapy/1.8.0 (+https://scrapy.org)
CategoryWeb scraper
First seen2019-11-11 05:12:58
Last seen2019-12-09 03:06:09
IP addresses3
Walk from
123.25.21.157mail.soft365.com.vn VN
14.238.15.74static.vnpt.vnVN
47.90.241.15747.90.241.157US
 

Scrapy/1.1.3

UseragentstringScrapy/1.1.3 (+http://scrapy.org)
CategoryWeb scraper
First seen2018-03-10 17:09:30
Last seen2019-11-29 01:27:42
IP addresses5
Walk from
116.202.83.62?DE
88.99.164.194?DE
138.201.34.140?DE
195.201.85.124?DE
138.201.36.40?DE
 

Scrapy/1.7.3

UseragentstringScrapy/1.7.3 (+https://scrapy.org)
CategoryWeb scraper
First seen2019-08-16 06:35:10
Last seen2019-10-17 07:53:46
IP addresses3
Walk from
78.47.48.49?DE
128.91.91.211hpcc017.wharton.upenn.edu US
128.91.88.38hpcc032.wharton.upenn.edu US
 

Scrapy/1.5.1

UseragentstringScrapy/1.5.1 (+https://scrapy.org)
CategoryWeb scraper
First seen2018-09-05 13:34:29
Last seen2019-09-08 13:00:38
IP addresses19
Walk from
121.156.47.204? KR
195.3.144.210colo-200.altnet.lv LV
189.120.120.204bd7878cc.virtua.com.br BR
195.3.144.185colo-185.altnet.lv LV
83.137.2.249?GB
78.145.209.112host-78-145-209-112.as13285.netGB
208.110.66.34?US
188.92.74.189? LV
37.223.66.137?ES
37.223.62.144?ES

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.7.1

UseragentstringScrapy/1.7.1 (+https://scrapy.org)
CategoryWeb scraper
First seen2019-08-06 08:42:08
Last seen2019-08-27 11:30:05
IP addresses2
Walk from
147.92.33.234147.92.33.234 HK
8.44.182.23?US
 

Scrapy/1.5.2

UseragentstringScrapy/1.5.2 (+https://scrapy.org)
CategoryWeb scraper
First seen2019-02-03 19:06:16
Last seen2019-07-07 06:18:17
IP addresses3
Walk from
130.203.136.75? US
185.47.220.131? CZ
188.175.125.148188.175.125.148 CZ
 

Scrapy/1.0.3

UseragentstringScrapy/1.0.3 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2015-12-14 08:00:00
Last seen2018-05-11 07:16:04
IP addresses21
Walk from
54.160.194.98ec2-54-160-194-98.compute-1.amazonaws.comUS
61.164.42.140?CN
139.59.29.77?IN
52.88.187.57ec2-52-88-187-57.us-west-2.compute.amazonaws.comUS
108.61.242.65108.61.242.65.vultr.comUS
46.166.188.231?NL
52.40.148.43ec2-52-40-148-43.us-west-2.compute.amazonaws.comUS
54.69.99.109ec2-54-69-99-109.us-west-2.compute.amazonaws.comUS
108.83.93.215108-83-93-215.lightspeed.wepbfl.sbcglobal.net US
108.61.205.28108.61.205.28.vultr.comUS

 
.. and more .. (All IP Addresses available download to users with subscription. ... Learn More)
 

Scrapy/1.3.0

UseragentstringScrapy/1.3.0 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-30 01:37:54
Last seen2018-04-27 06:46:07
IP addresses5
Walk from
13.57.36.202ec2-13-57-36-202.us-west-1.compute.amazonaws.comUS
104.218.141.98?US
23.106.83.199?US
23.81.67.216?US
136.243.66.162?DE
 

Scrapy/1.1.2

UseragentstringScrapy/1.1.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-09-10 01:45:54
Last seen2018-03-19 11:01:04
IP addresses8
Walk from
13.113.121.129ec2-13-113-121-129.ap-northeast-1.compute.amazonaws.comJP
52.197.188.46ec2-52-197-188-46.ap-northeast-1.compute.amazonaws.comJP
52.68.219.226ec2-52-68-219-226.ap-northeast-1.compute.amazonaws.comJP
13.113.12.64ec2-13-113-12-64.ap-northeast-1.compute.amazonaws.comJP
52.199.78.154ec2-52-199-78-154.ap-northeast-1.compute.amazonaws.comJP
52.199.104.67ec2-52-199-104-67.ap-northeast-1.compute.amazonaws.comJP
52.43.49.188ec2-52-43-49-188.us-west-2.compute.amazonaws.comUS
52.25.226.122ec2-52-25-226-122.us-west-2.compute.amazonaws.comUS
 

Scrapy/1.3.2

UseragentstringScrapy/1.3.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-12-20 10:34:45
Last seen2017-12-20 10:34:45
IP addresses1
Walk from
111.200.195.186111.200.195.186 CN
 

Scrapy/1.3.3

UseragentstringScrapy/1.3.3 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-02-08 11:05:24
Last seen2017-11-07 09:12:10
IP addresses8
Walk from
13.114.207.110ec2-13-114-207-110.ap-northeast-1.compute.amazonaws.comJP
52.196.202.55ec2-52-196-202-55.ap-northeast-1.compute.amazonaws.comJP
34.208.16.193ec2-34-208-16-193.us-west-2.compute.amazonaws.comUS
107.23.10.123ec2-107-23-10-123.compute-1.amazonaws.comUS
54.167.210.113ec2-54-167-210-113.compute-1.amazonaws.comUS
176.27.243.42b01bf32a.bb.sky.com GB
124.72.95.10? CN
45.33.43.32li993-32.members.linode.comUS
 

Scrapy/0.16.5

UseragentstringScrapy/0.16.5 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-08-18 20:17:35
Last seen2017-08-16 09:26:14
IP addresses2
Walk from
98.100.180.20rrcs-98-100-180-20.central.biz.rr.comUS
67.53.77.196rrcs-67-53-77-196.west.biz.rr.comUS
 

Scrapy/0.24.5

UseragentstringScrapy/0.24.5 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-12-19 09:53:17
Last seen2017-04-14 00:30:33
IP addresses5
Walk from
52.197.129.56ec2-52-197-129-56.ap-northeast-1.compute.amazonaws.comJP
52.69.69.138ec2-52-69-69-138.ap-northeast-1.compute.amazonaws.comJP
52.192.241.191ec2-52-192-241-191.ap-northeast-1.compute.amazonaws.comJP
107.23.56.124ec2-107-23-56-124.compute-1.amazonaws.comUS
52.69.82.104ec2-52-69-82-104.ap-northeast-1.compute.amazonaws.comJP
 

Scrapy/1.2.2

UseragentstringScrapy/1.2.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-12 23:59:17
Last seen2017-03-26 23:59:17
IP addresses1
Walk from
52.200.30.172ec2-52-200-30-172.compute-1.amazonaws.comUS
 

Scrapy/1.0.5.post4

UseragentstringScrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
CategoryWeb scraper
First seen2017-01-07 23:32:39
Last seen2017-01-07 23:32:39
IP addresses1
Walk from
98.126.65.56?US
 

Scrapy/1.2.1

UseragentstringScrapy/1.2.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-11-09 23:25:03
Last seen2016-12-13 16:37:29
IP addresses3
Walk from
5.230.153.214?GB
178.18.151.162? CZ
54.213.151.77ec2-54-213-151-77.us-west-2.compute.amazonaws.comUS
 

Scrapy/1.1.1

UseragentstringScrapy/1.1.1 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-09-11 07:43:33
Last seen2016-12-04 14:34:58
IP addresses3
Walk from
138.201.52.218?DE
134.96.225.227colossus02.cs.uni-saarland.de DE
192.240.14.1host01.fna.fujitsu.comUS
 

Scrapy/1.1.0

UseragentstringScrapy/1.1.0 (+http://scrapy.org)
CategoryWeb scraper
Respects robots.txtNo
First seen2016-06-12 20:00:00
Last seen2016-09-01 16:03:46
IP addresses4
Walk from
138.201.57.140?DE
148.251.133.219?DE
192.240.14.1host01.fna.fujitsu.comUS
92.249.223.12192-249-223-121.pool.digikabel.huHU
 

Scrapy/1.0.4

UseragentstringScrapy/1.0.4 (+http://scrapy.org)
CategoryWeb scraper
First seen2016-01-27 06:10:00
Last seen2016-02-19 11:08:30
IP addresses2
Walk from
103.14.245.38103.14.245.38SG
219.92.50.109gbk-50-109.tm.net.my MY
 

Scrapy/1.0.1

UseragentstringScrapy/1.0.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-02-07 05:07:09
Last seen2016-02-10 14:16:41
IP addresses3
Walk from
194.150.168.95kaputte.li DE
46.105.110.61ns3435966.ip-46-105-110.euFR
158.69.26.33ns517020.ip-158-69-26.netCA
 

Scrapy/0.24.0

UseragentstringScrapy/0.24.0 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-08-08 09:15:00
Last seen2015-12-05 09:29:21
IP addresses2
Walk from
114.212.190.201cs-cloud.nju.edu.cn CN
54.72.183.135ec2-54-72-183-135.eu-west-1.compute.amazonaws.comUS
 

Scrapy/0.24.6

UseragentstringScrapy/0.24.6 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-08-19 21:13:23
Last seen2015-08-19 21:13:23
IP addresses1
Walk from
188.40.45.131?DE
 

Scrapy/0.25.1

UseragentstringScrapy/0.25.1 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-04-08 18:52:07
Last seen2015-04-08 18:52:08
IP addresses1
Walk from
69.124.113.52ool-457c7134.dyn.optonline.netUS
 

Scrapy/0.24.4

UseragentstringScrapy/0.24.4 (+http://scrapy.org)
CategoryWeb scraper
First seen2015-12-01 07:19:29
Last seen2014-12-10 23:03:17
IP addresses2
Walk from
97.74.4.58ip-97-74-4-58.ip.secureserver.netUS
54.72.183.135ec2-54-72-183-135.eu-west-1.compute.amazonaws.comIE
 

Scrapy/0.22.0

UseragentstringScrapy/0.22.2 (+http://scrapy.org)
CategoryWeb scraper
First seen2014-02-18 10:40:06
Last seen2014-06-18 20:56:00
IP addresses2
Walk from
201.204.48.114201.204.48.114 CR
64.53.192.27d53-64-27-192.nap.wideopenwest.comUS
 
Among our clients
View more...
 SAP  
 MailChimp  
 Dailymotion SA  
 Akamai Technologies, Inc.  
 Oracle  
 PayPal Holdings, Inc.  

Recently Added UAS

Products

Resources

About Us