为啥第一台电脑不能返回数据,第二台电脑 ,第一台电脑被反爬后不能返回数据

2018-06-19 14:08:21 +08:00
 bestehen
第一个:
curl -v 'https://www.qichacha.com/gongsi_getList' -H 'cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; hasShow=1; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529381569060%2C%22updated%22%3A%201529381569062%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22www.baidu.com%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529376747; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529170947,1529201010,1529202769,1529381570; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529381570' -H 'origin: https://www.qichacha.com' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' -H 'accept: */*' -H 'referer: https://www.qichacha.com/' -H 'authority: www.qichacha.com' -H 'x-requested-with: XMLHttpRequest' --data $'key=\u767e\u5ea6&type=0' --compressed
* About to connect() to www.qichacha.com port 443 (#0)
* Trying 42.81.4.218...
* Connected to www.qichacha.com (42.81.4.218) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.qichacha.com,OU=IT,O=苏州企查查网络科技有限公司,L=苏州市,ST=江苏省,C=CN
* start date: Jun 16 00:00:00 2017 GMT
* expire date: Jun 15 23:59:59 2020 GMT
* common name: *.qichacha.com
* issuer: CN=GeoTrust SSL CA - G3,O=GeoTrust Inc.,C=US
> POST /gongsi_getList HTTP/1.1
> Host: www.qichacha.com
> cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; hasShow=1; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529381569060%2C%22updated%22%3A%201529381569062%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22www.baidu.com%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529376747; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529170947,1529201010,1529202769,1529381570; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529381570
> origin: https://www.qichacha.com
> accept-encoding: gzip, deflate, br
> accept-language: en-US,en;q=0.9
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36
> content-type: application/x-www-form-urlencoded; charset=UTF-8
> accept: */*
> referer: https://www.qichacha.com/
> authority: www.qichacha.com
> x-requested-with: XMLHttpRequest
> Content-Length: 17
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 200 OK
< Server: Tengine
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Tue, 19 Jun 2018 05:53:51 GMT
< Vary: Accept-Encoding
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Content-Encoding: gzip
< Via: cache19.l2nu20-3[116,200-0,M], cache40.l2nu20-3[117,0], cache8.cn247[132,200-0,M], cache8.cn247[133,0]
< X-Cache: MISS TCP_MISS dirn:-2:-2 mlen:-1
< X-Swift-SaveTime: Tue, 19 Jun 2018 05:53:52 GMT
< X-Swift-CacheTime: 0
< Timing-Allow-Origin: *
< EagleId: 2a51048815293876318761329e


第二个

curl -v 'https://www.qichacha.com/gongsi_getList' -H 'cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; hasShow=1; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529170947,1529201010,1529202769,1529381570; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529382147; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529386744674%2C%22updated%22%3A%201529386772675%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529386773' -H 'origin: https://www.qichacha.com' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' -H 'accept: */*' -H 'referer: https://www.qichacha.com/' -H 'authority: www.qichacha.com' -H 'x-requested-with: XMLHttpRequest' --data $'key=\u767e\u5ea6&type=0' --compressed
* Trying 42.81.4.217...
* TCP_NODELAY set
* Connected to www.qichacha.com (42.81.4.217) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.qichacha.com
* Server certificate: GeoTrust SSL CA - G3
* Server certificate: GeoTrust Global CA
> POST /gongsi_getList HTTP/1.1
> Host: www.qichacha.com
> cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; hasShow=1; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529170947,1529201010,1529202769,1529381570; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529382147; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529386744674%2C%22updated%22%3A%201529386772675%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529386773
> origin: https://www.qichacha.com
> accept-encoding: gzip, deflate, br
> accept-language: en-US,en;q=0.9
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36
> content-type: application/x-www-form-urlencoded; charset=UTF-8
> accept: */*
> referer: https://www.qichacha.com/
> authority: www.qichacha.com
> x-requested-with: XMLHttpRequest
> Content-Length: 17
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 200 OK
< Server: Tengine
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Tue, 19 Jun 2018 05:51:11 GMT
< Vary: Accept-Encoding
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Content-Encoding: gzip
< Via: cache19.l2em21-1[138,200-0,M], cache2.l2em21-1[139,0], cache8.cn247[169,200-0,M], cache5.cn247[170,0]
< X-Cache: MISS TCP_MISS dirn:-2:-2 mlen:-1
< X-Swift-SaveTime: Tue, 19 Jun 2018 05:51:11 GMT
< X-Swift-CacheTime: 0
< Timing-Allow-Origin: *
< EagleId: 2a51048515293874710697396e
<
* Curl_http_done: called premature == 0
* Connection #0 to host www.qichacha.com left intact
[{"KeyNo":"3f603703d59a04cbe427e5825099a565","Name":"<em>\u767e\u5ea6<\/em>\u5728\u7ebf\u7f51\u7edc\u6280\u672f(\u5317\u4eac)\u6709\u9650\u516c\u53f8","Reason":"\u80a1\u7968\u7b80\u79f0","Value":"<em>\u767e\u5ea6<\/em>","OperName":null,"ImageUrl":null},{"KeyNo":"576c21e3468a6b178bbf291e4820e896","Name":"\u5317\u4eac<em>\u767e\u5ea6<\/em>\u7f51\u8baf\u79d1\u6280\u6709\u9650\u516c\u53f8","Reason":"\u516c\u53f8\u540d\u79f0","Value":"\u5317\u4eac<em>\u767e\u5ea6<\/em>\u7f51\u8baf\u79d1\u6280\u6709\u9650\u516c\u53f8","OperName":null,"ImageUrl":null},{"KeyNo":"040087950737026999780939d6a623e9","Name":"<em>\u767e\u5ea6<\/em>\u56fd\u9645\u79d1\u6280(\u6df1\u5733)\u6709\u9650\u516c\u53f8","Reason":"\u516c\u53f8\u540d\u79f0","Value":"<em>\u767e\u5ea6<\/em>\u56fd\u9645\u79d1\u6280(\u6df1\u5733)\u6709\u9650\u516c\u53f8","OperName":null,"ImageUrl":null},{"KeyNo":"9459ee4a7789af50354b26dfc971c28a","Name":"<em>\u767e\u5ea6<\/em>\u79fb\u4fe1\u7f51\u7edc\u6280\u672f(\u5317\u4eac)\u6709\u9650\u516c\u53f8","Reason":"\u516c\u53f8\u540d\u79f0","Value":"<em>\u767e\u5ea6<\/em>\u79fb\u4fe1\u7f51\u7edc\u6280\u672f(\u5317\u4eac)\u6709\u9650\u516c\u53f8","OperName":null,"ImageUrl":null},{"KeyNo":"587d870f88a25bc849102850fcef9c0e","Name":"<em>\u767e\u5ea6<\/em>\u65f6\u4ee3\u7f51\u7edc\u6280\u672f(\u5317\u4eac)\u6709\u9650\u516c\u53f8","Reason":"\u516c\u53f8\u540d\u79f0","Value":"<em>\u767e\u5ea6<\/em>\u65f6\u4ee3\u7f51\u7edc\u6280\u672f(\u5317\u4eac)\u6709\u9650\u516c\u53f8","OperName":null,"ImageUrl":null}]%
1720 次点击
所在节点    Python
3 条回复
woscaizi
2018-06-19 14:31:45 +08:00
IP 被限制
opengps
2018-06-20 08:01:51 +08:00
这些爬虫起家的网站,都会有反爬虫策略的
bestehen
2018-06-22 01:21:47 +08:00
@woscaizi 我用一样的 ip 不一样结果啊

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/464085

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX