服务器资源有限，自己建立了一个共享，最近发现貌似 google 的爬虫一直在爬，有什么好的解决方案吗，
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
This topic created in 3380 days ago, the information mentioned may be changed or developed.
3 replies • 2017-03-17 10:03:53 +08:00
 |
1
windfarer Mar 17, 2017 via Android  1
加个 robot.txt
|
 |
2
15015613 Mar 17, 2017 via Android 1
Robot.txt
|
 |
3
GG668v26Fd55CP5W Mar 17, 2017 via iPhone
按 ua 屏蔽
|
Select Language