服务器资源有限，自己建立了一个共享，最近发现貌似 google 的爬虫一直在爬，有什么好的解决方案吗，
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
This topic created in 3522 days ago, the information mentioned may be changed or developed.
3 replies • 2017-03-17 10:03:53 +08:00
 |
1
加个 robot.txt
|
1 |
2
Robot.txt
|
 |
GG668v26Fd55CP5WMar 17, 2017 via iPhone
3
按 ua 屏蔽
|
Select Language