scrapy 抓取西刺网站一直提示 TCP connection timed out: 110:错误

[al0ne]

错误日志：

2015-10-21 11:39:10+0800 [xici] DEBUG: Retrying <GET http://www.xicidaili.com/nn/4> (failed 2 times): TCP connection timed out: 110: Connection timed out.
2015-10-21 11:39:10+0800 [xici] DEBUG: Retrying <GET http://www.xicidaili.com/nn/5> (failed 2 times): TCP connection timed out: 110: Connection timed out.
2015-10-21 11:39:11+0800 [xici] DEBUG: Retrying <GET http://www.xicidaili.com/nn/6> (failed 2 times): TCP connection timed out: 110: Connection timed out.
2015-10-21 11:39:11+0800 [xici] DEBUG: Retrying <GET http://www.xicidaili.com/nn/7> (failed 2 times): TCP connection timed out: 110: Connection timed out.
^C2015-10-21 11:39:40+0800 [scrapy] INFO: Received SIGINT, shutting down gracefully. Send again to force
2015-10-21 11:39:40+0800 [xici] INFO: Closing spider (shutdown)
^C2015-10-21 11:39:41+0800 [scrapy] INFO: Received SIGINT twice, forcing unclean shutdown
2015-10-21 11:39:41+0800 [xici] DEBUG: Retrying <GET http://www.xicidaili.com/nn/8> (failed 2 times): An error occurred while connecting: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.

setting.py 设置:

BOT_NAME = 'xici'
SPIDER_MODULES = ['xici.spiders']
NEWSPIDER_MODULE = 'xici.spiders'
DBKWARGS={'db':'python','user':'root', 'passwd':'12344321',
'host':'localhost','use_unicode':True, 'charset':'utf8'}

ITEM_PIPELINES = {
'xici.pipelines.XiciPipeline': 300,
}
DOWNLOAD_DELAY = 0.25
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.7'

[leavic]

我抓 SIS 的时候偶尔也会出这个问题，我猜测是网络问题，你也知道 SIS 是要走代理的。
不过 scrapy 的框架好像很智能，一旦网络连接恢复了，就会重新开始抓取，不过已经抓过的不会重复抓，只是会从中断的位置往回倒退检查一遍，然后继续从中断的位置开始。