ubuntu 下 python 使用 selenium + PhantomJS 时出错

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

已注册用户请登录

这是一个创建于 3349 天前的主题，其中的信息可能已经有所发展或是发生改变。

因为爬取的网站用到了AJAX，所以我想要使用selenium抓取数据。但是用selenium速度太慢，于是使用PhantomJS来加快速度。但是使用时会报错，各位帮忙看看吧。
爬取的网站是这个http://www.ncbi.nlm.nih.gov/pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)
爬的内容是每篇论文的链接。

ps:selenium用的是pip install -U selenium安装的，应该是最新版吧。Phantomjs用的是1.9.7版

#coding=utf-8
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time
import re

domain = "http://www.ncbi.nlm.nih.gov/"
url_tail = "pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)"
url = domain + url_tail

browser = webdriver.PhantomJS()
browser.get(url)


def extract_data(browser):
    links = browser.find_elements_by_css_selector("div.rprt div.rslt p.title a")
    return [link.get_attribute("href") for link in links]

page_start, page_end = 1, 3

'''每次都从page_start开始'''
page_number_box = browser.find_element_by_xpath("//*[@id='pageno']").clear()
page_number_box_cleared = browser.find_element_by_xpath("//*[@id='pageno']")
page_number_box_cleared.send_keys(str(page_start) + Keys.RETURN)

'''建立文件links.txt,模式为append'''
with open('links.txt', 'a') as f:
    for page in range(page_start, page_end+1):
        f.write("page" + str(page) + '\n')
        WebDriverWait(browser, 20).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, "div.rprt p.title"))
        )
        '''写入链接尾部的数字'''
        for line in extract_data(browser):
            line_re = re.findall('\d+', line)
            f.write(str(line_re) + '\n')

        print "page %d" % page


        time.sleep(5)
        '''点击下一页'''
        browser.find_element_by_css_selector("div.pagination a.next").click()

browser.close()

报错：

Traceback (most recent call last):
  File "scrape_url.py", line 25, in <module>
    page_number_box = browser.find_element_by_xpath("//*[@id='pageno']").clear()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 232, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 664, in find_element
    {'using': by, 'value': value})['value']
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 175, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with xpath '//*[@id='pageno']'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"101","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:46413","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"xpath\", \"sessionId\": \"99f43900-cb27-11e4-99bd-f371f24cc826\", \"value\": \"//*[@id='pageno']\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/99f43900-cb27-11e4-99bd-f371f24cc826/element"}}
Screenshot: available via screen

先谢过各位了。

import

Selenium

pageno

2 条回复 • 2015-03-16 00:19:04 +08:00

gaotongfei

2015-03-15 23:46:38 +08:00

其实最奇怪的是我几天前用原样的代码运行是正常的。。。
而且我只要把`browser = webdriver.PhantomJS()` 改为`browser = webdriver.Firefox()`就可以运行，但是实在太慢了。5万多页不知道要爬到什么时候。而且中途还经常抛出异常(可能是因为网渣)。

gaotongfei

2015-03-16 00:19:04 +08:00

啊哈，在stackoverflow上找到了一个解决方法。
屏蔽掉css，图片和js，虽然PhantomJS还是不能用，但是确实速度变快了，目的达到就好。

```
firefox_profile = webdriver.FirefoxProfile()
firefox_profile.set_preference("browser.download.folderList", 2)
firefox_profile.set_preference("permissions.default.stylesheet", 2)
firefox_profile.set_preference("permissions.default.image", 2)
firefox_profile.set_preference("javascript.enable", False)

browser = webdriver.Firefox(firefox_profile=firefox_profile)
```

http://stackoverflow.com/questions/20892768/how-to-speed-up-browsing-in-selenium-firefox
http://stackoverflow.com/questions/17462884/is-selenium-slow-or-is-my-code-wrong