Python 多进程

[Meli55a]

需求：批量检测 url 有效性

我有一个文本中有 10w 条网址，需要检测所有 url 是否能打开，并将能打开的 url 保存到文本中，不了解 python 的多进程，网上看了一些文章，有些懵逼，求指点

原代码如下：

import requests

url_result_success = []
url_result_failed = []

headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, compress',
    'Accept-Language': 'en-us;q=0.5,en;q=0.3',
    'Cache-Control': 'max-age=0',
    'Connection': 'keep-alive',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0'
}

with open(r'urls.txt', 'r') as f:
    for url in f:
        url = url.strip()
        print url
        try:
            response = requests.get(url, headers=headers, allow_redirects=True, timeout=5)
            if response.status_code != 200:
                raise requests.RequestException(u"Status code error: {}".format(response.status_code))
        except requests.RequestException as e:
            url_result_failed.append(url)
            continue
        url_result_success.append(url)

with open(r'valid_urls.txt', 'a+') as f:
    for url in url_result_success:
        url = url.strip()
        f.write(url + '\n')

[relic]

grequests 了解下

[qiudays]

golang 了解下

[ox180]

@orangeade 足够方便