一个简单的多线程python抓取网页信息程序

一个简单的多线程python抓取网页信息程序中出现list溢出提示，具体代码和错误提示在下面有描述，望高手们解惑，非常感谢


错误日志：

代码每次运行到spider函数内的content=each.xpath()这个地方的时候就报错：

1. Traceback (most recent call last):
   
2. File "E:/data/C++/Python/����ѧԺ/XPath-and-multithreading-crawler_v1/Դ��/tiebaspider.py", line 50, in <module>
   
3. results = pool.map(spider, page)
   
4. File "E:\data\C++\Python\python2.7.9\lib\multiprocessing\pool.py", line 251, in map
   
5. return self.map_async(func, iterable, chunksize).get()
   
6. File "E:\data\C++\Python\python2.7.9\lib\multiprocessing\pool.py", line 558, in get
   
7. raise self._value
   
8. IndexError: list index out of range

复制代码

以下为源代码

1. #-*-coding:utf8-*-
   
2. 
   
3. from lxml import etree
   
4. from multiprocessing.dummy import Pool as ThreadPool
   
5. import requests
   
6. import json
   
7. import sys
   
8. reload(sys)
   
9. 
   
10. 
    
11. sys.setdefaultencoding('utf-8')
    
12. '''重新运行之前请删除content.txt，因为文件操作使用追加方式，会导致内容太多。'''
    
13. 
    
14. 
    
15. def towrite(contentdict):
    
16.     f.writelines(u'回帖时间:' + str(contentdict['topic_reply_time']) + '\n')
    
17.     f.writelines(u'回帖内容:' + unicode(contentdict['topic_reply_content']) + '\n')
    
18.     f.writelines(u'回帖人:' + contentdict['user_name'] + '\n\n')
    
19. 
    
20. 
    
21. def spider(url):
    
22.     html = requests.get(url)
    
23.     selector = etree.HTML(html.text)
    
24.     print selector
    
25.     content_field = selector.xpath('//div[@class="l_post j_l_post l_post_bright  "]')
    
26.     item = {}
    
27.     for each in content_field:
    
28.         reply_info = json.loads(each.xpath('@data-field')[0].replace('"', ''))
    
29.         author = reply_info['author']['user_name']
    
30.         reply_time = reply_info['content']['date']
    
31.         content = each.xpath('div[@class="d_post_content_main"]/div/cc/\
    
32.         div[@class="d_post_content j_d_post_content"]/text()')[0]
    
33.         print (content)
    
34.         print (reply_time)
    
35.         print (author)
    
36.         item['user_name'] = author
    
37.         item['topic_reply_content'] = content
    
38.         item['topic_reply_time'] = reply_time
    
39.         towrite(item)
    
40. 
    
41. if __name__ == '__main__':
    
42.     pool = ThreadPool(2)
    
43.     f = open('content.txt', 'a')
    
44.     page = []
    
45.     for i in range(1, 21):
    
46.         newpage = 'http://tieba.baidu.com/p/3522395718?pn=' + str(i)
    
47.         page.append(newpage)
    
48. 
    
49.     results = pool.map(spider, page)
    
50.     pool.close()
    
51.     pool.join()
    
52.     f.close()

复制代码

有大神在吗，麻烦帮忙看看啊，非常感谢