找回密码
 立即注册

QQ登录

只需一步,快速开始

扫一扫,访问微社区

查看: 1934|回复: 6

scrapy爬虫pipeline无法启用,求大神帮忙

1

主题

4

帖子

4

积分

贫民

积分
4
duron101 发表于 2016-10-12 22:26:41 | 显示全部楼层 |阅读模式
我用pyQT+scrapy写了个爬虫,想爬mm.taobao.com的照片试试看,结果,能解析网址,无法进入pipeline。如下图标红。

=========================pipelines.py==============================
# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import os
import urllib.request
import scrapy
from UItest  import  settings
from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem


class TaoBMMPipeline(ImagesPipeline):

    def get_media_requests(self, item, info):
        print("RUN!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")        #此处根本无法执行,求告知
        
for image_url in item['image_urls']:
            
yield scrapy.Request(image_url)
    def file_path(self, request, response=None, info=None):
        image_guid = request.url.split(
'/')[-1]
        
return 'full/%s' % (image_guid)

   
def item_completed(self, results, item, info):
        image_paths = [x[
'path'] for ok, x in results if ok]
        
if not image_paths:
            
raise DropItem("Item contains no images")
        
#item['image_paths'] = image_paths
        print(image_paths)
        
return item

=========================================items.py================================================================import scrapy


class TaoBMMItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    image_urls = scrapy.Field()
    images  = scrapy.Field()
====================================================FirstSpider.py=========================================from scrapy.spiders import Spider
from UItest.UItest.items import  TaoBMMItem
from scrapy.selector import Selector
import scrapy
import hashlib
from scrapy.http import Request
class TaoBMMSpider(Spider):
    name = "TaoBMM"
    allowed_domains = ["img.alicdn.com","mm.taobao.com"]
    start_urls = ["https://mm.taobao.com/self/aiShow.htm?spm=719.7763510.1998643336.1.fpUCYD&userId=2859309431"]

    def parse(self, response):
        print("开始爬取……")
        sel = Selector(response)
        item = TaoBMMItem()
        item['image_urls'] = sel.xpath('//p//img//@src').extract()
        print('************this is image url:****************', item['image_urls'])
        yield item

============================================settings.py====================================================
BOT_NAME = 'UItest'

SPIDER_MODULES = ['UItest.spiders']
NEWSPIDER_MODULE = 'UItest.spiders'

#ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
ITEM_PIPELINES = {'UItest.pipelines.TaoBMMPipeline': 1}
IMAGES_STORE = 'E:'
DOWNLOAD_DELAY = 0.2
IMAGES_MIN_HEIGHT = 300
IMAGES_MIN_WIDTH = 300
ROBOTSTXT_OBEY = True

==================================================run.py=============================================这个是点击按钮启动爬虫的代码,我只列出关键部分如下:def startSpider(self):
    self.BtnOk.setEnabled(False)
    from twisted.internet import reactor
    from scrapy.utils.log import configure_logging
    configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
    runner = CrawlerRunner({'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36'})
    d = runner.crawl(TaoBMMSpider)
    d.addBoth(lambda _: reactor.stop())
    reactor.run()
================================我是输出=================================
输出是:
OK is clicked!  
INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats']
INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
INFO: Enabled item pipelines:
[]
INFO: Spider opened
INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
DEBUG: Crawled (200) <GET https://mm.taobao.com/self/aiSho ... erId=2859309431> (referer: None)
开始爬取……
************this is image url:**************** ['//img.alicdn.com/imgextra/i2/2859309431/TB1nyssKVXXXXcrXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1DYCCNFXXXXXuXFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1chmDNFXXXXcWXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1wP5vNFXXXXb_XFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB19u61NpXXXXa9XpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1rlLINpXXXXbFXVXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1Upz7NpXXXXctXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1Yv2yNpXXXXcJaXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1gRvPNpXXXXbEXFXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1Isj2NpXXXXbhXpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1uQj7NpXXXXbDXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB18wzxNpXXXXczaXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1FR2ZNpXXXXagXpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1E6EaNpXXXXXoXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1Edb.NpXXXXagXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1Zh6yNpXXXXc4aXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1p0bGNpXXXXcCXVXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB19n_1NpXXXXXDXpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1GFb2NpXXXXa2XpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1IArFNpXXXXcXXVXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1l4L8NpXXXXbmXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1AxTANpXXXXbCaXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1KrfPNpXXXXcvXFXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1RRrxNpXXXXcWaXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1vx2zMVXXXXX0XXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1.9DzMVXXXXXlXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1lJW4MVXXXXXgaFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1gfnlMVXXXXbwXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1oW6pMVXXXXbnXFXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i2/2859309431/TB1canxMVXXXXbpXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1A4O4MVXXXXcQapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB179u8MVXXXXajapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1Zrm_MVXXXXXPapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1gNS9MVXXXXXGapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1vj5yLpXXXXctaXXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i1/2859309431/TB1K49FLpXXXXbEXVXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i4/2859309431/TB1QnaOLpXXXXX1XFXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i3/2859309431/TB1qtOALpXXXXaraXXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i3/2859309431/TB1S.mOLpXXXXX0XFXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i2/2859309431/TB1iVrdLXXXXXc2XVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1cCjBLXXXXXXUXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB11SS_LXXXXXbTaXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1iavnLXXXXXXIXFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1BDzdLXXXXXbSXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1kpDBLXXXXXaqXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1W5rfLXXXXXbeXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1X6nyLXXXXXaCXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1is5iLXXXXXceapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1swehLXXXXXcGapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB12nqoLXXXXXbNaXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1gVSxLXXXXXbpXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1LWmDLXXXXXXMXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB158ewLXXXXXXLXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1yveGLXXXXXcNXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB18LqfLXXXXXbbapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1nyssKVXXXXcrXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1DYCCNFXXXXXuXFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1chmDNFXXXXcWXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1wP5vNFXXXXb_XFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB19u61NpXXXXa9XpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1rlLINpXXXXbFXVXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1Upz7NpXXXXctXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1Yv2yNpXXXXcJaXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1gRvPNpXXXXbEXFXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1Isj2NpXXXXbhXpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1uQj7NpXXXXbDXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB18wzxNpXXXXczaXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1FR2ZNpXXXXagXpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1E6EaNpXXXXXoXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1Edb.NpXXXXagXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1Zh6yNpXXXXc4aXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1p0bGNpXXXXcCXVXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB19n_1NpXXXXXDXpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1GFb2NpXXXXa2XpXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1IArFNpXXXXcXXVXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1l4L8NpXXXXbmXXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1AxTANpXXXXbCaXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1KrfPNpXXXXcvXFXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1RRrxNpXXXXcWaXXXXXXXXXXX_!!0-tstar.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1vx2zMVXXXXX0XXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1.9DzMVXXXXXlXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1lJW4MVXXXXXgaFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1gfnlMVXXXXbwXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1oW6pMVXXXXbnXFXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i2/2859309431/TB1canxMVXXXXbpXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1A4O4MVXXXXcQapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB179u8MVXXXXajapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1Zrm_MVXXXXXPapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1gNS9MVXXXXXGapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1vj5yLpXXXXctaXXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i1/2859309431/TB1K49FLpXXXXbEXVXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i4/2859309431/TB1QnaOLpXXXXX1XFXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i3/2859309431/TB1qtOALpXXXXaraXXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i3/2859309431/TB1S.mOLpXXXXX0XFXXXXXXXXXX_!!2-tstar.png', '//img.alicdn.com/imgextra/i2/2859309431/TB1iVrdLXXXXXc2XVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1cCjBLXXXXXXUXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB11SS_LXXXXXbTaXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1iavnLXXXXXXIXFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1BDzdLXXXXXbSXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB1kpDBLXXXXXaqXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i1/2859309431/TB1W5rfLXXXXXbeXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1X6nyLXXXXXaCXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1is5iLXXXXXceapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB1swehLXXXXXcGapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i2/2859309431/TB12nqoLXXXXXbNaXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1gVSxLXXXXXbpXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1LWmDLXXXXXXMXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB158ewLXXXXXXLXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i4/2859309431/TB1yveGLXXXXXcNXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg', '//img.alicdn.com/imgextra/i3/2859309431/TB18LqfLXXXXXbbapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg']

DEBUG: Scraped from <200 https://mm.taobao.com/self/aiSho ... erId=2859309431>
{'image_urls': ['//img.alicdn.com/imgextra/i2/2859309431/TB1nyssKVXXXXcrXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1DYCCNFXXXXXuXFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1chmDNFXXXXcWXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1wP5vNFXXXXb_XFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB19u61NpXXXXa9XpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1rlLINpXXXXbFXVXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1Upz7NpXXXXctXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1Yv2yNpXXXXcJaXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1gRvPNpXXXXbEXFXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1Isj2NpXXXXbhXpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1uQj7NpXXXXbDXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB18wzxNpXXXXczaXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1FR2ZNpXXXXagXpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1E6EaNpXXXXXoXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1Edb.NpXXXXagXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1Zh6yNpXXXXc4aXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1p0bGNpXXXXcCXVXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB19n_1NpXXXXXDXpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1GFb2NpXXXXa2XpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1IArFNpXXXXcXXVXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1l4L8NpXXXXbmXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1AxTANpXXXXbCaXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1KrfPNpXXXXcvXFXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1RRrxNpXXXXcWaXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1vx2zMVXXXXX0XXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1.9DzMVXXXXXlXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1lJW4MVXXXXXgaFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1gfnlMVXXXXbwXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1oW6pMVXXXXbnXFXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1canxMVXXXXbpXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1A4O4MVXXXXcQapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB179u8MVXXXXajapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1Zrm_MVXXXXXPapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1gNS9MVXXXXXGapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1vj5yLpXXXXctaXXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1K49FLpXXXXbEXVXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1QnaOLpXXXXX1XFXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1qtOALpXXXXaraXXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1S.mOLpXXXXX0XFXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1iVrdLXXXXXc2XVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1cCjBLXXXXXXUXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB11SS_LXXXXXbTaXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1iavnLXXXXXXIXFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1BDzdLXXXXXbSXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1kpDBLXXXXXaqXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1W5rfLXXXXXbeXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1X6nyLXXXXXaCXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1is5iLXXXXXceapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1swehLXXXXXcGapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB12nqoLXXXXXbNaXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1gVSxLXXXXXbpXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1LWmDLXXXXXXMXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB158ewLXXXXXXLXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1yveGLXXXXXcNXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB18LqfLXXXXXbbapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1nyssKVXXXXcrXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1DYCCNFXXXXXuXFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1chmDNFXXXXcWXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1wP5vNFXXXXb_XFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB19u61NpXXXXa9XpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1rlLINpXXXXbFXVXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1Upz7NpXXXXctXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1Yv2yNpXXXXcJaXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1gRvPNpXXXXbEXFXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1Isj2NpXXXXbhXpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1uQj7NpXXXXbDXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB18wzxNpXXXXczaXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1FR2ZNpXXXXagXpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1E6EaNpXXXXXoXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1Edb.NpXXXXagXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1Zh6yNpXXXXc4aXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1p0bGNpXXXXcCXVXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB19n_1NpXXXXXDXpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1GFb2NpXXXXa2XpXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1IArFNpXXXXcXXVXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1l4L8NpXXXXbmXXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1AxTANpXXXXbCaXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1KrfPNpXXXXcvXFXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1RRrxNpXXXXcWaXXXXXXXXXXX_!!0-tstar.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1vx2zMVXXXXX0XXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1.9DzMVXXXXXlXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1lJW4MVXXXXXgaFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1gfnlMVXXXXbwXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1oW6pMVXXXXbnXFXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1canxMVXXXXbpXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1A4O4MVXXXXcQapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB179u8MVXXXXajapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1Zrm_MVXXXXXPapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1gNS9MVXXXXXGapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1vj5yLpXXXXctaXXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1K49FLpXXXXbEXVXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1QnaOLpXXXXX1XFXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1qtOALpXXXXaraXXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1S.mOLpXXXXX0XFXXXXXXXXXX_!!2-tstar.png',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1iVrdLXXXXXc2XVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1cCjBLXXXXXXUXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB11SS_LXXXXXbTaXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1iavnLXXXXXXIXFXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1BDzdLXXXXXbSXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB1kpDBLXXXXXaqXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i1/2859309431/TB1W5rfLXXXXXbeXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1X6nyLXXXXXaCXXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1is5iLXXXXXceapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB1swehLXXXXXcGapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i2/2859309431/TB12nqoLXXXXXbNaXXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1gVSxLXXXXXbpXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1LWmDLXXXXXXMXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB158ewLXXXXXXLXVXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i4/2859309431/TB1yveGLXXXXXcNXpXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg',
                '//img.alicdn.com/imgextra/i3/2859309431/TB18LqfLXXXXXbbapXXXXXXXXXX_!!0-tstar.jpg_620x10000.jpg']}
INFO: Closing spider (finished)
INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 359,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 11658,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 10, 12, 14, 17, 32, 570380),
'item_scraped_count': 1,
'log_count/DEBUG': 2,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2016, 10, 12, 14, 17, 32, 116354)}
INFO: Spider closed (finished)


回复

使用道具 举报

47

主题

999

帖子

1050

积分

管理员

Rank: 9Rank: 9Rank: 9

积分
1050

活跃会员热心会员最佳导师默默耕耘突出贡献优秀版主荣誉管理论坛元老

blueelwang 发表于 2016-10-13 20:49:51 | 显示全部楼层
并没有看到错误提示, 建议使用log方法来调试, 而不是print, 这个print不一定能够打印出来
回复 支持 1 反对 0

使用道具 举报

1

主题

4

帖子

4

积分

贫民

积分
4
duron101  楼主| 发表于 2016-10-13 22:48:29 | 显示全部楼层
可是为什么是Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)?
回复 支持 反对

使用道具 举报

1

主题

4

帖子

4

积分

贫民

积分
4
duron101  楼主| 发表于 2016-10-13 23:00:25 | 显示全部楼层
blueelwang 发表于 2016-10-13 20:49
并没有看到错误提示, 建议使用log方法来调试, 而不是print, 这个print不一定能够打印出来 ...

输出的红色部分:
INFO: Enabled item pipelines:
[]
INFO: Spider opened
INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
是否意思是pipelines未启用?
爬取失败?
回复 支持 反对

使用道具 举报

47

主题

999

帖子

1050

积分

管理员

Rank: 9Rank: 9Rank: 9

积分
1050

活跃会员热心会员最佳导师默默耕耘突出贡献优秀版主荣誉管理论坛元老

blueelwang 发表于 2016-10-14 19:55:25 | 显示全部楼层
duron101 发表于 2016-10-13 23:00
输出的红色部分:
INFO: Enabled item pipelines:
[]

并不表示失败, 只是说待抓取的页面数为0
回复 支持 反对

使用道具 举报

1

主题

4

帖子

4

积分

贫民

积分
4
duron101  楼主| 发表于 2016-10-22 22:14:37 | 显示全部楼层
blueelwang 发表于 2016-10-14 19:55
并不表示失败, 只是说待抓取的页面数为0

我尝试了无数次,现在解决的情况如下,请版主帮忙分析下问题:
1、利用程序点击按钮,在按钮函数中执行:
process = CrawlerProcess({'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36'})
        process.crawl(TaoBMMSpider)
        process.start()
启动爬虫时,爬虫可以运行,但是无法用pipeline爬取数据,只能在parse回调函数中对url逐个处理,将读取的图片写入到文件中。
2、在命令行中直接利用scrapy crawl TaoBMM,则可以启用pipeline,下载图片数据。
这是为何?
如果版主方便,我可以将工程文件发送给您信箱,请帮忙分析下。
回复 支持 反对

使用道具 举报

0

主题

5

帖子

5

积分

贫民

积分
5
wei20124047 发表于 2018-1-26 17:27:47 | 显示全部楼层
INFO: Enabled item pipelines:
[]
INFO: Spider opened
INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
这一句好像并不意味着错误  我今天是属于上一步骤之后下一步spider closeed  后面看了是整个程序有小错误 比如start_urls 写成了start_url 等等之类的错误
回复 支持 反对

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

快速回复 返回顶部 返回列表