热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

分布式爬虫时出现异常:redis.exceptions.ResponseError:DISCARDwithoutMULTI

在使用

  1. 在使用
    1
    scrapy-redis

    进行分布式爬虫过程中出现了这个异常,不是一开始就出现,而是爬虫过程中突然出现的。用了 5 台机器同时爬取。




  2. 异常信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Traceback (most recent call last):

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/commands/crawl.py", line 58, in run

    self.crawler_process.start()

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 291, in start

    reactor.run(installSignalHandlers=False)  # blocking call

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 1267, in run

    self.mainLoop()

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 1276, in mainLoop

    self.runUntilCurrent()

--- ---

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 902, in runUntilCurrent

    call.func(*call.args, **call.kw)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/utils/reactor.py", line 41, in __call__

    return self._func(*self._a, **self._kw)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/core/engine.py", line 122, in _next_request

    if not self._next_request_from_scheduler(spider):

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/core/engine.py", line 149, in _next_request_from_scheduler

    request = slot.scheduler.next_request()

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy_redis_bloomfilter/scheduler.py", line 175, in next_request

    request = self.queue.pop(block_pop_timeout)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy_redis/queue.py", line 115, in pop

    results, count = pipe.execute()

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3443, in execute

    return execute(conn, stack, raise_on_error)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3339, in _execute_transaction

    self.immediate_execute_command('DISCARD')

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3275, in immediate_execute_command

    return self.parse_response(conn, command_name, **options)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3402, in parse_response

    self, connection, command_name, **options)

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 768, in parse_response

    respOnse= connection.read_response()

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/connection.py", line 638, in read_response

    raise response

redis.exceptions.ResponseError: DISCARD without MULTI



   



推荐阅读
author-avatar
2674224139_8a6503
这个家伙很懒,什么也没留下!
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有