热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

Offsetvs.limit:batchsizeistoolarge,sizemustbelessthanorequalto

Hi,in latest elasticdump is bug with "--offset" defined. Limit is defined to 10000. There is probabl

Hi,

in latest elasticdump is bug with "--offset" defined. Limit is defined to 10000. There is probably unwanted correlation from offset to limit.

Error message: Batch size is too large, size must be less than or equal to: [10000] but was [5120100].

  • ElasticDump version - 3.0.2

  • Elasticsearch version - 5.0.1

  • Full Command:

1
2
3
4
5
6
7
8
elasticdump \

  --input="http://10.10.10.1:9200/logstash-2017.01.03" \

  --output="http://10.10.10.2:9200/logstash-2017.01.03" \

  --offset=5120000 \

  --limit=10000 \

  --timeout=3600000 \

  --ignore-errors \

  --type=data


  • Command output:

1
2
3
4
5
6
7
8
9
10
11
12
Thu, 05 Jan 2017 09:58:51 GMT | starting dump

Thu, 05 Jan 2017 09:58:51 GMT | Warning: offseting 5120000 rows.

Thu, 05 Jan 2017 09:58:51 GMT |   * Using an offset doesn't guarantee that the offset rows have already been written, please refer to the HELP text.

Thu, 05 Jan 2017 09:58:51 GMT | Error Emitted => {"error":{"root_cause":[{"type":"query_phase_execution_exception","reason":"Batch size is too large, size must be less than or equal to: [10000] but was [5120100]. Scroll

batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetc

h","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-2017.01.03","node":"vdJLjO9uT1OEsTn9lIPRqw","reason":{"type":"query_phase_execution_exception","reason":"Batch size is too large, size must be less than or

equal to: [10000] but was [5120100]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."}}],"caused_by":{"type":"query_phase_execution_ex

ception","reason":"Batch size is too large, size must be less than or equal to: [10000] but was [5120100]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window]

index level setting."}},"status":500}

Thu, 05 Jan 2017 09:58:51 GMT | got 0 objects from source elasticsearch (offset: 5120000)

Thu, 05 Jan 2017 09:58:51 GMT | Total Writes: 0

Thu, 05 Jan 2017 09:58:51 GMT | dump complete

Thank you for your check.

该提问来源于开源项目:elasticsearch-dump/elasticsearch-dump


  1. You can increase
    1
    index.max_result_window

    (memory bound + not the most efficient option)


  2. Use the
    1
    --searchBody

    to filter the resultset a more manageable subset


Please re-open if you do not feel the question has been answered sufficiently.





   



推荐阅读
author-avatar
Because_of_you龙
这个家伙很懒,什么也没留下!
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有