V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
V2EX 提问指南
codepark
V2EX  ›  问与答

关于 kafka 日志清理策略的问题

  •  
  •   codepark · 2019-11-26 18:37:50 +08:00 · 3133 次点击
    这是一个创建于 1584 天前的主题,其中的信息可能已经有所发展或是发生改变。

    现象: 搭建了一个 kafka 服务, 使用 kafka-python 包正常生产数据, 但是 kafka 过五分钟就把我的 topic 删除掉. 但是配置 log 的已经配置了, 我认为 kafka 不应该删除我的 topic 历史数据.

    关于 log 清理的配置文件:

    ############################# Log Flush Policy #############################

    log.flush.interval.messages=10000

    log.flush.interval.ms=1000

    ############################# Log Retention Policy #############################

    log.retention.hours=168

    log.retention.bytes=-1

    log.segment.bytes=1073741824

    log.retention.check.interval.ms=300000

    打印的 log:

    [2019-11-26 10:19:11,478] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log) [2019-11-26 10:19:11,485] INFO [ProducerStateManager partition=falcon_source_created-0] Writing producer snapshot at offset 16 (kafka.log.ProducerStateManager) [2019-11-26 10:19:11,487] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Rolled new log segment at offset 16 in 8 ms. (kafka.log.Log) [2019-11-26 10:19:11,488] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Scheduling log segment [baseOffset 0, size 5861] for deletion. (kafka.log.Log) [2019-11-26 10:19:11,491] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Incrementing log start offset to 16 (kafka.log.Log) [2019-11-26 10:19:11,497] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log) [2019-11-26 10:19:11,499] INFO [ProducerStateManager partition=event_mapped-0] Writing producer snapshot at offset 16 (kafka.log.ProducerStateManager) [2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Rolled new log segment at offset 16 in 2 ms. (kafka.log.Log) [2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Scheduling log segment [baseOffset 0, size 6452] for deletion. (kafka.log.Log) [2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Incrementing log start offset to 16 (kafka.log.Log) [2019-11-26 10:20:11,489] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Deleting segment 0 (kafka.log.Log) [2019-11-26 10:20:11,493] INFO Deleted log /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.log.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Deleting segment 0 (kafka.log.Log) [2019-11-26 10:20:11,500] INFO Deleted log /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.log.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,503] INFO Deleted offset index /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.index.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,503] INFO Deleted offset index /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.index.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,504] INFO Deleted time index /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,504] INFO Deleted time index /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)

    查看一些资料发现应该跟他的偏移量的删除策略有关~ 但是我设置了 LOG_CLEANER_ENABLE 关闭删除策略, kafka 依然会删除我的 topic

    1 条回复    2019-11-26 19:40:27 +08:00
    amwyyyy
        1
    amwyyyy  
       2019-11-26 19:40:27 +08:00
    默认的配置,数据是可以保存 7 天。你这个日志有点像 topic 被删了,或者调整分区位置。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   我们的愿景   ·   实用小工具   ·   3173 人在线   最高记录 6543   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 25ms · UTC 12:19 · PVG 20:19 · LAX 05:19 · JFK 08:19
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.