FAQ
Hi,

I'm currently looking into implementing a load shedding strategy into
Flink's Kafka consumer.

Therefore, I would like to allow users to request the latest offset of the
subscribed TopicPartitions, so that they can
a) determine the lag
b) maybe set the next fetch offset to the latest offset (or anything in
between)

With the SimpleConsumer, there is the getOffsetsBefore() method, which
allows to request the largest offset.
For the new KafkaConsumer, I only found the seekToEnd() and
seekToBeginning() methods, but they don't allow to get the actual end or
beginning.


Are there plans to add such an API to the KafkaConsumer again?
Can you suggest any workarounds?

Regards,
Robert

Search Discussions

  • Gwen Shapira at Jan 22, 2016 at 4:22 am
    Hi Robert!

    Jason is the expert, and I hope he'll respond soon.

    Meanwhile: I think that you can do what you are trying to do by:
    1. call position() to get the current position you are consuming
    2. call seekToEnd() and then position(), which will give you the last
    position at the point in which you called seekToEnd()
    3. Use another seek() call to go to the offset you really want to get to.

    I don't think there are current plans to add getOffsetsBefore, but maybe we
    need it for the use-case you specified.
    I think the developer mailing list (or a JIRA) will be a better place for
    an API discussion.

    Gwen
    On Wed, Jan 20, 2016 at 3:09 AM, Robert Metzger wrote:

    Hi,

    I'm currently looking into implementing a load shedding strategy into
    Flink's Kafka consumer.

    Therefore, I would like to allow users to request the latest offset of the
    subscribed TopicPartitions, so that they can
    a) determine the lag
    b) maybe set the next fetch offset to the latest offset (or anything in
    between)

    With the SimpleConsumer, there is the getOffsetsBefore() method, which
    allows to request the largest offset.
    For the new KafkaConsumer, I only found the seekToEnd() and
    seekToBeginning() methods, but they don't allow to get the actual end or
    beginning.


    Are there plans to add such an API to the KafkaConsumer again?
    Can you suggest any workarounds?

    Regards,
    Robert
  • Jason Gustafson at Jan 22, 2016 at 7:37 pm
    The offset API is one of the known gaps in the new consumer. There is a
    JIRA (KAFKA-1332), but we might need a KIP to make that change now that the
    API is released. For now, Gwen's suggestion is the only way to do it.

    -Jason
    On Thu, Jan 21, 2016 at 8:22 PM, Gwen Shapira wrote:

    Hi Robert!

    Jason is the expert, and I hope he'll respond soon.

    Meanwhile: I think that you can do what you are trying to do by:
    1. call position() to get the current position you are consuming
    2. call seekToEnd() and then position(), which will give you the last
    position at the point in which you called seekToEnd()
    3. Use another seek() call to go to the offset you really want to get to.

    I don't think there are current plans to add getOffsetsBefore, but maybe we
    need it for the use-case you specified.
    I think the developer mailing list (or a JIRA) will be a better place for
    an API discussion.

    Gwen
    On Wed, Jan 20, 2016 at 3:09 AM, Robert Metzger wrote:

    Hi,

    I'm currently looking into implementing a load shedding strategy into
    Flink's Kafka consumer.

    Therefore, I would like to allow users to request the latest offset of the
    subscribed TopicPartitions, so that they can
    a) determine the lag
    b) maybe set the next fetch offset to the latest offset (or anything in
    between)

    With the SimpleConsumer, there is the getOffsetsBefore() method, which
    allows to request the largest offset.
    For the new KafkaConsumer, I only found the seekToEnd() and
    seekToBeginning() methods, but they don't allow to get the actual end or
    beginning.


    Are there plans to add such an API to the KafkaConsumer again?
    Can you suggest any workarounds?

    Regards,
    Robert
  • Robert Metzger at Jan 25, 2016 at 10:35 am
    Thank you for your replies Gwen and Jason.

    Lets continue the discussion in the JIRA.

    On Fri, Jan 22, 2016 at 8:37 PM, Jason Gustafson wrote:

    The offset API is one of the known gaps in the new consumer. There is a
    JIRA (KAFKA-1332), but we might need a KIP to make that change now that the
    API is released. For now, Gwen's suggestion is the only way to do it.

    -Jason
    On Thu, Jan 21, 2016 at 8:22 PM, Gwen Shapira wrote:

    Hi Robert!

    Jason is the expert, and I hope he'll respond soon.

    Meanwhile: I think that you can do what you are trying to do by:
    1. call position() to get the current position you are consuming
    2. call seekToEnd() and then position(), which will give you the last
    position at the point in which you called seekToEnd()
    3. Use another seek() call to go to the offset you really want to get to.

    I don't think there are current plans to add getOffsetsBefore, but maybe we
    need it for the use-case you specified.
    I think the developer mailing list (or a JIRA) will be a better place for
    an API discussion.

    Gwen

    On Wed, Jan 20, 2016 at 3:09 AM, Robert Metzger <rmetzger@apache.org>
    wrote:
    Hi,

    I'm currently looking into implementing a load shedding strategy into
    Flink's Kafka consumer.

    Therefore, I would like to allow users to request the latest offset of the
    subscribed TopicPartitions, so that they can
    a) determine the lag
    b) maybe set the next fetch offset to the latest offset (or anything in
    between)

    With the SimpleConsumer, there is the getOffsetsBefore() method, which
    allows to request the largest offset.
    For the new KafkaConsumer, I only found the seekToEnd() and
    seekToBeginning() methods, but they don't allow to get the actual end
    or
    beginning.


    Are there plans to add such an API to the KafkaConsumer again?
    Can you suggest any workarounds?

    Regards,
    Robert

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categorieskafka
postedJan 20, '16 at 11:09a
activeJan 25, '16 at 10:35a
posts4
users3
websitekafka.apache.org
irc#kafka

People

Translate

site design / logo © 2021 Grokbase