Grokbase Groups Kafka dev March 2013
FAQ

[Kafka-dev] [jira] [Commented] (KAFKA-156) Messages should not be dropped when brokers are unavailable

David Almroth (JIRA)
Mar 9, 2013 at 3:43 pm
[ https://issues.apache.org/jira/browse/KAFKA-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597981#comment-13597981 ]

David Almroth commented on KAFKA-156:
-------------------------------------

This is feature is very important for me. I am intrested in migrating from Scribe to Kafka and this feature is the only thing keeping me from starting now.

It looks like isse https://issues.apache.org/jira/browse/KAFKA-789 is a duplicate of this.

Messages should not be dropped when brokers are unavailable
-----------------------------------------------------------

Key: KAFKA-156
URL: https://issues.apache.org/jira/browse/KAFKA-156
Project: Kafka
Issue Type: Improvement
Reporter: Sharad Agarwal
Fix For: 0.8


When none of the broker is available, producer should spool the messages to disk and keep retrying for brokers to come back.
This will also enable brokers upgrade/maintenance without message loss.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
reply

Search Discussions

3 responses

  • Matan Safriel (JIRA) at Mar 9, 2013 at 5:19 pm
    [ https://issues.apache.org/jira/browse/KAFKA-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598000#comment-13598000 ]

    Matan Safriel commented on KAFKA-156:
    -------------------------------------

    I was led to believe this is most probably not planned for 0.8.
    Just to comment that currently in 0.8 a message set asynchronously sent may partially fail, in the sense that only some of its destinations (determined by topic & partition) will fail while others succeed. That seems to mean a 'one log spool' for all messages (if all messages are spooled before attempting to send them) requires non-sequential management of such a log, or overheads over-proportional to the proportion of failed messages. Perhaps a log per topic would simplify the algorithm but then there's partitions as well...
    Messages should not be dropped when brokers are unavailable
    -----------------------------------------------------------

    Key: KAFKA-156
    URL: https://issues.apache.org/jira/browse/KAFKA-156
    Project: Kafka
    Issue Type: Improvement
    Reporter: Sharad Agarwal
    Fix For: 0.8


    When none of the broker is available, producer should spool the messages to disk and keep retrying for brokers to come back.
    This will also enable brokers upgrade/maintenance without message loss.
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Matan Safriel (JIRA) at Mar 9, 2013 at 5:19 pm
    [ https://issues.apache.org/jira/browse/KAFKA-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598000#comment-13598000 ]

    Matan Safriel edited comment on KAFKA-156 at 3/9/13 5:17 PM:
    -------------------------------------------------------------

    I was led to believe this is most probably not planned for 0.8 now.
    Just to comment that currently in 0.8 a message set asynchronously sent may partially fail, in the sense that only some of its destinations (determined by topic & partition) will fail while others succeed. That seems to mean a 'one log spool' for all messages (if all messages are spooled before attempting to send them) requires non-sequential management of such a log, or overheads over-proportional to the proportion of failed messages. Perhaps a log per topic would simplify the algorithm but then there's partitions as well...

    was (Author: matan):
    I was led to believe this is most probably not planned for 0.8.
    Just to comment that currently in 0.8 a message set asynchronously sent may partially fail, in the sense that only some of its destinations (determined by topic & partition) will fail while others succeed. That seems to mean a 'one log spool' for all messages (if all messages are spooled before attempting to send them) requires non-sequential management of such a log, or overheads over-proportional to the proportion of failed messages. Perhaps a log per topic would simplify the algorithm but then there's partitions as well...
    Messages should not be dropped when brokers are unavailable
    -----------------------------------------------------------

    Key: KAFKA-156
    URL: https://issues.apache.org/jira/browse/KAFKA-156
    Project: Kafka
    Issue Type: Improvement
    Reporter: Sharad Agarwal
    Fix For: 0.8


    When none of the broker is available, producer should spool the messages to disk and keep retrying for brokers to come back.
    This will also enable brokers upgrade/maintenance without message loss.
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Jun Rao (JIRA) at Mar 11, 2013 at 3:17 pm
    [ https://issues.apache.org/jira/browse/KAFKA-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598881#comment-13598881 ]

    Jun Rao commented on KAFKA-156:
    -------------------------------

    Yes, this is likely a post 0.8 item.
    Messages should not be dropped when brokers are unavailable
    -----------------------------------------------------------

    Key: KAFKA-156
    URL: https://issues.apache.org/jira/browse/KAFKA-156
    Project: Kafka
    Issue Type: Improvement
    Reporter: Sharad Agarwal
    Fix For: 0.8


    When none of the broker is available, producer should spool the messages to disk and keep retrying for brokers to come back.
    This will also enable brokers upgrade/maintenance without message loss.
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post

1 user in discussion

Jun Rao (JIRA): 4 posts