Grokbase Groups Kafka users June 2016
FAQ
Hi All,

Does anyone have any experience of using kafka behind a load balancer?

Would this work? Are there any reasons why you would not want to do it?

Thanks!

Search Discussions

  • Tom Crayford at Jun 3, 2016 at 12:19 pm
    Hi,

    Kafka is designed to distribute traffic between brokers itself. It's
    naturally distributed and does not need, and indeed will not work behind a
    load balancer. I'd recommend reading the docs for more, but
    http://kafka.apache.org/documentation.html#design_loadbalancing is a good
    start.

    Thanks

    Tom Crayford
    Heroku Kafka
    On Fri, Jun 3, 2016 at 1:15 PM, cs user wrote:

    Hi All,

    Does anyone have any experience of using kafka behind a load balancer?

    Would this work? Are there any reasons why you would not want to do it?

    Thanks!
  • Cs user at Jun 3, 2016 at 12:37 pm
    Hi Tom,

    That's great, I thought as much, thanks for taking the time to respond,
    much appreciated!

    Cheers
    On Fri, Jun 3, 2016 at 1:18 PM, Tom Crayford wrote:

    Hi,

    Kafka is designed to distribute traffic between brokers itself. It's
    naturally distributed and does not need, and indeed will not work behind a
    load balancer. I'd recommend reading the docs for more, but
    http://kafka.apache.org/documentation.html#design_loadbalancing is a good
    start.

    Thanks

    Tom Crayford
    Heroku Kafka
    On Fri, Jun 3, 2016 at 1:15 PM, cs user wrote:

    Hi All,

    Does anyone have any experience of using kafka behind a load balancer?

    Would this work? Are there any reasons why you would not want to do it?

    Thanks!
  • Ewen Cheslack-Postava at Jun 5, 2016 at 1:04 am
    Note, however, that a load balancer can be useful for bootstrapping
    purposes, i.e. use it for the bootstrap.servers setting to have a single
    consistent value for the setting but allow the broker list to change over
    time. From there, as Tom says, it'll start using broker hostnames and
    automatically target the specific brokers it needs to communicate with.

    -Ewen
    On Fri, Jun 3, 2016 at 5:37 AM, cs user wrote:

    Hi Tom,

    That's great, I thought as much, thanks for taking the time to respond,
    much appreciated!

    Cheers
    On Fri, Jun 3, 2016 at 1:18 PM, Tom Crayford wrote:

    Hi,

    Kafka is designed to distribute traffic between brokers itself. It's
    naturally distributed and does not need, and indeed will not work behind a
    load balancer. I'd recommend reading the docs for more, but
    http://kafka.apache.org/documentation.html#design_loadbalancing is a good
    start.

    Thanks

    Tom Crayford
    Heroku Kafka
    On Fri, Jun 3, 2016 at 1:15 PM, cs user wrote:

    Hi All,

    Does anyone have any experience of using kafka behind a load balancer?

    Would this work? Are there any reasons why you would not want to do it?

    Thanks!


    --
    Thanks,
    Ewen
  • Todd Palino at Jun 5, 2016 at 4:57 am
    Yep, what Ewen said. We have all of our Kafka clusters behind hardware load
    balancers. Producers (and eventually consumers, once we switch to the new
    consumer) get configured with those VIPs. It’s better than providing a list
    of brokers for the cluster, because we often change the particular brokers
    in the cluster.

    Note that there is a difference in behavior between the older producer
    library and the current one. The old producer would make all metadata
    requests through the VIP (the broker list), and then connect directly to
    the individual brokers for produce/fetch. The current producer uses the VIP
    (bootstrap servers) once at startup to populate the list the list of
    brokers for the cluster and then makes metadata and all other requests to
    those brokers directly (not through the VIP).

    As a side note here, has anyone validated the behavior of the clients when
    the bootstrap.servers is a round robin DNS entry?

    -Todd

    On Sun, Jun 5, 2016 at 6:34 AM, Ewen Cheslack-Postava wrote:

    Note, however, that a load balancer can be useful for bootstrapping
    purposes, i.e. use it for the bootstrap.servers setting to have a single
    consistent value for the setting but allow the broker list to change over
    time. From there, as Tom says, it'll start using broker hostnames and
    automatically target the specific brokers it needs to communicate with.

    -Ewen
    On Fri, Jun 3, 2016 at 5:37 AM, cs user wrote:

    Hi Tom,

    That's great, I thought as much, thanks for taking the time to respond,
    much appreciated!

    Cheers
    On Fri, Jun 3, 2016 at 1:18 PM, Tom Crayford wrote:

    Hi,

    Kafka is designed to distribute traffic between brokers itself. It's
    naturally distributed and does not need, and indeed will not work
    behind
    a
    load balancer. I'd recommend reading the docs for more, but
    http://kafka.apache.org/documentation.html#design_loadbalancing is a good
    start.

    Thanks

    Tom Crayford
    Heroku Kafka
    On Fri, Jun 3, 2016 at 1:15 PM, cs user wrote:

    Hi All,

    Does anyone have any experience of using kafka behind a load
    balancer?
    Would this work? Are there any reasons why you would not want to do
    it?
    Thanks!


    --
    Thanks,
    Ewen


    --
    *Todd Palino*
    Staff Site Reliability Engineer
    Data Infrastructure Streaming



    linkedin.com/in/toddpalino

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categorieskafka
postedJun 3, '16 at 12:15p
activeJun 5, '16 at 4:57a
posts5
users4
websitekafka.apache.org
irc#kafka

People

Translate

site design / logo © 2021 Grokbase