Search Discussions

116 discussions - 641 posts

  • Hi all, Over at the Wikimedia Foundation, we're trying to figure out the best way to do our ETL from Kafka into Hadoop. We don't currently use Avro and I'm not sure if we are going to. I came across ...
    Aug 7, 2013 at 4:42 pm
    Aug 13, 2013 at 11:17 pm
  • Hi team, When I delete a topic, the topic is deleted from zookeeper but its log files are not deleted from Brokers. When I restart a broker, the broker will try to sync the log files whose topic has ...
    Yu, LiboYu, Libo
    Aug 22, 2013 at 5:31 pm
    Aug 24, 2013 at 4:16 am
  • Hi team, We have three kafka brokers in a production cluster. We use replication factor 3 for all topics. We notice quite frequently one broker is not in isr. Sometimes after it is restarted, it will ...
    Yu, LiboYu, Libo
    Aug 21, 2013 at 2:51 pm
    Aug 23, 2013 at 1:26 pm
  • Hi, I tried to run kafka-preferred-replica-election.sh on our kafka cluster. But I got this expection: Failed to start preferred replica election org.I0Itec.zkclient.exception.ZkNoNodeException ...
    Yu, LiboYu, Libo
    Aug 20, 2013 at 3:36 pm
    Aug 22, 2013 at 7:08 pm
  • We have 3 node kafka cluster. I initially created 4 topics. I wrote small shell script to create 150 topics. TOPICS=$(< $1) for topic in $TOPICS do echo "/usr/local/kafka/bin/kafka-create-topic.sh ...
    Vadim KeylisVadim Keylis
    Aug 14, 2013 at 6:04 am
    Aug 16, 2013 at 4:04 am
  • So my understanding is num of active streams that a consumer can utilize is number of partitions in topic. This is fine if we consumer from specific topic. But if we consumer from TopicFilter, I ...
    Rajasekar ElangoRajasekar Elango
    Aug 29, 2013 at 3:03 pm
    Oct 3, 2013 at 3:11 pm
  • According to the Kafka 8 documentation under broker configuration. There are these parameters and their definitions. log.retention.bytes -1 The maximum size of the log before deleting it ...
    Paul ChristianPaul Christian
    Aug 17, 2013 at 4:33 am
    Aug 28, 2013 at 8:04 pm
  • Hey guys! We recently deployed our kafka data pipeline application over the weekend and it is working out quite well once we ironed out all the issues. There is one behavior that we've noticed that ...
    Ian FriedmanIan Friedman
    Aug 24, 2013 at 4:59 pm
    Aug 26, 2013 at 9:43 pm
  • Hey guys, I designed my consumer app (running on 0.7) to run with autocommit off and commit manually once it was done processing a record. The intent was so that if a consumer died while processing a ...
    Ian FriedmanIan Friedman
    Aug 15, 2013 at 12:22 am
    Aug 21, 2013 at 3:54 am
  • Hi team, This is our current use case: Assume there is a topic with multiple partitions. 1 Create a connector first and create multiple streams from the connector for a topic. 2 Create multiple ...
    Yu, LiboYu, Libo
    Aug 29, 2013 at 5:28 pm
    Oct 4, 2013 at 4:11 pm
  • Hello everyone, We recently increased number of partitions from 4 to 16 and after that console producer mostly fails with LeaderNotAvailableException and exits after 3 tries: Here is last few lines ...
    Rajasekar ElangoRajasekar Elango
    Aug 27, 2013 at 3:54 pm
    Aug 29, 2013 at 4:16 pm
  • I'm using the kafka.javaapi.producer.Producer class from a java client. I'm wondering if it ever makes sense to refresh a producer by stopping it and creating a new one, for example in response to a ...
    Jason RosenbergJason Rosenberg
    Aug 23, 2013 at 11:10 pm
    Aug 26, 2013 at 7:21 am
  • Can someone break down how message serialization would work with Avro? I've read instead of adding a schema to every single event it would be wise to add some sort of fingerprint with each message to ...
    Aug 20, 2013 at 2:08 pm
    Aug 22, 2013 at 12:24 pm
  • Hi , I am trying to enable gzip compression for my events. But after I switched compression.codec to "1" I found the produced events were even not be persisted to disk log file. Of course, the ...
    Lu XuechaoLu Xuechao
    Aug 29, 2013 at 8:40 am
    Aug 30, 2013 at 7:23 am
  • Good afternoon. We are running kafka on centos linux. I enabled controlled shutdown in the property file. We are starting/stopping kafka using init script. The init script will issue term signal ...
    Vadim KeylisVadim Keylis
    Aug 18, 2013 at 10:33 pm
    Aug 21, 2013 at 4:55 am
  • Hi, We are using kafka-0.7.2 with zookeeper (3.4.5) Our cluster configuration: 3 brokers on 3 different machines. Each broker machine has a zookeeper instance running as well. We are currently facing ...
    Ramakrishna MallyaRamakrishna Mallya
    Aug 14, 2013 at 11:30 pm
    Aug 16, 2013 at 2:42 pm
  • Is there a way to stop a malicious user to connect directly to a kafka broker and send any messages? Could we have the brokers to accept a message to a list of know IPs?
    Calvin LeiCalvin Lei
    Aug 29, 2013 at 5:03 pm
    Sep 2, 2013 at 8:14 pm
  • 8


    Can you view Kafka metrics via JConsole? I've tried connecting to port 9999 with no such luck?
    Aug 29, 2013 at 10:42 pm
    Aug 29, 2013 at 11:54 pm
  • Somehow I am getting my instances of kafka to crash. I started kafka instances one by one and they started successfully. Later it some how two of 3 instances became completely unresponsive. The ...
    Vadim KeylisVadim Keylis
    Aug 27, 2013 at 5:13 am
    Aug 28, 2013 at 2:49 pm
  • Hi, When a broker is not in a topic's ISR, will it try to catch up to go back to ISR itself? Or do we have to restart it? We can increase replica.lag.time.max.ms and replica.lag.max.messages to let ...
    Yu, LiboYu, Libo
    Aug 23, 2013 at 6:34 pm
    Aug 27, 2013 at 4:03 pm
  • Hi, I am using Kafka 0.7.1, and using the low-level SyncProducer to send messages to a *single* partition from a *single* thread. The client sends messages that contain sequential numbers so it is ...
    Ross BlackRoss Black
    Aug 23, 2013 at 4:32 am
    Aug 25, 2013 at 12:01 pm
  • Hi, We are using zookeeper 3.3.6 with kafka 0.7.2. We have a topic with 8 partitions on each of 3 brokers that we are consuming with a consumer group with multiple threads. We are using the following ...
    Drew DaughertyDrew Daugherty
    Aug 13, 2013 at 11:58 pm
    Aug 17, 2013 at 4:36 am
  • Hi, We are using mirrormaker to replicate data between two kafka clusters. I am seeing huge difference in size of log in data dir between the broker in source cluster vs broker in destination ...
    Rajasekar ElangoRajasekar Elango
    Aug 22, 2013 at 9:15 pm
    Aug 23, 2013 at 4:24 pm
  • Hi, I am wondering if my leader broker crash, how to get it back to ISR after restart kafak ? In the initial status the kafka-list-topic.sh shows: topic: failover-test partition: 0 leader: 0 ...
    James WuJames Wu
    Aug 20, 2013 at 11:26 am
    Aug 22, 2013 at 12:22 am
  • Hi, This is more of a thought question than a problem that I need support for. I have trying out Kafka 0.8.0-beta1 with replication. For our user case we want to try and guarantee that our consumers ...
    Bryan BaugherBryan Baugher
    Aug 14, 2013 at 4:52 pm
    Aug 16, 2013 at 8:47 pm
  • We are using kafka08 beta1. We had 6 kafka in the tier. I have replication set to 3. Then 3 servers were removed using control shutdown method. I am getting the error bellow after that. What went ...
    Vadim KeylisVadim Keylis
    Aug 7, 2013 at 5:09 am
    Aug 7, 2013 at 3:12 pm
  • Hi all, Since I couldn't find any other way to publish kafka metrics to ganglia from kafka 0.8 (beta), I just published on github a super-simple ganglia metrics reporter for Kafka. It is configurable ...
    Maxime BrugidouMaxime Brugidou
    Aug 22, 2013 at 3:43 pm
    Nov 1, 2013 at 6:08 pm
  • Is it common/necessary to load balance the connections your producers use? I'm looking at the Ruby gem and it doesn't look like this is handled via the gem.
    Aug 29, 2013 at 4:02 am
    Aug 29, 2013 at 4:28 pm
  • Hello, I am getting started with Kafka and the various getting started guides are getting me close, but Ive run into some basic issues. Ive followed these guides ...
    Robert HeiseRobert Heise
    Aug 27, 2013 at 6:36 pm
    Aug 28, 2013 at 4:00 am
  • I ran command to create 300 topics sequentially. Kafka create topic script return with topic created successfully message. However running kafka-list-topics.sh script showed to me that no leaders ...
    Vadim KeylisVadim Keylis
    Aug 22, 2013 at 5:02 pm
    Aug 24, 2013 at 5:48 am
  • Hi, i am having problems using kafka as a dependency in sbt. with this simple build.sbt: name := "kafka-dependency-test" scalaVersion := "2.9.2" libraryDependencies += "org.apache.kafka" % ...
    Lorenz KniesLorenz Knies
    Aug 14, 2013 at 7:50 am
    Aug 22, 2013 at 3:31 am
  • I have a cluster of 3 kafka servers. Replication factor is 3. Two out of 3 servers were shutdown and traffic was sent to only one server that was up. I brought second host up and it says according to ...
    Vadim KeylisVadim Keylis
    Aug 19, 2013 at 7:27 pm
    Aug 21, 2013 at 5:16 pm
  • I am trying to setup kafka service and connect to zookeeper that would be shared with Other projects. Can someone advice how to configure namespace in kafka and zookeeper. Thanks so much Sent from my ...
    Vadim KeylisVadim Keylis
    Aug 9, 2013 at 10:25 pm
    Aug 10, 2013 at 10:20 pm
  • I wonder why at-least-once guarantee is easier to maintain than exactly-once (in that the latter requires 2PC while the former does not , according to ...
    Aug 7, 2013 at 11:01 pm
    Sep 4, 2013 at 6:07 am
  • 1) Should a producer be aware of which broker to write to or is this somehow managed by Kafka itself. For example, If I have 2 brokers with a configured partition size of 1 will my messages be ...
    Aug 29, 2013 at 6:10 pm
    Aug 29, 2013 at 8:41 pm
  • I have a situation where two high level consumers are being created to consume a single topic. There is only one partition for the topic, so I understand that only one Consumer will end up owning the ...
    Dennis HallerDennis Haller
    Aug 22, 2013 at 3:54 pm
    Aug 22, 2013 at 5:46 pm
  • Does LinkedIn include the SHA of the schema into the header of each Avro message they write or do they wrap the avro message and prepend the SHA? In either case, how does the Hadoop consumer know ...
    Aug 22, 2013 at 3:15 am
    Aug 22, 2013 at 4:52 pm
  • I have not been able to find reliable advice regarding how many partitions should exist on a single broker. How many partitions have you used, and what kind of throughput have you seen? Thanks in ...
    Tom BrownTom Brown
    Aug 21, 2013 at 2:07 pm
    Aug 21, 2013 at 6:26 pm
  • Hi, In Kafka source code, kafka.utils.TestUtils is defined in src/core/src/test/scala/unit/kafka/utils/TestUtils.scala, but the kafka_2.8.0-0.8.0-beta1.jar doesn't include kafka.utils.TestUtils. I'm ...
    Wu, Jiang2Wu, Jiang2
    Aug 19, 2013 at 4:09 pm
    Aug 19, 2013 at 10:51 pm
  • Chris commented in another thread about the poor compression performance in 0.8, even with snappy. Indeed if I run the linear log write throughput test on my laptop I see 75MB/sec with no compression ...
    Jay KrepsJay Kreps
    Aug 3, 2013 at 5:20 am
    Aug 16, 2013 at 2:34 pm
  • We are getting hanged on consumer side when we try to consume data using Scala API. We found it when we did load test.Attaching thread dump,please let us know if we have any fix. Thanks, Balaji
    Seshadri, BalajiSeshadri, Balaji
    Aug 14, 2013 at 8:50 pm
    Aug 15, 2013 at 4:55 am
  • Are all Kafka clients working with the latest version of Kafka? I tried the kafka-rb client and a simple example listed in the README but I keep getting a nasty error require 'kafka' producer = ...
    Aug 10, 2013 at 9:55 pm
    Aug 12, 2013 at 1:51 pm
  • These sounds like great steps. A couple of votes and questions: 1. Moving serialization out and basing it all off of byte[] for key and payload makes sense. Echoing a response below, we've ended up ...
    Chris HogueChris Hogue
    Aug 2, 2013 at 7:29 pm
    Aug 3, 2013 at 1:50 pm
  • Any error in the broker's state-change log? Thanks, Jun On Wed, Jul 31, 2013 at 2:32 PM, Nandigam, Sujitha wrote:
    Jun RaoJun Rao
    Aug 1, 2013 at 4:53 am
    Aug 2, 2013 at 4:11 am
  • Hi Jun, It's been some time since you last post, is this patch available now. Also we are doing following manual steps to update existing topic to use new partitions. 1) stop all zookeepers 2) stop ...
    Rajasekar ElangoRajasekar Elango
    Aug 28, 2013 at 3:47 pm
    Aug 28, 2013 at 10:30 pm
  • I am trying to tune mirrormaker configurations based on this doc ...
    Rajasekar ElangoRajasekar Elango
    Aug 22, 2013 at 8:45 pm
    Aug 23, 2013 at 2:58 pm
  • Hi, This is from kafka faq: * Each partition is not consumed by more than one consumer thread/process in each consumer group. This allows to have each process consume in a single threaded fashion to ...
    Yu, LiboYu, Libo
    Aug 21, 2013 at 7:31 pm
    Aug 22, 2013 at 2:42 pm
  • Hello guys. I am not clear about leader election rules in case of server or service failure. If one of the servers went down that was a leader for some partitions. Will new leader be elected ...
    Vadim KeylisVadim Keylis
    Aug 16, 2013 at 1:58 am
    Aug 19, 2013 at 12:57 am
  • We have cluster of 3 Kafka nodes. All topics created with replication factor of 3. Lets say two crashed for what ever reason. I ran tool prefer replica election leaders script. it look to me as ...
    Vadim KeylisVadim Keylis
    Aug 15, 2013 at 1:10 am
    Aug 15, 2013 at 9:00 pm
  • Everyone, I need a little help figuring out how buffers are allocated in Kafka consumers ver 0.8. What are the proper settings for a consumer that needs to receive a single message that is 210 MB in ...
    Eric SitesEric Sites
    Aug 15, 2013 at 4:12 am
    Aug 15, 2013 at 2:58 pm
Group Navigation
period‹ prev | Aug 2013 | next ›
Group Overview
groupusers @

89 users for August 2013

Jun Rao: 91 posts Neha Narkhede: 55 posts Yu, Libo: 46 posts Vadim Keylis: 45 posts Jay Kreps: 42 posts Mark: 27 posts Wang Guozhang: 27 posts Rajasekar Elango: 21 posts Philip O'Toole: 18 posts Ian Friedman: 17 posts Andrew Otto: 15 posts Jason Rosenberg: 14 posts Joe Stein: 14 posts Joel Koshy: 12 posts Eric Sites: 11 posts Chris Hogue: 8 posts Wu, Jiang2: 8 posts James Wu: 7 posts Lu Xuechao: 7 posts Tejas Patil: 6 posts
show more