FAQ

Search Discussions

77 discussions - 340 posts

  • Hello all, I am writing a producer client for the v0.8 protocol. I have the wire format encoding working, and much of the functionality works well. However, I have made too many assumptions along the ...
    Ben fleisBen fleis
    Nov 23, 2012 at 4:14 pm
    Dec 10, 2012 at 3:29 pm
  • I've been doing some testing, with an async producer. It seems, if I start up the producer, with no zk cluster present, it does what I expect, that is it waits for a limited time looking for the zk ...
    Jason RosenbergJason Rosenberg
    Nov 19, 2012 at 9:23 pm
    Nov 20, 2012 at 7:19 pm
  • Hey Guys, I was talking with Jay, and he recommended I forward some feedback along. I have been playing with Kafka 0.8 this week, and am feeling the pain in the lack of Maven support for it ...
    Chris RiccominiChris Riccomini
    Nov 20, 2012 at 11:28 pm
    Nov 26, 2012 at 5:05 am
  • I want to suggest kafka should create only one instance of ZkClient globally because ZkClient is thread safe and it will make many users easily customize kafka source code for Zookeeper. In our ...
    Bae, Jae HyeonBae, Jae Hyeon
    Nov 19, 2012 at 6:36 am
    Nov 20, 2012 at 6:42 pm
  • Have folks implemented large installations on Kafka on Amazon EC2. I am looking for best practices. Like the kind of nodes, EBS vs Instance store etc., -- ..Senthil "If there's anything more ...
    Senthilvel RangaswamySenthilvel Rangaswamy
    Nov 17, 2012 at 1:05 am
    Nov 20, 2012 at 5:13 pm
  • Hi, I am trying to write a custom monitoring script for our Kafka setup and would like some help understanding how to interpret the JMX attributes. In our setup, the consumers are writing their ...
    Mike HeffnerMike Heffner
    Nov 20, 2012 at 3:11 am
    Nov 21, 2012 at 10:16 pm
  • Hi, Installed 0.8.0 yesterday, 3 physical machines, 9 brokers running (3 per machine). 2 topics, with 3 replicas each the console producer/consumer examples work fine. When I run my producer logic I ...
    Chris CurtinChris Curtin
    Nov 27, 2012 at 3:18 pm
    Nov 27, 2012 at 8:39 pm
  • Hi. In the kafka broker (version 0.7.0) log I see occasionally following error message FATAL Halting due to unrecoverable I/O error while handling producer request: Unexpected end of ZLIB input ...
    Dmitri PriimakDmitri Priimak
    Nov 28, 2012 at 7:35 pm
    Dec 1, 2012 at 2:25 am
  • Hi, I'm wondering if there's a reason kafka currently seems to depend on zookeeper 3.3.4. While the current stable version for zookeeper is 3.4.4. Are there known issues with upgrading to the newer ...
    Jason RosenbergJason Rosenberg
    Nov 13, 2012 at 7:31 am
    Nov 14, 2012 at 12:47 am
  • Hi, I noticed several errors when writing to a topic with 5 partitions. It looks like the data was written to all 3 brokers, but I get the following errors: 9961 [main] DEBUG ...
    Chris CurtinChris Curtin
    Nov 27, 2012 at 9:12 pm
    Mar 6, 2013 at 3:52 pm
  • Hi, First, thanks for 0.8.0 I'm really impressed with the redundancy and simplification of the producer and consumer models. I've upgraded my consumers from 0.7.2 to 0.8.0 and have some questions. I ...
    Chris CurtinChris Curtin
    Nov 28, 2012 at 4:37 pm
    Dec 3, 2012 at 5:56 pm
  • Hi, I am running the console demo comes with 0.7.2 in getting started guide. All is working fine. Then I use stdio redirect a file of 30K lines of messages into the producer and I received an error ...
    Jamie WangJamie Wang
    Nov 26, 2012 at 10:28 pm
    Nov 30, 2012 at 5:10 am
  • We have a topic whose consumers, on startup, only need to consume from the head of the topic. They don't need to consume old messages they may have missed. We would like to use the high-level ...
    Bob CottonBob Cotton
    Nov 9, 2012 at 1:05 pm
    Nov 16, 2012 at 12:22 am
  • During the implementation of a custom Encoder/Decoder we noticed that should the Decoder throw an exception, the KafkaStream that it is in use becomes invalid. Searching the mailing list indicates ...
    Bob CottonBob Cotton
    Nov 9, 2012 at 4:18 pm
    Nov 13, 2012 at 11:48 pm
  • (I suspect this a bug, but thought I'd check with the group before filing) If I create consumer streams using a topic filter, and request more threads than are actually allocated (based on the ...
    Jason RosenbergJason Rosenberg
    Nov 6, 2012 at 12:35 am
    Nov 7, 2012 at 4:06 pm
  • Getting transactional support is quite hard problem. There will always be corner cases where the solution will not work, unless you want to go down the path of 2PC, paxos, etc which ofcourse will ...
    Rohit PrasadRohit Prasad
    Nov 2, 2012 at 10:12 pm
    Mar 28, 2013 at 4:30 pm
  • All, We're having an issue with Zookeeper, which has nothing to do with Kafka, but my consumers don't appear to be attempting to connect with the two nodes that are up. I specify my zk.connect as ...
    Sybrandy, CaseySybrandy, Casey
    Nov 28, 2012 at 7:41 pm
    Dec 10, 2012 at 4:30 pm
  • Just thought of a way to do transactions in Kafka. I think this solution would cover the most common types of transactions. However, it's often useful to run an idea by a second set of eyes. I am ...
    Tom BrownTom Brown
    Nov 14, 2012 at 8:13 pm
    Nov 28, 2012 at 11:31 pm
  • Hi all, I'm encountering a problem where i'm getting far too many duplicate messages being sent to my kafka setup (statically, using broker.list), being picked up by zk-based consumers. I've provided ...
    Mark GraboisMark Grabois
    Nov 16, 2012 at 5:59 pm
    Nov 27, 2012 at 4:54 am
  • from what I read, the message offset is the byte position of the message in the log file that Kafka writes to the logs are rotated and eventually deleted by Kafka ...does this mean the message offset ...
    Paul GarnerPaul Garner
    Nov 22, 2012 at 3:25 pm
    Nov 22, 2012 at 9:55 pm
  • Hi, We're currently having 1 broker to 10 consumers setup pointing to 5 ZK node. Load would be in 500GB data / day, and following is the generic configuration we're using as, brokerid=0 port=9092 ...
    MuthukumarMuthukumar
    Nov 19, 2012 at 5:35 pm
    Nov 20, 2012 at 8:37 pm
  • Hi, Where can I see some notes regarding differences between 0.7.2 and 0.8? Mainly, incompatibilities between the versions. Thanks, Danny
    Danny YeshurunDanny Yeshurun
    Nov 8, 2012 at 1:39 pm
    Nov 13, 2012 at 5:45 am
  • My consumers randomly hang. They essentially end up waiting on a kafka stream: There is plenty of data on the brokers and a ratio of about 1 consumer for 5 partitions. Any thoughts on how to keep the ...
    Juan ValenciaJuan Valencia
    Nov 29, 2012 at 7:48 pm
    Nov 29, 2012 at 9:36 pm
  • I am using the kafka-proiducer-shell.sh script without the --compress option however my messages seem to be gzipped the docs say compression is off by default ...
    Paul GarnerPaul Garner
    Nov 22, 2012 at 3:27 pm
    Nov 27, 2012 at 6:14 pm
  • Hi, Everyone, Thanks to people who contributed to the project, we have made significant progress in Kafka replication. Now, we'd like people to give it a try. Please check out revision 1411070 from ...
    Jun RaoJun Rao
    Nov 26, 2012 at 5:24 pm
    Nov 26, 2012 at 10:49 pm
  • Hello, We want to process messages from a single KafkaStream in a number of processes. Is it possible to have this code executing in multiple threads against the same stream? for (message <- stream) ...
    David RossDavid Ross
    Nov 21, 2012 at 5:41 am
    Nov 21, 2012 at 3:56 pm
  • Hi I am wondering that we do really need the following producer configuration val reconnectInterval = Utils.getInt(props, "reconnect.interval", 30000) /** negative reconnect time interval means ...
    Bae, Jae HyeonBae, Jae Hyeon
    Nov 18, 2012 at 11:37 pm
    Nov 19, 2012 at 3:46 pm
  • I am tinkering with the java async producer from kafka 7.2 and I am trying to write a client that will never block when sending messages to a kafka broker (dropping them would be better than ...
    Paul MacklesPaul Mackles
    Nov 15, 2012 at 8:55 pm
    Nov 16, 2012 at 12:29 am
  • Hello, We have a low-volume topic (~75msgs/sec) for which we would like to have a low propagation delay from producer to consumer. We have 3 brokers, each with a default of 4 partitions each. for a ...
    Bob CottonBob Cotton
    Nov 8, 2012 at 1:56 am
    Nov 8, 2012 at 10:30 pm
  • One more possible race might happen when the partition number is fixed but consumer(s) are added/removed For example: If I have a consumer reading data from two partitions (partition one and ...
    Guy PelegGuy Peleg
    Nov 1, 2012 at 7:11 am
    Nov 2, 2012 at 2:54 pm
  • Hi All, I am new to Kafka and we are planning to use Kafka as message broker. I am wondering if there are java API documentation for Kafka. I noticed that current API doc is written for Scala user. I ...
    Subhash AgrawalSubhash Agrawal
    Nov 29, 2012 at 11:18 pm
    Nov 30, 2012 at 9:19 pm
  • Hello, I am watching the video of your meetup, where Jun is going over the new .8 replica feature. Since the replica's for a master broker are in zookeeper, when there is a hardware crash it will ...
    S AhmedS Ahmed
    Nov 30, 2012 at 7:23 pm
    Nov 30, 2012 at 8:16 pm
  • Can someone go over how a consumer goes about reading from a broker? example: 1. connect to zookeeper, get information on the broker that corresponds to the topic/partition Then what does it do, does ...
    S AhmedS Ahmed
    Nov 28, 2012 at 3:14 pm
    Nov 29, 2012 at 6:15 pm
  • I just checked out the latest 0.8 branch, and ran ./sbt make-pom. I notice that some of the dependencies generated are either not released versions, e.g.: <dependency <groupId ...
    Jason RosenbergJason Rosenberg
    Nov 27, 2012 at 1:30 am
    Nov 28, 2012 at 12:45 am
  • Hi, all The code is like this: ConsumerConnector consumer = ...; Map<String, Integer topicCountMap = ...; Map<String, List<KafkaStream<Message consumerMap = null; KafkaStream<Message stream = null ...
    Sun liweiSun liwei
    Nov 22, 2012 at 5:04 pm
    Nov 26, 2012 at 5:18 am
  • Hi, I am new to using Kafka. I read all the documentations and followed the quickstart steps. I was able to run the sample kafka system. Looking through the Kafka directories extracted from the tar ...
    Jamie WangJamie Wang
    Nov 20, 2012 at 10:44 pm
    Nov 26, 2012 at 4:55 am
  • We graduated! -Jay ---------- Forwarded message ---------- From: Doug Cutting <<span class="m_body_email_addr" title="f81f6724508041482a3f001f17d74a91" cutting@apache.org</span Date: Wed, Nov 21, ...
    Jay KrepsJay Kreps
    Nov 21, 2012 at 10:15 pm
    Nov 23, 2012 at 8:15 pm
  • Hi, I would like to expose some of the kafka stats that appear in the current kafka jmx mbeans. In our system we are using the yammer metrics library (instead of polling jmx), so I'd like to wrap the ...
    Jason RosenbergJason Rosenberg
    Nov 20, 2012 at 10:05 pm
    Nov 21, 2012 at 6:14 pm
  • Hi In the case that I like to add more kafka brokers to distribute the traffic, I am seeing producer cannot recognize the newly created brokers. When I look at the code, the producer can update its ...
    Bae, Jae HyeonBae, Jae Hyeon
    Nov 2, 2012 at 10:15 pm
    Nov 8, 2012 at 5:57 pm
  • Hi, Foe the sender, I wrote a encoder for MessagePack. Why the consumer doesn't need the accompanying decoder? Thanks.
    Hussein BaghdadiHussein Baghdadi
    Nov 6, 2012 at 12:25 pm
    Nov 8, 2012 at 5:27 am
  • Hi, What I am trying to do is create a Consumer Java Service where you can send a request to the Java Service asking for say 15 messages for a topic, then if 15 messages were returned we assume there ...
    Andrew KellbergAndrew Kellberg
    Nov 1, 2012 at 1:48 am
    Nov 2, 2012 at 2:46 pm
  • Dear All, I am using async producer to publish event messages to a topic. Each message is about 2KB in size and I intend to publish about 1 million messages, i.e. about 2GB of message volumes. This ...
    Pankaj MisraPankaj Misra
    Nov 1, 2012 at 6:16 am
    Nov 1, 2012 at 3:33 pm
  • Hi, All We encounter an issue with Kafka of messages loss: We have Kafka 0.7.0 installed in cluster mode on three nodes(on partition per node configured). We are using kafka java API to send bulk of ...
    Boris koganBoris kogan
    Nov 29, 2012 at 1:11 pm
    Nov 29, 2012 at 6:14 pm
  • Hi everyone, I have a question regarding an issue related to OffsetOutOfRange exception. I have an app that consumes messages from a topic, this app uses node.js and node-kafka behind. Our data ...
    Radek GruchalskiRadek Gruchalski
    Nov 26, 2012 at 3:17 pm
    Nov 27, 2012 at 9:24 pm
  • Hi, I am seeing behaviour which I am not expecting when using topic filters. TopicFilter sourceTopicFilter = new Whitelist("pageviews"); List<KafkaStream<Message streams = ...
    Michal HarisMichal Haris
    Nov 20, 2012 at 12:45 pm
    Nov 21, 2012 at 7:04 pm
  • Hi! I am a zookeeper based consumer using the brod library (which is a python library for Kafka). I stop and start consumers using a simple shell script. I have noticed that when I kill and ...
    Varun VijayaraghavanVarun Vijayaraghavan
    Nov 14, 2012 at 4:52 pm
    Nov 19, 2012 at 7:35 pm
  • Hi folks, Do any of you have examples of unix init.d scripts that you use for starting and stopping Kafka? I am putting one together, but thought I'd ask if folks had any that might call out edge ...
    James A. RobinsonJames A. Robinson
    Nov 15, 2012 at 6:31 pm
    Nov 16, 2012 at 1:30 am
  • Suppose I'm iterating through messages from a topic with 6 messages. I consume message three messages and then commit (or an auto-commit happens). message 1 message 2 message 3 <--- commit message 4 ...
    David ArthurDavid Arthur
    Nov 8, 2012 at 2:23 pm
    Nov 8, 2012 at 4:33 pm
  • Hi, I have a situation where I want to reliably shut-down a consumer, but it seems to hang in certain conditions, namely: - If autocommit.enabled = true - If the current zk instance is not available ...
    Jason RosenbergJason Rosenberg
    Nov 5, 2012 at 7:51 pm
    Nov 5, 2012 at 8:39 pm
  • Hi, Is there any security enforcement built into Kafka broker that the system can have control who can send and consume messages? I understand kafka is most likely installed/deployed behind ...
    Jamie WangJamie Wang
    Nov 30, 2012 at 7:29 pm
    Nov 30, 2012 at 7:47 pm
Group Navigation
period‹ top | Nov 2012 | next ›
Group Overview
groupusers @
categorieskafka
discussions77
posts340
users65
websitekafka.apache.org
irc#kafka

65 users for November 2012

Jun Rao: 53 posts Neha Narkhede: 39 posts Jason Rosenberg: 35 posts Jay Kreps: 22 posts Joel Koshy: 16 posts David Arthur: 14 posts Chris Curtin: 12 posts Bae, Jae Hyeon: 9 posts Michal Haris: 9 posts S Ahmed: 8 posts Dmitri Priimak: 7 posts Paul Garner: 7 posts Bob Cotton: 6 posts Jamie Wang: 6 posts Mike Heffner: 6 posts James A. Robinson: 5 posts Tom Brown: 5 posts Evan Chan: 4 posts Mark Grabois: 4 posts Rohit Prasad: 4 posts
show more