FAQ

Search Discussions

49 discussions - 238 posts

  • I wish to use either the ConsumerConnector or the SimpleConsumer to read messages from all partitions across multiple brokers using a fixed number of threads (hopefully leveraging asynchronous IO for ...
    Tom BrownTom Brown
    Dec 19, 2012 at 6:49 pm
    Dec 23, 2012 at 7:54 pm
  • During my due diligence to assess use of Kafka for both our activity and log message streams, I would like to ask the project committers and community users about using Kafka with Node.js. Yes, I am ...
    Christopher AlexanderChristopher Alexander
    Dec 20, 2012 at 4:47 pm
    Dec 28, 2012 at 8:27 pm
  • Hi, So I followed the instructions from here: https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup So I pulled down the latest from github, ran; sbt open it up in idea, and builds fine ...
    S AhmedS Ahmed
    Dec 11, 2012 at 3:32 am
    Dec 12, 2012 at 10:58 pm
  • A few people have asked about getting a logo for Kafka. I think this would be a great idea! Oliver has kindly offered to have a someone at Datadog take a shot at it. If that doesn't pan out I am ...
    Jay KrepsJay Kreps
    Dec 19, 2012 at 6:03 pm
    Dec 19, 2012 at 11:19 pm
  • Hi, I am looking for a S3 based consumer, which can write all the received events to S3 bucket (say every minute). Something similar to Flume HDFSSink ...
    Pratyush ChandraPratyush Chandra
    Dec 27, 2012 at 9:42 am
    Dec 29, 2012 at 6:15 am
  • Hi all, In consumer we use a topic count map to set how many stream we want to split the topic. What's the relation between kafka partition number and consumer stream number? Seems stream number ...
    永辉 赵永辉 赵
    Dec 17, 2012 at 3:01 pm
    Mar 14, 2013 at 2:00 am
  • Hi, I run 'kafka-0.7.2' on a single node and today I got some error logs below. ... [2012-12-26 08:51:18,496] INFO End registering broker topic /brokers/topics/jnits/1 (kafka.server.KafkaZooKeeper) ...
    XingcanXingcan
    Dec 27, 2012 at 1:42 am
    Dec 28, 2012 at 10:32 pm
  • Hi, I'm trying to promote Kafka for our centralized log aggregation/metrics system and have set up a proof of concept based on 0.7.2 which seems to work very well for our purposes but the ...
    Johan LundahlJohan Lundahl
    Dec 14, 2012 at 12:43 pm
    Dec 19, 2012 at 5:58 pm
  • Could you share what changed in zkclient-20120522.jar? I found that watchers were canceled when the zookeeper connection was interrupted and reconnected with another application. If the new ...
    Bae, Jae HyeonBae, Jae Hyeon
    Dec 5, 2012 at 6:27 pm
    Dec 5, 2012 at 7:33 pm
  • Hello All, I am in the early stages of exploring the use of Kafka for large-scale application I am authoring. Generally, the documentation has been pretty good and quite pleased with getting things ...
    Christopher AlexanderChristopher Alexander
    Dec 19, 2012 at 4:00 pm
    Dec 20, 2012 at 4:48 pm
  • Anyone have a python/avro producer that slurps up records from a flat file (i.e., mix of string and binary data) and publishes to Kafka they would be willing to share? Starting to think this might be ...
    Joseph CrottyJoseph Crotty
    Dec 19, 2012 at 3:38 am
    Dec 19, 2012 at 4:36 pm
  • Hello, Is it possible to fetch messages from the Kafka message queue since a specific time? For example, a user may subscribe to a topic and the producer will continuously publish messages related to ...
    Jason HuangJason Huang
    Dec 17, 2012 at 6:23 pm
    Dec 17, 2012 at 9:38 pm
  • Hi, We are incorporating Kafka as part of our centralized log aggregation service for a clustered server system. Our current thinking on the design as follows: Each of our clustered server will ...
    Jamie WangJamie Wang
    Dec 12, 2012 at 10:26 pm
    Dec 14, 2012 at 6:13 am
  • Hello, We have a strange issue on one of our systems that we just can't seem to figure out. We loaded in 5000 records yesterday and have not loaded any since then into our brokers. Examining the logs ...
    Sybrandy, CaseySybrandy, Casey
    Dec 12, 2012 at 5:52 pm
    Dec 14, 2012 at 6:10 am
  • Hi, I read the literature somewhere that the windows platform is supported for running kafka. But I didn't find any *.bat command files in the 0.72 release I downloaded. I searched the net and found ...
    Jamie WangJamie Wang
    Dec 5, 2012 at 12:59 am
    Dec 5, 2012 at 9:45 pm
  • http://kafka.apache.org/ I will set up redirects from the old to the new. If folks could sanity check the links, email addresses, english, and so on that would be swell. -Jay
    Jay KrepsJay Kreps
    Dec 14, 2012 at 9:30 pm
    Dec 16, 2012 at 6:28 pm
  • Hello, How production-ready and how GA-compatible is 0.8 HEAD now? Context: We're starting to use Kafka in a few projects at Sematext (one of them is in my sig below) and are using Kafka 0.7.2. One ...
    Otis GospodneticOtis Gospodnetic
    Dec 8, 2012 at 1:50 pm
    Dec 10, 2012 at 10:26 pm
  • Is it possible to log/see which broker, and perhaps partition, a producer sent a message to? I'm using the SyncProducer if that matters.
    Sybrandy, CaseySybrandy, Casey
    Dec 10, 2012 at 3:26 pm
    Dec 10, 2012 at 6:09 pm
  • Hi All, I am new to Kafka broker and realized that Kafka broker does not enforce client authentication at connection or message level. To avoid DOS attack, we are planning to implement security ...
    Subhash AgrawalSubhash Agrawal
    Dec 7, 2012 at 9:53 pm
    Dec 10, 2012 at 3:31 pm
  • Curious what kind of uptime have you guys experienced using kafka? What sort of monitoring do you suggest should be in place for kafka? If the service crashes, does it usually make sense to have ...
    S AhmedS Ahmed
    Dec 28, 2012 at 3:28 pm
    Jan 30, 2013 at 4:56 am
  • Hi , I am reading the design doc for Kafka. But I can not find the algorithm which illustrates such scenario. When a new broker is registered in zookeeper, how do all the brokers negotiates with all ...
    王国栋王国栋
    Dec 28, 2012 at 2:31 am
    Dec 30, 2012 at 5:45 pm
  • Hi, I am new to Kafka. I am exploring ways to pump events from http browser(using javascript) or over tcp (say using node js) to broker. Currently I see, only scala based producer in source code ...
    Pratyush ChandraPratyush Chandra
    Dec 20, 2012 at 9:16 am
    Dec 21, 2012 at 6:07 pm
  • Hi, I have some questions about the batch size for producer. How big should be the batch size? What happens if producer client crashes before batch is full and messages were still sitting in producer ...
    Subhash AgrawalSubhash Agrawal
    Dec 20, 2012 at 10:47 pm
    Dec 21, 2012 at 6:22 am
  • Hi All, I've scanned the code a little (0.7.2) but couldn't find a parameter to force the broker to bind to a specific network interface rather than listen on all. Was this added to 0.8? I was hoping ...
    Matan AmirMatan Amir
    Dec 13, 2012 at 7:51 pm
    Dec 17, 2012 at 11:55 pm
  • Any objections to my deleting the old github kafka repository. We kept it around for posterity and to preserve the version control history. But I found that it confuses people as to where we are ...
    Jay KrepsJay Kreps
    Dec 17, 2012 at 7:05 pm
    Dec 17, 2012 at 8:02 pm
  • Hello, I am trying to distribute work across several nodes using Kafka. I have 3 brokers each with 16 partitions. I have 8 worker servers listening with one message stream on the same topic. I expect ...
    David RossDavid Ross
    Dec 13, 2012 at 7:18 am
    Dec 14, 2012 at 6:14 am
  • Hi guys: Just checked out the 0.8 branch: https://svn.apache.org/repos/asf/kafka/branches/0.8 and ./sbt update does not work. (I have a machine with no previous installations of scala), I am ...
    John WangJohn Wang
    Dec 6, 2012 at 9:24 pm
    Dec 13, 2012 at 3:44 pm
  • I was reading (or watching) how linkedin uses kafka to track page views. I'm trying to imagine this in practise, where linkedin probably has 100's of web servers serving requests, and each server is ...
    S AhmedS Ahmed
    Dec 9, 2012 at 8:48 pm
    Dec 10, 2012 at 7:15 pm
  • Hi, I am searching for a Kafka Producer that is able to write to a Kafka Cluster with replication, so that there is no data lost when one broker is going down. This functionality has been added in ...
    Tobias FeldhausTobias Feldhaus
    Dec 19, 2012 at 10:03 am
    Dec 20, 2012 at 7:00 am
  • Hi all, I find SenseDB uses kafka 0.7.6,but I didn't find this branch or tag in github. There are only 0.8, 0.7.2, 0.7.1 and 0.7 branches. Thanks, Yonghui
    永辉 赵永辉 赵
    Dec 17, 2012 at 3:00 pm
    Dec 17, 2012 at 4:38 pm
  • Hello, A coworker is having an issue where he's writing a plugin for ElasticSearch and is using the Kafka Log4J appender to log errors to a topic. However, if for some reason the broker is missing, ...
    Sybrandy, CaseySybrandy, Casey
    Dec 11, 2012 at 2:59 pm
    Dec 11, 2012 at 6:01 pm
  • Hi, I am trying kafka for one of POC. It seems to working well, but I'm having a small doubt about distribution of messages among consumers in same consumer group. Below is my setup details. 1. Kafka ...
    Jain RahulJain Rahul
    Dec 11, 2012 at 10:20 am
    Dec 11, 2012 at 5:32 pm
  • Hi , One of my colleagues wants to subscribe to this email group. But he can not send subscription email to <span class="m_body_email_addr" title="4f118bb1bc75eda000f73ffcf4d159ce" ...
    王国栋王国栋
    Dec 5, 2012 at 2:24 am
    Dec 5, 2012 at 5:25 am
  • Since kafka is written in scala, when using a java client (producer or consumer), do you pay a slight performance hit because you guys are internally mapping back/forth to map a java type to a scala ...
    S AhmedS Ahmed
    Dec 29, 2012 at 3:05 am
    Jan 1, 2013 at 1:21 am
  • Hi, Everyone, Just want to let people know that there will be a Kafka presentation (focusing on replication) at ApacheCon in Feb 2013. http://na.apachecon.com/schedule/presentation/115/ We also plan ...
    Jun RaoJun Rao
    Dec 29, 2012 at 12:51 am
    Dec 29, 2012 at 8:53 am
  • Hi! I'm playing with kafka 0.7.2 using the following setup: 3 zk nodes ensemble 2 brokers: * num_partitions:3 * topic.partition.count.map=test-topic:5 My producer connects to brokers using ...
    Samuel García MartínezSamuel García Martínez
    Dec 28, 2012 at 12:29 pm
    Dec 28, 2012 at 10:38 pm
  • Hello, I am writing a simple java client that uses SimpleConsumer to fetch messages from a Kafka Server: .................. KafkaConnectionParams connection = new KafkaConnectionParams() ...
    Jason HuangJason Huang
    Dec 21, 2012 at 3:59 pm
    Dec 21, 2012 at 4:26 pm
  • Hi, Some time ago people on this list asked if we could add Kafka to search-hadoop.com and we just did that yesterday. You can see Kafka and all other projects there if you go to ...
    Otis GospodneticOtis Gospodnetic
    Dec 18, 2012 at 6:57 pm
    Dec 18, 2012 at 7:24 pm
  • I was hoping someone could give a birds eye view of how zookeeper interacts with kafka (server, producer, client). Once everything is running, how often do the various processes communicate with ...
    S AhmedS Ahmed
    Dec 17, 2012 at 6:49 pm
    Dec 17, 2012 at 7:05 pm
  • Hi, I've read about Kafka being used for huge paces and volumes of messages. My scenario is a bit different, I'm looking for mostly only persistence of a message queue, where the queue is usually not ...
    Matan SafrielMatan Safriel
    Dec 16, 2012 at 10:24 am
    Dec 16, 2012 at 4:28 pm
  • Hi When one of the broker instances is dead, I can see the producer can acknowledge the dead broker and refresh its producer pool not to send the data to the dead broker. But I don't find any code in ...
    Bae, Jae HyeonBae, Jae Hyeon
    Dec 15, 2012 at 6:21 am
    Dec 16, 2012 at 1:48 am
  • Hi All, I just noticed that consumer need to have zookeeper URL. It does not use broker.list property. Can somebody please confirm if this is the case. Thanks Subhash A.
    Subhash AgrawalSubhash Agrawal
    Dec 13, 2012 at 11:43 pm
    Dec 14, 2012 at 6:17 am
  • Hello, I am interested in learning more about Kafka. I just sent a subscription request to <span class="m_body_email_addr" title="4f118bb1bc75eda000f73ffcf4d159ce" ...
    Jason HuangJason Huang
    Dec 13, 2012 at 8:56 pm
    Dec 13, 2012 at 9:11 pm
  • I'm not a infra guy, and was wondering if anyone has written a upstart script to start/stop kafka, and have it run on server reboots etc.
    S AhmedS Ahmed
    Dec 12, 2012 at 8:40 pm
    Dec 13, 2012 at 6:40 am
  • kafka.common.InvalidMessageSizeException: invalid message size: 1717662313 only received bytes: 835 at 395( possible causes (1) a single message larger than the fetch size; (2) log corruption ) I am ...
    Corbin HoenesCorbin Hoenes
    Dec 12, 2012 at 4:53 am
    Dec 12, 2012 at 5:33 pm
  • Hi! I'm playing with kafka using the following setup: 3 zk nodes ensemble 2 brokers: * num_partitions:3 * topic.partition.count.map=test-topic:5 My producer connects to brokers using zk.connect. When ...
    Samuel García MartínezSamuel García Martínez
    Dec 28, 2012 at 4:26 pm
    Dec 28, 2012 at 4:26 pm
  • Hi, Everyone, We have moved Kafka repo to git. The location of the new repo is at: https://git-wip-us.apache.org/repos/asf/kafka.git Please follow the following instruction on how to use it ...
    Jun RaoJun Rao
    Dec 14, 2012 at 6:43 pm
    Dec 14, 2012 at 6:43 pm
  • Okay I have fixed all the problems people found that I am aware of: 1. Correlation id used throughout 2. Brokers in metadata api not repeated per-partition 3. Removed creator id from broker because ...
    Jay KrepsJay Kreps
    Dec 7, 2012 at 6:33 am
    Dec 7, 2012 at 6:33 am
  • Thanks -- Prasanth
    Prasanth JPrasanth J
    Dec 3, 2012 at 3:24 pm
    Dec 3, 2012 at 3:24 pm
Group Navigation
period‹ prev | Dec 2012 | next ›
Group Overview
groupusers @
categorieskafka
discussions49
posts238
users49
websitekafka.apache.org
irc#kafka

49 users for December 2012

Jun Rao: 37 posts Jay Kreps: 24 posts Neha Narkhede: 16 posts David Arthur: 13 posts S Ahmed: 12 posts Sybrandy, Casey: 12 posts 永辉 赵: 9 posts Ben fleis: 7 posts Christopher Alexander: 6 posts Jamie Wang: 6 posts Jason Huang: 6 posts Subhash Agrawal: 6 posts Joel Koshy: 5 posts Pratyush Chandra: 5 posts Tom Brown: 5 posts Bae, Jae Hyeon: 4 posts Johan Lundahl: 4 posts Radek Gruchalski: 4 posts 王国栋: 4 posts Apoorva Gaurav: 3 posts
show more