FAQ

Kafka > users > February 2016

Search Discussions

165 discussions - 532 posts

  • Hey guys, first post here so bare with me Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here http://kafka.apache.org/documentation.html#security_sasl and i seem to be very close, ...
    Oleg ZhurakouskyOleg Zhurakousky
    Feb 23, 2016 at 7:02 pm
    Feb 24, 2016 at 9:01 pm
  • This is the first candidate for release of Apache Kafka 0.9.0.1. This a bug fix release that fixes 70 issues. Release Notes for the 0.9.0.1 release ...
    Jun RaoJun Rao
    Feb 12, 2016 at 2:55 am
    Feb 17, 2016 at 8:07 pm
  • Hi, We were using kafka for a while now. We have been using the binary release 2.10-0.8.2.1 . But we have been needing a encrypted communication between our publishers and subscribers. So we got ...
    Nazario ParsacalaNazario Parsacala
    Feb 1, 2016 at 4:52 pm
    Feb 1, 2016 at 8:09 pm
  • I am writing a Kafka consumer client using the document at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol One place where I am having problems is the fetch request ...
    Rajiv KurianRajiv Kurian
    Feb 4, 2016 at 11:21 pm
    Feb 5, 2016 at 7:38 pm
  • Hi, I use kafka.tools.ConsumerOffsetChecker to view the consumer offset status, and can get the correct output, along with a warning: ConsumerOffsetChecker is deprecated and will be dropped in ...
    AmoxicillinAmoxicillin
    Feb 19, 2016 at 5:35 am
    Feb 19, 2016 at 9:48 am
  • Hello, Is it viable to use infinite-retention Kafka topics as a master data store? I'm not talking massive volumes of data here, but still potentially extending into tens of terabytes. Are there any ...
    Ted SwerveTed Swerve
    Feb 15, 2016 at 5:08 pm
    Feb 17, 2016 at 6:34 pm
  • Using kafka 0.9.0.0, if there are multiple consumers in a group and one consumer pauses the topic+partition it's consuming, does that allow/cause other consumers in the group to pick it up?
    Craig wCraig w
    Feb 22, 2016 at 3:28 pm
    Feb 24, 2016 at 6:03 pm
  • When publishing messages to Kafka, you make a choice between at-most-once and at-least-once delivery, depending on whether you wait for acknowledgments and whether you retry on failures. In most ...
    Andrew SchofieldAndrew Schofield
    Feb 19, 2016 at 9:54 pm
    Feb 23, 2016 at 12:06 am
  • Any ideas as to which property should I set to enable Zookeeper re-connection? I have the following properties defined for my consumer (High Level Consumer API). Is this enough for a automatic ...
    Joe SanJoe San
    Feb 15, 2016 at 8:22 pm
    Feb 18, 2016 at 6:39 pm
  • Should your number of brokers be atleast one more then your replication factor of your topic(s)? So if I have a replication factor of 2, I need atleast 3 brokers? Thanks, Sean
    Sean Morris (semorris)Sean Morris (semorris)
    Feb 16, 2016 at 2:08 pm
    Feb 17, 2016 at 7:42 pm
  • Good Afternoon Gentlemen I am new to Kafka and am attempting to build Kafka ..so I downloaded latest kafka-0.9.0.0-src.tgz distro from http://kafka.apache.org/downloads.html and then built it with ...
    Martin GaintyMartin Gainty
    Feb 5, 2016 at 6:28 pm
    Feb 11, 2016 at 3:21 pm
  • Is this intentioal in the Kafka 0.8.2.0 version, org.apache.kafka.clients.consumer.KafkaConsumer, the method: public Map<String, ConsumerRecords<K, V poll(long timeout) { return null; } Which version ...
    Joe SanJoe San
    Feb 9, 2016 at 4:00 pm
    Feb 10, 2016 at 4:57 am
  • Hi team, What is the best way to verify a specific Kafka node functions properly? Telnet the port is one of the approach but I don't think it tells me whether or not the broker can still receive/send ...
    Tao xiaoTao xiao
    Feb 26, 2016 at 11:38 am
    Mar 5, 2016 at 11:12 pm
  • *Use Case: Disaster Recovery & Re-indexing SOLR* I'm using Kafka to hold messages from a service that prepares "documents" for SOLR. A second micro service (a consumer) requests these messages, does ...
    John BickerstaffJohn Bickerstaff
    Feb 17, 2016 at 7:16 pm
    Feb 23, 2016 at 11:21 pm
  • Hi, I'm trying to use the new Consumer API with this example : https://github.com/apache/kafka/tree/trunk/examples/src/main/java/kafka/examples With a Producer I sent 1000 messages to my Kafka ...
    Péricé RobinPéricé Robin
    Feb 17, 2016 at 5:54 pm
    Feb 22, 2016 at 2:45 pm
  • Hello all, I have a question about Kafka Streams, which I’m evaluating for a new project. (I know it’s still a work in progress but it might work anyway for this project.) I’m new to Kafka in ...
    Avi FlaxAvi Flax
    Feb 18, 2016 at 9:00 pm
    Feb 19, 2016 at 8:54 pm
  • Hi, We are getting the below error when trying to use a Java new producer client. Please let us know the reason for this error - Error message: [2016-02-18 15:41:06,182] DEBUG Accepted connection ...
    Srikrishna AllaSrikrishna Alla
    Feb 18, 2016 at 10:55 pm
    Feb 19, 2016 at 4:44 pm
  • Hello All, New Kafka user here. What is the best way to write Kafka data into HDFS? I have looked into following options and found that Flume is quickest and easiest to setup. 1. Flume 2. KaBoom 3 ...
    R PR P
    Feb 11, 2016 at 1:09 am
    Feb 11, 2016 at 10:56 pm
  • Hi, Everyone, We have fixed a few critical bugs since 0.9.0.0 was released and are still investigating a few more issues. The current list of issues tracked for 0.9.0.1 can be found below. Among ...
    Jun RaoJun Rao
    Feb 5, 2016 at 6:28 pm
    Feb 8, 2016 at 9:05 am
  • Is there a way to detect the broker version (even at a high level 0.8 vs 0.9) using the kafka-clients Java library? -- Best regards, Marko www.kafkatool.com
    MarkoMarko
    Feb 4, 2016 at 11:01 pm
    Feb 6, 2016 at 4:09 pm
  • Hi, We've just updated to the 0.9 client & broker, and we're suddenly seeing a lot of log spam in the consumers: 2016-02-05T03:31:34,182 | INFO | o.a.k.c.c.i.AbstractCoordinator [kafkaspout-thread-0] ...
    Simon CooperSimon Cooper
    Feb 5, 2016 at 10:02 am
    Feb 5, 2016 at 7:15 pm
  • I am trying to use kafka-streams (I know it isn’t released yet but seems to be fairly far on in its implementation and I would like to use that rather than samza for this project as my project ...
    Tom DearmanTom Dearman
    Feb 3, 2016 at 3:34 pm
    Feb 4, 2016 at 4:25 pm
  • I am using the Java Kafka 0.9 client. When I subscribe to a topic I provide a ConsumerRebalanceListener. In the "onPartitionsAssigned" method I am doing this: partitions.foreach( (tp: TopicPartition) ...
    Craig wCraig w
    Feb 24, 2016 at 3:10 pm
    Feb 24, 2016 at 6:53 pm
  • Hello all. I'm developing a kafka client and have a question about kafka server guarantees. A statement from ...
    Ivan DyachkovIvan Dyachkov
    Feb 17, 2016 at 6:55 pm
    Feb 18, 2016 at 7:42 pm
  • Dear Kafka users, I'm looking for some case studies around using Kafka on big projects. Specifically, I'm looking for some architectural insights into how I could orchestrate my data pipeline using ...
    Joe SanJoe San
    Feb 3, 2016 at 3:15 pm
    Feb 3, 2016 at 8:10 pm
  • Hi, I've been noticing that a restarted consumer in 0.9 will start consuming from the last committed offset (inclusive). This means that any restarted consumer will get the last read (and committed) ...
    Adam KunickiAdam Kunicki
    Feb 1, 2016 at 9:35 pm
    Feb 2, 2016 at 6:29 pm
  • Hi, One of our usage of kafka is to tolerate arbitrary consumer crash without losing or duplicating messages. So in our code we manually commit offset after successfully persisted the consumer ...
    Han JUHan JU
    Feb 1, 2016 at 5:59 pm
    Feb 2, 2016 at 5:34 pm
  • Hello, I am having the following problem trying to delete a topic. The topic was auto-created with a default.replication.factor = 1, but my test cluster has only 1 machine, so now when I start kafka ...
    Franco GiacosaFranco Giacosa
    Feb 23, 2016 at 10:28 am
    Feb 24, 2016 at 6:31 pm
  • Hi I'm the technical co-founder of a startup in San Francisco and I've been investigating using Kafka for some time. I'm thinking about biting the bullet and integrating it soon into our system. My ...
    Kyle MathewsKyle Mathews
    Feb 15, 2016 at 12:07 am
    Feb 15, 2016 at 8:33 am
  • Hi all! Right now I am working on reactive streams connector to kafka. I am using new client and found strange behavior of commitAsync method which not calling callbacks at all at some cases. I ...
    Alexey RomanchukAlexey Romanchuk
    Feb 5, 2016 at 2:36 pm
    Feb 8, 2016 at 3:38 am
  • I'm running kafka_2.11-0.9.0.0 and a java-based producer/consumer. With messages ~70 KB everything works fine. However, after the producer enqueues a larger, 70 MB message, kafka appears to stop ...
    Tech BolekTech Bolek
    Feb 3, 2016 at 2:04 am
    Feb 5, 2016 at 9:08 am
  • Hi All, I see that the —num.producers is removed from the Kafka 0.9 MirrorMaker. Why is this? How can we create multiple producer threads for publishing in MirrorMaker? Thanks, Tushar
    Mhaskar, TusharMhaskar, Tushar
    Feb 2, 2016 at 7:49 pm
    Feb 5, 2016 at 4:35 am
  • Hi, I am trying to navigate through the protocol and I am seeing some inconsistencies with the data. I am trying to parse out the MetadataResponse and I am seeing bytes in between where they ...
    Heath IvieHeath Ivie
    Feb 4, 2016 at 2:01 am
    Feb 4, 2016 at 5:11 pm
  • Hi, I ran into a scenario where one of my brokers would continually shutdown, with the error message: [2016-02-25 00:29:39,236] FATAL [ReplicaFetcherThread-0-1], Halting because log truncation is not ...
    James ChengJames Cheng
    Feb 25, 2016 at 9:47 pm
    Mar 16, 2016 at 6:07 am
  • Hello, I have an issue with the timing of Kafka. As and when the time increases with load testing, we see the increase in "request queue time". What is the "request queue time" ? Some basic config we ...
    AwagleAwagle
    Feb 29, 2016 at 10:30 pm
    Mar 1, 2016 at 10:14 pm
  • hi all, i'm having trouble processing a topic which includes paths to images which need to be downloaded and saved to disk (each takes ~3-5 seconds) and several are received on each poll within this ...
    Guven DemirGuven Demir
    Feb 25, 2016 at 4:54 pm
    Feb 26, 2016 at 10:57 pm
  • Hello, I've been experimenting with mirror maker and am a bit confused about some behavior I'm seeing. I have two simple single broker clusters setup locally, source and target. The source cluster ...
    Luke SteensenLuke Steensen
    Feb 25, 2016 at 12:17 am
    Feb 25, 2016 at 4:02 pm
  • Hello, We are using 0.8.2 kafka version. We are running 5 brokers in the prod cluster. Each topic is having two partitions. We are seeing some issues with some topic partitions. For some topic ...
    Raju BairishettiRaju Bairishetti
    Feb 22, 2016 at 3:44 am
    Feb 25, 2016 at 3:48 am
  • I am running into a curious problem with auto creation of topics and would appreciate any insight into what could be causing it. Some context first: We are using Confluent 2.0 platform and I have an ...
    Venkatesan SubramanianVenkatesan Subramanian
    Feb 19, 2016 at 6:43 pm
    Feb 23, 2016 at 1:36 am
  • Hi, I'm desperately trying to run Kafka 0.9.0.0 on Solaris 5.10 (Kafka 0.8.2 is working fine). I use 1 Broker with default configuration just as in the quickstart guide. The Broker is not working and ...
    Lukas BanduchLukas Banduch
    Feb 19, 2016 at 12:11 pm
    Feb 22, 2016 at 9:15 pm
  • Hi Everyone, I am encountering this exception similar to Saurabh's report earlier today as I try to scale up a Storm - Kafka output via the KafkaBolt (i.e., add more KafkaBolt executors) ...
    John YostJohn Yost
    Feb 18, 2016 at 1:18 pm
    Feb 22, 2016 at 6:21 pm
  • My company is waiting for the new consumer to move out of "beta" mode before using it. Does anyone know if it is still considered beta? -- Allen Michael Chan
    Allen chanAllen chan
    Feb 19, 2016 at 9:06 pm
    Feb 19, 2016 at 9:46 pm
  • Hi! The Kafka 0.9 protocol guide https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression explicitly states that there should be only ...
    Oleksiy KrivosheyOleksiy Krivoshey
    Feb 16, 2016 at 11:33 am
    Feb 17, 2016 at 6:39 pm
  • Hello! I'm trying to use kafka for long-running tasks processing. The tasks can be very short (less than a second) or very long (about 10 minutes). I've got one consumer group for the single queue, ...
    Насыров РенатНасыров Ренат
    Feb 16, 2016 at 3:15 pm
    Feb 16, 2016 at 5:35 pm
  • Hi, I'm having trouble with some recurring stalling connections to kafka. What I see as a symptom is that some consumers lag behind and most times restarting the consumer doesn't help. (occasionally ...
    Maximiliano Patricio MéndezMaximiliano Patricio Méndez
    Feb 10, 2016 at 5:39 pm
    Feb 15, 2016 at 5:04 pm
  • How could I get rid of this warning? log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties). log4j:WARN Please initialize the log4j system properly. Any ideas how to get ...
    Joe SanJoe San
    Feb 12, 2016 at 7:33 am
    Feb 12, 2016 at 11:58 am
  • Hi Raja, We seem to be encountering the same problem you were seeing where our producer thread becomes blocked for some reason. We also see our producer queue is full, and for some reason, the ...
    Yu, DebbieYu, Debbie
    Feb 6, 2016 at 3:17 am
    Feb 8, 2016 at 11:34 pm
  • Hi, In order to get metrics through JMX, we have exported JMX_PORT=8099. But when we are trying to delete the topics from kafka, we are getting following issue: Error: Exception thrown by the ...
    Shishir ShivhareShishir Shivhare
    Feb 8, 2016 at 4:46 pm
    Feb 8, 2016 at 5:14 pm
  • Kafka users, The code below is something that I have to write to a Topic! def publishMessage(tsDataPoints: Seq[DataPoint]): Future[Unit] = { Future { logger.info(s"Persisting ${tsDataPoints.length} ...
    Joe SanJoe San
    Feb 4, 2016 at 6:11 am
    Feb 4, 2016 at 6:42 am
  • We have shut down some nodes from our cluster yesterday and now we are seeing tons of these in the log, 2016-02-24 18:11:23 INFO SyncProducer:68 - Connected to ...
    Kim ChewKim Chew
    Feb 24, 2016 at 6:25 pm
    Feb 29, 2016 at 6:09 pm
Feb 2016
1234
Group Navigation
period‹ prev | Feb 2016 | next ›
Group Overview
groupusers @
categorieskafka
discussions165
posts532
users182
websitekafka.apache.org
irc#kafka

182 users for February 2016

Joe San: 27 posts Jason Gustafson: 26 posts Ismael Juma: 16 posts Ewen Cheslack-Postava: 14 posts Tao xiao: 12 posts Alex Loddengaard: 11 posts Rajiv Kurian: 11 posts Christian Posta: 10 posts Craig w: 10 posts Martin Gainty: 10 posts Oleg Zhurakousky: 10 posts Alexis Midon: 9 posts Ben Stopford: 9 posts Jens Rantil: 9 posts Nazario Parsacala: 9 posts Adam Kunicki: 8 posts Avi Flax: 8 posts Franco Giacosa: 8 posts Guozhang Wang: 8 posts Jun Rao: 7 posts
show more
Archives