FAQ

Search Discussions

138 discussions - 527 posts

  • Hi, I'm prototyping with the new consumer API of kafka 0.9 and I'm particularly interested in the `ConsumerRebalanceListener`. My test setup is like the following: - 5M messages pre-loaded in one ...
    Han JUHan JU
    Jan 22, 2016 at 3:13 pm
    Jan 27, 2016 at 1:19 am
  • KAFKA-3006 is under review, and would change some commonly used signatures in the Kafka client library. The idea behind the proposal is to provide a unified way of interacting with anything sequence ...
    Pierre-Yves RitschardPierre-Yves Ritschard
    Jan 26, 2016 at 9:00 am
    Mar 8, 2016 at 1:33 am
  • Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0. Here's a runnable gist illustrating the problem: https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires Kafka on ...
    Krzysztof CiesielskiKrzysztof Ciesielski
    Jan 21, 2016 at 6:46 pm
    Jan 28, 2016 at 2:17 am
  • Hi Kafka team, I want to know if I can plug-in my own security protocol to Kafka to implement project specific authentication mechanism. The current supported authentication protocols, SASL/GSSAPI ...
    Tao xiaoTao xiao
    Jan 18, 2016 at 3:16 am
    Jan 22, 2016 at 1:39 am
  • Hello. I set up the Kafka testbed environment on my VirtualBox, which simply has a Kafka broker. I tested the simple consumer & producer scripts, aka kafka-console-consumer.sh and ...
    BYEONG-GI KIMBYEONG-GI KIM
    Jan 20, 2016 at 7:57 am
    Jan 21, 2016 at 3:54 am
  • Hi, I'm trying to make sure I understand this statement in the docs: "Each broker partition is consumed by a single consumer within a given consumer group. The consumer must establish its ownership ...
    Jason J. W. WilliamsJason J. W. Williams
    Jan 16, 2016 at 1:34 am
    Jan 19, 2016 at 11:00 pm
  • Hi team, I have a scenario where I want to write new offset for a list of topics on demand. The list of topics is unknown until runtime and the interval between each commit is undetermined. what ...
    Tao xiaoTao xiao
    Jan 4, 2016 at 11:19 am
    Jan 6, 2016 at 9:10 pm
  • Hello, We've run into a bit of a head-scratcher with a new kafka deployment and I'm curious if anyone has any ideas. A little bit of background: this deployment uses "immutable infrastructure" on ...
    Luke SteensenLuke Steensen
    Jan 12, 2016 at 10:50 pm
    Jan 25, 2016 at 10:30 pm
  • I was trying to upgrade an 0.8.2.1 broker cluster to 0.9.0.0 by following the instructions here: http://kafka.apache.org/documentation.html#upgrade After upgrading one broker, with ...
    Dave PetersonDave Peterson
    Jan 15, 2016 at 2:04 am
    Jan 25, 2016 at 4:42 am
  • Hi , I am trying to run MirrorMaker from 0.9broker as source and 0.8.2.1 broker as destination. MirrorMaker is run from source(I.e. Using 0.9’s code) I am getting the following error message after it ...
    Mhaskar, TusharMhaskar, Tushar
    Jan 19, 2016 at 11:13 pm
    Jan 21, 2016 at 2:16 pm
  • dear all, I found a guide in Kafka protocol (Kafka Protocol Guide <https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetRequest ), the guide ...
    Muqtafi AkhmadMuqtafi Akhmad
    Jan 18, 2016 at 12:07 pm
    Jan 20, 2016 at 4:19 pm
  • Hi there, Kafka's `kafka-reassign-partitions.sh` tool currently has no mechanism for removing brokers. However, it does have the ability to generate partition plans across arbitrary sets of brokers, ...
    Tom CrayfordTom Crayford
    Jan 6, 2016 at 3:15 pm
    Jan 14, 2016 at 5:23 pm
  • Hi! I am testing Kafka's performance with large messages and would like to specify maximum request size when I run kafka-producer-perf-test.sh: ./kafka-producer-perf-test.sh --broker-list ...
    Andrej VladimirovichAndrej Vladimirovich
    Jan 8, 2016 at 4:04 pm
    Jan 11, 2016 at 7:36 pm
  • Hi , We are trying to write a Kafka-connect connector for Mongodb. The issue is, MongoDB does not provide an entire changed document for update operations, It just provides the modified fields. if ...
    Sunny ShahSunny Shah
    Jan 27, 2016 at 6:21 am
    Feb 6, 2016 at 10:37 am
  • I’m creating a custom Kafka Connect source connector, and I’m running into a situation for which Kafka Connect doesn’t seem to provide a solution out of the box. I thought I’d first post to the users ...
    Randall HauchRandall Hauch
    Jan 28, 2016 at 6:11 am
    Jan 29, 2016 at 10:57 pm
  • We are trying to deploy kafka into EC2 and one of the requirement from infosec is to have kafka encryption at rest (stored with encrypted value). We also need to be able to rotate encryption keys and ...
    Josh WoJosh Wo
    Jan 19, 2016 at 11:34 pm
    Jan 24, 2016 at 10:16 am
  • Kafka Users, How can I create a kafka topic programatically? I would like to create the topics when I initialize my application. It should also be in such a way that if the topic already exists, the ...
    Joe SanJoe San
    Jan 20, 2016 at 12:19 pm
    Jan 20, 2016 at 7:12 pm
  • Hi! Pondering how to configure Kafka clusters and avoid having too many machines to manage.. Would it be recommended to run say a 3 node kafka cluster where you also run your 3 node zookeeper cluster ...
    Erik ForsbergErik Forsberg
    Jan 14, 2016 at 11:06 am
    Jan 18, 2016 at 10:08 pm
  • we're using kafka-python, weighing pykafka, and wondering if there's another that is bettor to use. does confluent endorse or recommend a particular python package (psorry for the alliteration)? doug
    Doug TommDoug Tomm
    Jan 9, 2016 at 12:32 am
    Jan 12, 2016 at 4:02 pm
  • I might have sent this recently, but was not able to receive mail from this list (fixed) --- Hello- We have a use case where we're trying to create a topic, delete, then recreate with the same topic ...
    Brenden CobbBrenden Cobb
    Jan 5, 2016 at 3:28 am
    Jan 5, 2016 at 8:47 pm
  • We are using the new kafka consumer with the following config (as logged by kafka) metric.reporters = [] metadata.max.age.ms = 300000 value.deserializer = class ...
    Rajiv KurianRajiv Kurian
    Jan 26, 2016 at 5:20 am
    Jan 27, 2016 at 6:26 pm
  • Hi All, I am facing issues deleting config for a topic in Kafka 0.9. Anybody facing the same issue? ./kafka-topics.sh —zookeeper XX.XX.XX.XXX:2181 --alter --topic testTopic --deleteConfig ...
    Mhaskar, TusharMhaskar, Tushar
    Jan 13, 2016 at 6:52 pm
    Jan 13, 2016 at 9:36 pm
  • Hi all, Is KAFKA-725 Broker Exception: Attempt to read with a maximum offset less than start offset <https://issues.apache.org/jira/browse/KAFKA-725 still valid ? We are seeing a similar issue when ...
    Manu ZhangManu Zhang
    Jan 19, 2016 at 12:41 pm
    Jan 28, 2016 at 8:39 am
  • Hi I have a straightforward piece of code that creates a consumer (Kafka 0.9.0.0). Properties props = new Properties(); props.put("zookeeper.connect", zookeeperServers) ...
    Cosmin MargineanCosmin Marginean
    Jan 7, 2016 at 10:23 pm
    Jan 8, 2016 at 11:06 pm
  • Hi, I am building my application using the rest proxy and I wanted to get some clarification on the documentation. "Because consumers are stateful, any consumer instances created with the REST API ...
    Heath IvieHeath Ivie
    Jan 7, 2016 at 8:55 pm
    Jan 8, 2016 at 5:02 pm
  • Hi, The at-least-once delivery is generated in part by the network fails and the retries (that may generate duplicates) right? In the event of a duplicated (there was an error but the first message ...
    Franco GiacosaFranco Giacosa
    Jan 30, 2016 at 12:18 pm
    Feb 2, 2016 at 6:30 pm
  • Hi All, How to use Kafka 0.9 MirrorMaker to produce data to different topic name. Gwen has informed that --message.handler option needs to be used, but I couldn't find any documentation related to ...
    Mhaskar, TusharMhaskar, Tushar
    Jan 27, 2016 at 10:08 pm
    Jan 29, 2016 at 8:05 pm
  • Hi All, I was using the new Kafka Consumer to fetch messages in this way: while (true) { ConsumerRecords<Object, T records = kafkaConsumer.poll(Long.MAX_VALUE); // do nothing if records are empty ...
    Yifan YingYifan Ying
    Jan 28, 2016 at 9:19 pm
    Jan 28, 2016 at 10:26 pm
  • Hi All, For Kafka 0.8.2, I was using kafka-consumer-offset-checker.sh to get a consumer group's offset . Now I am testing against Kafka 0.9 , but the same tool always gave me Exiting due to ...
    Yifan YingYifan Ying
    Jan 26, 2016 at 1:04 am
    Jan 26, 2016 at 7:05 am
  • Hi, I found a problem that says Memory records is not writable in MirrorMaker when producer sending out messages: [2016-01-08 18:52:30,729] FATAL [mirrormaker-thread-32] Mirror maker thread failure ...
    Samuel ZhouSamuel Zhou
    Jan 9, 2016 at 1:14 am
    Jan 25, 2016 at 9:04 pm
  • Hi, on the documentation for commitSync it says the following about the CommitFailedException * @throws org.apache.kafka.clients.consumer.CommitFailedException if the commit failed and cannot be ...
    Franco GiacosaFranco Giacosa
    Jan 15, 2016 at 11:26 am
    Jan 21, 2016 at 11:11 pm
  • Hello, We historically have been a RabbitMQ environment, but we're looking at using Kafka for a new project and I'm wondering if the following topology/setup would work well in Kafka (for RMQ we'd ...
    Jason J. W. WilliamsJason J. W. Williams
    Jan 14, 2016 at 9:38 pm
    Jan 18, 2016 at 7:11 pm
  • Hello, In our project we have a very strong requirement to protect all data, all the time. Even when the data is “in-rest” on disk, it needs to be protected. We’ve been trying to figure out how to ...
    Bruno RassaertsBruno Rassaerts
    Jan 14, 2016 at 10:23 pm
    Jan 15, 2016 at 5:29 pm
  • Hi , I am running into an issue where I cannot register new consumers. The server is consistently return error code 40901: "Consumer cannot subscribe the the specified target because it has already ...
    Heath IvieHeath Ivie
    Jan 12, 2016 at 11:08 pm
    Jan 13, 2016 at 8:45 pm
  • We have recently upgraded some of our applications to use the Kafka 0.8.2 Java producers from the old Java wrappers over Scala producers. We've noticed these log messages on our application since the ...
    Rajiv KurianRajiv Kurian
    Jan 11, 2016 at 9:08 pm
    Jan 13, 2016 at 5:16 pm
  • Our 2 node kafka cluster has become unhealthy. We're running zookeeper as a 3 node system, which very light load. What seems to be happening is in the controller log we get a ZK session expire ...
    Dillian MurpheyDillian Murphey
    Jan 12, 2016 at 11:05 pm
    Jan 13, 2016 at 7:21 am
  • Hi, I tried executing the following, bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-console-sink.properties I created a file ...
    Shiti SaxenaShiti Saxena
    Jan 12, 2016 at 6:15 am
    Jan 12, 2016 at 7:27 pm
  • Hi, I have a question regarding the Kafka 0.9 Consumer Group . I manually commit offsets using the Kafka 0.9 Consumer created with a consumer group. However, after my app restarted totally from ...
    Wang, HowardWang, Howard
    Jan 11, 2016 at 3:42 pm
    Jan 11, 2016 at 9:09 pm
  • Hi, I'm having trouble sending produce requests to a Kafka 0.9.0.0 broker cluster consisting of 4 brokers with IDs 0, 1, 2, and 3. All 4 brokers are running locally on my CentOS 7 development box, ...
    DaveDave
    Jan 11, 2016 at 8:32 am
    Jan 11, 2016 at 7:05 pm
  • Hey! So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this morning, and it seems like everything went smoothly. As the last step when I changed the inter.broker.protocol.version to ...
    Stephen PowisStephen Powis
    Jan 8, 2016 at 7:39 am
    Jan 11, 2016 at 2:14 pm
  • Hi guys I’ve been seeing this in our logs a few times, when initially bootstrapping the system. WARN 2016-01-08 10:59:24,232 [pool-3-thread-11] [none] o.a.kafka.clients.NetworkClient: Error while ...
    Cosmin MargineanCosmin Marginean
    Jan 8, 2016 at 11:20 am
    Jan 8, 2016 at 5:28 pm
  • Hi, Does Kafka Connect have an API which can be used by applications to start Kafka Connect, add/remove Connectors? I also do not want to use the REST API and do not want to start the REST server ...
    Shiti SaxenaShiti Saxena
    Jan 6, 2016 at 5:12 am
    Jan 7, 2016 at 8:12 pm
  • I want to use the new 0.9 consumer for a particular application. My use case is the following: i) The TopicPartition I need to poll has a short log say 10 mins odd (log.retention.minutes is set to ...
    Rajiv KurianRajiv Kurian
    Jan 6, 2016 at 5:37 pm
    Jan 7, 2016 at 4:46 am
  • Hi All, On our dev kafka environment our brokers went down with a topic on it, can we just reassign the partitions to another broker? Kafka 0.9 Thanks Ben -- This email, including attachments, is ...
    Ben DavisonBen Davison
    Jan 5, 2016 at 4:32 pm
    Jan 6, 2016 at 11:00 pm
  • Hi all, In the kafka docs, it mentions that MirrorMaker will automatically create topics on the mirror cluster if the right options are set. Does it automatically delete topics on the mirror that are ...
    Simon CooperSimon Cooper
    Jan 5, 2016 at 1:53 pm
    Jan 5, 2016 at 6:51 pm
  • My version is kafka_2.11-0.9.0.0. I find that the kafka listen on multi tcp port on a linux server. [gdata@gdataqosconnd2 kafka_2.11-0.9.0.0]$ netstat -plnt|grep java (Not all processes could be ...
    Costa xuCosta xu
    Jan 29, 2016 at 3:58 am
    Feb 2, 2016 at 1:22 am
  • Hi Guys, I've set up a test cluster using 3 Vagrant nodes each running a Kafka broker and zookeeper instance. Zookeepers are all linked properly and the brokers appear to be registering properly ...
    Jason J. W. WilliamsJason J. W. Williams
    Jan 24, 2016 at 5:33 am
    Jan 25, 2016 at 10:42 pm
  • Hi, I'm currently looking into implementing a load shedding strategy into Flink's Kafka consumer. Therefore, I would like to allow users to request the latest offset of the subscribed ...
    Robert MetzgerRobert Metzger
    Jan 20, 2016 at 11:09 am
    Jan 25, 2016 at 10:35 am
  • Dear Kafka Users, I have been experimenting with Kafka and I'm currently trying to come up with a design for the set of topics that we would need for our use case. We are reading a set of signals ...
    Joe SanJoe San
    Jan 19, 2016 at 7:47 am
    Jan 19, 2016 at 10:10 pm
  • Hi Kafka users, I have an issue with saving Kafka offsets to Zookeeper through OffsetCommitRequest. It's the same issue I found unanswered on SO, so I kindly borrow the description ...
    Krzysztof ZarzyckiKrzysztof Zarzycki
    Jan 19, 2016 at 8:03 am
    Jan 19, 2016 at 7:34 pm
Group Navigation
period‹ prev | Jan 2016 | next ›
Group Overview
groupusers @
categorieskafka
discussions138
posts527
users142
websitekafka.apache.org
irc#kafka

142 users for January 2016

Ismael Juma: 28 posts Jason Gustafson: 28 posts Ewen Cheslack-Postava: 21 posts Tao xiao: 21 posts Gwen Shapira: 20 posts Guozhang Wang: 15 posts Rajiv Kurian: 15 posts Dana Powers: 13 posts Franco Giacosa: 13 posts Heath Ivie: 13 posts Jason Williams: 13 posts Luke Steensen: 11 posts Cosmin Marginean: 9 posts Jens Rantil: 9 posts Joe San: 9 posts Stephen Powis: 9 posts BYEONG-GI KIM: 8 posts Stevo Slavić: 8 posts Alex Loddengaard: 7 posts Dillian Murphey: 7 posts
show more