Grokbase Groups Kafka dev June 2016
FAQ
[ https://issues.apache.org/jira/browse/KAFKA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337883#comment-15337883 ]

Ismael Juma commented on KAFKA-1789:
------------------------------------

To add to Gwen's comment, the Scala producers have been deprecated and the new Java producer is recommended instead.
Issue with Async producer
-------------------------

Key: KAFKA-1789
URL: https://issues.apache.org/jira/browse/KAFKA-1789
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.0, 0.8.1
Reporter: devendra tagare
Priority: Critical

Hi,
We are using an async producer to send data to a kafka cluster.The event rate at peak is around 250 events/second of size 25KB each.
In the producer code base we have added specific debug statements to capture the time taken to create a producer,create a keyed message with a byte payload & send the message.
We have added the below properties to the producerConfig
queue.enqueue.timeout.ms=20
send.buffer.bytes=1024000
topic.metadata.refresh.interval.ms=30000
Based on the documentation, producer.send() queues the message on the async producer's queue.
So, ideally if the queue is full then the enqueue operation should result in an kafka.common.QueueFullException in 20 ms.
The logs indicate that the enqueue operation is taking more than 20ms (takes around 250ms) without throwing any exceptions.
Is there any other property that could conflict with queue.enqueue.timeout.ms which is causing this behavior ?
Or is it possible that the queue is not full & yet the producer.send() call is still taking around 200ms under peak load ?
Also, could you suggest any other alternatives so that we can either enforce a timeout or throw an exception in-case the async producer is taking more than a specified amount of time.
Regards,
Dev


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 2 of 2 | next ›
Discussion Overview
groupdev @
categorieskafka
postedJun 18, '16 at 11:32a
activeJun 18, '16 at 1:32p
posts2
users1
websitekafka.apache.org
irc#kafka

1 user in discussion

Ismael Juma (JIRA): 2 posts

People

Translate

site design / logo © 2019 Grokbase