Is RabbitMQ a good fit for writing a bunch of log data into or would something like Kafka be better suited for this?

Search Discussions

  • Dave Lambley at Jun 8, 2013 at 8:13 am
    Yes, it does this well. If you're wanting your application to log directly into it, it's worth adding some buffering such as with ZeroMQ to eliminate blocking.


    Dave


    --
    Sent using from a tiny keypad.


    -----Original Message-----
    From: Mark <static.void.dev@gmail.com>
    Sender: rabbitmq-discuss-bounces at lists.rabbitmq.com
    Date: Fri, 7 Jun 2013 17:24:25
    To: Discussions about RabbitMQ<rabbitmq-discuss@lists.rabbitmq.com>
    Reply-To: Discussions about RabbitMQ <rabbitmq-discuss@lists.rabbitmq.com>
    Subject: [rabbitmq-discuss] RabbitMQ for log data


    Is RabbitMQ a good fit for writing a bunch of log data into or would something like Kafka be better suited for this?
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Mark at Jun 8, 2013 at 7:10 pm
    I'm guessing if I didn't want directly to it I would need to write some sort of producer to tail some logs and ship them over to Rabbit?


    On Jun 8, 2013, at 1:13 AM, dave.lambley at gmail.com wrote:

    Yes, it does this well. If you're wanting your application to log directly into it, it's worth adding some buffering such as with ZeroMQ to eliminate blocking.

    Dave

    --
    Sent using from a tiny keypad.

    -----Original Message-----
    From: Mark <static.void.dev@gmail.com>
    Sender: rabbitmq-discuss-bounces at lists.rabbitmq.com
    Date: Fri, 7 Jun 2013 17:24:25
    To: Discussions about RabbitMQ<rabbitmq-discuss@lists.rabbitmq.com>
    Reply-To: Discussions about RabbitMQ <rabbitmq-discuss@lists.rabbitmq.com>
    Subject: [rabbitmq-discuss] RabbitMQ for log data

    Is RabbitMQ a good fit for writing a bunch of log data into or would something like Kafka be better suited for this?
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Alexis Richardson at Jun 10, 2013 at 9:55 am
    Some examples...


    https://www.google.co.uk/search?q=graylog2+rabbitmq&oq=graylog2+rabbitmq&aqs=chrome.0.57j0.6173j0&sourceid=chrome&ie=UTF-8





    On Sat, Jun 8, 2013 at 8:10 PM, Mark wrote:
    I'm guessing if I didn't want directly to it I would need to write some sort of producer to tail some logs and ship them over to Rabbit?
    On Jun 8, 2013, at 1:13 AM, dave.lambley at gmail.com wrote:

    Yes, it does this well. If you're wanting your application to log directly into it, it's worth adding some buffering such as with ZeroMQ to eliminate blocking.

    Dave

    --
    Sent using from a tiny keypad.

    -----Original Message-----
    From: Mark <static.void.dev@gmail.com>
    Sender: rabbitmq-discuss-bounces at lists.rabbitmq.com
    Date: Fri, 7 Jun 2013 17:24:25
    To: Discussions about RabbitMQ<rabbitmq-discuss@lists.rabbitmq.com>
    Reply-To: Discussions about RabbitMQ <rabbitmq-discuss@lists.rabbitmq.com>
    Subject: [rabbitmq-discuss] RabbitMQ for log data

    Is RabbitMQ a good fit for writing a bunch of log data into or would something like Kafka be better suited for this?
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Dave Lambley at Jun 11, 2013 at 10:16 pm
    That's already been done by a couple of things, try Logstash (
    http://logstash.net/ ), or Message::Passing (
    https://metacpan.org/release/Message-Passing ) if you prefer considerably
    lighter-weight Perl.


    Dave




    On 8 June 2013 20:10, Mark wrote:

    I'm guessing if I didn't want directly to it I would need to write some
    sort of producer to tail some logs and ship them over to Rabbit?
    On Jun 8, 2013, at 1:13 AM, dave.lambley at gmail.com wrote:

    Yes, it does this well. If you're wanting your application to log
    directly into it, it's worth adding some buffering such as with ZeroMQ to
    eliminate blocking.
    Dave

    --
    Sent using from a tiny keypad.

    -----Original Message-----
    From: Mark <static.void.dev@gmail.com>
    Sender: rabbitmq-discuss-bounces at lists.rabbitmq.com
    Date: Fri, 7 Jun 2013 17:24:25
    To: Discussions about RabbitMQ<rabbitmq-discuss@lists.rabbitmq.com>
    Reply-To: Discussions about RabbitMQ <
    rabbitmq-discuss at lists.rabbitmq.com>
    Subject: [rabbitmq-discuss] RabbitMQ for log data

    Is RabbitMQ a good fit for writing a bunch of log data into or would
    something like Kafka be better suited for this?
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130611/7cd60b8c/attachment.htm>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJun 8, '13 at 12:24a
activeJun 11, '13 at 10:16p
posts5
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase