FAQ
I'm approaching deploying WARs and other deployment artifacts into my cloud using RabbitMQ (of course! ;) by sending a message containing a URL the consumer could use to download the necessary resource, but I was wondering if I should not so easily dismiss chunking my artifacts and sending them in pieces into the cloud, rather than sending a reference to an HTTP-accessible resource which must be downloaded and deployed?

Does anyone use chunking for sending large objects via messages? Is it better or worse than forcing consumers to download their own resource? How do you keep messages in order and make sure your artifacts don't get corrupted? I've never done async chunking before so thought I'd see if anyone has experience in this area...

Thanks!

Jon Brisbin
Portal Webmaster
NPC International, Inc.

Search Discussions

  • Marek Majkowski at May 10, 2010 at 1:36 pm

    On Fri, May 7, 2010 at 20:25, Jon Brisbin wrote:
    I'm approaching deploying WARs and other deployment artifacts into my cloud using RabbitMQ (of course! ;) by sending a message containing a URL the consumer could use to download the necessary resource, but I was wondering if I should not so easily dismiss chunking my artifacts and sending them in pieces into the cloud, rather than sending a reference to an HTTP-accessible resource which must be downloaded and deployed?

    Does anyone use chunking for sending large objects via messages? Is it better or worse than forcing consumers to download their own resource? How do you keep messages in order and make sure your artifacts don't get corrupted? I've never done async chunking before so thought I'd see if anyone has experience in this area...

    Putting a large file over RabbitMQ obviously will work, but I'd say
    it's a bad idea. It's just easier to put the big things to some
    temporary storage, and transfer only light requests over message bus.

    Putting large stuff on the bus (throughput) can probably effect in a
    degraded latency. On the other hand, if you aren't worried about
    latency it might work - and you correctly identified possible
    complications.

    Marek Majkowski
  • Tony Garnock-Jones at May 11, 2010 at 9:42 am

    Marek Majkowski wrote:
    Putting a large file over RabbitMQ obviously will work, but I'd say
    it's a bad idea. It's just easier to put the big things to some
    temporary storage, and transfer only light requests over message bus.
    Furthermore, until the new persister lands, sending *really* big messages will
    cause problems with memory exhaustion.

    HTTP has all sorts of neat machinery for reliably transferring very large
    payloads a piece at a time, resuming aborted transfers, and so on. It even has
    a verb, DELETE, that can be pressed into service as a poor-man's
    acknowledge-and-release-message signal (which can work especially nicely on a
    server file system supporting hard links: give every potential receiver a
    distinct URL to retrieve and delete, and you get a kind of automatic garbage
    collection). I'd go with HTTP for transferring large chunks of data, and the
    messaging system for carrying the URLs.

    The problems with that system I can think of: 1. when is it safe to stop
    serving a piece of information from the HTTP server? and 2. how do you get the
    chunk up to the HTTP server in the first place? Not every client can run an
    httpd, and HTTP sadly (!) lacks resumable PUT/POST.

    To solve 1: If you know you have a single receiver, use DELETE. If you know you
    have exactly n receivers, hardlink the file on the server a few times for
    URL-uniqueness, and use DELETE. If you don't know the set of receivers, you're
    in trouble, and a timeout is likely your only sensible option.

    To solve 2: Either run an httpd on clients wishing to send Very Large Files
    (with associated firewall and configuration headaches), or use ReverseHTTP to
    be httpd-like without the firewall and configuration hassle, or put a CGI
    script or similar on the server that supports a protocol you can use to
    reliably upload files in a resumable fashion for later download by other clients.

    Or yeah, if it's not going to kill your server, just send the big files as
    large messages. Resumability of partial transfers goes out the window, of course.

    Tony

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedMay 7, '10 at 7:25p
activeMay 11, '10 at 9:42a
posts3
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase