FAQ
Hi,

I consider offering a push service for real-time data. It's AJAX based
and requires a consistent connection.
I'm not sure about the memory usage of this. I plan to run the
catalyst app as fastcgi server. Does every consistent connection
create an extra fastcgi process? Which means an additional ~30mb (if
that's the size of a fastcgi process)?

moritz

Search Discussions

  • Brian Kirkbride at Mar 5, 2008 at 6:39 pm

    Moritz Onken wrote:
    Hi,

    I consider offering a push service for real-time data. It's AJAX based
    and requires a consistent connection.
    I'm not sure about the memory usage of this. I plan to run the catalyst
    app as fastcgi server. Does every consistent connection create an extra
    fastcgi process? Which means an additional ~30mb (if that's the size of
    a fastcgi process)?

    moritz
    If it's AJAX then you are really polling as opposed to pushing. In
    this case Lighttpd or Nginx are your friends. They can handle all of
    the connections (with a long KeepAlive) and only talk to the backend
    FastCGI when a poll request comes through from the AJAX client.

    If you're really pushing with a constantly connected client, you'll
    need a FastCGI process per client. The difference is who initiates
    communication over the channel. If the server can send to the client
    at any moment, that's true push. Most people get along with a rapidly
    polling client that says "Anything for me?" every so often.

    In general polling is wasteful of bandwidth, while push is wasteful of
    server-side resources.

    Best,
    Brian
  • Moritz Onken at Mar 5, 2008 at 7:48 pm
    I'm combining ajax with push. So it's a real push service, the website
    is not pulling the data on a constant time rate.

    A process per client is really bad. I'll need to serve hundreds of
    client at the same time. Seems like push is not an option...

    Am 05.03.2008 um 19:39 schrieb Brian Kirkbride:
    Moritz Onken wrote:
    Hi,
    I consider offering a push service for real-time data. It's AJAX
    based and requires a consistent connection.
    I'm not sure about the memory usage of this. I plan to run the
    catalyst app as fastcgi server. Does every consistent connection
    create an extra fastcgi process? Which means an additional ~30mb
    (if that's the size of a fastcgi process)?
    moritz
    If it's AJAX then you are really polling as opposed to pushing. In
    this case Lighttpd or Nginx are your friends. They can handle all
    of the connections (with a long KeepAlive) and only talk to the
    backend FastCGI when a poll request comes through from the AJAX
    client.

    If you're really pushing with a constantly connected client, you'll
    need a FastCGI process per client. The difference is who initiates
    communication over the channel. If the server can send to the
    client at any moment, that's true push. Most people get along with
    a rapidly polling client that says "Anything for me?" every so often.

    In general polling is wasteful of bandwidth, while push is wasteful
    of server-side resources.

    Best,
    Brian

    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
  • Jay Shirley at Mar 5, 2008 at 8:13 pm

    On Wed, Mar 5, 2008 at 11:48 AM, Moritz Onken wrote:
    I'm combining ajax with push. So it's a real push service, the website
    is not pulling the data on a constant time rate.

    A process per client is really bad. I'll need to serve hundreds of
    client at the same time. Seems like push is not an option...
    Take a look at cometd: http://cometd.com/
  • Peter Edwards at Mar 5, 2008 at 10:47 pm

    Jay wrote:
    On Wed, Mar 5, 2008 at 11:48 AM, Moritz Onken wrote:
    I'm combining ajax with push. So it's a real push service, the website
    is not pulling the data on a constant time rate.

    A process per client is really bad. I'll need to serve hundreds of
    client at the same time. Seems like push is not an option...
    Take a look at cometd: http://cometd.com/
    Thanks for mentioning that, it looks interesting, particularly as it's
    hooked up to perlbal.

    Moritz, I've done something similar before using POE to handle many
    connections more cheaply than Apache (thanks to Matt Trout for the
    suggestion). It was to serve XML formatted requests though you could return
    JSON formatted data just as easily
    http://search.cpan.org/perldoc?POE::Component::Server::TCP

    Regards, Peter
    http://perl.dragonstaff.co.uk
  • Moritz Onken at Mar 6, 2008 at 10:08 am
    But I still need a lot of code which is run by catalyst, like
    authentication and of course the whole dbic schema.
    What about Catalyst::Engine::HTTP::POE?

    cometd.com is not very informative ...

    Am 05.03.2008 um 23:47 schrieb Peter Edwards:
    Jay wrote:
    On Wed, Mar 5, 2008 at 11:48 AM, Moritz Onken
    wrote:
    I'm combining ajax with push. So it's a real push service, the
    website
    is not pulling the data on a constant time rate.

    A process per client is really bad. I'll need to serve hundreds of
    client at the same time. Seems like push is not an option...
    Take a look at cometd: http://cometd.com/
    Thanks for mentioning that, it looks interesting, particularly as it's
    hooked up to perlbal.

    Moritz, I've done something similar before using POE to handle many
    connections more cheaply than Apache (thanks to Matt Trout for the
    suggestion). It was to serve XML formatted requests though you could
    return
    JSON formatted data just as easily
    http://search.cpan.org/perldoc?POE::Component::Server::TCP

    Regards, Peter
    http://perl.dragonstaff.co.uk


    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
  • Peter Edwards at Mar 6, 2008 at 1:14 pm

    Am 05.03.2008 um 23:47 schrieb Peter Edwards:
    Jay wrote:
    On Wed, Mar 5, 2008 at 11:48 AM, Moritz Onken
    <onken@houseofdesign.de>
    wrote:
    I'm combining ajax with push. So it's a real push service, the
    website is not pulling the data on a constant time rate.

    A process per client is really bad. I'll need to serve hundreds of
    client at the same time. Seems like push is not an option...
    Take a look at cometd: http://cometd.com/
    Moritz, I've done something similar before using POE to handle many
    connections more cheaply than Apache (thanks to Matt Trout for the
    suggestion). It was to serve XML formatted requests though you could
    return JSON formatted data just as easily
    http://search.cpan.org/perldoc?POE::Component::Server::TCP
    But I still need a lot of code which is run by catalyst, like
    authentication and of course the whole dbic schema.
    What about Catalyst::Engine::HTTP::POE?

    cometd.com is not very informative ...
    In the article http://www.irishdev.com/NewsArticle.aspx?id!66
    see that diagram of an SOA style architecture?
    http://alex.dojotoolkit.org/wp-content/Comet.png

    Browser UI = HTML + Javascript

    Comet client = Javascript driven by timed events

    talks persistent TCP to

    Comet event bus = proxy server, e.g. perlbal or nginx or POE TCP daemon

    talks HTTP or XML-RPC to

    Server-side processing = your Catalyst app


    You need to split the management of the many persistent TCP connections from
    the generation of content.

    In a typical web server setup you achieve this by using a scalable caching
    proxy front end to do the TCP socket handling and a Catalyst backend for the
    content.

    In your case you might need something a bit smarter in the front end, hence
    my mention of a custom POE daemon. Or you could write a plugin for perlbal
    http://search.cpan.org/dist/Perlbal/.

    You could make an XML-RPC or HTTP request from the front end to Catalyst to
    authenticate, and thereafter run periodic front to backend GETs and cache
    them (in the "comet event bus" part) ready to serve to the comet client via
    TCP.

    That way you don't need to tie up a 30MB Catalyst process just to hold open
    a socket, are spreading the request load evenly across time and can write it
    all in Perl.

    Alternatively, if you have an enormous load the for the proxy you could use
    nginx and write a C module to call out to your backend
    http://www.riceonfire.org/emiller/nginx-modules-guide.html


    Regards, Peter
    http://perl.dragonstaff.co.uk
  • Jonathan Rockway at Mar 6, 2008 at 9:38 pm

    * On Thu, Mar 06 2008, Moritz Onken wrote:
    But I still need a lot of code which is run by catalyst, like
    authentication and of course the whole dbic schema.
    What about Catalyst::Engine::HTTP::POE?
    Theoretically this should work. When your application is in the state
    of doing IO (sending data to the client, holding a socket open), other
    requests can run. That sounds like it is what you want.

    There are things that will block POE, though; handling a request of
    course, as well as things that are technically IO (waiting for your
    database to return a query). That can be fixed, though.

    Regards,
    Jonathan Rockway
  • Patrick Donelan at Mar 10, 2008 at 11:26 pm
    If you're looking for in-depth discussion of Comet techniques I'd suggest
    you have a look at cometdaily.com.

    Cheers,

    Patrick
    On 3/6/08, Moritz Onken wrote:

    But I still need a lot of code which is run by catalyst, like
    authentication and of course the whole dbic schema.
    What about Catalyst::Engine::HTTP::POE?

    cometd.com is not very informative ...

    Am 05.03.2008 um 23:47 schrieb Peter Edwards:

    Jay wrote:
    On Wed, Mar 5, 2008 at 11:48 AM, Moritz Onken
    <onken@houseofdesign.de>
    wrote:
    I'm combining ajax with push. So it's a real push service, the
    website
    is not pulling the data on a constant time rate.

    A process per client is really bad. I'll need to serve hundreds of
    client at the same time. Seems like push is not an option...
    Take a look at cometd: http://cometd.com/
    Thanks for mentioning that, it looks interesting, particularly as it's
    hooked up to perlbal.

    Moritz, I've done something similar before using POE to handle many
    connections more cheaply than Apache (thanks to Matt Trout for the
    suggestion). It was to serve XML formatted requests though you could
    return
    JSON formatted data just as easily
    http://search.cpan.org/perldoc?POE::Component::Server::TCP

    Regards, Peter
    http://perl.dragonstaff.co.uk


    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive:
    http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/

    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive:
    http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20080311/ac26ffc9/attachment.htm
  • Brian Kirkbride at Mar 5, 2008 at 8:31 pm

    Moritz Onken wrote:
    I'm combining ajax with push. So it's a real push service, the website
    is not pulling the data on a constant time rate.

    A process per client is really bad. I'll need to serve hundreds of
    client at the same time. Seems like push is not an option...
    Ah... after I clicked send I thought "what if it is a true PUSH using
    Ajax to listen?" Most people think of Ajax polling as a push, though
    it's just looks like push with some granularity. My apologies for
    assuming...

    Have you checked out COMET? See http://cometd.com/ for discussion of
    all the issues with push. I haven't looked at it in a long time, but
    it was an attempt to deal with your issue. The Lighttpd developers
    are working on something similar called mod_mailbox. I don't pretend
    to understand anything about either :)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcatalyst @
categoriescatalyst, perl
postedMar 5, '08 at 5:23p
activeMar 10, '08 at 11:26p
posts10
users6
websitecatalystframework.org
irc#catalyst

People

Translate

site design / logo © 2022 Grokbase