FAQ
Mostly thinking out loud here... but we welcome feedback if we're off
track...

Okay, after some perl -D:NTYProf tester.pl with 200 iterations:
70% of the time is taken up in five modules:
1) SQL::Abstract
2) DBIx::Class::ResultSet
3) Class::Accessor::Grouped
4) DBIx::Class::Storage::DBI
5) HTML::FormHandler::Field
...because in 200 iterations they were called millions of times each. (The
form requested has several select/option menus.)

Most of the data won't change rapidly -- e.g. client lists, countries,
employees just to populate a few popup menus -- so it seems like an
opportunity to benefit from caching the results, and only invalidating the
cache when a new record gets updated/deleted/added. Whole tables could be
cached without having to return to SQL/DBIC for every form-screen.

So we're looking for documentation on a best-of-breed approach for doing
this in a Catalyst environment...

Looking at Catalyst::TraitFor::Model::DBIC::Schema::Caching it looks like
you specify when the cache will expire:
$c->model('DB::Table')->search({ foo => 'bar' }, { cache_for => 18000 });
whereas we would prefer being able to invalidate-cache-for-table-XYZ-now.

Aha, DBIx::Class::Cursor::Cached does include a
$rs->cursor->clear_cache;
method, we'll be tinkering with that, next!



On Tue, Apr 12, 2011 at 8:14 AM, will trillich
wrote:
Thanks for the tips, Peter -- and for
http://dragonstaff.blogspot.com/2009/05/testing-with-perl-catalyst.html!
Got some cranking to do...

On Tue, Apr 12, 2011 at 2:30 AM, Peter Edwards wrote:
On 12 April 2011 02:53, will trillich wrote:

Hi folks -- this may be more of a FormHandler question than a Catalyst
question but I thought I'd check here to see if it's just us:

We've been using HTML::FormHandler and are basically happy with it...
until the performance issue mentioned below hit us. Any Catalystas running
into 50-second turnaround time with H:FH?

[info] Request took 51.956100s (0.019/s)

.------------------------------------------------------------+-----------.
Action | Time
+------------------------------------------------------------+-----------+
/auto | 0.000181s

/auth | 0.001857s

/ticket/base | 0.004652s

/ticket/item | 0.005865s

/ticket/edit | 51.88091s

Base:EDIT | 51.88050s

get FORM | 0.000078s

process FORM | *
51.87286s* |
/end | 0.000290s
'------------------------------------------------------------+-----------'

Turnaround time ranges from 6 seconds to 50+ seconds, with no discernable
pattern to the delay. (We can edit the same record multiple times and get
wildly differing lags.)


Run your test server with perl -d:NYTProf myapp.pl and see which
routines use the time http://search.cpan.org/perldoc?Devel::NYTProf .
Maybe it is blocking on DNS network lookups.
Or if the time isused around the database calls, run with DBIC_TRACE=1
perl myapp.pl and watch and see which are the slow ones, then run your
database query optimiser like EXPLAIN SELECT foo;
http://dev.mysql.com/doc/refman/5.0/en/explain.html
http://www.postgresql.org/docs/8.3/static/sql-explain.html
Are other processes/users accessing the same database? If so check for
lock competition. Also disk space.

Regards, Peter



_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive:
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

--
11 cheers for binary!


--
11 cheers for binary!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20110413/5fb0a1e6/attachment.htm

Search Discussions

  • Darren Duncan at Apr 13, 2011 at 6:19 am

    will trillich wrote:
    Okay, after some perl -D:NTYProf tester.pl <http://tester.pl> with 200
    iterations:
    70% of the time is taken up in five modules:
    1) SQL::Abstract
    2) DBIx::Class::ResultSet
    3) Class::Accessor::Grouped
    4) DBIx::Class::Storage::DBI
    5) HTML::FormHandler::Field
    ...because in 200 iterations they were called millions of times each.
    (The form requested has several select/option menus.)
    So, several 5000s (millions / 200) of calls for a single screen? That sounds
    like a lot for one screen. Do you need that much? -- Darren Duncan
  • Will Trillich at Apr 14, 2011 at 4:15 pm

    On Wed, Apr 13, 2011 at 1:19 AM, Darren Duncan wrote:

    will trillich wrote:
    70% of the time is taken up in five modules:
    1) SQL::Abstract
    2) DBIx::Class::ResultSet
    3) Class::Accessor::Grouped
    4) DBIx::Class::Storage::DBI
    5) HTML::FormHandler::Field
    ...because in 200 iterations they were called millions of times each. (The
    form requested has several select/option menus.)
    So, several 5000s (millions / 200) of calls for a single screen? That
    sounds like a lot for one screen. Do you need that much? -- Darren Duncan
    My question is similar -- why five thousand or more calls for rendering one
    page? The code does simple things like
    my $form = $self->form;
    # parse_form_for_numerics():
    # change any $1,234 to 1234, parse 12-apr-09 to a date 2009-04-12, etc
    # and stuff results into $c->req->params
    &parse_form_for_numerics( $c, $form );
    my $processed = $form->process(
    item => $item,
    params => $c->req->params,
    );
    if ( $form->has_errors) { ... }
    return unless $processed;
    # since we're testing rendering a fresh page there's no processing, that's
    it

    So why would this simple code generate 5,000+ calls to SQL::Abstract methods
    or to DBIx::Class::Resultset methods? Even commenting-out
    #parse_form_for_numerics()
    it's about the same. Still looking into this.

    --
    11 cheers for binary!
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20110414/c53eb4e9/attachment.htm
  • Hernan Lopes at Apr 14, 2011 at 5:53 pm
    i remember you mentioned something about many to many select options, try
    disabling those.. and then does the form works faster?
    if so, then the problem is mostly like in there ... try populating your
    options manually

    On Thu, Apr 14, 2011 at 1:15 PM, will trillich
    wrote:
    On Wed, Apr 13, 2011 at 1:19 AM, Darren Duncan wrote:

    will trillich wrote:
    70% of the time is taken up in five modules:
    1) SQL::Abstract
    2) DBIx::Class::ResultSet
    3) Class::Accessor::Grouped
    4) DBIx::Class::Storage::DBI
    5) HTML::FormHandler::Field
    ...because in 200 iterations they were called millions of times each.
    (The form requested has several select/option menus.)
    So, several 5000s (millions / 200) of calls for a single screen? That
    sounds like a lot for one screen. Do you need that much? -- Darren Duncan

    My question is similar -- why five thousand or more calls for rendering one
    page? The code does simple things like
    my $form = $self->form;
    # parse_form_for_numerics():
    # change any $1,234 to 1234, parse 12-apr-09 to a date 2009-04-12, etc
    # and stuff results into $c->req->params
    &parse_form_for_numerics( $c, $form );
    my $processed = $form->process(
    item => $item,
    params => $c->req->params,
    );
    if ( $form->has_errors) { ... }
    return unless $processed;
    # since we're testing rendering a fresh page there's no processing, that's
    it

    So why would this simple code generate 5,000+ calls to SQL::Abstract
    methods
    or to DBIx::Class::Resultset methods? Even commenting-out
    #parse_form_for_numerics()
    it's about the same. Still looking into this.

    --
    11 cheers for binary!

    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive:
    http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20110414/49256450/attachment.htm
  • Nicholas Wehr at Apr 14, 2011 at 6:17 pm
    From what I recall - the caching trait doesn't play with joins. From what I
    can tell - neither does H::FH; Can anyone confirm this? It might make more
    sense for your queries to always specify a deep join/prefetch option to
    reduce the number of queries.

    I see this statement as well:
    "This feature is new. It doesn't handle relationships yet, and the
    interfaces are still subject to change."
    ... which leads me to believe that efficient joins aren't implemented

    http://search.cpan.org/~gshank/HTML-FormHandler-Model-DBIC-0.14/lib/HTML/FormHandler/TraitFor/DBICFields.pm

    <http://search.cpan.org/~gshank/HTML-FormHandler-Model-DBIC-0.14/lib/HTML/FormHandler/TraitFor/DBICFields.pm>
    cheers,
    -nw
    On Thu, Apr 14, 2011 at 10:53 AM, Hernan Lopes wrote:

    i remember you mentioned something about many to many select options, try
    disabling those.. and then does the form works faster?
    if so, then the problem is mostly like in there ... try populating your
    options manually

    On Thu, Apr 14, 2011 at 1:15 PM, will trillich <
    will.trillich@serensoft.com> wrote:
    On Wed, Apr 13, 2011 at 1:19 AM, Darren Duncan wrote:

    will trillich wrote:
    70% of the time is taken up in five modules:
    1) SQL::Abstract
    2) DBIx::Class::ResultSet
    3) Class::Accessor::Grouped
    4) DBIx::Class::Storage::DBI
    5) HTML::FormHandler::Field
    ...because in 200 iterations they were called millions of times each.
    (The form requested has several select/option menus.)
    So, several 5000s (millions / 200) of calls for a single screen? That
    sounds like a lot for one screen. Do you need that much? -- Darren Duncan

    My question is similar -- why five thousand or more calls for rendering
    one page? The code does simple things like
    my $form = $self->form;
    # parse_form_for_numerics():
    # change any $1,234 to 1234, parse 12-apr-09 to a date 2009-04-12, etc
    # and stuff results into $c->req->params
    &parse_form_for_numerics( $c, $form );
    my $processed = $form->process(
    item => $item,
    params => $c->req->params,
    );
    if ( $form->has_errors) { ... }
    return unless $processed;
    # since we're testing rendering a fresh page there's no processing, that's
    it

    So why would this simple code generate 5,000+ calls to SQL::Abstract
    methods
    or to DBIx::Class::Resultset methods? Even commenting-out
    #parse_form_for_numerics()
    it's about the same. Still looking into this.

    --
    11 cheers for binary!

    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive:
    http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive:
    http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20110414/42a22aa3/attachment.htm
  • Bill Crawford at Apr 13, 2011 at 9:07 am

    On 13 April 2011 06:42, will trillich wrote:
    Mostly thinking out loud here... but we welcome feedback if we're off
    track...
    Okay, after some perl -D:NTYProf tester.pl with 200 iterations:
    70% of the time is taken up in five modules:
    1) SQL::Abstract
    2) DBIx::Class::ResultSet
    3) Class::Accessor::Grouped
    4) DBIx::Class::Storage::DBI
    5) HTML::FormHandler::Field
    ...because in 200 iterations they were called millions of times each. (The
    form requested has several select/option menus.)

    Most of the data won't change rapidly -- e.g. client lists, countries,
    employees just to populate a few popup menus -- so it seems like an
    opportunity to benefit from caching the results, and only invalidating the
    cache when a new record gets updated/deleted/added. Whole tables could be
    cached without having to return to SQL/DBIC for every form-screen.
    So we're looking for documentation on a best-of-breed approach for doing
    this in a Catalyst environment...
    What might help you there is a H::FH field attribute "do_not_reload => 1".

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcatalyst @
categoriescatalyst, perl
postedApr 13, '11 at 5:42a
activeApr 14, '11 at 6:17p
posts6
users5
websitecatalystframework.org
irc#catalyst

People

Translate

site design / logo © 2021 Grokbase