FAQ
I'm running into out of memory problems processing a large number of
elements from a query and I'm lacking a good understanding of how
to make sure that Cayenne can garbage collect it's objects.

Perhaps there are some do's and don'ts, but I don't know what they are at this
point.

For my purposes it doesn't matter how much memory Cayenne uses to
handle a small number of objects, but it becomes a problem if
the memory consumption is proportional to the # of objects returned
from a query.

I'm running Cayenne 3.x

1. Consider the code below. Will all the objects from the query
be locked in memory?

query=new SelectQuery(Foo.class);
query.setPageSize(1000); /* does this matter? */
/* Does performQuery return a clever implementation of
a List that does not lock the CayenneDataObjects into memory?
*/
l=context.performQuery(query);
while (i.hasNext())
{
Foo f=(Foo) i.next();

.... do some processing here that ...


}

2. When does peak memory usage occur above?

3. Can a CayenneDataObject be garbage collected as long as it's
DataContext is in scope?

I'm wondering because as far as I have understood the two if
statements below are equivalent when a & b are CayenneDataObjects
in the same DataContext.

That is if 'a' can be garbage collected during the lifetime of the
DataContext, then 'a' is points to different objects during the
lifetime of CayenneDataObject's and a==b just 'barely' works.
If any information is stored in 'a' then it can be garbage collected
during the lifetime of the CayenneDataContext.

if (a==b)

if (a.getObjectId().equals(b.getObjectId()))

4. Should I consider some "clever" implementation of a query that
ensures that only a small amount of CayenneDataObject's are
locked in memory at the time?

Some ideas:

- use performIteratedQuery() w/setPageSize()
- ResultIterator.nextDataRow() is a bit strange: it returns a Map<>, but
perhaps I can safely upcast that.
- use objectFromdataRow()


--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer

Search Discussions

  • Øyvind Harboe at Jun 20, 2008 at 8:08 am
    I've been looking a bit at IncrementalFaultList and I'm thinking it
    might be worthwhile to try the following implementation to
    minimize the # of objects stuck in memory:


    List l;
    query.setPageSize(1000);
    l=getDataContext().performQuery(query);

    while (l.size()>0)
    {
    t=l.remove(0);

    /* do processing here.... */
    }




    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer
  • Andrus Adamchik at Jun 20, 2008 at 8:27 am
    We thought about automatically swapping out pages from the
    IncrementalFaultList as it is getting filled. At the time we had
    doubts about the efficiency of algorithms under consideration in all
    different circumstances where pagination might be used. Now that I
    mentioned weak references, this can be another algorithm for dealing
    with it - store each "page" as a weak reference, so once its objects
    stop being referenced, it will become a candidate for gc...

    Andrus

    On Jun 20, 2008, at 11:07 AM, Øyvind Harboe wrote:
    I've been looking a bit at IncrementalFaultList and I'm thinking it
    might be worthwhile to try the following implementation to
    minimize the # of objects stuck in memory:


    List l;
    query.setPageSize(1000);
    l=getDataContext().performQuery(query);

    while (l.size()>0)
    {
    t=l.remove(0);

    /* do processing here.... */
    }




    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer
  • Øyvind Harboe at Jun 20, 2008 at 8:29 am

    On Fri, Jun 20, 2008 at 10:26 AM, Andrus Adamchik wrote:
    We thought about automatically swapping out pages from the
    IncrementalFaultList as it is getting filled. At the time we had doubts
    about the efficiency of algorithms under consideration in all different
    circumstances where pagination might be used. Now that I mentioned weak
    references, this can be another algorithm for dealing with it - store each
    "page" as a weak reference, so once its objects stop being referenced, it
    will become a candidate for gc...
    Hmm.... I'd love it if adding a setPageSize() was all that was needed to
    solve these peak memory usage problems.

    I think I'll tinker a bit with a patch...


    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer
  • Andrus Adamchik at Jun 20, 2008 at 8:34 am
    If we implement swapping out pages, 'setPageSize()' will indeed be the
    only thing a user would care about.

    Andrus

    On Jun 20, 2008, at 11:29 AM, Øyvind Harboe wrote:

    On Fri, Jun 20, 2008 at 10:26 AM, Andrus Adamchik
    wrote:
    We thought about automatically swapping out pages from the
    IncrementalFaultList as it is getting filled. At the time we had
    doubts
    about the efficiency of algorithms under consideration in all
    different
    circumstances where pagination might be used. Now that I mentioned
    weak
    references, this can be another algorithm for dealing with it -
    store each
    "page" as a weak reference, so once its objects stop being
    referenced, it
    will become a candidate for gc...
    Hmm.... I'd love it if adding a setPageSize() was all that was
    needed to
    solve these peak memory usage problems.

    I think I'll tinker a bit with a patch...


    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer
  • Andrus Adamchik at Jun 20, 2008 at 8:22 am

    On Jun 20, 2008, at 10:45 AM, Øyvind Harboe wrote:

    1. Consider the code below. Will all the objects from the query
    be locked in memory?

    query=new SelectQuery(Foo.class);
    query.setPageSize(1000); /* does this matter? */
    /* Does performQuery return a clever implementation of
    a List that does not lock the CayenneDataObjects into memory?
    */
    l=context.performQuery(query);
    while (i.hasNext())
    {
    Foo f=(Foo) i.next();

    .... do some processing here that ...


    }

    2. When does peak memory usage occur above?
    At the end of the the iterator when all pages are resolved.
    3. Can a CayenneDataObject be garbage collected as long as it's
    DataContext is in scope?

    I'm wondering because as far as I have understood the two if
    statements below are equivalent when a & b are CayenneDataObjects
    in the same DataContext.
    It is stored using a weak reference, so yes, it can be garbage
    collected if there are no other references to the object in the code.
    That is if 'a' can be garbage collected during the lifetime of the
    DataContext, then 'a' is points to different objects during the
    lifetime of CayenneDataObject's and a==b just 'barely' works.
    Not "barely". It always works. If you have var "a" pointing to a given
    object, that fact alone will prevent garbage collection within the
    scope of "a" definition.

    4. Should I consider some "clever" implementation of a query that
    ensures that only a small amount of CayenneDataObject's are
    locked in memory at the time?

    Some ideas:

    - use performIteratedQuery() w/setPageSize()
    - ResultIterator.nextDataRow() is a bit strange: it returns a Map<>,
    but
    perhaps I can safely upcast that.
    - use objectFromdataRow()

    Yes, use 'performIteratedQuery' - it was designed for serial
    processing of arbitrary size lists.

    Andrus
  • Øyvind Harboe at Jun 20, 2008 at 8:32 am

    On Fri, Jun 20, 2008 at 10:22 AM, Andrus Adamchik wrote:


    On Jun 20, 2008, at 10:45 AM, Øyvind Harboe wrote:

    1. Consider the code below. Will all the objects from the query
    be locked in memory?

    query=new SelectQuery(Foo.class);
    query.setPageSize(1000); /* does this matter? */
    /* Does performQuery return a clever implementation of
    a List that does not lock the CayenneDataObjects into memory?
    */
    l=context.performQuery(query);
    while (i.hasNext())
    {
    Foo f=(Foo) i.next();

    .... do some processing here that ...


    }

    2. When does peak memory usage occur above?
    At the end of the the iterator when all pages are resolved.
    Ah! Thanks! I'll have to change the places in the code that
    does this for *large* lists then.
    3. Can a CayenneDataObject be garbage collected as long as it's
    DataContext is in scope?

    I'm wondering because as far as I have understood the two if
    statements below are equivalent when a & b are CayenneDataObjects
    in the same DataContext.
    It is stored using a weak reference, so yes, it can be garbage collected if
    there are no other references to the object in the code.
    That is if 'a' can be garbage collected during the lifetime of the
    DataContext, then 'a' is points to different objects during the
    lifetime of CayenneDataObject's and a==b just 'barely' works.
    Not "barely". It always works. If you have var "a" pointing to a given
    object, that fact alone will prevent garbage collection within the scope of
    "a" definition.
    a==b vs. q.getObjectId().equals(b.getObjectId()) was a bit of a
    non-sequitor, I'll post
    a seperate message.

    4. Should I consider some "clever" implementation of a query that
    ensures that only a small amount of CayenneDataObject's are
    locked in memory at the time?

    Some ideas:

    - use performIteratedQuery() w/setPageSize()
    - ResultIterator.nextDataRow() is a bit strange: it returns a Map<>, but
    perhaps I can safely upcast that.
    - use objectFromdataRow()

    Yes, use 'performIteratedQuery' - it was designed for serial processing of
    arbitrary size lists.
    I want to have a stab at creating a patch that has weak references for paged
    queries... it has less impact on the application code.


    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriescayenne
postedJun 20, '08 at 7:46a
activeJun 20, '08 at 8:34a
posts7
users2
websitecayenne.apache.org

2 users in discussion

Øyvind Harboe: 4 posts Andrus Adamchik: 3 posts

People

Translate

site design / logo © 2021 Grokbase