FAQ
Hi,

I have asked this question a number of ways but I still have a very
serious problem with Cayenne-specific memory management configuration
associated with Tomcat.

The problem with debugging is that given that I have very little
visibility into the Cayenne memory management it is extremely
difficult to debug this using conventional strategies. Also I am not
requesting anything from the system that is terribly exceptional so I
am attempting to use default settings as much as possible.

My strategy is to ask the experts for a Cayenne configuration and
standard memory management steps I should take to conform to the new
Cayenne memory management design intentions.

Problem:
1. I have essentially a webstore, three tier design with Tomcat,
Cayenne and MySQL.
2. When after only a few queries of products, tomcat freezes up and
reports out of memory errors.

I have attempted to configure the caching strategy ask best as I can
understand from the docks but this only gets me a few more hours of
usage before the out of memory errors. (I tried the SHARED_CACHE).
The NO_CACHE strategy is worse.

I would appreciate a set of steps (aka a primer) that should handle a
website with a lot of fetches of hundreds of data objects (i.e.
products) and very few updates.

Note: My gut feeling is that I am not properly managing the data
object array properly and it is leaking memory.

I would appreciate any input, but I would first like to know what the
minimum require steps are for managing at data object result set
ArrayList so as to properly cache and then properly free the memory
after it is no longer needed.

Context: Tomcat, MySQL, Cayenne 3.0M6

Thanks,
Joe Baldwin

Search Discussions

  • Mike Kienenberger at Sep 16, 2009 at 6:26 pm
    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with? I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request? Or at least per session?

    3) I seem to remember that the cache strategy is configurable. Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?
    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin wrote:
    Hi,

    I have asked this question a number of ways but I still have a very serious
    problem with Cayenne-specific memory management configuration associated
    with Tomcat.

    The problem with debugging is that given that I have very little visibility
    into the Cayenne memory management it is extremely difficult to debug this
    using conventional strategies.  Also I am not requesting anything from the
    system that is terribly exceptional so I am attempting to use default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and standard
    memory management steps I should take to conform to the new Cayenne memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat, Cayenne and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I can
    understand from the docks but this only gets me a few more hours of usage
    before the out of memory errors.  (I tried the SHARED_CACHE). The NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should handle a
    website with a lot of fetches of hundreds of data objects (i.e. products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what the
    minimum require steps are for managing at data object result set ArrayList
    so as to properly cache and then properly free the memory after it is no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Joe Baldwin at Sep 16, 2009 at 6:50 pm
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue. On my development box (OSX) I set it
    to -Xms128m -Xmx128m (basically arbitrary). So when I went to the
    remote hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product fetches
    and only a few product inserts and updates) and it runs out of memory
    *very* fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for 64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one. I am
    using BaseContext.getThreadObjectContext() based on recommendations (I
    converted all the old DataContext refs to BaseContext, but I don't
    really understand it from reading the docs) and am *not* releasing it
    at the end of session. Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of
    memory. I recently removed *all* the SHARED_CACHE and it ran out of
    memory very fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with? I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request? Or at least per
    session?

    3) I seem to remember that the cache strategy is configurable. Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a very
    serious
    problem with Cayenne-specific memory management configuration
    associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to
    debug this
    using conventional strategies. Also I am not requesting anything
    from the
    system that is terribly exceptional so I am attempting to use default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat,
    Cayenne and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I can
    understand from the docks but this only gets me a few more hours of
    usage
    before the out of memory errors. (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should handle a
    website with a lot of fetches of hundreds of data objects (i.e.
    products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data
    object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after it
    is no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Michael Gentry at Sep 16, 2009 at 6:57 pm
    Is your hosting company giving you a private dedicated box (or VM) or
    is your application shared with other applications running in Tomcat?
    If the latter, that would skew things, I think. Also, if you are
    running a 64-bit JVM, then it'll use more memory. It won't be 2x
    more, but it'll be more.

    In your Cayenne Model, under the DataDomain, what is the size of your
    object cache?

    On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin wrote:
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue.  On my development box (OSX) I set it to
    -Xms128m -Xmx128m (basically arbitrary).  So when I went to the remote
    hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product fetches
    and only a few product inserts and updates) and it runs out of memory *very*
    fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for 64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one.  I am using
    BaseContext.getThreadObjectContext() based on recommendations (I converted
    all the old DataContext refs to BaseContext, but I don't really understand
    it from reading the docs) and am *not* releasing it at the end of session.
    Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of memory.  I
    recently removed *all* the SHARED_CACHE and it ran out of memory very fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with?   I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request?  Or at least per session?

    3) I seem to remember that the cache strategy is configurable.  Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a very
    serious
    problem with Cayenne-specific memory management configuration associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to debug
    this
    using conventional strategies.  Also I am not requesting anything from
    the
    system that is terribly exceptional so I am attempting to use default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat, Cayenne
    and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I can
    understand from the docks but this only gets me a few more hours of usage
    before the out of memory errors.  (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should handle a
    website with a lot of fetches of hundreds of data objects (i.e. products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after it is no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Joe Baldwin at Sep 16, 2009 at 7:16 pm
    Michael,

    Thanks for responding so fast (I appreciate it).

    1. The box is shared but the Tomcat server is dedicated to my project
    only.
    2. The box is 64 bits (Linux 2.6.25-14.fc9.x86_64 (amd64))
    3. They have three plans available: 64MB, 128MB, and 256MB

    Since we only have one - two users right now I selected 64MB until it
    goes live. But based on the recommendations today, we are immediately
    upgrading to 128MB.

    FYI: I have only about 500 products (and am anticipating 5000
    products). I do not use any BLOB's in the database (we only use file
    system references to pictures and audio). So I am very concerned that
    I missed something fundamental (if you are able to handle 10,000
    objects easily).

    Questions:
    1. The WebHost POC asked me to ask you for a recommendation for Xmx.
    2. In the event I made a programming error: please let me know how to
    properly release the memory from a result set (ArrayList) in this
    scenario. I am a tad confused with the BaseContext management and
    could have made a mistake there.

    Thanks,
    Joe

    PS other than this Memory Management issue Cayenne 3M6 has been rock-
    solid!!


    On Sep 16, 2009, at 2:57 PM, Michael Gentry wrote:

    Is your hosting company giving you a private dedicated box (or VM) or
    is your application shared with other applications running in Tomcat?
    If the latter, that would skew things, I think. Also, if you are
    running a 64-bit JVM, then it'll use more memory. It won't be 2x
    more, but it'll be more.

    In your Cayenne Model, under the DataDomain, what is the size of your
    object cache?


    On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin
    wrote:
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue. On my development box (OSX) I set
    it to
    -Xms128m -Xmx128m (basically arbitrary). So when I went to the
    remote
    hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product
    fetches
    and only a few product inserts and updates) and it runs out of
    memory *very*
    fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web
    recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for
    64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my
    assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one. I am
    using
    BaseContext.getThreadObjectContext() based on recommendations (I
    converted
    all the old DataContext refs to BaseContext, but I don't really
    understand
    it from reading the docs) and am *not* releasing it at the end of
    session.
    Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of
    memory. I
    recently removed *all* the SHARED_CACHE and it ran out of memory
    very fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with? I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request? Or at least per
    session?

    3) I seem to remember that the cache strategy is configurable. Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a very
    serious
    problem with Cayenne-specific memory management configuration
    associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to
    debug
    this
    using conventional strategies. Also I am not requesting anything
    from
    the
    system that is terribly exceptional so I am attempting to use
    default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat,
    Cayenne
    and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I
    can
    understand from the docks but this only gets me a few more hours
    of usage
    before the out of memory errors. (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should
    handle a
    website with a lot of fetches of hundreds of data objects (i.e.
    products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data
    object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what
    the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after
    it is no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Michael Gentry at Sep 16, 2009 at 8:53 pm
    Joe, keep in mind that a 64-bit Linux doesn't mean you'll be running a
    64-bit JVM.

    Can you add a -Dcom.sun.management.jmxremote to your VM startup
    arguments for Tomcat and then use JConsole to connect to it?

    On Wed, Sep 16, 2009 at 3:15 PM, Joe Baldwin wrote:
    Michael,

    Thanks for responding so fast (I appreciate it).

    1. The box is shared but the Tomcat server is dedicated to my project only.
    2. The box is 64 bits (Linux 2.6.25-14.fc9.x86_64 (amd64))
    3. They have three plans available: 64MB, 128MB, and 256MB

    Since we only have one - two users right now I selected 64MB until it goes
    live.  But based on the recommendations today, we are immediately upgrading
    to 128MB.

    FYI: I have only about 500 products (and am anticipating 5000 products).  I
    do not use any BLOB's in the database (we only use file system references to
    pictures and audio).  So I am very concerned that I missed something
    fundamental (if you are able to handle 10,000 objects easily).

    Questions:
    1. The WebHost POC asked me to ask you for a recommendation for Xmx.
    2. In the event I made a programming error: please let me know how to
    properly release the memory from a result set (ArrayList) in this scenario.
    I am a tad confused with the BaseContext management and could have made a
    mistake there.

    Thanks,
    Joe

    PS other than this Memory Management issue Cayenne 3M6 has been rock-solid!!


    On Sep 16, 2009, at 2:57 PM, Michael Gentry wrote:

    Is your hosting company giving you a private dedicated box (or VM) or
    is your application shared with other applications running in Tomcat?
    If the latter, that would skew things, I think.  Also, if you are
    running a 64-bit JVM, then it'll use more memory.  It won't be 2x
    more, but it'll be more.

    In your Cayenne Model, under the DataDomain, what is the size of your
    object cache?


    On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue.  On my development box (OSX) I set it to
    -Xms128m -Xmx128m (basically arbitrary).  So when I went to the remote
    hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product
    fetches
    and only a few product inserts and updates) and it runs out of memory
    *very*
    fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for 64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one.  I am using
    BaseContext.getThreadObjectContext() based on recommendations (I
    converted
    all the old DataContext refs to BaseContext, but I don't really
    understand
    it from reading the docs) and am *not* releasing it at the end of
    session.
    Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of memory.
    I
    recently removed *all* the SHARED_CACHE and it ran out of memory very
    fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with?   I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request?  Or at least per
    session?

    3) I seem to remember that the cache strategy is configurable.  Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a very
    serious
    problem with Cayenne-specific memory management configuration
    associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to debug
    this
    using conventional strategies.  Also I am not requesting anything from
    the
    system that is terribly exceptional so I am attempting to use default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat,
    Cayenne
    and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I can
    understand from the docks but this only gets me a few more hours of
    usage
    before the out of memory errors.  (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should handle a
    website with a lot of fetches of hundreds of data objects (i.e.
    products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after it is
    no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Michael Gentry at Sep 16, 2009 at 9:29 pm
    FWIW, I just did a little monitoring with JConsole with my current
    development setup: Eclipse, Jetty, Cayenne 3.0M6, Tapesty 5.1, MySQL.
    This new application I'm working on sounds similar to yours. Fairly
    lightweight. After everything loaded in, I was using 20-21 MB of
    memory and it stayed steady, even after doing about 100 queries in
    Cayenne (I tend to pull back 1-7 records per set-of-queries, but
    closer to 2-3 on average). I'm using session-based data contexts and
    on-demand data contexts. The memory footprint was fine and I'm not
    caching (I go get fresh data every query). I'm not sure why you are
    seeing the anomaly you are seeing unless you just need a bit more RAM
    for Tomcat to be stable.

    mrg
  • Joe Baldwin at Sep 16, 2009 at 10:08 pm
    Michael,

    Thank you for your patience on this. This is the worst problem I have
    encountered and I believe that it is probably something I have
    misunderstood (and failed to implement).
    FWIW, I just did a little monitoring with JConsole with my current
    development setup: Eclipse, Jetty, Cayenne 3.0M6, Tapesty 5.1, MySQL.
    Yes I am able to monitor with JConsole on my development machine (but
    not yet on the webhost). With shared cache ON, and testing with
    50-100 concurrent users, it starts out at about 10-15 MB and then
    spikes to 70-100MB. Most of the time after the Tomcat idle period
    (i.e. 15 min) it GC down to about 20MB-15MB.

    This new application I'm working on sounds similar to yours. Fairly
    lightweight. After everything loaded in, I was using 20-21 MB of
    memory and it stayed steady, even after doing about 100 queries in
    Cayenne (I tend to pull back 1-7 records per set-of-queries, but
    closer to 2-3 on average).
    The app sounds similar, but my JConsole reports a huge spike which is
    only released after the idle period. (This is with the Cache set to ON.)

    I'm using session-based data contexts and
    on-demand data contexts. The memory footprint was fine and I'm not
    caching (I go get fresh data every query). I'm not sure why you are
    seeing the anomaly you are seeing unless you just need a bit more RAM
    for Tomcat to be stable.
    Mike Kienenberger recommended that I handle this via a filter. I must
    admit that I am still not totally comfortable with BaseContext and
    could have made a mistake. I did not want to go in this direction
    until I am sure that I have the blue-print for the correct solution.

    So how do I implement you session-based data context configuration.
    (Please send explicit code as it appears that I am just using the
    default BaseContext.)

    Thanks,
    Joe
  • Michael Gentry at Sep 17, 2009 at 1:19 pm
    Hi Joe,

    The fact that you are seeing a spike with 50-100 concurrent users
    doesn't surprise me. After your sessions timeout, the memory usage
    goes back down. This seems expected to me.

    As for my session-based filter, I'm pretty much using what Cayenne
    provides. In my web.xml file for the servlet engine (Tomcat in your
    case), I have:

    <filter>
    <filter-name>Cayenne Filter</filter-name>
    <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
    </filter>
    <filter-mapping>
    <filter-name>Cayenne Filter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    To get your context after that:

    private ObjectContext objectContext = BaseContext.getThreadObjectContext();

    I think the filter actually creates a DataContext, so you could do a cast there.

    On Wed, Sep 16, 2009 at 6:07 PM, Joe Baldwin wrote:
    Michael,

    Thank you for your patience on this.  This is the worst problem I have
    encountered and I believe that it is probably something I have misunderstood
    (and failed to implement).
    FWIW, I just did a little monitoring with JConsole with my current
    development setup: Eclipse, Jetty, Cayenne 3.0M6, Tapesty 5.1, MySQL.
    Yes I am able to monitor with JConsole on my development machine (but not
    yet on the webhost).  With shared cache ON, and testing with 50-100
    concurrent users, it starts out at about 10-15 MB and then spikes to
    70-100MB.  Most of the time after the Tomcat idle period (i.e. 15 min) it GC
    down to about 20MB-15MB.

    This new application I'm working on sounds similar to yours.  Fairly
    lightweight.  After everything loaded in, I was using 20-21 MB of
    memory and it stayed steady, even after doing about 100 queries in
    Cayenne (I tend to pull back 1-7 records per set-of-queries, but
    closer to 2-3 on average).
    The app sounds similar, but my JConsole reports a huge spike which is only
    released after the idle period. (This is with the Cache set to ON.)

    I'm using session-based data contexts and
    on-demand data contexts.  The memory footprint was fine and I'm not
    caching (I go get fresh data every query).  I'm not sure why you are
    seeing the anomaly you are seeing unless you just need a bit more RAM
    for Tomcat to be stable.
    Mike Kienenberger recommended that I handle this via a filter.  I must admit
    that I am still not totally comfortable with BaseContext and could have made
    a mistake.  I did not want to go in this direction until I am sure that I
    have the blue-print for the correct solution.

    So how do I implement you session-based data context configuration.  (Please
    send explicit code as it appears that I am just using the default
    BaseContext.)

    Thanks,
    Joe

  • Joe Baldwin at Sep 17, 2009 at 1:43 pm
    Michael,

    I just checked my web.xml and it appears am using the same filter

    <!-- Cayenne -->
    <context-param>
    <param-name>cayenne.configuration.path</param-name>
    <param-value>/WEB-INF/config/cayenne-files</param-value>
    </context-param>
    <filter>
    <filter-name>CayenneFilter</filter-name>
    <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</
    filter-class>
    </filter>
    <filter-mapping>
    <filter-name>CayenneFilter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    In the code I am accessing the BaseContext via the following call:
    ObjectContext oc = BaseContext.getThreadObjectContext();

    So, unless I have misread, it appears we are using almost exactly the
    same code.

    Questions:
    1. I am not sure what you mean by "you could do a cast there".
    2. Should I release this BaseContext and create a new one (as has been
    suggested) or should I simple rely on the BaseContext to manage the
    memory?
    3. Is there some way to message the BaseContext to determine if it has
    properly released memory after a session is complete?

    Thanks,
    Joe




    On Sep 17, 2009, at 9:18 AM, Michael Gentry wrote:

    Hi Joe,

    The fact that you are seeing a spike with 50-100 concurrent users
    doesn't surprise me. After your sessions timeout, the memory usage
    goes back down. This seems expected to me.

    As for my session-based filter, I'm pretty much using what Cayenne
    provides. In my web.xml file for the servlet engine (Tomcat in your
    case), I have:

    <filter>
    <filter-name>Cayenne Filter</filter-name>
    <filter-
    class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-
    class>
    </filter>
    <filter-mapping>
    <filter-name>Cayenne Filter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    To get your context after that:

    private ObjectContext objectContext =
    BaseContext.getThreadObjectContext();

    I think the filter actually creates a DataContext, so you could do a
    cast there.
  • Michael Gentry at Sep 17, 2009 at 1:58 pm
    1) DataContext dc = (DataContext) BaseContext.getThreadObjectContext();

    2) The BaseContext (which is really a DataContext) will stay around as
    long as the session does. I suppose you could always bind a new
    context to the thread (to allow the old one to GC) or you could
    invalidate objects when you are done using them. However, given that
    you said you are reading in relatively few records, this may not be a
    big win.

    3) Once your session is gone, it would be hard to message the context
    since it is also gone.

    On Thu, Sep 17, 2009 at 9:42 AM, Joe Baldwin wrote:
    Michael,

    I just checked my web.xml and it appears am using the same filter

    <!-- Cayenne -->
    <context-param>
    <param-name>cayenne.configuration.path</param-name>
    <param-value>/WEB-INF/config/cayenne-files</param-value>
    </context-param>
    <filter>
    <filter-name>CayenneFilter</filter-name>

    <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
    </filter>
    <filter-mapping>
    <filter-name>CayenneFilter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    In the code I am accessing the BaseContext via the following call:
    ObjectContext oc = BaseContext.getThreadObjectContext();

    So, unless I have misread, it appears we are using almost exactly the same
    code.

    Questions:
    1. I am not sure what you mean by "you could do a cast there".
    2. Should I release this BaseContext and create a new one (as has been
    suggested) or should I simple rely on the BaseContext to manage the memory?
    3. Is there some way to message the BaseContext to determine if it has
    properly released memory after a session is complete?

    Thanks,
    Joe




    On Sep 17, 2009, at 9:18 AM, Michael Gentry wrote:

    Hi Joe,

    The fact that you are seeing a spike with 50-100 concurrent users
    doesn't surprise me.  After your sessions timeout, the memory usage
    goes back down.  This seems expected to me.

    As for my session-based filter, I'm pretty much using what Cayenne
    provides.  In my web.xml file for the servlet engine (Tomcat in your
    case), I have:

    <filter>
    <filter-name>Cayenne Filter</filter-name>

    <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
    </filter>
    <filter-mapping>
    <filter-name>Cayenne Filter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    To get your context after that:

    private ObjectContext objectContext =
    BaseContext.getThreadObjectContext();

    I think the filter actually creates a DataContext, so you could do a cast
    there.
  • Joe Baldwin at Sep 17, 2009 at 2:23 pm
    Michael
    2) The BaseContext (which is really a DataContext) will stay around as
    long as the session does. I suppose you could always bind a new
    context to the thread (to allow the old one to GC) or you could
    invalidate objects when you are done using them. However, given that
    you said you are reading in relatively few records, this may not be a
    big win.
    Sorry I mislead you on this one. Even for one or two users (the
    current usage until we go live), the query result sets are returning
    about 500 products, but we anticipate scaling to 5000 in the future.
    We are only updating about 5 every day (I say this because there was a
    mention of this in the docs concerning BaseContext memory management.)

    Also, each Product has a list of Photo references (file system paths),
    Audio references, and Video references. But I think these do not trip
    the fault until they are actually viewed (if I understand the docs).

    RE bind a new context to a thread

    I do not want to do this if this is not your normal procedure. The
    problem for the customer part is that I don't know exactly where they
    get a session and so this could make things more complex.

    3) Once your session is gone, it would be hard to message the
    context since it is also gone.

    Well now that is a good point. :) Again, I was under the impression
    that the default behavior was to have only *one* BaseContext
    (DataContext) per application. But you are saying that each session
    gets a new BaseContext.

    If that is true then wouldn't all the Data Objects fetched via that
    context be released at the end of the session? If this is true then
    my analysis is *ALL* wrong. If the BaseContext goes away at the end
    of each session, and if the DataObjects go away with it, then the
    problem lies only within the session.

    Do I have this logic correct or did I misunderstand you?

    Thanks,
    Joe



    On Sep 17, 2009, at 9:58 AM, Michael Gentry wrote:

    1) DataContext dc = (DataContext) BaseContext.getThreadObjectContext
    ();

    2) The BaseContext (which is really a DataContext) will stay around as
    long as the session does. I suppose you could always bind a new
    context to the thread (to allow the old one to GC) or you could
    invalidate objects when you are done using them. However, given that
    you said you are reading in relatively few records, this may not be a
    big win.

    3) Once your session is gone, it would be hard to message the context
    since it is also gone.


    On Thu, Sep 17, 2009 at 9:42 AM, Joe Baldwin
    wrote:
    Michael,

    I just checked my web.xml and it appears am using the same filter

    <!-- Cayenne -->
    <context-param>
    <param-name>cayenne.configuration.path</param-name>
    <param-value>/WEB-INF/config/cayenne-files</param-value>
    </context-param>
    <filter>
    <filter-name>CayenneFilter</filter-name>

    <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</
    filter-class>
    </filter>
    <filter-mapping>
    <filter-name>CayenneFilter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    In the code I am accessing the BaseContext via the following call:
    ObjectContext oc = BaseContext.getThreadObjectContext();

    So, unless I have misread, it appears we are using almost exactly
    the same
    code.

    Questions:
    1. I am not sure what you mean by "you could do a cast there".
    2. Should I release this BaseContext and create a new one (as has
    been
    suggested) or should I simple rely on the BaseContext to manage the
    memory?
    3. Is there some way to message the BaseContext to determine if it
    has
    properly released memory after a session is complete?

    Thanks,
    Joe




    On Sep 17, 2009, at 9:18 AM, Michael Gentry wrote:

    Hi Joe,

    The fact that you are seeing a spike with 50-100 concurrent users
    doesn't surprise me. After your sessions timeout, the memory usage
    goes back down. This seems expected to me.

    As for my session-based filter, I'm pretty much using what Cayenne
    provides. In my web.xml file for the servlet engine (Tomcat in your
    case), I have:

    <filter>
    <filter-name>Cayenne Filter</filter-name>

    <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</
    filter-class>
    </filter>
    <filter-mapping>
    <filter-name>Cayenne Filter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    To get your context after that:

    private ObjectContext objectContext =
    BaseContext.getThreadObjectContext();

    I think the filter actually creates a DataContext, so you could do
    a cast
    there.
  • Michael Gentry at Sep 17, 2009 at 2:52 pm
    If you are fetching 500-5000 products at a time, I'd seriously
    consider using pagination since it is unlikely they will look at
    500-5000 products at a time. On your SelectQuery object, do a
    setPageSize(10) -- or some other reasonable number (how ever many
    products you show on one page). This will reduce the memory
    footprint. See:

    http://cayenne.apache.org/doc/paginated-queries.html

    With a paginated query, Cayenne fetches all of the PKs in (500-5000),
    but then only fetches one page of data objects (say 10, using the PKs
    it fetched previously) as you are looking at them, which will be more
    efficient for large sets where you don't use all of the data at one
    time.

    As for one context/application, this is not what the thread context
    (which you are using) does. The Cayenne filter creates or restores a
    session-based context that is kept around for that user for the life
    of the session. For some applications this makes perfect sense. You
    might want to keep their shopping cart objects around from
    request-to-request, for example. For a catalog type application,
    though, I personally would probably use a brand-new context in each
    request for fetching catalog data, letting it go at the end of the
    request. This will allow those contexts and data objects to be
    garbage collected much sooner. When needing to copy something from
    the temporary catalog context to the user's session context (because
    Cayenne needs related objects in the same context), you need to use
    localObject():

    http://cayenne.apache.org/doc/moving-objects-between-contexts.html

    I know that was long-winded, but I hope it gave you some ideas.

    Also, you said, "I do not want to do this if this is not your normal
    procedure." There isn't really a normal procedure. Each application
    has requirements that drive how you approach it. In one application I
    kept 10,000+ records in an application-level object that was shared by
    all sessions. These records were read-mostly and expensive to read in
    (required several minutes due to the number of joins) and I cached
    them and controlled access to them. This approach made sense -- for
    that application.

    mrg

    PS. Add the paginated queries first and see how much that buys you.
    That should be easy to do. Changing the way you are using the context
    will be much more time consuming.

    On Thu, Sep 17, 2009 at 10:22 AM, Joe Baldwin wrote:
    Michael
    2) The BaseContext (which is really a DataContext) will stay around as
    long as the session does.  I suppose you could always bind a new
    context to the thread (to allow the old one to GC) or you could
    invalidate objects when you are done using them.  However, given that
    you said you are reading in relatively few records, this may not be a
    big win.
    Sorry I mislead you on this one.  Even for one or two users (the current
    usage until we go live), the query result sets are returning about 500
    products, but we anticipate scaling to 5000 in the future.  We are only
    updating about 5 every day (I say this because there was a mention of this
    in the docs concerning BaseContext memory management.)

    Also, each Product has a list of Photo references (file system paths), Audio
    references, and Video references.  But I think these do not trip the fault
    until they are actually viewed (if I understand the docs).

    RE bind a new context to a thread

    I do not want to do this if this is not your normal procedure.  The problem
    for the customer part is that I don't know exactly where they get a session
    and so this could make things more complex.

    3) Once your session is gone, it would be hard to message the context
    since it is also gone.

    Well now that is a good point. :)   Again, I was under the impression that
    the default behavior was to have only *one* BaseContext (DataContext) per
    application.  But you are saying that each session gets a new BaseContext.

    If that is true then wouldn't all the Data Objects fetched via that context
    be released at the end of the session?  If this is true then my analysis is
    *ALL* wrong.  If the BaseContext goes away at the end of each session, and
    if the DataObjects go away with it, then the problem lies only within the
    session.

    Do I have this logic correct or did I misunderstand you?

    Thanks,
    Joe



    On Sep 17, 2009, at 9:58 AM, Michael Gentry wrote:

    1) DataContext dc = (DataContext) BaseContext.getThreadObjectContext();

    2) The BaseContext (which is really a DataContext) will stay around as
    long as the session does.  I suppose you could always bind a new
    context to the thread (to allow the old one to GC) or you could
    invalidate objects when you are done using them.  However, given that
    you said you are reading in relatively few records, this may not be a
    big win.

    3) Once your session is gone, it would be hard to message the context
    since it is also gone.


    On Thu, Sep 17, 2009 at 9:42 AM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Michael,

    I just checked my web.xml and it appears am using the same filter

    <!-- Cayenne -->
    <context-param>
    <param-name>cayenne.configuration.path</param-name>
    <param-value>/WEB-INF/config/cayenne-files</param-value>
    </context-param>
    <filter>
    <filter-name>CayenneFilter</filter-name>


    <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
    </filter>
    <filter-mapping>
    <filter-name>CayenneFilter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    In the code I am accessing the BaseContext via the following call:
    ObjectContext oc = BaseContext.getThreadObjectContext();

    So, unless I have misread, it appears we are using almost exactly the
    same
    code.

    Questions:
    1. I am not sure what you mean by "you could do a cast there".
    2. Should I release this BaseContext and create a new one (as has been
    suggested) or should I simple rely on the BaseContext to manage the
    memory?
    3. Is there some way to message the BaseContext to determine if it has
    properly released memory after a session is complete?

    Thanks,
    Joe




    On Sep 17, 2009, at 9:18 AM, Michael Gentry wrote:

    Hi Joe,

    The fact that you are seeing a spike with 50-100 concurrent users
    doesn't surprise me.  After your sessions timeout, the memory usage
    goes back down.  This seems expected to me.

    As for my session-based filter, I'm pretty much using what Cayenne
    provides.  In my web.xml file for the servlet engine (Tomcat in your
    case), I have:

    <filter>
    <filter-name>Cayenne Filter</filter-name>


    <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
    </filter>
    <filter-mapping>
    <filter-name>Cayenne Filter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    To get your context after that:

    private ObjectContext objectContext =
    BaseContext.getThreadObjectContext();

    I think the filter actually creates a DataContext, so you could do a
    cast
    there.
  • Joe Baldwin at Sep 17, 2009 at 6:18 pm
    Michael,
    With a paginated query, Cayenne fetches all of the PKs in
    (500-5000), but then only fetches one page of data objects (say 10,
    using the PKs it fetched previously) as you are looking at them,
    which will be more efficient for large sets where you don't use all
    of the data at one time.
    Yes this is a really good idea. I have been using it for the past few
    months now. However, I was also using caching (which your docs say
    interferes with BaseContext "weak reference" memory management).

    My initial plan was to cache the products that are simply being looked
    at by customers. However, I thought that might be a problem if as
    customers look different products the cache will simply grow until it
    runs out of memory. So I am not sure if this is a good idea anymore.
    Is this cache ever released after a period of time?

    Second, I am investigating a comment by the Webhost tech support, who
    said: "watch out for singletons". Well the only thing that might be a
    problem that I have done is to implement a Factory class, which is
    essentially a set of class methods that I have written to accomplish
    to most repeated fetches. Would these fetches using static methods (I
    use only local variables in these static methods) cause any problems
    with BaseContext memory managment?

    So, if these are not the problems then I guess all I can do is
    increase the Xms.

    Thanks,
    Joe







    If you are fetching 500-5000 products at a time, I'd seriously
    consider using pagination since it is unlikely they will look at
    500-5000 products at a time. On your SelectQuery object, do a
    setPageSize(10) -- or some other reasonable number (how ever many
    products you show on one page). This will reduce the memory
    footprint. See:

    http://cayenne.apache.org/doc/paginated-queries.html

    With a paginated query, Cayenne fetches all of the PKs in (500-5000),
    but then only fetches one page of data objects (say 10, using the PKs
    it fetched previously) as you are looking at them, which will be more
    efficient for large sets where you don't use all of the data at one
    time.

    As for one context/application, this is not what the thread context
    (which you are using) does. The Cayenne filter creates or restores a
    session-based context that is kept around for that user for the life
    of the session. For some applications this makes perfect sense. You
    might want to keep their shopping cart objects around from
    request-to-request, for example. For a catalog type application,
    though, I personally would probably use a brand-new context in each
    request for fetching catalog data, letting it go at the end of the
    request. This will allow those contexts and data objects to be
    garbage collected much sooner. When needing to copy something from
    the temporary catalog context to the user's session context (because
    Cayenne needs related objects in the same context), you need to use
    localObject():

    http://cayenne.apache.org/doc/moving-objects-between-contexts.html

    I know that was long-winded, but I hope it gave you some ideas.

    Also, you said, "I do not want to do this if this is not your normal
    procedure." There isn't really a normal procedure. Each application
    has requirements that drive how you approach it. In one application I
    kept 10,000+ records in an application-level object that was shared by
    all sessions. These records were read-mostly and expensive to read in
    (required several minutes due to the number of joins) and I cached
    them and controlled access to them. This approach made sense -- for
    that application.

    mrg

    PS. Add the paginated queries first and see how much that buys you.
    That should be easy to do. Changing the way you are using the context
    will be much more time consuming.
  • Michael Gentry at Sep 17, 2009 at 6:28 pm
    What is the size of your object cache? Look under the DataDomain. If
    it is a large number, try lowering it.

    On Thu, Sep 17, 2009 at 2:18 PM, Joe Baldwin wrote:
    Michael,
    With a paginated query, Cayenne fetches all of the PKs in (500-5000), but
    then only fetches one page of data objects (say 10, using the PKs it fetched
    previously) as you are looking at them, which will be more efficient for
    large sets where you don't use all of the data at one time.
    Yes this is a really good idea.  I have been using it for the past few
    months now.  However, I was also using caching (which your docs say
    interferes with BaseContext "weak reference" memory management).

    My initial plan was to cache the products that are simply being looked at by
    customers.  However, I thought that might be a problem if as customers look
    different products the cache will simply grow until it runs out of memory.
    So I am not sure if this is a good idea anymore.  Is this cache ever
    released after a period of time?

    Second, I am investigating a comment by the Webhost tech support, who said:
    "watch out for singletons".  Well the only thing that might be a problem
    that I have done is to implement a Factory class, which is essentially a set
    of class methods that I have written to accomplish to most repeated fetches.
    Would these fetches using static methods (I use only local variables in
    these static methods) cause any problems with BaseContext memory managment?

    So, if these are not the problems then I guess all I can do is increase the
    Xms.

    Thanks,
    Joe







    If you are fetching 500-5000 products at a time, I'd seriously
    consider using pagination since it is unlikely they will look at
    500-5000 products at a time.  On your SelectQuery object, do a
    setPageSize(10) -- or some other reasonable number (how ever many
    products you show on one page).  This will reduce the memory
    footprint.  See:

    http://cayenne.apache.org/doc/paginated-queries.html

    With a paginated query, Cayenne fetches all of the PKs in (500-5000),
    but then only fetches one page of data objects (say 10, using the PKs
    it fetched previously) as you are looking at them, which will be more
    efficient for large sets where you don't use all of the data at one
    time.

    As for one context/application, this is not what the thread context
    (which you are using) does.  The Cayenne filter creates or restores a
    session-based context that is kept around for that user for the life
    of the session.  For some applications this makes perfect sense.  You
    might want to keep their shopping cart objects around from
    request-to-request, for example.  For a catalog type application,
    though, I personally would probably use a brand-new context in each
    request for fetching catalog data, letting it go at the end of the
    request.  This will allow those contexts and data objects to be
    garbage collected much sooner.  When needing to copy something from
    the temporary catalog context to the user's session context (because
    Cayenne needs related objects in the same context), you need to use
    localObject():

    http://cayenne.apache.org/doc/moving-objects-between-contexts.html

    I know that was long-winded, but I hope it gave you some ideas.

    Also, you said, "I do not want to do this if this is not your normal
    procedure."  There isn't really a normal procedure.  Each application
    has requirements that drive how you approach it.  In one application I
    kept 10,000+ records in an application-level object that was shared by
    all sessions.  These records were read-mostly and expensive to read in
    (required several minutes due to the number of joins) and I cached
    them and controlled access to them.  This approach made sense -- for
    that application.

    mrg

    PS. Add the paginated queries first and see how much that buys you.
    That should be easy to do.  Changing the way you are using the context
    will be much more time consuming.
  • Joe Baldwin at Sep 17, 2009 at 6:41 pm
    Michael
    What is the size of your object cache? Look under the DataDomain. If
    it is a large number, try lowering it.

    Query Cache Factory: org.apache.cayenne.cache.MapQueryCacheFactory
    Size of Object Cache: 10000
    Use Shared Cache: checked

    I do not know if 10,000 is small, medium, or large.

    What do you recommend?

    Joe
  • Michael Gentry at Sep 17, 2009 at 6:53 pm
    That's probably bigger than you want. Try 500 just to see how it
    works. Adjusting that number is black magic. :-)

    On Thu, Sep 17, 2009 at 2:40 PM, Joe Baldwin wrote:
    Michael
    What is the size of your object cache?  Look under the DataDomain.  If
    it is a large number, try lowering it.

    Query Cache Factory: org.apache.cayenne.cache.MapQueryCacheFactory
    Size of Object Cache: 10000
    Use Shared Cache: checked

    I do not know if 10,000 is small, medium, or large.

    What do you recommend?

    Joe
  • Joe Baldwin at Sep 17, 2009 at 7:02 pm
    500? So you are saying I am off by two orders of magnitude.

    OK.

    I will let you know what the cauldron conjures up.




    On Sep 17, 2009, at 2:53 PM, Michael Gentry wrote:

    That's probably bigger than you want. Try 500 just to see how it
    works. Adjusting that number is black magic. :-)


    On Thu, Sep 17, 2009 at 2:40 PM, Joe Baldwin
    wrote:
    Michael
    What is the size of your object cache? Look under the
    DataDomain. If
    it is a large number, try lowering it.

    Query Cache Factory: org.apache.cayenne.cache.MapQueryCacheFactory
    Size of Object Cache: 10000
    Use Shared Cache: checked

    I do not know if 10,000 is small, medium, or large.

    What do you recommend?

    Joe
  • Michael Gentry at Sep 17, 2009 at 7:55 pm
    Well, two orders of magnitude would be 50,000. :-)

    I just made 500 up. Maybe 2500 is a good number for you. Maybe 50.
    Probably not 50,000. Like I said, it is kind of black magic and
    depends a lot on your requirements and application. Part of your
    requirements are to be leaner in terms of memory usage, so you need to
    make the cache smaller.

    Maybe this page will help a little?

    http://cayenne.apache.org/doc/individual-object-caching.html

    On Thu, Sep 17, 2009 at 3:01 PM, Joe Baldwin wrote:
    500?  So you are saying I am off by two orders of magnitude.

    OK.

    I will let you know what the cauldron conjures up.




    On Sep 17, 2009, at 2:53 PM, Michael Gentry wrote:

    That's probably bigger than you want.  Try 500 just to see how it
    works.  Adjusting that number is black magic.  :-)


    On Thu, Sep 17, 2009 at 2:40 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Michael
    What is the size of your object cache?  Look under the DataDomain.  If
    it is a large number, try lowering it.

    Query Cache Factory: org.apache.cayenne.cache.MapQueryCacheFactory
    Size of Object Cache: 10000
    Use Shared Cache: checked

    I do not know if 10,000 is small, medium, or large.

    What do you recommend?

    Joe
  • Joe Baldwin at Sep 17, 2009 at 7:28 pm
    Michael,

    BTW, what symptoms will I see if the Object Cache size is too small?

    Is this sort of a temporary cache? So if I have the page size set to
    50, then can I assume that I have 10 pages in the temporary cache?

    Thanks,
    Joe


    On Sep 17, 2009, at 2:53 PM, Michael Gentry wrote:

    That's probably bigger than you want. Try 500 just to see how it
    works. Adjusting that number is black magic. :-)


    On Thu, Sep 17, 2009 at 2:40 PM, Joe Baldwin
    wrote:
    Michael
    What is the size of your object cache? Look under the
    DataDomain. If
    it is a large number, try lowering it.

    Query Cache Factory: org.apache.cayenne.cache.MapQueryCacheFactory
    Size of Object Cache: 10000
    Use Shared Cache: checked

    I do not know if 10,000 is small, medium, or large.

    What do you recommend?

    Joe
  • Aristedes Maniatis at Sep 18, 2009 at 3:54 am
    To remove some of the black magic can I suggest a tool like YourKit? It is very helpful for figuring out where all your memory goes.

    Also, consider a common approach is to have one 'read-only' [1] shared context which persists for the life of the application and is used to show data (products, etc) to users. Then a small context created per session which contains just the objects which that user is modifying (invoice, invoice lines, payment, contact, etc) within the session. Certainly that isn't the only approach, but it is pretty common.

    Ari Maniatis


    [1] There is no such thing as a 'read-only' context in Cayenne. Read-only just describes the way you use it.


    On 18/09/09 4:53 AM, Michael Gentry wrote:
    That's probably bigger than you want. Try 500 just to see how it
    works. Adjusting that number is black magic. :-)


    On Thu, Sep 17, 2009 at 2:40 PM, Joe Baldwinwrote:
    Michael
    What is the size of your object cache? Look under the DataDomain. If
    it is a large number, try lowering it.

    Query Cache Factory: org.apache.cayenne.cache.MapQueryCacheFactory
    Size of Object Cache: 10000
    Use Shared Cache: checked

    I do not know if 10,000 is small, medium, or large.

    What do you recommend?

    Joe
    --

    -------------------------->
    Aristedes Maniatis
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
  • Joe Baldwin at Sep 18, 2009 at 4:11 am
    Ari,

    thanks for the YourKit suggestion.
    Also, consider a common approach is to have one 'read-only' [1]
    shared context which persists for the life of the application and is
    used to show data (products, etc) to users. Then a small context
    created per session which contains just the objects which that user
    is modifying (invoice, invoice lines, payment, contact, etc) within
    the session. Certainly that isn't the only approach, but it is
    pretty common.
    That is pretty much what I was wanting to do (after listening to all
    the input). However, I am not sure how to implement this design in a
    webapp with session and a Cayenne filter that is creating the
    DataContext for me (automagically as they say).

    I suspect if I implement this sort of simple design (as about 95% of
    the result sets will be "read-only"), I could probably manage the
    memory much better.

    So how do I create a shared context? Is this going to be associated
    with an Application scoped singleton? Also how do I create the smaller
    contexts and make sure they are GC'd after the session is terminated?

    Thanks for the input,
    Joe



    On Sep 17, 2009, at 11:53 PM, Aristedes Maniatis wrote:

    To remove some of the black magic can I suggest a tool like YourKit?
    It is very helpful for figuring out where all your memory goes.

    Also, consider a common approach is to have one 'read-only' [1]
    shared context which persists for the life of the application and is
    used to show data (products, etc) to users. Then a small context
    created per session which contains just the objects which that user
    is modifying (invoice, invoice lines, payment, contact, etc) within
    the session. Certainly that isn't the only approach, but it is
    pretty common.

    Ari Maniatis


    [1] There is no such thing as a 'read-only' context in Cayenne. Read-
    only just describes the way you use it.


    On 18/09/09 4:53 AM, Michael Gentry wrote:
    That's probably bigger than you want. Try 500 just to see how it
    works. Adjusting that number is black magic. :-)


    On Thu, Sep 17, 2009 at 2:40 PM, Joe
    Baldwinwrote:
    Michael
    What is the size of your object cache? Look under the
    DataDomain. If
    it is a large number, try lowering it.

    Query Cache Factory: org.apache.cayenne.cache.MapQueryCacheFactory
    Size of Object Cache: 10000
    Use Shared Cache: checked

    I do not know if 10,000 is small, medium, or large.

    What do you recommend?

    Joe
    --

    -------------------------->
    Aristedes Maniatis
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
  • Aristedes Maniatis at Sep 18, 2009 at 4:47 am

    On 18/09/09 2:10 PM, Joe Baldwin wrote:
    So how do I create a shared context? Is this going to be associated with
    an Application scoped singleton? Also how do I create the smaller
    contexts and make sure they are GC'd after the session is terminated?
    Sorry I can't help with this one since our Cayenne application is actually a Swing client/server application and not a J2EE style web application you are creating. But I'm sure someone will have a recipie for you and to help others I'd like to encourage you to document your experiences on our wiki once you have it working nicely.

    Regards
    Ari

    --

    -------------------------->
    Aristedes Maniatis
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
  • Michael Gentry at Sep 18, 2009 at 1:24 pm

    On Fri, Sep 18, 2009 at 12:10 AM, Joe Baldwin wrote:
    That is pretty much what I was wanting to do (after listening to all the
    input). However, I am not sure how to implement this design in a webapp with
    session and a Cayenne filter that is creating the DataContext for me
    (automagically as they say).

    The one created through the Cayenne filter is the session-based one.
    Cayenne doesn't stop you from creating your own DataContexts to use
    elsewhere (that are separate from the session-based one):

    DataContext dc = DataContext.createDataContext();

    I suspect if I implement this sort of simple design (as about 95% of the
    result sets will be "read-only"), I could probably manage the memory much
    better.

    So how do I create a shared context? Is this going to be associated with an
    Application scoped singleton?

    I used a singleton that controlled access to my shared objects. I
    don't know how well that will scale since I didn't have to worry about
    scalability (I had 3-5 users searching large amounts of data). I also
    don't remember if DataContext is thread safe (what would happen it two
    different threads did a performQuery() on the same DataContext). I'm
    pretty sure I made my access methods on my singleton "synchronized" to
    keep it safe. (Again, don't know how well that will scale.)

    Also how do I create the smaller contexts and
    make sure they are GC'd after the session is terminated?

    DataContext dc = DataContext.createDataContext();


    The easiest (and probably cheapest at this point) thing is still going
    to be to add RAM. Seriously, what is the price difference between 128
    MB and 256 MB with your hosting provider? And then compare that
    difference to the cost of man hours trying to shave a few MB off your
    memory footprint. If it takes you 4 man weeks to shave 50 MB (wild
    guess) off your memory footprint, how many months of hosting at 256 MB
    vs 128 MB will that buy you?
  • Joe Baldwin at Sep 18, 2009 at 2:05 pm
    Michael,

    You have made some good points. I was reconsidering my design last
    night and coincidentally you came up with some of the same criticisms
    that I did (and more).
    I used a singleton that controlled access to my shared objects. I
    don't know how well that will scale since I didn't have to worry
    about scalability (I had 3-5 users searching large amounts of
    data). I also don't remember if DataContext is thread safe (what
    would happen it two different threads did a performQuery() on the
    same DataContext). I'm pretty sure I made my access methods on my
    singleton "synchronized" to keep it safe. (Again, don't know how
    well that will scale.)

    I was reconsidering this approach last night as well. I pictured
    having an Application scoped DataContext with 500 and it seemed
    reasonable, but then when I pictured one with 5000 products (and
    possibly faults associated with associated DataObjects) it started to
    appear to be not such a good design approach.
    The easiest (and probably cheapest at this point) thing is still
    going to be to add RAM.

    Well, I think I agree with you and bumping the Tomcat Xmx up to 256MB
    is the first thing I am going to change. I also think that I am going
    to go with your idea of not using a cache strategy on the large result
    sets (like the product searches). I am going to continue to use
    paging of 50 objects similar to what you are using.

    Questions:
    1. Please let me know if this last paragraph sounds reasonable or if I
    missed any of your recommendations.
    2. Could you please verify for me how to properly release the memory
    of a "read-only" result set within the scope of one session. (i.e.
    There might be 10-20 fetches and each new one will render the previous
    one unnecessary, so I would like to GC it on the fly, unless that is a
    bad idea). So with the new BaseContext weak references do I need to
    simply set the reference to the ArrayList (aka result set) to null, or
    do I need to explicitly release the DataObjects some other way.

    Thanks for all the help, I think these last two questions should be it.
    Joe





    On Sep 18, 2009, at 9:24 AM, Michael Gentry wrote:

    On Fri, Sep 18, 2009 at 12:10 AM, Joe Baldwin
    wrote:
    That is pretty much what I was wanting to do (after listening to
    all the
    input). However, I am not sure how to implement this design in a
    webapp with
    session and a Cayenne filter that is creating the DataContext for me
    (automagically as they say).

    The one created through the Cayenne filter is the session-based one.
    Cayenne doesn't stop you from creating your own DataContexts to use
    elsewhere (that are separate from the session-based one):

    DataContext dc = DataContext.createDataContext();

    I suspect if I implement this sort of simple design (as about 95%
    of the
    result sets will be "read-only"), I could probably manage the
    memory much
    better.

    So how do I create a shared context? Is this going to be associated
    with an
    Application scoped singleton?

    I used a singleton that controlled access to my shared objects. I
    don't know how well that will scale since I didn't have to worry about
    scalability (I had 3-5 users searching large amounts of data). I also
    don't remember if DataContext is thread safe (what would happen it two
    different threads did a performQuery() on the same DataContext). I'm
    pretty sure I made my access methods on my singleton "synchronized" to
    keep it safe. (Again, don't know how well that will scale.)

    Also how do I create the smaller contexts and
    make sure they are GC'd after the session is terminated?

    DataContext dc = DataContext.createDataContext();


    The easiest (and probably cheapest at this point) thing is still going
    to be to add RAM. Seriously, what is the price difference between 128
    MB and 256 MB with your hosting provider? And then compare that
    difference to the cost of man hours trying to shave a few MB off your
    memory footprint. If it takes you 4 man weeks to shave 50 MB (wild
    guess) off your memory footprint, how many months of hosting at 256 MB
    vs 128 MB will that buy you?
  • Michael Gentry at Sep 18, 2009 at 2:47 pm
    1. Sounded like a good first step.
    2. You have two basic options. Kick the objects out of the
    DataContext or replace the entire DataContext.

    To kick the objects out of the DataContext, I think you want
    unregisterObjects(Collection objects), but it might be
    invalidateObjects(Collection objects). I think unregisterObjects()
    gives them the boot and invalidateObjects() keeps them around, but
    flushes the cache (they become HOLLOW). I could have that wrong,
    though. :-) I'd try unregisterObjects() first.

    If you are done with everything in the thread/session DataContext, you
    can always replace it.
    BaseContext.bindThreadObjectContext(DataContext.createDataContext())
    or BaseContext.bindThreadObjectContext(null). Setting it to null
    might not be as safe if anything needs it after you null it out in
    that request.

    I suspect the latter will be the easiest for you ...


    On Fri, Sep 18, 2009 at 10:04 AM, Joe Baldwin wrote:
    Michael,

    You have made some good points.  I was reconsidering my design last night
    and coincidentally you came up with some of the same criticisms that I did
    (and more).
    I used a singleton that controlled access to my shared objects.  I don't
    know how well that will scale since I didn't have to worry about scalability
    (I had 3-5 users searching large amounts of data).  I also don't remember if
    DataContext is thread safe (what would happen it two different threads did a
    performQuery() on the same DataContext).  I'm pretty sure I made my access
    methods on my singleton "synchronized" to keep it safe.  (Again, don't know
    how well that will scale.)

    I was reconsidering this approach last night as well.  I pictured having an
    Application scoped DataContext with 500 and it seemed reasonable, but then
    when I pictured one with 5000 products (and possibly faults associated with
    associated DataObjects) it started to appear to be not such a good design
    approach.
    The easiest (and probably cheapest at this point) thing is still going to
    be to add RAM.

    Well, I think I agree with you and bumping the Tomcat Xmx up to 256MB  is
    the first thing I am going to change.  I also think that I am going to go
    with your idea of not using a cache strategy on the large result sets (like
    the product searches).  I am going to continue to use paging of 50 objects
    similar to what you are using.

    Questions:
    1. Please let me know if this last paragraph sounds reasonable or if I
    missed any of your recommendations.
    2. Could you please verify for me how to properly release the memory of a
    "read-only" result set within the scope of one session. (i.e. There might be
    10-20 fetches and each new one will render the previous one unnecessary, so
    I would like to GC it on the fly, unless that is a bad idea).  So with the
    new BaseContext weak references do I need to simply set the reference to the
    ArrayList (aka result set) to null, or do I need to explicitly release the
    DataObjects some other way.

    Thanks for all the help, I think these last two questions should be it.
    Joe





    On Sep 18, 2009, at 9:24 AM, Michael Gentry wrote:

    On Fri, Sep 18, 2009 at 12:10 AM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    That is pretty much what I was wanting to do (after listening to all the
    input). However, I am not sure how to implement this design in a webapp
    with
    session and a Cayenne filter that is creating the DataContext for me
    (automagically as they say).

    The one created through the Cayenne filter is the session-based one.
    Cayenne doesn't stop you from creating your own DataContexts to use
    elsewhere (that are separate from the session-based one):

    DataContext dc = DataContext.createDataContext();

    I suspect if I implement this sort of simple design (as about 95% of the
    result sets will be "read-only"), I could probably manage the memory much
    better.

    So how do I create a shared context? Is this going to be associated with
    an
    Application scoped singleton?

    I used a singleton that controlled access to my shared objects.  I
    don't know how well that will scale since I didn't have to worry about
    scalability (I had 3-5 users searching large amounts of data).  I also
    don't remember if DataContext is thread safe (what would happen it two
    different threads did a performQuery() on the same DataContext).  I'm
    pretty sure I made my access methods on my singleton "synchronized" to
    keep it safe.  (Again, don't know how well that will scale.)

    Also how do I create the smaller contexts and
    make sure they are GC'd after the session is terminated?

    DataContext dc = DataContext.createDataContext();


    The easiest (and probably cheapest at this point) thing is still going
    to be to add RAM.  Seriously, what is the price difference between 128
    MB and 256 MB with your hosting provider?  And then compare that
    difference to the cost of man hours trying to shave a few MB off your
    memory footprint.  If it takes you 4 man weeks to shave 50 MB (wild
    guess) off your memory footprint, how many months of hosting at 256 MB
    vs 128 MB will that buy you?
  • Andrus Adamchik at Sep 21, 2009 at 2:29 pm

    On Sep 18, 2009, at 6:53 AM, Aristedes Maniatis wrote:
    To remove some of the black magic can I suggest a tool like YourKit?
    I second this suggestion. Without a profiler you'd have to have a
    really good intuition to guess where the memory leaks are
    accumulating. It can really be anything...

    Andrus
  • Mike Kienenberger at Sep 16, 2009 at 7:11 pm
    1.) 128 still seems small to me. I don't think I run anything at
    less than 256.
    On the other hand, We have an app with 1000s of customers that uses
    512Mb, I think. So 1500 seems excessive.

    2.) BaseContext.getThreadObjectContext() just tells how you're
    getting a context. It doesn't tell how it's managed. Do you have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end? Or does it do this per
    session? I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter). That could be part of
    the problem.
    On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin wrote:
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue.  On my development box (OSX) I set it to
    -Xms128m -Xmx128m (basically arbitrary).  So when I went to the remote
    hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product fetches
    and only a few product inserts and updates) and it runs out of memory *very*
    fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for 64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one.  I am using
    BaseContext.getThreadObjectContext() based on recommendations (I converted
    all the old DataContext refs to BaseContext, but I don't really understand
    it from reading the docs) and am *not* releasing it at the end of session.
    Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of memory.  I
    recently removed *all* the SHARED_CACHE and it ran out of memory very fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with?   I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request?  Or at least per session?

    3) I seem to remember that the cache strategy is configurable.  Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a very
    serious
    problem with Cayenne-specific memory management configuration associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to debug
    this
    using conventional strategies.  Also I am not requesting anything from
    the
    system that is terribly exceptional so I am attempting to use default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat, Cayenne
    and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I can
    understand from the docks but this only gets me a few more hours of usage
    before the out of memory errors.  (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should handle a
    website with a lot of fetches of hundreds of data objects (i.e. products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after it is no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Joe Baldwin at Sep 16, 2009 at 7:30 pm
    Mike,

    RE BaseContext.getThreadObjectContext()

    Of course, this could be my problem. I was using DataContext until 3.0
    then converted over to BaseContext.

    2.) BaseContext.getThreadObjectContext() just tells how you're
    getting a context. It doesn't tell how it's managed. Do you have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end? Or does it do this per
    session?
    I do not know how it is being initialized in the webapp (in my
    experimental non-web apps I explicitly intitialize it, but in the
    webapp it is already initialized).

    I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter). That could be part of
    the problem.

    That sounds plausible. I could not find an example of how to
    initialize this and manage it as you suggest in a web app. I have not
    created a servlet filter but have follow the Cayenne docs for
    configuration of the web.xml.

    If this is insufficient then I agree, this could be the problem.
    Unfortunately, I have not found docs on how to accomplish what you are
    recommending.

    Joe



    On Sep 16, 2009, at 3:10 PM, Mike Kienenberger wrote:

    1.) 128 still seems small to me. I don't think I run anything at
    less than 256.
    On the other hand, We have an app with 1000s of customers that uses
    512Mb, I think. So 1500 seems excessive.

    2.) BaseContext.getThreadObjectContext() just tells how you're
    getting a context. It doesn't tell how it's managed. Do you have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end? Or does it do this per
    session? I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter). That could be part of
    the problem.

    On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin
    wrote:
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue. On my development box (OSX) I set
    it to
    -Xms128m -Xmx128m (basically arbitrary). So when I went to the
    remote
    hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product
    fetches
    and only a few product inserts and updates) and it runs out of
    memory *very*
    fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web
    recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for
    64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my
    assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one. I am
    using
    BaseContext.getThreadObjectContext() based on recommendations (I
    converted
    all the old DataContext refs to BaseContext, but I don't really
    understand
    it from reading the docs) and am *not* releasing it at the end of
    session.
    Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of
    memory. I
    recently removed *all* the SHARED_CACHE and it ran out of memory
    very fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with? I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request? Or at least per
    session?

    3) I seem to remember that the cache strategy is configurable. Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a very
    serious
    problem with Cayenne-specific memory management configuration
    associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to
    debug
    this
    using conventional strategies. Also I am not requesting anything
    from
    the
    system that is terribly exceptional so I am attempting to use
    default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat,
    Cayenne
    and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I
    can
    understand from the docks but this only gets me a few more hours
    of usage
    before the out of memory errors. (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should
    handle a
    website with a lot of fetches of hundreds of data objects (i.e.
    products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data
    object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what
    the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after
    it is no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Mike Kienenberger at Sep 16, 2009 at 7:38 pm
    I think there's a default filter provided by Cayenne you can specify
    in your config file, but it really comes down to something as simple
    as this to make it per request:

    public void doFilter(ServletRequest servletRequest,
    ServletResponse servletResponse, FilterChain chain)
    throws IOException, ServletException
    {
    // set base context on thread

    chain.doFilter(servletRequest, servletResponse);

    // remove base context from thread
    }

    Attaching a simple one I wrote a long time ago that perserves the
    DataContext across a session (but insures it's in a clean state at the
    end of each request).

    On Wed, Sep 16, 2009 at 3:29 PM, Joe Baldwin wrote:
    Mike,

    RE BaseContext.getThreadObjectContext()

    Of course, this could be my problem. I was using DataContext until 3.0 then
    converted over to BaseContext.

    2.)  BaseContext.getThreadObjectContext() just tells how you're
    getting a context.  It doesn't tell how it's managed.   Do you have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end?   Or does it do this per
    session?
    I do not know how it is being initialized in the webapp (in my experimental
    non-web apps I explicitly intitialize it, but in the webapp it is already
    initialized).

    I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter).   That could be part of
    the problem.

    That sounds plausible.  I could not find an example of how to initialize
    this and manage it as you suggest in a web app.  I have not created a
    servlet filter but have follow the Cayenne docs for configuration of the
    web.xml.

    If this is insufficient then I agree, this could be the problem.
    Unfortunately, I have not found docs on how to accomplish what you are
    recommending.

    Joe



    On Sep 16, 2009, at 3:10 PM, Mike Kienenberger wrote:

    1.)  128 still seems small to me.   I don't think I run anything at
    less than 256.
    On the other hand, We have an app with 1000s of customers that uses
    512Mb, I think.   So 1500 seems excessive.

    2.)  BaseContext.getThreadObjectContext() just tells how you're
    getting a context.  It doesn't tell how it's managed.   Do you have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end?   Or does it do this per
    session?   I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter).   That could be part of
    the problem.

    On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue.  On my development box (OSX) I set it to
    -Xms128m -Xmx128m (basically arbitrary).  So when I went to the remote
    hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product
    fetches
    and only a few product inserts and updates) and it runs out of memory
    *very*
    fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for 64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one.  I am using
    BaseContext.getThreadObjectContext() based on recommendations (I
    converted
    all the old DataContext refs to BaseContext, but I don't really
    understand
    it from reading the docs) and am *not* releasing it at the end of
    session.
    Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of memory.
    I
    recently removed *all* the SHARED_CACHE and it ran out of memory very
    fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with?   I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request?  Or at least per
    session?

    3) I seem to remember that the cache strategy is configurable.  Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a very
    serious
    problem with Cayenne-specific memory management configuration
    associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to debug
    this
    using conventional strategies.  Also I am not requesting anything from
    the
    system that is terribly exceptional so I am attempting to use default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat,
    Cayenne
    and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I can
    understand from the docks but this only gets me a few more hours of
    usage
    before the out of memory errors.  (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should handle a
    website with a lot of fetches of hundreds of data objects (i.e.
    products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after it is
    no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Joe Baldwin at Sep 16, 2009 at 8:40 pm
    Mike,

    I looked through your code for the filter and have a few questions.

    1. How does this filter differ from the default Cayenne filter?
    2. It appears that your filter is doing a similar task to Cayenne
    filter. The docs say:
    A Servlet Filter that binds session DataContext to the current
    request thread. During the request application code
    without any knowledge of the servlet environment can access DataContext
    viaDataContext.getThreadDataContext() method.

    Is this correct?
    3. I have never written a filter and want to make sure that this is
    the problem before attempting this kind of change.

    It appears that your strategy is to create a new DataContext (or
    BaseContext) for each session. If so, then I am confused because I
    thought that the new Memory Management strategy for Cayenne 3.0 was to
    avoid having to do this.

    Am I missing something about the basic usage of Cayenne?

    Thanks,
    Joe


    On Sep 16, 2009, at 3:37 PM, Mike Kienenberger wrote:

    I think there's a default filter provided by Cayenne you can specify
    in your config file, but it really comes down to something as simple
    as this to make it per request:

    public void doFilter(ServletRequest servletRequest,
    ServletResponse servletResponse, FilterChain chain)
    throws IOException, ServletException
    {
    // set base context on thread

    chain.doFilter(servletRequest, servletResponse);

    // remove base context from thread
    }

    Attaching a simple one I wrote a long time ago that perserves the
    DataContext across a session (but insures it's in a clean state at the
    end of each request).


    On Wed, Sep 16, 2009 at 3:29 PM, Joe Baldwin
    wrote:
    Mike,

    RE BaseContext.getThreadObjectContext()

    Of course, this could be my problem. I was using DataContext until
    3.0 then
    converted over to BaseContext.

    2.) BaseContext.getThreadObjectContext() just tells how you're
    getting a context. It doesn't tell how it's managed. Do you
    have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end? Or does it do this per
    session?
    I do not know how it is being initialized in the webapp (in my
    experimental
    non-web apps I explicitly intitialize it, but in the webapp it is
    already
    initialized).

    I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter). That could be part
    of
    the problem.

    That sounds plausible. I could not find an example of how to
    initialize
    this and manage it as you suggest in a web app. I have not created a
    servlet filter but have follow the Cayenne docs for configuration
    of the
    web.xml.

    If this is insufficient then I agree, this could be the problem.
    Unfortunately, I have not found docs on how to accomplish what you
    are
    recommending.

    Joe



    On Sep 16, 2009, at 3:10 PM, Mike Kienenberger wrote:

    1.) 128 still seems small to me. I don't think I run anything at
    less than 256.
    On the other hand, We have an app with 1000s of customers that uses
    512Mb, I think. So 1500 seems excessive.

    2.) BaseContext.getThreadObjectContext() just tells how you're
    getting a context. It doesn't tell how it's managed. Do you
    have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end? Or does it do this per
    session? I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter). That could be part
    of
    the problem.

    On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin <jfbaldwin@earthlink.net
    wrote:
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue. On my development box (OSX) I set
    it to
    -Xms128m -Xmx128m (basically arbitrary). So when I went to the
    remote
    hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product
    fetches
    and only a few product inserts and updates) and it runs out of
    memory
    *very*
    fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web
    recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for
    64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my
    assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one. I
    am using
    BaseContext.getThreadObjectContext() based on recommendations (I
    converted
    all the old DataContext refs to BaseContext, but I don't really
    understand
    it from reading the docs) and am *not* releasing it at the end of
    session.
    Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of
    memory.
    I
    recently removed *all* the SHARED_CACHE and it ran out of memory
    very
    fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to
    start
    with? I don't know what the default is these days, but in the
    old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request? Or at least per
    session?

    3) I seem to remember that the cache strategy is configurable.
    Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a
    very
    serious
    problem with Cayenne-specific memory management configuration
    associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to
    debug
    this
    using conventional strategies. Also I am not requesting
    anything from
    the
    system that is terribly exceptional so I am attempting to use
    default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new
    Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat,
    Cayenne
    and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up
    and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as
    I can
    understand from the docks but this only gets me a few more
    hours of
    usage
    before the out of memory errors. (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should
    handle a
    website with a lot of fetches of hundreds of data objects (i.e.
    products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the
    data object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know
    what the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after
    it is
    no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
    <DataContextManagerFilter.java>
  • Mike Kienenberger at Sep 16, 2009 at 8:51 pm
    That filter is for an old cayenne project (1.2?)

    Mine primarily differs in that it doesn't just insure that the session
    Context is bound but it also verifies that the context is in a clean
    state after each request and dumps the dirty objects if not.

    If you wanted a simple request-based DataContext, the filter would
    probably be only a couple of lines of code: create context and put in
    thread, call filter chain, remove context from thread.
    On Wed, Sep 16, 2009 at 4:39 PM, Joe Baldwin wrote:
    Mike,

    I looked through your code for the filter and have a few questions.

    1. How does this filter differ from the default Cayenne filter?
    2. It appears that your filter is doing a similar task to Cayenne filter.
    The docs say:
    A Servlet Filter that binds session DataContext to the current
    request thread. During the request application code
    without any knowledge of the servlet environment can access
    DataContext
    viaDataContext.getThreadDataContext() method.

    Is this correct?
    3. I have never written a filter and want to make sure that this is the
    problem before attempting this kind of change.

    It appears that your strategy is to create a new DataContext (or
    BaseContext) for each session.  If so, then  I am confused because I thought
    that the new Memory Management strategy for Cayenne 3.0 was to avoid having
    to do this.

    Am I missing something about the basic usage of Cayenne?

    Thanks,
    Joe


    On Sep 16, 2009, at 3:37 PM, Mike Kienenberger wrote:

    I think there's a default filter provided by Cayenne you can specify
    in your config file, but it really comes down to something as simple
    as this to make it per request:

    public void doFilter(ServletRequest servletRequest,
    ServletResponse servletResponse, FilterChain chain)
    throws IOException, ServletException
    {
    // set base context on thread

    chain.doFilter(servletRequest, servletResponse);

    // remove base context from thread
    }

    Attaching a simple one I wrote a long time ago that perserves the
    DataContext across a session (but insures it's in a clean state at the
    end of each request).


    On Wed, Sep 16, 2009 at 3:29 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Mike,

    RE BaseContext.getThreadObjectContext()

    Of course, this could be my problem. I was using DataContext until 3.0
    then
    converted over to BaseContext.

    2.)  BaseContext.getThreadObjectContext() just tells how you're
    getting a context.  It doesn't tell how it's managed.   Do you have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end?   Or does it do this per
    session?
    I do not know how it is being initialized in the webapp (in my
    experimental
    non-web apps I explicitly intitialize it, but in the webapp it is already
    initialized).

    I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter).   That could be part of
    the problem.

    That sounds plausible.  I could not find an example of how to initialize
    this and manage it as you suggest in a web app.  I have not created a
    servlet filter but have follow the Cayenne docs for configuration of the
    web.xml.

    If this is insufficient then I agree, this could be the problem.
    Unfortunately, I have not found docs on how to accomplish what you are
    recommending.

    Joe



    On Sep 16, 2009, at 3:10 PM, Mike Kienenberger wrote:

    1.)  128 still seems small to me.   I don't think I run anything at
    less than 256.
    On the other hand, We have an app with 1000s of customers that uses
    512Mb, I think.   So 1500 seems excessive.

    2.)  BaseContext.getThreadObjectContext() just tells how you're
    getting a context.  It doesn't tell how it's managed.   Do you have a
    servlet filter that creates a new ObjectContext at the start of a
    request and then clears it out at the end?   Or does it do this per
    session?   I've never looked at the method, but it might default to
    creating one permanent data context per thread if you don't do
    anything else (like set up a servlet filter).   That could be part of
    the problem.

    On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Caveat: Apparently I am not as well. :)

    1.) I looked at the 65M issue.  On my development box (OSX) I set it to
    -Xms128m -Xmx128m (basically arbitrary).  So when I went to the remote
    hosting company, I purchase a similar amount.
    a. However, we are doing *very* little work (lots of product
    fetches
    and only a few product inserts and updates) and it runs out of memory
    *very*
    fast which I *assume* means it is my code, but I don't know.
    b. I am doing research and here is a "web recommendation" (for all
    that is worth)
    Whenever possible, Unidata recommends
    -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for 64-bit
    systems.
    c. Since the host is 64-bit, I am wondering whether my assumptions
    may be off for 64-bit systems.

    2. DataContext: Sorry, but I am getting confused on this one.  I am
    using
    BaseContext.getThreadObjectContext() based on recommendations (I
    converted
    all the old DataContext refs to BaseContext, but I don't really
    understand
    it from reading the docs) and am *not* releasing it at the end of
    session.
    Not quite sure of how to do this properly.

    3. Don't know how to set the cache to retain N number of objects. I
    experimented with
    query.setPageSize(RowsPerPage);
    query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    query.setCacheGroups("product", "ProductList");

    This seemed to help quite a bit but I still eventually ran out of
    memory.
    I
    recently removed *all* the SHARED_CACHE and it ran out of memory very
    fast.

    Thanks for your input
    Joe


    On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:

    Caveat: I'm not really an expert on Cayenne memory management.

    1) Are you allocating enough heap memory to the app server to start
    with?   I don't know what the default is these days, but in the old
    days, an application by default only gets 64Mb of memory -- that's
    pretty small.

    2) Are you using a new DataContext per request?  Or at least per
    session?

    3) I seem to remember that the cache strategy is configurable.  Have
    you configured a cache that only retains N number of objects for a
    suitable value of N?

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net>
    wrote:
    Hi,

    I have asked this question a number of ways but I still have a very
    serious
    problem with Cayenne-specific memory management configuration
    associated
    with Tomcat.

    The problem with debugging is that given that I have very little
    visibility
    into the Cayenne memory management it is extremely difficult to debug
    this
    using conventional strategies.  Also I am not requesting anything
    from
    the
    system that is terribly exceptional so I am attempting to use default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and
    standard
    memory management steps I should take to conform to the new Cayenne
    memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat,
    Cayenne
    and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and
    reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I can
    understand from the docks but this only gets me a few more hours of
    usage
    before the out of memory errors.  (I tried the SHARED_CACHE). The
    NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should handle a
    website with a lot of fetches of hundreds of data objects (i.e.
    products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data
    object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what the
    minimum require steps are for managing at data object result set
    ArrayList
    so as to properly cache and then properly free the memory after it is
    no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
    <DataContextManagerFilter.java>
  • Michael Gentry at Sep 16, 2009 at 6:40 pm
    My biggest Cayenne-based application (to-date) was using Cayenne 2.x.
    I started on Tomcat, but then switched to Jetty for development. I
    didn't have any problems with either, nor when I deployed into
    WebLogic. I was reading 10,000+ active/cached records from the
    database, too, which sounds like many more than you are using. What
    web framework are you using? Also, make sure you have enough memory
    allocated to Tomcat. You could have too small of a maximum memory
    amount allocated to Tomcat.

    mrg

    On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin wrote:
    Hi,

    I have asked this question a number of ways but I still have a very serious
    problem with Cayenne-specific memory management configuration associated
    with Tomcat.

    The problem with debugging is that given that I have very little visibility
    into the Cayenne memory management it is extremely difficult to debug this
    using conventional strategies.  Also I am not requesting anything from the
    system that is terribly exceptional so I am attempting to use default
    settings as much as possible.

    My strategy is to ask the experts for a Cayenne configuration and standard
    memory management steps I should take to conform to the new Cayenne memory
    management design intentions.

    Problem:
    1. I have essentially a webstore, three tier design with Tomcat, Cayenne and
    MySQL.
    2. When after only a few queries of products, tomcat freezes up and reports
    out of memory errors.

    I have attempted to configure the caching strategy ask best as I can
    understand from the docks but this only gets me a few more hours of usage
    before the out of memory errors.  (I tried the SHARED_CACHE). The NO_CACHE
    strategy is worse.

    I would appreciate a set of steps (aka a primer) that should handle a
    website with a lot of fetches of hundreds of data objects (i.e. products)
    and very few updates.

    Note: My gut feeling is that I am not properly managing the data object
    array properly and it is leaking memory.

    I would appreciate any input, but I would first like to know what the
    minimum require steps are for managing at data object result set ArrayList
    so as to properly cache and then properly free the memory after it is no
    longer needed.

    Context: Tomcat, MySQL, Cayenne 3.0M6

    Thanks,
    Joe Baldwin
  • Michael Gentry at Sep 16, 2009 at 6:43 pm
    I'm not sure if this will help, but ...

    http://wiki.apache.org/tomcat/FAQ/Memory

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriescayenne
postedSep 16, '09 at 6:11p
activeSep 21, '09 at 2:29p
posts34
users5
websitecayenne.apache.org

People

Translate

site design / logo © 2022 Grokbase