FAQ
In the apps I'm working on I can generally sort tables into two
categories:

1. tables that change as part of normal operation of the database
and/or where the number of records is too large to fit into memory

2. various and sundry small tables. These have a small number
of records, change infrequently and can easily fit into memory. Loading
these tables take an instant(lest I'd categorize them as #1 tables).

#1 has relationships to #2 entries. These are constantly resolved via
sql queries.
At a guess I'd say that 95% of all queries are to resolve relationships from
#1 to #2. Ideally I should be able to mark a table as #2 via a configuration
option in my application as part of hardcoded tweaking of the app.

Is there a way to load #2 permanently into memory so as to avoid
SQL statements to resolve relationships from #1 to #2?


Since loading all #2 tables take an instant, they could be dumped if
more than N seconds has passed since they were last used.

--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer

Search Discussions

  • Scott Anderson at Jun 30, 2008 at 1:42 pm
    Within a given context, unique database rows exist as unique objects. You could cache these objects with strong references, which would prevent them from being GC'd.

    -----Original Message-----
    From: oyvindharboe@gmail.com On Behalf Of Øyvind Harboe
    Sent: Monday, June 30, 2008 9:06 AM
    To: cayenne-user@incubator.apache.org
    Subject: Loading small tables into memory for the application lifetime

    In the apps I'm working on I can generally sort tables into two
    categories:

    1. tables that change as part of normal operation of the database
    and/or where the number of records is too large to fit into memory

    2. various and sundry small tables. These have a small number
    of records, change infrequently and can easily fit into memory. Loading
    these tables take an instant(lest I'd categorize them as #1 tables).

    #1 has relationships to #2 entries. These are constantly resolved via
    sql queries.
    At a guess I'd say that 95% of all queries are to resolve relationships from
    #1 to #2. Ideally I should be able to mark a table as #2 via a configuration
    option in my application as part of hardcoded tweaking of the app.

    Is there a way to load #2 permanently into memory so as to avoid
    SQL statements to resolve relationships from #1 to #2?


    Since loading all #2 tables take an instant, they could be dumped if
    more than N seconds has passed since they were last used.

    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer
  • Michael Gentry at Jun 30, 2008 at 2:29 pm
    http://cayenne.apache.org/doc/moving-objects-between-contexts.html
    On Mon, Jun 30, 2008 at 9:06 AM, Øyvind Harboe wrote:
    In the apps I'm working on I can generally sort tables into two
    categories:

    1. tables that change as part of normal operation of the database
    and/or where the number of records is too large to fit into memory

    2. various and sundry small tables. These have a small number
    of records, change infrequently and can easily fit into memory. Loading
    these tables take an instant(lest I'd categorize them as #1 tables).

    #1 has relationships to #2 entries. These are constantly resolved via
    sql queries.
    At a guess I'd say that 95% of all queries are to resolve relationships from
    #1 to #2. Ideally I should be able to mark a table as #2 via a configuration
    option in my application as part of hardcoded tweaking of the app.

    Is there a way to load #2 permanently into memory so as to avoid
    SQL statements to resolve relationships from #1 to #2?


    Since loading all #2 tables take an instant, they could be dumped if
    more than N seconds has passed since they were last used.

    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer
  • Michael Gentry at Jun 30, 2008 at 2:46 pm
    Actually, I think in Cayenne 3 that it'll automatically pull an object
    into the DataContext if it exists in another DataContext. My memory
    is a bit rusty, though, but I'm sure someone will correct me if I am
    wrong. But in a nutshell, make a separate/global DataContext for your
    small tables and cache the data in it.

    On Mon, Jun 30, 2008 at 10:28 AM, Michael Gentry wrote:
    http://cayenne.apache.org/doc/moving-objects-between-contexts.html
    On Mon, Jun 30, 2008 at 9:06 AM, Øyvind Harboe wrote:
    In the apps I'm working on I can generally sort tables into two
    categories:

    1. tables that change as part of normal operation of the database
    and/or where the number of records is too large to fit into memory

    2. various and sundry small tables. These have a small number
    of records, change infrequently and can easily fit into memory. Loading
    these tables take an instant(lest I'd categorize them as #1 tables).

    #1 has relationships to #2 entries. These are constantly resolved via
    sql queries.
    At a guess I'd say that 95% of all queries are to resolve relationships from
    #1 to #2. Ideally I should be able to mark a table as #2 via a configuration
    option in my application as part of hardcoded tweaking of the app.

    Is there a way to load #2 permanently into memory so as to avoid
    SQL statements to resolve relationships from #1 to #2?


    Since loading all #2 tables take an instant, they could be dumped if
    more than N seconds has passed since they were last used.

    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer
  • Andrus Adamchik at Jun 30, 2008 at 2:49 pm

    On Jun 30, 2008, at 5:45 PM, Michael Gentry wrote:

    I think in Cayenne 3 that it'll automatically pull an object
    into the DataContext if it exists in another DataContext.
    Not true. (well only true for transient unregistered objects).
    Generally you still need to use 'localObject'.

    Andrus
  • Michael Gentry at Jun 30, 2008 at 3:12 pm
    OK, thanks. I wasn't certain of that by any means and am too buried
    in other stuff to look it up.

    The pattern of using a dedicated DataContext for the small/cached
    tables and using localObject to make joins to them would still apply,
    though.

    mrg


    On Mon, Jun 30, 2008 at 10:49 AM, Andrus Adamchik
    wrote:
    On Jun 30, 2008, at 5:45 PM, Michael Gentry wrote:

    I think in Cayenne 3 that it'll automatically pull an object
    into the DataContext if it exists in another DataContext.
    Not true. (well only true for transient unregistered objects). Generally you
    still need to use 'localObject'.

    Andrus
  • Mike Kienenberger at Jun 30, 2008 at 3:33 pm
    I don't remember the exact details, so this might be wrong.

    But what about putting the constant tables into a parent DataContext,
    then doing all of your work in the child DataContext? Does that
    remove the need for using localObject, but still give you the
    flexibility to toss the child context objects?

    On Mon, Jun 30, 2008 at 11:11 AM, Michael Gentry wrote:
    OK, thanks. I wasn't certain of that by any means and am too buried
    in other stuff to look it up.

    The pattern of using a dedicated DataContext for the small/cached
    tables and using localObject to make joins to them would still apply,
    though.

    mrg


    On Mon, Jun 30, 2008 at 10:49 AM, Andrus Adamchik
    wrote:
    On Jun 30, 2008, at 5:45 PM, Michael Gentry wrote:

    I think in Cayenne 3 that it'll automatically pull an object
    into the DataContext if it exists in another DataContext.
    Not true. (well only true for transient unregistered objects). Generally you
    still need to use 'localObject'.

    Andrus
  • Andrus Adamchik at Jun 30, 2008 at 3:48 pm

    On Jun 30, 2008, at 6:33 PM, Mike Kienenberger wrote:

    But what about putting the constant tables into a parent DataContext,
    then doing all of your work in the child DataContext? Does that
    remove the need for using localObject, but still give you the
    flexibility to toss the child context objects?
    That won't work directly, you still have to get a local object (it
    won't be fetched from DB, and rather resolved from memory, but still).
    But here is another way to work with lookup tables in 3.0 - use shared
    caching... So locally you'd do this:

    // you don't have to make it static; just wanted to point out
    // that the query object itself is reusable (as long as there is no
    parameters)
    // This query can be mapped in the Modeler just as well.
    static final SelectQuery countriesQuery = new
    SelectQuery(Country.class);

    static {
    countriesQuery.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
    }


    public void myNonStaticMethod() {
    ...
    List localCountries = context.performQuery(countriesQuery);
    }

    There is some overhead with the shared cache access, but still with
    this approach you don't have to mess with 'localObject'. I'd say this
    is an ideal separation of code from configuration.

    Andrus
  • Øyvind Harboe at Jun 30, 2008 at 5:29 pm
    I guess what I'm looking for is to have a writeback cache where entire
    tables are loaded into memory upon startup of the application.

    The tables and their content don't actually take up that much memory +
    the memory is allocated upon startup so there are no surprises later
    on by memory peak usage. Actual memory usage doesn't matter as much
    as being able to know precisely what to set maximum heap to & knowing
    how much physical memory to install.

    Of course that would require Cayenne to be able to run queries against
    tables in memory...

    Cayenne is capable of resolving to-one relationships for objects that
    are in memory, but to-many require a query to the database even
    if all the resulting records are cached in memory already.


    Or... I could be barking up the wrong tree... again.... I've been
    pondering whether to modify the logger not to show queries that
    take less than n ms. I never measured how long it actually
    takes to log a query.... This sort of caching is what I would have
    expected databases to have done to death...

    --
    Øyvind Harboe
    http://www.zylin.com/zy1000.html
    ARM7 ARM9 XScale Cortex
    JTAG debugger and flash programmer

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriescayenne
postedJun 30, '08 at 1:07p
activeJun 30, '08 at 5:29p
posts9
users5
websitecayenne.apache.org

People

Translate

site design / logo © 2022 Grokbase