FAQ
Hi,

I'm using an iterated query to process a huge amount of data which cannot be loaded at once.
This query creates its own transaction and binds it to the current thread.

This means that when I process the data I received from the iterated query all queries use
the transaction created by the iterated query.

So any updates executed while the iterated query is running will be committed when the
iterated query (actually the Result Iterator) is closed. So the iterator is committing the
updates for me what I don't want.

I know I can create my own transaction but this doesn't help me. I want to be able to
create a transaction and within that transaction I want to run the iterated query and
while in the query I want to do updates which are committed when I commit my own
transaction.

Question: Is there a way to prevent the iterated query to 'hyjack' my transaction?

pseudo code

create datacontext (my transaction)
run iterated query (overrides my transaction)
process data
do some updates (will use the transaction of the iterator)
close iterator (will commit my updates. I don't want this)
commit my updates (does nothing as its already committed by the iterator)

tx

Hans

PS: I'm using cayenne 2.0.4

Search Discussions

  • Andrus Adamchik at Jan 5, 2011 at 6:29 pm
    Haven't thought about this scenario deeply... How about this:

    In the simplest case you can keep your changes in the DataContext while iterating over the result (DataContext by itself is not permanently bound to a transaction). And then commit them after iteration is finished. This will work if only some objects have changes and you have enough memory to keep them in a DataContext.

    Or if you need to flush data to DB in smaller chunks without committing a DB transaction, you can still do it by temporary re-bidning your own transaction to the current thread for the duration of commit, and then restoring it to the iterated query transaction when update is done.

    Andrus
    On Dec 29, 2010, at 3:25 PM, Hans Pikkemaat wrote:
    Hi,

    I'm using an iterated query to process a huge amount of data which cannot be loaded at once.
    This query creates its own transaction and binds it to the current thread.

    This means that when I process the data I received from the iterated query all queries use
    the transaction created by the iterated query.

    So any updates executed while the iterated query is running will be committed when the
    iterated query (actually the Result Iterator) is closed. So the iterator is committing the
    updates for me what I don't want.

    I know I can create my own transaction but this doesn't help me. I want to be able to
    create a transaction and within that transaction I want to run the iterated query and
    while in the query I want to do updates which are committed when I commit my own
    transaction.

    Question: Is there a way to prevent the iterated query to 'hyjack' my transaction?

    pseudo code

    create datacontext (my transaction)
    run iterated query (overrides my transaction)
    process data
    do some updates (will use the transaction of the iterator)
    close iterator (will commit my updates. I don't want this)
    commit my updates (does nothing as its already committed by the iterator)

    tx

    Hans

    PS: I'm using cayenne 2.0.4
  • Hans Pikkemaat at Jan 6, 2011 at 8:48 am
    Hi,

    After some investigation it comes down to two problems

    1)
    An example.
    I have 2 libraries. One library calls the other one.
    The first one is using the iterated query to get some data. It will call the second
    library to process the data.

    This second library however is not aware of a iterated query being used and it
    simply creates a new DataContext and will commit it when it sees fit. But it doesn't
    get a new transaction as there is already one bound to the thread.

    This would effectively mean that every application that uses cayenne has to check
    the bound transaction and if it exists create a new one and restore it later
    (assuming he wants its own transaction). This is a weird situation.

    2)
    An example.
    I have 2 libraries. One library calls the other one.
    The first one is setting a task record to 'in progress'. Then it calls the second
    library. The second one is using an iterated query but it is wrapped in an Iterator
    such that the caller doesn't see the details. This however means that when an
    exception occurs that the ResultIterator is not closed. This means that the transaction
    is not unbound and as such never committed. The commit on the trasaction created
    by the calling library then also doesn't work.

    Now you could say that this is a design flaw. I partly agree. I should not wrap the
    ResultIterator in an iterator as this prevents access to the close method.
    So should I create a new iterator class with the close method and let this one
    'bleed' through to the user? This also seems not a nice solution to me.

    An thoughts?

    tx

    Hans
    On 1/5/11 7:29 PM, Andrus Adamchik wrote:
    Haven't thought about this scenario deeply... How about this:

    In the simplest case you can keep your changes in the DataContext while iterating over the result (DataContext by itself is not permanently bound to a transaction). And then commit them after iteration is finished. This will work if only some objects have changes and you have enough memory to keep them in a DataContext.

    Or if you need to flush data to DB in smaller chunks without committing a DB transaction, you can still do it by temporary re-bidning your own transaction to the current thread for the duration of commit, and then restoring it to the iterated query transaction when update is done.

    Andrus
    On Dec 29, 2010, at 3:25 PM, Hans Pikkemaat wrote:
    Hi,

    I'm using an iterated query to process a huge amount of data which cannot be loaded at once.
    This query creates its own transaction and binds it to the current thread.

    This means that when I process the data I received from the iterated query all queries use
    the transaction created by the iterated query.

    So any updates executed while the iterated query is running will be committed when the
    iterated query (actually the Result Iterator) is closed. So the iterator is committing the
    updates for me what I don't want.

    I know I can create my own transaction but this doesn't help me. I want to be able to
    create a transaction and within that transaction I want to run the iterated query and
    while in the query I want to do updates which are committed when I commit my own
    transaction.

    Question: Is there a way to prevent the iterated query to 'hyjack' my transaction?

    pseudo code

    create datacontext (my transaction)
    run iterated query (overrides my transaction)
    process data
    do some updates (will use the transaction of the iterator)
    close iterator (will commit my updates. I don't want this)
    commit my updates (does nothing as its already committed by the iterator)

    tx

    Hans

    PS: I'm using cayenne 2.0.4
  • Andrus Adamchik at Jan 6, 2011 at 9:08 am

    On Jan 6, 2011, at 10:48 AM, Hans Pikkemaat wrote:

    One library calls the other one.
    The first one is using the iterated query to get some data. It will call the second
    library to process the data.
    IMO this first library (iterator control code) should be the place that does transaction switching, transparently to the second library. E.g. in lib1:

    try {
    while(it.hasNext()) {
    Map nextRow = it.next();

    // save current tx in a local ivar
    try {
    // set current tx to NULL
    // call lib2
    }
    finally {
    // restore tx
    }

    }
    }
    finally {
    it.close();
    }
    Now you could say that this is a design flaw.
    IMO it is.

    Andrus
  • Hans Pikkemaat at Jan 6, 2011 at 1:21 pm
    Hi,

    That sounds logical but the issue could be that the calling party is simply not aware that the
    called library is using cayenne.

    This would mean that an application which is calling some library always should clear
    (and keep) the transaction. This is not very logical.

    And what about the other example with the exception? Is there a way to call the close
    method on the result iterator without exposing the ResultIterator to the calling party?

    tx

    Hans
    On 1/6/11 10:08 AM, Andrus Adamchik wrote:
    On Jan 6, 2011, at 10:48 AM, Hans Pikkemaat wrote:

    One library calls the other one.
    The first one is using the iterated query to get some data. It will call the second
    library to process the data.
    IMO this first library (iterator control code) should be the place that does transaction switching, transparently to the second library. E.g. in lib1:

    try {
    while(it.hasNext()) {
    Map nextRow = it.next();

    // save current tx in a local ivar
    try {
    // set current tx to NULL
    // call lib2
    }
    finally {
    // restore tx
    }

    }
    }
    finally {
    it.close();
    }
    Now you could say that this is a design flaw.
    IMO it is.

    Andrus

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriescayenne
postedDec 29, '10 at 1:26p
activeJan 6, '11 at 1:21p
posts5
users2
websitecayenne.apache.org

2 users in discussion

Hans Pikkemaat: 3 posts Andrus Adamchik: 2 posts

People

Translate

site design / logo © 2022 Grokbase