FAQ
I'm running a process which uses python numeric arrays extensively and also
does some calculations in c. All objects are created in python or through
the Numeric library and then passed to the c routines. I'm dealing with very
large sets of data.

I'm getting the following exceptions in the early part of the program:

Exception exceptions.AssertionError: <exceptions.AssertionError instance at
0x02639C10> in <function remove at 0x025CF5B0> ignored

...and then after about at the 1060 iteration of the main processing loop I
get the following error:

Fatal Python error: deallocating None

Questions:
1. Is there any way to catch this error and close my database connection so
my uncommitted transactions get saved to the db
2. When I profile the larger dataset the .prof file does not get generated.
Is there any way to incrementally build the .prof file
3. Any ideas what the errors means and how can I debug/catch them?

vinj

Search Discussions

  • John Hunter at May 16, 2004 at 10:22 pm
    "Vineet" == Vineet Jain <vineet at eswap.com> writes:
    Vineet> Exception exceptions.AssertionError:
    Vineet> <exceptions.AssertionError instance at 0x02639C10> in
    Vineet> <function remove at 0x025CF5B0> ignored

    Do you know what assertion is failing in the remove function and can
    you fix it?

    Vineet> Any ideas what the errors means and how can I
    Vineet> debug/catch them?

    The most likely thing is that you have made a mistake in your
    reference counting, perhaps decrementing a reference where you
    shouldn't have, or creating a python object and failing to incref it.
    Have you written a python extension module? If so did you do it by
    hand or are you using one of the standard wrapper generators like SWIG
    or pyrex? If the latter, it could be a bug in the wrapper generator.
    If not, it might help to post some of your code which acts on the
    PyObjects.

    JDH
  • Vineet Jain at May 19, 2004 at 9:56 pm

    or pyrex? If the latter, it could be a bug in the wrapper generator.
    If not, it might help to post some of your code which acts on the
    PyObjects.
    I've been able to recreate the problem with the following code segment. It
    seems that the problem comes when you have multple db files open at the same
    time. In my application I have many database files open at the same time.
    I'm not using threads so all access to the db files are done sequenetially.

    At the end of the program I get the following error:

    Fatal Python error: deallocating None

    This application has requested the Runtime to terminate it in an unusual
    way.
    Please contact the application's support team for more information.

    If I change the numInnerLoop to 5, 10, 20, or 30 it works however anything
    over 35 gives the above error. In my actual application, when run for a
    larger set, the error can come in the middle of the program and is causing
    me a lot of grief.

    Any help would be really appreciated.




    -------------------Problem Code----------------------------
    import sqlite
    import gc
    import sys, os

    #initialize database

    dbFileName = "c:/test.db"
    numRecords = 1000

    os.remove(dbFileName)
    conn = sqlite.connect(database=dbFileName,autocommit=0)
    cur = conn.cursor()
    cur.execute("create table t (field1, field2, field3, field4, field5, field6,
    field7)")
    for i in range(numRecords):
    cur.execute("insert into t values(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0)")
    conn.commit()
    conn.close()

    numInnerLoop = 50
    conn = [None]*numInnerLoop
    cur = [None]*numInnerLoop
    tempStore = [None]*numInnerLoop
    qry = "select * from t"

    for k in range(1):
    for i in range(numInnerLoop):
    conn[i] = sqlite.connect(database=dbFileName,autocommit=1)
    cur[i] = conn[i].cursor()

    sys.stdout.write('OuterLoop: %d Inner Loop : %d\r' % (k, i))

    tempStore[i] = []
    cur[i].execute(qry)
    dbRowList = cur[i].fetchall()
    for j in range(numRecords):
    for l in range(6):
    tempStore[i].append(float(dbRowList[j][l]))

    dbRowList = None
    gc.collect()

    for i in range(numInnerLoop):
    conn[i].close()


    On the other hand when I work with only one database connection. Things work
    fine:

    --------------------------Ok
    Code----------------------------------------------------
    import sqlite
    import gc
    import sys, os

    #initialize database

    tempStore = [None]*400
    dbFileName = "c:/test.db"


    os.remove(dbFileName)
    conn = sqlite.connect(database=dbFileName,autocommit=0)
    cur = conn.cursor()
    cur.execute("create table t (field1, field2, field3, field4, field5, field6,
    field7)")
    for i in range(10000):
    cur.execute("insert into t values(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0)")
    conn.commit()
    conn.close()

    conn = sqlite.connect(database=dbFileName,autocommit=1)
    cur = conn.cursor()

    qry = "select * from t"
    for k in range(1):
    for i in range(300):

    sys.stdout.write('OuterLoop: %d Inner Loop : %d\r' % (k, i))

    tempStore[i] = []
    cur.execute(qry)
    dbRowList = cur.fetchall()
    for j in range(10000):
    for l in range(6):
    tempStore[i].append(float(dbRowList[j][l]))

    dbRowList = None
    gc.collect()

    conn.close()
  • Terry Reedy at May 19, 2004 at 10:54 pm
    "Vineet Jain" <vineet at eswap.com> wrote in
    message news:KPENJIMKHJPNKOLFIHBNKEPNCJAA.vineet at eswap.com...
    I've been able to recreate the problem with the following code segment. It
    seems that the problem comes when you have multple db files open at the same
    time. In my application I have many database files open at the same time.
    I'm not using threads so all access to the db files are done
    sequenetially.
    At the end of the program I get the following error:

    Fatal Python error: deallocating None

    This application has requested the Runtime to terminate it in an unusual
    way.
    Please contact the application's support team for more information.
    Have you done so? In this case, the 'application' is, it would seem,
    sqlite.
    If I change the numInnerLoop to 5, 10, 20, or 30 it works however anything
    over 35 gives the above error. In my actual application, when run for a
    larger set, the error can come in the middle of the program and is causing
    me a lot of grief.
    Do the docs for sqlite specify max open connections?
    If not, maybe send them your example.
    If so, ask if limit can be raised.
    In the meanwhile, stay under it.

    Terry J. Reedy
  • Vineet Jain at May 19, 2004 at 11:51 pm
    I've tried to close all the db connections after I use them and still et the
    problem. I still get the problem if I add:

    cur = None
    conn.close()

    after I retrieve each reult set.




    -----Original Message-----
    From: python-list-bounces+vineet=eswap.com at python.org
    [mailto:python-list-bounces+vineet=eswap.com at python.org]On Behalf Of
    Terry Reedy
    Sent: Wednesday, May 19, 2004 5:54 PM
    To: python-list at python.org
    Cc: pysqlite-users at lists.sourceforge.net
    Subject: Re: Help with large data set and fatal python error (pysqlite)



    "Vineet Jain" <vineet at eswap.com> wrote in
    message news:KPENJIMKHJPNKOLFIHBNKEPNCJAA.vineet at eswap.com...
    I've been able to recreate the problem with the following code segment. It
    seems that the problem comes when you have multple db files open at the same
    time. In my application I have many database files open at the same time.
    I'm not using threads so all access to the db files are done
    sequenetially.
    At the end of the program I get the following error:

    Fatal Python error: deallocating None

    This application has requested the Runtime to terminate it in an unusual
    way.
    Please contact the application's support team for more information.
    Have you done so? In this case, the 'application' is, it would seem,
    sqlite.
    If I change the numInnerLoop to 5, 10, 20, or 30 it works however anything
    over 35 gives the above error. In my actual application, when run for a
    larger set, the error can come in the middle of the program and is causing
    me a lot of grief.
    Do the docs for sqlite specify max open connections?
    If not, maybe send them your example.
    If so, ask if limit can be raised.
    In the meanwhile, stay under it.

    Terry J. Reedy
  • Slawomir Nowaczyk at May 20, 2004 at 11:59 am
    On Wed, 19 May 2004 17:56:37 -0400 "Vineet Jain" wrote:
    #> I've been able to recreate the problem with the following code segment. It
    #> seems that the problem comes when you have multple db files open at the same
    #> time. In my application I have many database files open at the same time.
    #> I'm not using threads so all access to the db files are done sequenetially.

    #> At the end of the program I get the following error:

    #> Fatal Python error: deallocating None

    #> This application has requested the Runtime to terminate it in an unusual
    #> way.
    #> Please contact the application's support team for more information.

    #> If I change the numInnerLoop to 5, 10, 20, or 30 it works however anything
    #> over 35 gives the above error. In my actual application, when run for a
    #> larger set, the error can come in the middle of the program and is causing
    #> me a lot of grief.

    #> Any help would be really appreciated.

    I don't know how much help would it be, but the code below is working
    fine for me - for as much as "numInnerLoop = 150" and "k in range(5)".
    So it would seem like a problem specific to your system.

    I am using sqlite.version '0.5.0' on Windows XP. What is your
    configuration?

    One wild guess would be that you are reaching the system limit of open
    files (assuming each connection means opening a file, which seems
    likely). I recall Windows 9x having something like this, but I do not
    remember details and don't know how it looks like in newer versions.

    You may also want to check if you are able to open many connections
    with stand-alone sqlite program. That should, at least, give you a
    hint whom should you send a bug report :-) Even if there is a reason
    for sqlite to fail, it should do it more gracefully.

    -------------------Problem Code----------------------------
    import sqlite
    import gc
    import sys, os

    #initialize database

    dbFileName = "c:/test.db"
    numRecords = 1000

    os.remove(dbFileName)
    conn = sqlite.connect(database=dbFileName,autocommit=0)
    cur = conn.cursor()
    cur.execute("create table t (field1,field2,field3,field4,field5,field6,field7)")
    for i in range(numRecords):
    cur.execute("insert into t values(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0)")
    conn.commit()
    conn.close()

    numInnerLoop = 50
    conn = [None]*numInnerLoop
    cur = [None]*numInnerLoop
    tempStore = [None]*numInnerLoop
    qry = "select * from t"

    for k in range(1):
    for i in range(numInnerLoop):
    conn[i] = sqlite.connect(database=dbFileName,autocommit=1)
    cur[i] = conn[i].cursor()

    sys.stdout.write('OuterLoop: %d Inner Loop : %d\r' % (k, i))

    tempStore[i] = []
    cur[i].execute(qry)
    dbRowList = cur[i].fetchall()
    for j in range(numRecords):
    for l in range(6):
    tempStore[i].append(float(dbRowList[j][l]))

    dbRowList = None
    gc.collect()

    for i in range(numInnerLoop):
    conn[i].close()

    --
    Best wishes,
    Slawomir Nowaczyk
    ( Slawomir.Nowaczyk at cs.lth.se )

    My goal in life is to be the sort of person my dog thinks I am.
  • Vineet Jain at May 20, 2004 at 9:30 pm

    I don't know how much help would it be, but the code below is working
    fine for me - for as much as "numInnerLoop = 150" and "k in range(5)".
    So it would seem like a problem specific to your system.
    Thanks. what python version do you have? I tried the code on three different
    version and it gave the same
    problem:

    1. Windows XP Pro, Dell 8600, Sqlite 0.5.0
    2. Windows Advanced Server, Dell Server, Sqlite 0.5.0
    3. Windows XP, dell lattitude, Sqlite 0.50

    I still get the same problem. I also tried the same thing with sqlite cvs
    snapshot and that did not help.

    In the mean time, since I have to get this working, I'm in process on using
    pyrex to access the sqlite db directly.
  • Gerhard Häring at May 20, 2004 at 10:43 pm
    This bug is now fixed in PySQLite CVS.

    See

    http://sourceforge.net/tracker/index.php?funcÞtail&aid…2362&group_idT058&atidG2543

    for the history of this bug.

    -- Gerhard

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedMay 15, '04 at 2:27p
activeMay 20, '04 at 10:43p
posts8
users5
websitepython.org

People

Translate

site design / logo © 2022 Grokbase