FAQ
hi,
    I am running a python script and it will create a file name like filename0.0.0 and If I run it again then new file will create one more like filename0.0.1...... my code is-


i = 0
for i in range(1000):
     try:
         with open('filename%d.%d.%d.json'%(0,0,i,)): pass
         continue
     except IOError:
         dataFile = file('filename%d.%d.%d.json'%(0,0,i,), 'a+')
         break
But It will take more time after creating many files, So i want to store value of last var "i" in a variable so that when i run my script again then I can use it. for example-
                  my last created file is filename0.0.27 then it should store 27 in a variable and when i run again then new file should be created 0.0.28 according to last value "27", so that i could save time and it can create file fast..


Please give me suggestion for it.. How is it possible?
Thanks

Search Discussions

  • Cameron Simpson at Jun 6, 2013 at 11:19 am

    On 06Jun2013 03:50, Avnesh Shakya wrote:
    hi,
    I am running a python script and it will create a file name like filename0.0.0 and If I run it again then new file will create one more like filename0.0.1...... my code is-

    i = 0
    for i in range(1000):
    try:
    with open('filename%d.%d.%d.json'%(0,0,i,)): pass
    continue
    except IOError:
    dataFile = file('filename%d.%d.%d.json'%(0,0,i,), 'a+')
    break
    But It will take more time after creating many files, So i want to store value of last var "i" in a variable so that when i run my script again then I can use it. for example-
    my last created file is filename0.0.27 then it should store 27 in a variable and when i run again then new file should be created 0.0.28 according to last value "27", so that i could save time and it can create file fast..

    Please give me suggestion for it.. How is it possible?

    Write it to a file? Read the file next time the script runs?


    BTW, trying to open zillions of files is slow.
    But using listdir to read the directory you can see all the names.
    Pick the next free one (and then test anyway).
    --
    Cameron Simpson <cs@zip.com.au>


    The mark must be robust enough to survive MP3 transmission over the Internet,
    but remain inaudible when played on the yet to be launched DVD-Audio players.
    - the SDMI audio watermarkers literally ask for the impossible, since all
       audio compressors aim to pass _only_ human perceptible data
       http://www.newscientist.com/news/news.jsp?id=ns224836
  • Avnesh Shakya at Jun 6, 2013 at 12:06 pm
    Thanks.




    On Thu, Jun 6, 2013 at 4:49 PM, Cameron Simpson wrote:

    On 06Jun2013 03:50, Avnesh Shakya wrote:
    hi,
    I am running a python script and it will create a file name like
    filename0.0.0 and If I run it again then new file will create one more like
    filename0.0.1...... my code is-
    i = 0
    for i in range(1000):
    try:
    with open('filename%d.%d.%d.json'%(0,0,i,)): pass
    continue
    except IOError:
    dataFile = file('filename%d.%d.%d.json'%(0,0,i,), 'a+')
    break
    But It will take more time after creating many files, So i want to store
    value of last var "i" in a variable so that when i run my script again then
    I can use it. for example-
    my last created file is filename0.0.27 then it should
    store 27 in a variable and when i run again then new file should be created
    0.0.28 according to last value "27", so that i could save time and it can
    create file fast..
    Please give me suggestion for it.. How is it possible?
    Write it to a file? Read the file next time the script runs?

    BTW, trying to open zillions of files is slow.
    But using listdir to read the directory you can see all the names.
    Pick the next free one (and then test anyway).
    --
    Cameron Simpson <cs@zip.com.au>

    The mark must be robust enough to survive MP3 transmission over the
    Internet,
    but remain inaudible when played on the yet to be launched DVD-Audio
    players.
    - the SDMI audio watermarkers literally ask for the impossible, since all
    audio compressors aim to pass _only_ human perceptible data
    http://www.newscientist.com/news/news.jsp?id=ns224836
    --
    http://mail.python.org/mailman/listinfo/python-list
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-list/attachments/20130606/f5c44dd5/attachment.html>
  • Dave Angel at Jun 6, 2013 at 12:14 pm

    On 06/06/2013 06:50 AM, Avnesh Shakya wrote:
    hi,
    I am running a python script and it will create a file name like filename0.0.0 and If I run it again then new file will create one more like filename0.0.1...... my code is-

    i = 0

    Redundant initialization of i.

    for i in range(1000):
    try:
    with open('filename%d.%d.%d.json'%(0,0,i,)): pass
    continue
    except IOError:
    dataFile = file('filename%d.%d.%d.json'%(0,0,i,), 'a+')
    break
    But It will take more time after creating many files, So i want to store value of last var "i" in a variable

    There are no variables once the program ends. You mean you want to
    store it in the file. That's known as persistent storage, and in the
    general case you could use pickle or something like that. But in your
    simple case, the easiest thing would be to simply write the last value
    of i out to a file in the same directory.


    Then when your program starts, it opens that extra file and reads in the
    value of i. And uses that for the starting value in the loop.


       so that when i run my script again then I can use it. for example-
    my last created file is filename0.0.27 then it should store 27 in a variable and when i run again then new file should be created 0.0.28 according to last value "27", so that i could save time and it can create file fast..

    Please give me suggestion for it.. How is it possible?
    Thanks

    Incidentally, instead of opening each one, why not check its existence?
       Should be quicker, and definitely clearer.


    Entirely separate suggestion, since I dislike having extra housekeeping
    files that aren't logically necessary, and that might become out of synch :


    If you're planning on having the files densely populated (meaning no
    gaps in the numbering), then you could use a binary search to find the
    last one. Standard algorithm would converge with 10 existence checks if
    you have a limit of 1000 files.


    --
    DaveA
  • Chris Angelico at Jun 6, 2013 at 3:37 pm

    On Thu, Jun 6, 2013 at 10:14 PM, Dave Angel wrote:
    If you're planning on having the files densely populated (meaning no gaps in
    the numbering), then you could use a binary search to find the last one.
    Standard algorithm would converge with 10 existence checks if you have a
    limit of 1000 files.

    Or, if you can dedicate a directory to those files, you could go even simpler:


    dataFile = open('filename0.0.%d.json'%len(os.listdir()), 'w')


    The number of files currently existing equals the number of the next file.


    ChrisA
  • MRAB at Jun 6, 2013 at 3:54 pm

    On 06/06/2013 16:37, Chris Angelico wrote:
    On Thu, Jun 6, 2013 at 10:14 PM, Dave Angel wrote:
    If you're planning on having the files densely populated (meaning no gaps in
    the numbering), then you could use a binary search to find the last one.
    Standard algorithm would converge with 10 existence checks if you have a
    limit of 1000 files.
    Or, if you can dedicate a directory to those files, you could go even simpler:

    dataFile = open('filename0.0.%d.json'%len(os.listdir()), 'w')

    The number of files currently existing equals the number of the next file.
    Assuming no gaps.
  • Chris Angelico at Jun 6, 2013 at 3:56 pm

    On Fri, Jun 7, 2013 at 1:54 AM, MRAB wrote:
    On 06/06/2013 16:37, Chris Angelico wrote:
    On Thu, Jun 6, 2013 at 10:14 PM, Dave Angel wrote:

    If you're planning on having the files densely populated (meaning no gaps
    in
    the numbering), then you could use a binary search to find the last one.
    Standard algorithm would converge with 10 existence checks if you have a
    limit of 1000 files.

    Or, if you can dedicate a directory to those files, you could go even
    simpler:

    dataFile = open('filename0.0.%d.json'%len(os.listdir()), 'w')

    The number of files currently existing equals the number of the next file.
    Assuming no gaps.

    Which is also a stated assumption of the binary search. The only
    additional assumption of the file-count method is that there be no
    other files in that directory.


    ChrisA
  • Jussi Piitulainen at Jun 6, 2013 at 12:46 pm

    Avnesh Shakya writes:


    I am running a python script and it will create a file name like
    filename0.0.0 and If I run it again then new file will create one
    more like filename0.0.1...... my code is-

    i = 0
    for i in range(1000):
    try:
    with open('filename%d.%d.%d.json'%(0,0,i,)): pass
    continue
    except IOError:
    dataFile = file('filename%d.%d.%d.json'%(0,0,i,), 'a+')
    break

    But It will take more time after creating many files, So i want to
    store value of last var "i" in a variable so that when i run my
    script again then I can use it. for example- my last created file is
    filename0.0.27 then it should store 27 in a variable and when i run
    again then new file should be created 0.0.28 according to last value
    "27", so that i could save time and it can create file fast..

    You could get a list of all filenames that match the pattern. Extract
    the last components as numbers, and add 1 to the maximum.


       i = 1 + max(int(name.split('.')[-1])
                   for name in glob.glob('filename.0.0.*))


    That assumes that there already is at least one such file and all such
    files have a last component that can be parsed as an int. Take an
    appropriate amount of care.


    Or you could also create a file, say lastname.0.0.31, to track the
    name, and when you find it there, create filename.0.0.32 and replace
    lastname.0.0.32; panic if there is more than one lastname.0.0.*, or
    fewer than one.


    Or as above but track with nextname.0.0.31 to create filename.0.0.31
    and replace the tracking name with nextname.0.0.32 for the next file.


    Or save the number somewhere else.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJun 6, '13 at 10:50a
activeJun 6, '13 at 3:56p
posts8
users6
websitepython.org

People

Translate

site design / logo © 2022 Grokbase