FAQ
Dear perl users,

I read a large file into a hash table. This hash table is used only once in the subroutine. I am very concern the memory usage, So I plan to realse the memory after used the hash table. The way I used is:

my %hash = ();

#read the large file into the hash table
...

#release memory
%hash = ();

Do you think this way will really release memory usage?

By the way, I found the hash table use much more memory than the array. I think hash table and array use different data structrue. it need much more memory to matain the hash tabe structure. I hope what I find is useful.

Thanks

Org


_________________________________________________________________
Net yourself a bargain. Find great deals on eBay.
http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Frover%2Eebay%2Ecom%2Frover%2F1%2F705%2D10129%2D5668%2D323%2F4%3Fid%3D10&_t=763807330&_r=hotmailTAGLINES&_m=EXT

Search Discussions

  • Mr. Shawn H. Corey at Dec 10, 2008 at 2:52 am

    On Wed, 2008-12-10 at 13:15 +1030, org chen wrote:
    I read a large file into a hash table. This hash table is used only
    once in the subroutine. I am very concern the memory usage, So I plan
    to realse the memory after used the hash table. The way I used is:

    my %hash = ();

    #read the large file into the hash table
    ...

    #release memory
    %hash = ();

    Do you think this way will really release memory usage?
    Yes.

    If you declare the has inside the sub, it will be released when the sub
    is exited. That is, unless you create a reference to it and store the
    reference in an outside variable.

    Example:

    #!/usr/bin/perl

    use strict;
    use warnings;

    sub foo {
    my %hash1 = ( a => 1, b => 2 ); # This will be release on exit
    my %hash2 = ( x => 3.14, y => 1.76 );

    return \%hash2; # %hash2 will NOT be released until later
    }



    --
    Just my 0.00000002 million dollars worth,
    Shawn

    The key to success is being too stupid to realize you can fail.
  • Rob Dixon at Dec 10, 2008 at 3:00 am

    org chen wrote:

    I read a large file into a hash table. This hash table is used only once in
    the subroutine. I am very concern the memory usage, So I plan to realse the
    memory after used the hash table. The way I used is:

    my %hash = ();

    #read the large file into the hash table
    ...

    #release memory
    %hash = ();

    Do you think this way will really release memory usage?

    By the way, I found the hash table use much more memory than the array. I
    think hash table and array use different data structrue. it need much more
    memory to matain the hash tabe structure. I hope what I find is useful.
    Yes, that will flag all of the memory used by the hash for release.

    But there must be a way to rewrite the subroutine so that it doesn't copy the
    entire file into memory and then release it. Also, if your file is really so big
    and requires frequent access then you should think about creating a database.

    Please show us your subroutine and describe the contents of the file so that we
    can help you further.

    Rob
  • Org chen at Dec 10, 2008 at 3:51 am
    Hi,

    Thanks for the confirmation and suggestion.

    I had tried Database, it do not work well for me, since the database need more data structure to maintain the data, it take longer time or more memory to add or remove data and matain the index table. In some case the pure TEXT file or array is much efficient, such as sorting.

    It is nice there is a place we can discuss these issus.

    Org.

    Date: Wed, 10 Dec 2008 02:59:47 +0000> From: rob.dixon@gmx.com> To: beginners@perl.org> CC: org.chen@live.com.au> Subject: Re: how to release memory from a hash table or array> > org chen wrote:> > > > I read a large file into a hash table. This hash table is used only once in> > the subroutine. I am very concern the memory usage, So I plan to realse the> > memory after used the hash table. The way I used is:> > > > my %hash = ();> > > > #read the large file into the hash table> > ...> > > > #release memory> > %hash = ();> > > > Do you think this way will really release memory usage?> > > > By the way, I found the hash table use much more memory than the array. I > > think hash table and array use different data structrue. it need much more > > memory to matain the hash tabe structure. I hope what I find is useful.> > Yes, that will flag all of the memory used by the hash for release.> > But there must be a way to rewrite the subroutine so that it doesn't copy the> entire file into memory and then release it. Also, if your file is really so big> and requires frequent access then you should think about creating a database.> > Please show us your subroutine and describe the contents of the file so that we> can help you further.> > Rob
    _________________________________________________________________
    Holiday cheer from Messenger. Download free emoticons today!
    http://livelife.ninemsn.com.au/article.aspx?id=669758
  • Rob Dixon at Dec 10, 2008 at 4:22 am

    org chen wrote:

    Thanks for the confirmation and suggestion.

    I had tried Database, it do not work well for me, since the database need
    more data structure to maintain the data, it take longer time or more memory
    to add or remove data and matain the index table. In some case the pure TEXT
    file or array is much efficient, such as sorting.

    It is nice there is a place we can discuss these issus.
    You haven't shown us your data. But I think that if you don't spend the time to
    organize your data into a database then you will instead have to spend much
    longer writing your program. And when it is finished it will probably run more
    slowly than if it was data-oriented.

    You should also consider whether there there will ever be another program
    written that needs access to the same data. If so then there is no question that
    it belongs in a database.

    You're welcome to discuss your problems here, but please try to describe it
    carefully so that we can best help you.

    Rob
  • Dr.Ruud at Dec 10, 2008 at 8:36 pm

    org chen schreef:

    I read a large file into a hash table. This hash table is used only
    once in the subroutine. I am very concern the memory usage, So I plan
    to realse the memory after used the hash table. The way I used is:

    my %hash = ();

    #read the large file into the hash table
    ...

    #release memory
    %hash = ();

    Do you think this way will really release memory usage?
    Define "release" (or rather "really release").

    It can get marked as free in a Perl sense, but not (necessarily) be
    returned to the OS. So from the OS point of view it is still used by the
    perl process that (once) allocated it. This all depends on how your perl
    is compiled too, like with what memory manager.

    By the way, I found the hash table use much more memory than the
    array.
    Show us your code.

    --
    Affijn, Ruud

    "Gewoon is een tijger."

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupbeginners @
categoriesperl
postedDec 10, '08 at 2:46a
activeDec 10, '08 at 8:36p
posts6
users4
websiteperl.org

People

Translate

site design / logo © 2021 Grokbase