At 9:10 PM +0200 5/14/04, Richard Jelinek wrote:
Is there any reason why store grabs that much memory? Looks like if a
clone of the data structure would take place in memory.
Doubling seems extreme, but Storable does keep a large hash that
seems to point to every piece of data you are storing. I don't know
if it's for referencing counting, storing data offsets or both,
didn't go that far. We were storing a hash with 70,000 items, each
pointing at hash of probably 7 items or so. (Not *my* idea of an
optimal data structure.) So roughly half a million items. At free
time Storable was freeing 1.5 million hash entries.

I was recently in the internals because we were getting memory faults
on store in 5.8.0. The cleanup routine was looping through a large
hash, setting the items to undefined prior to freeing the hash, and
the standard perl iterator routine was missing 2 or 3 items that the
undef routine was finding--which resulted in a segfault. Only seemed
to happen on Solaris (8 and 9), not on Linux or MacOSX, and from
looking at the code I suspect it's fixed in 5.8.1 and later--all of
that code has changed a bit to better support Storable. In the
meantime I just patched our copy of Storable to access the underlying
hash structures directly.

P.S. There's something odd about someone with a domain name of
"petamem" complaining performance on low memory machines :-).

Kee Hinckley
http://www.messagefire.com/ Next Generation Spam Defense
http://commons.somewhere.com/buzz/ Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 9 of 19 | next ›
Discussion Overview
groupperl5-porters @
postedMay 14, '04 at 7:10p
activeJul 7, '04 at 10:45a



site design / logo © 2022 Grokbase