[Tutor] ElementTree - reading large XML files as file handles

Reading a 1.3 GB file on a machine with .99 GB RAM is certainly pushing
things. To parse it into an ElementTree will probably double or triple
your memory requirements.
Do you need the whole parsed tree at once or can you process it a little
at a time? If not, maybe this will help:
http://effbot.org/zone/element-iterparse.htm

Kent

If time is the problem, you might want to look at:

- cElementTree -- See notes about cElementTree on this page:
http://effbot.org/zone/elementtree-13-intro.htm

- lxml -- http://codespeak.net/lxml/

If size/resources/memory are the issue, as must be the case for
you, then SAX can be a solution. But, switching to SAX requires a
very radical redesign of your application.

You might also want to investigate pulldom. It's in the Python
standard library. A quote:

"PullDOM has 80% of the speed of SAX and 80% of the convenience
of the DOM. There are still circumstances where you might need
SAX (speed freak!) or DOM (complete random access). But IMO
there are a lot more circumstances where the PullDOM middle
ground is exactly what you need."

The Python standard documentation on pulldom is next to none, but
here are several links:

http://www.prescod.net/python/pulldom.html
http://www.ibm.com/developerworks/xml/library/x-tipulldom.html
http://www.idealliance.org/papers/dx_xml03/papers/06-02-03/06-02-03.html
http://www.idealliance.org/papers/dx_xml03/papers/06-02-03/06-02-03.html#pull

Hope this helps.

Dave

Reading a 1.3 GB file on a machine with .99 GB RAM is certainly pushing
things. To parse it into an ElementTree will probably double or triple
your memory requirements.
Do you need the whole parsed tree at once or can you process it a little
at a time? If not, maybe this will help:
http://effbot.org/zone/element-iterparse.htm

Kent