[Python] [ANN] pyxser-1.2r --- Python-Object to XML serialization module

You mean, the whole XML document is serialised with that encoding, right?

Stefan

Stefan Behnel <stefan_ml at behnel.de>
on Monday 24 August 2009 03:16
wrote in comp.lang.python:

Well, if you call pyxser with:

xmldocstr = pyxser.serialize(obj = someobj, enc = "utf-8", depth = 3)

pyxser will serialize someobj into XML document (xmldocstr) using the
utf-8 encoding with a depth of three levels in the object tree. If the
object tree has unicode objects, those objects will be encoded into
utf-8 too. And yes, it means that unicode objects are encoded into the
encoding that the XML document encoding has, and as you say, the whole
XML document has one encoding. There is no mixing of byte encoded strings
with different encodings in the outout document.

When the object is restored, by using pyxser.unserialize:

pyobj = pyxser.unserialize(obj = xmldocstr, enc = "utf-8")

Unicode objects are restored as unicode objects, by decoding the utf-8
strings on the document. Byte string objects are restored as byte string
objects. And the other types of objects are restored as their original
type.

Another issue is the fact that if you have mixed some encodings in byte
strings objects in your object tree, such as iso-8859-1 and utf-8, and
you try to serialize that object, pyxser will output to stdout the
serialization errors by trying to handle those mixed encodings which are
not regarding the document encoding.

There are some restrictions, if the class of the object is declared on
the __main__ module, pyxser can not handle it... This is a bug but I'm
expecting to solve it in near future.

A depth level of zero (depth = 0) will serialize the complete object
tree with a fixed limit of 50 levels, pyxser can handle cross referenced
objects and circular references.

Sorry if don't answer more questions until we reach the night --- here
in Chile --- the night, I must go to my job right now, and there I don't
have access to usenet or google groups.
Best regards,
- --
.O. | Daniel Molina Wegener | FreeBSD & Linux
..O | dmw [at] coder [dot] cl | Open Standards
OOO | http://coder.cl/ | FOSS Developer

Ok, that's what I hoped anyway. It just wasn't clear from your description.

But this is XML, right? What do you need to pass the encoding for at this
point?

There shouldn't be any serialisation errors (unless you try to recode byte
strings on the way out, which is a no-no for arbitrary user input). All you
have to do is properly escape the byte string so that it passes the XML
encoding step.

One trick to do that is to decode the byte string as ISO-8859-1 and
serialise the result as a normal Unicode string. Then you can re-encode the
unicode string on input back to ISO-8859-1.

I choose ISO-8859-1 here because it has the well-defined side-effect of
mapping byte values directly to Unicode characters with an identical code
point value. So you do not risk any failures or data loss.

Stefan

Stefan Behnel <stefan_ml at behnel.de>
on Monday 24 August 2009 09:00
wrote in comp.lang.python:

The user may want a different encoding, other than utf-8, it can
be any encoding supported by libxml2.
Yup, but if the encodings are mixed inside Python byte strings, I think
that there is no way to know which encoding are using them. This may cause
XML serialization errors, by having a different encoding that the user
have set as the document encoding.
Sure, but if there are Python byte strings (not Unicode strings), ones
encoded in big5 and others in iso-8859-1 inside the object tree, the
XML serialization would throw errors on the encoding conversion, by
setting those bytes inside the document...
Thanks for commenting, and sorry for the late answer. This day was
busy...

Best regards,
- --
.O. | Daniel Molina Wegener | FreeBSD & Linux
..O | dmw [at] coder [dot] cl | Open Standards
OOO | http://coder.cl/ | FOSS Developer

I really meant what I wrote: this is XML. The encoding is well defined in
the XML declaration at the start of the document (and will default to UTF-8
if not provided). Passing it externally will allow users to override that,
which doesn't make any sense at all.

Yes, but only if you try to recode the strings (which, as I said, is a no-no).

No, I really meant: decoding from ISO-8859-1 to Unicode, for all byte
strings, regardless of their encoding (since you can't even know if they
represent encoded text at all). So you get a unicode string that you can
serialise to the target encoding, although it may result in character
references (&#xyz;) being output. But you won't get any errors, at least.

On the way in, you get a unicode string again, which you can encode to
ISO-8859-1 to get the original byte string back.

Stefan

Hmm, having written that, I guess it's actually best to encode byte strings
as base64 instead. Otherwise, null bytes and other special byte values
won't pass.

I also think that if the user wants readable output for text strings, it's
reasonable to require Unicode input instead of byte strings. Handling text
in byte strings is just too error prone.

Still, you may have to sanitize text input to make sure it doesn't contain
special characters either. Take a look at the way lxml does it in the
apihelpers.pxi source file, or read the XML spec on character content.

Stefan

Stefan Behnel <stefan_ml at behnel.de>
on Tuesday 25 August 2009 01:23
wrote in comp.lang.python:

Thanks, I will look for that. I must to do a better implementation on
handling byte strings, since would be many cases on where encoded strings
are mixed. For example different database inputs with different
encodings --- if those byte strings are not readed as Unicode strings.

Both sanitizing and base64 encoding are good options, and also, both are
readable from other platforms. The problem with later implementations of
pyxser was that it was using *RawUnicodeEscape* which is not readable from
other platforms.
Best regards,
- --
.O. | Daniel Molina Wegener | FreeBSD & Linux
..O | dmw [at] coder [dot] cl | Open Standards
OOO | http://coder.cl/ | FOSS Developer