FAQ

Quoth Sean:
I'm really new to dealing with unicode, so please bear with me. I'm
trying to add unicode support to a program I'm working on, and I'm
getting stuck a little when printing a unicode string to a file. I
know I have to encode the string using an encoding (UTF-8, UTF-16,
latin-1, etc). The problem is that I don't know how to determine what
the *right* encoding to use on a particular string is. The way I
understand it, utf-8 will handle any unicode data, but it will
translate characters not in the standard ASCII set to fit within the
8-bit character table. [...]
Actually characters outside ASCII get turned into multibyte
sequences by UTF-8. To use an example that came up here a little
while back:
u = u'\N{DEGREE SIGN}' # U+00B0; not in ASCII
u
u'\xb0'
u.encode('utf-8') # two-byte sequence
'\xc2\xb0'

UTF-8 is capable of representing any Unicode character without
information loss, so if you need to deal with arbitrary Unicode
it's a good choice. (It also has other pleasant properties.)
[...] My problem is I'm handling data from a lot of
different encodings (latin, eastern, asian, etc) and I can't allow
data in the strings to be changed. I also can't (at least I don't
know how to) determine what encodings the strings are using. IE, I
don't know what strings are from what languages. [...]
I think I detect a confusion or two here.

To encode is to turn (a sequence of) characters into (a sequence
of) bytes, and to decode is the reverse. An encoding is a scheme
for doing these things; it need not be strongly associated with
any particular language. (Though often an encoding can only
represent certain characters, and as a result is only useful for
those languages which use just those characters.)

A Unicode string (that is, a Python object of type 'unicode', such
as u'foo') is a sequence of characters, not bytes. It therefore
is not in any particular encoding.

A normal string (that is, a Python object of type 'str', such as
'foo') is a sequence of bytes, not characters. It can be
interpreted as a sequence of characters only if an encoding is
used to decode it. (Usually ASCII is assumed if you do not
specify one explicitly.)

So, if you have a bunch of normal strings and don't know what
encodings they're in, you're hooped. But if you have a bunch of
Unicode strings, it doesn't make sense to ask what encodings
they're in.

Now, as for not allowing the data in your strings to be changed:
If you mean you need to preserve the same sequence of characters,
then it's okay to change the encoding. You'll almost certainly
want the file you produce to be all in one encoding, so you'll
want an encoding which can represent any character you might
encounter -- UTF-8, for example.

But if you mean you need to preserve the exact byte sequences,
you're hooped. (Unless you'd be happy to have the output file be
a mishmash of different encodings and so virtually unusable.)
[...] Is there any way to
determine, from the unicode string itself, what encoding I need to use
to prevent data loss? Or do I need to find a way to determine
beforehand what encoding they are using when they are read in?
You will have to know the encoding at the input stage; use that
information to decode the bytes into a Unicode string. Then
assemble the Unicode strings to be output and encode them in, for
example, UTF-8.

In principle you might be able to look at the characters in a
Unicode string and determine some "least encoding" which could
deal with all the characters in it. But there's not much point to
this, imho; just use UTF-8, which can handle anything.

--
Steven Taschuk staschuk at telusplanet.net
Receive them ignorant; dispatch them confused. (Weschler's Teaching Motto)

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 4 of 6 | next ›
Discussion Overview
grouppython-list @
categoriespython
postedApr 29, '03 at 8:42p
activeApr 30, '03 at 3:27p
posts6
users4
websitepython.org

People

Translate

site design / logo © 2022 Grokbase