Unicode¶
Legacy Wiki Page
This page was migrated from the old MoinMoin-based wiki. Information may be outdated or no longer applicable. For current documentation, see python.org.
Resources to help you learn how to handle Unicode in your Python programs:
General Unicode Resources¶
The Absolute Minimum Every Software Developer Must Know about Unicode - short intro to Unicode
Unicode â Wikipedia unicode entry
Python-Specific Resources¶
Standard Reference¶
Search the Python reference for:
unichr builtin
string handling - example:
u'Hello\u0020World !'unicodedata module
regular expressions - see the
(?u)flag, and there.UNICODEconstantexceptions -
UnicodeEncodeError
Tutorials¶
Python and Unicode (pdf talk) He also has a brief tutorial.
Unicode for Programmers - Java and Python info
Python Unicode Objects - brief notes
Sample Code¶
HTMLifying and UnHTMLifying - see atomef.py
Pitfalls¶
The standard encodings list is for the current version of python. GB2312 (PRC Chinese,) for example, is in Python2.4, but not in Python2.2, nor Python2.3.
Supported Encodings¶
There is a list of standard encodings in the Python documentation.
Encodings can be registered at runtime, as well, with the codecs module.
Python2.4 supports many codecs that 2.2 and 2.3 do not, including Chinese bg2312.
Encodings are specified in files found in a directory called âencodingsâ; one way to find the encodings with your Python distribution is to check the contents of this directory:
>>> import encodings, os
>>> [n for n in os.listdir(os.path.dirname(encodings.__file__))
... if n[0] != '_' and n.endswith('.py')]
['aliases.py', 'ascii.py', 'base64_codec.py', 'charmap.py', 'cp037.py', ...]
Another is to list aliases from the encodings module.
>>> import encodings
>>> from encodings import aliases
>>> aliases.aliases
{'base64': 'base64_codec', 'us_ascii': 'ascii', ...}
Contributors: LionKimbro, FredDrake, JĂŒrgenHermann.
âThe Truth about Unicode in Pythonâ¶
Discussion¶
Hereâs a conversation that I had on CommunityWiki; Iâd like to bring the main ideas into here.
Conversation between Lion and Bayle¶
That looks like 32-bits per character, so Iâd say itâs some form of little-endian utf-32.
And for some strange reason, Python only comes with âutf-8â and âutf-16â as valid âdecodeâ values.
You can:
1 >>> bytes = "H\x00i\x00\n\x00"
2 >>> unistring = bytes.decode('utf-16')
3 >>> print unistring
4 u'Hi\n'
You can do that for either âutf-8â or âutf-16.â But for some reason, I canât say âutf-32â or âutf-32LEâ (LE=little endian). I have no idea why. I also donât know how it is that my Python programs are producing UTF-32 for you..!
Iâve been wanting to diagram how Python unicode works, like how I diagrammed itâs time use, and regex use.
Basically, âencodeâ is meant to be called from unicode data, and âdecodeâ is meant to be called from bytes data. Continuing from above:
1 >>> bytes
2 'H\x00i\x00\n\x00'
3 >>> unistring = bytes.decode('utf-16')
4 >>> unistring
5 u'Hi\n'
6 >>> unistring.encode('utf-8')
7 'Hi\n'
8 >>> unistring.encode('utf-16')
9 '\xff\xfeH\x00i\x00\n\x00'
10 >>> unistring.encode('utf-32')
11 Traceback (most recent call last):
12 File "<stdin>", line 1, in ?
13 LookupError: unknown encoding: utf-32
Iâm guessing that the â\xffâ at the beginning of the utf-16 encoding is a byte-order marker, saying âthis is little endian.â
I learned about unicode stuff about 2-3 weeks ago. I kept notes about what I thought were the largest mental misconceptions, and what were the most revealing ways of thinking about it. Sadly, Iâve forgotten about all that. (Shouldâa documented it in the wiki!)
In Python, the data in a unicode or byte string is exactly the same. The difference is only in how Python treats and presents the data. I found it super-helpful to not think about what the console said, or work with the console, because the console lies. That is, the characters go through conversions even being printed to the screen: Your console has an understanding of encoding, and your fonts have an understanding of encoding, and I had a lot of difficulty seperating it out.
I had a lot easier time thinking about the concepts, instead of the concrete representations. (Which is opposite my usual course of thinking.)
âDecoded,â to Pythonâs mind, is data being treated as unicode data. âEncoded,â to Pythonâs mind, is data being treated as bytes. The data isnât actually changing form, at all. Itâs just the treatment of the same data that is being changed. But there is no actual conversion taking place.
So, you only ever run âdecodeâ on a byte string. (Another thing: Donât think of native Python strings as âstrings.â Think of them as âbytes.â And indeed: In the new Python 3.0, theyâre calling it just that: strings are called âbytesâ in Python3, and unicode strings are called just âstringsâ in Python3.)
So you can decode bytes, and encode unicode strings.
Donât think about decoding unicode strings, and donât think about encoding bytes. The bytes are already coded. Only unicode strings live in pure, abstract, heavenly, platonic form. There is no code there, only perfect clarity. (At least, thatâs how Python makes it seem for you.)
Again, sadly, I have no idea how to get from UTF-32 to Python unicode. I donât see the path. I saw something somewhere about being able to compile something in to your Python.
That said, if Iâm actually serving UTF-32 to you somehow,⊠âŠthen thereâs probably a way I just donât know.
(Discussion continued)¶
On side-notes, I think the diagrams Iâve posted for WorkingWithTime and RegularExpressions were eaten up in the transition to MoinMoin 1.3; Iâll repost them soon, after I get my own wiki upgraded to 1.3.
â LionKimbro 2005-02-06 02:56:28