Child pages
  • unicoding
Skip to end of metadata
Go to start of metadata

I stored the same text file in various encodings with notepad (a fine tool for unicode diagnostics). I read in those files in python, as plain 8-bit streams. No surprises there:

Here is a breakdown of the BOMs (http://en.wikipedia.org/wiki/Byte-order_mark):

UTF-8

EF BB BF

UTF-16 BE ("unicode big endian")

FE FF

UTF-16 LE ("unicode")

FF FE

decode works as expected:

Of course it happened again: program works okay, unit-tests suck and must be debugged.

At this point, I have my unicoder (18:36), and the unit-tests show that it works.

  • No labels