[go-nuts] How to handle UTF-16 LE XML

You have to do it in CharsetReader; In the example you pasted,
charset.NewReaderLabel handles the decoding/encoding for you. Internally Go
only handles utf-8.

I think this should be the real encoding; in your case utf-16. The value
found in the <?xml?> tag is passed to CharsetReader as first argument. The
second argument is the reader itself.

CharsetReader must return the Reader of the utf-8 encoded contents, or an
error. This is exactly (and not incidentally) what NewReaderLabel does :)

Thanks for your clarification. My problem now seems to be the BOM. The code
looks like this:

b, _ := ioutil.ReadAll(xmlFile)
text := strings.NewReader(string(b))
decoder := xml.NewDecoder(text)
decoder.CharsetReader = charset.NewReaderLabel


When I print out the text variable I get:

&{??<?xml version="1.0" encoding="UTF-16"?>


At the start of the file. The two leading question marks are probably the
BOM marks. I get the error message:

XML syntax error on line 1: invalid UTF-8

from the decoder....

charset encoding should be able to handle BOM (because the unicode
transforms it uses do). two things to try: see what DetermineEncoding
says about your text, and then add your test file to the
sniffTestCases inside the charset package's charset_test.go.

if the latter sounds like too much, just print the first line of your
"text" variable fmt'd using %q and let's see exactly those bytes are.

Hi Andrey,

here is the printout with %q of the first few characters:

&{"\xff\xfe<\x00?\x00x\x00m\x00l\x00
\x00v\x00e\x00r\x00s\x00i\x00o\x00n\x00=\x00\"\x001\x00.\


I will try your suggestions later on and let you know what´s going on.



On Thursday, September 3, 2015 at 5:36:32 PM UTC+2, andrey mirtchovski
wrote:

The problem lies with encoding/xml's design: in order to use the
charset reader the xml library needs to examine the first line of text
from the xml file (where the encoding is specified). unfortunately
that first line contains invalid UTF-8 already, and libxml barfs
before it even figures out what the encoding should be to pass it to
our charset reader.

To solve this we can cheat and pass the input through the charset
reader before it goes to encoding/xml. Unfortunately in that case the
XML library will find an UTF-16 encoding specified in the xml headers
and will complain that it has no charset reader to convert that. We
solve this by supplying a dummy charset reader.

This is inelegant but I don't see another solution, at least not a
quick one. There is talk of redesigning xml, so if you file a bug
there may be a chance to get this fixed somehow in the library.

I've attached the test program and the test utf-16le encoded file I
created from your initial input. Sample run below:

$ hexdump -C bomtest.txt
00000000 ff fe 3c 00 3f 00 78 00 6d 00 6c 00 20 00 76 00 |..<.?.x.m.l. .v.|
00000010 65 00 72 00 73 00 69 00 6f 00 6e 00 3d 00 22 00 |e.r.s.i.o.n.=.".|
00000020 31 00 2e 00 30 00 22 00 20 00 65 00 6e 00 63 00 |1...0.". .e.n.c.|
00000030 6f 00 64 00 69 00 6e 00 67 00 3d 00 22 00 55 00 |o.d.i.n.g.=.".U.|
00000040 54 00 46 00 2d 00 31 00 36 00 22 00 3f 00 3e 00 |T.F.-.1.6.".?.>.|
00000050 20 00 20 00 0d 00 0a 00 3c 00 4f 00 75 00 74 00 | . .....<.O.u.t.|
00000060 65 00 72 00 3e 00 3c 00 49 00 6e 00 6e 00 65 00 |e.r.>.<.I.n.n.e.|
00000070 72 00 3e 00 74 00 65 00 73 00 74 00 3c 00 2f 00 |r.>.t.e.s.t.<./.|
00000080 49 00 6e 00 6e 00 65 00 72 00 3e 00 3c 00 2f 00 |I.n.n.e.r.>.<./.|
00000090 4f 00 75 00 74 00 65 00 72 00 3e 00 0d 00 0a 00 |O.u.t.e.r.>.....|
000000a0 3c 00 2f 00 78 00 6d 00 6c 00 3e 00 0d 00 0a 00 |<./.x.m.l.>.....|
000000b0
$ go run t.go
test
$

Thank you very much, for your workaround code, it works !!

Also many thanks to all who posted in order to solve the problem. You are a
special community, always willing to help.

Overengeneered. os.File already implements io.Reader,
so just do

decoder := xml.NewDecoder(xmlFile)
decoder.CharsetReader = charset.NewReaderLabel
OK, so I'd then employ buffering and its ability to "peek" at the data,
literally, and discard it, if needed:
http://play.golang.org/p/zGrNnYRkPF

Adding Andy and Marcel for their thoughts re how a UTF-16 charset
reader from golang.org/x/net/html/charset should handle BOMs.

As I understand the WHATWG spec, the BOMs are to be left in the decoded output, to be ignored by the tokenizer. But I can’t imagine what problem it would cause for the decoder to remove them.