Hello everyone!
Having some troubles decoding data with invalid bytes. From encoding.go:
type Encoding interface {
         // NewDecoder returns a transformer that converts to UTF-8.
         // Transforming source bytes that are not of that encoding will not
         // result in an error per se. Each byte that cannot be transcoded
         // be represented in the output by the UTF-8 encoding of '\uFFFD',
         // replacement rune.
         NewDecoder() transform.Transformer

So I expect that decoder will not stop on first error and just replaces
invalid bytes with U+FFFD.
Can I do something to force this behavior?
My code is very simple:
t := japanese.ShiftJIS
reader := transform.NewReader(file, t.NewDecoder())
data, err := ioutil.ReadAll(reader)
if err != nil {

Resulting in:
*japanese: invalid Shift JIS encoding*

I tried some workaround, but I think it's ugly and potentially bugged as
far as I can see:
func ForceDecode(src []byte, t transform.Transformer) (result []byte,
skipped int) {
         src0, src1 := 0, len(src)
         for src0 != src1 {
                 buf, n, err := transform.Bytes(t, src[src0:src1])
                 src0 += n
                 result = append(result, buf...)
                 if err != nil {

Any advice would be appreciated.

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 3 | next ›
Discussion Overview
groupgolang-nuts @
postedDec 2, '15 at 3:21p
activeDec 19, '15 at 2:40p

3 users in discussion

Rin Tohsaka: 1 post Nigel Tao: 1 post Mpvl: 1 post



site design / logo © 2022 Grokbase