FAQ
Hello,

I'm serialising a struct into a []byte slice (for base64-encoding into a
cookie, but that's not relevant). There are several ways to do this,
such as formatting the data into a string with fmt and converting that
to []byte, using append to construct the []byte slice, or using a
bytes.Buffer.

I guess using fmt and converting the resulting string to []byte looks
easier than manually constructing the []byte slice, but perhaps this is
the slowest method. I'm not sure how one should weigh the use of strings
vs. []byte slices.

(I'm ignoring solutions such as the gob package here because I'd like
the data to be accessible to other languages.)

I was wondering which method of serialising to (plain) text is the most
idiomatic.

Opinions?



Some sample code:

type TimeStamp struct {
Stamp time.Time
Seqno uint16
Host []byte
Hmac []byte
}


func (ts *TimeStamp) String() string {
return fmt.Sprintf("%d.%d %d %s",
ts.Stamp.Unix(), ts.Stamp.Nanosecond(), ts.Seqno, ts.Host)
}


func (ts *TimeStamp) Bytes() []byte {
epoch := make([]byte, 20) // unix(10) + '.' + nsec(9)
epoch = strconv.AppendInt(epoch, ts.Stamp.Unix(), 10)
epoch = append(epoch, '.')
epoch = strconv.AppendInt(epoch, int64(ts.Stamp.Nanosecond()), 10)

var buf bytes.Buffer
buf.Write(epoch)
buf.WriteByte(0x20)
buf.Write(strconv.AppendUint(nil, uint64(ts.Seqno), 10))
buf.WriteByte(0x20)
buf.Write(ts.Host)

return buf.Bytes()
}

--

Search Discussions

  • Jonathan Pittman at Sep 24, 2012 at 4:00 am
    If you want relatively plain text, JSON usually looks rather nice and works
    with other languages.
    On Sun, Sep 23, 2012 at 10:29 PM, wrote:


    Hello,

    I'm serialising a struct into a []byte slice (for base64-encoding into a
    cookie, but that's not relevant). There are several ways to do this, such
    as formatting the data into a string with fmt and converting that to
    []byte, using append to construct the []byte slice, or using a bytes.Buffer.

    I guess using fmt and converting the resulting string to []byte looks
    easier than manually constructing the []byte slice, but perhaps this is the
    slowest method. I'm not sure how one should weigh the use of strings vs.
    []byte slices.

    (I'm ignoring solutions such as the gob package here because I'd like the
    data to be accessible to other languages.)

    I was wondering which method of serialising to (plain) text is the most
    idiomatic.

    Opinions?



    Some sample code:

    type TimeStamp struct {
    Stamp time.Time
    Seqno uint16
    Host []byte
    Hmac []byte
    }


    func (ts *TimeStamp) String() string {
    return fmt.Sprintf("%d.%d %d %s",
    ts.Stamp.Unix(), ts.Stamp.Nanosecond(), ts.Seqno, ts.Host)
    }


    func (ts *TimeStamp) Bytes() []byte {
    epoch := make([]byte, 20) // unix(10) + '.' + nsec(9)
    epoch = strconv.AppendInt(epoch, ts.Stamp.Unix(), 10)
    epoch = append(epoch, '.')
    epoch = strconv.AppendInt(epoch, int64(ts.Stamp.Nanosecond()), 10)

    var buf bytes.Buffer
    buf.Write(epoch)
    buf.WriteByte(0x20)
    buf.Write(strconv.AppendUint(**nil, uint64(ts.Seqno), 10))
    buf.WriteByte(0x20)
    buf.Write(ts.Host)

    return buf.Bytes()
    }

    --

    --
  • Brian Slesinsky at Sep 24, 2012 at 4:14 am
    You've combined a few different questions. If you're asking which format to
    use, JSON is a good choice if you prefer plain text or want to communicate
    with JavaScript, or protocol buffers work well as a binary format for
    communicating with other server-side languages.

    To write output to a byte stream efficiently, the idiom is to take an
    io.Writer as a parameter rather than returning a string or []byte. So if
    you're using fmt then fmt.Fprintf will be better than fmt.Sprintf.

    If you want to make your type printable using the fmt package, you have the
    choice of implementing the Stringer, GoStringer, or Formatter interfaces in
    the fmt package. For large amounts of data, the Formatter interface will be
    more efficient since it gives you an io.Writer (the State type implements
    Writer) and avoids the intermediate conversion to a string.

    - Brian
    On Sunday, September 23, 2012 7:29:11 PM UTC-7, (unknown) wrote:


    Hello,

    I'm serialising a struct into a []byte slice (for base64-encoding into a
    cookie, but that's not relevant). There are several ways to do this,
    such as formatting the data into a string with fmt and converting that
    to []byte, using append to construct the []byte slice, or using a
    bytes.Buffer.

    I guess using fmt and converting the resulting string to []byte looks
    easier than manually constructing the []byte slice, but perhaps this is
    the slowest method. I'm not sure how one should weigh the use of strings
    vs. []byte slices.

    (I'm ignoring solutions such as the gob package here because I'd like
    the data to be accessible to other languages.)

    I was wondering which method of serialising to (plain) text is the most
    idiomatic.

    Opinions?



    Some sample code:

    type TimeStamp struct {
    Stamp time.Time
    Seqno uint16
    Host []byte
    Hmac []byte
    }


    func (ts *TimeStamp) String() string {
    return fmt.Sprintf("%d.%d %d %s",
    ts.Stamp.Unix(), ts.Stamp.Nanosecond(), ts.Seqno, ts.Host)
    }


    func (ts *TimeStamp) Bytes() []byte {
    epoch := make([]byte, 20) // unix(10) + '.' + nsec(9)
    epoch = strconv.AppendInt(epoch, ts.Stamp.Unix(), 10)
    epoch = append(epoch, '.')
    epoch = strconv.AppendInt(epoch, int64(ts.Stamp.Nanosecond()), 10)

    var buf bytes.Buffer
    buf.Write(epoch)
    buf.WriteByte(0x20)
    buf.Write(strconv.AppendUint(nil, uint64(ts.Seqno), 10))
    buf.WriteByte(0x20)
    buf.Write(ts.Host)

    return buf.Bytes()
    }
    --
  • Wouter-golang at Sep 24, 2012 at 5:59 pm
    First of all, thanks for your replies!

    I'm generating a short UUID-like time stamp, so I think JSON might be
    too verbose for this purpose.

    I admit my question wasn't very concrete.

    The fmt package makes it easy to get a string representation of a struct
    – it's basically a one-liner (it feels higher-level).

    However, many hash, encryption and encoding (e.g. base64) functions
    quite logically take (or return) a []byte slice or buffer.

    Some Go packages freely mix strings and []byte slices depending on
    convenience, converting back and forth; especially converting fmt string
    output to []byte.

    My question is if it's ok to convert fmt string to []byte, or would it
    be better to stick with []byte if that's what's needed down the line and
    emulate fmt formatting by using append/appendInt/buffers directly on
    []byte, avoiding conversions and memory overhead.

    Since there is a lot of overlap between strings and []byte (the
    functions in their respective modules are pretty similar too), it's not
    always clear which to prefer.

    --
  • Kyle Lemons at Sep 24, 2012 at 6:50 pm
    Usually if you are working with []byte, you'll have a buffer or stream into
    which you can Fprint and avoid the conversion
    On Mon, Sep 24, 2012 at 8:57 AM, wrote:


    First of all, thanks for your replies!

    I'm generating a short UUID-like time stamp, so I think JSON might be too
    verbose for this purpose.

    I admit my question wasn't very concrete.

    The fmt package makes it easy to get a string representation of a struct –
    it's basically a one-liner (it feels higher-level).

    However, many hash, encryption and encoding (e.g. base64) functions quite
    logically take (or return) a []byte slice or buffer.

    Some Go packages freely mix strings and []byte slices depending on
    convenience, converting back and forth; especially converting fmt string
    output to []byte.

    My question is if it's ok to convert fmt string to []byte, or would it be
    better to stick with []byte if that's what's needed down the line and
    emulate fmt formatting by using append/appendInt/buffers directly on
    []byte, avoiding conversions and memory overhead.

    Since there is a lot of overlap between strings and []byte (the functions
    in their respective modules are pretty similar too), it's not always clear
    which to prefer.

    --

    --
  • Brian Slesinsky at Sep 24, 2012 at 7:23 pm
    I'm generating a short UUID-like time stamp, so I think JSON might be too
    verbose for this purpose.

    Yes, that's true. JSON would be more appropriate if you had a collection of
    timestamps; you could store each timestamp in a string literal in the
    larger document.

    My question is if it's ok to convert fmt string to []byte, or would it
    be better to stick with []byte if that's what's needed down the line and
    emulate fmt formatting by using append/appendInt/buffers directly on
    []byte, avoiding conversions and memory overhead.
    In general, rather than appending to []byte, you should append to an
    io.Writer and let the caller decide what the destination should be; it
    might be a []byte but it could be a file or socket.

    Since there is a lot of overlap between strings and []byte (the
    functions in their respective modules are pretty similar too), it's not
    always clear which to prefer.
    There's nothing wrong with converting from string to byte[] - just remember
    that it does a copy. If performance matters to you, you'll want to avoid
    repeatedly copying the same data, which can result in an O(n^2) algorithm
    if you have a recursive data structure.

    A timestamp converts to constant-size output so it's probably not that bad
    unless it's used a lot. Appending to an io.Writer will be more important
    for variable-sized output.

    - Brian

    --
  • Wouter-golang at Sep 25, 2012 at 11:37 pm

    On 24.09.2012 22:23, Brian Slesinsky wrote:
    [...]

    A timestamp converts to constant-size output so it's probably not
    that bad unless it's used a lot. Appending to an io.Writer will be
    more important for variable-sized output.
    Yup. I guess I'd better go and benchmark one or two things... I've
    already noticed buffers seem to be significantly slower than []byte
    slices for small data.

    Thanks for the help guys!

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedSep 24, '12 at 3:19a
activeSep 25, '12 at 11:37p
posts7
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase