FAQ
Hello,

I am trying to see the compression ratio of a json string. Is this program
correct. gzipping the json string yields 94% reduction in size?

http://play.golang.org/p/QEmtBskEji

package main

import (
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"strconv"
)

const (
_ = iota
KB ByteSize = 1 << (10 * iota)
MB
)

type ByteSize float64

func (b ByteSize) String() string {
switch {
case b >= MB:
return fmt.Sprintf("%.2fMB", b/MB)
case b >= KB:
return fmt.Sprintf("%.2fKB", b/KB)
}
return fmt.Sprintf("%.2fB", b)
}

type Type struct {
I int64
N string
B bool
M int
}
type Cont struct {
I int64
A int64
B int64
C Type
D string
E int

F string
G int64
H int
J int
}

func main() {

var (
c Cont
tp Type
)

cmap := make(map[string]Cont)
for i := 0; i < 1000; i++ {

tp = Type{
I: int64(8888888888888 + i),
N: "ASASASDFSFSFFFSFS",
B: false,
M: 25,
}
c = Cont{
I: int64(9999999999999 + i),
A: int64(8888888888888),
B: int64(9999999999999 + i),
C: tp,
D: "A",
E: 1,
F: "ASASDASASASASASA",
G: int64(8888888888888),
H: 10,
J: 1,
}

cmap[strconv.FormatInt(c.I, 10)] = c
}

b, err := json.Marshal(&cmap)
if err != nil {
panic(err)
}

fmt.Printf("Json string : %s \n", ByteSize(len(string(b))))

buf := new(bytes.Buffer)
gz := gzip.NewWriter(buf)

_, err = gz.Write(b)
if err != nil {
panic(err)
}

err = gz.Close()
if err != nil {
panic(err)
}

fmt.Printf("Buf : %s \n", ByteSize(buf.Len()))
fmt.Printf("Savings: %.2f %% \n",
100*(1-(float64(buf.Len())/float64(len(string(b))))))

}

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Chris dollin at Oct 8, 2013 at 11:02 am
    On 8 October 2013 11:56, bsr wrote:

    Hello,
    I am trying to see the compression ratio of a json string. Is this program
    correct. gzipping the json string yields 94% reduction in size?

    http://play.golang.org/p/QEmtBskEji


    I don't know whether the program is right, but that's a lot of lot of lot
    of
    repetition in the string you're zipping up; it doesn't seem unreasonable
    that it can compress by a factor of 20. This will of course not be a
    /typical/ compression ratio.

    Chris

    --
    Chris "allusive" Dollin

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Bsr at Oct 8, 2013 at 11:35 am
    Thanks Chris. I was trying to add randomness, and made this change. (Still,
    there are repetition in Json keys.)
    http://play.golang.org/p/39XiZ0swzg

      the output is way different for local and play server. Is there anything
    to do with changes recently in std lib?


    play server:

    Json string : 201.17KB Buf : 17.18KB Savings: 91.46 %


    local:

    Json string : 201.17KB
    Buf : 47.03KB
    Savings: 76.62 %


    go version
    go version go1.1.2 darwin/amd64




    On Tuesday, October 8, 2013 7:02:04 AM UTC-4, chris dollin wrote:

    On 8 October 2013 11:56, bsr <bsr...@gmail.com <javascript:>> wrote:

    Hello,
    I am trying to see the compression ratio of a json string. Is this
    program correct. gzipping the json string yields 94% reduction in size?

    http://play.golang.org/p/QEmtBskEji


    I don't know whether the program is right, but that's a lot of lot of lot
    of
    repetition in the string you're zipping up; it doesn't seem unreasonable
    that it can compress by a factor of 20. This will of course not be a
    /typical/ compression ratio.

    Chris

    --
    Chris "allusive" Dollin
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Coda Hale at Oct 8, 2013 at 3:59 pm
    play.golang.org is sandboxed, which means crypto/rand (which usually reads
    from /dev/urandom) returns all zeros:

    http://play.golang.org/p/TLrvkC6NfH

    This adds dramatically less entropy to the input, allowing for higher
    compression ratios.

    On Tue, Oct 8, 2013 at 4:34 AM, bsr wrote:

    Thanks Chris. I was trying to add randomness, and made this change.
    (Still, there are repetition in Json keys.)
    http://play.golang.org/p/39XiZ0swzg

    the output is way different for local and play server. Is there anything
    to do with changes recently in std lib?


    play server:

    Json string : 201.17KB Buf : 17.18KB Savings: 91.46 %


    local:

    Json string : 201.17KB
    Buf : 47.03KB
    Savings: 76.62 %


    go version
    go version go1.1.2 darwin/amd64




    On Tuesday, October 8, 2013 7:02:04 AM UTC-4, chris dollin wrote:

    On 8 October 2013 11:56, bsr wrote:

    Hello,
    I am trying to see the compression ratio of a json string. Is this
    program correct. gzipping the json string yields 94% reduction in size?

    http://play.golang.org/p/**QEmtBskEji<http://play.golang.org/p/QEmtBskEji>


    I don't know whether the program is right, but that's a lot of lot of
    lot of
    repetition in the string you're zipping up; it doesn't seem unreasonable
    that it can compress by a factor of 20. This will of course not be a
    /typical/ compression ratio.

    Chris

    --
    Chris "allusive" Dollin
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Coda Hale
    http://codahale.com

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Bsr at Oct 8, 2013 at 4:08 pm
    great explanation. thank you very much.
    On Tuesday, October 8, 2013 11:59:45 AM UTC-4, Coda Hale wrote:

    play.golang.org is sandboxed, which means crypto/rand (which usually
    reads from /dev/urandom) returns all zeros:

    http://play.golang.org/p/TLrvkC6NfH

    This adds dramatically less entropy to the input, allowing for higher
    compression ratios.


    On Tue, Oct 8, 2013 at 4:34 AM, bsr <bsr...@gmail.com <javascript:>>wrote:
    Thanks Chris. I was trying to add randomness, and made this change.
    (Still, there are repetition in Json keys.)
    http://play.golang.org/p/39XiZ0swzg

    the output is way different for local and play server. Is there anything
    to do with changes recently in std lib?


    play server:

    Json string : 201.17KB Buf : 17.18KB Savings: 91.46 %


    local:

    Json string : 201.17KB
    Buf : 47.03KB
    Savings: 76.62 %


    go version
    go version go1.1.2 darwin/amd64




    On Tuesday, October 8, 2013 7:02:04 AM UTC-4, chris dollin wrote:

    On 8 October 2013 11:56, bsr wrote:

    Hello,
    I am trying to see the compression ratio of a json string. Is this
    program correct. gzipping the json string yields 94% reduction in size?

    http://play.golang.org/p/**QEmtBskEji<http://play.golang.org/p/QEmtBskEji>


    I don't know whether the program is right, but that's a lot of lot of
    lot of
    repetition in the string you're zipping up; it doesn't seem unreasonable
    that it can compress by a factor of 20. This will of course not be a
    /typical/ compression ratio.

    Chris

    --
    Chris "allusive" Dollin
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Coda Hale
    http://codahale.com
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 8, '13 at 10:56a
activeOct 8, '13 at 4:08p
posts5
users3
websitegolang.org

3 users in discussion

Bsr: 3 posts Chris dollin: 1 post Coda Hale: 1 post

People

Translate

site design / logo © 2021 Grokbase