FAQ
Now I'm writing a net filesystem (like GFS) in golang. In my test, when put
many files( 10 million or more ) in the filesystem, the memory cost is up
to 4GB, and every a few minutes, the filesystem server will disconnect to
the client for about 10 seconds, because my server an client check heart
beat every two second.

I debug it, found:
1. during that time, the filesystem server is frozen
2. In cpu profile, "scanblock" cost more than 60% cpu time

so I guess that, golang runtime do GC every few minute, and GC cost much
time,. During GC, the memory and the goroutine freeze, server
can't response heart beats, so disconnected.

To exclude the possibility of memory leak, I wrote a small program to verify
it.

The program make a big map in memory, which bigger than 10GB, and heart
beat every second. when it lost heart beat, it will log

type fsnode struct {
id uint32
ctime uint32
mtime uint32
atime uint32
nodtype byte
mode uint16
uid uint32
gid uint32
}

var srvlog *syslog.Writer

func checkActive() {
checkTick := time.Tick(time.Second)
var timeNow, timePre int64
for {
select {
case <-checkTick:
timeNow = time.Now().Unix()
if timePre != 0 && timeNow - timePre != 1 {
srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
}
timePre = timeNow
}
}
}

func memLoad(){
fstab := make( map[uint32]*fsnode )
for i:=0; i<20000000; i++ {
nd := new(fsnode)
fstab[uint32(i)] = nd
}
}

func main() {
srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

go checkActive()
memLoad()
fmt.Scanln()

}


I run this program for more than 3 hours, during that time, every ten
minute it will occur a "heart beat lose"

This is a very simple data structure: only a big map in memory, so I think
maybe it's a GC Problem


So guys, have you ever met this kind of problem? I don't know much about
GC, I used to be a C programmer on embedded systems, golang is the first
and the only GC language I even used.

I'll be very thankful if you can tell me why or how to deal with it




--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Feeling4t at Apr 23, 2013 at 5:14 pm
    well, it's very interesting that, when I add runtime.GOMAXPROCS(4), the
    disconnect down form 3min a time to one hour a time
    so, maybe the scheduler also has problem?


    在 2013年4月24日星期三UTC+8上午1时10分15秒,feeling4t写道:

    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I think
    maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much about
    GC, I used to be a C programmer on embedded systems, golang is the first
    and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • ⚛ at Apr 23, 2013 at 5:33 pm
    Are you using Go 1.0.3 or Go tip?
    On Tuesday, April 23, 2013 7:10:15 PM UTC+2, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I think
    maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much about
    GC, I used to be a C programmer on embedded systems, golang is the first
    and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Feeling4t at Apr 24, 2013 at 5:15 am
    well. I'm using Go 1, need I update to go 1.03?

    在 2013年4月24日星期三UTC+8上午1时33分22秒,⚛写道:
    Are you using Go 1.0.3 or Go tip?
    On Tuesday, April 23, 2013 7:10:15 PM UTC+2, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much about
    GC, I used to be a C programmer on embedded systems, golang is the first
    and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Voidlogic7 at Apr 23, 2013 at 6:57 pm
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I think
    maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much about
    GC, I used to be a C programmer on embedded systems, golang is the first
    and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Feeling4t at Apr 24, 2013 at 5:22 am
    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much about
    GC, I used to be a C programmer on embedded systems, golang is the first
    and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Apr 24, 2013 at 5:24 am
    Run your program with GOGCTRACE=1 and post the results to this thread,
    that will show if GC pause times are affecting your program.
    On Wed, Apr 24, 2013 at 3:22 PM, feeling4t wrote:

    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server will disconnect to
    the client for about 10 seconds, because my server an client check heart
    beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server can't response
    heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to
    verify it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much about
    GC, I used to be a C programmer on embedded systems, golang is the first and
    the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it


    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Feeling4t at Apr 24, 2013 at 8:34 am
    how to use it, is it in runtime? I did't find it. Or just
    define GOGCTRACE as a const?

    在 2013年4月24日星期三UTC+8下午1时24分27秒,Dave Cheney写道:
    Run your program with GOGCTRACE=1 and post the results to this thread,
    that will show if GC pause times are affecting your program.
    On Wed, Apr 24, 2013 at 3:22 PM, feeling4t wrote:

    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test,
    when
    put many files( 10 million or more ) in the filesystem, the memory
    cost is
    up to 4GB, and every a few minutes, the filesystem server will
    disconnect to
    the client for about 10 seconds, because my server an client check
    heart
    beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost
    much
    time,. During GC, the memory and the goroutine freeze, server can't
    response
    heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to
    verify it.

    The program make a big map in memory, which bigger than 10GB, and
    heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about
    GC, I used to be a C programmer on embedded systems, golang is the
    first and
    the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it


    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Apr 24, 2013 at 8:37 am
    It is an environment variable, like GOPATH.


    On 24/04/2013, at 6:34 PM, feeling4t wrote:


    how to use it, is it in runtime? I did't find it. Or just define GOGCTRACE as a const?

    在 2013年4月24日星期三UTC+8下午1时24分27秒,Dave Cheney写道:
    Run your program with GOGCTRACE=1 and post the results to this thread,
    that will show if GC pause times are affecting your program.
    On Wed, Apr 24, 2013 at 3:22 PM, feeling4t wrote:

    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server will disconnect to
    the client for about 10 seconds, because my server an client check heart
    beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server can't response
    heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to
    verify it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much about
    GC, I used to be a C programmer on embedded systems, golang is the first and
    the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it


    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Voidlogic7 at Apr 24, 2013 at 5:48 am
    I'm not familar with the details of the Go map inplementation, but in
    general when a hashmap has too many collisions (and thus the linked lists
    or b-trees attached to each bucket are too big) it must expand and re-hash
    all the existing items into the new memory space.

    Based on what you are saying below, allocating a hashmap with enough
    capacity for the keys you are adding solved the issue. To me this indicates
    the pauses you were seeing were probably related to map
    growth/expansion/GC. In general if you are going to put more than a handful
    of items into a map, you should always use the version of make() that
    supplies the capacity.

    In regards to if Go 1.1 would fix your original code- it may, as it has an
    improved map implementation; however, the best solution is to do what your
    new code does and supply a initial capacity to make() when creating your
    map.
    On Wednesday, April 24, 2013 12:22:14 AM UTC-5, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Feeling4t at Apr 24, 2013 at 8:44 am
    I don't think map growth or re-hash cause this problem. My test is after
    fill millions of elements into the map, and during my test, the map won't
    be modify by my code.
    I think GC is the mainly reason, scanblock cost more then 60% cpu time.

    And it's hard for me to solve this problem by pre-allocate memory before
    insert element to the map, because the program I'm writing is a net
    filesystem, I can't know how many files users will write in


    在 2013年4月24日星期三UTC+8下午1时48分07秒,voidl...@gmail.com写道:
    I'm not familar with the details of the Go map inplementation, but in
    general when a hashmap has too many collisions (and thus the linked lists
    or b-trees attached to each bucket are too big) it must expand and re-hash
    all the existing items into the new memory space.

    Based on what you are saying below, allocating a hashmap with enough
    capacity for the keys you are adding solved the issue. To me this indicates
    the pauses you were seeing were probably related to map
    growth/expansion/GC. In general if you are going to put more than a handful
    of items into a map, you should always use the version of make() that
    supplies the capacity.

    In regards to if Go 1.1 would fix your original code- it may, as it has an
    improved map implementation; however, the best solution is to do what your
    new code does and supply a initial capacity to make() when creating your
    map.
    On Wednesday, April 24, 2013 12:22:14 AM UTC-5, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost
    much time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • ⚛ at Apr 24, 2013 at 6:22 am
    It is impossible for the garbage collector to quickly traverse huge
    hashmaps of type map[uint32]*fsnode because the traversal causes many CPU
    cache misses. Running your memLoad() function on my notebook results in 40%
    cache miss ratio. This means that a garbage collection is mostly waiting
    for data to arrive into the CPU cache. The memory access pattern is random
    because of hashing in the hashmap implementation.

    I would suggest to split the huge hashmap into multiple smaller hashmaps,
    or to replace the hashmap with a different data structure in your program,
    in order to avoid cache misses.
    On Wednesday, April 24, 2013 7:22:14 AM UTC+2, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Feeling4t at Apr 24, 2013 at 9:32 am
    well, I think the traversing of memory indeed takes much of time, because
    in cpu profile, I can see scanblock takes more than 60% cpu time.

    And I think, there may be a problem about the scheduler, when I set
    runtime.GOMAXPROCS(1), the disconnect happens every 3 minute, and last for
    more than 10 seconds. But when I set runtime.GOMAXPROCS(4), it's much
    better, the disconnect happens every one hour, and last for a few seconds

    Both of the runtime sets cost much time in scanblock, more than 60% cpu
    time.

    在 2013年4月24日星期三UTC+8下午2时22分06秒,⚛写道:
    It is impossible for the garbage collector to quickly traverse huge
    hashmaps of type map[uint32]*fsnode because the traversal causes many CPU
    cache misses. Running your memLoad() function on my notebook results in 40%
    cache miss ratio. This means that a garbage collection is mostly waiting
    for data to arrive into the CPU cache. The memory access pattern is random
    because of hashing in the hashmap implementation.

    I would suggest to split the huge hashmap into multiple smaller hashmaps,
    or to replace the hashmap with a different data structure in your program,
    in order to avoid cache misses.
    On Wednesday, April 24, 2013 7:22:14 AM UTC+2, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost
    much time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • ⚛ at Apr 24, 2013 at 9:39 am

    On Wednesday, April 24, 2013 11:32:32 AM UTC+2, feeling4t wrote:
    well, I think the traversing of memory indeed takes much of time, because
    in cpu profile, I can see scanblock takes more than 60% cpu time.

    And I think, there may be a problem about the scheduler, when I set
    runtime.GOMAXPROCS(1), the disconnect happens every 3 minute, and last for
    more than 10 seconds. But when I set runtime.GOMAXPROCS(4), it's much
    better, the disconnect happens every one hour, and last for a few seconds
    If runtime.GOMAXPROCS(4) is better than GOMAXPROCS(1) it means there isn't
    any problem in the scheduler.

    Both of the runtime sets cost much time in scanblock, more than 60% cpu
    time.

    在 2013年4月24日星期三UTC+8下午2时22分06秒,⚛写道:
    It is impossible for the garbage collector to quickly traverse huge
    hashmaps of type map[uint32]*fsnode because the traversal causes many CPU
    cache misses. Running your memLoad() function on my notebook results in 40%
    cache miss ratio. This means that a garbage collection is mostly waiting
    for data to arrive into the CPU cache. The memory access pattern is random
    because of hashing in the hashmap implementation.

    I would suggest to split the huge hashmap into multiple smaller hashmaps,
    or to replace the hashmap with a different data structure in your program,
    in order to avoid cache misses.
    On Wednesday, April 24, 2013 7:22:14 AM UTC+2, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test,
    when put many files( 10 million or more ) in the filesystem, the memory
    cost is up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost
    much time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and
    heart beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Feeling4t at Apr 24, 2013 at 9:47 am
    Err..... sorry, I mean, maybe it can be solve by different runtime setting?

    在 2013年4月24日星期三UTC+8下午5时39分36秒,⚛写道:
    On Wednesday, April 24, 2013 11:32:32 AM UTC+2, feeling4t wrote:


    well, I think the traversing of memory indeed takes much of time, because
    in cpu profile, I can see scanblock takes more than 60% cpu time.

    And I think, there may be a problem about the scheduler, when I set
    runtime.GOMAXPROCS(1), the disconnect happens every 3 minute, and last for
    more than 10 seconds. But when I set runtime.GOMAXPROCS(4), it's much
    better, the disconnect happens every one hour, and last for a few seconds
    If runtime.GOMAXPROCS(4) is better than GOMAXPROCS(1) it means there isn't
    any problem in the scheduler.

    Both of the runtime sets cost much time in scanblock, more than 60% cpu
    time.

    在 2013年4月24日星期三UTC+8下午2时22分06秒,⚛写道:
    It is impossible for the garbage collector to quickly traverse huge
    hashmaps of type map[uint32]*fsnode because the traversal causes many CPU
    cache misses. Running your memLoad() function on my notebook results in 40%
    cache miss ratio. This means that a garbage collection is mostly waiting
    for data to arrive into the CPU cache. The memory access pattern is random
    because of hashing in the hashmap implementation.

    I would suggest to split the huge hashmap into multiple smaller
    hashmaps, or to replace the hashmap with a different data structure in your
    program, in order to avoid cache misses.
    On Wednesday, April 24, 2013 7:22:14 AM UTC+2, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test,
    when put many files( 10 million or more ) in the filesystem, the memory
    cost is up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost
    much time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and
    heart beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it




    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • ⚛ at Apr 24, 2013 at 6:56 am

    On Wednesday, April 24, 2013 7:22:14 AM UTC+2, feeling4t wrote:
    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?
    GC of hashmaps in Go tip is slower than in Go 1. Optimizing the GC
    implementation requires substantial changes to the code. Go 1.1beta source
    code doesn't allow such changes any more, so the performance will only
    improve when Go 1.2+ arrives.

    An option is to replace map[uint32]*fsnode with []*fsnode. This should run
    much faster than map[uint32]*fsnode.

    Reallocation costs can be avoided by introducing a hierarchy into the data
    structure, such as [][256]*fsnode or a tree.

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost much
    time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Khr at Apr 24, 2013 at 7:35 am
    Both the old and new map implementations do incremental growing, so you
    shouldn't see large pauses even if you give a small (or no) size hint to
    make().

    Traversing the map during GC shouldn't cause many cache misses in and of
    itself, as traversals are done in order (for both old and new
    implementations). However, since the map value is a pointer, you have to
    take one miss per element to collect each value. A possible improvement,
    if your semantics allow it, is to use a map[uint32]fsnode instead of
    map[uint32]*fsnode. It will waste some space but should make GC scanning
    faster.

    I'll investigate some more tomorrow. I'm curious where the time is going.
    On Tuesday, April 23, 2013 11:56:31 PM UTC-7, ⚛ wrote:
    On Wednesday, April 24, 2013 7:22:14 AM UTC+2, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?
    GC of hashmaps in Go tip is slower than in Go 1. Optimizing the GC
    implementation requires substantial changes to the code. Go 1.1beta source
    code doesn't allow such changes any more, so the performance will only
    improve when Go 1.2+ arrives.

    An option is to replace map[uint32]*fsnode with []*fsnode. This should run
    much faster than map[uint32]*fsnode.

    Reallocation costs can be avoided by introducing a hierarchy into the data
    structure, such as [][256]*fsnode or a tree.

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test, when
    put many files( 10 million or more ) in the filesystem, the memory cost is
    up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost
    much time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and heart
    beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Feeling4t at Apr 24, 2013 at 9:35 am
    In the cpu profile, I can see more than 60% samples hit scanblock.

    Set runtime.GOMAXPROCS(1) to runtime.GOMAXPROCS(4) will be much better,
    disconnect happens much less often, but scanblock still cost more than 60%

    在 2013年4月24日星期三UTC+8下午3时08分14秒,k...@golang.org写道:
    Both the old and new map implementations do incremental growing, so you
    shouldn't see large pauses even if you give a small (or no) size hint to
    make().

    Traversing the map during GC shouldn't cause many cache misses in and of
    itself, as traversals are done in order (for both old and new
    implementations). However, since the map value is a pointer, you have to
    take one miss per element to collect each value. A possible improvement,
    if your semantics allow it, is to use a map[uint32]fsnode instead of
    map[uint32]*fsnode. It will waste some space but should make GC scanning
    faster.

    I'll investigate some more tomorrow. I'm curious where the time is going.
    On Tuesday, April 23, 2013 11:56:31 PM UTC-7, ⚛ wrote:
    On Wednesday, April 24, 2013 7:22:14 AM UTC+2, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?
    GC of hashmaps in Go tip is slower than in Go 1. Optimizing the GC
    implementation requires substantial changes to the code. Go 1.1beta source
    code doesn't allow such changes any more, so the performance will only
    improve when Go 1.2+ arrives.

    An option is to replace map[uint32]*fsnode with []*fsnode. This should
    run much faster than map[uint32]*fsnode.

    Reallocation costs can be avoided by introducing a hierarchy into the
    data structure, such as [][256]*fsnode or a tree.

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test,
    when put many files( 10 million or more ) in the filesystem, the memory
    cost is up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost
    much time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and
    heart beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • ⚛ at Apr 24, 2013 at 9:47 am
    Because type fsnode has no pointers, using []fsnode or [][N]fsnode will
    allow the garbage collector to finish almost immediately. To keep GC cost
    close to zero, data types that refer to filesystem nodes should use uint32
    instead of *fsnode.
    On Wednesday, April 24, 2013 11:35:36 AM UTC+2, feeling4t wrote:


    In the cpu profile, I can see more than 60% samples hit scanblock.

    Set runtime.GOMAXPROCS(1) to runtime.GOMAXPROCS(4) will be much better,
    disconnect happens much less often, but scanblock still cost more than 60%

    在 2013年4月24日星期三UTC+8下午3时08分14秒,k...@golang.org写道:
    Both the old and new map implementations do incremental growing, so you
    shouldn't see large pauses even if you give a small (or no) size hint to
    make().

    Traversing the map during GC shouldn't cause many cache misses in and of
    itself, as traversals are done in order (for both old and new
    implementations). However, since the map value is a pointer, you have to
    take one miss per element to collect each value. A possible improvement,
    if your semantics allow it, is to use a map[uint32]fsnode instead of
    map[uint32]*fsnode. It will waste some space but should make GC scanning
    faster.

    I'll investigate some more tomorrow. I'm curious where the time is going.
    On Tuesday, April 23, 2013 11:56:31 PM UTC-7, ⚛ wrote:
    On Wednesday, April 24, 2013 7:22:14 AM UTC+2, feeling4t wrote:


    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?
    GC of hashmaps in Go tip is slower than in Go 1. Optimizing the GC
    implementation requires substantial changes to the code. Go 1.1beta source
    code doesn't allow such changes any more, so the performance will only
    improve when Go 1.2+ arrives.

    An option is to replace map[uint32]*fsnode with []*fsnode. This should
    run much faster than map[uint32]*fsnode.

    Reallocation costs can be avoided by introducing a hierarchy into the
    data structure, such as [][256]*fsnode or a tree.

    在 2013年4月24日星期三UTC+8上午2时57分37秒,voidl...@gmail.com写道:
    What happens if you use this version of the memLoad function?

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    nodes := make([]fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    fstab[uint32(i)] = &nodes[i]
    }
    }

    Also are you using tip / 1.1 beta 2?

    On Tuesday, April 23, 2013 12:10:15 PM UTC-5, feeling4t wrote:


    Now I'm writing a net filesystem (like GFS) in golang. In my test,
    when put many files( 10 million or more ) in the filesystem, the memory
    cost is up to 4GB, and every a few minutes, the filesystem server
    will disconnect to the client for about 10 seconds, because my server an
    client check heart beat every two second.

    I debug it, found:
    1. during that time, the filesystem server is frozen
    2. In cpu profile, "scanblock" cost more than 60% cpu time

    so I guess that, golang runtime do GC every few minute, and GC cost
    much time,. During GC, the memory and the goroutine freeze, server
    can't response heart beats, so disconnected.

    To exclude the possibility of memory leak, I wrote a small program to verify
    it.

    The program make a big map in memory, which bigger than 10GB, and
    heart beat every second. when it lost heart beat, it will log

    type fsnode struct {
    id uint32
    ctime uint32
    mtime uint32
    atime uint32
    nodtype byte
    mode uint16
    uid uint32
    gid uint32
    }

    var srvlog *syslog.Writer

    func checkActive() {
    checkTick := time.Tick(time.Second)
    var timeNow, timePre int64
    for {
    select {
    case <-checkTick:
    timeNow = time.Now().Unix()
    if timePre != 0 && timeNow - timePre != 1 {
    srvlog.Notice(fmt.Sprintf("NOT Active for %d", timeNow - timePre))
    }
    timePre = timeNow
    }
    }
    }

    func memLoad(){
    fstab := make( map[uint32]*fsnode )
    for i:=0; i<20000000; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    func main() {
    srvlog ,_ = rsyslog.New( rsyslog.LOG_ERR , "GolangGC" )

    go checkActive()
    memLoad()
    fmt.Scanln()

    }


    I run this program for more than 3 hours, during that time, every ten
    minute it will occur a "heart beat lose"

    This is a very simple data structure: only a big map in memory, so I
    think maybe it's a GC Problem


    So guys, have you ever met this kind of problem? I don't know much
    about GC, I used to be a C programmer on embedded systems, golang is the
    first and the only GC language I even used.

    I'll be very thankful if you can tell me why or how to deal with it




    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Monnand at Apr 24, 2013 at 5:54 pm

    On 04/24/2013 01:22 AM, feeling4t wrote:
    I just change it like this:

    func memLoad() {
    const LOAD = 20000000
    fstab := make(map[uint32]*fsnode, LOAD)
    for i := 0; i < LOAD; i++ {
    nd := new(fsnode)
    fstab[uint32(i)] = nd
    }
    }

    and heart beats lose doesn't happen any more

    I'm using Go 1, does the GC or scheduler improved in later version?
    Is that go 1.0.3 or go 1.1 beta?

    Yes, it improved a lot as far as I know.

    You can run `go version` to check the version.

    Regards,
    -Monnand

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedApr 23, '13 at 5:10p
activeApr 24, '13 at 5:54p
posts20
users6
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase