FAQ
Could someone have a look through tcp.go<https://github.com/liamzebedee/go-qrp/blob/feature-tcp/tcp.go>?
When running the TCP test, large amounts of memory are consumed which
eventually leads to an application crash. It's somewhere in this code, but
I don't know where.

--

Search Discussions

  • Peter S at Oct 27, 2012 at 4:28 am
    Have you tried memory profiling? It could give some useful hints.
    http://blog.golang.org/2011/06/profiling-go-programs.html

    Just glancing at the source, I noticed that it seems to be creating
    goroutines in a quasi-infinite loop at line 117. This could very easily
    cause memory exhaustion.

    Peter

    On Sat, Oct 27, 2012 at 10:52 AM, liamzebedee wrote:

    Could someone have a look through tcp.go<https://github.com/liamzebedee/go-qrp/blob/feature-tcp/tcp.go>?
    When running the TCP test, large amounts of memory are consumed which
    eventually leads to an application crash. It's somewhere in this code, but
    I don't know where.

    --

    --
  • Liamzebedee at Oct 27, 2012 at 6:19 am
    Regarding the goroutine creation - I don't believe that this is the issue,
    as I intend the *conns* channel to only return a value when a connection
    has been accepted. Is this not the case?
    On Saturday, October 27, 2012 2:28:32 PM UTC+10, speter wrote:

    Have you tried memory profiling? It could give some useful hints.
    http://blog.golang.org/2011/06/profiling-go-programs.html

    Just glancing at the source, I noticed that it seems to be creating
    goroutines in a quasi-infinite loop at line 117. This could very easily
    cause memory exhaustion.

    Peter


    On Sat, Oct 27, 2012 at 10:52 AM, liamzebedee <liamz...@gmail.com<javascript:>
    wrote:
    Could someone have a look through tcp.go<https://github.com/liamzebedee/go-qrp/blob/feature-tcp/tcp.go>?
    When running the TCP test, large amounts of memory are consumed which
    eventually leads to an application crash. It's somewhere in this code, but
    I don't know where.

    --

    --
  • Peter S at Oct 27, 2012 at 5:56 am
    You're right, sorry about that. This is the problem when looking for
    potential issues without any clues -- you have to go through a lot of
    things without knowing what is really relevant, and most of them will turn
    out to be false alarms. You should profile the program and it will tell
    exactly what the increasing memory amounts are used for. (Another thing I
    noticed is that "activeConnections" are never deleted, but without a memory
    profile it is just another shot in the dark.)

    Peter


    On Sat, Oct 27, 2012 at 2:16 PM, liamzebedee wrote:

    Regarding the goroutine creation - I don't believe that this is the issue,
    as I intend the *conns* channel to only return a value when a connection
    has been accepted. Is this not the case?

    On Saturday, October 27, 2012 2:28:32 PM UTC+10, speter wrote:

    Have you tried memory profiling? It could give some useful hints.
    http://blog.golang.org/2011/**06/profiling-go-programs.html<http://blog.golang.org/2011/06/profiling-go-programs.html>

    Just glancing at the source, I noticed that it seems to be creating
    goroutines in a quasi-infinite loop at line 117. This could very easily
    cause memory exhaustion.

    Peter

    On Sat, Oct 27, 2012 at 10:52 AM, liamzebedee wrote:

    Could someone have a look through tcp.go<https://github.com/liamzebedee/go-qrp/blob/feature-tcp/tcp.go>?
    When running the TCP test, large amounts of memory are consumed which
    eventually leads to an application crash. It's somewhere in this code, but
    I don't know where.

    --

    --
    --
  • Peter S at Oct 27, 2012 at 6:32 am
    Speaking of activeConnections, you will probably need to add some kind of
    synchronization as it is read and modified concurrently from multiple
    goroutines.

    Peter
    On Sat, Oct 27, 2012 at 2:49 PM, Peter S wrote:

    You're right, sorry about that. This is the problem when looking for
    potential issues without any clues -- you have to go through a lot of
    things without knowing what is really relevant, and most of them will turn
    out to be false alarms. You should profile the program and it will tell
    exactly what the increasing memory amounts are used for. (Another thing I
    noticed is that "activeConnections" are never deleted, but without a memory
    profile it is just another shot in the dark.)

    Peter



    On Sat, Oct 27, 2012 at 2:16 PM, liamzebedee wrote:

    Regarding the goroutine creation - I don't believe that this is the
    issue, as I intend the *conns* channel to only return a value when a
    connection has been accepted. Is this not the case?

    On Saturday, October 27, 2012 2:28:32 PM UTC+10, speter wrote:

    Have you tried memory profiling? It could give some useful hints.
    http://blog.golang.org/2011/**06/profiling-go-programs.html<http://blog.golang.org/2011/06/profiling-go-programs.html>

    Just glancing at the source, I noticed that it seems to be creating
    goroutines in a quasi-infinite loop at line 117. This could very easily
    cause memory exhaustion.

    Peter

    On Sat, Oct 27, 2012 at 10:52 AM, liamzebedee wrote:

    Could someone have a look through tcp.go<https://github.com/liamzebedee/go-qrp/blob/feature-tcp/tcp.go>?
    When running the TCP test, large amounts of memory are consumed which
    eventually leads to an application crash. It's somewhere in this code, but
    I don't know where.

    --

    --
    --
  • DisposaBoy at Oct 27, 2012 at 10:39 am

    On Saturday, October 27, 2012 2:52:09 AM UTC+1, liamzebedee wrote:
    Could someone have a look through tcp.go<https://github.com/liamzebedee/go-qrp/blob/feature-tcp/tcp.go>?
    When running the TCP test, large amounts of memory are consumed which
    eventually leads to an application crash. It's somewhere in this code, but
    I don't know where.

    In WriteTo you have `tcpConn, err := net.DialTCP(addr.Network(),
    node.connectAddr, addrConnectTo)` I don't think that does what you thin
    it's doing... because you enter that block tcpConn is nil but here you're
    not initializing it, you're creating a new variable and sending that new
    variable off in a goroutine. after the if statement, you're then using the
    `nil` tcpConn
    Other than that, check your error handling etc. especially check those
    infinite loops, especially what happens to them when there is an error. You
    also have to look for infiinite recursion, e.g a calls be, b initializes y
    which iinvolves calling b again. Also run go vet on those files, it's
    possible that due to bad arguments to functions likes fmt.Printf you may
    not be getting correct output

    --
  • Liamzebedee at Oct 28, 2012 at 12:21 am
    Ok so I've been staring at this for ages now, and have taken into
    consideration all the comments made in this thread. After some commenting
    out of all the client stuff, and just leaving the server instantiation code
    I've managed to narrow down my memory surge to a single line -
    server.Stop().

    println("=== Running TCP Test")

    // Server
    server, err := CreateNodeTCP("tcp", "127.0.0.1:50000", "127.0.0.1:50001")
    if err != nil {
    print("SERVER: Can't create server node -", err.Error())
    t.FailNow()
    }
    println("SERVER: Server created")
    add := new(AddService)
    server.Register(add)
    println("SERVER: Add service registered")

    go func() {
    err := server.ListenAndServe()
    if err != nil {
    println("SERVER:", "Error serving -", err.Error())
    t.FailNow()
    }
    }()
    defer server.Stop()
    time.Sleep(2 * time.Second)

    server.Stop causes my Windows machine to consume all available memory
    (2GB). The weird thing is that even though I defer it, it immediately
    takes effect without waiting for time.Sleep.

    --
  • Liamzebedee at Oct 28, 2012 at 12:32 am
    Oh damn.
    For TCPNode, there exists no Stop function, only a Close function.
    For UDPNode however, there is a Stop function.

    Will update as situation progresses.
    On Sunday, October 28, 2012 10:21:08 AM UTC+10, liamzebedee wrote:

    Ok so I've been staring at this for ages now, and have taken into
    consideration all the comments made in this thread. After some commenting
    out of all the client stuff, and just leaving the server instantiation code
    I've managed to narrow down my memory surge to a single line -
    server.Stop().

    println("=== Running TCP Test")

    // Server
    server, err := CreateNodeTCP("tcp", "127.0.0.1:50000", "127.0.0.1:50001")
    if err != nil {
    print("SERVER: Can't create server node -", err.Error())
    t.FailNow()
    }
    println("SERVER: Server created")
    add := new(AddService)
    server.Register(add)
    println("SERVER: Add service registered")

    go func() {
    err := server.ListenAndServe()
    if err != nil {
    println("SERVER:", "Error serving -", err.Error())
    t.FailNow()
    }
    }()
    defer server.Stop()
    time.Sleep(2 * time.Second)

    server.Stop causes my Windows machine to consume all available memory
    (2GB). The weird thing is that even though I defer it, it immediately
    takes effect without waiting for time.Sleep.
    --
  • Jesse McNelis at Oct 28, 2012 at 2:50 am
    On Sat, Oct 27, 2012 at 12:52 PM, liamzebedee wrote
    Could someone have a look through tcp.go? When running the TCP test, large
    amounts of memory are consumed which eventually leads to an application
    crash. It's somewhere in this code, but I don't know where.
    In handleConnection(),

    buffer, err := reader.ReadBytes(0)
    read := len(buffer)
    if err == io.EOF && read == 0 {
    // No packet, no cry
    continue
    }
    Means that if the connection is closed by the remote end this will
    just loop infinitely since the error will always be io.EOF.


    --
    =====================
    http://jessta.id.au

    --
  • Liamzebedee at Oct 28, 2012 at 3:03 am
    Shouldn't the *continue *statement go back to the start of the for loop,
    where the following select statement<https://github.com/liamzebedee/go-qrp/blob/5daa5086cf57f455303cb42dd084b98e0c40783e/tcp.go#L139>can receive the channel signal to stop reading packets?
    On Sunday, October 28, 2012 12:51:04 PM UTC+10, Jesse McNelis wrote:

    On Sat, Oct 27, 2012 at 12:52 PM, liamzebedee <liamz...@gmail.com<javascript:>>
    wrote
    Could someone have a look through tcp.go? When running the TCP test, large
    amounts of memory are consumed which eventually leads to an application
    crash. It's somewhere in this code, but I don't know where.
    In handleConnection(),

    buffer, err := reader.ReadBytes(0)
    read := len(buffer)
    if err == io.EOF && read == 0 {
    // No packet, no cry
    continue
    }
    Means that if the connection is closed by the remote end this will
    just loop infinitely since the error will always be io.EOF.


    --
    =====================
    http://jessta.id.au
    --
  • Liamzebedee at Oct 28, 2012 at 3:08 am
    Also, on an important side note, in my latest commit<https://github.com/liamzebedee/go-qrp/blob/feature-tcp/tcp.go>(in which I rewrote the code for tcp.go) my TCPListener isn'ttiming out<https://github.com/liamzebedee/go-qrp/blob/feature-tcp/tcp.go#L109>.
    Anyone know why?

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 27, '12 at 2:54a
activeOct 28, '12 at 3:08a
posts11
users4
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase