FAQ
I'm having trouble containing the memory usage of an application that
serially parses through ~100GB lines of logs and writes out unique factors
out to github.com/syndtr/goleveldb -- Most of the information in each line
is duplicative so I'm attempting to throw away any variables pointing to
the string, but per the memory profile, I'm seemingly unable to do so as I
can see the problem allocations come from bufio.Scanner.Text(). Those
allocations should be cleaned up (as from what I can tell, I'm not
referencing the string or any slices of the string anymore), but they still
exist in my running program and are scaling 1:1 with the amount of data
parsed.

I'm fairly certain it's a bug in my code, but is there any tool or feature
that I'm missing that would help me identify what pointer I'm still holding
that is keeping this memory from being garbage collected?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Matthew Zimmerman at May 4, 2015 at 5:48 pm
    And of course a few minutes after I sent this I found it...

    As suspected, I was still referring to a slice of that string although it
    was being passed around through about 5 different data structures before
    arriving at it's final destination.

    Sorry for the noise.
    On Mon, May 4, 2015 at 1:43 PM Matthew Zimmerman wrote:

    I'm having trouble containing the memory usage of an application that
    serially parses through ~100GB lines of logs and writes out unique factors
    out to github.com/syndtr/goleveldb -- Most of the information in each
    line is duplicative so I'm attempting to throw away any variables pointing
    to the string, but per the memory profile, I'm seemingly unable to do so as
    I can see the problem allocations come from bufio.Scanner.Text(). Those
    allocations should be cleaned up (as from what I can tell, I'm not
    referencing the string or any slices of the string anymore), but they still
    exist in my running program and are scaling 1:1 with the amount of data
    parsed.

    I'm fairly certain it's a bug in my code, but is there any tool or feature
    that I'm missing that would help me identify what pointer I'm still holding
    that is keeping this memory from being garbage collected?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMay 4, '15 at 5:43p
activeMay 4, '15 at 5:48p
posts2
users1
websitegolang.org

1 user in discussion

Matthew Zimmerman: 2 posts

People

Translate

site design / logo © 2022 Grokbase