FAQ
One of my small programmes to retrieve protein sequence data from NCBI
recently failed for user with:

attempting to retrieve 500 records starting from 787000 with 5 retries.
retrieved records with 0 retries... writing out.
attempting to retrieve 500 records starting from 787500 with 5 retries.
retrieved records with 0 retries... writing out.
attempting to retrieve 500 records starting from 788000 with 5 retries.
failed to buffer on attempt 0... error: unexpected EOF ... retrying.
failed to buffer on attempt 1... error: unexpected EOF ... retrying.
failed to buffer on attempt 2... error: unexpected EOF ... retrying.
failed to buffer on attempt 3... error: unexpected EOF ... retrying.
failed to buffer on attempt 4... error: unexpected EOF ... retrying.

The program calls a set of convenience wrappers around http.Get in
github.com/biogo/ncbi/entrez and then does an io.Copy of the returned
http.Response.Body to a buffer.

https://github.com/biogo/examples/blob/master/fetch/fetch.go#L88

The concrete http.Response.Body returned is *http.bodyEOFSignal.

Under what conditions would the behaviour above be expected? What does
an io.ErrUnexpectedError mean in this context?

The reason this is a concern is that a perl run-alike does not fail. My
suspicion is that the perl code is just ignoring the error and dropping
these failures. Does this sound like a reasonable explanation?

Repeating the failure is time consuming given that the failure only
occurred after 788K records. This takes over night. I am trying a repeat
though.

thanks
Dan

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Roger peppe at Mar 22, 2016 at 7:29 am
    Are you using HTTPS? I've regularly seen io.ErrUnexpectedEOF from the TLS
    layer in the past.
    On 22 Mar 2016 00:15, "Dan Kortschak" wrote:

    One of my small programmes to retrieve protein sequence data from NCBI
    recently failed for user with:

    attempting to retrieve 500 records starting from 787000 with 5 retries.
    retrieved records with 0 retries... writing out.
    attempting to retrieve 500 records starting from 787500 with 5 retries.
    retrieved records with 0 retries... writing out.
    attempting to retrieve 500 records starting from 788000 with 5 retries.
    failed to buffer on attempt 0... error: unexpected EOF ... retrying.
    failed to buffer on attempt 1... error: unexpected EOF ... retrying.
    failed to buffer on attempt 2... error: unexpected EOF ... retrying.
    failed to buffer on attempt 3... error: unexpected EOF ... retrying.
    failed to buffer on attempt 4... error: unexpected EOF ... retrying.

    The program calls a set of convenience wrappers around http.Get in
    github.com/biogo/ncbi/entrez and then does an io.Copy of the returned
    http.Response.Body to a buffer.

    https://github.com/biogo/examples/blob/master/fetch/fetch.go#L88

    The concrete http.Response.Body returned is *http.bodyEOFSignal.

    Under what conditions would the behaviour above be expected? What does
    an io.ErrUnexpectedError mean in this context?

    The reason this is a concern is that a perl run-alike does not fail. My
    suspicion is that the perl code is just ignoring the error and dropping
    these failures. Does this sound like a reasonable explanation?

    Repeating the failure is time consuming given that the failure only
    occurred after 788K records. This takes over night. I am trying a repeat
    though.

    thanks
    Dan

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dan Kortschak at Mar 22, 2016 at 7:38 am
    No, it's all http.

    https://godoc.org/github.com/biogo/ncbi/entrez#pkg-constants

    On 22/03/2016, at 5:58 PM, "roger peppe" wrote:


    Are you using HTTPS? I've regularly seen io.ErrUnexpectedEOF from the TLS layer in the past.

    On 22 Mar 2016 00:15, "Dan Kortschak" wrote:
    One of my small programmes to retrieve protein sequence data from NCBI
    recently failed for user with:

    attempting to retrieve 500 records starting from 787000 with 5 retries.
    retrieved records with 0 retries... writing out.
    attempting to retrieve 500 records starting from 787500 with 5 retries.
    retrieved records with 0 retries... writing out.
    attempting to retrieve 500 records starting from 788000 with 5 retries.
    failed to buffer on attempt 0... error: unexpected EOF ... retrying.
    failed to buffer on attempt 1... error: unexpected EOF ... retrying.
    failed to buffer on attempt 2... error: unexpected EOF ... retrying.
    failed to buffer on attempt 3... error: unexpected EOF ... retrying.
    failed to buffer on attempt 4... error: unexpected EOF ... retrying.

    The program calls a set of convenience wrappers around http.Get in
    github.com/biogo/ncbi/entrez<http://github.com/biogo/ncbi/entrez> and then does an io.Copy of the returned
    http.Response.Body to a buffer.

    https://github.com/biogo/examples/blob/master/fetch/fetch.go#L88

    The concrete http.Response.Body returned is *http.bodyEOFSignal.

    Under what conditions would the behaviour above be expected? What does
    an io.ErrUnexpectedError mean in this context?

    The reason this is a concern is that a perl run-alike does not fail. My
    suspicion is that the perl code is just ignoring the error and dropping
    these failures. Does this sound like a reasonable explanation?

    Repeating the failure is time consuming given that the failure only
    occurred after 788K records. This takes over night. I am trying a repeat
    though.

    thanks
    Dan

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Mar 23, 2016 at 4:45 am
    Maybe chunked or deflate encoding?
    On Tue, Mar 22, 2016, 00:38 Dan Kortschak wrote:

    No, it's all http.

    https://godoc.org/github.com/biogo/ncbi/entrez#pkg-constants

    On 22/03/2016, at 5:58 PM, "roger peppe" wrote:

    Are you using HTTPS? I've regularly seen io.ErrUnexpectedEOF from the TLS
    layer in the past.
    On 22 Mar 2016 00:15, "Dan Kortschak" wrote:

    One of my small programmes to retrieve protein sequence data from NCBI
    recently failed for user with:

    attempting to retrieve 500 records starting from 787000 with 5 retries.
    retrieved records with 0 retries... writing out.
    attempting to retrieve 500 records starting from 787500 with 5 retries.
    retrieved records with 0 retries... writing out.
    attempting to retrieve 500 records starting from 788000 with 5 retries.
    failed to buffer on attempt 0... error: unexpected EOF ... retrying.
    failed to buffer on attempt 1... error: unexpected EOF ... retrying.
    failed to buffer on attempt 2... error: unexpected EOF ... retrying.
    failed to buffer on attempt 3... error: unexpected EOF ... retrying.
    failed to buffer on attempt 4... error: unexpected EOF ... retrying.

    The program calls a set of convenience wrappers around http.Get in
    github.com/biogo/ncbi/entrez and then does an io.Copy of the returned
    http.Response.Body to a buffer.

    https://github.com/biogo/examples/blob/master/fetch/fetch.go#L88

    The concrete http.Response.Body returned is *http.bodyEOFSignal.

    Under what conditions would the behaviour above be expected? What does
    an io.ErrUnexpectedError mean in this context?

    The reason this is a concern is that a perl run-alike does not fail. My
    suspicion is that the perl code is just ignoring the error and dropping
    these failures. Does this sound like a reasonable explanation?

    Repeating the failure is time consuming given that the failure only
    occurred after 788K records. This takes over night. I am trying a repeat
    though.

    thanks
    Dan
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dan Kortschak at Mar 23, 2016 at 5:23 am
    The queries specify text, but maybe they deflate it (chunking seems
    unlikely). I'll have a look. Thanks.
    On Wed, 2016-03-23 at 04:45 +0000, Matt Harden wrote:
    Maybe chunked or deflate encoding?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dan Kortschak at Mar 23, 2016 at 11:52 pm
    Yes. It is chunked. I'll capture a response that results in a failure
    and post it here. This may take a while.

    thanks
    On Wed, 2016-03-23 at 15:53 +1030, Dan Kortschak wrote:
    The queries specify text, but maybe they deflate it (chunking seems
    unlikely). I'll have a look. Thanks.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Mar 24, 2016 at 7:18 am
    The server could be doing chunked encoding wrong and terminating the
    chunked response without the required 0 byte EOF chunk.

    On Thu, Mar 24, 2016 at 10:52 AM, Dan Kortschak wrote:

    Yes. It is chunked. I'll capture a response that results in a failure
    and post it here. This may take a while.

    thanks
    On Wed, 2016-03-23 at 15:53 +1030, Dan Kortschak wrote:
    The queries specify text, but maybe they deflate it (chunking seems
    unlikely). I'll have a look. Thanks.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dan Kortschak at Apr 7, 2016 at 4:46 am

    On Thu, 2016-03-24 at 18:18 +1100, Brad Fitzpatrick wrote:
    The server could be doing chunked encoding wrong and terminating the
    chunked response without the required 0 byte EOF chunk.
    I finally have a captured http.Response that caused the error. The full
    value is viewable at
    https://gist.github.com/kortschak/adcfc50063638f811342499738ac7edb

    (this gist also has the previous successful http.Response).

    For a reprise on the problem, the response body is used as the source in
    an io.Copy call that fails with an io.UnexpectedEOF error.

    It would be nice to be able to get an idea of why this is happening with
    a view to preventing it.

    thanks
    Dan

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nick Craig-Wood at Apr 7, 2016 at 12:16 pm
    I've been tracking down a bug reported by an rclone user which has
    boiled down to exactly this: https://github.com/ncw/rclone/issues/415

    Which turned into this PR: https://github.com/aws/aws-sdk-go/pull/620

    The user was doing transfers over HTTPS but the stream was being cut off
    for some reason.
    On 07/04/16 05:46, Dan Kortschak wrote:
    On Thu, 2016-03-24 at 18:18 +1100, Brad Fitzpatrick wrote:
    The server could be doing chunked encoding wrong and terminating the
    chunked response without the required 0 byte EOF chunk.
    I finally have a captured http.Response that caused the error. The full
    value is viewable at
    https://gist.github.com/kortschak/adcfc50063638f811342499738ac7edb

    (this gist also has the previous successful http.Response).

    For a reprise on the problem, the response body is used as the source in
    an io.Copy call that fails with an io.UnexpectedEOF error.

    It would be nice to be able to get an idea of why this is happening with
    a view to preventing it.

    thanks
    Dan

    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Apr 7, 2016 at 1:31 pm

    On Thu, Apr 7, 2016 at 2:46 PM, Dan Kortschak wrote:
    On Thu, 2016-03-24 at 18:18 +1100, Brad Fitzpatrick wrote:
    The server could be doing chunked encoding wrong and terminating the
    chunked response without the required 0 byte EOF chunk.
    I finally have a captured http.Response that caused the error. The full
    value is viewable at
    https://gist.github.com/kortschak/adcfc50063638f811342499738ac7edb

    Yeah, it looks truncated, so UnexpectedEOF is the right answer.

    Note that it has "Transfer-Encoding: chunked" but none of the responses end
    in the final EOF 0-byte chunk, which would look like "0\r\n\r\n" on the
    wire (or 30 0d 0a 0d 0a).

    (this gist also has the previous successful http.Response).

    For a reprise on the problem, the response body is used as the source in
    an io.Copy call that fails with an io.UnexpectedEOF error.

    It would be nice to be able to get an idea of why this is happening with
    a view to preventing it.

    thanks
    Dan
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dan Kortschak at Apr 8, 2016 at 2:28 am

    On Thu, 2016-04-07 at 23:30 +1000, Brad Fitzpatrick wrote:
    Yeah, it looks truncated, so UnexpectedEOF is the right answer.

    Note that it has "Transfer-Encoding: chunked" but none of the
    responses end
    in the final EOF 0-byte chunk, which would look like "0\r\n\r\n" on
    the
    wire (or 30 0d 0a 0d 0a).
    So the correct response from me would be to file an issue with that
    service?

    Thanks for looking at this.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Apr 8, 2016 at 2:34 am

    On Thu, Apr 7, 2016 at 7:28 PM, Dan Kortschak wrote:
    On Thu, 2016-04-07 at 23:30 +1000, Brad Fitzpatrick wrote:
    Yeah, it looks truncated, so UnexpectedEOF is the right answer.

    Note that it has "Transfer-Encoding: chunked" but none of the
    responses end
    in the final EOF 0-byte chunk, which would look like "0\r\n\r\n" on
    the
    wire (or 30 0d 0a 0d 0a).
    So the correct response from me would be to file an issue with that
    service?
    Yeah. Does it happen 100% of the time on that one particular GET URL?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dan Kortschak at Apr 8, 2016 at 3:32 am

    On Thu, 2016-04-07 at 19:34 -0700, Brad Fitzpatrick wrote:
    Yeah. Does it happen 100% of the time on that one particular GET URL?
    When it fails once it fails 100%, but the same GET URL may work in other
    queries. The GET is part of a series of fetches of results from a
    database query that is 552MB of text in total. So I guess the answer is
    yes, but the context is different for different invocations.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dave Cheney at Apr 8, 2016 at 4:34 am
    Is it documentaries that the HTTP package has to return ErrUnexpectedEOF? The chunked reader probably has enough context to return a more descriptive error. I don't know if that would break the compatibility promise.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMar 22, '16 at 12:15a
activeApr 8, '16 at 4:34a
posts14
users6
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase