FAQ
Hi,

I'm creating a base tar file that I then need to modify in further
operations for a set of configurations. Basically I create the initial tar
in Go and then subsequent modifications involve duplicating the tar and
omitting certain files.

When I create the tar I use the file pntr to keep of track of where the
file and the header live (allowing me to skip them later) and then spit out
a struct called ArchiveInfo which lists the names, the index and the length
in bytes.

My issue is that when I even try to make a 1-1 copy I seem to be creating a
badly formed binary file, as the tar fails to extract. Should I be using a
more low level way to create a complete copy?

My output function is https://gist.github.com/callumj/10573518

Thanks in advance!
Callum

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Tamás Gulácsi at Apr 13, 2014 at 11:02 am
    Maybe anneasier solution is to create the copy with tar, too and add headers, data as needed. More high-level than copying only the needed bytes, but harder to corrupt the tar.
    For low-level things be careful with bufio and other bufferings: they corrupt your positions...

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Gyepi SAM at Apr 13, 2014 at 11:16 am

    On Sun, Apr 13, 2014 at 12:45:00AM -0700, Callum Jones wrote:
    I'm creating a base tar file that I then need to modify in further
    operations for a set of configurations. Basically I create the initial tar
    in Go and then subsequent modifications involve duplicating the tar and
    omitting certain files.

    When I create the tar I use the file pntr to keep of track of where the
    file and the header live (allowing me to skip them later) and then spit out
    a struct called ArchiveInfo which lists the names, the index and the length
    in bytes.

    My issue is that when I even try to make a 1-1 copy I seem to be creating a
    badly formed binary file, as the tar fails to extract. Should I be using a
    more low level way to create a complete copy?

    My output function is https://gist.github.com/callumj/10573518
    All is not clear from your output function so can only answer with questions.

    Does 'Item' represent headers and data or just data?
    Needs to be both.

    Is Item.Length accurate? Does it include padding to round off data chunks.
    IIRC tar records are in 512 chunks.

    Have you confirmed that the base tar file works?
    Does the copy work when you copy one file? Two files?

    BTW, you should check for errors from io.CopyN().

    Off topic, but your code could be more idiomatic.
    I'd be happy to explain further if you're interested.

    -Gyepi

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Callum Jones at Apr 13, 2014 at 12:29 pm
    Hey guys,

    Item represents both, before writing the header I query the underlying file
    pntr for the current position (using Seek(0,1)) and after writing the
    contents (after the header) I query for the current position and use the
    diff as the length.

    Do I need to perform the padding if I'm querying the underlying io writer
    resource for positions?

    Good point on the error checking, will do.

    As for my code style, do you have any links to good resources for great Go
    styles?

    Thanks for your help Tamás and Gyepi.
    Callum

    On Sun, Apr 13, 2014 at 7:16 PM, Gyepi SAM wrote:
    On Sun, Apr 13, 2014 at 12:45:00AM -0700, Callum Jones wrote:
    I'm creating a base tar file that I then need to modify in further
    operations for a set of configurations. Basically I create the initial tar
    in Go and then subsequent modifications involve duplicating the tar and
    omitting certain files.

    When I create the tar I use the file pntr to keep of track of where the
    file and the header live (allowing me to skip them later) and then spit out
    a struct called ArchiveInfo which lists the names, the index and the length
    in bytes.

    My issue is that when I even try to make a 1-1 copy I seem to be
    creating a
    badly formed binary file, as the tar fails to extract. Should I be using a
    more low level way to create a complete copy?

    My output function is https://gist.github.com/callumj/10573518
    All is not clear from your output function so can only answer with
    questions.

    Does 'Item' represent headers and data or just data?
    Needs to be both.

    Is Item.Length accurate? Does it include padding to round off data chunks.
    IIRC tar records are in 512 chunks.

    Have you confirmed that the base tar file works?
    Does the copy work when you copy one file? Two files?

    BTW, you should check for errors from io.CopyN().

    Off topic, but your code could be more idiomatic.
    I'd be happy to explain further if you're interested.

    -Gyepi

    --

    Thanks,

    Callum Jones

    callum@callumj.com

    +61 407 058 404

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Tamás Gulácsi at Apr 13, 2014 at 1:24 pm
    Your code should work iff your offsets and sizes correct. Tar is 512 byte blocked format, so you have to pad your last block accordingly. And finish the file with an empty (zero) block.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Gyepi SAM at Apr 13, 2014 at 1:55 pm

    On Sun, Apr 13, 2014 at 08:29:07PM +0800, Callum Jones wrote:
    Item represents both, before writing the header I query the underlying file
    pntr for the current position (using Seek(0,1)) and after writing the
    contents (after the header) I query for the current position and use the
    diff as the length.
    You need to Flush or Close the data writer first to get an accurate position.
    Header should be 512 bytes and data should be some multiple of 512 bytes.
    Do I need to perform the padding if I'm querying the underlying io writer
    resource for positions?
    Not if you're getting the correct values ;)
    Good point on the error checking, will do.

    As for my code style, do you have any links to good resources for great Go
    styles?
    The Language Spec and Effective Go cover most of it. The wiki and other
    resources round things out. The standard libs also make for good reading.

    -Gyepi

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • OpenNota at Apr 13, 2014 at 2:23 pm

    On Sunday, April 13, 2014 7:29:07 PM UTC+7, Callum Jones wrote:

    As for my code style, do you have any links to good resources for great Go
    styles?
    I follow this guide:
    http://code.google.com/p/go-wiki/wiki/CodeReviewComments

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Gyepi SAM at Apr 13, 2014 at 3:06 pm

    On Sun, Apr 13, 2014 at 07:23:54AM -0700, OpenNota wrote:
    On Sunday, April 13, 2014 7:29:07 PM UTC+7, Callum Jones wrote:
    As for my code style, do you have any links to good resources for great Go
    styles?
    I follow this guide:
    http://code.google.com/p/go-wiki/wiki/CodeReviewComments
    This is useful, but I was thinking of more fundamental stuff such as:

    1. Removing stutter and using simpler constructs.

       mapped := make(map[string]bool)
       dirLen := len(basedir)
       for _, item := range contents {
         mapped[item[dirLen:]] = true
       }

    2. Replacing the proliferation of error variables with a single one.

    3. Removing extra parens from condition expressions. Their presence tells me
    that an introduction to gofmt may be in order.

    Simple and idiomatic code is easier to write, understand, and debug.

    -Gyepi

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Callum Jones at Apr 13, 2014 at 5:40 pm
    Looks like I can solve it by initialising a tar.Writer, doing nothing with
    it and then closing it but I'm curious what else I need to do?
    On Sunday, April 13, 2014 3:45:00 PM UTC+8, Callum Jones wrote:

    Hi,

    I'm creating a base tar file that I then need to modify in further
    operations for a set of configurations. Basically I create the initial tar
    in Go and then subsequent modifications involve duplicating the tar and
    omitting certain files.

    When I create the tar I use the file pntr to keep of track of where the
    file and the header live (allowing me to skip them later) and then spit out
    a struct called ArchiveInfo which lists the names, the index and the length
    in bytes.

    My issue is that when I even try to make a 1-1 copy I seem to be creating
    a badly formed binary file, as the tar fails to extract. Should I be using
    a more low level way to create a complete copy?

    My output function is https://gist.github.com/callumj/10573518

    Thanks in advance!
    Callum
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Gyepi SAM at Apr 13, 2014 at 6:27 pm

    On Sun, Apr 13, 2014 at 12:52:08AM -0700, Callum Jones wrote:
    Looks like I can solve it by initialising a tar.Writer, doing nothing with
    it and then closing it but I'm curious what else I need to do?
    Without knowing what you *are* doing, it's hard to know what you *should* be
    doing.

    In any case, that doesn't sound right, even if it happens to "work."

    This sort of thing intrigues me as a starting point to finding out
    why and scares me as stopping point because it's no answer at all.

    -Gyepi

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Callum Jones at Apr 14, 2014 at 12:01 pm
    Hey guys,

    So I'm not extremely confident in screwing around with the tar archive, so
    for this first pass I'm going to just copy the whole reader stream as a new
    writer stream with io.Copy and then append new items onto it as needed. If
    the item exists I'll will just use the existing tar implementation in Go to
    build a new tar archive.

    I think the source of my tar failures were the padding, especially when
    dropping say the first item in the archive and leaving the padding
    remaining and causing parser errors when read back. As the for the hacky
    fix with opening a tar writer, doing nothing and closing out I think it was
    solving the issue where I wasn't copying the trailer across.

    Thanks for the help guys, especially on the go styling guide.

    On Mon, Apr 14, 2014 at 2:27 AM, Gyepi SAM wrote:
    On Sun, Apr 13, 2014 at 12:52:08AM -0700, Callum Jones wrote:
    Looks like I can solve it by initialising a tar.Writer, doing nothing with
    it and then closing it but I'm curious what else I need to do?
    Without knowing what you *are* doing, it's hard to know what you *should*
    be
    doing.

    In any case, that doesn't sound right, even if it happens to "work."

    This sort of thing intrigues me as a starting point to finding out
    why and scares me as stopping point because it's no answer at all.

    -Gyepi


    --

    Thanks,

    Callum Jones

    callum@callumj.com

    +61 407 058 404

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedApr 13, '14 at 7:45a
activeApr 14, '14 at 12:01p
posts11
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase