FAQ
https://github.com/dmitris/prewrite - a simple script to rewrite import
statements and package import comments based on a given prefix. Yet
another vendoring tool :))

I noticed that all Go path rewriting I needed was done by either adding or
removing a prefix - for example, add "go.corp.yahoo.com/x/" to all the
import paths of packages not in standard library as well as package import
comments (as in https://golang.org/s/go14customimport). Occasionally I
wanted to do a reverse operation - to remove the "go.corp.yahoo.com/x/"
prefix from all the import statements and package comments (for example, to
diff with the upstream). I looked at a few existing tools - but found that
often they do "too much" or make assumptions that may not fit all the use
cases (such as a specific output directory or source tree structure), and
some do not [yet] rewrite the package import comments. What I missed was a
glorified sed or Perl script (but in Go!) that would have structure
awareness - that is, would rewrite strings only in the import blocks and
the package import comments, not any other matches in comments etc. Of
course, it was also as good excuse as any to play with the nice Go AST
library :)

The output of the processing is gofmt'ed (to have all the files in the
canonical form) and the reversal of the rewriting with the -r command-line
switch should produce the same code as the original.

Dmitry



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Daniel Theophanes at Apr 29, 2015 at 3:34 pm
    Hi Dmitry,

    Looks like a nice fit for your use case! It looks like it doesn't move any
    package, and that a package move would be taken care of with a different
    tool.

    Before someone uses this outside a corporate environment I would want it to
    record what rewrites it did. You're familiar with my kardianos/vendor tool;
    if I added a tool specific field in the config called "VendorPath" to the
    root of the file, and a flag to not copy directories it could cover your
    use case. What do you think? I ask because there are 101 dep tools, maybe
    we can work together rather say not invented here.

    Daniel

    On Wednesday, April 29, 2015 at 7:17:26 AM UTC-7, Dmitry Savintsev wrote:

    https://github.com/dmitris/prewrite - a simple script to rewrite import
    statements and package import comments based on a given prefix. Yet
    another vendoring tool :))

    I noticed that all Go path rewriting I needed was done by either adding or
    removing a prefix - for example, add "go.corp.yahoo.com/x/" to all the
    import paths of packages not in standard library as well as package import
    comments (as in https://golang.org/s/go14customimport). Occasionally I
    wanted to do a reverse operation - to remove the "go.corp.yahoo.com/x/"
    prefix from all the import statements and package comments (for example, to
    diff with the upstream). I looked at a few existing tools - but found that
    often they do "too much" or make assumptions that may not fit all the use
    cases (such as a specific output directory or source tree structure), and
    some do not [yet] rewrite the package import comments. What I missed was a
    glorified sed or Perl script (but in Go!) that would have structure
    awareness - that is, would rewrite strings only in the import blocks and
    the package import comments, not any other matches in comments etc. Of
    course, it was also as good excuse as any to play with the nice Go AST
    library :)

    The output of the processing is gofmt'ed (to have all the files in the
    canonical form) and the reversal of the rewriting with the -r command-line
    switch should produce the same code as the original.

    Dmitry


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Savintsev at Apr 29, 2015 at 4:26 pm
    Hi Daniel,

    You are right - the assumption is that the moving (or copying) the source
    tree would be done by a different tool or command for more flexibility.
    (https://godoc.org/golang.org/x/tools/cmd/gomvpkg moves the (sub)tree - but
    sometimes I want to copy instead of moving, or just need to rewrite the
    import and check them into the internal source control, etc.)

    I have vendor installed and looked at it. I think there are some
    differences in purpose and scope - I think vendor and several other tools
    are trying to do much more than source rewriting (which is not at all a
    value judgement). Some comments:
    Copying third-party vendor packages into a local project structure
    requires a meta-data file describing the vendor's packages.
    It depends - I think an approach "convention over configuration" is also
    possible. I have not used any meta-data file for vendoring so far. There
    can be different levels of vendoring - corporate, team/group, per-project
    etc. For the corporate level, there would be one copy of third-party
    packages per the corporate repository - Brad described that in his post
    that started the "epic" golang-dev "Dependencies and vendoring" thread
    https://groups.google.com/d/msg/golang-dev/nMWoEAG55v8/iJGgur7W_SEJ. It
    can also happen that a team or group decides that they need their own
    vendoring world - or a specific project would organize and store its
    dependencies. What I wanted to provide with prewrite is a lightweight
    "lowest common denominator" tool - something to reliably add (or remove) a
    specific prefix to all the imports and the "vanity" package import comments
    - and let other tools and layers to organize or post-process the results
    (for example, by copying or moving source trees, checking the modified code
    into the source control, running tests / firing up the CD pipeline, etc.)
    The vendor packages should be placed in an "internal" directory.
    I understand the value of a convention but can think of the cases where it
    can be too restrictive. For example, I may want to or need to refer to the
    vendored packages as git.external.company.com/github.com/xyz/foobar . With
    the prewrite, I can just run: prewrite -p git.external.company.com -v
    <target_root> to add that prefix to all the imports.

    Dmitry
    On Wednesday, April 29, 2015 at 5:33:54 PM UTC+2, Daniel Theophanes wrote:

    Hi Dmitry,

    Looks like a nice fit for your use case! It looks like it doesn't move any
    package, and that a package move would be taken care of with a different
    tool.

    Before someone uses this outside a corporate environment I would want it
    to record what rewrites it did. You're familiar with my kardianos/vendor
    tool; if I added a tool specific field in the config called "VendorPath" to
    the root of the file, and a flag to not copy directories it could cover
    your use case. What do you think? I ask because there are 101 dep tools,
    maybe we can work together rather say not invented here.

    Daniel

    On Wednesday, April 29, 2015 at 7:17:26 AM UTC-7, Dmitry Savintsev wrote:

    https://github.com/dmitris/prewrite - a simple script to rewrite import
    statements and package import comments based on a given prefix. Yet
    another vendoring tool :))

    I noticed that all Go path rewriting I needed was done by either adding
    or removing a prefix - for example, add "go.corp.yahoo.com/x/" to all
    the import paths of packages not in standard library as well as package
    import comments (as in https://golang.org/s/go14customimport).
    Occasionally I wanted to do a reverse operation - to remove the "
    go.corp.yahoo.com/x/" prefix from all the import statements and package
    comments (for example, to diff with the upstream). I looked at a few
    existing tools - but found that often they do "too much" or make
    assumptions that may not fit all the use cases (such as a specific output
    directory or source tree structure), and some do not [yet] rewrite the
    package import comments. What I missed was a glorified sed or Perl script
    (but in Go!) that would have structure awareness - that is, would rewrite
    strings only in the import blocks and the package import comments, not any
    other matches in comments etc. Of course, it was also as good excuse as
    any to play with the nice Go AST library :)

    The output of the processing is gofmt'ed (to have all the files in the
    canonical form) and the reversal of the rewriting with the -r command-line
    switch should produce the same code as the original.

    Dmitry


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roger peppe at Apr 29, 2015 at 6:17 pm
    Have you looked at github.com/rogpeppe/govers ?
    On 29 Apr 2015 15:17, "Dmitry Savintsev" wrote:

    https://github.com/dmitris/prewrite - a simple script to rewrite import
    statements and package import comments based on a given prefix. Yet
    another vendoring tool :))

    I noticed that all Go path rewriting I needed was done by either adding or
    removing a prefix - for example, add "go.corp.yahoo.com/x/" to all the
    import paths of packages not in standard library as well as package import
    comments (as in https://golang.org/s/go14customimport). Occasionally I
    wanted to do a reverse operation - to remove the "go.corp.yahoo.com/x/"
    prefix from all the import statements and package comments (for example, to
    diff with the upstream). I looked at a few existing tools - but found that
    often they do "too much" or make assumptions that may not fit all the use
    cases (such as a specific output directory or source tree structure), and
    some do not [yet] rewrite the package import comments. What I missed was a
    glorified sed or Perl script (but in Go!) that would have structure
    awareness - that is, would rewrite strings only in the import blocks and
    the package import comments, not any other matches in comments etc. Of
    course, it was also as good excuse as any to play with the nice Go AST
    library :)

    The output of the processing is gofmt'ed (to have all the files in the
    canonical form) and the reversal of the rewriting with the -r command-line
    switch should produce the same code as the original.

    Dmitry



    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Savintsev at Apr 30, 2015 at 7:18 am
    Have you looked at github.com/rogpeppe/govers ?
    Of course - that's what I used for import rewriting and also recommended to
    my colleagues before prewrite. But it does not rewrite the package import
    comments so I had to use an editor global replace or other scripts to
    modify lines
    like https://github.com/golang/net/blob/master/context/context.go#L37:
    package context // import "golang.org/x/net/context"

    Also with govers I cannot specify the target directory (has to be the
    current one) and have to do more typing -
    "govers -d -m golang.org/x go.corp.yahoo.com/x/golang.org/x" vs. "prewrite
    -p go.corp.yahoo.com/x". I also had to repeat the govers command for every
    FQDN: golang.org, github.com, gopkg.in, h12.me etc. That was probably the
    main reason for prewrite - I wanted a tool that would add a specified
    prefix to ALL third-party imports without having to specify each one
    individually, so that it would cover even the vanity domains that don't
    exist today but will pop-up tomorrow. I think prewrite will help to avoid
    the need to modify the vendoring script for such changes.

    Also the "BUG" comment
    in https://github.com/rogpeppe/govers/blob/master/govers.go#L134 was
    somewhat worrisome. Let's suppose that I have functional tests (or some
    other special cases) behind a "// +build functional" constraint. If govers
    is used for rewriting, I believe the imports in those files would not be
    rewritten and when you run functional tests in the vendored repository,
    they would fail because of the original ones. Because of such edge cases,
    I wanted more of a "dumb" text-based search-and-replace (with a thin layer
    of intelligence to limit replacements to imports and package import
    comments) rather than a full-blown Go-based refactoring engine such as the
    go/build package.

    I believe the primary purpose of govers was to allow bumping up of the
    pkg.in versions - I had in mind strictly prefix-oriented reversible
    rewriting. I will continue to use govers for suffix modifications when
    needed.

    By the way, prewrite is based on a small AST manipulation library
    http://godoc.org/github.com/dmitris/prewrite/astmod which can also be used
    on its own. The main rewriting functionality is abstracted into Rewrite,
    RewriteImports and RewriteImportComments functions. Maybe govers could use
    it to add rewriting of package import comments? I would appreciate
    feedback and improvement suggestions about the API or the implementation.

    Dmitry
    On Wednesday, April 29, 2015 at 8:18:55 PM UTC+2, rog wrote:

    Have you looked at github.com/rogpeppe/govers ?
    On 29 Apr 2015 15:17, "Dmitry Savintsev" <dsav...@gmail.com <javascript:>>
    wrote:
    https://github.com/dmitris/prewrite - a simple script to rewrite import
    statements and package import comments based on a given prefix. Yet
    another vendoring tool :))

    I noticed that all Go path rewriting I needed was done by either adding
    or removing a prefix - for example, add "go.corp.yahoo.com/x/" to all
    the import paths of packages not in standard library as well as package
    import comments (as in https://golang.org/s/go14customimport).
    Occasionally I wanted to do a reverse operation - to remove the "
    go.corp.yahoo.com/x/" prefix from all the import statements and package
    comments (for example, to diff with the upstream). I looked at a few
    existing tools - but found that often they do "too much" or make
    assumptions that may not fit all the use cases (such as a specific output
    directory or source tree structure), and some do not [yet] rewrite the
    package import comments. What I missed was a glorified sed or Perl script
    (but in Go!) that would have structure awareness - that is, would rewrite
    strings only in the import blocks and the package import comments, not any
    other matches in comments etc. Of course, it was also as good excuse as
    any to play with the nice Go AST library :)

    The output of the processing is gofmt'ed (to have all the files in the
    canonical form) and the reversal of the rewriting with the -r command-line
    switch should produce the same code as the original.

    Dmitry



    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedApr 29, '15 at 2:17p
activeApr 30, '15 at 7:18a
posts5
users3
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase