FAQ
Here is a one-liner I just wrote to delete duplicate lines in a file.

perl -ni.bak -e 'unless ($last eq $_){print $_};$last=$_;' theFile

Going with the TMTOWTDI credo, I was just curious if anyone knew of a better
way.

Thanks,
David

Search Discussions

  • Matt Lyon at Jul 30, 2001 at 5:34 pm
    here's one for the group... is there a module ala getopt::long that will do
    the arg-getting as well as command-line completion with a predefined
    list/hash/etc... of keywords?

    -mL
    -- Life would be so much easier if its source code was well-commented.
  • Paul Johnson at Jul 30, 2001 at 6:15 pm

    On Mon, Jul 30, 2001 at 10:36:50AM -0700, Matt Lyon wrote:
    here's one for the group... is there a module ala getopt::long that will do
    the arg-getting as well as command-line completion with a predefined
    list/hash/etc... of keywords?
    Command line completion is done by a shell rather than an application.
    A good one will do programmable completion. I suggest zsh.

    --
    Paul Johnson - paul@pjcj.net
    http://www.pjcj.net
  • Paul at Jul 30, 2001 at 5:44 pm

    --- David Blevins wrote:
    Here is a one-liner I just wrote to delete duplicate lines in a file.

    perl -ni.bak -e 'unless ($last eq $_){print $_};$last=$_;' theFile
    That requires that it be sorted, doesn't it?
    Going with the TMTOWTDI credo, I was just curious if anyone knew of a
    better way.
    For *S*M*A*L*L* files, I use a hash.

    perl -ni.bak -e 'unless ($hit{$_}){$hit{$_}++,print $_}' theFile

    That preserves the order of nonduplicated lines, but eats a lot of
    memory if the file is very big at all.

    If you don't want to preserve the order, try "sort -u theFile"
    ( though that isn't Perl =o)



    __________________________________________________
    Do You Yahoo!?
    Make international calls for as low as $.04/minute with Yahoo! Messenger
    http://phonecard.yahoo.com/
  • Carl Rogers at Jul 30, 2001 at 5:54 pm
    One word of caution.... it looks to me like this will catch duplicates
    lines, just as long as the duplicate lines follow each other. You may want
    to do some kind of a sort process prior to running this line of code.
    Only reason I bring this up... I've been bitten by this same problem in the
    past:)
    I've used rpsort (I think it's freeware/shareware) to sort an entire line
    (not exactly efficient for larger files, but it seems to do the trick),
    I also got some help from someone on this list who provided the following
    (this didn't require a sort process prior to running it)

    while (<INPUTFILE>){
    if (not $seen{$_}) {
    $seen{$_} = 1;
    print OUTFILE;
    }
    else {
    }
    }

    I wish I could tell you why/how it works (I'm *still* working my way up to
    newbie status), but it does. (Magic??).. It'll take longer for big files,
    but again, it does the trick.

    HTH
    Carl
    At 12:30 PM 7/30/2001 -0500, David Blevins wrote:
    Here is a one-liner I just wrote to delete duplicate lines in a file.

    perl -ni.bak -e 'unless ($last eq $_){print $_};$last=$_;' theFile

    Going with the TMTOWTDI credo, I was just curious if anyone knew of a better
    way.

    Thanks,
    David


    --
    To unsubscribe, e-mail: beginners-unsubscribe@perl.org
    For additional commands, e-mail: beginners-help@perl.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupbeginners @
categoriesperl
postedJul 30, '01 at 5:30p
activeJul 30, '01 at 6:15p
posts5
users5
websiteperl.org

People

Translate

site design / logo © 2022 Grokbase