FAQ
Hello,

How can I check if a file is binary or text?

There was some easy way but I forgot it..


Thanks in adv.

Search Discussions

  • Bromden at Aug 13, 2003 at 10:50 am

    How can I check if a file is binary or text?
    import os
    f = os.popen('file -bi test.py', 'r')
    f.read().startswith('text')
    1

    (btw, f.read() returns 'text/x-java; charset=us-ascii\n')

    --
    bromden[at]gazeta.pl
  • Bromden at Aug 13, 2003 at 11:03 am

    f = os.popen('file -bi test.py', 'r')
    f.read().startswith('text')
    sorry, it's not general, since "file -i" returns
    "application/x-shellscript" for shell scripts,
    it's better to go like that:
    import os
    f = os.popen('file test.py', 'r')
    f.read().find('text') != -1
    --
    bromden[at]gazeta.pl
  • Sami Viitanen at Aug 13, 2003 at 11:17 am
    Works well in Unix but I'm making a script that works on both
    Unix and Windows.

    Win doesn't have that 'file -bi' command.

    "bromden" <bromden at gazeta.pl.no.spam> wrote in message
    news:bhd559$ku9$1 at absinth.dialog.net.pl...
    How can I check if a file is binary or text?
    import os
    f = os.popen('file -bi test.py', 'r')
    f.read().startswith('text')
    1

    (btw, f.read() returns 'text/x-java; charset=us-ascii\n')

    --
    bromden[at]gazeta.pl
  • Michael Peuser at Aug 13, 2003 at 12:23 pm
    Hi,
    yes there is more than just Unix in the world ;-)
    Windows directories have no means to specify their contents type in any way.
    The approved method is using three-letter extensions, though this rule is
    not strictly followed (lot of files without extension nowadays!)

    When I had a similar problem I read 1000 characters, counted the amount of
    <32 and >255 characters and classified it "binary when this qota exceeded
    20%. I have no idea whether it will work good with chinese unicode files or
    some funny depositories or project files that store uncompressed texts....

    KIndly
    Michael P

    "Sami Viitanen" <none at none.net> schrieb im Newsbeitrag
    news:v7p_a.1558$k4.32814 at news2.nokia.com...
    Works well in Unix but I'm making a script that works on both
    Unix and Windows.

    Win doesn't have that 'file -bi' command.

    "bromden" <bromden at gazeta.pl.no.spam> wrote in message
    news:bhd559$ku9$1 at absinth.dialog.net.pl...
    How can I check if a file is binary or text?
    import os
    f = os.popen('file -bi test.py', 'r')
    f.read().startswith('text')
    1

    (btw, f.read() returns 'text/x-java; charset=us-ascii\n')

    --
    bromden[at]gazeta.pl
  • Karl Scalet at Aug 13, 2003 at 12:39 pm

    Michael Peuser schrieb:
    Hi,
    yes there is more than just Unix in the world ;-)
    Windows directories have no means to specify their contents type in any way.
    That's even more true with linux/unix, as there is no need to do
    any stuff like line-terminator conversion.
    The approved method is using three-letter extensions, though this rule is
    not strictly followed (lot of files without extension nowadays!)

    When I had a similar problem I read 1000 characters, counted the amount of
    <32 and >255 characters and classified it "binary when this qota exceeded
    20%. I have no idea whether it will work good with chinese unicode files or
    some funny depositories or project files that store uncompressed texts....
    based on the idea from Mr. "bromden", why not use mimetypes.MimeTypes()
    and guess_type('file://...') and analye the returned string.
    This should work on windows / linux / unix / whatever.


    Karl

    KIndly
    Michael P

    "Sami Viitanen" <none at none.net> schrieb im Newsbeitrag
    news:v7p_a.1558$k4.32814 at news2.nokia.com...
    Works well in Unix but I'm making a script that works on both
    Unix and Windows.

    Win doesn't have that 'file -bi' command.

    "bromden" <bromden at gazeta.pl.no.spam> wrote in message
    news:bhd559$ku9$1 at absinth.dialog.net.pl...
    How can I check if a file is binary or text?
    import os
    f = os.popen('file -bi test.py', 'r')
    f.read().startswith('text')
    1

    (btw, f.read() returns 'text/x-java; charset=us-ascii\n')

    --
    bromden[at]gazeta.pl
  • John Machin at Aug 13, 2003 at 10:32 pm
    "Michael Peuser" <mpeuser at web.de> wrote in message news:<bhdaks$f92$07$1 at news.t-online.com>...
    When I had a similar problem I read 1000 characters, counted the amount of
    <32 and >255 characters and classified it "binary when this qota exceeded
    How many characters > 255 did you get? Did you mean 127? If so, what
    about accented characters ... like umlauts?

    On a slightly more serious note, CR, LF, HT and FF would have to be
    considered "text" but their ordinal values are < 32.

    What was the problem that you thought you were solving?
  • Peter Hansen at Aug 13, 2003 at 2:39 pm

    Sami Viitanen wrote:
    How can I check if a file is binary or text?

    There was some easy way but I forgot it..
    First you need to define what you mean by binary and text.
    Is a file "text" simply because it contains only the
    printable (in ASCII) bytes between 31 and 127, plus
    CR and/or LF, or do you have a more complex definition
    in mind.

    Better yet, what do you need the information for? Maybe
    the answer to that will show us the proper path to take.
  • Trent Mick at Aug 13, 2003 at 5:56 pm
    [Sami Viitanen wrote]
    Hello,

    How can I check if a file is binary or text?

    There was some easy way but I forgot it..
    Generally I define a text file as "it has no null bytes". I think this
    is a pretty safe definition (I would be interested to hear practical
    experience to the contrary). Assuming that, then:

    def is_binary(filename):
    """Return true iff the given filename is binary.

    Raises an EnvironmentError if the file does not exist or cannot be
    accessed.
    """
    fin = open(filename, 'rb')
    try:
    CHUNKSIZE = 1024
    while 1:
    chunk = fin.read(CHUNKSIZE)
    if '\0' in chunk: # found null byte
    return 1
    if len(chunk) < CHUNKSIZE:
    break # done
    finally:
    fin.close()

    return 0

    Cheers,
    Trent


    --
    Trent Mick
    TrentM at ActiveState.com
  • Graham Fawcett at Aug 13, 2003 at 6:28 pm

    Trent Mick wrote:
    [Sami Viitanen wrote]

    Hello,

    How can I check if a file is binary or text?

    There was some easy way but I forgot it..
    Generally I define a text file as "it has no null bytes". I think this
    is a pretty safe definition (I would be interested to hear practical
    experience to the contrary).
    Dangerous assumption. Even if many or most binary files contain NULs, it
    doesn't mean that they all do.

    It is trivial to create a non-text file that has no NULs.

    f = open('no_zeroes.bin', 'rb')
    for x in range(1, 256):
    f.write(chr(x))
    f.close()

    Sami, I would suggest that you need to stop thinking in terms of tools,
    and instead think in terms of the problem you're trying to solve. Why do
    you need to (or think you need to) determine whether a file is "binary"
    or "text"? Why would your application fail if it received a
    (binary/text) file when it expected a (text/binary) one?

    My guess is that the trait you are trying to identify will prove not to
    be "binary or text", but something more application-specific.

    -- Graham

    P.S. Sami, it's very bad form to "make up" an e-mail address, such as
    <none at none.net>. I'm sure the owners of the none.net domain would agree.
    Can't you provide a real address?
  • John Machin at Aug 13, 2003 at 11:04 pm
    Graham Fawcett <fawcett at teksavvy.com> wrote in message news:<mailman.1060799361.14244.python-list at python.org>...
    It is trivial to create a non-text file that has no NULs.

    f = open('no_zeroes.bin', 'rb')
    for x in range(1, 256):
    f.write(chr(x))
    f.close()
    I tried this but it didn't work. It said:

    IOError: [Errno 2] No such file or directory: 'no_zeroes.bin'.

    So I thought I had to be persistent but after doing it a few more times it said:

    SerialIdiotError: What I tell you three times is true.
    NotLispingError: You need 'wb' as in 'wascally wabbit'

    This is very strange behaviour -- does my computer have worms?
  • Graham Fawcett at Aug 14, 2003 at 1:03 am

    John Machin wrote:
    Graham Fawcett <fawcett at teksavvy.com> wrote in message news:<mailman.1060799361.14244.python-list at python.org>...

    It is trivial to create a non-text file that has no NULs.

    f = open('no_zeroes.bin', 'rb')
    for x in range(1, 256):
    f.write(chr(x))
    f.close()
    I tried this but it didn't work. It said:

    IOError: [Errno 2] No such file or directory: 'no_zeroes.bin'.

    So I thought I had to be persistent but after doing it a few more times it said:

    SerialIdiotError: What I tell you three times is true.
    NotLispingError: You need 'wb' as in 'wascally wabbit'

    This is very strange behaviour -- does my computer have worms?
    No, but my brain does. Glad you caught my typo.

    However, it looks like your computer definitely has an AttitudeError!

    -- Graham
  • JanC at Aug 16, 2003 at 2:27 am

    Graham Fawcett <fawcett at teksavvy.com> schreef:

    P.S. Sami, it's very bad form to "make up" an e-mail address, such as
    <none at none.net>. I'm sure the owners of the none.net domain would agree.
    Very true.
    Can't you provide a real address?
    Some non-real addresses are allowed/harmless too:
    - everything ending with the .invalid TLD
    e.g.: none at none.invalid
    - me at privacy.net (the owner of the domain gave his permission)

    --
    JanC

    "Be strict when sending and tolerant when receiving."
    RFC 1958 - Architectural Principles of the Internet - section 3.9
  • Grant Edwards at Aug 13, 2003 at 6:10 pm

    In article <AFm_a.9725$g4.189983 at news1.nokia.com>, Sami Viitanen wrote:

    How can I check if a file is binary or text?
    In order to provide an answer, you'll have to define "binary"
    and "text".
    There was some easy way but I forgot it..
    To _me_ a file isn't "binary" or "text". Those are two modes
    you can use to read a file. The file itself is neutral on the
    matter. At least under Windows and Unix. VMS and FILES-11
    contained a _lot_ more meta-data and actually did have several
    different fundamental file types (fixed length records,
    variable length records, byte-stream, etc.).

    --
    Grant Edwards grante Yow! Will it improve my
    at CASH FLOW?
    visi.com
  • Peter Hansen at Aug 13, 2003 at 6:24 pm

    Trent Mick wrote:
    [Sami Viitanen wrote]
    Hello,

    How can I check if a file is binary or text?

    There was some easy way but I forgot it..
    Generally I define a text file as "it has no null bytes". I think this
    is a pretty safe definition (I would be interested to hear practical
    experience to the contrary).
    "Contains only printable characters" is probably a more useful definition
    of text in many cases. I can't say off the top of my head exactly when
    either definition might be a problem.... wait, how about this one: in
    CVS, if you don't have a file that is effectively line-oriented, human
    readable information, you probably don't want to let it be treated as
    "text" and stored as diffs. In that situation, "contains primarily
    printable characters organized in lines" is probably a more thorough,
    though less deterministic, definition.

    -Peter
  • Grant Edwards at Aug 13, 2003 at 7:33 pm

    In article <3F3A8275.8B6EE8C4 at engcorp.com>, Peter Hansen wrote:

    "Contains only printable characters" is probably a more useful definition
    of text in many cases.
    The definition of "printable" is dependent on the character
    set, that will have to be specified.

    --
    Grant Edwards grante Yow! World War Three can
    at be averted by adherence
    visi.com to a strictly enforced
    dress code!
  • Peter Hansen at Aug 13, 2003 at 9:53 pm

    Grant Edwards wrote:
    In article <3F3A8275.8B6EE8C4 at engcorp.com>, Peter Hansen wrote:
    "Contains only printable characters" is probably a more useful definition
    of text in many cases.
    The definition of "printable" is dependent on the character
    set, that will have to be specified.
    That's why I said "printable (in ASCII)" in another message, so I
    definitely agree. The problem was rather under-specified. :-)
  • Brian Lenihan at Aug 14, 2003 at 6:32 am
    Peter Hansen <peter at engcorp.com> wrote in message news:<3F3A8275.8B6EE8C4 at engcorp.com>...
    "Contains only printable characters" is probably a more useful definition
    of text in many cases. I can't say off the top of my head exactly when
    either definition might be a problem.... wait, how about this one: in
    CVS, if you don't have a file that is effectively line-oriented, human
    readable information, you probably don't want to let it be treated as
    "text" and stored as diffs. In that situation, "contains primarily
    printable characters organized in lines" is probably a more thorough,
    though less deterministic, definition.
    We check for binary files in our CVS commitprep script like this:

    look for -kb arg
    open the file in binary mode, read 4k fom the file and...

    for i in range(len(buff)):
    a = ord(buff[i])
    if (a < 8) or (a > 13 and a < 32) or (a > 126):
    non_text = non_text + 1

    If 10 percent of the characters are found to be non-text, we reject
    the file if it was not commited with the -kb flag, or print a warning
    if the file appears to be text but is being checked in as a binary.

    We don't bother checking for charsets other than ascii, because
    localized files have to be checked in as binaries or bad things
    (tm) happen.
  • John Machin at Aug 13, 2003 at 10:42 pm
    Trent Mick <trentm at ActiveState.com> wrote in message news:<mailman.1060797503.18604.python-list at python.org>...
    Generally I define a text file as "it has no null bytes". I think this
    is a pretty safe definition (I would be interested to hear practical
    experience to the contrary).
    Data file written by C program which has an off-by-one error and is
    including a trailing '\0' byte ...
  • Peter Hansen at Aug 14, 2003 at 2:10 am

    John Machin wrote:
    Trent Mick <trentm at ActiveState.com> wrote in message news:<mailman.1060797503.18604.python-list at python.org>...
    Generally I define a text file as "it has no null bytes". I think this
    is a pretty safe definition (I would be interested to hear practical
    experience to the contrary).
    Data file written by C program which has an off-by-one error and is
    including a trailing '\0' byte ...
    To be fair, I'd call that a "binary" file in any case, or at least
    a defective text file...
  • Sami Viitanen at Aug 14, 2003 at 8:13 am
    Thanks for the answers.

    To be more specific I'm making a script that should
    identify binary files as binary and text files as text.

    The script is for automating CVS commands and
    with CVS you have to add the -kb flag to
    add (or import) binary files. (because it can't itself
    determine what type the file is). If binary file is not
    added with -kb the results are awful.

    Script example usage:
    -import.py <directory_name>

    Script makes list of all files under that directory
    and then determines each files filetype. After that
    all files are added with Add command and binary
    files get that additional -kb automatically.


    "Sami Viitanen" <none at none.net> wrote in message
    news:AFm_a.9725$g4.189983 at news1.nokia.com...
    Hello,

    How can I check if a file is binary or text?

    There was some easy way but I forgot it..


    Thanks in adv.
  • Sami Viitanen at Aug 14, 2003 at 8:30 am
    I think I've found what I need.

    The module is called magic.py.

    It show if the file is data or text.

    It can be downloaded from http://www.demonseed.net/~jp/code/


    "Sami Viitanen" <none at none.net> wrote in message
    news:gwH_a.1649$k4.34358 at news2.nokia.com...
    Thanks for the answers.

    To be more specific I'm making a script that should
    identify binary files as binary and text files as text.

    The script is for automating CVS commands and
    with CVS you have to add the -kb flag to
    add (or import) binary files. (because it can't itself
    determine what type the file is). If binary file is not
    added with -kb the results are awful.

    Script example usage:
    -import.py <directory_name>

    Script makes list of all files under that directory
    and then determines each files filetype. After that
    all files are added with Add command and binary
    files get that additional -kb automatically.


    "Sami Viitanen" <none at none.net> wrote in message
    news:AFm_a.9725$g4.189983 at news1.nokia.com...
    Hello,

    How can I check if a file is binary or text?

    There was some easy way but I forgot it..


    Thanks in adv.
  • David C. Fox at Aug 14, 2003 at 2:04 pm

    Sami Viitanen wrote:
    Thanks for the answers.

    To be more specific I'm making a script that should
    identify binary files as binary and text files as text.

    The script is for automating CVS commands and
    with CVS you have to add the -kb flag to
    add (or import) binary files. (because it can't itself
    determine what type the file is). If binary file is not
    added with -kb the results are awful.
    You should note that the question of when to use -kb is not simply based
    on the contents of the file, but on whether you want CVS/RCS to try to
    merge conflicting versions.

    For example, I recently added some files containing pickled objects
    (used as test data sets for a regression test) to the CVS repository for
    my project. Although the pickle files are in fact all printable text, a
    CVS/RCS merge of two valid pickle files won't yield a valid pickle file.
    Therefore, I used -kb to ensure that the developer would always be
    forced to choose a version in the event of a version conflict.

    David
  • Peter Hansen at Aug 14, 2003 at 3:15 pm

    "David C. Fox" wrote:
    Sami Viitanen wrote:
    Thanks for the answers.

    To be more specific I'm making a script that should
    identify binary files as binary and text files as text.

    The script is for automating CVS commands and
    with CVS you have to add the -kb flag to
    add (or import) binary files. (because it can't itself
    determine what type the file is). If binary file is not
    added with -kb the results are awful.
    You should note that the question of when to use -kb is not simply based
    on the contents of the file, but on whether you want CVS/RCS to try to
    merge conflicting versions.

    For example, I recently added some files containing pickled objects
    (used as test data sets for a regression test) to the CVS repository for
    my project. Although the pickle files are in fact all printable text, a
    CVS/RCS merge of two valid pickle files won't yield a valid pickle file.
    Therefore, I used -kb to ensure that the developer would always be
    forced to choose a version in the event of a version conflict.
    Exactly. We had the same issue with the project files for the Codewright
    text editor. They are sort of like Windows .INI files, but merging such
    files leads to complete disaster, including inability to run Codewright
    until the files are manually fixed or removed!

    -Peter
  • Grant Edwards at Aug 14, 2003 at 2:28 pm

    In article <gwH_a.1649$k4.34358 at news2.nokia.com>, Sami Viitanen wrote:

    To be more specific I'm making a script that should
    identify binary files as binary and text files as text.
    That's "more specific"? ;)

    --
    Grant Edwards grante Yow! I hope I
    at bought the right
    visi.com relish... zzzzzzzzz...

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedAug 13, '03 at 8:29a
activeAug 16, '03 at 2:27a
posts25
users12
websitepython.org

People

Translate

site design / logo © 2022 Grokbase