FAQ
Hi to all Hackers,

=========================================================================
On Wed, 2 Jul 1997, Thomas G. Lockhart wrote:

I have already committed patches to the source tree which had been
contributed by Tatsuo Ishii <t-ishii@sra.co.jp>.
Tatsuo Ishii's patch (I think it was his path?) repaired the "wrong-reversed"
byte order in pqcomprim.c into the "correct-reversed" byte order.

As a result all Front-ends using "pqlib" will work on all platforms. I
personally would call this as "workaround" - it leads to correct behaviour
inside the postgres world - real good work - but...

But applications using direct socket I/O, reading an "u_short/u_long"
and translating it into a system defined "u_short/u_long" by using
"ntohs/ntohl" will NOT get correct values!
E. g. the PostODBC driver is such an application.

I don't know why, in addition to the system defines htonl/htons and
ntohl/ntohs, the postgres specific "ntopl/ntops" and "ptonl/ptons" ('p'
stands for postgres) translation have been defined. May be there is a
reason which I don't know, but removing ALL the ENDIAN stuff AND using the
systems htonl/htons/ntohl/ntohs (as done in my patch pqcomprim.patch) ALL
PLATFORMS should talk to ALL PLATFORMS (including MSWindows). In other
words: a pqPutShort will be received as u_short (using ntohs) regrardless
of the local endianess.

======================================================================
On Thu, 3 Jul 1997, Thomas G. Lockhart wrote:


We understand that point. Have you actually tested the patches released
a few days ago? They are in

ftp://postgresql.org/pub/6.2/postgresql-v6.1-chngs-p1.tar.gz
I did not compile it, but have read it. It should be the same problem as
mentionend above - I think.
The various versions of AIX have _dominated_ the porting e-mail traffic
the last few weeks. Does anyone have any indications that Tatsuo's
patches do not fix the endian problems for non-AIX systems? He clearly
tested on many platforms.
The reason for the "dominating" traffic was the "incorrect" implementation
of the ENDIAN behaviour, which has been "repaired" by Tatsuo.
But IMHO it's only a "postgres-internal workaround" -
and a workaround DOES WORK!
but it does not solve the general "network to host" problem with
other applications.

========================================================================
On Thu, 3 Jul 1997, Frank Dana wrote:

I assume it's because there's some other function that consumes ENDIAN
but not machine.h -- and the FOO_ENDIAN defines are respresented
differently in postgres.h and AIX's machine.h. <sigh> That's what I
get for rushing into patches -- as soon as I stopped coding and started
thinking, I realized there was a good chance the patches would create
other problems with ENDIAN representation on AIX 3.2.5.

What if we moved the FOO_ENDIAN defines to include/ports/bar.h? Then we
could put the proper logic in aix.h to deal with 3.2.5 vs 4.1 without
mucking up the other ports' include files. It seems to make sense,
actually -- endianness really a "machine-defined" thing, ripe for
placement in the port-specific include files.
To put the FOO_ENDIAN's into the port specific, is IMHO hiding the
original problem in an include file, which is alredy solved in the
operating systems include files. IMHO it's better to use the (commonly
used) hton*/ntoh* functions/macros. Only these function should cover the
endianess.

========================================================================
On Fri, 4 Jul 1997, Tatsuo Ishii wrote:

Among these, Linux 2.0.29 is the only Little endian machine.
I checked all of flowing combinations:

Client Server
----------------------
Linux MkLinux
Linux SunOS
MkLinux Linux
MkLinux SunOS
SunOS Linux
SunOS MkLinux

Other platforms have been tested by our local user groups. I hope
additional reports will come soon.
Yes, they all are all right, because they all use the same libpq with
Tatsuo's patch.

=====================================================================

Now for the last:
I, Gerhard Reithofer vote for using the system defined hton*/ntoh* functions
and removing the *ENDIAN defines. If they are not present on a platform,
the functions should #define'd in the way, as they are defined in all
other OS's.
(BTW: The hton*/ntoh* -- functions are used in the pqcomprim *ENDIAN
#defines anyway, so you still get in trouble when your system does not
define them...)

Christian Czezatke has implemented this feature into a temporary PostODBC
version and waits still for an "official" statement from the developers
how these problem will be covered in future. He votes for the system
conversions too.

We all hope, that we soon have a stable, uniqe version of PostgreSQL. The
PostgreSQL hackers do real great work for many, many users and I'm only
one them which have to say "many, many thanks for your work".

Best regards,
Gerhardr Reithofer

- ----------------------------------------------------------------------
BYTEORDER(3) Linux Programmer's Manual BYTEORDER(3)

NAME
htonl, htons, ntohl, ntohs - convert values between host
and network byte order

Why do we re-invent the wheel?
- ----------------------------------------------------------------------

+------------------+ +--- gerhardr@tech-edv.co.at ----+
T.B.Reithofer \ Gerhard | Technical Sofware Developement |
Staatsbahnstr. 100 \ Reithofer | CAD/CAM/CAE/CAQ/CAP +----------+
A-2136 Laa/Thaya +----------+ Mechanical CAD +----+
Tel +43-2522/8726, Fax +43-2522/87268 +---------+
+---------------------------------------+

------------------------------

Search Discussions

  • Thomas G. Lockhart at Jul 4, 1997 at 11:04 pm

    I have already committed patches to the source tree which had been
    contributed by Tatsuo Ishii <t-ishii@sra.co.jp>.
    Tatsuo Ishii's patch (I think it was his path?) repaired the "wrong-reversed"
    byte order in pqcomprim.c into the "correct-reversed" byte order.
    OK, so your conclusion is that the consistant, working byte ordering is
    backwards from what would be obtained by using the "work-alike" --
    actually the original -- routines provided on some or most systems.
    But applications using direct socket I/O, reading an "u_short/u_long"
    and translating it into a system defined "u_short/u_long" by using
    "ntohs/ntohl" will NOT get correct values!
    E. g. the PostODBC driver is such an application.
    I don't know why, in addition to the system defines htonl/htons and
    ntohl/ntohs, the postgres specific "ntopl/ntops" and "ptonl/ptons" ('p'
    stands for postgres) translation have been defined.
    I don't know why either. I can't recall the exact name of the person
    contributing the big-endian/little-endian interface (Martin L?). We
    should hear his opinion on the best course of action for the long term.

    For the v6.1 patch release, does the little-endian behavior of the
    interface cause fatal trouble for ODBC drivers and other add-on software
    (Gerhard suggests that it does)? If so, we should hear from the original
    implementor to confirm that using the system-provided routines is
    possible, then move toward Gerhard's solution.

    Comments?

    - Tom

    ------------------------------
  • Bruce Momjian at Jul 5, 1997 at 2:54 am

    Hi to all Hackers,

    =========================================================================
    On Wed, 2 Jul 1997, Thomas G. Lockhart wrote:

    I have already committed patches to the source tree which had been
    contributed by Tatsuo Ishii <t-ishii@sra.co.jp>.
    Tatsuo Ishii's patch (I think it was his path?) repaired the "wrong-reversed"
    byte order in pqcomprim.c into the "correct-reversed" byte order.
    As a result all Front-ends using "pqlib" will work on all platforms. I
    personally would call this as "workaround" - it leads to correct behavior
    inside the postgres world - real good work - but...

    But applications using direct socket I/O, reading an "u_short/u_long"
    and translating it into a system defined "u_short/u_long" by using
    "ntohs/ntohl" will NOT get correct values!
    E. g. the PostODBC driver is such an application.
    This is correct. For some reason that no one knows, the Postgres byte
    order is OPPOSITE from the standard network byte order.

    I think it got confusing when we added the "ntopl/ntops" and
    "ptonl/ptons" macros, because they do the opposite from the standard
    functions.

    We plan 'some day' to repair this, and make our byte order that same as
    everyone else's standard byte order, but we need to do that in a major
    release, and we need to give all the tool providers a 'heads up' on the
    change so they can have new versions available before a general release.


    - --
    Bruce Momjian
    maillist@candle.pha.pa.us

    ------------------------------

    End of hackers-digest V1 #410
    *****************************
  • Gerhard Reithofer at Jul 5, 1997 at 5:00 pm
    On Fri, 4 Jul 1997, Bruce Momjian wrote:

    ... CUT ...
    We plan 'some day' to repair this, and make our byte order that same as
    everyone else's standard byte order, but we need to do that in a major
    release, and we need to give all the tool providers a 'heads up' on the
    change so they can have new versions available before a general release.
    Thanks for your quick reply.

    I'd like to state the position of PostODBC at the moment:

    PostODBC V0.21 works until PostgreSQL V6.0 and with PostgreSQL V6.1 on
    little endian machines (like Linux) and may work (? - I don't really know)
    with Tatsuo's patch with big endian machines.

    An not-official solution is available via email (gerhardr@tech-edv.co.at)
    whichs works with PostgreSQL V6.1 on all plattforms, if the patch
    "pqprimcom.path" is applied. Available is the source or a compiled 16-bit
    version. I will include the patch too, if requested.

    If a new PostgreSQL version will be released, an official PostODBC version
    will be available (usually within 1 or 2 days).

    ========================================================================
    BTW Christian Czezatke suggested to include a version identification of
    the postgres frontend from the client side.

    E.g. the identification package could be used for this. The client could
    send a required version number and the frontend could refuse the
    connection if not identical. Another way could be to query the frontend
    version from the client side.

    The idea behind is, that changes in the communication protocol and/or
    SQL syntax could be implemented into "one" driver, which responds
    depending on the frontend version.

    Any suggestions?

    Gerhard Reithofer

    +------------------+ +--- gerhardr@tech-edv.co.at ----+
    T.B.Reithofer \ Gerhard | Technical Sofware Developement |
    Staatsbahnstr. 100 \ Reithofer | CAD/CAM/CAE/CAQ/CAP +----------+
    A-2136 Laa/Thaya +----------+ Mechanical CAD +----+
    Tel +43-2522/8726, Fax +43-2522/87268 +---------+
    +---------------------------------------+

    ------------------------------
  • Tatsuo Ishii at Jul 7, 1997 at 5:40 am

    This is correct. For some reason that no one knows, the Postgres byte
    order is OPPOSITE from the standard network byte order.

    I think it got confusing when we added the "ntopl/ntops" and
    "ptonl/ptons" macros, because they do the opposite from the standard
    functions.

    We plan 'some day' to repair this, and make our byte order that same as
    everyone else's standard byte order, but we need to do that in a major
    release, and we need to give all the tool providers a 'heads up' on the
    change so they can have new versions available before a general release.
    I'm glad to hear you have a plan to use the standard network byte
    order in the future. And I completely agree with you that the
    migration should be done in a major release.

    BTW, regarding the endian patch I'd like to add new facts.

    o Jim Bennett's Java-libpq interface
    (http://www.tara-lu.com/~jimb/jsql/) does not work with PostgreSQL 6.1
    but works fine with both PostgreSQL 6.0 and 6.1 the endian patch
    applied.

    o Kataoka (kataoka@matsu.interwiz.koganei.tokyo.jp) and Yoshihiko
    Ichikawa (ichikawa@is.ocha.ac.jp) reported that by applying the endian
    patch, i386-solaris and sparc-soalris successfully communicated with
    each other(both were compiled by gcc-2.7.2).
    - ---
    Tatuso Ishii
    t-ishii@sra.co.jp

    ------------------------------
  • Bruce Momjian at Jul 7, 1997 at 3:16 pm

    This is correct. For some reason that no one knows, the Postgres byte
    order is OPPOSITE from the standard network byte order.

    I think it got confusing when we added the "ntopl/ntops" and
    "ptonl/ptons" macros, because they do the opposite from the standard
    functions.

    We plan 'some day' to repair this, and make our byte order that same as
    everyone else's standard byte order, but we need to do that in a major
    release, and we need to give all the tool providers a 'heads up' on the
    change so they can have new versions available before a general release.
    I'm glad to hear you have a plan to use the standard network byte
    order in the future. And I completely agree with you that the
    migration should be done in a major release.

    BTW, regarding the endian patch I'd like to add new facts.

    o Jim Bennett's Java-libpq interface
    (http://www.tara-lu.com/~jimb/jsql/) does not work with PostgreSQL 6.1
    but works fine with both PostgreSQL 6.0 and 6.1 the endian patch
    applied.

    o Kataoka (kataoka@matsu.interwiz.koganei.tokyo.jp) and Yoshihiko
    Ichikawa (ichikawa@is.ocha.ac.jp) reported that by applying the endian
    patch, i386-solaris and sparc-soalris successfully communicated with
    each other(both were compiled by gcc-2.7.2).
    OK, good news.

    Now that we have reports of a complete Endian fix for 6.1, can we
    schedule a release of 6.1p1?

    - --
    Bruce Momjian
    maillist@candle.pha.pa.us

    ------------------------------
  • Thomas G. Lockhart at Jul 7, 1997 at 4:01 pm

    This is correct. For some reason that no one knows, the Postgres byte
    order is OPPOSITE from the standard network byte order.
    We plan 'some day' to repair this, and make our byte order that same as
    everyone else's standard byte order, but we need to do that in a major
    release...
    Tatsuo, apparently the only remaining question regarding the endian
    patches is where to place the endian definitions if endian.h is not
    available. You placed the definitions in postgres.h, but expressed
    reservations, and Gerhard suggests that perhaps individual <platform>.h
    files would be better (his machine had trouble with postgres.h as-is).

    How many of your test platforms actually required the BYTE_ORDER or
    BIG/LITTLE_ENDIAN definitions? There is at least one platform (AIX,
    *sigh*) which provides the definitions but does not have an endian.h,
    and which will need an extra #include to find it (very platform
    specific, so should go in aix.h?). If there are only one or two other
    platforms which do not have endian.h, then it seems that the
    <platform>.h file might be the best place to locate this for the v6.1
    patch release.

    What is your opinion? Everyone??

    - Tom

    ------------------------------
  • Tatsuo Ishii at Jul 8, 1997 at 4:52 am

    This is correct. For some reason that no one knows, the Postgres byte
    order is OPPOSITE from the standard network byte order.
    We plan 'some day' to repair this, and make our byte order that same as
    everyone else's standard byte order, but we need to do that in a major
    release...
    Tatsuo, apparently the only remaining question regarding the endian
    patches is where to place the endian definitions if endian.h is not
    available. You placed the definitions in postgres.h, but expressed
    reservations, and Gerhard suggests that perhaps individual <platform>.h
    files would be better (his machine had trouble with postgres.h as-is).

    How many of your test platforms actually required the BYTE_ORDER or
    BIG/LITTLE_ENDIAN definitions? There is at least one platform (AIX,
    *sigh*) which provides the definitions but does not have an endian.h,
    and which will need an extra #include to find it (very platform
    specific, so should go in aix.h?). If there are only one or two other
    platforms which do not have endian.h, then it seems that the
    <platform>.h file might be the best place to locate this for the v6.1
    patch release.

    What is your opinion? Everyone??

    - Tom
    We checked some platforms. Here are the result:

    Followings have BIG/LITTLE_ENDIAN definitions in endian.h.
    o BSD4.4 lite derived platforms (FreeBSD, NetBSD) and BSD/OS 2.x have
    /usr/include/machine/endian.h
    o NEWS OS 4.2.1 --> /usr/include/machine/endian.h

    Followings have BIG/LITTLE_ENDIAN definitions but in different place.
    o NEWS OS 6.1.2 --> /usr/include/sys/byteorder.h
    o NEC EWS UX/V7.1 --> /usr/include/sys/byteorder.h

    Followings do not have BIG/LITTLE_ENDIAN definitions.
    o Solaris2.4/Solaris2.5.x/gcc
    BIG/LITTLE_ENDIAN are defined in gnu/lib/gcc-lib/*/include/sys/byteorder.h
    but the name of definitions are different(__BIG_ENDIAN__ , __LITTLE_ENDIAN__)
    o Solaris2.5.x/cc does not have any BIG/LITTLE_ENDIAN definitions
    o SunOS 4.1.x cc/gcc does not have any BIG/LITTLE_ENDIAN definitions

    Note that if the bind package is installed, there will be complete
    BIG/LITTLE_ENDIAN definitions in /usr/include/arpa/nameser.h. May be
    we should check the existance of nameser.h in configure script.
    - --
    Tatsuo Ishii
    t-ishii@sra.co.jp

    ------------------------------
  • Bruce Momjian at Jul 8, 1997 at 5:16 am

    We checked some platforms. Here are the result:

    Followings have BIG/LITTLE_ENDIAN definitions in endian.h.
    o BSD4.4 lite derived platforms (FreeBSD, NetBSD) and BSD/OS 2.x have
    /usr/include/machine/endian.h
    o NEWS OS 4.2.1 --> /usr/include/machine/endian.h

    Followings have BIG/LITTLE_ENDIAN definitions but in different place.
    o NEWS OS 6.1.2 --> /usr/include/sys/byteorder.h
    o NEC EWS UX/V7.1 --> /usr/include/sys/byteorder.h

    Followings do not have BIG/LITTLE_ENDIAN definitions.
    o Solaris2.4/Solaris2.5.x/gcc
    BIG/LITTLE_ENDIAN are defined in gnu/lib/gcc-lib/*/include/sys/byteorder.h
    but the name of definitions are different(__BIG_ENDIAN__ , __LITTLE_ENDIAN__)
    o Solaris2.5.x/cc does not have any BIG/LITTLE_ENDIAN definitions
    o SunOS 4.1.x cc/gcc does not have any BIG/LITTLE_ENDIAN definitions

    Note that if the bind package is installed, there will be complete
    BIG/LITTLE_ENDIAN definitions in /usr/include/arpa/nameser.h. May be
    we should check the existance of nameser.h in configure script.
    With these kind of results, perhaps we should just write a test program
    to determine the proper ENDIAN value. Shouldn't be too hard, and will
    be 100% portable.

    - --
    Bruce Momjian
    maillist@candle.pha.pa.us

    ------------------------------
  • Gerhard Reithofer at Jul 8, 1997 at 10:17 am
    Hi to all Hackers,
    On Mon, 7 Jul 1997, Thomas G. Lockhart wrote:

    Tatsuo, apparently the only remaining question regarding the endian
    patches is where to place the endian definitions if endian.h is not
    available. You placed the definitions in postgres.h, but expressed
    reservations, and Gerhard suggests that perhaps individual <platform>.h
    files would be better (his machine had trouble with postgres.h as-is).

    How many of your test platforms actually required the BYTE_ORDER or
    BIG/LITTLE_ENDIAN definitions? There is at least one platform (AIX,
    *sigh*) which provides the definitions but does not have an endian.h,
    and which will need an extra #include to find it (very platform
    specific, so should go in aix.h?). If there are only one or two other
    platforms which do not have endian.h, then it seems that the
    <platform>.h file might be the best place to locate this for the v6.1
    patch release.

    What is your opinion? Everyone??
    =======================================================================
    Which processors?

    IMHO most processor types use big endian representation. E.g. the old
    IBM/RS6000, the new Power PC's, Sun Sparc, all 68x processors
    (Motorola)... All little endians, that I know are Intel 86x and DEC Alpha
    AXP's.
    ========================================================================
    Where to implement?

    There's not place for implementation. It's usually done by the system.
    For network communication the include file <netinet/in.h> defines the
    stuff. For type definitions you should include <unistd.h> before
    <netinet/in.h> (may be obsolete on some systems?).

    The programmer should NOT deal with endians and all #define *ENDIAN*
    should be removed from the code.

    A little test program shows that:
    ==============================================================
    #include <unistd.h>
    #include <netinet/in.h>

    #define NUMBER 0x1234
    #define LUMBER 0xDeadBeef

    main()
    {
    unsigned short int ns;
    unsigned long int nl;
    system("echo Machine type is `uname`.");
    ns=htons(NUMBER);
    printf("The short number %x is in the net %x.\n",NUMBER,ns);
    nl=htonl(LUMBER);
    printf("The long number %lx is in the net %lx.\n",LUMBER,nl);
    }
    ===================================================================

    gerhardr@aix530:/home/gerhardr > ./a.out
    Machine type is AIX.
    The short number 1234 is in the net 1234.
    The long number deadbeef is in the net deadbeef.

    gerhardr@server:/home/gerhardr > ./a.out
    Machine type is Linux.
    The short number 1234 is in the net 3412.
    The long number deadbeef is in the net efbeadde.

    ... it's an Intel - a PPC Linux should react like in the 1st example.

    Best regards,
    Gerhard

    +------------------+ +--- gerhardr@tech-edv.co.at ----+
    T.B.Reithofer \ Gerhard | Technical Sofware Developement |
    Staatsbahnstr. 100 \ Reithofer | CAD/CAM/CAE/CAQ/CAP +----------+
    A-2136 Laa/Thaya +----------+ Mechanical CAD +----+
    Tel +43-2522/8726, Fax +43-2522/87268 +---------+
    +---------------------------------------+

    ------------------------------
  • Frank Dana at Jul 8, 1997 at 1:28 pm

    With Shakespearian flourish, Bruce Momjian writes:
    [Endian stats]
    With these kind of results, perhaps we should just write a test program
    to determine the proper ENDIAN value. Shouldn't be too hard, and will
    be 100% portable.
    That's what Perl does, and it may be the best method. It protects
    against situations like in AIX where the ENDIAN defines are in an
    unexpected place and are sometimes included, sometimes not. Just test
    and define a PG_HOSTORDER appropriately from config.h.

    -Frank

    - --
    Frank R. Dana, Jr. Senior Associate Programmer
    danaf@ans.net ANS Communications
    (914) 789-5449 100 Clearbrook Rd. Elmsford, NY 10523
    Pager: 800-946-4646 pin# 1420717

    ------------------------------
  • D'Arcy J.M. Cain at Jul 8, 1997 at 2:30 pm
    Thus spake Bruce Momjian
    With these kind of results, perhaps we should just write a test program
    to determine the proper ENDIAN value. Shouldn't be too hard, and will
    be 100% portable.
    Here's a thought. Why not write a routine that first calls the standard
    byte ordering macros then reverses the result? Let the system get it into
    a known state and operate on that. It's only temporary anyway, right?

    - --
    D'Arcy J.M. Cain | Democracy is three wolves
    darcy@{druid.net|vex.net} | and a sheep voting on
    +1 416 424 2871 (DoD#0082) (eNTP) | what's for dinner.
    -- http://www.druid.net/darcy --

    ------------------------------
  • Martin J. Laubach at Jul 9, 1997 at 2:30 pm
    So... to avoid another endian-desaster -- anyone on the list
    who would like to work with me on a redesign of the comm protocol?
    I'd like to have someone cross-check design and implementation,
    possibly testing stuff out on different plattforms.

    Pleeeeeeeeeeeeease?

    mjl

    ------------------------------

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppgsql-hackers @
categoriespostgresql
postedJul 4, '97 at 8:55a
activeJul 9, '97 at 2:30p
posts13
users7
websitepostgresql.org...
irc#postgresql

People

Translate

site design / logo © 2021 Grokbase