FAQ
Hi,

I've encountered difficulties utilising the mail() function properly
under a NIX environments while conforming to RFC 2822. There two
specific issues, one is a code problem the other a documentation
issue they are intertwined so I thought best run it by here and see
that people understand the issue before I head to the bug tracker.

As you are no doubt aware mail() operates using the system sendmail
interface under NIX environments, and under Windows it either utilises a
sendmail binary or more commonly the SMTP feature. The problem stems
from confusion and inappropriate application of the SMTP standards to
the sendmail interface.

Under a NIX environment the sendmail interface operates taking an e-mail
message constructed using the system line ending, LF. I've confirmed
this applies to the Postfix sendmail interface [1], and could ask other
vendors.

Under Windows either the sendmail binary or SMTP is utilised. The
correct line endings for SMTP are certainly CR-LF, sendmail I guess
could be either.

The documentation mentions to use CR-LF as the line endings for the
$additional_headers parameters whilst the function implementation
utilises LFs [2] which results in a mix of line endings, which is worse
than incorrect line endings. There is a 7 year old bug open about this
inconsistency [3] and as of yet nobody has fixed it, I hope this can be
rectified.

The second issue I have is quite a show stopper with regards to properly
formatted emails under the NIX environment; email headers maybe no
longer than 998 characters but the advised cut off is 78, a long
header should be folded over two lines with some white space
indentation. [4] As the To and Subject headers are populated from the
function arguments of the same name they pass through additional checks
over the other headers. The problem arises with the SKIP_LONG_HEADER_SEP
macro [5] which only skips over the SMTP standard of CR-LF-WSP and not
LF-WSP required for NIX sendmail as a consequence the LF is replace with
a space effectively unfolding the folded line. It is therefore
impossible to correctly create an email using mail() with a large number
of recipients or a long subject, the effective limit of text shrinks
when encoding is used for non ASCII characters as well.

Can anyone provide me with any insight?

Regards,

Chris

[1] http://thread.gmane.org/gmane.mail.postfix.user/200784
[2]
http://svn.php.net/viewvc/php/php-src/branches/PHP_5_3/ext/standard/mail.c?view=markup#l312
[3] http://bugs.php.net/bug.php?id=15841
http://tools.ietf.org/html/rfc2822#section-2.2.3
[4] http://tools.ietf.org/html/rfc2822#section-2.2.3
[5]
http://svn.php.net/viewvc/php/php-src/branches/PHP_5_3/ext/standard/mail.c?view=markup#l56

--
Chris Smith

Search Discussions

  • Joey Smith at Aug 21, 2009 at 7:11 pm

    On Fri, Aug 21, 2009 at 04:55:31PM +0100, Chris Smith wrote:

    I've encountered difficulties utilising the mail() function properly
    under a NIX environments while conforming to RFC 2822. There two
    specific issues, one is a code problem the other a documentation
    issue they are intertwined so I thought best run it by here and see
    that people understand the issue before I head to the bug tracker.

    As you are no doubt aware mail() operates using the system sendmail
    interface under NIX environments, and under Windows it either utilises a
    sendmail binary or more commonly the SMTP feature. The problem stems
    from confusion and inappropriate application of the SMTP standards to
    the sendmail interface.

    Under a NIX environment the sendmail interface operates taking an e-mail
    message constructed using the system line ending, LF. I've confirmed
    this applies to the Postfix sendmail interface [1], and could ask other
    vendors.
    1) Maybe you could go back to Wietse and ask him to justify the
    seemingly contradictory assertions that "text is expected to be
    in native UNIX stream-LF format" [1] but "Postfix receives local
    submissions in (LF or CRLF) format" [2] and "Postfix looks at
    the first input line [to determine what format you are using]" [3]?
    It's hard to know what to tell PHP developers when we get mixed
    messages from someone like Wietse...

    As for "other vendors", I can tell you first-hand that sendmail and
    exim both handle "mixed line-endings" just fine. However, I agree that
    not sending them in the first place is the ideal scenario...
    Under Windows either the sendmail binary or SMTP is utilised. The
    correct line endings for SMTP are certainly CR-LF, sendmail I guess
    could be either.
    2) Perhaps this has changed, but last time I looked, the built-in
    SMTP client is the ONLY method available for mail() on Win32.
    The documentation mentions to use CR-LF as the line endings for the
    $additional_headers parameters whilst the function implementation
    utilises LFs [2] which results in a mix of line endings, which is worse
    than incorrect line endings. There is a 7 year old bug open about this
    inconsistency [3] and as of yet nobody has fixed it, I hope this can be
    rectified.
    3) I don't have an Apple platform for testing, what will happen on
    Mac if PHP_EOL is used as the separator for $additional_headers? I
    would like to change the documentation to say "Multiple extra
    headers should be separated with the PHP_EOL constant", but I'm not
    the least bit certain this is going to work correctly on Mac. I can
    tell you that on my machines (Linux, using a mix exim and sendmail
    as MTAs), it will not see the \r as a separator, but mixing \r\n and
    \n within the same message works just fine (another case of the
    ever-prevalent SMTP mantra of "Be permissive in what you accept, and
    strict in what you send"). If PHP_EOL can't be safely used, I imagine
    we'll have to document it as 'Use "\r\n" on Win32, and "\n"
    everywhere else', which I'd really rather not do - it seems hackish.

    The second issue I have is quite a show stopper with regards to properly
    formatted emails under the NIX environment; email headers maybe no
    longer than 998 characters but the advised cut off is 78, a long
    header should be folded over two lines with some white space
    indentation. [4] As the To and Subject headers are populated from the
    function arguments of the same name they pass through additional checks
    over the other headers. The problem arises with the SKIP_LONG_HEADER_SEP
    macro [5] which only skips over the SMTP standard of CR-LF-WSP and not
    LF-WSP required for NIX sendmail as a consequence the LF is replace with
    a space effectively unfolding the folded line. It is therefore
    impossible to correctly create an email using mail() with a large number
    of recipients or a long subject, the effective limit of text shrinks
    when encoding is used for non ASCII characters as well.
    4) I don't write/maintain an MTA, but it seems like you're conflating
    SMTP and the "local pipe to a sendmail binary" conversation where it
    should not be. If you're sending your emails to a local pipe, I don't
    think you should be wrapping your headers in the PHP code with a LF-WSP.
    The section of RFC2822 on folding long headers quite clearly states [4]:

    The general rule is that wherever this standard allows for
    folding white space (not simply WSP characters), a *CRLF* may
    be inserted before any WSP. [emphasis mine]

    Since PHP's mail() doesn't know if it's sending over SMTP or a local
    pipe, I honestly feel that SKIP_LONG_HEADER_SEP is doing the correct
    thing here - your MTA should be able to accept long headers via a
    local pipe and make sure it formats them appropriately before sending
    them on to their SMTP destination - PHP should only be concerned with
    SKIP_LONG_HEADER_SEP in a *pure* SMTP situation.

    Note: I just fed exim a 12000 character long subject from PHP and exim
    wrapped it before sending it to the other end of the SMTP conversation.
    Are you saying that Postfix does not do this?
  • Wietse Venema at Aug 21, 2009 at 8:41 pm

    Joey Smith:
    On Fri, Aug 21, 2009 at 04:55:31PM +0100, Chris Smith wrote:

    I've encountered difficulties utilising the mail() function properly
    under a NIX environments while conforming to RFC 2822. There two
    specific issues, one is a code problem the other a documentation
    issue they are intertwined so I thought best run it by here and see
    that people understand the issue before I head to the bug tracker.

    As you are no doubt aware mail() operates using the system sendmail
    interface under NIX environments, and under Windows it either utilises a
    sendmail binary or more commonly the SMTP feature. The problem stems
    from confusion and inappropriate application of the SMTP standards to
    the sendmail interface.

    Under a NIX environment the sendmail interface operates taking an e-mail
    message constructed using the system line ending, LF. I've confirmed
    this applies to the Postfix sendmail interface [1], and could ask other
    vendors.
    1) Maybe you could go back to Wietse and ask him to justify the
    seemingly contradictory assertions that "text is expected to be
    in native UNIX stream-LF format" [1] but "Postfix receives local
    submissions in (LF or CRLF) format" [2] and "Postfix looks at
    the first input line [to determine what format you are using]" [3]?
    It's hard to know what to tell PHP developers when we get mixed
    messages from someone like Wietse...
    The Postfix sendmail command prefers input in native UNIX stream-lf
    format. Postfix will jump some hoops for software that wants to
    use the non-native CRLF format. It uses a switch (going from using
    LF to using CRLF) and therefore it won't accept mixed line endings.
    4) I don't write/maintain an MTA, but it seems like you're conflating
    SMTP and the "local pipe to a sendmail binary" conversation where it
    If you want email to survive the limitations of SMTP, then you need
    to submit the email message in a format that is compatible with
    the limitations of SMTP. It is not the job of MTAs to automagically
    fold and unfold long lines.
    Note: I just fed exim a 12000 character long subject from PHP and exim
    wrapped it before sending it to the other end of the SMTP conversation.
    Are you saying that Postfix does not do this?
    I would expect that a user agent (such as /usr/bin/mail, Mutt etc.)
    would do the header folding, not the MTA. In fact, am not aware of
    any RFC that requires MTAs to wrap long message headers, but I am
    willing to be corrected by someone who can refer me to the document.

    Wietse
  • Daniel Convissor at Aug 22, 2009 at 1:23 am
    Hi:
    On Fri, Aug 21, 2009 at 04:41:31PM -0400, Wietse Venema wrote:

    The Postfix sendmail command prefers input in native UNIX stream-lf
    format. Postfix will jump some hoops for software that wants to
    use the non-native CRLF format. It uses a switch (going from using
    LF to using CRLF) and therefore it won't accept mixed line endings.
    That's my experience too. Along those lines, something seems amiss in
    mail.c in the 5.3 branch, and perhaps in other branches.

    Line 244 uses \r\n while line 246 uses \n.

    Those originating script header lines were added in revision 273168,
    though line 244 originally had the \r\n at the start of the line and a \n
    at the end. Then revision 282504 (in response to bug 48620) removed the
    starting \r\n but changed the trailing \n to an \r\n.

    A quick look at mail.c and the documentation seems to indicate \r\n is
    the goal, so line 246 likely needs changing from \n to \r\n.

    Thanks,

    --Dan

    --
    T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y
    data intensive web and database programming
    http://www.AnalysisAndSolutions.com/
    4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
  • Joey Smith at Aug 23, 2009 at 3:41 pm
    Wietse:
    On Fri, Aug 21, 2009 at 04:41:31PM -0400, Wietse Venema wrote:
    The Postfix sendmail command prefers input in native UNIX stream-lf
    format. Postfix will jump some hoops for software that wants to
    use the non-native CRLF format. It uses a switch (going from using
    LF to using CRLF) and therefore it won't accept mixed line endings.
    Thanks. I can understand not accepting mixed line endings, I just
    wanted to understand what exactly I should tell people since the two
    answers seemed to conflict with each other.
  • Davey Shafik at Aug 21, 2009 at 8:56 pm
    PHP_EOL is \n on OS X. So the \r worries are not a
    concern. PHP_EOL would be fine in this case, assuming
    the OSX sendmail is fine with it.

    - Davey

    On Aug 21, 2009, at 03:11 PM, Joey Smith wrote:

    <snip>
    3) I don't have an Apple platform for testing, what will happen on
    Mac if PHP_EOL is used as the separator for $additional_headers? I
    would like to change the documentation to say "Multiple extra
    headers should be separated with the PHP_EOL constant", but I'm not
    the least bit certain this is going to work correctly on Mac. I can
    tell you that on my machines (Linux, using a mix exim and sendmail
    as MTAs), it will not see the \r as a separator, but mixing \r\n and
    \n within the same message works just fine (another case of the
    ever-prevalent SMTP mantra of "Be permissive in what you accept, and
    strict in what you send"). If PHP_EOL can't be safely used, I imagine
    we'll have to document it as 'Use "\r\n" on Win32, and "\n"
    everywhere else', which I'd really rather not do - it seems hackish.
  • Wietse Venema at Aug 21, 2009 at 9:02 pm

    Joey Smith wrote:
    3) I don't have an Apple platform for testing, what will happen on
    Mac if PHP_EOL is used as the separator for $additional_headers? I
    would like to change the documentation to say "Multiple extra
    headers should be separated with the PHP_EOL constant", but I'm not
    the least bit certain this is going to work correctly on Mac. I can
    tell you that on my machines (Linux, using a mix exim and sendmail
    as MTAs), it will not see the \r as a separator, but mixing \r\n and
    \n within the same message works just fine (another case of the
    ever-prevalent SMTP mantra of "Be permissive in what you accept, and
    strict in what you send"). If PHP_EOL can't be safely used, I imagine
    we'll have to document it as 'Use "\r\n" on Win32, and "\n"
    everywhere else', which I'd really rather not do - it seems hackish.
    Davey Shafik:
    PHP_EOL is \n on OS X. So the \r worries are not a
    concern. PHP_EOL would be fine in this case, assuming
    the OSX sendmail is fine with it.
    Oops. I forgot to mention that Postfix is the default MTA for Mac
    OS X. That system is based off FreeBSD, and it uses the same text
    encoding as *NIX systems.

    Wietse
  • Chris Smith at Aug 21, 2009 at 9:38 pm

    Joey Smith wrote:
    1) Maybe you could go back to Wietse and ask him to justify the
    seemingly contradictory assertions that "text is expected to be
    in native UNIX stream-LF format" [1] but "Postfix receives local
    submissions in (LF or CRLF) format" [2] and "Postfix looks at
    the first input line [to determine what format you are using]" [3]?
    It's hard to know what to tell PHP developers when we get mixed
    messages from someone like Wietse...
    I think from his previous replies on the subject Postfix tolerates CRLF
    formatted submissions and prefers LF, which is understandable as it is a
    NIX program.
    As for "other vendors", I can tell you first-hand that sendmail and
    exim both handle "mixed line-endings" just fine. However, I agree that
    not sending them in the first place is the ideal scenario...
    Mixed line endings should never be passed to the mailer.
    2) Perhaps this has changed, but last time I looked, the built-in
    SMTP client is the ONLY method available for mail() on Win32.
    It is perfectly possible to use sendmail under Windows it's not really
    advertised though, I've not tested it either.
    3) I don't have an Apple platform for testing, what will happen on
    Mac if PHP_EOL is used as the separator for $additional_headers?
    Well you are entering the realms of pre-OSX systems there, not really a
    valid concern.
    4) I don't write/maintain an MTA, but it seems like you're conflating
    SMTP and the "local pipe to a sendmail binary" conversation where it
    should not be. If you're sending your emails to a local pipe, I don't
    think you should be wrapping your headers in the PHP code with a LF-WSP.
    I have to disagree RFC 2822 specifies the message format. While I agree
    it does specify that the end of line should be CRLF and that folding is
    conducted using the CRLF followed by WSP, this is not really a point of
    discussion suited to PHP it is for the MTA authors. The derivation from
    the RFC 2822 specification as far as I am aware is just line endings,
    which ever the MTA expects should be used. I think its a fair assumption
    to expect nearly all mailers to permit both CRLF and LF separated
    emails, however the mail() function only permits talking LF. As stated
    previously sending mixed line endings should be a total no go, as it
    stands then only LF can be used by headers passed to mail().
    Since PHP's mail() doesn't know if it's sending over SMTP or a local
    pipe, I honestly feel that SKIP_LONG_HEADER_SEP is doing the correct
    thing here - your MTA should be able to accept long headers via a
    local pipe and make sure it formats them appropriately before sending
    them on to their SMTP destination - PHP should only be concerned with
    SKIP_LONG_HEADER_SEP in a *pure* SMTP situation.
    It is simple to detect the method used, SMTP is used if sendmail_path is
    empty and the system is Windows, otherwise sendmail is used. The
    SKIP_LONG_HEADER_SEP was added for a reason back in 2003 in response to
    bug #22355. [1]
    Note: I just fed exim a 12000 character long subject from PHP and exim
    wrapped it before sending it to the other end of the SMTP conversation.
    Are you saying that Postfix does not do this?
    Did you try encoding a 12000 UTF-8 string using RFC 2047 [2], this is
    where the real problems emerge. Postfix will handle a really long
    subject just fine, but an encoded header reaching that long gets folded
    rendering it invalid. This is why it is imperative to be able to fold
    the Subject and To headers in PHP, which is where it should be done.

    Regards,

    Chris

    [1] http://bugs.php.net/bug.php?id=22355
    [2] http://tools.ietf.org/html/rfc2047

    --
    Chris Smith
  • Tullio Andreatta ML at Aug 24, 2009 at 8:48 am

    Joey Smith:
    It's hard to know what to tell PHP developers when we get mixed
    messages from someone like Wietse...
    No, it's not so hard.
    PHP developers must follow the standard:
    - use "network EOL" (CRNL) when speaking SMTP;
    - use "system EOL" (CRNL or NL or CR) when speaking with local MTA.
    Eventually mail() should convert "mixed line-endings" to "right
    line-endings", and "right line-endings" may be selected by the
    system administrator (a php.ini "mail_line_ending = system|network"
    switch?).
    As for "other vendors", I can tell you first-hand that sendmail and
    exim both handle "mixed line-endings" just fine.
    Actually a "real" PHP programmer is not using mail() at all, because
    nobody know where the application will be installed.


    --
    Tullio Andreatta

    Disclaimer: "Please treat this email message in a reasonable way, or we
    might get angry" ( http://www.goldmark.org/jeff/stupid-disclaimers )
  • Richard Lynch at Aug 28, 2009 at 9:09 pm

    On Fri, August 21, 2009 2:11 pm, Joey Smith wrote:
    2) Perhaps this has changed, but last time I looked, the built-in
    SMTP client is the ONLY method available for mail() on Win32.
    I believe some people have had success in installing sendmail-like
    software on Windows and using the 'linux' sendmail version.

    But I could be mis-remembering...

    I have no intention of commenting on the actual thrust of this \n \r\n
    \r mess that has been stinking up not only php mail() but the entire
    software industry for decades...
    Then Chris Smith wrote:
    It is simple to detect the method used, SMTP is used if sendmail_path
    is empty and the system is Windows, otherwise sendmail is used.
    I think you should revise this to be "if sendmail_path is empty".

    Many *nix folks use SMTP as well...

    --
    Some people ask for gifts here.
    I just want you to buy an Indie CD for yourself:
    http://cdbaby.com/search/from/lynch
  • Chris Smith at Aug 28, 2009 at 9:23 pm

    Richard Lynch wrote:
    Then Chris Smith wrote:
    It is simple to detect the method used, SMTP is used if sendmail_path
    is empty and the system is Windows, otherwise sendmail is used.
    I think you should revise this to be "if sendmail_path is empty".

    Many *nix folks use SMTP as well...
    The SMTP function of mail() only works under Windows. [1]

    Regards,

    Chris

    [1]
    http://svn.php.net/viewvc/php/php-src/tags/php_5_3_0/ext/standard/mail.c?view=markup#l252

    --
    Chris Smith
  • Richard Lynch at Sep 10, 2009 at 8:20 pm

    On Fri, August 28, 2009 4:23 pm, Chris Smith wrote:
    Richard Lynch wrote:
    Then Chris Smith wrote:
    It is simple to detect the method used, SMTP is used if
    sendmail_path
    is empty and the system is Windows, otherwise sendmail is used.
    I think you should revise this to be "if sendmail_path is empty".

    Many *nix folks use SMTP as well...
    The SMTP function of mail() only works under Windows. [1]

    Regards,

    Chris

    [1]
    http://svn.php.net/viewvc/php/php-src/tags/php_5_3_0/ext/standard/mail.c?view=markup#l252
    Sorry. I was thinking of Manuel Lemos' SMTP class oft-used on Un*x to
    provide SMTP support, rather than the built-in mail() function.

    --
    Some people ask for gifts here.
    I just want you to buy an Indie CD for yourself:
    http://cdbaby.com/search/from/lynch

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupphp-internals @
categoriesphp
postedAug 21, '09 at 5:00p
activeSep 10, '09 at 8:20p
posts12
users7
websitephp.net

People

Translate

site design / logo © 2022 Grokbase