FAQ
Hi All,

I have been hunting a BufferOverflowException on our production system for
the last while, and initially thought this was resolved by issue 53119, we
have subsequently upgraded to 7.0.32 and have still seen the issues.

Reading through the mailing list archive it seems I am not the only one
having issues with the NIO AJP Processor.

http://www.mailinglistarchive.com/html/users@tomcat.apache.org/2012-10/msg00477.html

Below is what we are seeing pretty regularly in our production environment.
This is probably load based, as we don't seem to see this in any of our
development environments.

Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at
org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
at
org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1122)
at org.apache.coyote.Response.doWrite(Response.java:504)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
at
org.apache.tomcat.util.buf.IntermediateOutputStream.write(C2BConverter.java:278)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at
org.apache.tomcat.util.buf.WriteConvertor.write(C2BConverter.java:242)
at java.io.BufferedWriter.write(BufferedWriter.java:188)
at
org.apache.tomcat.util.buf.C2BConverter.convert(C2BConverter.java:125)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:476)
at
org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:152)
at
org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
at
org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:336)
at
org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:353)
at ... 72 more

As mentioned in the mailing list, could this be an issue that just needs to
be re-applied to the nio version of the Processor from the original issue I
thought this was?

I don't really know why this is happening, so can't provide a test case but
do know this is happening pretty regularly.

Steve

Search Discussions

  • Mark Thomas at Jan 17, 2013 at 12:10 pm

    On 17/01/2013 11:07, Steve Mactaggart wrote:
    Hi All,

    I have been hunting a BufferOverflowException on our production system for
    the last while, and initially thought this was resolved by issue 53119, we
    have subsequently upgraded to 7.0.32 and have still seen the issues.

    Reading through the mailing list archive it seems I am not the only one
    having issues with the NIO AJP Processor.

    http://www.mailinglistarchive.com/html/users@tomcat.apache.org/2012-10/msg00477.html

    Below is what we are seeing pretty regularly in our production environment.
    This is probably load based, as we don't seem to see this in any of our
    development environments.

    Caused by: java.nio.BufferOverflowException
    at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
    at
    org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
    at
    org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1122)
    at org.apache.coyote.Response.doWrite(Response.java:504)
    at
    org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)
    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
    at
    org.apache.tomcat.util.buf.IntermediateOutputStream.write(C2BConverter.java:278)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
    at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
    at
    org.apache.tomcat.util.buf.WriteConvertor.write(C2BConverter.java:242)
    at java.io.BufferedWriter.write(BufferedWriter.java:188)
    at
    org.apache.tomcat.util.buf.C2BConverter.convert(C2BConverter.java:125)
    at
    org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:476)
    at
    org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:152)
    at
    org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
    at
    org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:336)
    at
    org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:353)
    at ... 72 more

    As mentioned in the mailing list, could this be an issue that just needs to
    be re-applied to the nio version of the Processor from the original issue I
    thought this was?
    Unlikely. I have taken a look at the AjpNioProcessor code and I don't
    see any exception paths that could lead to the writeBuffer not being
    cleared.
    I don't really know why this is happening, so can't provide a test case but
    do know this is happening pretty regularly.
    How regularly? How many requests an hour and how many errors like the
    above an hour? (I'm trying to figure out what additional logging could
    be added that provides useful information without filling your disks
    with logs).

    The two possibilities at this point are:
    a) Tomcat bug
    b) Application bug

    b would be caused by something (e.g. servlet, filter) retaining a
    reference to a request and using it after the request had finished
    processing.

    If you set the following system property:
    org.apache.catalina.connector.RECYCLE_FACADES

    to true (e.g. in setenv.[bat|sh]) and your application is retaining
    references you should start to see NPEs and the stack trace should point
    to where the problem is.

    If you still see BufferOverflowException with that setting in place then
    that would make a Tomcat bug more likely.

    Mark


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Konstantin Preißer at Jan 17, 2013 at 1:52 pm
    Hi,
    -----Original Message-----
    From: Mark Thomas
    Sent: Thursday, January 17, 2013 1:11 PM
    On 17/01/2013 11:07, Steve Mactaggart wrote:
    Hi All,

    I have been hunting a BufferOverflowException on our production system
    for
    the last while, and initially thought this was resolved by issue 53119,
    we
    have subsequently upgraded to 7.0.32 and have still seen the issues.

    Reading through the mailing list archive it seems I am not the only one
    having issues with the NIO AJP Processor.
    Yes, I got these BufferOverflowExceptions when switching from AJP-APR to
    AJP-NIO.
    I added a comment to the bug for the AJP-APR connector:
    https://issues.apache.org/bugzilla/show_bug.cgi?id=53119


    Mark,
    The two possibilities at this point are:
    a) Tomcat bug
    b) Application bug

    b would be caused by something (e.g. servlet, filter) retaining a
    reference to a request and using it after the request had finished
    processing.

    If you set the following system property:
    org.apache.catalina.connector.RECYCLE_FACADES

    to true (e.g. in setenv.[bat|sh]) and your application is retaining
    references you should start to see NPEs and the stack trace should point
    to where the problem is.

    If you still see BufferOverflowException with that setting in place then
    that would make a Tomcat bug more likely.
    I'm certain that my applications don't retain references to request objects,
    but I will also try your suggestion with setting the RECYCLE_FACADES
    property and switching back to the AJP-NIO connector, to see if these
    BufferOverflowExceptions still occur.

    Thanks!


    Regards,
    Konstantin Preißer


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Konstantin Preißer at Jan 18, 2013 at 2:12 pm
    Hi Mark,
    If you set the following system property:
    org.apache.catalina.connector.RECYCLE_FACADES

    to true (e.g. in setenv.[bat|sh]) and your application is retaining
    references you should start to see NPEs and the stack trace should point
    to where the problem is.

    If you still see BufferOverflowException with that setting in place then
    that would make a Tomcat bug more likely.
    Since I also got BufferOverflowExceptions with AJP-NIO, I tried your
    suggestion setting the system property
    "org.apache.catalina.connector.RECYCLE_FACADES" to "true" (by setting it in
    the Java options using tomcat7w.exe) and switched from AJP-BIO to the
    AJP-NIO connector.

    Today, I got a BufferOverflowException again, this time using the following
    environment:
    Windows Server 2008 R2 SP1 x64,
    Java 1.7.0_10 (64 bit),
    Tomcat 7.0.35,
    Frontend: IIS 7.5 with ISAPI Redirector 1.2.37


    In the logfile, the following stacktrace was written:

    Jan 18, 2013 2:43:21 PM org.apache.coyote.ajp.AjpNioProcessor process
    SEVERE: Error processing request
    java.nio.BufferOverflowException
    at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
    at
    org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
    at
    org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(Abstra
    ctAjpProcessor.java:1124)
    at org.apache.coyote.Response.doWrite(Response.java:505)
    at
    org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
    383)
    at
    org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462)
    at
    org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334)
    at
    org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283)
    at
    org.apache.catalina.connector.Response.finishResponse(Response.java:513)
    at
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434)
    at
    org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:184)
    at
    org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstrac
    tProtocol.java:589)
    at
    org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
    1680)
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
    10)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
    03)
    at java.lang.Thread.run(Thread.java:722)

    The log of the isapi-redirector contains for the given time (but not only)
    an error:
    [Fri Jan 18 14:43:21.904 2013] [5040:1780] [error]
    isapi_write_client::jk_isapi_plugin.c (1286): WriteClient failed with 995
    (0x000003e3)

    The isapi log contains about 43 such errors ("WriteClient failed with 995")
    per hour, which probably means that in such a case, the ISAPI redirector
    closes the AJP connection before the full response has been sent.


    I'm not sure how many requests my Tomcat does process, but I would guess
    about 50-100 requests per second. The BufferOverflowExceptions seems to
    occur rarely, it took ~ 12 hours after switching to the NIO connector for
    the exception to occur.


    Do you have any ideas?


    Thanks,

    Konstantin Preißer


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Mark Thomas at Jan 20, 2013 at 4:47 pm

    On 18/01/2013 14:11, Konstantin Preißer wrote:
    Hi Mark,
    If you set the following system property:
    org.apache.catalina.connector.RECYCLE_FACADES

    to true (e.g. in setenv.[bat|sh]) and your application is retaining
    references you should start to see NPEs and the stack trace should point
    to where the problem is.

    If you still see BufferOverflowException with that setting in place then
    that would make a Tomcat bug more likely.
    Since I also got BufferOverflowExceptions with AJP-NIO, I tried your
    suggestion setting the system property
    "org.apache.catalina.connector.RECYCLE_FACADES" to "true" (by setting it in
    the Java options using tomcat7w.exe) and switched from AJP-BIO to the
    AJP-NIO connector.

    Today, I got a BufferOverflowException again, this time using the following
    environment:
    Windows Server 2008 R2 SP1 x64,
    Java 1.7.0_10 (64 bit),
    Tomcat 7.0.35,
    Frontend: IIS 7.5 with ISAPI Redirector 1.2.37


    In the logfile, the following stacktrace was written:

    Jan 18, 2013 2:43:21 PM org.apache.coyote.ajp.AjpNioProcessor process
    SEVERE: Error processing request
    java.nio.BufferOverflowException
    at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
    at
    org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
    at
    org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(Abstra
    ctAjpProcessor.java:1124)
    at org.apache.coyote.Response.doWrite(Response.java:505)
    at
    org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
    383)
    at
    org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462)
    at
    org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334)
    at
    org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283)
    at
    org.apache.catalina.connector.Response.finishResponse(Response.java:513)
    at
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434)
    at
    org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:184)
    at
    org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstrac
    tProtocol.java:589)
    at
    org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
    1680)
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
    10)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
    03)
    at java.lang.Thread.run(Thread.java:722)

    The log of the isapi-redirector contains for the given time (but not only)
    an error:
    [Fri Jan 18 14:43:21.904 2013] [5040:1780] [error]
    isapi_write_client::jk_isapi_plugin.c (1286): WriteClient failed with 995
    (0x000003e3)

    The isapi log contains about 43 such errors ("WriteClient failed with 995")
    per hour, which probably means that in such a case, the ISAPI redirector
    closes the AJP connection before the full response has been sent.


    I'm not sure how many requests my Tomcat does process, but I would guess
    about 50-100 requests per second. The BufferOverflowExceptions seems to
    occur rarely, it took ~ 12 hours after switching to the NIO connector for
    the exception to occur.


    Do you have any ideas?
    Only that it makes it more likely that there is a problem somewhere in
    the NIO AJP connector.

    With that number of requests adding additional logging needs to be very
    carefully targeted. That usually means having a pretty good idea where
    the problem is in which case just fixing the problem is as easy to do.

    I think it is time you raised a Bugzilla report that brings together the
    information in this thread. With an idea of what causes the problem it
    should be possible to track down the bug by code inspection but from
    past experience that is not as easy as it sounds.

    Mark

    Thanks,

    Konstantin Preißer


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categoriestomcat
postedJan 17, '13 at 11:07a
activeJan 20, '13 at 4:47p
posts5
users3
websitetomcat.apache.org
irc#tomcat

People

Translate

site design / logo © 2018 Grokbase