FAQ
RPC Server send buffer retains size of largest response ever sent
------------------------------------------------------------------

Key: HADOOP-4802
URL: https://issues.apache.org/jira/browse/HADOOP-4802
Project: Hadoop Core
Issue Type: Bug
Components: ipc
Affects Versions: 0.19.0, 0.18.2
Reporter: stack


The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • stack (JIRA) at Dec 7, 2008 at 8:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HADOOP-4802:
    --------------------------

    Attachment: 4802.patch

    Suggested fix: reset BAOS if larger than N * initial buffer size.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Dec 8, 2008 at 6:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654517#action_12654517 ]

    Doug Cutting commented on HADOOP-4802:
    --------------------------------------

    I'd rather not see new configuration options for this. We should rather try hard to pick values that work well in all cases, if possible.

    I wonder if the simpler change of not re-using the buffer might work just as well. If we simply allocate a new buffer per request whose initial size is 10k. I would be surprised if that allocation would be a significant RPC bottleneck, but we should of course test that with a benchmark.

    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Dec 8, 2008 at 7:17 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654526#action_12654526 ]

    Raghu Angadi commented on HADOOP-4802:
    --------------------------------------

    I was thinking on the same lines. Allocation per request would not hurt (mostly because various other extra costs). Coule of improvements by subclassing BAOS:

    # Keep a fixed buffer 10kb. inside reset() replace the previous buffer with this. This avoids allocation for most RPCs.
    # extra benefit : this would allow us to avoid a copy if the response could be written to socket in-line (common case). Right now server always copies (through asByteArray()).

    Note that this jira will increase the buffer copies for large (multi-MB) responses because of copy each time buffer expands. (Partly negated by copies reduced by HADOOP-4797, since it takes multiple write() invocations to write all the data. in jdk, each write copies all the data!). A future improvement could be to write our own stream that uses a list of buffers.

    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Dec 8, 2008 at 9:56 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654591#action_12654591 ]

    Raghu Angadi commented on HADOOP-4802:
    --------------------------------------
    extra benefit : this would allow us to avoid a copy if the response could be written to socket in-line (common case).
    Once we have this (here or in a future jira), the fixed buffer could be a direct buffer and that will remove yet another copy.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Dec 9, 2008 at 8:28 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HADOOP-4802:
    --------------------------

    Attachment: 4802-v2.patch

    Thanks for the feedback lads.

    I like your suggestions Raghu. This patch removes configuration and makes an attempt at #1. #2 and the third suggestion of direct buffer to socket would take more work (though we should do them, yeah).
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Dec 9, 2008 at 8:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654951#action_12654951 ]

    stack commented on HADOOP-4802:
    -------------------------------

    Patch applies to 0.19 and to TRUNK.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Dec 9, 2008 at 9:11 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654964#action_12654964 ]

    Doug Cutting commented on HADOOP-4802:
    --------------------------------------

    I'm still not convinced we should do more than replace the buf.reset() with buf = new ByteArrayOutputStream() and remove the initialization of buf altogether.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Dec 9, 2008 at 10:35 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654990#action_12654990 ]

    stack commented on HADOOP-4802:
    -------------------------------
    I'm still not convinced we should do more than replace the buf.reset() with buf = new ByteArrayOutputStream() and remove the initialization of buf altogether.
    Above is predicated on our running a 'benchmark'. What would you suggest I run?

    Why create new objects when it can be avoided (when response < 10k)?


    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Dec 9, 2008 at 10:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654992#action_12654992 ]

    Raghu Angadi commented on HADOOP-4802:
    --------------------------------------

    yes, other buffering improvements are more work and better suited in a new jira. Btw, we don't need the check '{{buf != initialBuffer}}'.

    Couple of advantages to current patch :

    # we can clearly say this won't change performance for common case. Though I agree that creating new stream (i.e. two more allocations) each time won't be noticeable (except ay be more frequent partial G/C on benchmarks).
    # Subclassing BAOS (or a new output stream) will be required for other buffering improvements any way.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Dec 9, 2008 at 10:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HADOOP-4802:
    --------------------------

    Attachment: 4802-v3.patch

    v3 removes the unnecessary equality test (Thanks Raghu)
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Dec 9, 2008 at 10:51 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654997#action_12654997 ]

    Raghu Angadi commented on HADOOP-4802:
    --------------------------------------

    Also, the new class needs to be static.

    Subjective : I don't think we should have to use jira numbers in comments. I think this is a specific enough problem that comment or javadoc could explain it in enough detail. Readers don't need to go to jira.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Dec 9, 2008 at 11:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HADOOP-4802:
    --------------------------

    Attachment: 4802-v4-TRUNK.patch

    Made HBAOS static (Had to move it out of Server#Handler to do so). Removed pointer to this issue from comments (Can go to svn log if really needed).
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802-v4-TRUNK.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Dec 9, 2008 at 11:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655022#action_12655022 ]

    Raghu Angadi commented on HADOOP-4802:
    --------------------------------------

    +1 for the patch.

    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802-v4-TRUNK.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Dec 10, 2008 at 12:43 am
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655040#action_12655040 ]

    Doug Cutting commented on HADOOP-4802:
    --------------------------------------
    Why create new objects when it can be avoided (when response < 10k)?
    It's probably a premature optimization. It adds code. Less is more.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802-v4-TRUNK.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Dec 10, 2008 at 4:59 am
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655095#action_12655095 ]

    stack commented on HADOOP-4802:
    -------------------------------
    It's probably a premature optimization. It adds code. Less is more.
    Do the above add up to a -1 on v4 of the patch?
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802-v4-TRUNK.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Dec 10, 2008 at 6:01 am
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655106#action_12655106 ]

    Doug Cutting commented on HADOOP-4802:
    --------------------------------------
    Do the above add up to a -1 on v4 of the patch?
    No, more like a +0. Without benchmarking its safest to not change things much. Do we have a good pure RPC benchmark?
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802-v4-TRUNK.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Edward J. Yoon (JIRA) at Dec 12, 2008 at 7:17 am
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655913#action_12655913 ]

    Edward J. Yoon commented on HADOOP-4802:
    ----------------------------------------
    stack - 09/Dec/08 12:32 PM
    Patch applies to 0.19 and to TRUNK.
    I hope it'll be fixed at the 0.18.3 release.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802-v4-TRUNK.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Dec 13, 2008 at 10:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656347#action_12656347 ]

    Raghu Angadi commented on HADOOP-4802:
    --------------------------------------

    TestRpcCpu micro-benchmark attached to HADOOP-4797 can be used to asses the CPU penalty of this patch for larger buffers. May not matter much since we do expect extra copies with this patch for larger responses.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack
    Attachments: 4802-v2.patch, 4802-v3.patch, 4802-v4-TRUNK.patch, 4802.patch


    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Jan 13, 2009 at 7:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HADOOP-4802:
    --------------------------

    Attachment: (was: 4802-v4-TRUNK.patch)
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack

    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Jan 13, 2009 at 7:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HADOOP-4802:
    --------------------------

    Attachment: (was: 4802-v3.patch)
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack

    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Jan 13, 2009 at 7:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HADOOP-4802:
    --------------------------

    Attachment: (was: 4802.patch)
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack

    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Jan 13, 2009 at 7:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack updated HADOOP-4802:
    --------------------------

    Attachment: (was: 4802-v2.patch)
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack

    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Jan 13, 2009 at 7:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663442#action_12663442 ]

    stack commented on HADOOP-4802:
    -------------------------------

    Removed my patches. Write me offline if want to know why.
    RPC Server send buffer retains size of largest response ever sent
    ------------------------------------------------------------------

    Key: HADOOP-4802
    URL: https://issues.apache.org/jira/browse/HADOOP-4802
    Project: Hadoop Core
    Issue Type: Bug
    Components: ipc
    Affects Versions: 0.18.2, 0.19.0
    Reporter: stack

    The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop. This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered. If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently. If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedDec 7, '08 at 7:46p
activeJan 13, '09 at 7:49p
posts24
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

stack (JIRA): 24 posts

People

Translate

site design / logo © 2022 Grokbase