FAQ
DFS client can allow user to write data to the next block while uploading previous block to HDFS
------------------------------------------------------------------------------------------------

Key: HADOOP-1707
URL: https://issues.apache.org/jira/browse/HADOOP-1707
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: dhruba borthakur


The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.

A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • dhruba borthakur (JIRA) at Oct 9, 2007 at 4:24 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur reassigned HADOOP-1707:
    ----------------------------------------

    Assignee: dhruba borthakur

    I plan on removing the staging file altogether. The client will stream data to the datanodes directly, possibly in chunks of 64K memory buffers. Detail design to follow.
    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Oct 10, 2007 at 8:10 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533859 ]

    Doug Cutting commented on HADOOP-1707:
    --------------------------------------
    The client will stream data to the datanodes directly [ ... ]
    Some history to be aware of. Long ago writes were tee'd to datanodes directly, and the local file was only used to replay things. Switching it so that writes were always buffered to a local file had two advantages: it radically simplified the code (the tee multiplied the number of failure modes) and it improved performance & reliability. Each datanode had far fewer active connections, since blocks were written in a burst rather than as a trickle.

    How will you handle datanode failures? Since you have no local file to replay, won't those always cause an exception in the client? That will cause tasks to fail, which might be acceptable, now that things are overall more reliable, but, at the time I looked at this (again, long ago) datanode timeouts were frequent enough that this would cause job failure.
    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Oct 10, 2007 at 9:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533875 ]

    dhruba borthakur commented on HADOOP-1707:
    ------------------------------------------

    Thanks Doug for your comments.

    1. My thinking is as follows: the client has a bunch of small buffers. Say 2 buffers each of size 16K. When the first buffer is full, it writes that buffer to the first datanode in the pipeline. The client meanwhile can continue to fill up the remaining buffer(s). The first datanode, on receipt of this buffer, sends it to the next datanode in the pipeline and also writes it to its local disk.

    2. If a datanode fails to write a buffer to its disk, it is reported back to the client. The client removes this datanode from the pipeline and continues to write to the remaining two datanodes. The file in the bad datanode remains in the "tmp" directory.

    3. When the file is closed, the under-replicated blocks will be replicated by the namenode.

    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Oct 10, 2007 at 9:54 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533885 ]

    Doug Cutting commented on HADOOP-1707:
    --------------------------------------

    bq. If a datanode fails to write a buffer to its disk, it is reported back to the client. The client removes this datanode from the pipeline and continues to write to the remaining two datanodes. [ ... ] When the file is closed, the under-replicated blocks will be replicated by the namenode.

    I think the more typical failure mode will be a timeout. I'm also still not sure of the answer to my question: if the first datanode in the pipeline times out, does the write fail, throwing an exception to the client? Or does the client route around the first datanode in the pipeline and continue until all datanodes in the pipeline time out? If so, how can it be sure that the other datanodes have received their copies of prior chunks from the first datanode in the pipeline?

    Also, HADOOP-1927 states that we should fail as soon as any element in the pipeline fails. Do you agree? Currently this would be invisible to clients, since the entire block can be replayed to a new pipeline. But, without a local file, this would force us to fail the write when any element of the pipeline fails. Thoughts?
    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Oct 10, 2007 at 10:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533895 ]

    dhruba borthakur commented on HADOOP-1707:
    ------------------------------------------

    If the primary datanode fails the client can still replay the last-flushed-data buffer to the remaining datanodes. The client has to specify the offset in the block where this buffer contents has to be written. The datanode, given this offset-in-block, can determine whether to do the write or whether the write was already done. The pre-requisite is that a client holds on to a buffer until the write is complete on all known good datanodes.

    Another option would be to say that the application gets an error if the Primary datanode fails. Do you think that this is acceptable?

    I think HADOOP-1927 says that if a non-primary datanode dies, the client should detect it and possibly take appropriate action. Currently the client has no way of knowing whether any secondary datanodes have died.


    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Oct 10, 2007 at 11:21 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533901 ]

    Raghu Angadi commented on HADOOP-1707:
    --------------------------------------


    The jira description only talks about parallel write to datanodes. It does not require removed the temp file on client.

    How about just storing the block at the client like we do now and replay the data if the there is an error? It still allows parallel write to the client. This also does not need any changes/improvements to datanode protocol. Yes, removing the temp file would be better, but it is not worse than current implementation.
    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Oct 10, 2007 at 11:23 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533901 ]

    rangadi edited comment on HADOOP-1707 at 10/10/07 4:21 PM:
    ----------------------------------------------------------------

    The jira description only talks about parallel write to datanodes. It does not require removal of the temp file on client.

    How about just storing the block at the client like we do now and replay the data if the there is an error? It still allows parallel write to the client. This also does not need any changes/improvements to datanode protocol. Yes, removing the temp file would be better, but it is not worse than current implementation.

    was (Author: rangadi):

    The jira description only talks about parallel write to datanodes. It does not require removed the temp file on client.

    How about just storing the block at the client like we do now and replay the data if the there is an error? It still allows parallel write to the client. This also does not need any changes/improvements to datanode protocol. Yes, removing the temp file would be better, but it is not worse than current implementation.
    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Oct 10, 2007 at 11:27 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533903 ]

    dhruba borthakur commented on HADOOP-1707:
    ------------------------------------------

    The local disk cache implementation is similar to creating four replicas of the block and then deleting the excess replica when the block is done. This reduces overall cluster throughput and I would like to analyze ways of getting rid of it.

    I agree that I should change the name-description of this JIRA. Will do.
    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Oct 11, 2007 at 5:10 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534092 ]

    Doug Cutting commented on HADOOP-1707:
    --------------------------------------
    Another option would be to say that the application gets an error if the Primary datanode fails. Do you think that this is acceptable?
    Perhaps, if it only happens rarely. If, e.g., sorts generally complete on 900 nodes with no such failures, then this is probably acceptable. If the primary datanode is localhost, and if secondary failures are survivable, then this may work well enough.

    Otherwise, how do we recover when a datanode in the pipeline becomes unreachable? Will we use per-buffer acks? The primary datanode won't ack a buffer until all datanodes in the pipeline have it? Then if one datanode fails, we could route around it, initialize its copy of the block from one of the survivors, and continue. The acking will effectively add flow-control, which could be a feature, or could slow things. Datanodes may receive the same buffer twice, so buffers will need revision numbers or somesuch.
    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Bug
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Oct 11, 2007 at 9:35 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Issue Type: Improvement (was: Bug)
    DFS client can allow user to write data to the next block while uploading previous block to HDFS
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur

    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Oct 12, 2007 at 9:43 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Fix Version/s: 0.16.0
    Summary: Remove the DFS Client disk-based cache (was: DFS client can allow user to write data to the next block while uploading previous block to HDFS)
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Oct 12, 2007 at 10:24 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534427 ]

    dhruba borthakur commented on HADOOP-1707:
    ------------------------------------------

    I have the following proposal in mind:

    1. The Client uses a small pool of memory buffers per dfs-output stream. Say, 10 buffers of size 64K each.
    2. A write to the output stream actually copies the user data into one of the buffers, if available. Otherwise the user-write blocks.
    3. A separate thread (one per output stream), sends buffers that are full. Each buffer has metadata that contains a sequence number (locally generated on the client) , the length of the buffer and its offset in this block.
    4. Another thread(one per output stream) process incoming responses. The incoming response has the sequence number of the buffer that the datanode had processed. The client removes that buffer from its queue.
    5. The client gets an exception if the primary datanode fails. If a secondary datanode fails, the primary informs the client about this event.
    6. In any datanodes fail, the client removes it from the pipeline and resends all pending buffers to all known good datanodes.
    7. A target datanode remembers the last sequencenumber that it has previously processed. It forwards the buffer to the next datanode in the pipeline. If the datanode receives a buffer that it has not processed earlier, it writes it to local disk. When the response arrives, it forwards the response back to the client.




    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Oct 15, 2007 at 7:48 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534952 ]

    Doug Cutting commented on HADOOP-1707:
    --------------------------------------
    The client gets an exception if the primary datanode fails.
    Why can't it simply replace the primary with one of the secondary datanodes and proceed?
    If a secondary datanode fails, the primary informs the client about this event.
    Since a secondary will typically fail by timing out, the timeout used between the client and the primary must be longer than that used between the primary and secondary, so that the client waits long enough to hear about a failed secondary. And the timeout used between the application and the client must be longer yet. Right? Perhaps we should make all these timeouts proportional to a single configuration parameter, the application timeout?

    If we wish to ensure that blocks are sufficiently replicated, then we'll block on file close, right?

    Overall, this sounds like an approach worth trying.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Oct 30, 2007 at 4:22 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538657 ]

    dhruba borthakur commented on HADOOP-1707:
    ------------------------------------------

    I agree that that timeout issue does not have a very elegant solution. Here is a new proposal.

    The Client
    --------------
    1. The Client uses a small pool of memory buffers per dfs-output stream. Say, 10 buffers of size 64K each.
    2. A write to the output stream actually copies the user data into one of the buffers, if available. Otherwise the user-write blocks.
    3. A separate thread (one per output stream), sends buffers that are full. Each buffer has metadata that contains a sequence number (locally generated on the client) , the length of the buffer and its offset in this block.
    4. Another thread(one per output stream) process incoming responses. The incoming response has the sequence number of the buffer that the datanode had processed. The client removes that buffer from its queue.

    The Primary Datanode
    ------------------------------
    The primary datanode has two threads per stream. The first thread processes incoming packets from the client, writes them to the downstream datanode and writes them to local disk. The second thread processes responses from downstream datanodes and forwards them back to the client.

    This means that the client gets back an ack only when the packet is persisted on all datanodes. In the future this can be changed so that the client gets an ack when the data is persisted in dfs.replication.min number of datanodes.

    In case the primary datanode encounters an exception while writing to the downstream datanode, it declares the block as bad. It removes the immediate downstream datanode from the pipeline. It makes an RPC to the namenode to abandon the current blockId and*replace* the block id with a new one. It then establishes a new pipeline using the new blockid using the remaining datanodes. It then copies all the data from the local temporary block file to the downstream datanodes using the new blockId.

    The Secondary Datanodes
    ------------------------------------
    The Secondary datanode has two threads per stream. The first thread processes incoming packets from the upstream datanode, writes them to the downstream datanode and writes them to local disk. The second thread processes responses from downstream datanodes and forwards them back to the upstream datanode.

    Each secondary datanode sends its response as well forwards the response of all downstream datanodes.


    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Oct 30, 2007 at 5:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538867 ]

    Doug Cutting commented on HADOOP-1707:
    --------------------------------------

    This still appears to have the cascading timeout issue, no? Each stage in the pipeline must have a smaller timeout than the prior stage or else the whole pipeline will fail when any node fails. In particular, the client must use a much larger timeout, since it must permit the primary to potentially replay the entire block downstream. Perhaps there can be multiple kinds of acks, some which just indicate that the primary is still alive and others that indicate that replication is complete? (Acks might include the current level of replication.) That might help distinguish the cases where the primary has actually gone down from those where it is still doing productive work. Then one timeout could be used for communications, and a substantially longer one for awaiting replication.

    I also wonder whether, instead of having so many threads, we might implement this with async i/o. Much of the processing seems simple enough that maintaining a state object for each file being written and using a single thread that selects on sockets and then updates the state might be more efficient. Perhaps it will be simpler to write these with threads, then convert them to async?

    We discussed offline last week a different approach from what you've described here. In that, acks would only signal that the immediately downstream node had written the data, not all downstream nodes. Only at block end or flush would it check that sufficient replicas exist, with a different command. Why have you abandoned this plan?

    An intermediate approach might be to use buffer pools on each datanode in the pipeline. Each would write the buffer locally and queue it to be written downstream. The buffer would only be returned to the pool when both writes complete. A datanode could block when no buffers are available. That might improve throughput.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Oct 31, 2007 at 8:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer.patch

    This is a *very very preliminary* patch that packetize writes from clients. It does not do any error recovery at all.

    We discussed a proposal where datanodes do local recovery. If a datanode fails, the datanode immediately preceeding it will recreate the pipeline by ignoring the one that failed and connecting directly to the datanode that followed the one that failed. This approach has the disadvantage that in the case of multiple failures, two upstream datanodes might be in recovery and both of then might try to resend the block to a downstream datanode simultaneously. This might be a difficult case to handle.

    Also, the earlier proposal generated an exception to the client if the primary datanode fails. This might be a commonly occuring case. If we would want to avoid this problem, then the client has to do Recovery (over and above any datanodes doing local recovery). In this case, maybe it is better to drive the entire recovery from a single point : the client.

    The cascading timeouts issue has to be handled somehow. Your proposal of setting different timeouts for datanodes in the pipeline will work but it might be a little tricky to implement and debug. Another approach would be for each datanode to expose a new "ping" RPC. The Client, when it has to recover, "pings" each Datanode and determines which of them are not responding. This seems to work, isn't it?




    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Nov 2, 2007 at 5:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer2.patch
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Nov 6, 2007 at 12:13 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12540308 ]

    dhruba borthakur commented on HADOOP-1707:
    ------------------------------------------

    In the current trunk, the first datanode in the pipeline sets a timeout of 2 minutes. The second datanode sets a timeout of 1 minute, and so on. If a datanode does not receive a response from a downstream datanode within this timeout period, it declared the downsteam data as dead.

    In this patch for removing the client-side-disk buffer, the connection between datanodes in the pipeline could remain open for extended periods of time, especially for clients that are producing output slowly. I propose that we change the timeouts to behave as follows:

    1. Each datanode in the pipeline has the same timeout of 1 minute. If a datanode does not receive a response from a downstream datanode in 1 minute, it declares the downstream datanode as dead.
    2. Each datanode sends a heartbeat message to the upstream datanode once every half-timeout-period.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Nov 6, 2007 at 12:24 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12540311 ]

    Raghu Angadi commented on HADOOP-1707:
    --------------------------------------
    2. Each datanode sends a heartbeat message to the upstream datanode once every half-timeout-period.
    It might be better to call this 'KeepAlive' since it is per connection and avoids confusion with DataNode heartbeat.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Nov 13, 2007 at 9:09 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer6.patch

    This patch removes the client side disk buffer.

    1. FSConstants.java : Bumped up DATA_TRANSFER_VERSION.
    2. Daemon.java: Added a ThreadGroup to the Daemon class. All worker threads that process data transfers belong to this group. The shutdown of a datnode waits for the entire threadgroup to exit. Prior to this change, a datanode shutdown did not wait for the data transfer threads to exit.
    3. FSNamesystem.java: Allows a zero size file to have no blocks associated with it.
    4. DataChecksum.java: A utility method to return the size of a checksum header.
    5. FSDataset.java: The ongoingCreates data structure remembers the thread that is currently writing to a block. The writeToBlock() method (when the recovery flag is set) terminates any existing threads that might have been writing to a block before allowing a new thread to write to the same block.
    6. FSDataOutputStream.java: The unit test needed to extract the pipeline associated with a block. This is facilitated by exposing a new public API called getWrappedStream() that returns the underlying DFSOutputStream object.
    7. MiniDFSCluster.java: Allows stopping a particular datanode.
    8. DFSClient.java/DataNode.java: User data is transferred in the form of packets. Each Packet requires an ack from all datanodes. The DFSClient drives the entire recovery strategy. A keepalive is sent every READ_TIMEOUT/2 period on the response socket channel. Each packet is 64K in size and the client has a sliding window of 5MB per stream.
    9. TestDatanodeDeath.java: A unit test to trigger datanode deaths and DFSClient recovery.



    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Nov 16, 2007 at 12:11 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: DataTransferProtocol.doc

    A document that describes the streaming protocol used to transfer data among clients and datanodes.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, DataTransferProtocol.doc


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Nov 16, 2007 at 12:15 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: DataTransferProtocol.html

    The Data transfer Protocll document in html format.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Nov 19, 2007 at 7:06 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12543636 ]

    Doug Cutting commented on HADOOP-1707:
    --------------------------------------

    This protocol document is great to have! Can we get it converted into forrest-compatible XML and included in a reference section of the documentation when this patch is committed?
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Nov 21, 2007 at 10:13 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer7.patch

    Merged patch with latest trunk.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Nov 27, 2007 at 7:33 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer8.patch

    Fixed two bugs that were exposed while running random writer on a 100 node cluster.

    1. The code was such that it was waiting for the ResponseThread to exit while holding the lock on dataQueue. This caused a deadlock.

    2. The DFSClient was sending the packet to the first datanode before it inserted the packet into the ackQueue. Now, if the response from the datanode arrives before the DFSClient could enqueue the packet into the ackQueue it triggered an error. This situation is now avoided because the DFSClient first inserts the packet into the ackQueue before sending the packet to the datanode.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Nov 29, 2007 at 10:06 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546917 ]

    Konstantin Shvachko commented on HADOOP-1707:
    ---------------------------------------------

    Since you have just encountered that.
    The same problem will potentially be in the following 3 methods
    - nextBlockOutputStream()
    - locateFollowingBlock()
    - DFSOutputStream.close()

    where the client sleeps under a lock. In general a thread should wait() instead of sleep() under a lock.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Dec 3, 2007 at 11:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer9.patch

    Make patch compile with JDK 1.5
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Dec 5, 2007 at 7:44 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer10.patch

    Fixed a bug that was causing the client to hang if all datanodes in the pipeline reported an error. This situation was triggered while testing this patch on a 500 node cluster.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Dec 5, 2007 at 11:55 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer11.patch

    Merged with latest trunk.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Dec 6, 2007 at 3:40 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548911 ]

    Konstantin Shvachko commented on HADOOP-1707:
    ---------------------------------------------

    I think this patch has been tested quite thoroughly, and I don't see any algorithmic flaws in it.
    The logic is fairly complicated though, so imo
    # we need better documentation either in JavaDoc or at least in Jira.
    # it would be good if you could extract common actions for the client and the data-node into
    separate classes, not inner ones.

    =========== DFSClient.java
    - DFSClient: 4 unused variables, members.
    - DFSOutputStream.lb should be local variable.
    - processDatanodeError() and DFSOutputStream.close() have common code.
    - BlockReader.readChunk()
    {code}
    07/12/04 18:36:22 INFO fs.FSInputChecker: DFSClient readChunk got seqno 14 offsetInBlock 7168
    {code}
    Should be DEBUG.
    - More comments: What is e.g. dataQueue, ackQueue, bytesCurBlock?
    - Some new members in DFSOutputStream can be calculated from the other.
    No need to store them all. See e.g.
    {code}
    private int packetSize = 0;
    private int chunksPerPacket = 0;
    private int chunksPerBlock = 0;
    private int chunkSize = 0;
    {code}
    - In the line below "8" should be defined as a constant. Otherwise, the meaning of that is not clear.
    {code}
    chunkSize = bytesPerChecksum + 8; // user data + checksum
    {code}
    - currentPacket should be a local variable of writeChunk()
    - The 4 in the code snippet below looks misterious:
    {code}
    if (len + cklen + 4 > chunkSize) {
    {code}
    - why start ResponseProcessor in processDatanodeError()
    - some methods should be moved into new inner classes, like
    nextBlockOutputStream() should be a part of DataStreamer
    - Packet should be factored out to a separate class (named probably DataPacket).
    It should have serialization/deserialization methods for packet header, which should
    be reused in DFSClient and DataNodes for consistency in data transfer.
    It also should have methods readPacker() and writePacket()

    =========== DataNode.java
    - import org.apache.hadoop.io.Text; is redundant.
    - My Eclipse shows 5 variables that are "never read".
    - Rather than using "4" on several occasions a constant should be defined
    {code}
    SIZE_OF_INTEGER = Integer.SIZE / Byte.SIZE;
    {code}
    and used whenever required.
    - lastDataNodeRun() should not be public

    =========== FSDataset.java
    - writeToBlock(): These are two searches in a map instead of one.
    {code}
    if (ongoingCreates.containsKey(b)) {
    ActiveFile activeFile = ongoingCreates.get(b);
    {code}
    - unfinalizeBlock() I kinda find the name funny.

    =========== General
    - Convert comments like // .......... to JavaDoc /** ... */ style comments
    when used as method or class headers even if they are private.
    - Formatting. Tabs should be replaced by 2 spaces. Eg: ResponseProcessor.run(), DataStreamer.run()
    - Formatting. Long lines.

    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Dec 7, 2007 at 8:33 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer12.patch

    Setting the TCP buffer size to 64K (instead of the default of 8K) and setting tcpNoDelay() on the response socket improves performance by about 5%.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Dec 13, 2007 at 1:11 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer14.patch

    Merged with latest trunk. Also fixed a bug where an InterruptedException was being consumed silently, thus leading to long delays in timing-outs.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Mukund Madhugiri (JIRA) at Dec 14, 2007 at 11:32 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551977 ]

    Mukund Madhugiri commented on HADOOP-1707:
    ------------------------------------------

    I ran sort benchmark on 500 nodes and here is the data:

    trunk:
    * random writer: 0.405 hrs
    * sort: 1.508 hrs
    * sort validation: 0.333 hrs

    trunk + patch:
    * random writer: 0.534 hrs
    * sort: 1.808 hrs
    * sort validation: 0.408 hrs

    During the sort reduce phase, I observed some errors, but the sort eventually succeeded:
    * java.io.IOException: Could not get block locations. Aborting...
    * java.io.IOException: All datanodes are bad. Aborting...

    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Dec 14, 2007 at 11:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551982 ]

    dhruba borthakur commented on HADOOP-1707:
    ------------------------------------------

    Thanks Mukund. The errors are causingt he numbers to go up. i will dig into the logs and code to find the cause of the errors.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Dec 19, 2007 at 7:59 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer15.patch

    Found a race condition that was causing the client to close the connection before the datanodes had a chance to process the end-of-packet. This caused the datanode to treat it as an error condition, thus causing the client to do error recovery and re-send the outstanding packets to the remaining good datanodes. This was causing performance regression.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Dec 29, 2007 at 9:11 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer16.patch

    This patch fixes another performance degradation shown by the earlier patch. There was a race condition whereby an intermediate datanode in the pipeline was ignoring the response sent from the downstream datanode, always forwarding an "error" to the client.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 7, 2008 at 7:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer17.patch

    Merged patch with latest trunk. This patch has additional debugging that might help in getting to the cause of the performance degradation seen in one earlier run.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 8, 2008 at 5:58 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer18.patch

    The earlier patch had LOG levels set to debug by default.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 10, 2008 at 11:44 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer19.patch

    A sort on a 500 node cluster detected a data corruption. The code in datanode had a race whereby a block confirmation of a block did not have the correct size of the block. This caused the namenode to think that the block is shorter in length.

    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Mukund Madhugiri (JIRA) at Jan 11, 2008 at 5:43 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558041#action_12558041 ]

    Mukund Madhugiri commented on HADOOP-1707:
    ------------------------------------------

    Running on a 100 node cluster, with the patch clientDiskBuffer19.patch, the sort benchmark showed these results:
    *100 nodes*|*trunk*|*trunk + patch*|
    randomWriter (hrs)|0.44|0.45|
    sort (hrs)|1.03|1|
    sortValidation (hrs)|0.39|0.3|
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 11, 2008 at 5:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Status: Patch Available (was: Open)
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jan 12, 2008 at 4:11 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558245#action_12558245 ]

    Hadoop QA commented on HADOOP-1707:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12372949/clientDiskBuffer19.patch
    against trunk revision r611385.

    @author +1. The patch does not contain any @author tags.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new compiler warnings.

    findbugs -1. The patch appears to introduce 10 new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests -1. The patch failed contrib unit tests.

    Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1555/testReport/
    Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1555/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1555/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1555/console

    This message is automatically generated.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 13, 2008 at 7:59 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Status: Open (was: Patch Available)

    Cancel patch to fix findbugs warnings.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 13, 2008 at 8:01 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer20.patch

    Fix findbugs warnings. There are two findbugs warnings (about having two locks while invoking wait()) that are valid scenarios.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer20.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 13, 2008 at 8:03 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Status: Patch Available (was: Open)
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer20.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jan 13, 2008 at 10:31 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558378#action_12558378 ]

    Hadoop QA commented on HADOOP-1707:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12373048/clientDiskBuffer20.patch
    against trunk revision r611537.

    @author +1. The patch does not contain any @author tags.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new compiler warnings.

    findbugs -1. The patch appears to introduce 2 new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests -1. The patch failed contrib unit tests.

    Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1569/testReport/
    Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1569/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1569/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1569/console

    This message is automatically generated.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer20.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 14, 2008 at 8:57 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Status: Open (was: Patch Available)
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer20.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 14, 2008 at 8:59 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Attachment: clientDiskBuffer21.patch

    More findbugs issues and test failures.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer20.patch, clientDiskBuffer21.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 14, 2008 at 8:59 am
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-1707:
    -------------------------------------

    Status: Patch Available (was: Open)
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer20.patch, clientDiskBuffer21.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jan 14, 2008 at 5:56 pm
    [ https://issues.apache.org/jira/browse/HADOOP-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558696#action_12558696 ]

    Hadoop QA commented on HADOOP-1707:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12373081/clientDiskBuffer21.patch
    against trunk revision r611760.

    @author +1. The patch does not contain any @author tags.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new compiler warnings.

    findbugs -1. The patch appears to introduce 2 new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests -1. The patch failed contrib unit tests.

    Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1583/testReport/
    Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1583/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1583/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1583/console

    This message is automatically generated.
    Remove the DFS Client disk-based cache
    --------------------------------------

    Key: HADOOP-1707
    URL: https://issues.apache.org/jira/browse/HADOOP-1707
    Project: Hadoop
    Issue Type: Improvement
    Components: dfs
    Reporter: dhruba borthakur
    Assignee: dhruba borthakur
    Fix For: 0.16.0

    Attachments: clientDiskBuffer.patch, clientDiskBuffer10.patch, clientDiskBuffer11.patch, clientDiskBuffer12.patch, clientDiskBuffer14.patch, clientDiskBuffer15.patch, clientDiskBuffer16.patch, clientDiskBuffer17.patch, clientDiskBuffer18.patch, clientDiskBuffer19.patch, clientDiskBuffer2.patch, clientDiskBuffer20.patch, clientDiskBuffer21.patch, clientDiskBuffer6.patch, clientDiskBuffer7.patch, clientDiskBuffer8.patch, clientDiskBuffer9.patch, DataTransferProtocol.doc, DataTransferProtocol.html


    The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
    A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedAug 11, '07 at 12:40a
activeJan 17, '08 at 8:04p
posts89
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase