[
https://issues.apache.org/jira/browse/HADOOP-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618896#action_12618896 ]
Chris Douglas commented on HADOOP-3062:
---------------------------------------
The analysis should leverage HADOOP-3719, so this issue should cover the log4j appender emitting the HDFS and shuffling data. There are a few open questions and arguable assumptions:
* Should this count bytes successfully transferred separately from failed transfers? Should failed transfers be logged at all?
* The header/metadata/etc. traffic is assumed to be a negligible fraction of the total network traffic and irrelevant to the analysis for a particular job. The overall network utilization is also best measured using standard monitoring utilities that don't require any knowledge of Hadoop. This will focus on tracking block traffic over HDFS (reads, writes, replications) and map output fetched during the shuffle, only.
* For local reads, the source and destination IP will match. This should be sufficient to detect and discard during analysis of network traffic, but will not be sufficient to account for all reads from the local disk (counters and job history are likely better tools for this).
* Accounting for topology (to break down by racks, etc.) is best deferred to the analysis. Logging changes in topology would also be helpful, though I don't know whether Hadoop has sufficient information to do this in the general case.
* If job information is available (in the shuffle), should it be included in the entry? Doing this for HDFS is non-trivial, but would be invaluable to the analysis. I'm not certain how to do this, yet. Of course, replications and rebalancing won't include this, and HDFS reads prior to job submission (and all other traffic from JobClient) will likely be orphaned, as well.
* Should this include start/end entries so one can infer how long the transfer took?
* What about DistributedCache? Can it be ignored as part of the job setup, which is already omitted?
In general, the format will follow:
{noformat}
<log4j schema including timestamp, etc.> source: <src IP>, destination: <dst IP>, bytes: <bytes>, operation: <op enum>[, taskid: <TaskID>]
{noformat}
Where {{<(src|dst) IP>}} is the IP address of the source and destination nodes, {{<bytes>}} is a long, and {{<op enum>}} is one of {{HDFS_READ}}, {{HDFS_WRITE}}, {{HDFS_COPY}}, and {{MAPRED_SHUFFLE}}. {{HDFS_REPLACE}} should be redundant if {{HDFS_COPY}} is recorded (I think). The rebalancing traffic isn't relevant to job analysis, but if one is including sufficient information to determine the duration of each transfer it may be interesting. The TaskID should be sufficient, but one could argue that including the JobID would be useful as a point to join on.
Thoughts?
Need to capture the metrics for the network ios generate by dfs reads/writes and map/reduce shuffling and break them down by racks
------------------------------------------------------------------------------------------------------------------------------------
Key: HADOOP-3062
URL:
https://issues.apache.org/jira/browse/HADOOP-3062Project: Hadoop Core
Issue Type: Improvement
Components: metrics
Reporter: Runping Qi
In order to better understand the relationship between hadoop performance and the network bandwidth, we need to know
what the aggregated traffic data in a cluster and its breakdown by racks. With these data, we can determine whether the network
bandwidth is the bottleneck when certain jobs are running on a cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
