[HBase-dev] [jira] Created: (HBASE-1672) Map tasks not local to RS

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amandeep Khurana updated HBASE-1672:
------------------------------------

Component/s: regionserver
master
mapred

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amandeep Khurana updated HBASE-1672:
------------------------------------

Description:
The number of data local map tasks while scanning a table is only about 10% of the total map tasks...
My table had 280 regions and 13M records... The number of map tasks in the scan job were equal to the number of regions (280). Only 25 of them were data local tasks.

was:The number of data local map tasks is only about 10% of the total map tasks...

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-1672:
---------------------------------

Affects Version/s: 0.20.0
Fix Version/s: 0.19.4
0.20.0

Bringing in to 0.20.0 so someone can verify whether this works in trunk or not. I can do it later this week if no one else does.

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733256#action_12733256 ]

Jonathan Gray commented on HBASE-1672:
--------------------------------------

This needs to be tested on trunk, thought we had fixed this.

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733651#action_12733651 ]

Jean-Daniel Cryans commented on HBASE-1672:
-------------------------------------------

We already do this inside TableInputFormatBase:

{code}
String regionLocation = table.getRegionLocation(startKeys[startPos]).
getServerAddress().getHostname();
splits[i] = new TableSplit(this.table.getTableName(),
startKeys[startPos], ((i + 1) < realNumSplits) ? startKeys[lastPos]:
HConstants.EMPTY_START_ROW, regionLocation);
LOG.info("split: " + i + "->" + splits[i]);
{code}

I don't know if we can do anything more than that. One difference in HBase compared to mapred on HDFS is that a region is only on one node, not 3 which is the default replication factor. So being able to get the right map task on the right RS at the right moment may be difficult for the JobTracker.

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733774#action_12733774 ]

stack commented on HBASE-1672:
------------------------------

So, what is the indicator in the MR UI measuring? TT+DN locality? Or is it TT+RS? If the latter, and we are only 10% of the time doing TT mapper local to the region hosting server, then our TT+RS locality would seem to be broke -- or ineffective (either would be good to know).

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-1672.
--------------------------

Resolution: Cannot Reproduce

I ran a rowcounter job against a 100 region table of ~20M rows. Cluster was small (4 regionservers). Tasktrackers ran beside the RS. Every task was scheduled on the TT that was local to the RS ("Input Split Locations" always had same value as "Machine" in the taskdetails page).

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735207#action_12735207 ]

Jonathan Gray commented on HBASE-1672:
--------------------------------------

Thank you for researching, stack.

Next week we'll have a ton of MR running on trunk so will report if we find anything strange.

[ https://issues.apache.org/jira/browse/HBASE-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735302#action_12735302 ]

Amandeep Khurana commented on HBASE-1672:
-----------------------------------------

I had this issue in 0.19. Not facing the problem in 0.20 though.