FAQ
HI Guys

We configured fair scheduler with cdh4, Fair scheduler not work properly.
Map Task Capacity = 1380
Reduce Task Capacity = 720

We create two users tech and research, we configured equal weight 1 But, I
stared job in research user mapper will not allocated why?
please guide me guys.

<?xml version="1.0"?>
<allocations>
<pool name="tech">
<minMaps>5</minMaps>
<minReduces>5</minReduces>
<maxRunningJobs>30</maxRunningJobs>
<weight>1.0</weight>
</pool>
<pool name="research">
<minMaps>5</minMaps>
<minReduces>5</minReduces>
<maxRunningJobs>30</maxRunningJobs>
<weight>1.0</weight>
</pool>
</allocations>

Note: we have tested with Hadoop Stream job.

Fair Scheduler Administration Pools PoolRunning JobsMap TasksReduce
TasksScheduling
Mode Min ShareMax ShareRunningFair ShareMin ShareMax ShareRunningFair Share
research15-90690.05-00.0FAIR tech35-1266690.05-2424.0FAIR default00-00.00-0
0.0FAIR Running Jobs SubmittedJobIDUserNamePoolPriorityMap TasksReduce Tasks
FinishedRunningFair ShareWeightFinishedRunningFair ShareWeight Jan 16, 08:51
job_201301071639_2118<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2118>
tech streamjob5335328828469969152.jar 30466 / 53724583313.5 1.0 0 / 2400.0
1.0 Jan 16, 09:56
job_201301071639_2147<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2147>
research streamjob8832181817213433660.jar 4175 / 958190690.0 1.0 0 / 240
0.0 1.0 Jan 16, 10:01
job_201301071639_2148<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2148>
tech streamjob8773848575543653055.jar 1842 / 15484620313.5 1.0 0 / 2400.0
1.0 Jan 16, 10:08
job_201301071639_2155<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2155>
tech counterfactualsim-prod.eagle-EagleDepthSignalDisabled-prod.eagle 387
/ 4506363.0 1.0 0 / 242424.0 1.0

--

Search Discussions

  • Justin Workman at Jan 16, 2013 at 3:45 pm
    Looks like weight for both pools is equal and all map slots are used.
    Therefore I don't believe anyone has priority for the next slots. Try
    setting research weight to 2. This should allow research to take slots as
    tech released them.

    Sent from my iPhone

    On Jan 16, 2013, at 8:26 AM, Dhanasekaran Anbalagan wrote:

    HI Guys

    We configured fair scheduler with cdh4, Fair scheduler not work properly.
    Map Task Capacity = 1380
    Reduce Task Capacity = 720

    We create two users tech and research, we configured equal weight 1 But, I
    stared job in research user mapper will not allocated why?
    please guide me guys.

    <?xml version="1.0"?>
    <allocations>
    <pool name="tech">
    <minMaps>5</minMaps>
    <minReduces>5</minReduces>
    <maxRunningJobs>30</maxRunningJobs>
    <weight>1.0</weight>
    </pool>
    <pool name="research">
    <minMaps>5</minMaps>
    <minReduces>5</minReduces>
    <maxRunningJobs>30</maxRunningJobs>
    <weight>1.0</weight>
    </pool>
    </allocations>

    Note: we have tested with Hadoop Stream job.

    Fair Scheduler Administration Pools PoolRunning JobsMap TasksReduce
    TasksScheduling
    Mode Min ShareMax ShareRunningFair ShareMin ShareMax ShareRunningFair Share
    research15-90690.05-00.0FAIR tech35-1266690.05-2424.0FAIR default00-00.00-0
    0.0FAIR Running Jobs SubmittedJobIDUserNamePoolPriorityMap TasksReduce Tasks
    FinishedRunningFair ShareWeightFinishedRunningFair ShareWeight Jan 16, 08:51
    job_201301071639_2118<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2118>
    tech streamjob5335328828469969152.jar 30466 / 53724583313.5 1.0 0 / 2400.0
    1.0 Jan 16, 09:56
    job_201301071639_2147<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2147>
    research streamjob8832181817213433660.jar 4175 / 958190690.0 1.0 0 / 240
    0.0 1.0 Jan 16, 10:01
    job_201301071639_2148<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2148>
    tech streamjob8773848575543653055.jar 1842 / 15484620313.5 1.0 0 / 2400.0
    1.0 Jan 16, 10:08
    job_201301071639_2155<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2155>
    tech counterfactualsim-prod.eagle-EagleDepthSignalDisabled-prod.eagle 387
    / 4506363.0 1.0 0 / 242424.0 1.0

    --

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedJan 16, '13 at 3:26p
activeJan 16, '13 at 3:45p
posts2
users2
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase