We are using capacity scheduler to schedule resources among different queues
for 1 user (hadoop) only. We have set the queues to have equal share of the
resources. However, when 1st task starts in the first queue and is consuming
all the resources, the 2nd task starts in the 2nd queue will be starved from
reducer until the first task finished. A lot of processing is being stuck
when a large query is executing.
We are using 0.20.2 hive in amazon aws. We tried to use Fair Scheduler
before but it gives an error when the mapper gives no output (which is fine
in our use cases).
Anyone can give us some advice?