Grokbase Groups Hive user May 2011
FAQ
My cluster went corrupt-mode. I wiped it and deleted the Hive metastore and
started over. In the process, I did a "yum upgrade" which probably took me
from CDH3b4 to CDH3u0. Now everytime I submit a Hive query of complexity
requiring a map/reduce job*, I get this error:

2011-05-06 18:39:14,533 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201104081532_0509 with errors
java.lang.RuntimeException: Error while reading from task log url
at
org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
at
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
.....[snip].....
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for
URL:
http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at java.net.URL.openStream(URL.java:1010)
at
org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
... 16 more
Ended Job = job_201104081532_0509 with exception
'java.lang.RuntimeException(Error while reading from task log url)'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MapRedTask

It seems to me the key point in here is this:

Server returned HTTP response code: 400 for URL:
http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true

So I submitted that URL to my web browser, which said:

Problem accessing /tasklog. Reason: Argument attemptid is required

Does anyone have clues what this means?

--
Tim Ellis
Riot Games
* That is to say, simple queries like "select * from table limit 10" return
results just fine.

Search Discussions

  • Time Less at Jun 28, 2011 at 12:08 am
    Today I'm getting this error again. A Google search brought me back to...
    you guessed it... my own post. But this time no HDFS corruption. Bounced all
    services, namenode, jobtracker, datanodes, tasktrackers. Still same error.
    Here's what it looks like:

    *Fsck Output:
    *FSCK ended at Mon Jun 27 17:02:07 PDT 2011 in 818 milliseconds
    The filesystem under path '/' is HEALTHY

    *Hive Query Output:
    *-bash-3.2$ hive -e "select * from air_client_logs where loglevel =
    '[FATAL]'"
    Hive history file=/tmp/hdfs/hive_job_log_hdfs_201106271702_317571839.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201106271658_0002, Tracking URL =
    http://hadooptest1:50030/jobdetails.jsp?jobid=job_201106271658_0002
    Kill Command = /usr/lib/hadoop/bin/hadoop job
    -Dmapred.job.tracker=hadooptest1:54311 -kill job_201106271658_0002
    2011-06-27 17:02:39,874 Stage-1 map = 0%, reduce = 0%
    2011-06-27 17:03:02,111 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201106271658_0002 with errors
    java.lang.RuntimeException: Error while reading from task log url
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
    at
    org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:425)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    Caused by: java.io.IOException: Server returned HTTP response code: 400 for
    URL:
    http://hadooptest14:50060/tasklog?taskid=attempt_201106271658_0002_m_000008_0&all=true
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    at java.net.URL.openStream(URL.java:1010)
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
    ... 16 more
    Ended Job = job_201106271658_0002 with exception
    'java.lang.RuntimeException(Error while reading from task log url)'
    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    Again, trying to go to "
    http://hadooptest14:50060/tasklog?taskid=attempt_201106271658_0002_m_000008_0&all=true"
    returns that argument attemptid error.

    What am I doing wrong here? I appear to keep doing it, whatever it is.

    On Fri, May 6, 2011 at 6:47 PM, Time Less wrote:

    My cluster went corrupt-mode. I wiped it and deleted the Hive metastore and
    started over. In the process, I did a "yum upgrade" which probably took me
    from CDH3b4 to CDH3u0. Now everytime I submit a Hive query of complexity
    requiring a map/reduce job*, I get this error:

    2011-05-06 18:39:14,533 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201104081532_0509 with errors
    java.lang.RuntimeException: Error while reading from task log url
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
    at
    org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
    .....[snip].....
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    Caused by: java.io.IOException: Server returned HTTP response code: 400 for
    URL:
    http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    at java.net.URL.openStream(URL.java:1010)
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
    ... 16 more
    Ended Job = job_201104081532_0509 with exception
    'java.lang.RuntimeException(Error while reading from task log url)'
    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    It seems to me the key point in here is this:

    Server returned HTTP response code: 400 for URL:
    http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true

    So I submitted that URL to my web browser, which said:

    Problem accessing /tasklog. Reason: Argument attemptid is required

    Does anyone have clues what this means?

    --
    Tim Ellis
    Riot Games
    * That is to say, simple queries like "select * from table limit 10" return
    results just fine.

    --
    Tim
  • Sumanth V at Jun 28, 2011 at 12:14 am
    You are hitting this bug - https://issues.apache.org/jira/browse/HIVE-1579
    I consistently hit this bug for one of the Hive queries.


    Sumanth


    On Mon, Jun 27, 2011 at 5:08 PM, Time Less wrote:

    Today I'm getting this error again. A Google search brought me back to...
    you guessed it... my own post. But this time no HDFS corruption. Bounced all
    services, namenode, jobtracker, datanodes, tasktrackers. Still same error.
    Here's what it looks like:

    *Fsck Output:
    *FSCK ended at Mon Jun 27 17:02:07 PDT 2011 in 818 milliseconds
    The filesystem under path '/' is HEALTHY

    *Hive Query Output:
    *-bash-3.2$ hive -e "select * from air_client_logs where loglevel =
    '[FATAL]'"
    Hive history file=/tmp/hdfs/hive_job_log_hdfs_201106271702_317571839.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201106271658_0002, Tracking URL =
    http://hadooptest1:50030/jobdetails.jsp?jobid=job_201106271658_0002
    Kill Command = /usr/lib/hadoop/bin/hadoop job
    -Dmapred.job.tracker=hadooptest1:54311 -kill job_201106271658_0002
    2011-06-27 17:02:39,874 Stage-1 map = 0%, reduce = 0%
    2011-06-27 17:03:02,111 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201106271658_0002 with errors

    java.lang.RuntimeException: Error while reading from task log url
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
    at
    org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:425)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    Caused by: java.io.IOException: Server returned HTTP response code: 400 for
    URL:
    http://hadooptest14:50060/tasklog?taskid=attempt_201106271658_0002_m_000008_0&all=true

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    at java.net.URL.openStream(URL.java:1010)
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
    ... 16 more
    Ended Job = job_201106271658_0002 with exception
    'java.lang.RuntimeException(Error while reading from task log url)'

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    Again, trying to go to "
    http://hadooptest14:50060/tasklog?taskid=attempt_201106271658_0002_m_000008_0&all=true"
    returns that argument attemptid error.

    What am I doing wrong here? I appear to keep doing it, whatever it is.


    On Fri, May 6, 2011 at 6:47 PM, Time Less wrote:

    My cluster went corrupt-mode. I wiped it and deleted the Hive metastore
    and started over. In the process, I did a "yum upgrade" which probably took
    me from CDH3b4 to CDH3u0. Now everytime I submit a Hive query of complexity
    requiring a map/reduce job*, I get this error:

    2011-05-06 18:39:14,533 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201104081532_0509 with errors
    java.lang.RuntimeException: Error while reading from task log url
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
    at
    org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
    .....[snip].....
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    Caused by: java.io.IOException: Server returned HTTP response code: 400
    for URL:
    http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    at java.net.URL.openStream(URL.java:1010)
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
    ... 16 more
    Ended Job = job_201104081532_0509 with exception
    'java.lang.RuntimeException(Error while reading from task log url)'
    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    It seems to me the key point in here is this:

    Server returned HTTP response code: 400 for URL:
    http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true

    So I submitted that URL to my web browser, which said:

    Problem accessing /tasklog. Reason: Argument attemptid is required

    Does anyone have clues what this means?

    --
    Tim Ellis
    Riot Games
    * That is to say, simple queries like "select * from table limit 10"
    return results just fine.

    --
    Tim
  • Time Less at Jun 28, 2011 at 5:41 pm
    I'm having a hard time interpreting the JIRA - it seems to be saying that
    Hive is passing an incorrect "mapred.child.java.opts=XmxNNNM" parameter,
    missing the - ... Is that correct? I could dig into the Hive source code and
    look for this bit if that's so.

    I admit, I'm a Java newbie, so I'm not entirely certain I understand how
    Hive/Hadoop interact, but if Hive is passing incorrect memory arguments to
    Hadoop tasks, and they're OOMing, then it seems the sort of fix I could
    undertake successfully.

    On Mon, Jun 27, 2011 at 5:13 PM, Sumanth V wrote:

    You are hitting this bug - https://issues.apache.org/jira/browse/HIVE-1579
    I consistently hit this bug for one of the Hive queries.


    Sumanth


    On Mon, Jun 27, 2011 at 5:08 PM, Time Less wrote:

    Today I'm getting this error again. A Google search brought me back to...
    you guessed it... my own post. But this time no HDFS corruption. Bounced all
    services, namenode, jobtracker, datanodes, tasktrackers. Still same error.
    Here's what it looks like:

    *Fsck Output:
    *FSCK ended at Mon Jun 27 17:02:07 PDT 2011 in 818 milliseconds
    The filesystem under path '/' is HEALTHY

    *Hive Query Output:
    *-bash-3.2$ hive -e "select * from air_client_logs where loglevel =
    '[FATAL]'"
    Hive history file=/tmp/hdfs/hive_job_log_hdfs_201106271702_317571839.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201106271658_0002, Tracking URL =
    http://hadooptest1:50030/jobdetails.jsp?jobid=job_201106271658_0002
    Kill Command = /usr/lib/hadoop/bin/hadoop job
    -Dmapred.job.tracker=hadooptest1:54311 -kill job_201106271658_0002
    2011-06-27 17:02:39,874 Stage-1 map = 0%, reduce = 0%
    2011-06-27 17:03:02,111 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201106271658_0002 with errors

    java.lang.RuntimeException: Error while reading from task log url
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
    at
    org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:425)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    Caused by: java.io.IOException: Server returned HTTP response code: 400
    for URL:
    http://hadooptest14:50060/tasklog?taskid=attempt_201106271658_0002_m_000008_0&all=true

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    at java.net.URL.openStream(URL.java:1010)
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
    ... 16 more
    Ended Job = job_201106271658_0002 with exception
    'java.lang.RuntimeException(Error while reading from task log url)'

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    Again, trying to go to "
    http://hadooptest14:50060/tasklog?taskid=attempt_201106271658_0002_m_000008_0&all=true"
    returns that argument attemptid error.

    What am I doing wrong here? I appear to keep doing it, whatever it is.


    On Fri, May 6, 2011 at 6:47 PM, Time Less wrote:

    My cluster went corrupt-mode. I wiped it and deleted the Hive metastore
    and started over. In the process, I did a "yum upgrade" which probably took
    me from CDH3b4 to CDH3u0. Now everytime I submit a Hive query of complexity
    requiring a map/reduce job*, I get this error:

    2011-05-06 18:39:14,533 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201104081532_0509 with errors
    java.lang.RuntimeException: Error while reading from task log url
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
    at
    org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
    .....[snip].....
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    Caused by: java.io.IOException: Server returned HTTP response code: 400
    for URL:
    http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    at java.net.URL.openStream(URL.java:1010)
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
    ... 16 more
    Ended Job = job_201104081532_0509 with exception
    'java.lang.RuntimeException(Error while reading from task log url)'
    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    It seems to me the key point in here is this:

    Server returned HTTP response code: 400 for URL:
    http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true

    So I submitted that URL to my web browser, which said:

    Problem accessing /tasklog. Reason: Argument attemptid is required

    Does anyone have clues what this means?

    --
    Tim Ellis
    Riot Games
    * That is to say, simple queries like "select * from table limit 10"
    return results just fine.

    --
    Tim

    --
    Tim
  • Time Less at Aug 29, 2011 at 10:29 pm
    We ran into this YET AGAIN today, leading one of us to start checking
    configuration settings. We determined our TaskTracker JVM was configured
    with not too much memory (200MB?). We increased that (512MB?) and now the
    error is gone. So it appears it's a TaskTracker getting OOM killed.

    On Tue, Jun 28, 2011 at 10:40 AM, Time Less wrote:

    I'm having a hard time interpreting the JIRA - it seems to be saying that
    Hive is passing an incorrect "mapred.child.java.opts=XmxNNNM" parameter,
    missing the - ... Is that correct? I could dig into the Hive source code and
    look for this bit if that's so.

    I admit, I'm a Java newbie, so I'm not entirely certain I understand how
    Hive/Hadoop interact, but if Hive is passing incorrect memory arguments to
    Hadoop tasks, and they're OOMing, then it seems the sort of fix I could
    undertake successfully.


    On Mon, Jun 27, 2011 at 5:13 PM, Sumanth V wrote:

    You are hitting this bug -
    https://issues.apache.org/jira/browse/HIVE-1579
    I consistently hit this bug for one of the Hive queries.


    Sumanth


    On Mon, Jun 27, 2011 at 5:08 PM, Time Less wrote:

    Today I'm getting this error again. A Google search brought me back to...
    you guessed it... my own post. But this time no HDFS corruption. Bounced all
    services, namenode, jobtracker, datanodes, tasktrackers. Still same error.
    Here's what it looks like:

    *Fsck Output:
    *FSCK ended at Mon Jun 27 17:02:07 PDT 2011 in 818 milliseconds
    The filesystem under path '/' is HEALTHY

    *Hive Query Output:
    *-bash-3.2$ hive -e "select * from air_client_logs where loglevel =
    '[FATAL]'"
    Hive history file=/tmp/hdfs/hive_job_log_hdfs_201106271702_317571839.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201106271658_0002, Tracking URL =
    http://hadooptest1:50030/jobdetails.jsp?jobid=job_201106271658_0002
    Kill Command = /usr/lib/hadoop/bin/hadoop job
    -Dmapred.job.tracker=hadooptest1:54311 -kill job_201106271658_0002
    2011-06-27 17:02:39,874 Stage-1 map = 0%, reduce = 0%
    2011-06-27 17:03:02,111 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201106271658_0002 with errors

    java.lang.RuntimeException: Error while reading from task log url
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
    at
    org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:425)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    Caused by: java.io.IOException: Server returned HTTP response code: 400
    for URL:
    http://hadooptest14:50060/tasklog?taskid=attempt_201106271658_0002_m_000008_0&all=true

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    at java.net.URL.openStream(URL.java:1010)
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
    ... 16 more
    Ended Job = job_201106271658_0002 with exception
    'java.lang.RuntimeException(Error while reading from task log url)'

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    Again, trying to go to "
    http://hadooptest14:50060/tasklog?taskid=attempt_201106271658_0002_m_000008_0&all=true"
    returns that argument attemptid error.

    What am I doing wrong here? I appear to keep doing it, whatever it is.


    On Fri, May 6, 2011 at 6:47 PM, Time Less wrote:

    My cluster went corrupt-mode. I wiped it and deleted the Hive metastore
    and started over. In the process, I did a "yum upgrade" which probably took
    me from CDH3b4 to CDH3u0. Now everytime I submit a Hive query of complexity
    requiring a map/reduce job*, I get this error:

    2011-05-06 18:39:14,533 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201104081532_0509 with errors
    java.lang.RuntimeException: Error while reading from task log url
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:889)
    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:680)
    at
    org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
    .....[snip].....
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    Caused by: java.io.IOException: Server returned HTTP response code: 400
    for URL:
    http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
    at java.net.URL.openStream(URL.java:1010)
    at
    org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
    ... 16 more
    Ended Job = job_201104081532_0509 with exception
    'java.lang.RuntimeException(Error while reading from task log url)'
    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    It seems to me the key point in here is this:

    Server returned HTTP response code: 400 for URL:
    http://hadooptest3:50060/tasklog?taskid=attempt_201104081532_0509_m_000002_2&all=true

    So I submitted that URL to my web browser, which said:

    Problem accessing /tasklog. Reason: Argument attemptid is required

    Does anyone have clues what this means?

    --
    Tim Ellis
    Riot Games
    * That is to say, simple queries like "select * from table limit 10"
    return results just fine.

    --
    Tim

    --
    Tim


    --
    Tim

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedMay 7, '11 at 1:48a
activeAug 29, '11 at 10:29p
posts5
users2
websitehive.apache.org

2 users in discussion

Time Less: 4 posts Sumanth V: 1 post

People

Translate

site design / logo © 2022 Grokbase