Grokbase Groups Hive user July 2011
FAQ
I'm teaching myself hive and I'm having an issue with a simple select
statement.

I'm using Amazon's Elastic MapReduce.

I'm creating a table with some sample web log data I downloaded from a
sample given by Amazon.

I create this table:
CREATE EXTERNAL TABLE log_parser_stage (
remoteAddr STRING,
remoteLogname STRING,
user STRING,
time STRING,
request STRING,
status STRING,
bytes_string STRING,
referrer STRING,
browser STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = '^(\\S+) (\\S+) (\\S+)
\\[([\\w:/]+\\s[+\\-]\\d{4})\\] "(.+?)" (\\S+) (\\S+) "([^"]*)" "([^"]*)"',
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s")
LOCATION 's3://mys3bucket/';

I then run a simple select * from the table to insure there is data there:

hive> SELECT * FROM log_parser_stage LIMIT 10;
OK
66.249.67.3 - - 20/Jul/2009:20:12:22 -0700 GET
/gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15741&g2_returnName=photo
HTTP/1.1 302 5 - Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)
66.249.67.3 - - 20/Jul/2009:20:12:25 -0700 GET
/gallery/main.php?g2_itemId=15741&g2_fromNavId=x8fa12efc HTTP/1.1
200 8068 - Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
64.233.172.17 - - 20/Jul/2009:20:12:26 -0700 GET
/gwidgets/alexa.xml HTTP/1.1 200 2969 - Mozilla/5.0
(compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
74.125.74.193 - - 20/Jul/2009:20:13:01 -0700 GET
/gwidgets/alexa.xml HTTP/1.1 200 2969 - Mozilla/5.0
(compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
192.168.1.198 - - 20/Jul/2009:20:13:18 -0700 GET /
HTTP/1.1 200 17935 - Mozilla/5.0 (Macintosh; U; Intel Mac
OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0
Safari/530.17
192.168.1.198 - - 20/Jul/2009:20:13:18 -0700 GET
/style.css HTTP/1.1 200 1504 http://example.org/ Mozilla/5.0
(Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML,
like Gecko) Version/4.0 Safari/530.17
192.168.1.198 - - 20/Jul/2009:20:13:19 -0700 GET
/favicon.ico HTTP/1.1 404 146 http://example.org/
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us)
AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Safari/530.17
66.249.67.3 - - 20/Jul/2009:20:13:21 -0700 GET
/gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D30893&g2_returnName=photo
HTTP/1.1 302 5 - Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)
66.249.67.3 - - 20/Jul/2009:20:13:24 -0700 GET
/gallery/main.php?g2_itemId=30893&g2_fromNavId=xfc647d65 HTTP/1.1
200 8196 - Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
66.249.67.3 - - 20/Jul/2009:20:13:29 -0700 GET
/gallery/main.php?g2_view=search.SearchScan&g2_form%5BuseDefaultSettings%5D=1&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15789&g2_returnName=photo
HTTP/1.1 200 6360 - Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)
Time taken: 13.954 seconds

I then try to query for two fields:

hive> SELECT remoteAddr, browser FROM log_parser_stage LIMIT 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201107062035_0007, Tracking URL =
http://ip-10-36-79-98.ec2.internal:9100/jobdetails.jsp?jobid=job_201107062035_0007
Kill Command = /home/hadoop/.versions/0.20/bin/../bin/hadoop job
-Dmapred.job.tracker=ip-10-36-79-98.ec2.internal:9001 -kill
job_201107062035_0007
2011-07-06 22:06:21,137 Stage-1 map = 0%, reduce = 0%
2011-07-06 22:07:05,974 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201107062035_0007 with errors

Failed tasks with most(4) failures :
Task URL:
http://ip-10-36-79-98.ec2.internal:9100/taskdetails.jsp?jobid=job_201107062035_0007&tipid=task_201107062035_0007_m_000000

FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.ExecDriver

The following is from the job logs:

Meta VERSION="1" .
Job JOBID="job_201107062035_0006" JOBNAME="SELECT remoteAddr, browse\.\.\.log_parser_stage(Stage-1)" USER="hadoop" SUBMIT_TIME="1309989443667" JOBCONF="hdfs://ip-10-36-79-98\.ec2\.internal:9000/mnt/var/lib/hadoop/tmp/mapred/system/job_201107062035_0006/job\.xml" .
Job JOBID="job_201107062035_0006" JOB_PRIORITY="NORMAL" .
Job JOBID="job_201107062035_0006" LAUNCH_TIME="1309989444146" TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP" START_TIME="1309989444665" SPLITS="" .
MapAttempt TASK_TYPE="SETUP" TASKID="task_201107062035_0006_m_000002" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0" START_TIME="1309989445566" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
MapAttempt TASK_TYPE="SETUP" TASKID="task_201107062035_0006_m_000002" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0" TASK_STATUS="SUCCESS" FINISH_TIME="1309989447674" HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal" STATE_STRING="setup" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP" TASK_STATUS="SUCCESS" FINISH_TIME="1309989450674" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Job JOBID="job_201107062035_0006" JOB_STATUS="RUNNING" .
Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP" START_TIME="1309989450675" SPLITS="/default-rack/localhost" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0" START_TIME="1309989450701" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0" TASK_STATUS="FAILED" FINISH_TIME="1309989457869" HOSTNAME="ip-10-36-79-98\.ec2\.internal" ERROR="java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: Map operator initialization failed
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
\.\.\. 18 more
Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
\.\.\. 18 more
Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
at java\.security\.AccessController\.doPrivileged(Native Method)
at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
at java\.lang\.Class\.forName0(Native Method)
at java\.lang\.Class\.forName(Class\.java:247)
at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
\.\.\. 19 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1" START_TIME="1309989459732" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1" TASK_STATUS="FAILED" FINISH_TIME="1309989467109" HOSTNAME="ip-10-36-79-98\.ec2\.internal" ERROR="java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: Map operator initialization failed
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
\.\.\. 18 more
Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
\.\.\. 18 more
Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
at java\.security\.AccessController\.doPrivileged(Native Method)
at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
at java\.lang\.Class\.forName0(Native Method)
at java\.lang\.Class\.forName(Class\.java:247)
at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
\.\.\. 19 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2" START_TIME="1309989468821" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2" TASK_STATUS="FAILED" FINISH_TIME="1309989477378" HOSTNAME="ip-10-36-79-98\.ec2\.internal" ERROR="java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: Map operator initialization failed
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
\.\.\. 18 more
Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
\.\.\. 18 more
Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
at java\.security\.AccessController\.doPrivileged(Native Method)
at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
at java\.lang\.Class\.forName0(Native Method)
at java\.lang\.Class\.forName(Class\.java:247)
at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
\.\.\. 19 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3" START_TIME="1309989477905" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3" TASK_STATUS="FAILED" FINISH_TIME="1309989484902" HOSTNAME="ip-10-36-79-98\.ec2\.internal" ERROR="java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: Map operator initialization failed
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
\.\.\. 18 more
Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
\.\.\. 18 more
Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
at java\.security\.AccessController\.doPrivileged(Native Method)
at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
at java\.lang\.Class\.forName0(Native Method)
at java\.lang\.Class\.forName(Class\.java:247)
at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
\.\.\. 19 more
" .
Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP" TASK_STATUS="FAILED" FINISH_TIME="1309989484902" ERROR="java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: Map operator initialization failed
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
\.\.\. 18 more
Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
\.\.\. 18 more
Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
at java\.security\.AccessController\.doPrivileged(Native Method)
at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
at java\.lang\.Class\.forName0(Native Method)
at java\.lang\.Class\.forName(Class\.java:247)
at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
\.\.\. 19 more
" TASK_ATTEMPT_ID="" .
Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP" START_TIME="1309989486882" SPLITS="" .
MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201107062035_0006_m_000001" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0" START_TIME="1309989486898" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201107062035_0006_m_000001" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0" TASK_STATUS="SUCCESS" FINISH_TIME="1309989489057" HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal" STATE_STRING="cleanup" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP" TASK_STATUS="SUCCESS" FINISH_TIME="1309989489906" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Job JOBID="job_201107062035_0006" FINISH_TIME="1309989489907" JOB_STATUS="FAILED" FINISHED_MAPS="0" FINISHED_REDUCES="0" .

Any help is appreciated.

-Sal

Search Discussions

  • Sal Scalisi at Jul 7, 2011 at 8:05 pm
    I have some more information that may shed some light on this issue. I
    re-ran my query with hive configured to log to the console window. I
    noticed the following error from AWS:

    11/07/07 19:56:42 INFO amazonaws.request: Sending Request: HEAD
    https://mapreduce-dev.s3.amazonaws.com /log_parser%2Finput_%24folder%24
    Headers: (Authorization: AWS XXXX=, Date: Thu, 07 Jul 2011 19:56:42 GMT,
    Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
    11/07/07 19:56:42 INFO amazonaws.request: Received error response:
    Status Code: 404, AWS Request ID: XXXX, AWS Error Code: null, AWS Error
    Message: Not Found, S3 Extended Request ID:

    I've tried to make my S3 folders public, but that did not resolve the
    issue. If anyone has any insight, it is appreciated.

    -Sal

    On 7/6/2011 3:12 PM, Sal Scalisi wrote:
    I'm teaching myself hive and I'm having an issue with a simple select
    statement.

    I'm using Amazon's Elastic MapReduce.

    I'm creating a table with some sample web log data I downloaded from a
    sample given by Amazon.

    I create this table:
    CREATE EXTERNAL TABLE log_parser_stage (
    remoteAddr STRING,
    remoteLogname STRING,
    user STRING,
    time STRING,
    request STRING,
    status STRING,
    bytes_string STRING,
    referrer STRING,
    browser STRING
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    WITH SERDEPROPERTIES (
    "input.regex" = '^(\\S+) (\\S+) (\\S+)
    \\[([\\w:/]+\\s[+\\-]\\d{4})\\] "(.+?)" (\\S+) (\\S+) "([^"]*)" "([^"]*)"',
    "output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s")
    LOCATION 's3://mys3bucket/';

    I then run a simple select * from the table to insure there is data there:

    hive> SELECT * FROM log_parser_stage LIMIT 10;
    OK
    66.249.67.3 - - 20/Jul/2009:20:12:22 -0700 GET
    /gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15741&g2_returnName=photo
    HTTP/1.1 302 5 - Mozilla/5.0 (compatible;
    Googlebot/2.1; +http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:12:25 -0700 GET
    /gallery/main.php?g2_itemId=15741&g2_fromNavId=x8fa12efc HTTP/1.1
    200 8068 - Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)
    64.233.172.17 - - 20/Jul/2009:20:12:26 -0700 GET
    /gwidgets/alexa.xml HTTP/1.1 200 2969 - Mozilla/5.0
    (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
    74.125.74.193 - - 20/Jul/2009:20:13:01 -0700 GET
    /gwidgets/alexa.xml HTTP/1.1 200 2969 - Mozilla/5.0
    (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
    192.168.1.198 - - 20/Jul/2009:20:13:18 -0700 GET /
    HTTP/1.1 200 17935 - Mozilla/5.0 (Macintosh; U; Intel Mac
    OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0
    Safari/530.17
    192.168.1.198 - - 20/Jul/2009:20:13:18 -0700 GET
    /style.css HTTP/1.1 200 1504 http://example.org/ Mozilla/5.0
    (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML,
    like Gecko) Version/4.0 Safari/530.17
    192.168.1.198 - - 20/Jul/2009:20:13:19 -0700 GET
    /favicon.ico HTTP/1.1 404 146 http://example.org/
    Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us)
    AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Safari/530.17
    66.249.67.3 - - 20/Jul/2009:20:13:21 -0700 GET
    /gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D30893&g2_returnName=photo
    HTTP/1.1 302 5 - Mozilla/5.0 (compatible;
    Googlebot/2.1; +http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:13:24 -0700 GET
    /gallery/main.php?g2_itemId=30893&g2_fromNavId=xfc647d65 HTTP/1.1
    200 8196 - Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:13:29 -0700 GET
    /gallery/main.php?g2_view=search.SearchScan&g2_form%5BuseDefaultSettings%5D=1&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15789&g2_returnName=photo
    HTTP/1.1 200 6360 - Mozilla/5.0 (compatible;
    Googlebot/2.1; +http://www.google.com/bot.html)
    Time taken: 13.954 seconds

    I then try to query for two fields:

    hive> SELECT remoteAddr, browser FROM log_parser_stage LIMIT 10;
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201107062035_0007, Tracking URL =
    http://ip-10-36-79-98.ec2.internal:9100/jobdetails.jsp?jobid=job_201107062035_0007

    Kill Command = /home/hadoop/.versions/0.20/bin/../bin/hadoop job
    -Dmapred.job.tracker=ip-10-36-79-98.ec2.internal:9001 -kill
    job_201107062035_0007
    2011-07-06 22:06:21,137 Stage-1 map = 0%, reduce = 0%
    2011-07-06 22:07:05,974 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201107062035_0007 with errors

    Failed tasks with most(4) failures :
    Task URL:
    http://ip-10-36-79-98.ec2.internal:9100/taskdetails.jsp?jobid=job_201107062035_0007&tipid=task_201107062035_0007_m_000000


    FAILED: Execution Error, return code 2 from
    org.apache.hadoop.hive.ql.exec.ExecDriver

    The following is from the job logs:

    Meta VERSION="1" .
    Job JOBID="job_201107062035_0006" JOBNAME="SELECT remoteAddr,
    browse\.\.\.log_parser_stage(Stage-1)" USER="hadoop"
    SUBMIT_TIME="1309989443667"
    JOBCONF="hdfs://ip-10-36-79-98\.ec2\.internal:9000/mnt/var/lib/hadoop/tmp/mapred/system/job_201107062035_0006/job\.xml"
    .
    Job JOBID="job_201107062035_0006" JOB_PRIORITY="NORMAL" .
    Job JOBID="job_201107062035_0006" LAUNCH_TIME="1309989444146"
    TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
    Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP"
    START_TIME="1309989444665" SPLITS="" .
    MapAttempt TASK_TYPE="SETUP" TASKID="task_201107062035_0006_m_000002"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0"
    START_TIME="1309989445566"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="SETUP" TASKID="task_201107062035_0006_m_000002"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989447674"
    HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal"
    STATE_STRING="setup"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989450674"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Job JOBID="job_201107062035_0006" JOB_STATUS="RUNNING" .
    Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP"
    START_TIME="1309989450675" SPLITS="/default-rack/localhost" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0"
    START_TIME="1309989450701"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0"
    TASK_STATUS="FAILED" FINISH_TIME="1309989457869"
    HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1"
    START_TIME="1309989459732"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1"
    TASK_STATUS="FAILED" FINISH_TIME="1309989467109"
    HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2"
    START_TIME="1309989468821"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2"
    TASK_STATUS="FAILED" FINISH_TIME="1309989477378"
    HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3"
    START_TIME="1309989477905"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3"
    TASK_STATUS="FAILED" FINISH_TIME="1309989484902"
    HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP"
    TASK_STATUS="FAILED" FINISH_TIME="1309989484902"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " TASK_ATTEMPT_ID="" .
    Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP"
    START_TIME="1309989486882" SPLITS="" .
    MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201107062035_0006_m_000001"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0"
    START_TIME="1309989486898"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201107062035_0006_m_000001"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989489057"
    HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal"
    STATE_STRING="cleanup"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989489906"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Job JOBID="job_201107062035_0006" FINISH_TIME="1309989489907"
    JOB_STATUS="FAILED" FINISHED_MAPS="0" FINISHED_REDUCES="0" .

    Any help is appreciated.

    -Sal
  • Norbert Burger at Jul 7, 2011 at 8:30 pm
    There is likely a cleaner solution to this, but what happens if you add the
    Hive contrib JAR to the classpath at the beginning of your Hive sesssion?

    ADD JAR << fully-qualified path to HIVE contrib JAR >> , for example:

    ADD JAR /usr/lib/hive-0.7.0-cdh3u0/lib/hive-contrib-0.7.0-cdh3u0.jar

    Norbert
    On Thu, Jul 7, 2011 at 4:04 PM, Sal Scalisi wrote:

    I have some more information that may shed some light on this issue. I
    re-ran my query with hive configured to log to the console window. I
    noticed the following error from AWS:

    11/07/07 19:56:42 INFO amazonaws.request: Sending Request: HEAD
    https://mapreduce-dev.s3.amazonaws.com /log_parser%2Finput_%24folder%24
    Headers: (Authorization: AWS XXXX=, Date: Thu, 07 Jul 2011 19:56:42 GMT,
    Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
    11/07/07 19:56:42 INFO amazonaws.request: Received error response: Status
    Code: 404, AWS Request ID: XXXX, AWS Error Code: null, AWS Error Message:
    Not Found, S3 Extended Request ID:

    I've tried to make my S3 folders public, but that did not resolve the
    issue. If anyone has any insight, it is appreciated.

    -Sal

    On 7/6/2011 3:12 PM, Sal Scalisi wrote:
    I'm teaching myself hive and I'm having an issue with a simple select
    statement.

    I'm using Amazon's Elastic MapReduce.

    I'm creating a table with some sample web log data I downloaded from a
    sample given by Amazon.

    I create this table:
    CREATE EXTERNAL TABLE log_parser_stage (
    remoteAddr STRING,
    remoteLogname STRING,
    user STRING,
    time STRING,
    request STRING,
    status STRING,
    bytes_string STRING,
    referrer STRING,
    browser STRING
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    WITH SERDEPROPERTIES (
    "input.regex" = '^(\\S+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\]
    "(.+?)" (\\S+) (\\S+) "([^"]*)" "([^"]*)"',
    "output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s")
    LOCATION 's3://mys3bucket/';

    I then run a simple select * from the table to insure there is data there:

    hive> SELECT * FROM log_parser_stage LIMIT 10;
    OK
    66.249.67.3 - - 20/Jul/2009:20:12:22 -0700 GET
    /gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15741&g2_returnName=photo
    HTTP/1.1 302 5 - Mozilla/5.0 (compatible; Googlebot/2.1; +
    http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:12:25 -0700 GET
    /gallery/main.php?g2_itemId=15741&g2_fromNavId=x8fa12efc HTTP/1.1 200
    8068 - Mozilla/5.0 (compatible; Googlebot/2.1; +
    http://www.google.com/bot.html)
    64.233.172.17 - - 20/Jul/2009:20:12:26 -0700 GET
    /gwidgets/alexa.xml HTTP/1.1 200 2969 - Mozilla/5.0
    (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)

    74.125.74.193 - - 20/Jul/2009:20:13:01 -0700 GET
    /gwidgets/alexa.xml HTTP/1.1 200 2969 - Mozilla/5.0
    (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)

    192.168.1.198 - - 20/Jul/2009:20:13:18 -0700 GET /
    HTTP/1.1 200 17935 - Mozilla/5.0 (Macintosh; U; Intel Mac OS X
    10_5_7; en-us) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0
    Safari/530.17
    192.168.1.198 - - 20/Jul/2009:20:13:18 -0700 GET */style.css
    HTTP/1.1 200 1504 http://example.org/* Mozilla/5.0 (Macintosh;
    U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML, like Gecko)
    Version/4.0 Safari/530.17
    192.168.1.198 - - 20/Jul/2009:20:13:19 -0700 GET */favicon.ico
    HTTP/1.1 404 146 http://example.org/* Mozilla/5.0
    (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML, like
    Gecko) Version/4.0 Safari/530.17
    66.249.67.3 - - 20/Jul/2009:20:13:21 -0700 GET
    /gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D30893&g2_returnName=photo
    HTTP/1.1 302 5 - Mozilla/5.0 (compatible; Googlebot/2.1; +
    http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:13:24 -0700 GET
    /gallery/main.php?g2_itemId=30893&g2_fromNavId=xfc647d65 HTTP/1.1 200
    8196 - Mozilla/5.0 (compatible; Googlebot/2.1; +
    http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:13:29 -0700 GET
    /gallery/main.php?g2_view=search.SearchScan&g2_form%5BuseDefaultSettings%5D=1&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15789&g2_returnName=photo
    HTTP/1.1 200 6360 - Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)
    Time taken: 13.954 seconds

    I then try to query for two fields:

    hive> SELECT remoteAddr, browser FROM log_parser_stage LIMIT 10;
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201107062035_0007, Tracking URL =
    http://ip-10-36-79-98.ec2.internal:9100/jobdetails.jsp?jobid=job_201107062035_0007
    Kill Command = */home/hadoop/*.versions/0.20/bin/../bin/hadoop job
    -Dmapred.job.tracker=ip-10-36-79-98.ec2.internal:9001 -kill
    job_201107062035_0007
    2011-07-06 22:06:21,137 Stage-1 map = 0%, reduce = 0%
    2011-07-06 22:07:05,974 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201107062035_0007 with errors

    Failed tasks with most(4) failures :
    Task URL:
    http://ip-10-36-79-98.ec2.internal:9100/taskdetails.jsp?jobid=job_201107062035_0007&tipid=task_201107062035_0007_m_000000

    FAILED: Execution Error, return code 2 from
    org.apache.hadoop.hive.ql.exec.ExecDriver

    The following is from the job logs:

    Meta VERSION="1" .
    Job JOBID="job_201107062035_0006" JOBNAME="SELECT remoteAddr,
    browse\.\.\.log_parser_stage(Stage-1)" USER="hadoop"
    SUBMIT_TIME="1309989443667"
    JOBCONF="hdfs://ip-10-36-79-98\.ec2\.internal:9000/mnt/var/lib/hadoop/tmp/mapred/system/job_201107062035_0006/job\.xml"
    .
    Job JOBID="job_201107062035_0006" JOB_PRIORITY="NORMAL" .
    Job JOBID="job_201107062035_0006" LAUNCH_TIME="1309989444146"
    TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
    Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP"
    START_TIME="1309989444665" SPLITS="" .
    MapAttempt TASK_TYPE="SETUP" TASKID="task_201107062035_0006_m_000002"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0"
    START_TIME="1309989445566"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="SETUP" TASKID="task_201107062035_0006_m_000002"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0" TASK_STATUS="SUCCESS"
    FINISH_TIME="1309989447674"
    HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal" STATE_STRING="setup"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989450674"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Job JOBID="job_201107062035_0006" JOB_STATUS="RUNNING" .
    Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP"
    START_TIME="1309989450675" SPLITS="/default-rack/localhost" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0"
    START_TIME="1309989450701"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0" TASK_STATUS="FAILED"
    FINISH_TIME="1309989457869" HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed

    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1"
    START_TIME="1309989459732"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1" TASK_STATUS="FAILED"
    FINISH_TIME="1309989467109" HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed

    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2"
    START_TIME="1309989468821"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2" TASK_STATUS="FAILED"
    FINISH_TIME="1309989477378" HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed

    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3"
    START_TIME="1309989477905"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3" TASK_STATUS="FAILED"
    FINISH_TIME="1309989484902" HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed

    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP"
    TASK_STATUS="FAILED" FINISH_TIME="1309989484902"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed

    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " TASK_ATTEMPT_ID="" .
    Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP"
    START_TIME="1309989486882" SPLITS="" .
    MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201107062035_0006_m_000001"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0"
    START_TIME="1309989486898"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201107062035_0006_m_000001"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0" TASK_STATUS="SUCCESS"
    FINISH_TIME="1309989489057"
    HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal"
    STATE_STRING="cleanup"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989489906"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Job JOBID="job_201107062035_0006" FINISH_TIME="1309989489907"
    JOB_STATUS="FAILED" FINISHED_MAPS="0" FINISHED_REDUCES="0" .

    Any help is appreciated.

    -Sal
  • Sal Scalisi at Jul 7, 2011 at 9:09 pm
    That did it, thanks!
    On 7/7/2011 1:30 PM, Norbert Burger wrote:
    There is likely a cleaner solution to this, but what happens if you
    add the Hive contrib JAR to the classpath at the beginning of your
    Hive sesssion?

    ADD JAR << fully-qualified path to HIVE contrib JAR >> , for example:

    ADD JAR /usr/lib/hive-0.7.0-cdh3u0/lib/hive-contrib-0.7.0-cdh3u0.jar

    Norbert

    On Thu, Jul 7, 2011 at 4:04 PM, Sal Scalisi wrote:

    I have some more information that may shed some light on this
    issue. I re-ran my query with hive configured to log to the
    console window. I noticed the following error from AWS:

    11/07/07 19:56:42 INFO amazonaws.request: Sending Request: HEAD
    https://mapreduce-dev.s3.amazonaws.com
    /log_parser%2Finput_%24folder%24 Headers: (Authorization: AWS
    XXXX=, Date: Thu, 07 Jul 2011 19:56:42 GMT, Content-Type:
    application/x-www-form-urlencoded; charset=utf-8, )
    11/07/07 19:56:42 INFO amazonaws.request: Received error response:
    Status Code: 404, AWS Request ID: XXXX, AWS Error Code: null, AWS
    Error Message: Not Found, S3 Extended Request ID:

    I've tried to make my S3 folders public, but that did not resolve
    the issue. If anyone has any insight, it is appreciated.

    -Sal

    On 7/6/2011 3:12 PM, Sal Scalisi wrote:
    I'm teaching myself hive and I'm having an issue with a simple
    select statement.

    I'm using Amazon's Elastic MapReduce.

    I'm creating a table with some sample web log data I downloaded
    from a sample given by Amazon.

    I create this table:
    CREATE EXTERNAL TABLE log_parser_stage (
    remoteAddr STRING,
    remoteLogname STRING,
    user STRING,
    time STRING,
    request STRING,
    status STRING,
    bytes_string STRING,
    referrer STRING,
    browser STRING
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    WITH SERDEPROPERTIES (
    "input.regex" = '^(\\S+) (\\S+) (\\S+)
    \\[([\\w:/]+\\s[+\\-]\\d{4})\\] "(.+?)" (\\S+) (\\S+) "([^"]*)"
    "([^"]*)"',
    "output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s
    %8$s %9$s")
    LOCATION 's3://mys3bucket/';

    I then run a simple select * from the table to insure there is
    data there:

    hive> SELECT * FROM log_parser_stage LIMIT 10;
    OK
    66.249.67.3 - - 20/Jul/2009:20:12:22 -0700
    GET
    /gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15741&g2_returnName=photo
    HTTP/1.1 302 5 - Mozilla/5.0 (compatible;
    Googlebot/2.1; +http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:12:25 -0700
    GET /gallery/main.php?g2_itemId=15741&g2_fromNavId=x8fa12efc
    HTTP/1.1 200 8068 - Mozilla/5.0 (compatible;
    Googlebot/2.1; +http://www.google.com/bot.html)
    64.233.172.17 - - 20/Jul/2009:20:12:26 -0700
    GET /gwidgets/alexa.xml HTTP/1.1 200 2969 -
    Mozilla/5.0 (compatible) Feedfetcher-Google;
    (+http://www.google.com/feedfetcher.html)
    74.125.74.193 - - 20/Jul/2009:20:13:01 -0700
    GET /gwidgets/alexa.xml HTTP/1.1 200 2969 -
    Mozilla/5.0 (compatible) Feedfetcher-Google;
    (+http://www.google.com/feedfetcher.html)
    192.168.1.198 - - 20/Jul/2009:20:13:18 -0700
    GET / HTTP/1.1 200 17935 - Mozilla/5.0 (Macintosh; U;
    Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML, like
    Gecko) Version/4.0 Safari/530.17
    192.168.1.198 - - 20/Jul/2009:20:13:18 -0700
    GET //style.css HTTP/1.1 200 1504 http://example.org//
    Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us)
    AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Safari/530.17
    192.168.1.198 - - 20/Jul/2009:20:13:19 -0700
    GET //favicon.ico HTTP/1.1 404 146
    http://example.org// Mozilla/5.0 (Macintosh; U; Intel Mac OS X
    10_5_7; en-us) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0
    Safari/530.17
    66.249.67.3 - - 20/Jul/2009:20:13:21 -0700
    GET
    /gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D30893&g2_returnName=photo
    HTTP/1.1 302 5 - Mozilla/5.0 (compatible;
    Googlebot/2.1; +http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:13:24 -0700
    GET /gallery/main.php?g2_itemId=30893&g2_fromNavId=xfc647d65
    HTTP/1.1 200 8196 - Mozilla/5.0 (compatible;
    Googlebot/2.1; +http://www.google.com/bot.html)
    66.249.67.3 - - 20/Jul/2009:20:13:29 -0700
    GET
    /gallery/main.php?g2_view=search.SearchScan&g2_form%5BuseDefaultSettings%5D=1&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15789&g2_returnName=photo
    HTTP/1.1 200 6360 - Mozilla/5.0 (compatible;
    Googlebot/2.1; +http://www.google.com/bot.html)
    Time taken: 13.954 seconds

    I then try to query for two fields:

    hive> SELECT remoteAddr, browser FROM log_parser_stage LIMIT 10;
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201107062035_0007, Tracking URL =
    http://ip-10-36-79-98.ec2.internal:9100/jobdetails.jsp?jobid=job_201107062035_0007

    Kill Command = //home/hadoop//.versions/0.20/bin/../bin/hadoop
    job -Dmapred.job.tracker=ip-10-36-79-98.ec2.internal:9001 -kill
    job_201107062035_0007
    2011-07-06 22:06:21,137 Stage-1 map = 0%, reduce = 0%
    2011-07-06 22:07:05,974 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201107062035_0007 with errors

    Failed tasks with most(4) failures :
    Task URL:
    http://ip-10-36-79-98.ec2.internal:9100/taskdetails.jsp?jobid=job_201107062035_0007&tipid=task_201107062035_0007_m_000000
    <http://ip-10-36-79-98.ec2.internal:9100/taskdetails.jsp?jobid=job_201107062035_0007&tipid=task_201107062035_0007_m_000000>


    FAILED: Execution Error, return code 2 from
    org.apache.hadoop.hive.ql.exec.ExecDriver

    The following is from the job logs:

    Meta VERSION="1" .
    Job JOBID="job_201107062035_0006" JOBNAME="SELECT remoteAddr,
    browse\.\.\.log_parser_stage(Stage-1)" USER="hadoop"
    SUBMIT_TIME="1309989443667"
    JOBCONF="hdfs://ip-10-36-79-98\.ec2\.internal:9000/mnt/var/lib/hadoop/tmp/mapred/system/job_201107062035_0006/job\.xml"
    .
    Job JOBID="job_201107062035_0006" JOB_PRIORITY="NORMAL" .
    Job JOBID="job_201107062035_0006" LAUNCH_TIME="1309989444146"
    TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
    Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP"
    START_TIME="1309989444665" SPLITS="" .
    MapAttempt TASK_TYPE="SETUP"
    TASKID="task_201107062035_0006_m_000002"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0"
    START_TIME="1309989445566"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="SETUP"
    TASKID="task_201107062035_0006_m_000002"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989447674"
    HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal"
    STATE_STRING="setup"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989450674"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Job JOBID="job_201107062035_0006" JOB_STATUS="RUNNING" .
    Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP"
    START_TIME="1309989450675" SPLITS="/default-rack/localhost" .
    MapAttempt TASK_TYPE="MAP"
    TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0"
    START_TIME="1309989450701"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP"
    TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0"
    TASK_STATUS="FAILED" FINISH_TIME="1309989457869"
    HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator
    initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at
    sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP"
    TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1"
    START_TIME="1309989459732"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP"
    TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1"
    TASK_STATUS="FAILED" FINISH_TIME="1309989467109"
    HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator
    initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at
    sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP"
    TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2"
    START_TIME="1309989468821"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP"
    TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2"
    TASK_STATUS="FAILED" FINISH_TIME="1309989477378"
    HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator
    initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at
    sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP"
    TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3"
    START_TIME="1309989477905"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP"
    TASKID="task_201107062035_0006_m_000000"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3"
    TASK_STATUS="FAILED" FINISH_TIME="1309989484902"
    HOSTNAME="ip-10-36-79-98\.ec2\.internal"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator
    initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at
    sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP"
    TASK_STATUS="FAILED" FINISH_TIME="1309989484902"
    ERROR="java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at
    org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at
    sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at
    sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at
    org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator
    initialization failed
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException:
    java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException:
    org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at
    sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at
    org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at
    org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " TASK_ATTEMPT_ID="" .
    Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP"
    START_TIME="1309989486882" SPLITS="" .
    MapAttempt TASK_TYPE="CLEANUP"
    TASKID="task_201107062035_0006_m_000001"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0"
    START_TIME="1309989486898"
    TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327"
    HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="CLEANUP"
    TASKID="task_201107062035_0006_m_000001"
    TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989489057"
    HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal"
    STATE_STRING="cleanup"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP"
    TASK_STATUS="SUCCESS" FINISH_TIME="1309989489906"
    COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
    Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Job JOBID="job_201107062035_0006" FINISH_TIME="1309989489907"
    JOB_STATUS="FAILED" FINISHED_MAPS="0" FINISHED_REDUCES="0" .

    Any help is appreciated.

    -Sal
  • Prashanth R at Jul 7, 2011 at 9:16 pm
    to add it globally to have better persistence and control, you could
    add it in hive-site.xml

    <property>
    <name>hive.aux.jars.path</name>
    <value>file:///${HIVE_HOME/auxlib/all_new.jar</value>
    </property>
    On Thu, Jul 7, 2011 at 2:09 PM, Sal Scalisi wrote:

    That did it, thanks!

    On 7/7/2011 1:30 PM, Norbert Burger wrote:

    There is likely a cleaner solution to this, but what happens if you add the Hive contrib JAR to the classpath at the beginning of your Hive sesssion?

    ADD JAR << fully-qualified path to HIVE contrib JAR >> , for example:

    ADD JAR /usr/lib/hive-0.7.0-cdh3u0/lib/hive-contrib-0.7.0-cdh3u0.jar

    Norbert
    On Thu, Jul 7, 2011 at 4:04 PM, Sal Scalisi wrote:

    I have some more information that may shed some light on this issue.  I re-ran my query with hive configured to log to the console window.  I noticed the following error from AWS:

    11/07/07 19:56:42 INFO amazonaws.request: Sending Request: HEAD https://mapreduce-dev.s3.amazonaws.com /log_parser%2Finput_%24folder%24 Headers: (Authorization: AWS XXXX=, Date: Thu, 07 Jul 2011 19:56:42 GMT, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
    11/07/07 19:56:42 INFO amazonaws.request: Received error response: Status Code: 404, AWS Request ID: XXXX, AWS Error Code: null, AWS Error Message: Not Found, S3 Extended Request ID:

    I've tried to make my S3 folders public, but that did not resolve the issue.  If anyone has any insight, it is appreciated.

    -Sal

    On 7/6/2011 3:12 PM, Sal Scalisi wrote:
    I'm teaching myself hive and I'm having an issue with a simple select statement.

    I'm using Amazon's Elastic MapReduce.

    I'm creating a table with some sample web log data I downloaded from a sample given by Amazon.

    I create this table:
    CREATE EXTERNAL TABLE log_parser_stage (
    remoteAddr    STRING,
    remoteLogname STRING,
    user          STRING,
    time          STRING,
    request       STRING,
    status        STRING,
    bytes_string  STRING,
    referrer      STRING,
    browser       STRING
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    WITH SERDEPROPERTIES (
    "input.regex" = '^(\\S+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] "(.+?)" (\\S+) (\\S+) "([^"]*)" "([^"]*)"',
    "output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s")
    LOCATION 's3://mys3bucket/';

    I then run a simple select * from the table to insure there is data there:

    hive> SELECT * FROM log_parser_stage LIMIT 10;
    OK
    66.249.67.3     -       -       20/Jul/2009:20:12:22 -0700      GET /gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15741&g2_returnName=photo HTTP/1.1   302     5       -       Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    66.249.67.3     -       -       20/Jul/2009:20:12:25 -0700      GET /gallery/main.php?g2_itemId=15741&g2_fromNavId=x8fa12efc HTTP/1.1   200     8068    -       Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    64.233.172.17   -       -       20/Jul/2009:20:12:26 -0700      GET /gwidgets/alexa.xml HTTP/1.1        200     2969    -       Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
    74.125.74.193   -       -       20/Jul/2009:20:13:01 -0700      GET /gwidgets/alexa.xml HTTP/1.1        200     2969    -       Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
    192.168.1.198   -       -       20/Jul/2009:20:13:18 -0700      GET / HTTP/1.1  200     17935   -       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Safari/530.17
    192.168.1.198   -       -       20/Jul/2009:20:13:18 -0700      GET /style.css HTTP/1.1 200     1504    http://example.org/     Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Safari/530.17
    192.168.1.198   -       -       20/Jul/2009:20:13:19 -0700      GET /favicon.ico HTTP/1.1       404     146     http://example.org/     Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Safari/530.17
    66.249.67.3     -       -       20/Jul/2009:20:13:21 -0700      GET /gallery/main.php?g2_controller=exif.SwitchDetailMode&g2_mode=detailed&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D30893&g2_returnName=photo HTTP/1.1   302     5       -       Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    66.249.67.3     -       -       20/Jul/2009:20:13:24 -0700      GET /gallery/main.php?g2_itemId=30893&g2_fromNavId=xfc647d65 HTTP/1.1   200     8196    -       Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    66.249.67.3     -       -       20/Jul/2009:20:13:29 -0700      GET /gallery/main.php?g2_view=search.SearchScan&g2_form%5BuseDefaultSettings%5D=1&g2_return=%2Fgallery%2Fmain.php%3Fg2_itemId%3D15789&g2_returnName=photo HTTP/1.1    200     6360    -       Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    Time taken: 13.954 seconds

    I then try to query for two fields:

    hive> SELECT remoteAddr, browser FROM log_parser_stage LIMIT 10;
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201107062035_0007, Tracking URL = http://ip-10-36-79-98.ec2.internal:9100/jobdetails.jsp?jobid=job_201107062035_0007
    Kill Command = /home/hadoop/.versions/0.20/bin/../bin/hadoop job  -Dmapred.job.tracker=ip-10-36-79-98.ec2.internal:9001 -kill job_201107062035_0007
    2011-07-06 22:06:21,137 Stage-1 map = 0%,  reduce = 0%
    2011-07-06 22:07:05,974 Stage-1 map = 100%,  reduce = 100%
    Ended Job = job_201107062035_0007 with errors

    Failed tasks with most(4) failures :
    Task URL: http://ip-10-36-79-98.ec2.internal:9100/taskdetails.jsp?jobid=job_201107062035_0007&tipid=task_201107062035_0007_m_000000

    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver

    The following is from the job logs:

    Meta VERSION="1" .
    Job JOBID="job_201107062035_0006" JOBNAME="SELECT remoteAddr, browse\.\.\.log_parser_stage(Stage-1)" USER="hadoop" SUBMIT_TIME="1309989443667" JOBCONF="hdfs://ip-10-36-79-98\.ec2\.internal:9000/mnt/var/lib/hadoop/tmp/mapred/system/job_201107062035_0006/job\.xml" .
    Job JOBID="job_201107062035_0006" JOB_PRIORITY="NORMAL" .
    Job JOBID="job_201107062035_0006" LAUNCH_TIME="1309989444146" TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
    Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP" START_TIME="1309989444665" SPLITS="" .
    MapAttempt TASK_TYPE="SETUP" TASKID="task_201107062035_0006_m_000002" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0" START_TIME="1309989445566" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="SETUP" TASKID="task_201107062035_0006_m_000002" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000002_0" TASK_STATUS="SUCCESS" FINISH_TIME="1309989447674" HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal" STATE_STRING="setup" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Task TASKID="task_201107062035_0006_m_000002" TASK_TYPE="SETUP" TASK_STATUS="SUCCESS" FINISH_TIME="1309989450674" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Job JOBID="job_201107062035_0006" JOB_STATUS="RUNNING" .
    Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP" START_TIME="1309989450675" SPLITS="/default-rack/localhost" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0" START_TIME="1309989450701" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_0" TASK_STATUS="FAILED" FINISH_TIME="1309989457869" HOSTNAME="ip-10-36-79-98\.ec2\.internal" ERROR="java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1" START_TIME="1309989459732" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_1" TASK_STATUS="FAILED" FINISH_TIME="1309989467109" HOSTNAME="ip-10-36-79-98\.ec2\.internal" ERROR="java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2" START_TIME="1309989468821" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_2" TASK_STATUS="FAILED" FINISH_TIME="1309989477378" HOSTNAME="ip-10-36-79-98\.ec2\.internal" ERROR="java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3" START_TIME="1309989477905" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="MAP" TASKID="task_201107062035_0006_m_000000" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000000_3" TASK_STATUS="FAILED" FINISH_TIME="1309989484902" HOSTNAME="ip-10-36-79-98\.ec2\.internal" ERROR="java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " .
    Task TASKID="task_201107062035_0006_m_000000" TASK_TYPE="MAP" TASK_STATUS="FAILED" FINISH_TIME="1309989484902" ERROR="java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:359)
    at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:312)
    at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 5 more
    Caused by: java\.lang\.RuntimeException: Error in configuring object
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:93)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.java:64)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\.java:117)
    at org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
    \.\.\. 10 more
    Caused by: java\.lang\.reflect\.InvocationTargetException
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native Method)
    at sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\.java:39)
    at sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAccessorImpl\.java:25)
    at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
    at org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.java:88)
    \.\.\. 13 more
    Caused by: java\.lang\.RuntimeException: Map operator initialization failed
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:110)
    \.\.\. 18 more
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:291)
    at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.configure(ExecMapper\.java:80)
    \.\.\. 18 more
    Caused by: java\.lang\.ClassNotFoundException: org\.apache\.hadoop\.hive\.contrib\.serde2\.RegexSerDe
    at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
    at java\.security\.AccessController\.doPrivileged(Native Method)
    at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:307)
    at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
    at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:248)
    at java\.lang\.Class\.forName0(Native Method)
    at java\.lang\.Class\.forName(Class\.java:247)
    at org\.apache\.hadoop\.conf\.Configuration\.getClassByName(Configuration\.java:778)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.initObjectInspector(MapOperator\.java:181)
    at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.setChildren(MapOperator\.java:249)
    \.\.\. 19 more
    " TASK_ATTEMPT_ID="" .
    Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP" START_TIME="1309989486882" SPLITS="" .
    MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201107062035_0006_m_000001" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0" START_TIME="1309989486898" TRACKER_NAME="tracker_ip-10-36-79-98\.ec2\.internal:localhost\.localdomain/127\.0\.0\.1:51327" HTTP_PORT="9103" .
    MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201107062035_0006_m_000001" TASK_ATTEMPT_ID="attempt_201107062035_0006_m_000001_0" TASK_STATUS="SUCCESS" FINISH_TIME="1309989489057" HOSTNAME="/default-rack/ip-10-36-79-98\.ec2\.internal" STATE_STRING="cleanup" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Task TASKID="task_201107062035_0006_m_000001" TASK_TYPE="CLEANUP" TASK_STATUS="SUCCESS" FINISH_TIME="1309989489906" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
    Job JOBID="job_201107062035_0006" FINISH_TIME="1309989489907" JOB_STATUS="FAILED" FINISHED_MAPS="0" FINISHED_REDUCES="0" .

    Any help is appreciated.

    -Sal


    --
    - Prash
  • Jonathan Hwang at Jul 8, 2011 at 1:07 am
    Hi there,
    I got my hadoop all setup writing out sequence file with LZO compression. Using the following:
    mapred.output.compress=true
    mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec

    How do I define my table so it will write out compressed data and be able to read in compressed data during my Hive queries?

    CREATE EXTERNAL TABLE foo (
    columnA string,
    columnB string )
    PARTITIONED BY (date string)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
    LOCATION '/path/to/hive/tables/foo';

    Thanks,
    Jon

    This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
  • Aaron Sun at Jul 8, 2011 at 1:18 am
    You can use this one

    STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
    OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"

    On Thu, Jul 7, 2011 at 6:06 PM, wrote:

    Hi there,
    I got my hadoop all setup writing out sequence file with LZO compression.
    Using the following:
    mapred.output.compress=true
    mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec

    How do I define my table so it will write out compressed data and be able
    to read in compressed data during my Hive queries?

    CREATE EXTERNAL TABLE foo (
    columnA string,
    columnB string )
    PARTITIONED BY (date string)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
    LOCATION '/path/to/hive/tables/foo';

    Thanks,
    Jon

    This message is for the designated recipient only and may contain
    privileged, proprietary, or otherwise private information. If you have
    received it in error, please notify the sender immediately and delete the
    original. Any other use of the email by you is prohibited.
  • Steven Wong at Jul 8, 2011 at 1:42 am
    When writing, set hive.exec.compress.output=true also.

    When reading, nothing special needs to be done.


    -----Original Message-----
    From: jonathan.hwang@accenture.com
    Sent: Thursday, July 07, 2011 6:07 PM
    To: user@hive.apache.org
    Subject: Lzo compression on Hive table

    Hi there,
    I got my hadoop all setup writing out sequence file with LZO compression. Using the following:
    mapred.output.compress=true
    mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec

    How do I define my table so it will write out compressed data and be able to read in compressed data during my Hive queries?

    CREATE EXTERNAL TABLE foo (
    columnA string,
    columnB string )
    PARTITIONED BY (date string)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
    LOCATION '/path/to/hive/tables/foo';

    Thanks,
    Jon

    This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedJul 6, '11 at 10:13p
activeJul 8, '11 at 1:42a
posts8
users6
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase