FAQ
streaming job with Control characters in the command causes runtime exception in the job tracker
------------------------------------------------------------------------------------------------

Key: HADOOP-709
URL: http://issues.apache.org/jira/browse/HADOOP-709
Project: Hadoop
Issue Type: Bug
Components: contrib/streaming
Reporter: dhruba borthakur
Assigned To: dhruba borthakur


I run a streaming job with a Ctrl-A character in the command line option as follows:

$HSTREAMING -mapper "/bin/sort -t'^A'" -input "input/*" -output output

It causes a exeception as listed below. If i run the same command without the Ctrl-A character the command completes successfully.

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException: org.xml.sax.SAXParseException: Character reference "&#1" is an invalid XML character.
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:551)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:472)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:453)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:201)
at org.apache.hadoop.mapred.JobConf.getUser(JobConf.java:175)
at org.apache.hadoop.mapred.JobInProgress.(JobTracker.java:1016)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:514)

at org.apache.hadoop.ipc.Client$Connection.run(Client.java:248)



--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • dhruba borthakur (JIRA) at Nov 11, 2006 at 12:40 am
    [ http://issues.apache.org/jira/browse/HADOOP-709?page=comments#action_12448910 ]

    dhruba borthakur commented on HADOOP-709:
    -----------------------------------------

    This seems be a documented bug with jdk 1.5:

    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6470090

    streaming job with Control characters in the command causes runtime exception in the job tracker
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-709
    URL: http://issues.apache.org/jira/browse/HADOOP-709
    Project: Hadoop
    Issue Type: Bug
    Components: contrib/streaming
    Reporter: dhruba borthakur
    Assigned To: dhruba borthakur

    I run a streaming job with a Ctrl-A character in the command line option as follows:
    $HSTREAMING -mapper "/bin/sort -t'^A'" -input "input/*" -output output
    It causes a exeception as listed below. If i run the same command without the Ctrl-A character the command completes successfully.
    Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException: org.xml.sax.SAXParseException: Character reference "&#1" is an invalid XML character.
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:551)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:472)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:453)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:201)
    at org.apache.hadoop.mapred.JobConf.getUser(JobConf.java:175)
    at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:88)
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1016)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:514)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:248)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • dhruba borthakur (JIRA) at Nov 11, 2006 at 1:52 am
    [ http://issues.apache.org/jira/browse/HADOOP-709?page=comments#action_12448922 ]

    dhruba borthakur commented on HADOOP-709:
    -----------------------------------------

    Another sun issue:
    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6371411

    Our streaming code creates an XML file for the job. This file as an entry of the form:
    <property><name>stream.map.streamprocessor</name><value>/bin/sort -t'&#1;'</value></property>

    One option would be that user-input be encoded in Base64 before being XML-ised. This is necessary to support non-ascii user-input.
    streaming job with Control characters in the command causes runtime exception in the job tracker
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-709
    URL: http://issues.apache.org/jira/browse/HADOOP-709
    Project: Hadoop
    Issue Type: Bug
    Components: contrib/streaming
    Reporter: dhruba borthakur
    Assigned To: dhruba borthakur

    I run a streaming job with a Ctrl-A character in the command line option as follows:
    $HSTREAMING -mapper "/bin/sort -t'^A'" -input "input/*" -output output
    It causes a exeception as listed below. If i run the same command without the Ctrl-A character the command completes successfully.
    Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException: org.xml.sax.SAXParseException: Character reference "&#1" is an invalid XML character.
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:551)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:472)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:453)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:201)
    at org.apache.hadoop.mapred.JobConf.getUser(JobConf.java:175)
    at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:88)
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1016)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:514)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:248)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • dhruba borthakur (JIRA) at Nov 18, 2006 at 12:08 am
    [ http://issues.apache.org/jira/browse/HADOOP-709?page=all ]

    dhruba borthakur updated HADOOP-709:
    ------------------------------------

    Attachment: ctrlchars_in_streaming

    URLEncode the streaming related key-value pairs in the job.xml file.
    stream.map.streamprocessor
    stream.reduce.streamprocessor
    stream.combine.streamprocessor


    streaming job with Control characters in the command causes runtime exception in the job tracker
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-709
    URL: http://issues.apache.org/jira/browse/HADOOP-709
    Project: Hadoop
    Issue Type: Bug
    Components: contrib/streaming
    Reporter: dhruba borthakur
    Assigned To: dhruba borthakur
    Attachments: ctrlchars_in_streaming


    I run a streaming job with a Ctrl-A character in the command line option as follows:
    $HSTREAMING -mapper "/bin/sort -t'^A'" -input "input/*" -output output
    It causes a exeception as listed below. If i run the same command without the Ctrl-A character the command completes successfully.
    Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException: org.xml.sax.SAXParseException: Character reference "&#1" is an invalid XML character.
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:551)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:472)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:453)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:201)
    at org.apache.hadoop.mapred.JobConf.getUser(JobConf.java:175)
    at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:88)
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1016)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:514)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:248)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • dhruba borthakur (JIRA) at Nov 18, 2006 at 12:10 am
    [ http://issues.apache.org/jira/browse/HADOOP-709?page=all ]

    dhruba borthakur updated HADOOP-709:
    ------------------------------------

    Status: Patch Available (was: Open)

    URLEncode the streaming related key-value pairs in the job.xml file.
    stream.map.streamprocessor
    stream.reduce.streamprocessor
    stream.combine.streamprocessor

    Reviewed: mahadev.
    streaming job with Control characters in the command causes runtime exception in the job tracker
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-709
    URL: http://issues.apache.org/jira/browse/HADOOP-709
    Project: Hadoop
    Issue Type: Bug
    Components: contrib/streaming
    Reporter: dhruba borthakur
    Assigned To: dhruba borthakur
    Attachments: ctrlchars_in_streaming


    I run a streaming job with a Ctrl-A character in the command line option as follows:
    $HSTREAMING -mapper "/bin/sort -t'^A'" -input "input/*" -output output
    It causes a exeception as listed below. If i run the same command without the Ctrl-A character the command completes successfully.
    Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException: org.xml.sax.SAXParseException: Character reference "&#1" is an invalid XML character.
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:551)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:472)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:453)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:201)
    at org.apache.hadoop.mapred.JobConf.getUser(JobConf.java:175)
    at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:88)
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1016)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:514)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:248)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at Nov 21, 2006 at 12:19 am
    [ http://issues.apache.org/jira/browse/HADOOP-709?page=all ]

    Doug Cutting updated HADOOP-709:
    --------------------------------

    Status: Resolved (was: Patch Available)
    Fix Version/s: 0.9.0
    Resolution: Fixed

    I just committed this. Thanks, Dhruba!
    streaming job with Control characters in the command causes runtime exception in the job tracker
    ------------------------------------------------------------------------------------------------

    Key: HADOOP-709
    URL: http://issues.apache.org/jira/browse/HADOOP-709
    Project: Hadoop
    Issue Type: Bug
    Components: contrib/streaming
    Reporter: dhruba borthakur
    Assigned To: dhruba borthakur
    Fix For: 0.9.0

    Attachments: ctrlchars_in_streaming


    I run a streaming job with a Ctrl-A character in the command line option as follows:
    $HSTREAMING -mapper "/bin/sort -t'^A'" -input "input/*" -output output
    It causes a exeception as listed below. If i run the same command without the Ctrl-A character the command completes successfully.
    Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException: org.xml.sax.SAXParseException: Character reference "&#1" is an invalid XML character.
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:551)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:472)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:453)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:201)
    at org.apache.hadoop.mapred.JobConf.getUser(JobConf.java:175)
    at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:88)
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1016)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:514)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:248)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedNov 10, '06 at 10:35p
activeNov 21, '06 at 12:19a
posts6
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Doug Cutting (JIRA): 6 posts

People

Translate

site design / logo © 2022 Grokbase