FAQ
Hello,
I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

I've recently upgraded to CM4 and everything went fine except the host
inspector fails on all instances with the following errors:

- host1: org.codehaus.jackson.JsonParseException: Unexpected character
('I' (code 73)): expected a valid value (number, String, array, object,
'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26; line:
1, column: 2]
- host2: org.codehaus.jackson.JsonParseException: Unexpected character
('I' (code 73)): expected a valid value (number, String, array, object,
'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17; line:
1, column: 2]
- host3: org.codehaus.jackson.JsonParseException: Unexpected character
('I' (code 73)): expected a valid value (number, String, array, object,
'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc; line:
1, column: 2]
- host4: org.codehaus.jackson.JsonParseException: Unexpected character
('I' (code 73)): expected a valid value (number, String, array, object,
'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63; line:
1, column: 2]
- host5: org.codehaus.jackson.JsonParseException: Unexpected character
('I' (code 73)): expected a valid value (number, String, array, object,
'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3; line:
1, column: 2]

The hosts are communicating with the server and on the host status page I
see that the nodes reply to heart beats.

Looking at the cloudera-scm-server.log I also see these lines (only a
summary):
2012-07-04 11:14:02,291 INFO
[2131163116@scm-web-48:service.ServiceHandlerRegistry@411] Executing
command Inspector [].
2012-07-04 11:14:02,293 INFO
[2131163116@scm-web-48:service.ServiceHandlerRegistry@593] Executing host
command hostInspector []. Host: DbHost{id=5, hostId=host1,
rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
configContainer=ALL_HOSTS, configGener
ation=1}
2012-07-04 11:14:02,985 WARN
[44314049@agentServer-13:cmf.AgentProtocolImpl@273] Received Process
Heartbeat for unknown (or duplicate) process. Ignoring.id=338 name=host-
inspector host=host
2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
Command (Inspector, 966) is still active on service null.
2012-07-04 11:14:18,015 INFO
[CommandPusher:service.AbstractOneOffHostCommand@127] Successful
'hostInspector'
2012-07-04 11:14:24,317 ERROR
[CommandPusher:inspector.InspectorCommand@122] Unexpected error
deserializing child output: org.codehaus.jackson.JsonParseException:
Unexpecte
d character ('I' (code 73)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
73)): expected a valid value (number, String
, array, object, 'true', 'false' or 'null')
at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
at
com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
at
com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
at
com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
at
com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
at
com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
at
com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
at
com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
at
com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
at
com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character
('I' (code 73)): expected a valid value (number, String, array, object,
'true', 'false' or 'null')
at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
at
org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
at
org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
at
org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
at
org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
at
org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
at
com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
... 8 more

Any ideas?

Thansk!

Search Discussions

  • bc Wong at Jul 5, 2012 at 3:59 pm

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann wrote:

    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the host
    inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status page I
    see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command hostInspector
    []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown (or
    duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
    Command (Inspector, 966) is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR [CommandPusher:inspector.InspectorCommand@122]
    Unexpected error deserializing child output:
    org.codehaus.jackson.JsonParseException: Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on each
    of the 5 hosts. And they each produce a JSON result string, which seems to
    be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't completed
    yet. Wait another minute and retry.) Could you please hit that result data
    url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer
  • Galed Friedmann at Jul 5, 2012 at 4:30 pm
    Hi,
    Checking the command ID returns this:
    {
    "id" : 1084,
    "name" : "hostInspector",
    "startTime" : "2012-07-05T16:15:07.190Z",
    "endTime" : "2012-07-05T16:15:22.862Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Inspector ran successfully.",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1084/download",
    "hostRef" : {
    "hostId" : "hadoop-node"
    },
    "parent" : {
    "id" : 1085,
    "name" : "Inspector",
    "startTime" : "2012-07-05T16:15:07.109Z",
    "endTime" : "2012-07-05T16:15:34.677Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Command completed with 5/5 successful subcommands",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1085/download"
    },
    "children" : {
    "items" : [ ]
    }
    }

    I tried both URLs, the first one (1084) returned this:
    Invalid request. Process: 393-host-inspector files ('inspector',)

    The second URL (1085) returned this:

    {
    "allHostsDnsMessages" : [ ],
    "etcHostsMessages" : [ ],
    "failedHostsWithError" : {
    "ip-10-7-29-79.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@38f6ac70; line: 1,
    column: 2]",
    "ip-10-73-94-108.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ab30a47; line: 1,
    column: 2]",
    "ip-10-47-54-22.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@3067e1a1; line: 1,
    column: 2]",
    "ip-10-64-165-124.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@740e29da; line: 1,
    column: 2]",
    "domU-12-31-39-15-64-0F.compute-1.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ddde2bc; line: 1,
    column: 2]"
    },
    "failedLocalhostSanityMessages" : [ ],
    "hdfsUidsInconsistent" : null,
    "hostDnsChecksSuccesses" : 0,
    "hostDnsMessages" : [ ],
    "kernelVersionErrors" : [ ],
    "otherErrors" : [ ],
    "successfulSubcommands" : 0,
    "timeInspectionMessage" : null,
    "timeInspectionTimezoneMessage" : null,
    "userGroupErrors" : [ ],
    "versionHostsByCDHVersion" : [ {
    "hostCDHVersion" : "NONE",
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionHostsByComponentSet" : [ {
    "componentInfo" : [ ],
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionMessages" : [ {
    "annotation" : "CONCERNING",
    "args" : [ "5", "NONE" ],
    "messageId" : "message.inspector.version.nHostsConcerningCdhVersion"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.oneComponentSet"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.daemonsVersionsMatchServer"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.agentVersionsMatchServer"
    } ]
    }
    On Thu, Jul 5, 2012 at 6:59 PM, bc Wong wrote:

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann <galed.friedmann@onavo.com
    wrote:
    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the host
    inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status page I
    see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command
    hostInspector []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown (or
    duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
    Command (Inspector, 966) is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR
    [CommandPusher:inspector.InspectorCommand@122] Unexpected error
    deserializing child output: org.codehaus.jackson.JsonParseException:
    Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on each
    of the 5 hosts. And they each produce a JSON result string, which seems to
    be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't
    completed yet. Wait another minute and retry.) Could you please hit that
    result data url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer
  • Galed Friedmann at Jul 12, 2012 at 7:00 am
    Hi,
    Any ideas maybe?

    We want to upgrade to CDH4 using CM and it's quite difficult because it
    can't detect the Hadoop version..

    Thanks again

    On Thu, Jul 5, 2012 at 7:30 PM, Galed Friedmann
    wrote:
    Hi,
    Checking the command ID returns this:
    {
    "id" : 1084,
    "name" : "hostInspector",
    "startTime" : "2012-07-05T16:15:07.190Z",
    "endTime" : "2012-07-05T16:15:22.862Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Inspector ran successfully.",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1084/download",
    "hostRef" : {
    "hostId" : "hadoop-node"
    },
    "parent" : {
    "id" : 1085,
    "name" : "Inspector",
    "startTime" : "2012-07-05T16:15:07.109Z",
    "endTime" : "2012-07-05T16:15:34.677Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Command completed with 5/5 successful subcommands",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1085/download
    "
    },
    "children" : {
    "items" : [ ]
    }
    }

    I tried both URLs, the first one (1084) returned this:
    Invalid request. Process: 393-host-inspector files ('inspector',)

    The second URL (1085) returned this:

    {
    "allHostsDnsMessages" : [ ],
    "etcHostsMessages" : [ ],
    "failedHostsWithError" : {
    "ip-10-7-29-79.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@38f6ac70; line: 1,
    column: 2]",
    "ip-10-73-94-108.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ab30a47; line: 1,
    column: 2]",
    "ip-10-47-54-22.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@3067e1a1; line: 1,
    column: 2]",
    "ip-10-64-165-124.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@740e29da; line: 1,
    column: 2]",
    "domU-12-31-39-15-64-0F.compute-1.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ddde2bc; line: 1,
    column: 2]"
    },
    "failedLocalhostSanityMessages" : [ ],
    "hdfsUidsInconsistent" : null,
    "hostDnsChecksSuccesses" : 0,
    "hostDnsMessages" : [ ],
    "kernelVersionErrors" : [ ],
    "otherErrors" : [ ],
    "successfulSubcommands" : 0,
    "timeInspectionMessage" : null,
    "timeInspectionTimezoneMessage" : null,
    "userGroupErrors" : [ ],
    "versionHostsByCDHVersion" : [ {
    "hostCDHVersion" : "NONE",
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionHostsByComponentSet" : [ {
    "componentInfo" : [ ],
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionMessages" : [ {
    "annotation" : "CONCERNING",
    "args" : [ "5", "NONE" ],
    "messageId" : "message.inspector.version.nHostsConcerningCdhVersion"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.oneComponentSet"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.daemonsVersionsMatchServer"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.agentVersionsMatchServer"
    } ]
    }
    On Thu, Jul 5, 2012 at 6:59 PM, bc Wong wrote:

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the host
    inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status page
    I see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command
    hostInspector []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown (or
    duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
    Command (Inspector, 966) is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR
    [CommandPusher:inspector.InspectorCommand@122] Unexpected error
    deserializing child output: org.codehaus.jackson.JsonParseException:
    Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on
    each of the 5 hosts. And they each produce a JSON result string, which
    seems to be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't
    completed yet. Wait another minute and retry.) Could you please hit that
    result data url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer
  • Philip Zeyliger at Jul 18, 2012 at 6:50 pm
    You can continue on with the upgrade even if there are host inspector
    failures. If you're comfortable with sharing it, to debug this further,
    we'd want the server log (/var/log/cloudera-scm-server/*), and possibly a
    copy of your database (pgdump).

    -- Philip

    On Thu, Jul 12, 2012 at 12:00 AM, Galed Friedmann wrote:

    Hi,
    Any ideas maybe?

    We want to upgrade to CDH4 using CM and it's quite difficult because it
    can't detect the Hadoop version..

    Thanks again


    On Thu, Jul 5, 2012 at 7:30 PM, Galed Friedmann <galed.friedmann@onavo.com
    wrote:
    Hi,
    Checking the command ID returns this:
    {
    "id" : 1084,
    "name" : "hostInspector",
    "startTime" : "2012-07-05T16:15:07.190Z",
    "endTime" : "2012-07-05T16:15:22.862Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Inspector ran successfully.",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1084/download
    ",
    "hostRef" : {
    "hostId" : "hadoop-node"
    },
    "parent" : {
    "id" : 1085,
    "name" : "Inspector",
    "startTime" : "2012-07-05T16:15:07.109Z",
    "endTime" : "2012-07-05T16:15:34.677Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Command completed with 5/5 successful subcommands",
    "resultDataUrl" : "
    http://hadoop-master:7180/cmf/command/1085/download"
    },
    "children" : {
    "items" : [ ]
    }
    }

    I tried both URLs, the first one (1084) returned this:
    Invalid request. Process: 393-host-inspector files ('inspector',)

    The second URL (1085) returned this:

    {
    "allHostsDnsMessages" : [ ],
    "etcHostsMessages" : [ ],
    "failedHostsWithError" : {
    "ip-10-7-29-79.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@38f6ac70; line: 1,
    column: 2]",
    "ip-10-73-94-108.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ab30a47; line: 1,
    column: 2]",
    "ip-10-47-54-22.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@3067e1a1; line: 1,
    column: 2]",
    "ip-10-64-165-124.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@740e29da; line: 1,
    column: 2]",
    "domU-12-31-39-15-64-0F.compute-1.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ddde2bc; line: 1,
    column: 2]"
    },
    "failedLocalhostSanityMessages" : [ ],
    "hdfsUidsInconsistent" : null,
    "hostDnsChecksSuccesses" : 0,
    "hostDnsMessages" : [ ],
    "kernelVersionErrors" : [ ],
    "otherErrors" : [ ],
    "successfulSubcommands" : 0,
    "timeInspectionMessage" : null,
    "timeInspectionTimezoneMessage" : null,
    "userGroupErrors" : [ ],
    "versionHostsByCDHVersion" : [ {
    "hostCDHVersion" : "NONE",
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionHostsByComponentSet" : [ {
    "componentInfo" : [ ],
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionMessages" : [ {
    "annotation" : "CONCERNING",
    "args" : [ "5", "NONE" ],
    "messageId" : "message.inspector.version.nHostsConcerningCdhVersion"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.oneComponentSet"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.daemonsVersionsMatchServer"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.agentVersionsMatchServer"
    } ]
    }
    On Thu, Jul 5, 2012 at 6:59 PM, bc Wong wrote:

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the host
    inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status page
    I see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command
    hostInspector []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown (or
    duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
    Command (Inspector, 966) is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR
    [CommandPusher:inspector.InspectorCommand@122] Unexpected error
    deserializing child output: org.codehaus.jackson.JsonParseException:
    Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on
    each of the 5 hosts. And they each produce a JSON result string, which
    seems to be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't
    completed yet. Wait another minute and retry.) Could you please hit that
    result data url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer
  • Galed Friedmann at Jul 19, 2012 at 6:55 am
    You mean to continue the upgrade from the command line and not through the
    manager itself? Because trying to complete the upgrade using the manager
    gives me an error that it can't determine the version of my CDH cluster.

    Thanks again
    On Wed, Jul 18, 2012 at 9:50 PM, Philip Zeyliger wrote:

    You can continue on with the upgrade even if there are host inspector
    failures. If you're comfortable with sharing it, to debug this further,
    we'd want the server log (/var/log/cloudera-scm-server/*), and possibly a
    copy of your database (pgdump).

    -- Philip



    On Thu, Jul 12, 2012 at 12:00 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hi,
    Any ideas maybe?

    We want to upgrade to CDH4 using CM and it's quite difficult because it
    can't detect the Hadoop version..

    Thanks again


    On Thu, Jul 5, 2012 at 7:30 PM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hi,
    Checking the command ID returns this:
    {
    "id" : 1084,
    "name" : "hostInspector",
    "startTime" : "2012-07-05T16:15:07.190Z",
    "endTime" : "2012-07-05T16:15:22.862Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Inspector ran successfully.",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1084/download
    ",
    "hostRef" : {
    "hostId" : "hadoop-node"
    },
    "parent" : {
    "id" : 1085,
    "name" : "Inspector",
    "startTime" : "2012-07-05T16:15:07.109Z",
    "endTime" : "2012-07-05T16:15:34.677Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Command completed with 5/5 successful
    subcommands",
    "resultDataUrl" : "
    http://hadoop-master:7180/cmf/command/1085/download"
    },
    "children" : {
    "items" : [ ]
    }
    }

    I tried both URLs, the first one (1084) returned this:
    Invalid request. Process: 393-host-inspector files ('inspector',)

    The second URL (1085) returned this:

    {
    "allHostsDnsMessages" : [ ],
    "etcHostsMessages" : [ ],
    "failedHostsWithError" : {
    "ip-10-7-29-79.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@38f6ac70; line:
    1, column: 2]",
    "ip-10-73-94-108.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ab30a47; line:
    1, column: 2]",
    "ip-10-47-54-22.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@3067e1a1; line:
    1, column: 2]",
    "ip-10-64-165-124.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@740e29da; line:
    1, column: 2]",
    "domU-12-31-39-15-64-0F.compute-1.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ddde2bc; line:
    1, column: 2]"
    },
    "failedLocalhostSanityMessages" : [ ],
    "hdfsUidsInconsistent" : null,
    "hostDnsChecksSuccesses" : 0,
    "hostDnsMessages" : [ ],
    "kernelVersionErrors" : [ ],
    "otherErrors" : [ ],
    "successfulSubcommands" : 0,
    "timeInspectionMessage" : null,
    "timeInspectionTimezoneMessage" : null,
    "userGroupErrors" : [ ],
    "versionHostsByCDHVersion" : [ {
    "hostCDHVersion" : "NONE",
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionHostsByComponentSet" : [ {
    "componentInfo" : [ ],
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionMessages" : [ {
    "annotation" : "CONCERNING",
    "args" : [ "5", "NONE" ],
    "messageId" : "message.inspector.version.nHostsConcerningCdhVersion"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.oneComponentSet"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.daemonsVersionsMatchServer"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.agentVersionsMatchServer"
    } ]
    }
    On Thu, Jul 5, 2012 at 6:59 PM, bc Wong wrote:

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the host
    inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status
    page I see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command
    hostInspector []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown
    (or duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
    Command (Inspector, 966) is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR
    [CommandPusher:inspector.InspectorCommand@122] Unexpected error
    deserializing child output: org.codehaus.jackson.JsonParseException:
    Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on
    each of the 5 hosts. And they each produce a JSON result string, which
    seems to be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't
    completed yet. Wait another minute and retry.) Could you please hit that
    result data url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer
  • bc Wong at Jul 19, 2012 at 7:33 am

    On Thu, Jul 5, 2012 at 9:30 AM, Galed Friedmann wrote:

    Hi,
    Checking the command ID returns this:
    {
    "id" : 1084,
    "name" : "hostInspector",
    "startTime" : "2012-07-05T16:15:07.190Z",
    "endTime" : "2012-07-05T16:15:22.862Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Inspector ran successfully.",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1084/download",
    "hostRef" : {
    "hostId" : "hadoop-node"
    },
    "parent" : {
    "id" : 1085,
    "name" : "Inspector",
    "startTime" : "2012-07-05T16:15:07.109Z",
    "endTime" : "2012-07-05T16:15:34.677Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Command completed with 5/5 successful subcommands",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1085/download
    "
    },
    "children" : {
    "items" : [ ]
    }
    }

    I tried both URLs, the first one (1084) returned this:
    Invalid request. Process: 393-host-inspector files ('inspector',)
    Hi Galed,

    So that's where the "unexpected character" comes from. It's trying to parse
    the "Invalid request ..." string as JSON.

    Host inspector runs an inspection process on each node. That per-node
    inspection is supposed to generate a JSON result. I'm guessing that things
    went really wrong and that inspection crashed. (It's probably something to
    do with the host environment.)

    You should find some
    "/var/run/cloudera-scm-agent/process/*-host-inspector/logs" directory on
    any node. (If not, trigger the upgrade again.) Could you please attach the
    two log files in that directory?

    Cheers,
    bc

    The second URL (1085) returned this:

    {
    "allHostsDnsMessages" : [ ],
    "etcHostsMessages" : [ ],
    "failedHostsWithError" : {
    "ip-10-7-29-79.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@38f6ac70; line: 1,
    column: 2]",
    "ip-10-73-94-108.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ab30a47; line: 1,
    column: 2]",
    "ip-10-47-54-22.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@3067e1a1; line: 1,
    column: 2]",
    "ip-10-64-165-124.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@740e29da; line: 1,
    column: 2]",
    "domU-12-31-39-15-64-0F.compute-1.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ddde2bc; line: 1,
    column: 2]"
    },
    "failedLocalhostSanityMessages" : [ ],
    "hdfsUidsInconsistent" : null,
    "hostDnsChecksSuccesses" : 0,
    "hostDnsMessages" : [ ],
    "kernelVersionErrors" : [ ],
    "otherErrors" : [ ],
    "successfulSubcommands" : 0,
    "timeInspectionMessage" : null,
    "timeInspectionTimezoneMessage" : null,
    "userGroupErrors" : [ ],
    "versionHostsByCDHVersion" : [ {
    "hostCDHVersion" : "NONE",
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionHostsByComponentSet" : [ {
    "componentInfo" : [ ],
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionMessages" : [ {
    "annotation" : "CONCERNING",
    "args" : [ "5", "NONE" ],
    "messageId" : "message.inspector.version.nHostsConcerningCdhVersion"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.oneComponentSet"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.daemonsVersionsMatchServer"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.agentVersionsMatchServer"
    } ]
    }
    On Thu, Jul 5, 2012 at 6:59 PM, bc Wong wrote:

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the host
    inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status page
    I see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command
    hostInspector []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown (or
    duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
    Command (Inspector, 966) is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR
    [CommandPusher:inspector.InspectorCommand@122] Unexpected error
    deserializing child output: org.codehaus.jackson.JsonParseException:
    Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected character
    ('I' (code 73)): expected a valid value (number, String, array, object,
    'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on
    each of the 5 hosts. And they each produce a JSON result string, which
    seems to be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't
    completed yet. Wait another minute and retry.) Could you please hit that
    result data url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer
  • Galed Friedmann at Jul 19, 2012 at 9:26 am
    Attached

    Thanks for the help
    On Thu, Jul 19, 2012 at 10:32 AM, bc Wong wrote:

    On Thu, Jul 5, 2012 at 9:30 AM, Galed Friedmann <galed.friedmann@onavo.com
    wrote:
    Hi,
    Checking the command ID returns this:
    {
    "id" : 1084,
    "name" : "hostInspector",
    "startTime" : "2012-07-05T16:15:07.190Z",
    "endTime" : "2012-07-05T16:15:22.862Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Inspector ran successfully.",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1084/download
    ",
    "hostRef" : {
    "hostId" : "hadoop-node"
    },
    "parent" : {
    "id" : 1085,
    "name" : "Inspector",
    "startTime" : "2012-07-05T16:15:07.109Z",
    "endTime" : "2012-07-05T16:15:34.677Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Command completed with 5/5 successful subcommands",
    "resultDataUrl" : "
    http://hadoop-master:7180/cmf/command/1085/download"
    },
    "children" : {
    "items" : [ ]
    }
    }

    I tried both URLs, the first one (1084) returned this:
    Invalid request. Process: 393-host-inspector files ('inspector',)
    Hi Galed,

    So that's where the "unexpected character" comes from. It's trying to
    parse the "Invalid request ..." string as JSON.

    Host inspector runs an inspection process on each node. That per-node
    inspection is supposed to generate a JSON result. I'm guessing that things
    went really wrong and that inspection crashed. (It's probably something to
    do with the host environment.)

    You should find some
    "/var/run/cloudera-scm-agent/process/*-host-inspector/logs" directory on
    any node. (If not, trigger the upgrade again.) Could you please attach the
    two log files in that directory?

    Cheers,
    bc

    The second URL (1085) returned this:

    {
    "allHostsDnsMessages" : [ ],
    "etcHostsMessages" : [ ],
    "failedHostsWithError" : {
    "ip-10-7-29-79.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@38f6ac70; line: 1,
    column: 2]",
    "ip-10-73-94-108.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ab30a47; line: 1,
    column: 2]",
    "ip-10-47-54-22.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@3067e1a1; line: 1,
    column: 2]",
    "ip-10-64-165-124.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@740e29da; line: 1,
    column: 2]",
    "domU-12-31-39-15-64-0F.compute-1.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ddde2bc; line: 1,
    column: 2]"
    },
    "failedLocalhostSanityMessages" : [ ],
    "hdfsUidsInconsistent" : null,
    "hostDnsChecksSuccesses" : 0,
    "hostDnsMessages" : [ ],
    "kernelVersionErrors" : [ ],
    "otherErrors" : [ ],
    "successfulSubcommands" : 0,
    "timeInspectionMessage" : null,
    "timeInspectionTimezoneMessage" : null,
    "userGroupErrors" : [ ],
    "versionHostsByCDHVersion" : [ {
    "hostCDHVersion" : "NONE",
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionHostsByComponentSet" : [ {
    "componentInfo" : [ ],
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionMessages" : [ {
    "annotation" : "CONCERNING",
    "args" : [ "5", "NONE" ],
    "messageId" : "message.inspector.version.nHostsConcerningCdhVersion"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.oneComponentSet"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.daemonsVersionsMatchServer"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.agentVersionsMatchServer"
    } ]
    }
    On Thu, Jul 5, 2012 at 6:59 PM, bc Wong wrote:

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the host
    inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status page
    I see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command
    hostInspector []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown (or
    duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
    Command (Inspector, 966) is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR
    [CommandPusher:inspector.InspectorCommand@122] Unexpected error
    deserializing child output: org.codehaus.jackson.JsonParseException:
    Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on
    each of the 5 hosts. And they each produce a JSON result string, which
    seems to be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't
    completed yet. Wait another minute and retry.) Could you please hit that
    result data url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer
  • bc Wong at Jul 19, 2012 at 5:29 pm
    Could it be possible that you're running an old version of CM agents?

    If it's not that, then could you run the host inspector again, and attach
    the logs from `/var/log/cloudera-scm-agent/'? Any node will do.

    Thanks,
    bc


    On Thu, Jul 19, 2012 at 2:26 AM, Galed Friedmann
    wrote:
    Attached

    Thanks for the help

    On Thu, Jul 19, 2012 at 10:32 AM, bc Wong wrote:

    On Thu, Jul 5, 2012 at 9:30 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hi,
    Checking the command ID returns this:
    {
    "id" : 1084,
    "name" : "hostInspector",
    "startTime" : "2012-07-05T16:15:07.190Z",
    "endTime" : "2012-07-05T16:15:22.862Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Inspector ran successfully.",
    "resultDataUrl" : "http://hadoop-master:7180/cmf/command/1084/download
    ",
    "hostRef" : {
    "hostId" : "hadoop-node"
    },
    "parent" : {
    "id" : 1085,
    "name" : "Inspector",
    "startTime" : "2012-07-05T16:15:07.109Z",
    "endTime" : "2012-07-05T16:15:34.677Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Command completed with 5/5 successful
    subcommands",
    "resultDataUrl" : "
    http://hadoop-master:7180/cmf/command/1085/download"
    },
    "children" : {
    "items" : [ ]
    }
    }

    I tried both URLs, the first one (1084) returned this:
    Invalid request. Process: 393-host-inspector files ('inspector',)
    Hi Galed,

    So that's where the "unexpected character" comes from. It's trying to
    parse the "Invalid request ..." string as JSON.

    Host inspector runs an inspection process on each node. That per-node
    inspection is supposed to generate a JSON result. I'm guessing that things
    went really wrong and that inspection crashed. (It's probably something to
    do with the host environment.)

    You should find some
    "/var/run/cloudera-scm-agent/process/*-host-inspector/logs" directory on
    any node. (If not, trigger the upgrade again.) Could you please attach the
    two log files in that directory?

    Cheers,
    bc

    The second URL (1085) returned this:

    {
    "allHostsDnsMessages" : [ ],
    "etcHostsMessages" : [ ],
    "failedHostsWithError" : {
    "ip-10-7-29-79.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@38f6ac70; line:
    1, column: 2]",
    "ip-10-73-94-108.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ab30a47; line:
    1, column: 2]",
    "ip-10-47-54-22.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@3067e1a1; line:
    1, column: 2]",
    "ip-10-64-165-124.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@740e29da; line:
    1, column: 2]",
    "domU-12-31-39-15-64-0F.compute-1.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ddde2bc; line:
    1, column: 2]"
    },
    "failedLocalhostSanityMessages" : [ ],
    "hdfsUidsInconsistent" : null,
    "hostDnsChecksSuccesses" : 0,
    "hostDnsMessages" : [ ],
    "kernelVersionErrors" : [ ],
    "otherErrors" : [ ],
    "successfulSubcommands" : 0,
    "timeInspectionMessage" : null,
    "timeInspectionTimezoneMessage" : null,
    "userGroupErrors" : [ ],
    "versionHostsByCDHVersion" : [ {
    "hostCDHVersion" : "NONE",
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionHostsByComponentSet" : [ {
    "componentInfo" : [ ],
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionMessages" : [ {
    "annotation" : "CONCERNING",
    "args" : [ "5", "NONE" ],
    "messageId" : "message.inspector.version.nHostsConcerningCdhVersion"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.oneComponentSet"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.daemonsVersionsMatchServer"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.agentVersionsMatchServer"
    } ]
    }
    On Thu, Jul 5, 2012 at 6:59 PM, bc Wong wrote:

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the host
    inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status
    page I see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command
    hostInspector []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown
    (or duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO [CommandPusher:command.CommandHelpers@191]
    Command (Inspector, 966) is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR
    [CommandPusher:inspector.InspectorCommand@122] Unexpected error
    deserializing child output: org.codehaus.jackson.JsonParseException:
    Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on
    each of the 5 hosts. And they each produce a JSON result string, which
    seems to be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't
    completed yet. Wait another minute and retry.) Could you please hit that
    result data url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer
  • Galed Friedmann at Jul 22, 2012 at 3:31 pm
    Well looks like that was the problem.
    The agents were upgraded using the upgrade wizard after I upgraded the
    master, but it seems that the wizard did not restart the agents and they
    were will running the old version. After I restarted all of the agents they
    now show their correct version and the host inspector is running as
    expected.

    Thanks for the help!
    On Thu, Jul 19, 2012 at 8:29 PM, bc Wong wrote:

    Could it be possible that you're running an old version of CM agents?

    If it's not that, then could you run the host inspector again, and attach
    the logs from `/var/log/cloudera-scm-agent/'? Any node will do.

    Thanks,
    bc


    On Thu, Jul 19, 2012 at 2:26 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Attached

    Thanks for the help

    On Thu, Jul 19, 2012 at 10:32 AM, bc Wong wrote:

    On Thu, Jul 5, 2012 at 9:30 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hi,
    Checking the command ID returns this:
    {
    "id" : 1084,
    "name" : "hostInspector",
    "startTime" : "2012-07-05T16:15:07.190Z",
    "endTime" : "2012-07-05T16:15:22.862Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Inspector ran successfully.",
    "resultDataUrl" : "
    http://hadoop-master:7180/cmf/command/1084/download",
    "hostRef" : {
    "hostId" : "hadoop-node"
    },
    "parent" : {
    "id" : 1085,
    "name" : "Inspector",
    "startTime" : "2012-07-05T16:15:07.109Z",
    "endTime" : "2012-07-05T16:15:34.677Z",
    "active" : false,
    "success" : true,
    "resultMessage" : "Command completed with 5/5 successful
    subcommands",
    "resultDataUrl" : "
    http://hadoop-master:7180/cmf/command/1085/download"
    },
    "children" : {
    "items" : [ ]
    }
    }

    I tried both URLs, the first one (1084) returned this:
    Invalid request. Process: 393-host-inspector files ('inspector',)
    Hi Galed,

    So that's where the "unexpected character" comes from. It's trying to
    parse the "Invalid request ..." string as JSON.

    Host inspector runs an inspection process on each node. That per-node
    inspection is supposed to generate a JSON result. I'm guessing that things
    went really wrong and that inspection crashed. (It's probably something to
    do with the host environment.)

    You should find some
    "/var/run/cloudera-scm-agent/process/*-host-inspector/logs" directory on
    any node. (If not, trigger the upgrade again.) Could you please attach the
    two log files in that directory?

    Cheers,
    bc

    The second URL (1085) returned this:

    {
    "allHostsDnsMessages" : [ ],
    "etcHostsMessages" : [ ],
    "failedHostsWithError" : {
    "ip-10-7-29-79.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@38f6ac70; line:
    1, column: 2]",
    "ip-10-73-94-108.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ab30a47; line:
    1, column: 2]",
    "ip-10-47-54-22.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@3067e1a1; line:
    1, column: 2]",
    "ip-10-64-165-124.ec2.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@740e29da; line:
    1, column: 2]",
    "domU-12-31-39-15-64-0F.compute-1.internal" :
    "org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String, array, object, 'true',
    'false' or 'null')\n at [Source: java.io.StringReader@4ddde2bc; line:
    1, column: 2]"
    },
    "failedLocalhostSanityMessages" : [ ],
    "hdfsUidsInconsistent" : null,
    "hostDnsChecksSuccesses" : 0,
    "hostDnsMessages" : [ ],
    "kernelVersionErrors" : [ ],
    "otherErrors" : [ ],
    "successfulSubcommands" : 0,
    "timeInspectionMessage" : null,
    "timeInspectionTimezoneMessage" : null,
    "userGroupErrors" : [ ],
    "versionHostsByCDHVersion" : [ {
    "hostCDHVersion" : "NONE",
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionHostsByComponentSet" : [ {
    "componentInfo" : [ ],
    "hosts" : [ "ip-10-7-29-79.ec2.internal",
    "ip-10-73-94-108.ec2.internal", "ip-10-47-54-22.ec2.internal",
    "ip-10-64-165-124.ec2.internal",
    "domU-12-31-39-15-64-0F.compute-1.internal" ]
    } ],
    "versionMessages" : [ {
    "annotation" : "CONCERNING",
    "args" : [ "5", "NONE" ],
    "messageId" : "message.inspector.version.nHostsConcerningCdhVersion"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.oneComponentSet"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.daemonsVersionsMatchServer"
    }, {
    "annotation" : "GOOD",
    "args" : [ ],
    "messageId" : "message.inspector.version.agentVersionsMatchServer"
    } ]
    }
    On Thu, Jul 5, 2012 at 6:59 PM, bc Wong wrote:

    On Wed, Jul 4, 2012 at 4:22 AM, Galed Friedmann <
    galed.friedmann@onavo.com> wrote:
    Hello,
    I'm running a cluster with 5 nodes, running CM4 and CDH3u4.

    I've recently upgraded to CM4 and everything went fine except the
    host inspector fails on all instances with the following errors:

    - host1: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@46c70d26;
    line: 1, column: 2]
    - host2: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@36305e17;
    line: 1, column: 2]
    - host3: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@217153cc;
    line: 1, column: 2]
    - host4: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@799a3e63;
    line: 1, column: 2]
    - host5: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7d7613f3;
    line: 1, column: 2]

    The hosts are communicating with the server and on the host status
    page I see that the nodes reply to heart beats.

    Looking at the cloudera-scm-server.log I also see these lines (only a
    summary):
    2012-07-04 11:14:02,291 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@411] Executing command Inspector [].
    2012-07-04 11:14:02,293 INFO [2131163116@scm-web-48
    :service.ServiceHandlerRegistry@593] Executing host command
    hostInspector []. Host: DbHost{id=5, hostId=host1,
    rackId=/default, hostName=host, ipAddress=10.0.0.0.3, status=NA,
    configContainer=ALL_HOSTS, configGener
    ation=1}
    2012-07-04 11:14:02,985 WARN [44314049@agentServer-13
    :cmf.AgentProtocolImpl@273] Received Process Heartbeat for unknown
    (or duplicate) process. Ignoring.id=338 name=host-
    inspector host=host
    2012-07-04 11:14:03,016 INFO
    [CommandPusher:command.CommandHelpers@191] Command (Inspector, 966)
    is still active on service null.
    2012-07-04 11:14:18,015 INFO
    [CommandPusher:service.AbstractOneOffHostCommand@127] Successful
    'hostInspector'
    2012-07-04 11:14:24,317 ERROR
    [CommandPusher:inspector.InspectorCommand@122] Unexpected error
    deserializing child output: org.codehaus.jackson.JsonParseException:
    Unexpecte
    d character ('I' (code 73)): expected a valid value (number, String,
    array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    com.cloudera.enterprise.JsonUtil$JsonRuntimeException:
    org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
    73)): expected a valid value (number, String
    , array, object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:72)
    at
    com.cloudera.enterprise.JsonUtil.valueFromBytes(JsonUtil.java:129)
    at
    com.cloudera.cmf.inspector.InspectorSerialization.toInspectorOutput(InspectorSerialization.java:35)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.postGather(InspectorCommand.java:120)
    at
    com.cloudera.cmf.command.inspector.InspectorCommand.update(InspectorCommand.java:101)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:290)
    at
    com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:202)
    at
    com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:121)
    at
    com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:94)
    Caused by: org.codehaus.jackson.JsonParseException: Unexpected
    character ('I' (code 73)): expected a valid value (number, String, array,
    object, 'true', 'false' or 'null')
    at [Source: java.io.StringReader@38f006af; line: 1, column: 2]
    at
    org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at
    org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at
    org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at
    org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at
    org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at
    org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at
    org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at
    com.cloudera.enterprise.JsonUtil.valueFromString(JsonUtil.java:70)
    ... 8 more

    Any ideas?

    Thansk!
    Hi Galed,

    Could you help gather the raw json? The inspection runs separately on
    each of the 5 hosts. And they each produce a JSON result string, which
    seems to be invalid. You'll need to run:

    $ curl -u 'admin:admin' -X POST http://
    <hostname>:7180/api/v1/cm/commands/inspectHosts

    That should give you a numeric command ID for the overall inspection
    command, and separate IDs for each per-host inspection commands. Wait about
    a minute for the commands to finish. Then:

    $ curl -u 'admin:admin' http://
    <hostname>:7180/api/v1/commands/<any_child_command_id>

    Now you should see a "resultDataUrl". (If not, the command hasn't
    completed yet. Wait another minute and retry.) Could you please hit that
    result data url and attach its contents?

    Thanks a lot!
    --
    bc Wong
    Cloudera Software Engineer

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedJul 4, '12 at 11:22a
activeJul 22, '12 at 3:31p
posts10
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase