FAQ
Hello,
   It seems that CM, via the "Deploy Client Configuration" button, is only
sending it to machines running an HDFS service. (For example, not the Hive
Server/Oozie server etc.)

   In my case CM is managing 16 hosts but only sending the client config out
to 6.

   Is this expected behavior?

Thanks

Search Discussions

  • Will Briggs at May 30, 2013 at 6:50 pm
    I'm fairly new to Hadoop and Cloudera, so take this with a grain of salt -
    but I believe that you can ensure the client config is pushed to any of
    your other non-MapReduce/HDFS hosts by adding a 'Gateway' role for them. If
    you need help with the steps to do that, let me know.

    -Will
    On Thursday, May 30, 2013 2:47:47 PM UTC-4, Joe Travaglini wrote:

    Hello,
    It seems that CM, via the "Deploy Client Configuration" button, is only
    sending it to machines running an HDFS service. (For example, not the Hive
    Server/Oozie server etc.)

    In my case CM is managing 16 hosts but only sending the client config
    out to 6.

    Is this expected behavior?

    Thanks
  • Darren Lo at May 30, 2013 at 7:56 pm
    Will's suggestion is correct. For each service, deploy client configs will
    only deploy to hosts with a role. You can add Gateway roles to a host to
    make sure they get your client configs.

    Thanks,
    Darren

    On Thu, May 30, 2013 at 11:50 AM, Will Briggs wrote:

    I'm fairly new to Hadoop and Cloudera, so take this with a grain of salt -
    but I believe that you can ensure the client config is pushed to any of
    your other non-MapReduce/HDFS hosts by adding a 'Gateway' role for them. If
    you need help with the steps to do that, let me know.

    -Will

    On Thursday, May 30, 2013 2:47:47 PM UTC-4, Joe Travaglini wrote:

    Hello,
    It seems that CM, via the "Deploy Client Configuration" button, is only
    sending it to machines running an HDFS service. (For example, not the Hive
    Server/Oozie server etc.)

    In my case CM is managing 16 hosts but only sending the client config
    out to 6.

    Is this expected behavior?

    Thanks

    --
    Thanks,
    Darren
  • Joe Travaglini at May 30, 2013 at 8:14 pm
    Darren,
       Perhaps I wasn't fully clear in my posts, so sorry about that.

       For these purposes, are MapReduce daemons not considered a role? Oozie
    daemons? Hive / Hue / Impala / Zookeeper daemons? My understanding is
    that they are. Am I mistaken there?

       All these services show up in the CM UI with good health, yet deploying
    client configs is only sending them to the servers with the DN and NN roles
    on them. The JT is not having the config pushed via the 'deploy client
    config' feature. I'm not going to get into the specifics here, but the TTs
    are on separate hosts than the DNs - I'm testing something.

       I understand that for each server, client configs will only deploy to
    hosts with a role. But that's not what I'm seeing in this new system.

       FWIW I've deployed 7 clusters in the past, the most recent one a few
    weeks ago. Only this new one is being problematic.

    -Joe

    On Thu, May 30, 2013 at 3:56 PM, Darren Lo wrote:

    Will's suggestion is correct. For each service, deploy client configs will
    only deploy to hosts with a role. You can add Gateway roles to a host to
    make sure they get your client configs.

    Thanks,
    Darren

    On Thu, May 30, 2013 at 11:50 AM, Will Briggs wrote:

    I'm fairly new to Hadoop and Cloudera, so take this with a grain of salt
    - but I believe that you can ensure the client config is pushed to any of
    your other non-MapReduce/HDFS hosts by adding a 'Gateway' role for them. If
    you need help with the steps to do that, let me know.

    -Will

    On Thursday, May 30, 2013 2:47:47 PM UTC-4, Joe Travaglini wrote:

    Hello,
    It seems that CM, via the "Deploy Client Configuration" button, is
    only sending it to machines running an HDFS service. (For example, not the
    Hive Server/Oozie server etc.)

    In my case CM is managing 16 hosts but only sending the client config
    out to 6.

    Is this expected behavior?

    Thanks

    --
    Thanks,
    Darren
  • Darren Lo at May 30, 2013 at 9:04 pm
    This isn't quite correct: "I understand that for each server, client
    configs will only deploy to hosts with a role."
    It should be "for each service, client configs will only deploy to hosts
    with a role for that service". For example, having an MR role on a host
    doesn't mean it gets Hive configs, just that it gets MR configs.

    TTs are part of the MapReduce service, DNs are part of the HDFS service. If
    you deploy client configs for MapReduce, then it will only deploy to hosts
    with MapReduce roles, such as TTs and MapReduce Gateway roles. Similarly
    for HDFS service and NN / SNN / DN and HDFS Gateway roles. When you deploy
    client configs at the cluster level, it is the same thing as deploying
    client configs for each service one by one.

    If you want particular services (such as HDFS, MapReduce, YARN, Hbase,
    Hive) to generate client configs on a particular host, then each of those
    services should have a gateway role on that host, or some other role for
    that service. Note that MapReduce, YARN, and HDFS share the same conf dir,
    and you can use the alternatives priority to decide which one is actually
    used.

    Thanks,
    Darren

    On Thu, May 30, 2013 at 1:14 PM, Joe Travaglini wrote:

    Darren,
    Perhaps I wasn't fully clear in my posts, so sorry about that.

    For these purposes, are MapReduce daemons not considered a role? Oozie
    daemons? Hive / Hue / Impala / Zookeeper daemons? My understanding is
    that they are. Am I mistaken there?

    All these services show up in the CM UI with good health, yet deploying
    client configs is only sending them to the servers with the DN and NN roles
    on them. The JT is not having the config pushed via the 'deploy client
    config' feature. I'm not going to get into the specifics here, but the TTs
    are on separate hosts than the DNs - I'm testing something.

    I understand that for each server, client configs will only deploy to
    hosts with a role. But that's not what I'm seeing in this new system.

    FWIW I've deployed 7 clusters in the past, the most recent one a few
    weeks ago. Only this new one is being problematic.

    -Joe

    On Thu, May 30, 2013 at 3:56 PM, Darren Lo wrote:

    Will's suggestion is correct. For each service, deploy client configs
    will only deploy to hosts with a role. You can add Gateway roles to a host
    to make sure they get your client configs.

    Thanks,
    Darren

    On Thu, May 30, 2013 at 11:50 AM, Will Briggs wrote:

    I'm fairly new to Hadoop and Cloudera, so take this with a grain of salt
    - but I believe that you can ensure the client config is pushed to any of
    your other non-MapReduce/HDFS hosts by adding a 'Gateway' role for them. If
    you need help with the steps to do that, let me know.

    -Will

    On Thursday, May 30, 2013 2:47:47 PM UTC-4, Joe Travaglini wrote:

    Hello,
    It seems that CM, via the "Deploy Client Configuration" button, is
    only sending it to machines running an HDFS service. (For example, not the
    Hive Server/Oozie server etc.)

    In my case CM is managing 16 hosts but only sending the client config
    out to 6.

    Is this expected behavior?

    Thanks

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren
  • Joe Travaglini at May 30, 2013 at 9:58 pm
    Darren,
       There was a typo - I meant 'service', not 'server'.

    In any event, I am deploying client configs globally, at the cluster level,
    and they are only getting pushed out to for the HDFS service when I look at
    the subcommands.

    Thank you for confirming that this is not expected behavior.

    On Thu, May 30, 2013 at 5:04 PM, Darren Lo wrote:

    This isn't quite correct: "I understand that for each server, client
    configs will only deploy to hosts with a role."
    It should be "for each service, client configs will only deploy to hosts
    with a role for that service". For example, having an MR role on a host
    doesn't mean it gets Hive configs, just that it gets MR configs.

    TTs are part of the MapReduce service, DNs are part of the HDFS service.
    If you deploy client configs for MapReduce, then it will only deploy to
    hosts with MapReduce roles, such as TTs and MapReduce Gateway roles.
    Similarly for HDFS service and NN / SNN / DN and HDFS Gateway roles. When
    you deploy client configs at the cluster level, it is the same thing as
    deploying client configs for each service one by one.

    If you want particular services (such as HDFS, MapReduce, YARN, Hbase,
    Hive) to generate client configs on a particular host, then each of those
    services should have a gateway role on that host, or some other role for
    that service. Note that MapReduce, YARN, and HDFS share the same conf dir,
    and you can use the alternatives priority to decide which one is actually
    used.

    Thanks,
    Darren

    On Thu, May 30, 2013 at 1:14 PM, Joe Travaglini wrote:

    Darren,
    Perhaps I wasn't fully clear in my posts, so sorry about that.

    For these purposes, are MapReduce daemons not considered a role? Oozie
    daemons? Hive / Hue / Impala / Zookeeper daemons? My understanding is
    that they are. Am I mistaken there?

    All these services show up in the CM UI with good health, yet deploying
    client configs is only sending them to the servers with the DN and NN roles
    on them. The JT is not having the config pushed via the 'deploy client
    config' feature. I'm not going to get into the specifics here, but the TTs
    are on separate hosts than the DNs - I'm testing something.

    I understand that for each server, client configs will only deploy to
    hosts with a role. But that's not what I'm seeing in this new system.

    FWIW I've deployed 7 clusters in the past, the most recent one a few
    weeks ago. Only this new one is being problematic.

    -Joe

    On Thu, May 30, 2013 at 3:56 PM, Darren Lo wrote:

    Will's suggestion is correct. For each service, deploy client configs
    will only deploy to hosts with a role. You can add Gateway roles to a host
    to make sure they get your client configs.

    Thanks,
    Darren

    On Thu, May 30, 2013 at 11:50 AM, Will Briggs wrote:

    I'm fairly new to Hadoop and Cloudera, so take this with a grain of
    salt - but I believe that you can ensure the client config is pushed to any
    of your other non-MapReduce/HDFS hosts by adding a 'Gateway' role for them.
    If you need help with the steps to do that, let me know.

    -Will

    On Thursday, May 30, 2013 2:47:47 PM UTC-4, Joe Travaglini wrote:

    Hello,
    It seems that CM, via the "Deploy Client Configuration" button, is
    only sending it to machines running an HDFS service. (For example, not the
    Hive Server/Oozie server etc.)

    In my case CM is managing 16 hosts but only sending the client
    config out to 6.

    Is this expected behavior?

    Thanks

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren
  • Darren Lo at May 30, 2013 at 10:03 pm
    Joe,

    That's very strange. If the command is still in progress, then it may not
    list all child commands that it will eventually run. Does the command fail?
    Is there any error or warning in the CM Server log? Are you sure you picked
    the cluster-level command and not the service-level one? What happens when
    you do the service-level deploy client configs for MR or another service?

    Thanks,
    Darren

    On Thu, May 30, 2013 at 2:58 PM, Joe Travaglini wrote:

    Darren,
    There was a typo - I meant 'service', not 'server'.

    In any event, I am deploying client configs globally, at the cluster
    level, and they are only getting pushed out to for the HDFS service when I
    look at the subcommands.

    Thank you for confirming that this is not expected behavior.

    On Thu, May 30, 2013 at 5:04 PM, Darren Lo wrote:

    This isn't quite correct: "I understand that for each server, client
    configs will only deploy to hosts with a role."
    It should be "for each service, client configs will only deploy to
    hosts with a role for that service". For example, having an MR role on a
    host doesn't mean it gets Hive configs, just that it gets MR configs.

    TTs are part of the MapReduce service, DNs are part of the HDFS service.
    If you deploy client configs for MapReduce, then it will only deploy to
    hosts with MapReduce roles, such as TTs and MapReduce Gateway roles.
    Similarly for HDFS service and NN / SNN / DN and HDFS Gateway roles. When
    you deploy client configs at the cluster level, it is the same thing as
    deploying client configs for each service one by one.

    If you want particular services (such as HDFS, MapReduce, YARN, Hbase,
    Hive) to generate client configs on a particular host, then each of those
    services should have a gateway role on that host, or some other role for
    that service. Note that MapReduce, YARN, and HDFS share the same conf dir,
    and you can use the alternatives priority to decide which one is actually
    used.

    Thanks,
    Darren


    On Thu, May 30, 2013 at 1:14 PM, Joe Travaglini <joe.travaglini@gmail.com
    wrote:
    Darren,
    Perhaps I wasn't fully clear in my posts, so sorry about that.

    For these purposes, are MapReduce daemons not considered a role?
    Oozie daemons? Hive / Hue / Impala / Zookeeper daemons? My
    understanding is that they are. Am I mistaken there?

    All these services show up in the CM UI with good health, yet
    deploying client configs is only sending them to the servers with the DN
    and NN roles on them. The JT is not having the config pushed via the
    'deploy client config' feature. I'm not going to get into the specifics
    here, but the TTs are on separate hosts than the DNs - I'm testing
    something.

    I understand that for each server, client configs will only deploy to
    hosts with a role. But that's not what I'm seeing in this new system.

    FWIW I've deployed 7 clusters in the past, the most recent one a few
    weeks ago. Only this new one is being problematic.

    -Joe

    On Thu, May 30, 2013 at 3:56 PM, Darren Lo wrote:

    Will's suggestion is correct. For each service, deploy client configs
    will only deploy to hosts with a role. You can add Gateway roles to a host
    to make sure they get your client configs.

    Thanks,
    Darren

    On Thu, May 30, 2013 at 11:50 AM, Will Briggs wrote:

    I'm fairly new to Hadoop and Cloudera, so take this with a grain of
    salt - but I believe that you can ensure the client config is pushed to any
    of your other non-MapReduce/HDFS hosts by adding a 'Gateway' role for them.
    If you need help with the steps to do that, let me know.

    -Will

    On Thursday, May 30, 2013 2:47:47 PM UTC-4, Joe Travaglini wrote:

    Hello,
    It seems that CM, via the "Deploy Client Configuration" button, is
    only sending it to machines running an HDFS service. (For example, not the
    Hive Server/Oozie server etc.)

    In my case CM is managing 16 hosts but only sending the client
    config out to 6.

    Is this expected behavior?

    Thanks

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMay 30, '13 at 6:47p
activeMay 30, '13 at 10:03p
posts7
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase