FAQ
Hello,

First of all thanks for the prompt response I have been getting from this
community. Really appreciate it.

I have a use case where I am trying to use the cloudera manager API to add
a new host to an existing cluster that already has bunch of parcels
deployed and activated. I am able to add the new host and assign roles to
it but not able to start them because it doesn't have those parcels
deployed. Is there anything in the API that will help to deploy parcels on
a specific host? I hunted for it in the API for quite a while but couldn't
find one. All I could find was using the ParcelsResource to deploy them on
the cluster itself.

Thanks,

--
Swarnim

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Vikram Bajaj at Mar 4, 2014 at 8:51 pm
    You have to re distribute the parcel, just like you did for the other
    hosts. If the hosts already have the parcel and is activated it will not
    distribute it to them, but only to the new hosts

      parcel = cluster.get_parcel(product, version)

    cmd = parcel.start_download()

    parcel.start_distribution()

    cmd = parcel.activate()


    Regards,

    Vikram

    On Tue, Mar 4, 2014 at 12:47 PM, kulkarni.swarnim@gmail.com wrote:

    Hello,

    First of all thanks for the prompt response I have been getting from this
    community. Really appreciate it.

    I have a use case where I am trying to use the cloudera manager API to add
    a new host to an existing cluster that already has bunch of parcels
    deployed and activated. I am able to add the new host and assign roles to
    it but not able to start them because it doesn't have those parcels
    deployed. Is there anything in the API that will help to deploy parcels on
    a specific host? I hunted for it in the API for quite a while but couldn't
    find one. All I could find was using the ParcelsResource to deploy them on
    the cluster itself.

    Thanks,

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Kulkarni Swarnim at Mar 4, 2014 at 9:05 pm
    Thanks Vikram. This is exactly what my expectation was as well.

    The point where I got stuck with this approach is since all these parcel
    commands(download, distribute, activate) are async in nature, in case of
    redistribution how do we ensure their successful completion? Even if the
    parcel is getting download/distributed only on new hosts, the state of
    parcel still remains "Activated" since it was previously activated on other
    hosts.

    Thanks,

    On Tue, Mar 4, 2014 at 2:51 PM, Vikram Bajaj wrote:

    You have to re distribute the parcel, just like you did for the other
    hosts. If the hosts already have the parcel and is activated it will not
    distribute it to them, but only to the new hosts

    parcel = cluster.get_parcel(product, version)

    cmd = parcel.start_download()

    parcel.start_distribution()

    cmd = parcel.activate()


    Regards,

    Vikram


    On Tue, Mar 4, 2014 at 12:47 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Hello,

    First of all thanks for the prompt response I have been getting from this
    community. Really appreciate it.

    I have a use case where I am trying to use the cloudera manager API to
    add a new host to an existing cluster that already has bunch of parcels
    deployed and activated. I am able to add the new host and assign roles to
    it but not able to start them because it doesn't have those parcels
    deployed. Is there anything in the API that will help to deploy parcels on
    a specific host? I hunted for it in the API for quite a while but couldn't
    find one. All I could find was using the ParcelsResource to deploy them on
    the cluster itself.

    Thanks,

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Vikram Bajaj at Mar 4, 2014 at 9:52 pm
    This should help

    def distribute_parcel(cluster, product, version):



         # If available remotely, start the downloading the parcel

         cmd = None

         while True:

             parcel = cluster.get_parcel(product, version)

             LOG.info('Parcel Status: %s' %(parcel.stage))

             if parcel.stage == 'UNAVAILABLE':

                 LOG.info('Giving CM time to find parcel information')

                 LOG.info('Parcel Status: %s' %(parcel.stage))

             if parcel.stage == 'AVAILABLE_REMOTELY':

                 LOG.info('Beginning Parcel %s : %s download.' %(product,
    version))

                 cmd = parcel.start_download()

             if parcel.stage == 'DOWNLOADING':

                 LOG.info("download progress: %s / %s" % (parcel.state.progress,
    parcel.state.totalProgress))

             if parcel.stage == 'DOWNLOADED':

                 LOG.info("Starting distribution of %s parcel version %s to
    cluster %s" % (product, version, cluster.name))

                 cmd = parcel.start_distribution()

             if parcel.stage == 'DISTRIBUTING':

                 LOG.info("distribution progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

             if parcel.stage == 'DISTRIBUTED':

                 LOG.info("Starting activation of %s parcel version %s for
    cluster %s" % (product, version, cluster.name))

                 cmd = parcel.activate()

             if parcel.stage == 'ACTIVATED':

                 LOG.info("activated - %s parcel version %s on cluster %s" %
    (product, version, cluster.name))

                 break

             if cmd:

                 if not cmd.wait(CMD_TIMEOUT).success:

                     LOG.critical("PAR:501 Time out during parcel operation.
    Exiting now. %s " %(cmd.resultMessage))

                     sys.exit(501)

             if parcel.state.errors:

                 LOG.critical('PAR:500 Exception in parcel operation %s Message
    %s' %(parcel.state.errors, cmd.resultMessage))

                 sys.exit(500)

             time.sleep(15) # check again in 15 seconds

    On Tue, Mar 4, 2014 at 1:04 PM, kulkarni.swarnim@gmail.com wrote:

    Thanks Vikram. This is exactly what my expectation was as well.

    The point where I got stuck with this approach is since all these parcel
    commands(download, distribute, activate) are async in nature, in case of
    redistribution how do we ensure their successful completion? Even if the
    parcel is getting download/distributed only on new hosts, the state of
    parcel still remains "Activated" since it was previously activated on other
    hosts.

    Thanks,

    On Tue, Mar 4, 2014 at 2:51 PM, Vikram Bajaj wrote:

    You have to re distribute the parcel, just like you did for the other
    hosts. If the hosts already have the parcel and is activated it will not
    distribute it to them, but only to the new hosts

    parcel = cluster.get_parcel(product, version)

    cmd = parcel.start_download()

    parcel.start_distribution()

    cmd = parcel.activate()


    Regards,

    Vikram


    On Tue, Mar 4, 2014 at 12:47 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Hello,

    First of all thanks for the prompt response I have been getting from
    this community. Really appreciate it.

    I have a use case where I am trying to use the cloudera manager API to
    add a new host to an existing cluster that already has bunch of parcels
    deployed and activated. I am able to add the new host and assign roles to
    it but not able to start them because it doesn't have those parcels
    deployed. Is there anything in the API that will help to deploy parcels on
    a specific host? I hunted for it in the API for quite a while but couldn't
    find one. All I could find was using the ParcelsResource to deploy them on
    the cluster itself.

    Thanks,

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.

    --
    Swarnim
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Kulkarni Swarnim at Mar 4, 2014 at 10:11 pm
    Thanks Vikram.

    So I am actually consuming the Java API but my code is very similar to what
    you have.

    So in this case, if I call the distribute_parcel method again to
    redistribute them, wouldn't the "parcel = cluster.get_parcel(product,
    version)" return state of parcel as ACTIVATED and then would directly break
    out of the loop? Or am I missing something. This is the exact problem I am
    facing.

    Thanks again.

    On Tue, Mar 4, 2014 at 3:52 PM, Vikram Bajaj wrote:

    This should help

    def distribute_parcel(cluster, product, version):



    # If available remotely, start the downloading the parcel

    cmd = None

    while True:

    parcel = cluster.get_parcel(product, version)

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'UNAVAILABLE':

    LOG.info('Giving CM time to find parcel information')

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'AVAILABLE_REMOTELY':

    LOG.info('Beginning Parcel %s : %s download.' %(product,
    version))

    cmd = parcel.start_download()

    if parcel.stage == 'DOWNLOADING':

    LOG.info("download progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DOWNLOADED':

    LOG.info("Starting distribution of %s parcel version %s to
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.start_distribution()

    if parcel.stage == 'DISTRIBUTING':

    LOG.info("distribution progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DISTRIBUTED':

    LOG.info("Starting activation of %s parcel version %s for
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.activate()

    if parcel.stage == 'ACTIVATED':

    LOG.info("activated - %s parcel version %s on cluster %s" %
    (product, version, cluster.name))

    break

    if cmd:

    if not cmd.wait(CMD_TIMEOUT).success:

    LOG.critical("PAR:501 Time out during parcel operation.
    Exiting now. %s " %(cmd.resultMessage))

    sys.exit(501)

    if parcel.state.errors:

    LOG.critical('PAR:500 Exception in parcel operation %s
    Message %s' %(parcel.state.errors, cmd.resultMessage))

    sys.exit(500)

    time.sleep(15) # check again in 15 seconds


    On Tue, Mar 4, 2014 at 1:04 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Thanks Vikram. This is exactly what my expectation was as well.

    The point where I got stuck with this approach is since all these parcel
    commands(download, distribute, activate) are async in nature, in case of
    redistribution how do we ensure their successful completion? Even if the
    parcel is getting download/distributed only on new hosts, the state of
    parcel still remains "Activated" since it was previously activated on other
    hosts.

    Thanks,

    On Tue, Mar 4, 2014 at 2:51 PM, Vikram Bajaj wrote:

    You have to re distribute the parcel, just like you did for the other
    hosts. If the hosts already have the parcel and is activated it will not
    distribute it to them, but only to the new hosts

    parcel = cluster.get_parcel(product, version)

    cmd = parcel.start_download()

    parcel.start_distribution()

    cmd = parcel.activate()


    Regards,

    Vikram


    On Tue, Mar 4, 2014 at 12:47 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Hello,

    First of all thanks for the prompt response I have been getting from
    this community. Really appreciate it.

    I have a use case where I am trying to use the cloudera manager API to
    add a new host to an existing cluster that already has bunch of parcels
    deployed and activated. I am able to add the new host and assign roles to
    it but not able to start them because it doesn't have those parcels
    deployed. Is there anything in the API that will help to deploy parcels on
    a specific host? I hunted for it in the API for quite a while but couldn't
    find one. All I could find was using the ParcelsResource to deploy them on
    the cluster itself.

    Thanks,

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.

    --
    Swarnim

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Vikram Bajaj at Mar 4, 2014 at 10:14 pm
    No, it does not at least in my scripts. I do have a script to add new
    dataNodes and it works as expected. i.e it distributes the parcel to the
    new hosts and activates it.

    You can test it out, by adding a new host from the cloudera manager and
    then check the parcel status from the cloudera manager parcels link.

    On Tue, Mar 4, 2014 at 2:11 PM, kulkarni.swarnim@gmail.com wrote:

    Thanks Vikram.

    So I am actually consuming the Java API but my code is very similar to
    what you have.

    So in this case, if I call the distribute_parcel method again to
    redistribute them, wouldn't the "parcel = cluster.get_parcel(product,
    version)" return state of parcel as ACTIVATED and then would directly break
    out of the loop? Or am I missing something. This is the exact problem I am
    facing.

    Thanks again.

    On Tue, Mar 4, 2014 at 3:52 PM, Vikram Bajaj wrote:

    This should help

    def distribute_parcel(cluster, product, version):



    # If available remotely, start the downloading the parcel

    cmd = None

    while True:

    parcel = cluster.get_parcel(product, version)

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'UNAVAILABLE':

    LOG.info('Giving CM time to find parcel information')

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'AVAILABLE_REMOTELY':

    LOG.info('Beginning Parcel %s : %s download.' %(product,
    version))

    cmd = parcel.start_download()

    if parcel.stage == 'DOWNLOADING':

    LOG.info("download progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DOWNLOADED':

    LOG.info("Starting distribution of %s parcel version %s to
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.start_distribution()

    if parcel.stage == 'DISTRIBUTING':

    LOG.info("distribution progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DISTRIBUTED':

    LOG.info("Starting activation of %s parcel version %s for
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.activate()

    if parcel.stage == 'ACTIVATED':

    LOG.info("activated - %s parcel version %s on cluster %s" %
    (product, version, cluster.name))

    break

    if cmd:

    if not cmd.wait(CMD_TIMEOUT).success:

    LOG.critical("PAR:501 Time out during parcel operation.
    Exiting now. %s " %(cmd.resultMessage))

    sys.exit(501)

    if parcel.state.errors:

    LOG.critical('PAR:500 Exception in parcel operation %s
    Message %s' %(parcel.state.errors, cmd.resultMessage))

    sys.exit(500)

    time.sleep(15) # check again in 15 seconds


    On Tue, Mar 4, 2014 at 1:04 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Thanks Vikram. This is exactly what my expectation was as well.

    The point where I got stuck with this approach is since all these parcel
    commands(download, distribute, activate) are async in nature, in case of
    redistribution how do we ensure their successful completion? Even if the
    parcel is getting download/distributed only on new hosts, the state of
    parcel still remains "Activated" since it was previously activated on other
    hosts.

    Thanks,

    On Tue, Mar 4, 2014 at 2:51 PM, Vikram Bajaj wrote:

    You have to re distribute the parcel, just like you did for the other
    hosts. If the hosts already have the parcel and is activated it will not
    distribute it to them, but only to the new hosts

    parcel = cluster.get_parcel(product, version)

    cmd = parcel.start_download()

    parcel.start_distribution()

    cmd = parcel.activate()


    Regards,

    Vikram


    On Tue, Mar 4, 2014 at 12:47 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Hello,

    First of all thanks for the prompt response I have been getting from
    this community. Really appreciate it.

    I have a use case where I am trying to use the cloudera manager API to
    add a new host to an existing cluster that already has bunch of parcels
    deployed and activated. I am able to add the new host and assign roles to
    it but not able to start them because it doesn't have those parcels
    deployed. Is there anything in the API that will help to deploy parcels on
    a specific host? I hunted for it in the API for quite a while but couldn't
    find one. All I could find was using the ParcelsResource to deploy them on
    the cluster itself.

    Thanks,

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.

    --
    Swarnim

    --
    Swarnim
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Kulkarni Swarnim at Mar 4, 2014 at 10:24 pm
    I'll try it out. Thanks Vikram for taking time to help me out.

    On Tue, Mar 4, 2014 at 4:14 PM, Vikram Bajaj wrote:

    No, it does not at least in my scripts. I do have a script to add new
    dataNodes and it works as expected. i.e it distributes the parcel to the
    new hosts and activates it.

    You can test it out, by adding a new host from the cloudera manager and
    then check the parcel status from the cloudera manager parcels link.


    On Tue, Mar 4, 2014 at 2:11 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Thanks Vikram.

    So I am actually consuming the Java API but my code is very similar to
    what you have.

    So in this case, if I call the distribute_parcel method again to
    redistribute them, wouldn't the "parcel = cluster.get_parcel(product,
    version)" return state of parcel as ACTIVATED and then would directly break
    out of the loop? Or am I missing something. This is the exact problem I am
    facing.

    Thanks again.

    On Tue, Mar 4, 2014 at 3:52 PM, Vikram Bajaj wrote:

    This should help

    def distribute_parcel(cluster, product, version):



    # If available remotely, start the downloading the parcel

    cmd = None

    while True:

    parcel = cluster.get_parcel(product, version)

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'UNAVAILABLE':

    LOG.info('Giving CM time to find parcel information')

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'AVAILABLE_REMOTELY':

    LOG.info('Beginning Parcel %s : %s download.' %(product,
    version))

    cmd = parcel.start_download()

    if parcel.stage == 'DOWNLOADING':

    LOG.info("download progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DOWNLOADED':

    LOG.info("Starting distribution of %s parcel version %s to
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.start_distribution()

    if parcel.stage == 'DISTRIBUTING':

    LOG.info("distribution progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DISTRIBUTED':

    LOG.info("Starting activation of %s parcel version %s for
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.activate()

    if parcel.stage == 'ACTIVATED':

    LOG.info("activated - %s parcel version %s on cluster %s" %
    (product, version, cluster.name))

    break

    if cmd:

    if not cmd.wait(CMD_TIMEOUT).success:

    LOG.critical("PAR:501 Time out during parcel
    operation. Exiting now. %s " %(cmd.resultMessage))

    sys.exit(501)

    if parcel.state.errors:

    LOG.critical('PAR:500 Exception in parcel operation %s
    Message %s' %(parcel.state.errors, cmd.resultMessage))

    sys.exit(500)

    time.sleep(15) # check again in 15 seconds


    On Tue, Mar 4, 2014 at 1:04 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Thanks Vikram. This is exactly what my expectation was as well.

    The point where I got stuck with this approach is since all these
    parcel commands(download, distribute, activate) are async in nature, in
    case of redistribution how do we ensure their successful completion? Even
    if the parcel is getting download/distributed only on new hosts, the state
    of parcel still remains "Activated" since it was previously activated on
    other hosts.

    Thanks,

    On Tue, Mar 4, 2014 at 2:51 PM, Vikram Bajaj wrote:

    You have to re distribute the parcel, just like you did for the other
    hosts. If the hosts already have the parcel and is activated it will not
    distribute it to them, but only to the new hosts

    parcel = cluster.get_parcel(product, version)

    cmd = parcel.start_download()

    parcel.start_distribution()

    cmd = parcel.activate()


    Regards,

    Vikram


    On Tue, Mar 4, 2014 at 12:47 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Hello,

    First of all thanks for the prompt response I have been getting from
    this community. Really appreciate it.

    I have a use case where I am trying to use the cloudera manager API
    to add a new host to an existing cluster that already has bunch of parcels
    deployed and activated. I am able to add the new host and assign roles to
    it but not able to start them because it doesn't have those parcels
    deployed. Is there anything in the API that will help to deploy parcels on
    a specific host? I hunted for it in the API for quite a while but couldn't
    find one. All I could find was using the ParcelsResource to deploy them on
    the cluster itself.

    Thanks,

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+unsubscribe@cloudera.org.

    --
    Swarnim

    --
    Swarnim

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Kulkarni Swarnim at Mar 5, 2014 at 5:05 pm
    I think I found out what's going on. So it seems like when a parcel is
    redistributed, it goes directly from a DISTRIBUTING stage to ACTIVATED
    stage.

    Current parcel stage: [ACTIVATED]
    Current parcel stage: [ACTIVATED] <----- Was already activated on the
    cluster
    Current parcel stage: [DISTRIBUTING] <----- Redistribution starts on new
    hosts
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    <snip>
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [ACTIVATED] <----- ??
    Current parcel stage: [ACTIVATED]
    Current parcel stage: [ACTIVATED]
    Current parcel stage: [ACTIVATED]

    My current code waits for it to move to a DISTRIBUTED state first before
    activating it. Since it never achieves that stage, it just sits there
    waiting.

    Is this something that should be expected or is a bug expected to be fixed
    in later CM versions?

    Thanks,

    On Tue, Mar 4, 2014 at 4:24 PM, kulkarni.swarnim@gmail.com wrote:

    I'll try it out. Thanks Vikram for taking time to help me out.

    On Tue, Mar 4, 2014 at 4:14 PM, Vikram Bajaj wrote:

    No, it does not at least in my scripts. I do have a script to add new
    dataNodes and it works as expected. i.e it distributes the parcel to the
    new hosts and activates it.

    You can test it out, by adding a new host from the cloudera manager and
    then check the parcel status from the cloudera manager parcels link.


    On Tue, Mar 4, 2014 at 2:11 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Thanks Vikram.

    So I am actually consuming the Java API but my code is very similar to
    what you have.

    So in this case, if I call the distribute_parcel method again to
    redistribute them, wouldn't the "parcel = cluster.get_parcel(product,
    version)" return state of parcel as ACTIVATED and then would directly break
    out of the loop? Or am I missing something. This is the exact problem I am
    facing.

    Thanks again.

    On Tue, Mar 4, 2014 at 3:52 PM, Vikram Bajaj wrote:

    This should help

    def distribute_parcel(cluster, product, version):



    # If available remotely, start the downloading the parcel

    cmd = None

    while True:

    parcel = cluster.get_parcel(product, version)

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'UNAVAILABLE':

    LOG.info('Giving CM time to find parcel information')

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'AVAILABLE_REMOTELY':

    LOG.info('Beginning Parcel %s : %s download.' %(product,
    version))

    cmd = parcel.start_download()

    if parcel.stage == 'DOWNLOADING':

    LOG.info("download progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DOWNLOADED':

    LOG.info("Starting distribution of %s parcel version %s to
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.start_distribution()

    if parcel.stage == 'DISTRIBUTING':

    LOG.info("distribution progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DISTRIBUTED':

    LOG.info("Starting activation of %s parcel version %s for
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.activate()

    if parcel.stage == 'ACTIVATED':

    LOG.info("activated - %s parcel version %s on cluster %s"% (product, version,
    cluster.name))

    break

    if cmd:

    if not cmd.wait(CMD_TIMEOUT).success:

    LOG.critical("PAR:501 Time out during parcel
    operation. Exiting now. %s " %(cmd.resultMessage))

    sys.exit(501)

    if parcel.state.errors:

    LOG.critical('PAR:500 Exception in parcel operation %s
    Message %s' %(parcel.state.errors, cmd.resultMessage))

    sys.exit(500)

    time.sleep(15) # check again in 15 seconds


    On Tue, Mar 4, 2014 at 1:04 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Thanks Vikram. This is exactly what my expectation was as well.

    The point where I got stuck with this approach is since all these
    parcel commands(download, distribute, activate) are async in nature, in
    case of redistribution how do we ensure their successful completion? Even
    if the parcel is getting download/distributed only on new hosts, the state
    of parcel still remains "Activated" since it was previously activated on
    other hosts.

    Thanks,

    On Tue, Mar 4, 2014 at 2:51 PM, Vikram Bajaj wrote:

    You have to re distribute the parcel, just like you did for the other
    hosts. If the hosts already have the parcel and is activated it will not
    distribute it to them, but only to the new hosts

    parcel = cluster.get_parcel(product, version)

    cmd = parcel.start_download()

    parcel.start_distribution()

    cmd = parcel.activate()


    Regards,

    Vikram


    On Tue, Mar 4, 2014 at 12:47 PM, kulkarni.swarnim@gmail.com <
    kulkarni.swarnim@gmail.com> wrote:
    Hello,

    First of all thanks for the prompt response I have been getting from
    this community. Really appreciate it.

    I have a use case where I am trying to use the cloudera manager API
    to add a new host to an existing cluster that already has bunch of parcels
    deployed and activated. I am able to add the new host and assign roles to
    it but not able to start them because it doesn't have those parcels
    deployed. Is there anything in the API that will help to deploy parcels on
    a specific host? I hunted for it in the API for quite a while but couldn't
    find one. All I could find was using the ParcelsResource to deploy them on
    the cluster itself.

    Thanks,

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+unsubscribe@cloudera.org.

    --
    Swarnim

    --
    Swarnim

    --
    Swarnim


    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Herman Chen at Mar 5, 2014 at 6:47 pm
    Hi kulkarni

    This is expected. The parcel was already and still is activated
    throughout the process, so as soon as distribution completes to the
    newly added hosts, it is back to activated.

    Herman

    On Wed, Mar 5, 2014 at 9:04 AM, kulkarni.swarnim@gmail.com
    wrote:
    I think I found out what's going on. So it seems like when a parcel is
    redistributed, it goes directly from a DISTRIBUTING stage to ACTIVATED
    stage.

    Current parcel stage: [ACTIVATED]
    Current parcel stage: [ACTIVATED] <----- Was already activated on the
    cluster
    Current parcel stage: [DISTRIBUTING] <----- Redistribution starts on new
    hosts
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    <snip>
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [DISTRIBUTING]
    Current parcel stage: [ACTIVATED] <----- ??
    Current parcel stage: [ACTIVATED]
    Current parcel stage: [ACTIVATED]
    Current parcel stage: [ACTIVATED]

    My current code waits for it to move to a DISTRIBUTED state first before
    activating it. Since it never achieves that stage, it just sits there
    waiting.

    Is this something that should be expected or is a bug expected to be fixed
    in later CM versions?

    Thanks,


    On Tue, Mar 4, 2014 at 4:24 PM, kulkarni.swarnim@gmail.com
    wrote:
    I'll try it out. Thanks Vikram for taking time to help me out.

    On Tue, Mar 4, 2014 at 4:14 PM, Vikram Bajaj wrote:

    No, it does not at least in my scripts. I do have a script to add new
    dataNodes and it works as expected. i.e it distributes the parcel to the new
    hosts and activates it.

    You can test it out, by adding a new host from the cloudera manager and
    then check the parcel status from the cloudera manager parcels link.


    On Tue, Mar 4, 2014 at 2:11 PM, kulkarni.swarnim@gmail.com
    wrote:
    Thanks Vikram.

    So I am actually consuming the Java API but my code is very similar to
    what you have.

    So in this case, if I call the distribute_parcel method again to
    redistribute them, wouldn't the "parcel = cluster.get_parcel(product,
    version)" return state of parcel as ACTIVATED and then would directly break
    out of the loop? Or am I missing something. This is the exact problem I am
    facing.

    Thanks again.


    On Tue, Mar 4, 2014 at 3:52 PM, Vikram Bajaj <paphillon@gmail.com>
    wrote:
    This should help

    def distribute_parcel(cluster, product, version):



    # If available remotely, start the downloading the parcel

    cmd = None

    while True:

    parcel = cluster.get_parcel(product, version)

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'UNAVAILABLE':

    LOG.info('Giving CM time to find parcel information')

    LOG.info('Parcel Status: %s' %(parcel.stage))

    if parcel.stage == 'AVAILABLE_REMOTELY':

    LOG.info('Beginning Parcel %s : %s download.' %(product,
    version))

    cmd = parcel.start_download()

    if parcel.stage == 'DOWNLOADING':

    LOG.info("download progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DOWNLOADED':

    LOG.info("Starting distribution of %s parcel version %s to
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.start_distribution()

    if parcel.stage == 'DISTRIBUTING':

    LOG.info("distribution progress: %s / %s" %
    (parcel.state.progress, parcel.state.totalProgress))

    if parcel.stage == 'DISTRIBUTED':

    LOG.info("Starting activation of %s parcel version %s for
    cluster %s" % (product, version, cluster.name))

    cmd = parcel.activate()

    if parcel.stage == 'ACTIVATED':

    LOG.info("activated - %s parcel version %s on cluster %s" %
    (product, version, cluster.name))

    break

    if cmd:

    if not cmd.wait(CMD_TIMEOUT).success:

    LOG.critical("PAR:501 Time out during parcel operation.
    Exiting now. %s " %(cmd.resultMessage))

    sys.exit(501)

    if parcel.state.errors:

    LOG.critical('PAR:500 Exception in parcel operation %s
    Message %s' %(parcel.state.errors, cmd.resultMessage))

    sys.exit(500)

    time.sleep(15) # check again in 15 seconds



    On Tue, Mar 4, 2014 at 1:04 PM, kulkarni.swarnim@gmail.com
    wrote:
    Thanks Vikram. This is exactly what my expectation was as well.

    The point where I got stuck with this approach is since all these
    parcel commands(download, distribute, activate) are async in nature, in case
    of redistribution how do we ensure their successful completion? Even if the
    parcel is getting download/distributed only on new hosts, the state of
    parcel still remains "Activated" since it was previously activated on other
    hosts.

    Thanks,


    On Tue, Mar 4, 2014 at 2:51 PM, Vikram Bajaj <paphillon@gmail.com>
    wrote:
    You have to re distribute the parcel, just like you did for the other
    hosts. If the hosts already have the parcel and is activated it will not
    distribute it to them, but only to the new hosts

    parcel = cluster.get_parcel(product, version)

    cmd = parcel.start_download()

    parcel.start_distribution()

    cmd = parcel.activate()


    Regards,

    Vikram



    On Tue, Mar 4, 2014 at 12:47 PM, kulkarni.swarnim@gmail.com
    wrote:
    Hello,

    First of all thanks for the prompt response I have been getting from
    this community. Really appreciate it.

    I have a use case where I am trying to use the cloudera manager API
    to add a new host to an existing cluster that already has bunch of parcels
    deployed and activated. I am able to add the new host and assign roles to it
    but not able to start them because it doesn't have those parcels deployed.
    Is there anything in the API that will help to deploy parcels on a specific
    host? I hunted for it in the API for quite a while but couldn't find one.
    All I could find was using the ParcelsResource to deploy them on the cluster
    itself.

    Thanks,

    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+unsubscribe@cloudera.org.


    --
    Swarnim


    --
    Swarnim


    --
    Swarnim



    --
    Swarnim

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMar 4, '14 at 8:47p
activeMar 5, '14 at 6:47p
posts9
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase