FAQ
Good morning all CM administrators,

    I've a CDH5.0 cluster, installed via CM Agent 5.0. I checked CM's
Parcels window and there are a few downloadable service's upgrade
available; namely CDH5.0.1 and Impala 1.3.1. I wish to download and upgrade
these services via CM but is gated by the Parcel Directory -
"/opt/cloudera/parcels" space issue.

    My cluster resides on AWS EC2 and CM installation defaulted each
instance with 10GB local FS space. When cluster first setup and configured,
approximately 5GB or 50% of every instance ( NameNode and DataNode ) local
FS in cluster is used. All is well at this point.

    When I perform a service upgrade via CM's Download-Distribute-Activate,
the instances' local FS gets exhausted, 95% because files get created in
folder "/opt/cloudera/parcels" and "/opt/cloudera/parcel-cache/". I
understand that upon activating a service upgrade, I can safely remove
files in "/opt/cloudera/parcel-cache/" but even with this cleanup, my
instances' local FS utilization is still up around 80%+. Impacts of this
scenario are 1) prevent efficient workflow execution, 2 ) no more space to
upgrade other services and 3 ) CM services' monitoring constantly showing
RED status due to local instance space low issue.

    My question is :

    1 - In order to alleviate the instance local FS space restriction issue,
I'm considering mount a new disk to each instance, NameNode and DataNode(s)
and set in CM, Parcel Directory to this new mount disk. For example
"/newmountdisk/cloudera/parcels". Hence all service upgrade package
distribution, files creation and activation will now on this new mount
disk. I will keep instance local FS for service logging only and staging
area for workflow processing. Will this configuration works ?

    Note : Upgrading to CDH5.0.1 via CM Parcel prevented my cluster's SQOOP
service from starting up due to some JDBC upgrade not allowable if my
memory didn't fail me.

Thanks,
Andy

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Tony Li Xu at May 15, 2014 at 3:32 pm
    Hi Andy:

    Even though I never needed to configure the "Local Parcel Repository Path"
    for my CM cluster, but I think this configuration should work. By default
    "Local Parcel Repository Path" is set to "/opt/cloudera/parcel-repo", you
    can redefine it. It is just a place where downloaded parcels are stored. As
    long as the mounted disk is available and permissions are correct I think
    you should be fine.

    Also in Cloudera documentation, it says clearly this variable is
    configurable. So as the path for the distributed parcels on the member
    nodes which is "/opt/cloudera/parcels".

    --
    Tony

    On Thu, May 15, 2014 at 11:00 AM, 'Andy Tey' via Cloudera Manager Users wrote:

    Good morning all CM administrators,

    I've a CDH5.0 cluster, installed via CM Agent 5.0. I checked CM's
    Parcels window and there are a few downloadable service's upgrade
    available; namely CDH5.0.1 and Impala 1.3.1. I wish to download and upgrade
    these services via CM but is gated by the Parcel Directory -
    "/opt/cloudera/parcels" space issue.

    My cluster resides on AWS EC2 and CM installation defaulted each
    instance with 10GB local FS space. When cluster first setup and configured,
    approximately 5GB or 50% of every instance ( NameNode and DataNode ) local
    FS in cluster is used. All is well at this point.

    When I perform a service upgrade via CM's Download-Distribute-Activate,
    the instances' local FS gets exhausted, 95% because files get created in
    folder "/opt/cloudera/parcels" and "/opt/cloudera/parcel-cache/". I
    understand that upon activating a service upgrade, I can safely remove
    files in "/opt/cloudera/parcel-cache/" but even with this cleanup, my
    instances' local FS utilization is still up around 80%+. Impacts of this
    scenario are 1) prevent efficient workflow execution, 2 ) no more space to
    upgrade other services and 3 ) CM services' monitoring constantly showing
    RED status due to local instance space low issue.

    My question is :

    1 - In order to alleviate the instance local FS space restriction
    issue, I'm considering mount a new disk to each instance, NameNode and
    DataNode(s) and set in CM, Parcel Directory to this new mount disk. For
    example "/newmountdisk/cloudera/parcels". Hence all service upgrade package
    distribution, files creation and activation will now on this new mount
    disk. I will keep instance local FS for service logging only and staging
    area for workflow processing. Will this configuration works ?

    Note : Upgrading to CDH5.0.1 via CM Parcel prevented my cluster's SQOOP
    service from starting up due to some JDBC upgrade not allowable if my
    memory didn't fail me.

    Thanks,
    Andy

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMay 15, '14 at 3:00p
activeMay 15, '14 at 3:32p
posts2
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Andy Tey: 1 post Tony Li Xu: 1 post

People

Translate

site design / logo © 2022 Grokbase