FAQ
We used Cloudera Manager Free to install CDH3 in our cluster. Very simple.
We are trying to upgrade to CDH4 but it seems as though the instructions
I've found assume cdh3 was installed manually.
here <https://ccp.cloudera.com/display/CDH4DOC/Upgrading+from+CDH3+to+CDH4>for
example. (back up old configs, make note of manual settings)

None of the config files on the hosts are configured manually since it was
all done through Cloudera Manager.
Can someone verify the steps? If I use the 13 steps to upgrade cdh3 to
cdh4 then I wouldn't need Cloudera Manager in the first place.
And it's documentation here
<https://ccp.cloudera.com/display/FREE4DOC/Cloudera+Manager+Free+Edition+Frequently+Asked+Questions#ClouderaManagerFreeEditionFrequentlyAskedQuestions-CanIupgradeCDHusingClouderaManager%3F>already
says it won't upgrade cdh and to use the other page.

Please advise.

Search Discussions

  • Joey Echeverria at Jun 7, 2012 at 2:50 pm
    We probably need to make the docs around this better.

    What you need to do is follow the steps from the CDH3 to CDH4 upgrade
    guide that involve removing the CDH3 packages and installing the CDH4
    packages. You don't have to worry about the config files in
    /etc/<project> as those weren't being used and you dont' need to
    install the packages that just contain service scripts (e.g.
    hadoop-hdfs-namenode).

    I'll file a documentation JIRA to make the steps more clear.

    -Joey
    On Thu, Jun 7, 2012 at 10:43 AM, Gary S wrote:
    We used Cloudera Manager Free to install CDH3 in our cluster. Very simple.
    We are trying to upgrade to CDH4 but it seems as though the instructions
    I've found assume cdh3 was installed manually.
    here for example. (back up old configs, make note of manual settings)

    None of the config files on the hosts are configured manually since it was
    all done through Cloudera Manager.
    Can someone verify the steps?  If I use the 13 steps to upgrade cdh3 to cdh4
    then I wouldn't need Cloudera Manager in the first place.
    And it's documentation here already says it won't upgrade cdh and to use the
    other page.

    Please advise.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Gary S at Jun 7, 2012 at 3:03 pm
    Thanks Joey. Would it be as simple as...
    1) Remove all components on all hosts (nn, snn, dn) per upgrade guide
    2) Apply steps 1-5 on all hosts per upgrade guide
    3) skip steps 6 thru 13?
    3) Install/Upgrade to Cloudera Manager 4
    4) Install to cluster as always?

    On the upgrade page, we're stuck at step 6 - upgrade the hdfs metadata.
    Since the hdfs-site.xml never had the dfs.name.dir, it defaults to the
    file:/// then we get:

    java.lang.IllegalArgumentException: Invalid URI for NameNode address (check
    fs.defaultFS): file:/// has no authority.

    Sorry to mix 2 issues into 1 thread. (upgrade if using CM plus exception on
    step 6)
    On Thursday, June 7, 2012 10:50:37 AM UTC-4, Joey Echeverria wrote:

    We probably need to make the docs around this better.

    What you need to do is follow the steps from the CDH3 to CDH4 upgrade
    guide that involve removing the CDH3 packages and installing the CDH4
    packages. You don't have to worry about the config files in
    /etc/<project> as those weren't being used and you dont' need to
    install the packages that just contain service scripts (e.g.
    hadoop-hdfs-namenode).

    I'll file a documentation JIRA to make the steps more clear.

    -Joey
    On Thu, Jun 7, 2012 at 10:43 AM, Gary S wrote:
    We used Cloudera Manager Free to install CDH3 in our cluster. Very simple.
    We are trying to upgrade to CDH4 but it seems as though the instructions
    I've found assume cdh3 was installed manually.
    here for example. (back up old configs, make note of manual settings)

    None of the config files on the hosts are configured manually since it was
    all done through Cloudera Manager.
    Can someone verify the steps? If I use the 13 steps to upgrade cdh3 to cdh4
    then I wouldn't need Cloudera Manager in the first place.
    And it's documentation here already says it won't upgrade cdh and to use the
    other page.

    Please advise.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Joey Echeverria at Jun 7, 2012 at 3:18 pm
    So, you can actually upgrade to CM 4 first. CM 4 can manage both CDH3
    and CDH4 clusters.

    So, I'd start with this:

    https://ccp.cloudera.com/display/FREE4DOC/Upgrade+from+Cloudera+Manager+3.7.x+to+Cloudera+Manager+4.0

    and then tackle the CDH4 upgrade after the fact.

    -Joey
    On Thu, Jun 7, 2012 at 11:02 AM, Gary S wrote:
    Thanks Joey.  Would it be as simple as...
    1) Remove all components on all hosts (nn, snn, dn) per upgrade guide
    2) Apply steps 1-5 on all hosts per upgrade guide
    3) skip steps 6 thru 13?
    3) Install/Upgrade to Cloudera Manager 4
    4) Install to cluster as always?

    On the upgrade page, we're stuck at step 6 - upgrade the hdfs metadata.
    Since the hdfs-site.xml never had the dfs.name.dir, it defaults to the
    file:/// then we get:

    java.lang.IllegalArgumentException: Invalid URI for NameNode address (check
    fs.defaultFS): file:/// has no authority.

    Sorry to mix 2 issues into 1 thread. (upgrade if using CM plus exception on
    step 6)

    On Thursday, June 7, 2012 10:50:37 AM UTC-4, Joey Echeverria wrote:

    We probably need to make the docs around this better.

    What you need to do is follow the steps from the CDH3 to CDH4 upgrade
    guide that involve removing the CDH3 packages and installing the CDH4
    packages. You don't have to worry about the config files in
    /etc/<project> as those weren't being used and you dont' need to
    install the packages that just contain service scripts (e.g.
    hadoop-hdfs-namenode).

    I'll file a documentation JIRA to make the steps more clear.

    -Joey
    On Thu, Jun 7, 2012 at 10:43 AM, Gary S wrote:
    We used Cloudera Manager Free to install CDH3 in our cluster. Very
    simple.
    We are trying to upgrade to CDH4 but it seems as though the instructions
    I've found assume cdh3 was installed manually.
    here for example. (back up old configs, make note of manual settings)

    None of the config files on the hosts are configured manually since it
    was
    all done through Cloudera Manager.
    Can someone verify the steps?  If I use the 13 steps to upgrade cdh3 to
    cdh4
    then I wouldn't need Cloudera Manager in the first place.
    And it's documentation here already says it won't upgrade cdh and to use
    the
    other page.

    Please advise.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Gary S at Jun 7, 2012 at 4:03 pm
    Ok. We've actually done that already. We had CM4 managing CDH3.

    So then the 'tackle the CDH4 upgrade' is still where I have concerns.
    Based on your other post you mentioned not worrying about the config files
    and the services. Makes sense.

    But with CM4 it seems like we could follow steps 1,2,3 on the upgrade guide
    page and then use CM4 to roll out CDH4 to the cluster.
    Main questions:
    1) if we try that, what happens to the metadata upgrade in steps 6 & 13?
    Will CM4 do it?
    2) What will CM4 do with existing data if we point the dfs.name.dir to
    existing data?

    Thanks.
    On Thursday, June 7, 2012 11:17:58 AM UTC-4, Joey Echeverria wrote:

    So, you can actually upgrade to CM 4 first. CM 4 can manage both CDH3
    and CDH4 clusters.

    So, I'd start with this:


    https://ccp.cloudera.com/display/FREE4DOC/Upgrade+from+Cloudera+Manager+3.7.x+to+Cloudera+Manager+4.0

    and then tackle the CDH4 upgrade after the fact.

    -Joey
    On Thu, Jun 7, 2012 at 11:02 AM, Gary S wrote:
    Thanks Joey. Would it be as simple as...
    1) Remove all components on all hosts (nn, snn, dn) per upgrade guide
    2) Apply steps 1-5 on all hosts per upgrade guide
    3) skip steps 6 thru 13?
    3) Install/Upgrade to Cloudera Manager 4
    4) Install to cluster as always?

    On the upgrade page, we're stuck at step 6 - upgrade the hdfs metadata.
    Since the hdfs-site.xml never had the dfs.name.dir, it defaults to the
    file:/// then we get:

    java.lang.IllegalArgumentException: Invalid URI for NameNode address (check
    fs.defaultFS): file:/// has no authority.

    Sorry to mix 2 issues into 1 thread. (upgrade if using CM plus exception on
    step 6)

    On Thursday, June 7, 2012 10:50:37 AM UTC-4, Joey Echeverria wrote:

    We probably need to make the docs around this better.

    What you need to do is follow the steps from the CDH3 to CDH4 upgrade
    guide that involve removing the CDH3 packages and installing the CDH4
    packages. You don't have to worry about the config files in
    /etc/<project> as those weren't being used and you dont' need to
    install the packages that just contain service scripts (e.g.
    hadoop-hdfs-namenode).

    I'll file a documentation JIRA to make the steps more clear.

    -Joey
    On Thu, Jun 7, 2012 at 10:43 AM, Gary S wrote:
    We used Cloudera Manager Free to install CDH3 in our cluster. Very
    simple.
    We are trying to upgrade to CDH4 but it seems as though the
    instructions
    I've found assume cdh3 was installed manually.
    here for example. (back up old configs, make note of manual settings)

    None of the config files on the hosts are configured manually since
    it
    was
    all done through Cloudera Manager.
    Can someone verify the steps? If I use the 13 steps to upgrade cdh3
    to
    cdh4
    then I wouldn't need Cloudera Manager in the first place.
    And it's documentation here already says it won't upgrade cdh and to
    use
    the
    other page.

    Please advise.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Joey Echeverria at Jun 7, 2012 at 4:13 pm

    1) if we try that, what happens to the metadata upgrade in steps 6 & 13?
    Will CM4 do it?
    That's a good question, I'm betting the answer is those still need to
    be done by hand. On the plus side, CM4 should deploy config files /etc
    as well as the /var/run location, so maybe you can do the metadata
    upgrade by hand still?
    2) What will CM4 do with existing data if we point the dfs.name.dir to
    existing data?
    It would likely error out. There are built in protections to not
    format and not start if there' existing (meta)data that's in the way.
    I'd still recommend backing up your dfs.name.dir (as with any upgrade)
    to make sure that you have a reasonable chance to back out if you run
    into any issues.

    -Joey
    On Thu, Jun 7, 2012 at 12:02 PM, Gary S wrote:
    Ok. We've actually done that already.  We had CM4 managing CDH3.

    So then the 'tackle the CDH4 upgrade' is still where I have concerns.  Based
    on your other post you mentioned not worrying about the config files and the
    services. Makes sense.

    But with CM4 it seems like we could follow steps 1,2,3 on the upgrade guide
    page and then use CM4 to roll out CDH4 to the cluster.
    Main questions:
    1) if we try that, what happens to the metadata upgrade in steps 6 & 13?
    Will CM4 do it?
    2) What will CM4 do with existing data if we point the dfs.name.dir to
    existing data?

    Thanks.

    On Thursday, June 7, 2012 11:17:58 AM UTC-4, Joey Echeverria wrote:

    So, you can actually upgrade to CM 4 first. CM 4 can manage both CDH3
    and CDH4 clusters.

    So, I'd start with this:


    https://ccp.cloudera.com/display/FREE4DOC/Upgrade+from+Cloudera+Manager+3.7.x+to+Cloudera+Manager+4.0

    and then tackle the CDH4 upgrade after the fact.

    -Joey
    On Thu, Jun 7, 2012 at 11:02 AM, Gary S wrote:
    Thanks Joey.  Would it be as simple as...
    1) Remove all components on all hosts (nn, snn, dn) per upgrade guide
    2) Apply steps 1-5 on all hosts per upgrade guide
    3) skip steps 6 thru 13?
    3) Install/Upgrade to Cloudera Manager 4
    4) Install to cluster as always?

    On the upgrade page, we're stuck at step 6 - upgrade the hdfs metadata.
    Since the hdfs-site.xml never had the dfs.name.dir, it defaults to the
    file:/// then we get:

    java.lang.IllegalArgumentException: Invalid URI for NameNode address
    (check
    fs.defaultFS): file:/// has no authority.

    Sorry to mix 2 issues into 1 thread. (upgrade if using CM plus exception
    on
    step 6)

    On Thursday, June 7, 2012 10:50:37 AM UTC-4, Joey Echeverria wrote:

    We probably need to make the docs around this better.

    What you need to do is follow the steps from the CDH3 to CDH4 upgrade
    guide that involve removing the CDH3 packages and installing the CDH4
    packages. You don't have to worry about the config files in
    /etc/<project> as those weren't being used and you dont' need to
    install the packages that just contain service scripts (e.g.
    hadoop-hdfs-namenode).

    I'll file a documentation JIRA to make the steps more clear.

    -Joey

    On Thu, Jun 7, 2012 at 10:43 AM, Gary S <gary.s.snider@gmail.com>
    wrote:
    We used Cloudera Manager Free to install CDH3 in our cluster. Very
    simple.
    We are trying to upgrade to CDH4 but it seems as though the
    instructions
    I've found assume cdh3 was installed manually.
    here for example. (back up old configs, make note of manual settings)

    None of the config files on the hosts are configured manually since
    it
    was
    all done through Cloudera Manager.
    Can someone verify the steps?  If I use the 13 steps to upgrade cdh3
    to
    cdh4
    then I wouldn't need Cloudera Manager in the first place.
    And it's documentation here already says it won't upgrade cdh and to
    use
    the
    other page.

    Please advise.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Philip Langdale at Jun 7, 2012 at 4:27 pm
    Hi Gary,

    It's partially manual and partially automated. You should do the following:

    1) Stop all your services
    1a) There is no need to delete any configuration from CM
    2) Uninstall CDH3 packages and install CDH4 packages on all your machines
    2a) You can't really do a 'yum upgrade' style upgrade as the packages
    conflict
    and yum doesn't do the right thing
    2b) You should avoid installing the init-script packages like
    hadoop-hdfs-datanode,
    etc. These will end up trying to start processes outside of CM - which is
    bad.
    3) Go back to CM and choose the "Upgrade Cluster" item from the actions
    menu for
    your cluster. It will do the necessary config updates and trigger the HDFS
    metadata
    upgrade as described in the main CDH4 docs.
    4) When you're happy, you can finalise your HDFS upgrade from the HDFS
    service's
    action menu.

    --phil


    On 7 June 2012 07:43, Gary S wrote:

    We used Cloudera Manager Free to install CDH3 in our cluster. Very simple.
    We are trying to upgrade to CDH4 but it seems as though the instructions
    I've found assume cdh3 was installed manually.
    here
    <https://ccp.cloudera.com/display/CDH4DOC/Upgrading+from+CDH3+to+CDH4>for
    example. (back up old configs, make note of manual settings)

    None of the config files on the hosts are configured manually since it was
    all done through Cloudera Manager.
    Can someone verify the steps? If I use the 13 steps to upgrade cdh3 to
    cdh4 then I wouldn't need Cloudera Manager in the first place.
    And it's documentation here
    <https://ccp.cloudera.com/display/FREE4DOC/Cloudera+Manager+Free+Edition+Frequently+Asked+Questions#ClouderaManagerFreeEditionFrequentlyAskedQuestions-CanIupgradeCDHusingClouderaManager%3F>already
    says it won't upgrade cdh and to use the other page.

    Please advise.
  • Gary S at Jun 11, 2012 at 4:41 pm
    That worked Phil.

    Just a note for anyone following this thread until the docs are updated...
    the HDFS metadata upgrade must be done manually through the UI. Just open
    the hdfs service and that option should be in the 'Actions' drop-down.

    Thanks Phil and Joey for your help. The upgrade was definitely easier with
    CM4 using the modified instructions.
    On Thursday, June 7, 2012 12:27:06 PM UTC-4, Philip Langdale wrote:

    Hi Gary,

    It's partially manual and partially automated. You should do the following:

    1) Stop all your services
    1a) There is no need to delete any configuration from CM
    2) Uninstall CDH3 packages and install CDH4 packages on all your machines
    2a) You can't really do a 'yum upgrade' style upgrade as the packages
    conflict
    and yum doesn't do the right thing
    2b) You should avoid installing the init-script packages like
    hadoop-hdfs-datanode,
    etc. These will end up trying to start processes outside of CM - which is
    bad.
    3) Go back to CM and choose the "Upgrade Cluster" item from the actions
    menu for
    your cluster. It will do the necessary config updates and trigger the HDFS
    metadata
    upgrade as described in the main CDH4 docs.
    4) When you're happy, you can finalise your HDFS upgrade from the HDFS
    service's
    action menu.

    --phil


    On 7 June 2012 07:43, Gary S wrote:

    We used Cloudera Manager Free to install CDH3 in our cluster. Very simple.
    We are trying to upgrade to CDH4 but it seems as though the instructions
    I've found assume cdh3 was installed manually.
    here
    <https://ccp.cloudera.com/display/CDH4DOC/Upgrading+from+CDH3+to+CDH4>for
    example. (back up old configs, make note of manual settings)

    None of the config files on the hosts are configured manually since it
    was all done through Cloudera Manager.
    Can someone verify the steps? If I use the 13 steps to upgrade cdh3 to
    cdh4 then I wouldn't need Cloudera Manager in the first place.
    And it's documentation here
    <https://ccp.cloudera.com/display/FREE4DOC/Cloudera+Manager+Free+Edition+Frequently+Asked+Questions#ClouderaManagerFreeEditionFrequentlyAskedQuestions-CanIupgradeCDHusingClouderaManager%3F>already
    says it won't upgrade cdh and to use the other page.

    Please advise.
  • Philip Langdale at Jun 11, 2012 at 5:06 pm
    Hi Gary,

    I'm glad it went smoothly. Strictly speaking you don't need to trigger the
    metadata upgrade
    manually - the cluster-level upgrade action, initiates that for you (it's
    available in the menu
    primarily for emergencies, in case it fails and you need to re-try), but
    you do need to do
    the Finalize manually (it would be dangerous for us to do that
    automatically, as it removes
    the rollback ability).

    --phil


    On 11 June 2012 09:41, Gary S wrote:

    That worked Phil.

    Just a note for anyone following this thread until the docs are updated...
    the HDFS metadata upgrade must be done manually through the UI. Just open
    the hdfs service and that option should be in the 'Actions' drop-down.

    Thanks Phil and Joey for your help. The upgrade was definitely easier
    with CM4 using the modified instructions.

    On Thursday, June 7, 2012 12:27:06 PM UTC-4, Philip Langdale wrote:

    Hi Gary,

    It's partially manual and partially automated. You should do the
    following:

    1) Stop all your services
    1a) There is no need to delete any configuration from CM
    2) Uninstall CDH3 packages and install CDH4 packages on all your machines
    2a) You can't really do a 'yum upgrade' style upgrade as the packages
    conflict
    and yum doesn't do the right thing
    2b) You should avoid installing the init-script packages like
    hadoop-hdfs-datanode,
    etc. These will end up trying to start processes outside of CM - which is
    bad.
    3) Go back to CM and choose the "Upgrade Cluster" item from the actions
    menu for
    your cluster. It will do the necessary config updates and trigger the
    HDFS metadata
    upgrade as described in the main CDH4 docs.
    4) When you're happy, you can finalise your HDFS upgrade from the HDFS
    service's
    action menu.

    --phil


    On 7 June 2012 07:43, Gary S wrote:

    We used Cloudera Manager Free to install CDH3 in our cluster. Very
    simple.
    We are trying to upgrade to CDH4 but it seems as though the instructions
    I've found assume cdh3 was installed manually.
    here
    <https://ccp.cloudera.com/display/CDH4DOC/Upgrading+from+CDH3+to+CDH4>for
    example. (back up old configs, make note of manual settings)

    None of the config files on the hosts are configured manually since it
    was all done through Cloudera Manager.
    Can someone verify the steps? If I use the 13 steps to upgrade cdh3 to
    cdh4 then I wouldn't need Cloudera Manager in the first place.
    And it's documentation here
    <https://ccp.cloudera.com/display/FREE4DOC/Cloudera+Manager+Free+Edition+Frequently+Asked+Questions#ClouderaManagerFreeEditionFrequentlyAskedQuestions-CanIupgradeCDHusingClouderaManager%3F>already
    says it won't upgrade cdh and to use the other page.

    Please advise.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedJun 7, '12 at 2:43p
activeJun 11, '12 at 5:06p
posts9
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase