How to upgrade to cdh4 if cloudera manager installed cdh3 in cluster

We probably need to make the docs around this better.

What you need to do is follow the steps from the CDH3 to CDH4 upgrade
guide that involve removing the CDH3 packages and installing the CDH4
packages. You don't have to worry about the config files in
/etc/<project> as those weren't being used and you dont' need to
install the packages that just contain service scripts (e.g.
hadoop-hdfs-namenode).

I'll file a documentation JIRA to make the steps more clear.

-Joey

Thanks Joey. Would it be as simple as...
1) Remove all components on all hosts (nn, snn, dn) per upgrade guide
2) Apply steps 1-5 on all hosts per upgrade guide
3) skip steps 6 thru 13?
3) Install/Upgrade to Cloudera Manager 4
4) Install to cluster as always?

On the upgrade page, we're stuck at step 6 - upgrade the hdfs metadata.
Since the hdfs-site.xml never had the dfs.name.dir, it defaults to the
file:/// then we get:

java.lang.IllegalArgumentException: Invalid URI for NameNode address (check
fs.defaultFS): file:/// has no authority.

Sorry to mix 2 issues into 1 thread. (upgrade if using CM plus exception on
step 6)

So, you can actually upgrade to CM 4 first. CM 4 can manage both CDH3
and CDH4 clusters.

So, I'd start with this:

https://ccp.cloudera.com/display/FREE4DOC/Upgrade+from+Cloudera+Manager+3.7.x+to+Cloudera+Manager+4.0

and then tackle the CDH4 upgrade after the fact.

-Joey

Ok. We've actually done that already. We had CM4 managing CDH3.

So then the 'tackle the CDH4 upgrade' is still where I have concerns.
Based on your other post you mentioned not worrying about the config files
and the services. Makes sense.

But with CM4 it seems like we could follow steps 1,2,3 on the upgrade guide
page and then use CM4 to roll out CDH4 to the cluster.
Main questions:
1) if we try that, what happens to the metadata upgrade in steps 6 & 13?
Will CM4 do it?
2) What will CM4 do with existing data if we point the dfs.name.dir to
existing data?

Thanks.

That's a good question, I'm betting the answer is those still need to
be done by hand. On the plus side, CM4 should deploy config files /etc
as well as the /var/run location, so maybe you can do the metadata
upgrade by hand still?
It would likely error out. There are built in protections to not
format and not start if there' existing (meta)data that's in the way.
I'd still recommend backing up your dfs.name.dir (as with any upgrade)
to make sure that you have a reasonable chance to back out if you run
into any issues.

-Joey

Hi Gary,

It's partially manual and partially automated. You should do the following:

1) Stop all your services
1a) There is no need to delete any configuration from CM
2) Uninstall CDH3 packages and install CDH4 packages on all your machines
2a) You can't really do a 'yum upgrade' style upgrade as the packages
conflict
and yum doesn't do the right thing
2b) You should avoid installing the init-script packages like
hadoop-hdfs-datanode,
etc. These will end up trying to start processes outside of CM - which is
bad.
3) Go back to CM and choose the "Upgrade Cluster" item from the actions
menu for
your cluster. It will do the necessary config updates and trigger the HDFS
metadata
upgrade as described in the main CDH4 docs.
4) When you're happy, you can finalise your HDFS upgrade from the HDFS
service's
action menu.

--phil

That worked Phil.

Just a note for anyone following this thread until the docs are updated...
the HDFS metadata upgrade must be done manually through the UI. Just open
the hdfs service and that option should be in the 'Actions' drop-down.

Thanks Phil and Joey for your help. The upgrade was definitely easier with
CM4 using the modified instructions.

Hi Gary,

I'm glad it went smoothly. Strictly speaking you don't need to trigger the
metadata upgrade
manually - the cluster-level upgrade action, initiates that for you (it's
available in the menu
primarily for emergencies, in case it fails and you need to re-try), but
you do need to do
the Finalize manually (it would be dangerous for us to do that
automatically, as it removes
the rollback ability).

--phil