Manually Upgrade CDH 5.2 in CM 5

I was interrupted again when upgrading CDH.

HDFS

This time, NameNode was not started, I have to bring them up and resume the upgrade progress.
I didn’t save any log about NN’s error, stop all HDFS components and ran ‘Upgrade HDFS Metadata‘, then start HDFS.

YARN

Next, YARN.
I started YARN, and then all other services. Hive went down, then YARN.
I checked CM’s monitor:

I found both instances of ResourceManager were ‘Standby‘.

Here is what I found from /var/log/hadoop-yarn/hadoop-cmf-yarn-RESOURCEMANAGER-hadoop4.xxx.com.log.out

Google helps a lot: http://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-5-YARN-Resource-Manager-HA-deadlock-in-Kerberos-cluster/td-p/14396

In /opt/cloudera/parcels/CDH/lib/zookeeper/bin/zkCli.sh,
Do

one by one, because zkCli.sh does not have wildcard support.

Hive

I just guessed that hive didn’t work because of YARN, but I was wrong.
I checked all hive related commands executed by CM:

So I stopped Hive, ran Update Hive Metastore NameNodes and Upgrade Hive Metastore Database Schema, none of them worked but with the error message above.
I got more from logs:

The schemaTool reminded me, I manually upgraded hive metastore in Feb: Hive MetaStore Schema Upgrade Failed When Upgrading CDH5.
But this time, dbType should be postgres instead of derby.(Derby is not supported by Impala, that’s why I changed to postgresql embedded in Cloudera Manager.)

I cann’t find the terminal output, but when I ran:

I found a similar output (only first few lines) to the blog post above, saying schemaTool was trying to connect to derby

I re-deploy hive’s client configuration, and checked /etc/hive/conf/hive-site.xml, and compared with /var/run/cloudera-scm-agent/process/4525-hive-HIVEMETASTORE/hive-site.xml,
xml under /etc uses hive metastore’s thrift server and that under CM’s running folder speicified the exact database connection. schemaTool uses the /etc one.
So I replaced /etc one with CM’s, and then redo upgradeSchema:

Same error as I saw in CM’s log, plpgsql does not exist. Fix this by:

You can find password from the xml I mentioned above of file like

If you meet error message saying OWNER_NAME or OWNER_TYPE already exists in table DBS, open /opt/cloudera/parcels/CDH/lib/hive/scripts/metastore/upgrade/postgres/016-HIVE-6386.postgres.sql and comment/delete the two ALTER TABLE lines.

Manually Upgrade CDH 5.2 in CM 5 by @sskaje: https://sskaje.me/2014/10/manually-upgrade-cdh-5-2-cm-5/

Incoming search terms:

Fix Alternatives for Cloudera Manager + CDH

Earlier post: Fix Hadoop Conf Alternatives for CDH5

I tried to upgrade Cloudera Manager + CDH 5.0.0 beta 1 and beta 2 from CM+CDH 4 then downgrade and delete, found many alternatives were installed on my small cluster, that made my lately installed CM+CDH 4 and CM+CDH 5 not working well, all because of the dirty uninstallation of CM + CDH 5 beta-s.

To fix these alternatives, I wrote a python script, read default alternative configurations, check all currently installed alternatives and delete broken links, install defaults and bring down priority, so we can use ‘Deploy Client Configuration’ in CM to set up the correct ones.

Repository: https://github.com/sskaje/cm_fix_alternatives

Tested only under centos 6.

Fix Alternatives for Cloudera Manager + CDH by @sskaje: https://sskaje.me/2014/04/fix-alternatives-cloudera-manager-cdh/

Fix Hadoop Conf Alternatives for CDH5

I’m using CDH5, upgraded failed from CDH4 and then reinstalled directly.
/etc/hadoop/conf is linked to /etc/hadoop/conf/conf.cloudera.mapreduce1/.
Deploy Client Configuration does not make it right.

The way fix it is manually set a new path and remove the old one, like:

But the next time you try Deploy Client Configuration would corrupt it again.

Continue reading “Fix Hadoop Conf Alternatives for CDH5” »

Fix Hadoop Conf Alternatives for CDH5 by @sskaje: https://sskaje.me/2014/02/fix-hadoop-conf-alternatives-cdh5/

Incoming search terms:

Hive MetaStore Schema Upgrade Failed When Upgrading CDH5

Manually upgrade:

Continue reading “Hive MetaStore Schema Upgrade Failed When Upgrading CDH5” »

Hive MetaStore Schema Upgrade Failed When Upgrading CDH5 by @sskaje: https://sskaje.me/2014/02/hive-metastore-schema-upgrade-failed-upgrading-cdh5/

Cloudera Archive Mirror Updated for CM5 & CDH5

Latest Updates @ https://sskaje.me/cloudera-mirror/

Cloudera just released it’s CDH 5 beta download here, this time their don’t use the beta.cloudera.com as non-release product’s repo.

URL: http://cloudera.rst.im/ RHEL/CentOS 6 Only.

You may download Cloudera Manager 4 installer from http://cloudera.rst.im/cm4/installer/latest/
And use http://cloudera.rst.im/cm4/redhat/6/x86_64/cm/4.7.2/ as your yum repo for CM 4.7.2.

Or Cloudera Manager 5 installer from http://cloudera.rst.im/cm5/installer/latest/,
And use http://cloudera.rst.im/cm4/redhat/6/x86_64/cm/5/ as your yum repo for CM 5.

Continue reading “Cloudera Archive Mirror Updated for CM5 & CDH5” »

Cloudera Archive Mirror Updated for CM5 & CDH5 by @sskaje: https://sskaje.me/2013/10/cloudera-archive-mirror-updated-for-cm5-cdh5/

Incoming search terms: