Ha | @sskaje

By @sskaje
Link: https://sskaje.me/2014/03/shark-0-9-1-snapshot-working-cdh-4-6-0/

The Shark 0.9.x(0.9.0, 0.9.1) are still pre-release: https://github.com/amplab/shark/releases

Previously supplied 0.9.0 Prebuild with Hadoop2, CDH4.5.0: shark-0.9.0-hadoop2-bin is not really working with CDH 4.5.0, so I tried to compile Build Shark 0.9 for CDH 4, Install Spark/Shark on CDH 4

Some days later, 0.9.1 is out at the https://github.com/amplab/shark/tree/branch-0.9, the patched hive is uploaded to maven repo and can be put in lib_managed now.

I have 5 nodes for my CDH 4 test, hadoop1 – hadoop5, HDFS NameNode HA on hadoop5 + hadoop4, nameservice1.
Spark master locates on hadoop5, works on all nodes.

Shark 0.9.1 Snapshot Working with CDH 4.6.0 Now by @sskaje: https://sskaje.me/2014/03/shark-0-9-1-snapshot-working-cdh-4-6-0/

By @sskaje
Link: https://sskaje.me/2014/02/cdh-4-ha-related-problems/

Problem

I meet ‘Failed to initialize High Availability state in ZooKeeper. This might be because a ZNode for this nameservice is already created. Either remove the ZNode, or to reuse the ZNode skip this step and simply start the NameNodes and Failover Controllers. To retry, use the “Initialize High Availability state in ZooKeeper” command available as a Failover Controller action.‘ When I was trying to enable automatic failover for HDFS in Cloudera Manager after HA being enabled.

Error logs:

Failed to initialize High Availability state in ZooKeeper from failovercontroller (hadoop5): 14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.39/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/lib/native
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.18.1.el6.x86_64
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-hdfs
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/519-failover-controller-initialize-znode
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop5.xxx.com:2181,hadoop4.xxx.com:2181,hadoop3.xxx.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@3219762f
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5.xxx.com/172.16.3.20:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Socket connection established to hadoop5.xxx.com/172.16.3.20:2181, initiating session
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5.xxx.com/172.16.3.20:2181, sessionid = 0x2447689aed10113, negotiated timeout = 5000
14/02/28 14:51:24 INFO ha.ActiveStandbyElector: Session connected.
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Session: 0x2447689aed10113 closed
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: EventThread shut down

Failed to initialize High Availability state in ZooKeeper from failovercontroller (hadoop5): 14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.39/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/lib/native

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.18.1.el6.x86_64

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-hdfs

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/519-failover-controller-initialize-znode

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop5.xxx.com:2181,hadoop4.xxx.com:2181,hadoop3.xxx.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@3219762f

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5.xxx.com/172.16.3.20:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Socket connection established to hadoop5.xxx.com/172.16.3.20:2181, initiating session

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5.xxx.com/172.16.3.20:2181, sessionid = 0x2447689aed10113, negotiated timeout = 5000

14/02/28 14:51:24 INFO ha.ActiveStandbyElector: Session connected.

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Session: 0x2447689aed10113 closed

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: EventThread shut down

CDH 4 HA Related Problems by @sskaje: https://sskaje.me/2014/02/cdh-4-ha-related-problems/