Shark 0.9.1 Snapshot Working with CDH 4.6.0 Now

By @sskaje
Link: https://sskaje.me/2014/03/shark-0-9-1-snapshot-working-cdh-4-6-0/

The Shark 0.9.x(0.9.0, 0.9.1) are still pre-release: https://github.com/amplab/shark/releases

Previously supplied 0.9.0 Prebuild with Hadoop2, CDH4.5.0: shark-0.9.0-hadoop2-bin is not really working with CDH 4.5.0, so I tried to compile Build Shark 0.9 for CDH 4, Install Spark/Shark on CDH 4

Some days later, 0.9.1 is out at the https://github.com/amplab/shark/tree/branch-0.9, the patched hive is uploaded to maven repo and can be put in lib_managed now.

I have 5 nodes for my CDH 4 test, hadoop1 – hadoop5, HDFS NameNode HA on hadoop5 + hadoop4, nameservice1.
Spark master locates on hadoop5, works on all nodes.

Problem

I tried to compile with:

export SCALA_HOME=/opt/cloudera/parcels/SPARK/lib/spark;
SHARK_HADOOP_VERSION=2.0.0-cdh4.5.0 ./sbt/sbt package

1 2	export SCALA_HOME=/opt/cloudera/parcels/SPARK/lib/spark; SHARK_HADOOP_VERSION=2.0.0-cdh4.5.0 ./sbt/sbt package

This works for the first build of SHark 0.9.0 with CDH 4.5.0, but for 0.9.1 or master branch, nameservice1 is resolvd by dns and hit a wildcard record makes namenode not accessible by Shark, meta data are ok.

Errors:

14/02/23 23:12:04 DEBUG fs.FileSystem: Creating filesystem for hdfs://nameservice1:8020/tmp/hive-hdfs/hive_2014-02-23_23-12-04_120_4758827093065246377
14/02/23 23:12:04 DEBUG conf.Configuration: java.io.IOException: config()
	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
	at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:60)
	at org.apache.hadoop.net.NetUtils.makeSocketAddr(NetUtils.java:186)
	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:166)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:198)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:88)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
	at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180)
	at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)
	at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)
	at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)
	at shark.SharkDriver.compile(SharkDriver.scala:210)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
	at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
	at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)
	at shark.SharkCliDriver.main(SharkCliDriver.scala)
14/02/23 23:12:04 DEBUG ipc.Client: The ping interval is60000ms.
14/02/23 23:12:04 DEBUG ipc.Client: Use SIMPLE authentication for protocol ClientProtocol
14/02/23 23:12:04 DEBUG ipc.Client: Connecting to nameservice1/xx.xx.xx.xx:8020
14/02/23 23:12:05 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 0 time(s).
14/02/23 23:12:06 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 1 time(s).
14/02/23 23:12:07 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 2 time(s).
14/02/23 23:12:08 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 3 time(s).
14/02/23 23:12:09 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 4 time(s).
14/02/23 23:12:10 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 5 time(s).
14/02/23 23:12:11 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 6 time(s).
14/02/23 23:12:12 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 7 time(s).
14/02/23 23:12:13 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 8 time(s).
14/02/23 23:12:14 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 9 time(s).
14/02/23 23:12:14 DEBUG ipc.Client: closing ipc connection to nameservice1/xx.xx.xx.xx:8020: Network is unreachable
java.net.SocketException: Network is unreachable
	at sun.nio.ch.Net.connect(Native Method)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
	at org.apache.hadoop.ipc.Client.call(Client.java:1050)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
	at $Proxy12.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
	at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
	at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180)
	at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)
	at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)
	at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)
	at shark.SharkDriver.compile(SharkDriver.scala:210)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
	at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
	at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)
	at shark.SharkCliDriver.main(SharkCliDriver.scala)
14/02/23 23:12:14 DEBUG ipc.Client: IPC Client (47) connection to nameservice1/xx.xx.xx.xx:8020 from hdfs: closed
FAILED: Hive Internal Error: java.lang.RuntimeException(java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable)
14/02/23 23:12:14 ERROR shark.SharkDriver: FAILED: Hive Internal Error: java.lang.RuntimeException(java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable)
java.lang.RuntimeException: java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable
	at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:190)
	at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)
	at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)
	at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)
	at shark.SharkDriver.compile(SharkDriver.scala:210)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
	at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
	at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)
	at shark.SharkCliDriver.main(SharkCliDriver.scala)
Caused by: java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
	at org.apache.hadoop.ipc.Client.call(Client.java:1075)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
	at $Proxy12.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
	at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
	at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180)
	... 13 more
Caused by: java.net.SocketException: Network is unreachable
	at sun.nio.ch.Net.connect(Native Method)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
	at org.apache.hadoop.ipc.Client.call(Client.java:1050)
	... 27 more

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

14/02/23 23:12:04 DEBUG fs.FileSystem: Creating filesystem for hdfs://nameservice1:8020/tmp/hive-hdfs/hive_2014-02-23_23-12-04_120_4758827093065246377

14/02/23 23:12:04 DEBUG conf.Configuration: java.io.IOException: config()

at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)

at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)

at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:60)

at org.apache.hadoop.net.NetUtils.makeSocketAddr(NetUtils.java:186)

at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:166)

at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:198)

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:88)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)

at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180)

at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)

at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)

at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)

at shark.SharkDriver.compile(SharkDriver.scala:210)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)

at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)

at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)

at shark.SharkCliDriver.main(SharkCliDriver.scala)

14/02/23 23:12:04 DEBUG ipc.Client: The ping interval is60000ms.

14/02/23 23:12:04 DEBUG ipc.Client: Use SIMPLE authentication for protocol ClientProtocol

14/02/23 23:12:04 DEBUG ipc.Client: Connecting to nameservice1/xx.xx.xx.xx:8020

14/02/23 23:12:05 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 0 time(s).

14/02/23 23:12:06 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 1 time(s).

14/02/23 23:12:07 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 2 time(s).

14/02/23 23:12:08 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 3 time(s).

14/02/23 23:12:09 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 4 time(s).

14/02/23 23:12:10 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 5 time(s).

14/02/23 23:12:11 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 6 time(s).

14/02/23 23:12:12 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 7 time(s).

14/02/23 23:12:13 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 8 time(s).

14/02/23 23:12:14 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 9 time(s).

14/02/23 23:12:14 DEBUG ipc.Client: closing ipc connection to nameservice1/xx.xx.xx.xx:8020: Network is unreachable

java.net.SocketException: Network is unreachable

at sun.nio.ch.Net.connect(Native Method)

at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)

at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)

at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)

at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)

at org.apache.hadoop.ipc.Client.call(Client.java:1050)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)

at $Proxy12.getProtocolVersion(Unknown Source)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)

at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)

at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)

at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)

at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180)

at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)

at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)

at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)

at shark.SharkDriver.compile(SharkDriver.scala:210)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)

at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)

at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)

at shark.SharkCliDriver.main(SharkCliDriver.scala)

14/02/23 23:12:14 DEBUG ipc.Client: IPC Client (47) connection to nameservice1/xx.xx.xx.xx:8020 from hdfs: closed

FAILED: Hive Internal Error: java.lang.RuntimeException(java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable)

14/02/23 23:12:14 ERROR shark.SharkDriver: FAILED: Hive Internal Error: java.lang.RuntimeException(java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable)

java.lang.RuntimeException: java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable

at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:190)

at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)

at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)

at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)

at shark.SharkDriver.compile(SharkDriver.scala:210)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)

at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)

at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)

at shark.SharkCliDriver.main(SharkCliDriver.scala)

Caused by: java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable

at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)

at org.apache.hadoop.ipc.Client.call(Client.java:1075)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)

at $Proxy12.getProtocolVersion(Unknown Source)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)

at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)

at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)

at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)

at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180)

... 13 more

Caused by: java.net.SocketException: Network is unreachable

at sun.nio.ch.Net.connect(Native Method)

at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)

at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)

at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)

at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)

at org.apache.hadoop.ipc.Client.call(Client.java:1050)

... 27 more

After several test base on different branches/commits, I tried to minimize the configuration:
1 Change the hive-site.xml, set fs.default.name and fs.defaultFS to hadoop5
2 Edit the shark-env.sh, comment the line

export MASTER="spark://hadoop5.xxx.com:7077"

1	export MASTER="spark://hadoop5.xxx.com:7077"

This time, shark is working at standalone mode and communicating directly with namenode, if there’s any error, that might cause the stupid HA problem.

New error came up:

14/03/04 18:00:15 DEBUG ipc.Client: closing ipc connection to hadoop5/172.16.3.20:8020: Server IPC version 7 cannot communicate with client version 4
org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
	at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:823)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:749)
14/03/04 18:00:15 DEBUG ipc.Client: IPC Client (47) connection to hadoop5/172.16.3.20:8020 from root: closed
14/03/04 18:00:15 DEBUG ipc.Client: IPC Client (47) connection to hadoop5/172.16.3.20:8020 from root: stopped, remaining connections 0
FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4)
14/03/04 18:00:15 ERROR shark.SharkDriver: FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4)
java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
	at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:190)
	at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)
	at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)
	at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:137)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)
	at shark.SharkDriver.compile(SharkDriver.scala:210)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
	at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
	at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)
	at shark.SharkCliDriver.main(SharkCliDriver.scala)
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
	at org.apache.hadoop.ipc.Client.call(Client.java:1070)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
	at $Proxy12.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
	at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
	at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180)
	... 13 more

14/03/04 18:00:15 DEBUG ipc.Client: closing ipc connection to hadoop5/172.16.3.20:8020: Server IPC version 7 cannot communicate with client version 4

org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4

at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:823)

at org.apache.hadoop.ipc.Client$Connection.run(Client.java:749)

14/03/04 18:00:15 DEBUG ipc.Client: IPC Client (47) connection to hadoop5/172.16.3.20:8020 from root: closed

14/03/04 18:00:15 DEBUG ipc.Client: IPC Client (47) connection to hadoop5/172.16.3.20:8020 from root: stopped, remaining connections 0

FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4)

14/03/04 18:00:15 ERROR shark.SharkDriver: FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4)

java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4

at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:190)

at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)

at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)

at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:137)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)

at shark.SharkDriver.compile(SharkDriver.scala:210)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)

at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)

at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)

at shark.SharkCliDriver.main(SharkCliDriver.scala)

Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4

at org.apache.hadoop.ipc.Client.call(Client.java:1070)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)

at $Proxy12.getProtocolVersion(Unknown Source)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)

at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)

at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)

at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)

at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180)

... 13 more

Another mismatched hadoop IPC server/client versions!

Last time I saw this was Install Spark/Shark on CDH 4.
I find all jars named like *hadoop* under shark/lib_managed/, found this:

./lib_managed/jars/org.apache.hadoop
./lib_managed/jars/org.apache.hadoop/hadoop-auth
./lib_managed/jars/org.apache.hadoop/hadoop-auth/hadoop-auth-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/cloudera-jets3t
./lib_managed/jars/org.apache.hadoop/cloudera-jets3t/cloudera-jets3t-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-nodemanager
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-nodemanager/hadoop-yarn-server-nodemanager-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-shuffle
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-shuffle/hadoop-mapreduce-client-shuffle-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-common
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-common/hadoop-mapreduce-client-common-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-jobclient
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-jobclient/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-core
./lib_managed/jars/org.apache.hadoop/hadoop-core/hadoop-core-1.0.4.jar
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-api
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-client
./lib_managed/jars/org.apache.hadoop/hadoop-client/hadoop-client-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-common
./lib_managed/jars/org.apache.hadoop/hadoop-common/hadoop-common-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-app
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-app/hadoop-mapreduce-client-app-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-client
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-client/hadoop-yarn-client-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-hdfs
./lib_managed/jars/org.apache.hadoop/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-common
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-common/hadoop-yarn-server-common-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-annotations
./lib_managed/jars/org.apache.hadoop/hadoop-annotations/hadoop-annotations-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-test
./lib_managed/jars/org.apache.hadoop/hadoop-test/hadoop-test-0.20.2.jar
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-core
./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-core/hadoop-mapreduce-client-core-2.0.0-cdh4.6.0.jar
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-common
./lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop

./lib_managed/jars/org.apache.hadoop/hadoop-auth

./lib_managed/jars/org.apache.hadoop/hadoop-auth/hadoop-auth-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/cloudera-jets3t

./lib_managed/jars/org.apache.hadoop/cloudera-jets3t/cloudera-jets3t-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-nodemanager

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-nodemanager/hadoop-yarn-server-nodemanager-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-shuffle

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-shuffle/hadoop-mapreduce-client-shuffle-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-common

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-common/hadoop-mapreduce-client-common-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-jobclient

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-jobclient/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-core

./lib_managed/jars/org.apache.hadoop/hadoop-core/hadoop-core-1.0.4.jar

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-api

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-client

./lib_managed/jars/org.apache.hadoop/hadoop-client/hadoop-client-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-common

./lib_managed/jars/org.apache.hadoop/hadoop-common/hadoop-common-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-app

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-app/hadoop-mapreduce-client-app-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-client

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-client/hadoop-yarn-client-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-hdfs

./lib_managed/jars/org.apache.hadoop/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-common

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-common/hadoop-yarn-server-common-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-annotations

./lib_managed/jars/org.apache.hadoop/hadoop-annotations/hadoop-annotations-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-test

./lib_managed/jars/org.apache.hadoop/hadoop-test/hadoop-test-0.20.2.jar

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-core

./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-core/hadoop-mapreduce-client-core-2.0.0-cdh4.6.0.jar

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-common

./lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.0.0-cdh4.6.0.jar

I don’t understand why there’s a ‘hadoop-core-1.0.4.jar‘.
I tried to delete this stupid file and copy ‘/opt/cloudera/parcels/CDH/lib/hadoop/client-0.20/hadoop-core-2.0.0-mr1-cdh4.6.0.jar‘ to ./lib_managed/jars/org.apache.hadoop/hadoop-core.

Second test working as standalone mode, it works!
Bring all configurations back, works again!

DO NOT FORGET TO SYNC LATEST SHARK BUILD TO ALL SERVERS!

Why?

Get back to compile issue.
I’ve set ‘SHARK_HADOOP_VERSION=2.0.0-cdh4.6.0’, why a hadoop-core-1.0.4.jar is downloaded and put into lib_managed?
I tried grep ‘1\.0\.4’ in the shark folder, many lines mostly under target/; sbt clean and package, still.
Changed project/ScalaBuild and bin/dev/run-tests-from-scratch, nothing changed.
Why?

http://spark.incubator.apache.org/docs/latest/hadoop-third-party-distributions.html

 SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.6.0 ./sbt/sbt package

1	SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.6.0 ./sbt/sbt package

Download

I’ve added a snapshot build(commit fe75a886) for CDH 4.6.0 working for me here: http://cloudera.rst.im/shark/.

Shark 0.9.1 Snapshot Working with CDH 4.6.0 Now by @sskaje: https://sskaje.me/2014/03/shark-0-9-1-snapshot-working-cdh-4-6-0/

Problem

Why?

Download

Related posts: