The Shark 0.9.x(0.9.0, 0.9.1) are still pre-release: https://github.com/amplab/shark/releases
Previously supplied 0.9.0 Prebuild with Hadoop2, CDH4.5.0: shark-0.9.0-hadoop2-bin is not really working with CDH 4.5.0, so I tried to compile Build Shark 0.9 for CDH 4, Install Spark/Shark on CDH 4
Some days later, 0.9.1 is out at the https://github.com/amplab/shark/tree/branch-0.9, the patched hive is uploaded to maven repo and can be put in lib_managed now.
I have 5 nodes for my CDH 4 test, hadoop1 – hadoop5, HDFS NameNode HA on hadoop5 + hadoop4, nameservice1.
Spark master locates on hadoop5, works on all nodes.
Problem
I tried to compile with:
1 2 |
export SCALA_HOME=/opt/cloudera/parcels/SPARK/lib/spark; SHARK_HADOOP_VERSION=2.0.0-cdh4.5.0 ./sbt/sbt package |
This works for the first build of SHark 0.9.0 with CDH 4.5.0, but for 0.9.1 or master branch, nameservice1 is resolvd by dns and hit a wildcard record makes namenode not accessible by Shark, meta data are ok.
Errors:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 |
14/02/23 23:12:04 DEBUG fs.FileSystem: Creating filesystem for hdfs://nameservice1:8020/tmp/hive-hdfs/hive_2014-02-23_23-12-04_120_4758827093065246377 14/02/23 23:12:04 DEBUG conf.Configuration: java.io.IOException: config() at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227) at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214) at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:60) at org.apache.hadoop.net.NetUtils.makeSocketAddr(NetUtils.java:186) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:166) at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:198) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:88) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180) at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231) at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059) at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279) at shark.SharkDriver.compile(SharkDriver.scala:210) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at shark.SharkCliDriver$.main(SharkCliDriver.scala:235) at shark.SharkCliDriver.main(SharkCliDriver.scala) 14/02/23 23:12:04 DEBUG ipc.Client: The ping interval is60000ms. 14/02/23 23:12:04 DEBUG ipc.Client: Use SIMPLE authentication for protocol ClientProtocol 14/02/23 23:12:04 DEBUG ipc.Client: Connecting to nameservice1/xx.xx.xx.xx:8020 14/02/23 23:12:05 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 0 time(s). 14/02/23 23:12:06 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 1 time(s). 14/02/23 23:12:07 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 2 time(s). 14/02/23 23:12:08 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 3 time(s). 14/02/23 23:12:09 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 4 time(s). 14/02/23 23:12:10 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 5 time(s). 14/02/23 23:12:11 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 6 time(s). 14/02/23 23:12:12 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 7 time(s). 14/02/23 23:12:13 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 8 time(s). 14/02/23 23:12:14 INFO ipc.Client: Retrying connect to server: nameservice1/xx.xx.xx.xx:8020. Already tried 9 time(s). 14/02/23 23:12:14 DEBUG ipc.Client: closing ipc connection to nameservice1/xx.xx.xx.xx:8020: Network is unreachable java.net.SocketException: Network is unreachable at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy12.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180) at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231) at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059) at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279) at shark.SharkDriver.compile(SharkDriver.scala:210) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at shark.SharkCliDriver$.main(SharkCliDriver.scala:235) at shark.SharkCliDriver.main(SharkCliDriver.scala) 14/02/23 23:12:14 DEBUG ipc.Client: IPC Client (47) connection to nameservice1/xx.xx.xx.xx:8020 from hdfs: closed FAILED: Hive Internal Error: java.lang.RuntimeException(java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable) 14/02/23 23:12:14 ERROR shark.SharkDriver: FAILED: Hive Internal Error: java.lang.RuntimeException(java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable) java.lang.RuntimeException: java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:190) at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231) at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059) at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:139) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279) at shark.SharkDriver.compile(SharkDriver.scala:210) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at shark.SharkCliDriver$.main(SharkCliDriver.scala:235) at shark.SharkCliDriver.main(SharkCliDriver.scala) Caused by: java.io.IOException: Call to nameservice1/xx.xx.xx.xx:8020 failed on local exception: java.net.SocketException: Network is unreachable at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107) at org.apache.hadoop.ipc.Client.call(Client.java:1075) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy12.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180) ... 13 more Caused by: java.net.SocketException: Network is unreachable at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206) at org.apache.hadoop.ipc.Client.call(Client.java:1050) ... 27 more |
After several test base on different branches/commits, I tried to minimize the configuration:
1 Change the hive-site.xml, set fs.default.name and fs.defaultFS to hadoop5
2 Edit the shark-env.sh, comment the line
1 |
export MASTER="spark://hadoop5.xxx.com:7077" |
This time, shark is working at standalone mode and communicating directly with namenode, if there’s any error, that might cause the stupid HA problem.
New error came up:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
14/03/04 18:00:15 DEBUG ipc.Client: closing ipc connection to hadoop5/172.16.3.20:8020: Server IPC version 7 cannot communicate with client version 4 org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4 at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:823) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:749) 14/03/04 18:00:15 DEBUG ipc.Client: IPC Client (47) connection to hadoop5/172.16.3.20:8020 from root: closed 14/03/04 18:00:15 DEBUG ipc.Client: IPC Client (47) connection to hadoop5/172.16.3.20:8020 from root: stopped, remaining connections 0 FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4) 14/03/04 18:00:15 ERROR shark.SharkDriver: FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4) java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4 at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:190) at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231) at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059) at shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:137) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279) at shark.SharkDriver.compile(SharkDriver.scala:210) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:330) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at shark.SharkCliDriver$.main(SharkCliDriver.scala:235) at shark.SharkCliDriver.main(SharkCliDriver.scala) Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4 at org.apache.hadoop.ipc.Client.call(Client.java:1070) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy12.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:180) ... 13 more |
Another mismatched hadoop IPC server/client versions!
Last time I saw this was Install Spark/Shark on CDH 4.
I find all jars named like *hadoop* under shark/lib_managed/, found this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
./lib_managed/jars/org.apache.hadoop ./lib_managed/jars/org.apache.hadoop/hadoop-auth ./lib_managed/jars/org.apache.hadoop/hadoop-auth/hadoop-auth-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/cloudera-jets3t ./lib_managed/jars/org.apache.hadoop/cloudera-jets3t/cloudera-jets3t-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-nodemanager ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-nodemanager/hadoop-yarn-server-nodemanager-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-shuffle ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-shuffle/hadoop-mapreduce-client-shuffle-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-common ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-common/hadoop-mapreduce-client-common-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-jobclient ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-jobclient/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-core ./lib_managed/jars/org.apache.hadoop/hadoop-core/hadoop-core-1.0.4.jar ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-api ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-client ./lib_managed/jars/org.apache.hadoop/hadoop-client/hadoop-client-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-common ./lib_managed/jars/org.apache.hadoop/hadoop-common/hadoop-common-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-app ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-app/hadoop-mapreduce-client-app-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-client ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-client/hadoop-yarn-client-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-hdfs ./lib_managed/jars/org.apache.hadoop/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-common ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-server-common/hadoop-yarn-server-common-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-annotations ./lib_managed/jars/org.apache.hadoop/hadoop-annotations/hadoop-annotations-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-test ./lib_managed/jars/org.apache.hadoop/hadoop-test/hadoop-test-0.20.2.jar ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-core ./lib_managed/jars/org.apache.hadoop/hadoop-mapreduce-client-core/hadoop-mapreduce-client-core-2.0.0-cdh4.6.0.jar ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-common ./lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.0.0-cdh4.6.0.jar |
I don’t understand why there’s a ‘hadoop-core-1.0.4.jar‘.
I tried to delete this stupid file and copy ‘/opt/cloudera/parcels/CDH/lib/hadoop/client-0.20/hadoop-core-2.0.0-mr1-cdh4.6.0.jar‘ to ./lib_managed/jars/org.apache.hadoop/hadoop-core.
Second test working as standalone mode, it works!
Bring all configurations back, works again!
DO NOT FORGET TO SYNC LATEST SHARK BUILD TO ALL SERVERS!
Why?
Get back to compile issue.
I’ve set ‘SHARK_HADOOP_VERSION=2.0.0-cdh4.6.0’, why a hadoop-core-1.0.4.jar is downloaded and put into lib_managed?
I tried grep ‘1\.0\.4’ in the shark folder, many lines mostly under target/; sbt clean and package, still.
Changed project/ScalaBuild and bin/dev/run-tests-from-scratch, nothing changed.
Why?
http://spark.incubator.apache.org/docs/latest/hadoop-third-party-distributions.html
1 |
SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.6.0 ./sbt/sbt package |
Download
I’ve added a snapshot build(commit fe75a886) for CDH 4.6.0 working for me here: http://cloudera.rst.im/shark/.