Archives for February 2014

CDH 4 HA Related Problems

By @sskaje
Link: https://sskaje.me/2014/02/cdh-4-ha-related-problems/

Problem

I meet ‘Failed to initialize High Availability state in ZooKeeper. This might be because a ZNode for this nameservice is already created. Either remove the ZNode, or to reuse the ZNode skip this step and simply start the NameNodes and Failover Controllers. To retry, use the “Initialize High Availability state in ZooKeeper” command available as a Failover Controller action.‘ When I was trying to enable automatic failover for HDFS in Cloudera Manager after HA being enabled.

Error logs:

Failed to initialize High Availability state in ZooKeeper from failovercontroller (hadoop5): 14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.39/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/lib/native
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.18.1.el6.x86_64
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-hdfs
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/519-failover-controller-initialize-znode
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop5.xxx.com:2181,hadoop4.xxx.com:2181,hadoop3.xxx.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@3219762f
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5.xxx.com/172.16.3.20:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Socket connection established to hadoop5.xxx.com/172.16.3.20:2181, initiating session
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5.xxx.com/172.16.3.20:2181, sessionid = 0x2447689aed10113, negotiated timeout = 5000
14/02/28 14:51:24 INFO ha.ActiveStandbyElector: Session connected.
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Session: 0x2447689aed10113 closed
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: EventThread shut down

Failed to initialize High Availability state in ZooKeeper from failovercontroller (hadoop5): 14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.39/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/lib/native

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.18.1.el6.x86_64

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-hdfs

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/519-failover-controller-initialize-znode

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop5.xxx.com:2181,hadoop4.xxx.com:2181,hadoop3.xxx.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@3219762f

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5.xxx.com/172.16.3.20:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Socket connection established to hadoop5.xxx.com/172.16.3.20:2181, initiating session

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5.xxx.com/172.16.3.20:2181, sessionid = 0x2447689aed10113, negotiated timeout = 5000

14/02/28 14:51:24 INFO ha.ActiveStandbyElector: Session connected.

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Session: 0x2447689aed10113 closed

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: EventThread shut down

Continue reading “CDH 4 HA Related Problems” »

CDH 4 HA Related Problems by @sskaje: https://sskaje.me/2014/02/cdh-4-ha-related-problems/

Create Windows 7/8 Bootable USB Drive on OS X

By @sskaje
Link: https://sskaje.me/2014/02/create-windows-7-8-bootable-usb-drive-on-osx/

This note works for Windows (e.g. Windows 7, Windows 8, …), Linux … ISO image only, and all commands in Terminal.

If you’re interested in using Disk Utility, try this: Create OS X Mavericks Bootable USB Drive from AppStore Release, but DON’T forget to unmount partitions.

To launch a Terminal, simply open Spotlight and type Terminal, in most cases, you can see Terminal listed after ‘Ter‘.

For Windows 7/8 or later, Boot Camp Assistant is the BEST tool.
In ‘Select Tasks‘, CHECK ‘Create a Windows 7 or later version disk’ and UNCHECK all others.
In ‘Create Bootable USB Drive for Windows Installation‘, choose the right usb destination disk.

Continue reading “Create Windows 7/8 Bootable USB Drive on OS X” »

Create Windows 7/8 Bootable USB Drive on OS X by @sskaje: https://sskaje.me/2014/02/create-windows-7-8-bootable-usb-drive-on-osx/

从GeoIP免费库里生成IP到省市编号的组合

By @sskaje
Link: https://sskaje.me/2014/02/geoip-china-province-city-code-database/

需求

生成一份从大陆地区的IP到省市编号的映射列表，用于数据分析。

准备工作

统计局提供的标准行政区划代码 http://www.stats.gov.cn/tjsj/tjbz/xzqhdm/，身份证前6位的编号。
从 http://dev.maxmind.com/geoip/geoip2/geolite2/ 下载CSV格式的城市数据库
新浪首页提供一个IP查询接口，可以输出TSV，JSONP，JSON的IP信息

Continue reading “从GeoIP免费库里生成IP到省市编号的组合” »

从GeoIP免费库里生成IP到省市编号的组合 by @sskaje: https://sskaje.me/2014/02/geoip-china-province-city-code-database/

Incoming search terms:

geoip 苏州

Prestodb Command Line Client Output Formats

By @sskaje
Link: https://sskaje.me/2014/02/prestodb-command-line-client-output-formats/

Output format is not listed in cli client help.

In presto-cli/src/main/java/com/facebook/presto/cli/ClientOptions.java:

    public enum OutputFormat
    {
        ALIGNED,
        VERTICAL,
        CSV,
        TSV,
        CSV_HEADER,
        TSV_HEADER
    }

public enum OutputFormat

{

ALIGNED,

VERTICAL,

CSV,

TSV,

CSV_HEADER,

TSV_HEADER

}

Output like:
Continue reading “Prestodb Command Line Client Output Formats” »

Prestodb Command Line Client Output Formats by @sskaje: https://sskaje.me/2014/02/prestodb-command-line-client-output-formats/

Incoming search terms:

presto output format

MySQL/Hive/Presto/Impala Transposition

By @sskaje
Link: https://sskaje.me/2014/02/mysql-hive-presto-impala-transposition/

Rows to Columns

Rows to Comma Separated String

FROM:
+-----+-----+
| uid | tag |
+-----+-----+
|   1 |   1 |
|   1 |   2 |
|   1 |   3 |
|   2 |   1 |
|   2 |   4 |
|   3 |   5 |
+-----+-----+
TO:
+-----+----------+
| uid | tag_list |
+-----+----------+
|   1 | 1,2,3    |
|   2 | 1,4      |
|   3 | 5        |
+-----+----------+

FROM:

+-----+-----+

| uid | tag |

+-----+-----+

| 1 | 1 |

| 1 | 2 |

| 1 | 3 |

| 2 | 1 |

| 2 | 4 |

| 3 | 5 |

+-----+-----+

TO:

+-----+----------+

| uid | tag_list |

+-----+----------+

| 1 | 1,2,3 |

| 2 | 1,4 |

| 3 | 5 |

+-----+----------+

MySQL

Use GROUP_CONCAT(). This function also works in Infobright.
Continue reading “MySQL/Hive/Presto/Impala Transposition” »

MySQL/Hive/Presto/Impala Transposition by @sskaje: https://sskaje.me/2014/02/mysql-hive-presto-impala-transposition/