Month: February 2014

CDH 4 HA Related Problems

Post author By sskaje
Post date February 28, 2014
No Comments on CDH 4 HA Related Problems

By @sskaje
Link: https://sskaje.me/2014/02/cdh-4-ha-related-problems/

Problem

I meet ‘Failed to initialize High Availability state in ZooKeeper. This might be because a ZNode for this nameservice is already created. Either remove the ZNode, or to reuse the ZNode skip this step and simply start the NameNodes and Failover Controllers. To retry, use the “Initialize High Availability state in ZooKeeper” command available as a Failover Controller action.‘ When I was trying to enable automatic failover for HDFS in Cloudera Manager after HA being enabled.

Error logs:

Failed to initialize High Availability state in ZooKeeper from failovercontroller (hadoop5): 14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.39/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/lib/native
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.18.1.el6.x86_64
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-hdfs
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/519-failover-controller-initialize-znode
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop5.xxx.com:2181,hadoop4.xxx.com:2181,hadoop3.xxx.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@3219762f
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5.xxx.com/172.16.3.20:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Socket connection established to hadoop5.xxx.com/172.16.3.20:2181, initiating session
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5.xxx.com/172.16.3.20:2181, sessionid = 0x2447689aed10113, negotiated timeout = 5000
14/02/28 14:51:24 INFO ha.ActiveStandbyElector: Session connected.
14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Session: 0x2447689aed10113 closed
14/02/28 14:51:24 INFO zookeeper.ClientCnxn: EventThread shut down

Failed to initialize High Availability state in ZooKeeper from failovercontroller (hadoop5): 14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.39/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/lib/native

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.18.1.el6.x86_64

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-hdfs

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/519-failover-controller-initialize-znode

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop5.xxx.com:2181,hadoop4.xxx.com:2181,hadoop3.xxx.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@3219762f

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5.xxx.com/172.16.3.20:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Socket connection established to hadoop5.xxx.com/172.16.3.20:2181, initiating session

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5.xxx.com/172.16.3.20:2181, sessionid = 0x2447689aed10113, negotiated timeout = 5000

14/02/28 14:51:24 INFO ha.ActiveStandbyElector: Session connected.

14/02/28 14:51:24 INFO zookeeper.ZooKeeper: Session: 0x2447689aed10113 closed

14/02/28 14:51:24 INFO zookeeper.ClientCnxn: EventThread shut down

CDH 4 HA Related Problems by @sskaje: https://sskaje.me/2014/02/cdh-4-ha-related-problems/

Tags CDH, cdh 4 hight availability, cdh ha, cdh high availability, CDH4, cdh4 ha, ha

OS X 笔记

Create Windows 7/8 Bootable USB Drive on OS X

Post author By sskaje
Post date February 27, 2014
No Comments on Create Windows 7/8 Bootable USB Drive on OS X

By @sskaje
Link: https://sskaje.me/2014/02/create-windows-7-8-bootable-usb-drive-on-osx/

This note works for Windows (e.g. Windows 7, Windows 8, …), Linux … ISO image only, and all commands in Terminal.

If you’re interested in using Disk Utility, try this: Create OS X Mavericks Bootable USB Drive from AppStore Release, but DON’T forget to unmount partitions.

To launch a Terminal, simply open Spotlight and type Terminal, in most cases, you can see Terminal listed after ‘Ter‘.

For Windows 7/8 or later, Boot Camp Assistant is the BEST tool.
In ‘Select Tasks‘, CHECK ‘Create a Windows 7 or later version disk’ and UNCHECK all others.
In ‘Create Bootable USB Drive for Windows Installation‘, choose the right usb destination disk.

Create Windows 7/8 Bootable USB Drive on OS X by @sskaje: https://sskaje.me/2014/02/create-windows-7-8-bootable-usb-drive-on-osx/

Tags 10.9, apple, bootable, linux, mavericks, OS X, usb, windows

杂七杂八

从GeoIP免费库里生成IP到省市编号的组合

By @sskaje
Link: https://sskaje.me/2014/02/geoip-china-province-city-code-database/

需求

生成一份从大陆地区的IP到省市编号的映射列表，用于数据分析。

准备工作

统计局提供的标准行政区划代码 http://www.stats.gov.cn/tjsj/tjbz/xzqhdm/，身份证前6位的编号。
从 http://dev.maxmind.com/geoip/geoip2/geolite2/ 下载CSV格式的城市数据库
新浪首页提供一个IP查询接口，可以输出TSV，JSONP，JSON的IP信息

从GeoIP免费库里生成IP到省市编号的组合 by @sskaje: https://sskaje.me/2014/02/geoip-china-province-city-code-database/

Incoming search terms:

Tags IP, IP database, IP库, IP归属地, IP查询, 免费IP库, 城市IP, 城市IP库, 查IP

Hadoop相关 PrestoDB 学习研究笔记

Prestodb Command Line Client Output Formats

Post author By sskaje
Post date February 26, 2014
No Comments on Prestodb Command Line Client Output Formats

By @sskaje
Link: https://sskaje.me/2014/02/prestodb-command-line-client-output-formats/

Output format is not listed in cli client help.

In presto-cli/src/main/java/com/facebook/presto/cli/ClientOptions.java:

    public enum OutputFormat
    {
        ALIGNED,
        VERTICAL,
        CSV,
        TSV,
        CSV_HEADER,
        TSV_HEADER
    }

public enum OutputFormat

{

ALIGNED,

VERTICAL,

CSV,

TSV,

CSV_HEADER,

TSV_HEADER

}

Output like:

Prestodb Command Line Client Output Formats by @sskaje: https://sskaje.me/2014/02/prestodb-command-line-client-output-formats/

Incoming search terms:

presto output format

Tags presto, presto cli, presto client, prestodb, prestodb client

Hadoop相关 Hive Impala PrestoDB 学习研究

MySQL/Hive/Presto/Impala Transposition

Post author By sskaje
Post date February 18, 2014
No Comments on MySQL/Hive/Presto/Impala Transposition

By @sskaje
Link: https://sskaje.me/2014/02/mysql-hive-presto-impala-transposition/

Rows to Columns

Rows to Comma Separated String

FROM:
+-----+-----+
| uid | tag |
+-----+-----+
|   1 |   1 |
|   1 |   2 |
|   1 |   3 |
|   2 |   1 |
|   2 |   4 |
|   3 |   5 |
+-----+-----+
TO:
+-----+----------+
| uid | tag_list |
+-----+----------+
|   1 | 1,2,3    |
|   2 | 1,4      |
|   3 | 5        |
+-----+----------+

FROM:

+-----+-----+

| uid | tag |

+-----+-----+

| 1 | 1 |

| 1 | 2 |

| 1 | 3 |

| 2 | 1 |

| 2 | 4 |

| 3 | 5 |

+-----+-----+

TO:

+-----+----------+

| uid | tag_list |

+-----+----------+

| 1 | 1,2,3 |

| 2 | 1,4 |

| 3 | 5 |

+-----+----------+

MySQL

Use GROUP_CONCAT(). This function also works in Infobright.

MySQL/Hive/Presto/Impala Transposition by @sskaje: https://sskaje.me/2014/02/mysql-hive-presto-impala-transposition/