Search Results for delete hdfs conf

Newer Documentation for HttpFS(Hadoop HDFS over HTTP)

I was trying to make rsyslog v8 communicating with hadoop hdfs directly via omhdfs, but failed as it’s said officially that omhdfs is not working with rsyslog v8 by now. UPDATE: OmHTTPFS, Another Rsyslog HDFS Output Plugin, https://github.com/rsyslog/rsyslog/tree/master/contrib/omhttpfs I was recommended to use HttpFS when setting up Hue in CDH. HttpFS is a http gateway … Continue reading “Newer Documentation for HttpFS(Hadoop HDFS over HTTP)”

Fix Hadoop Conf Alternatives for CDH5

I’m using CDH5, upgraded failed from CDH4 and then reinstalled directly. /etc/hadoop/conf is linked to /etc/hadoop/conf/conf.cloudera.mapreduce1/. Deploy Client Configuration does not make it right. The way fix it is manually set a new path and remove the old one, like:

ssh hadoop5 alternatives --set hadoop-conf /etc/hadoop/conf.cloudera.yarn;
ssh hadoop5 alternatives --remove hadoop-conf /etc/hadoop/conf.cloudera.mapreduce1/

1 2	ssh hadoop5 alternatives --set hadoop-conf /etc/hadoop/conf.cloudera.yarn; ssh hadoop5 alternatives --remove hadoop-conf /etc/hadoop/conf.cloudera.mapreduce1/

But the next time you try Deploy Client Configuration would corrupt it again.

[root@hadoop4 daily]# alternatives --config hadoop-conf

There are 5 programs which provide 'hadoop-conf'.

  Selection    Command
-----------------------------------------------
*+ 1           /etc/hadoop/conf.cloudera.mapreduce1
   2           /etc/hadoop/conf.cloudera.hdfs1
   3           /etc/hadoop/conf.cloudera.yarn
   4           /opt/cloudera/parcels/CDH-5.0.0-0.cdh5b1.p0.57/etc/hadoop/conf.empty
   5           /etc/hadoop/conf.cloudera.hdfs

[root@hadoop4 daily]# alternatives --config hadoop-conf

There are 5 programs which provide 'hadoop-conf'.

Selection Command

-----------------------------------------------

*+ 1 /etc/hadoop/conf.cloudera.mapreduce1

2 /etc/hadoop/conf.cloudera.hdfs1

3 /etc/hadoop/conf.cloudera.yarn

4 /opt/cloudera/parcels/CDH-5.0.0-0.cdh5b1.p0.57/etc/hadoop/conf.empty

5 /etc/hadoop/conf.cloudera.hdfs

… Continue reading “Fix Hadoop Conf Alternatives for CDH5”

Infobright 企业版数据导入和数据擦写实验

拿到一个IEE的试用版证书，试了下作为日志存储和计算的方案。统计数据查询就不用测了，ICE试试就能感受出来，比hive反正快了不少。这里主要还是想测试 INSERT / UPDATE / DELETE。实验环境的日志系统使用 rsyslog -> flume-ng -> IEE/HDFS. 使用Flume-ng自带的HDFS Sink写HDFS的方案一直很稳定，目录按天分，写脚本预先创建目录、加Hive分区，使用hive进行分析。但是由于可能对当天数据有统计需求hdfs.rollInterval设的比较小，目前是2分钟，每天都会有大量小文件，hive处理速度十分慢。 Flume-ng 找人写了个简单的入mysql的插件，单加了一个队列，把日志文件切分后按列送进mysql，插件要求数据库insert使用prepare批量处理insert。 Incoming search terms:infobright 压缩率Link to this post!

YARN NodeManager Failed to Start

I upgraded my CDH, one of my NodeNamager cannot be brought up. NullPointer Exceptions were found in error log:

2014-11-03 16:51:10,042 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED; cause:
java.lang.NullPointerException
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:252)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:235)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:250)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:445)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:492)
2014-11-03 16:51:10,053 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Applications still running : [application_1413440084214_10329]
2014-11-03 16:51:10,053 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: org.apache.hadoop.yarn.server.nodemanager.containermanager.logagg
2014-11-03 16:51:10,054 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: java.lang.NullPointerException
java.lang.NullPointerException
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:252)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:235)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:250)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:445)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:492)
2014-11-03 16:51:10,055 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
2014-11-03 16:51:10,055 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
2014-11-03 16:51:10,055 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2014-11-03 16:51:10,056 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
java.lang.NullPointerException
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:252)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:235)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:250)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:445)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:492)
2014-11-03 16:51:10,058 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at a1.hadoop.xxx.com/172.16.3.131
************************************************************/

2014-11-03 16:51:10,042 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED; cause:

java.lang.NullPointerException

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:252)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:235)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:250)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:445)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:492)

2014-11-03 16:51:10,053 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Applications still running : [application_1413440084214_10329]

2014-11-03 16:51:10,053 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: org.apache.hadoop.yarn.server.nodemanager.containermanager.logagg

2014-11-03 16:51:10,054 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: java.lang.NullPointerException