Tag: rsyslog omhdfs

I was trying to make rsyslog v8 communicating with hadoop hdfs directly via omhdfs, but failed as it’s said officially that omhdfs is not working with rsyslog v8 by now.

UPDATE: OmHTTPFS, Another Rsyslog HDFS Output Plugin, https://github.com/rsyslog/rsyslog/tree/master/contrib/omhttpfs

I was recommended to use HttpFS when setting up Hue in CDH. HttpFS is a http gateway for HDFS, originally developed by cloudera as hoop and then contributed to Apache foundation as a component of HDFS.

It’s named ‘Hadoop HDFS over HTTP’， HttpFS is a server that provides a REST HTTP gateway supporting all HDFS File System operations (read and write). And it is inteoperable with the webhdfs REST HTTP API. The latest doc can be found here.
Examples with cURL are given on that doc page:

$ curl http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt returns the contents of the HDFS /user/foo/README.txt file.
$ curl http://httpfs-host:14000/webhdfs/v1/user/foo?op=list returns the contents of the HDFS /user/foo directory in JSON format.
$ curl -X POST http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs creates the HDFS /user/foo.bar directory.

Try those, only things you can get is a HTTP 401.
You have to add, at least, a ‘user.name=xxx’ to identify yourself, then:

YOU ARE FOOLED!

The latest source of httpfs can be found in apache foundation’s svn: http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/.
Continue reading “Newer Documentation for HttpFS(Hadoop HDFS over HTTP)” »

Incoming search terms:

• httpfs windows client
• hdfsksSk
• No enum constant org apache hadoop fs http client HttpFSFileSystem Operation LIST