Still in beta, merged into rsyslog’s official repo.
Known issues or features to be supported:
Error codes returned;
Connection detect;
http keep-alive;
…
Code here: https://github.com/rsyslog/rsyslog/tree/master/contrib/omhttpfs
I was trying to make rsyslog v8 communicating with hadoop hdfs directly via omhdfs, but failed as it’s said officially that omhdfs is not working with rsyslog v8 by now.
UPDATE: OmHTTPFS, Another Rsyslog HDFS Output Plugin, https://github.com/rsyslog/rsyslog/tree/master/contrib/omhttpfs
I was recommended to use HttpFS when setting up Hue in CDH. HttpFS is a http gateway for HDFS, originally developed by cloudera as hoop and then contributed to Apache foundation as a component of HDFS.
It’s named ‘Hadoop HDFS over HTTP’， HttpFS is a server that provides a REST HTTP gateway supporting all HDFS File System operations (read and write). And it is inteoperable with the webhdfs REST HTTP API. The latest doc can be found here.
Examples with cURL are given on that doc page:
$ curl http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt returns the contents of the HDFS /user/foo/README.txt file.
$ curl http://httpfs-host:14000/webhdfs/v1/user/foo?op=list returns the contents of the HDFS /user/foo directory in JSON format.
$ curl -X POST http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs creates the HDFS /user/foo.bar directory.
Try those, only things you can get is a HTTP 401.
You have to add, at least, a ‘user.name=xxx’ to identify yourself, then:
|
1 2 |
curl 'http://172.16.3.20:14000/webhdfs/v1/?op=list&user.name=hdfs' {"RemoteException":{"message":"java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.LIST","exception":"QueryParamException","javaClassName":"com.sun.jersey.api.ParamException$QueryParamException"}} |
YOU ARE FOOLED!
The latest source of httpfs can be found in apache foundation’s svn: http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/.
omhttpfs is contrib-ed to rsyslog official repo as an alternative log-to-hdfs solution.
Omhdfs is not provided as an RPM from rsyslog’s offical yum repo, so I’m trying to build on a machine with hadoop installed.
My server is CentOS 6.5, with CDH 5.1 installed.
Download source from: http://www.rsyslog.com/files/download/rsyslog/rsyslog-8.2.2.tar.gz
Install dependencies:
|
1 |
yum install libestr libestr-devel liblogging liblogging-devel json-c-devel json-c libuuid-devel uuid libgcrypt-devel libgcrypt zlib-devel byacc byaccj |
Then I tried to:
|
1 |
./configure --enable-omhdfs --enable-zlib |
Failed on make.
Error message:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
omhdfs.c:80: error: expected specifier-qualifier-list before 'hdfsFS' omhdfs.c:121: error: expected ')' before 'fs' omhdfs.c:140: error: expected ')' before 'fs' omhdfs.c: In function 'fileObjConstruct': omhdfs.c:167: error: 'file_t' has no member named 'hdfsHost' omhdfs.c:168: error: 'file_t' has no member named 'fh' omhdfs.c:169: error: 'file_t' has no member named 'nUsers' omhdfs.c: In function 'fileObjAddUser': omhdfs.c:180: error: 'file_t' has no member named 'nUsers' omhdfs.c:181: error: 'file_t' has no member named 'nUsers' omhdfs.c:182: error: 'file_t' has no member named 'mut' omhdfs.c:183: error: 'file_t' has no member named 'nUsers' omhdfs.c: In function 'fileObjDestruct': omhdfs.c:190: error: 'file_t' has no member named 'nUsers' omhdfs.c:191: error: 'file_t' has no member named 'mut' omhdfs.c:194: error: 'file_t' has no member named 'hdfsHost' omhdfs.c:195: error: 'file_t' has no member named 'fh' omhdfs.c: In function 'filePrepare': omhdfs.c:210: warning: implicit declaration of function 'HDFSFileExists' omhdfs.c:210: error: 'file_t' has no member named 'fs' omhdfs.c:222: error: 'file_t' has no member named 'fs' omhdfs.c:223: warning: implicit declaration of function 'HDFSmkdir' omhdfs.c:223: error: 'file_t' has no member named 'fs' ... |