omhttpfs is contrib-ed to rsyslog official repo as an alternative log-to-hdfs solution.
Omhdfs is not provided as an RPM from rsyslog’s offical yum repo, so I’m trying to build on a machine with hadoop installed.
My server is CentOS 6.5, with CDH 5.1 installed.
Download source from: http://www.rsyslog.com/files/download/rsyslog/rsyslog-8.2.2.tar.gz
Install dependencies:
|
1 |
yum install libestr libestr-devel liblogging liblogging-devel json-c-devel json-c libuuid-devel uuid libgcrypt-devel libgcrypt zlib-devel byacc byaccj |
Then I tried to:
|
1 |
./configure --enable-omhdfs --enable-zlib |
Failed on make.
Error message:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
omhdfs.c:80: error: expected specifier-qualifier-list before 'hdfsFS' omhdfs.c:121: error: expected ')' before 'fs' omhdfs.c:140: error: expected ')' before 'fs' omhdfs.c: In function 'fileObjConstruct': omhdfs.c:167: error: 'file_t' has no member named 'hdfsHost' omhdfs.c:168: error: 'file_t' has no member named 'fh' omhdfs.c:169: error: 'file_t' has no member named 'nUsers' omhdfs.c: In function 'fileObjAddUser': omhdfs.c:180: error: 'file_t' has no member named 'nUsers' omhdfs.c:181: error: 'file_t' has no member named 'nUsers' omhdfs.c:182: error: 'file_t' has no member named 'mut' omhdfs.c:183: error: 'file_t' has no member named 'nUsers' omhdfs.c: In function 'fileObjDestruct': omhdfs.c:190: error: 'file_t' has no member named 'nUsers' omhdfs.c:191: error: 'file_t' has no member named 'mut' omhdfs.c:194: error: 'file_t' has no member named 'hdfsHost' omhdfs.c:195: error: 'file_t' has no member named 'fh' omhdfs.c: In function 'filePrepare': omhdfs.c:210: warning: implicit declaration of function 'HDFSFileExists' omhdfs.c:210: error: 'file_t' has no member named 'fs' omhdfs.c:222: error: 'file_t' has no member named 'fs' omhdfs.c:223: warning: implicit declaration of function 'HDFSmkdir' omhdfs.c:223: error: 'file_t' has no member named 'fs' ... |
The hdfs.h is not found.
The default CFLAGS in plugins/omhdfs/Makefile is ‘-g -O2‘, so I changed CFLAGS and LDFLAGS as below:
|
1 |
CFLAGS="-g -O2 -I/opt/cloudera/parcels/CDH/include/" LDFLAGS="-L/opt/cloudera/parcels/CDH/lib/hadoop/lib/native/ -fPIC" ./configure --enable-omhdfs --enable-zlib |
This time error message like:
|
1 2 3 |
/usr/bin/ld: /opt/cloudera/parcels/CDH/lib/hadoop/lib/native//libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC /opt/cloudera/parcels/CDH/lib/hadoop/lib/native//libhdfs.a: could not read symbols: Bad value collect2: ld returned 1 exit status |
I tried to add -fPIC to either CFLAGS or LDFLAGS or both two, same error.
The official module documentation page, http://www.rsyslog.com/doc/omhdfs.html, says java related environment variables should be set, so I set as following:
|
1 2 |
export JAVA_INCLUDES="-I/usr/java/jdk1.7.0_55-cloudera/include -I/usr/java/jdk1.7.0_55-cloudera/include/linux" export JAVA_LIBS="-L/usr/java/jdk1.7.0_55-cloudera/jre/lib/amd64 -L/usr/java/jdk1.7.0_55-cloudera/jre/lib/amd64/server -ljava -ljvm -lverify" |
Same configure as above, same error too.
I choose /opt/cloudera/parcels/CDH/lib/hadoop/lib/native/ just because I felt .h should be come along with .a, so I locate libhdfs and found the /opt/cloudera/parcels/CDH/lib/hadoop/lib/native/libhdfs.a.
|
1 2 3 4 5 6 |
[root@hadoop4 rsyslog-8.2.2]# locate libhdfs /opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hadoop/lib/native/libhdfs.a /opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/impala/lib/libhdfs.so /opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/impala/lib/libhdfs.so.0.0.0 /opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib64/libhdfs.so /opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib64/libhdfs.so.0.0.0 |
Then I tried to test if libhdfs.so can work with this.
|
1 |
CFLAGS="-g -O2 -I/opt/cloudera/parcels/CDH/include/" LDFLAGS="-L/opt/cloudera/parcels/CDH/lib64/" ./configure --enable-omhdfs --enable-zlib |
then make.
All done.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
... Making all in plugins/omhdfs make[2]: Entering directory `/root/rsyslog-8.2.2/plugins/omhdfs' CC omhdfs_la-omhdfs.lo omhdfs.c: In function 'fileWrite': omhdfs.c:315: warning: format '%u' expects type 'unsigned int', but argument 2 has type 'size_t' omhdfs.c: In function 'addData': omhdfs.c:375: warning: passing argument 3 of 'fileWrite' from incompatible pointer type omhdfs.c:295: note: expected 'size_t *' but argument is of type 'unsigned int *' omhdfs.c:377: warning: passing argument 3 of 'fileWrite' from incompatible pointer type omhdfs.c:295: note: expected 'size_t *' but argument is of type 'unsigned int *' omhdfs.c: In function 'endTransaction': omhdfs.c:451: warning: passing argument 3 of 'fileWrite' from incompatible pointer type omhdfs.c:295: note: expected 'size_t *' but argument is of type 'unsigned int *' omhdfs.c: In function 'doAction': omhdfs.c:299: warning: dereferencing pointer 'len.43' does break strict-aliasing rules omhdfs.c:315: warning: dereferencing pointer 'len.43' does break strict-aliasing rules omhdfs.c:316: warning: dereferencing pointer 'len.43' does break strict-aliasing rules omhdfs.c:317: warning: dereferencing pointer 'len.43' does break strict-aliasing rules omhdfs.c:318: warning: dereferencing pointer 'len.43' does break strict-aliasing rules omhdfs.c:324: warning: dereferencing pointer 'len.43' does break strict-aliasing rules omhdfs.c:377: note: initialized from here CCLD omhdfs.la make[2]: Leaving directory `/root/rsyslog-8.2.2/plugins/omhdfs' Making all in tests make[2]: Entering directory `/root/rsyslog-8.2.2/tests' make[2]: Nothing to be done for `all'. make[2]: Leaving directory `/root/rsyslog-8.2.2/tests' make[1]: Leaving directory `/root/rsyslog-8.2.2' |
Build omhdfs for Rsyslog by @sskaje: https://sskaje.me/2014/08/build-omhdfs-rsyslog/
Link to this post!