Still in beta, merged into rsyslog’s official repo.
Known issues or features to be supported:
Error codes returned;
Connection detect;
http keep-alive;
…
Code here: https://github.com/rsyslog/rsyslog/tree/master/contrib/omhttpfs
Update:
I wrote a test script, but most requests were lost.
|
1 2 3 4 5 6 7 8 |
<?php date_default_timezone_set('Asia/Shanghai'); openlog("logger", LOG_NDELAY | LOG_CONS | LOG_PID, LOG_LOCAL4); $c = 0; do { syslog(LOG_INFO, "test" . date("Y-m-d H:i:s")); } while(++$c < 100000); closelog(); |
Here are queue parameters supported by action() by default: http://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html
And updated config:
|
1 2 3 |
local4.* action(type="omhttpfs" host="127.0.0.1" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on" queue.filename="httpfs_q1" queue.size="1000000" queue.type="LinkedList" ) |
OmHTTPFS
Author: sskaje (sskaje@gmail.com, http://sskaje.me/)
OmHTTPFS is an Rsyslog plugin writting data to HDS via Hadoop HDFS over HTTP.
Hadoop HDFS over HTTP
Official site: Hadoop HDFS over HTTP
HTTPFS is not well documented. I tried to read its source and write an intro with examples, until I found Administering the file system by using HttpFS REST APIs by IBM.
OmHDFS for Rsyslog
Rsyslog provides a plugin named omHDFS which requires lots of work compiling and configuring, and it’s not that usable.
Here is what I tried before writting this omhttpfs: Build omhdfs for Rsyslog.
Rsyslog config
Legacy config NOT supported.
Example:
|
1 2 3 4 5 |
module(load="omhttpfs") template(name="hdfs_tmp_file" type="string" string="/tmp/%$YEAR%/test.log") template(name="hdfs_tmp_filecontent" type="string" string="%$YEAR%-%$MONTH%-%$DAY% %MSG% ==\n") local4.* action(type="omhttpfs" host="10.1.1.161" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on") local5.* action(type="omhttpfs" host="10.1.1.161" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on" template="hdfs_tmp_filecontent") |
Tested with CDH 5.2.0 + Rsyslog 8.6.0 on CentOS 7
# EOF