By @sskaje
Link: https://sskaje.me/2014/12/omhttpfs-rsyslog-hdfs-output-plugin/

Still in beta, merged into rsyslog’s official repo.

Known issues or features to be supported:
Error codes returned;
Connection detect;
http keep-alive;
…

Code here: https://github.com/rsyslog/rsyslog/tree/master/contrib/omhttpfs

Update:
I wrote a test script, but most requests were lost.

<?php
date_default_timezone_set('Asia/Shanghai');
openlog("logger", LOG_NDELAY | LOG_CONS | LOG_PID, LOG_LOCAL4);
$c = 0;
do {
syslog(LOG_INFO, "test" . date("Y-m-d H:i:s"));
} while(++$c < 100000);
closelog();

<?php

date_default_timezone_set('Asia/Shanghai');

openlog("logger", LOG_NDELAY | LOG_CONS | LOG_PID, LOG_LOCAL4);

$c = 0;

do {

syslog(LOG_INFO, "test" . date("Y-m-d H:i:s"));

} while(++$c < 100000);

closelog();

Here are queue parameters supported by action() by default: http://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html
And updated config:

local4.*    action(type="omhttpfs" host="127.0.0.1" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on"
        queue.filename="httpfs_q1" queue.size="1000000" queue.type="LinkedList"
)

local4.* action(type="omhttpfs" host="127.0.0.1" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on"

queue.filename="httpfs_q1" queue.size="1000000" queue.type="LinkedList"

)

OmHTTPFS

Author: sskaje (sskaje@gmail.com, http://sskaje.me/)

OmHTTPFS is an Rsyslog plugin writting data to HDS via Hadoop HDFS over HTTP.

Hadoop HDFS over HTTP

Official site: Hadoop HDFS over HTTP

HTTPFS is not well documented. I tried to read its source and write an intro with examples, until I found Administering the file system by using HttpFS REST APIs by IBM.

OmHDFS for Rsyslog

Rsyslog provides a plugin named omHDFS which requires lots of work compiling and configuring, and it’s not that usable.
Here is what I tried before writting this omhttpfs: Build omhdfs for Rsyslog.

Rsyslog config

Legacy config NOT supported.

Example:

module(load="omhttpfs")
template(name="hdfs_tmp_file" type="string" string="/tmp/%$YEAR%/test.log")
template(name="hdfs_tmp_filecontent" type="string" string="%$YEAR%-%$MONTH%-%$DAY% %MSG% ==\n")
local4.*    action(type="omhttpfs" host="10.1.1.161" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on")
local5.*    action(type="omhttpfs" host="10.1.1.161" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on" template="hdfs_tmp_filecontent")

module(load="omhttpfs")

template(name="hdfs_tmp_file" type="string" string="/tmp/%$YEAR%/test.log")

template(name="hdfs_tmp_filecontent" type="string" string="%$YEAR%-%$MONTH%-%$DAY% %MSG% ==\n")

local4.* action(type="omhttpfs" host="10.1.1.161" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on")

local5.* action(type="omhttpfs" host="10.1.1.161" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on" template="hdfs_tmp_filecontent")

Tested with CDH 5.2.0 + Rsyslog 8.6.0 on CentOS 7

# EOF

OmHTTPFS, Another Rsyslog HDFS Output Plugin by @sskaje: https://sskaje.me/2014/12/omhttpfs-rsyslog-hdfs-output-plugin/

OmHTTPFS

Hadoop HDFS over HTTP

OmHDFS for Rsyslog

Rsyslog config

Incoming search terms:

Related posts: