SphinxQL FACET is Awesome

FACET clause. This Sphinx specific extension enables faceted search with subtree optimization. It is capable of returning multiple result sets with a single SQL statement, without the need for complicated multi-queries. FACET clauses should be written at the very end of SELECT statements with spaces between them.

FACET {expr_list} [BY {expr_list}] [ORDER BY {expr | FACET()} {ASC | DESC}] [LIMIT [offset,] count]
SELECT * FROM test FACET brand_id FACET categories;
SELECT * FROM test FACET brand_name BY brand_id ORDER BY brand_name ASC FACET property;

The FACET Feature

SphinxQL FACET is Awesome by @sskaje: https://sskaje.me/2015/06/sphinxql-facet-awesome/

OS X 下采集程序调用:sample

之前在看某个软件,一直没找到注册算法的位置。然后就发现了这个命令行工具:https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/sample.1.html

sample — Profile a process during a time interval

sample is a command-line tool for gathering data about the running behavior of a process. It suspends
the process at specified intervals (by default, every 1 millisecond), records the call stacks of all
threads in the process at that time, then resumes the process. The analysis done by sample is called
sampling” because it only checks the state of the program at the sampling points. The analysis may
miss execution of some functions that are not executing during one of the samples, but sample still
provides useful data about commonly executing functions.

Profile的工具,作为逆向调试来说,再好用不过。

OS X 下采集程序调用:sample by @sskaje: https://sskaje.me/2015/04/os-%e4%b8%8b%e9%87%87%e9%9b%86%e7%a8%8b%e5%ba%8f%e8%b0%83%e7%94%a8%ef%bc%9asample/

Nginx 反向代理 过滤Cookie

一个项目需求需要过滤HTTP头中的特定cookie项目。
代理使用nginx搭建,编译参数如下:

其中用到了两个插件:

https://github.com/simpl/ngx_devel_kit
https://github.com/openresty/lua-nginx-module

参考示例:

参考过滤脚本:

Nginx 反向代理 过滤Cookie by @sskaje: https://sskaje.me/2015/03/nginx-proxy-filter-cookie/

Incoming search terms:

PHP解密AES

分析某客户端协议,发现如下代码:

这个和之前分析航旅纵横客户端时的代码类似,都是128位ECB。
当时只是用openssl看了看请求:

于是参考mcrypt的文档,写了简单的代码实现上边Java代码里的AES 128 ECB加密解密

PHP解密AES by @sskaje: https://sskaje.me/2015/02/php-aes-encrypt-decrypt/

Infobright 企业版数据导入和数据擦写实验

拿到一个IEE的试用版证书,试了下作为日志存储和计算的方案。统计数据查询就不用测了,ICE试试就能感受出来,比hive反正快了不少。
这里主要还是想测试 INSERT / UPDATE / DELETE

实验环境的日志系统使用 rsyslog -> flume-ng -> IEE/HDFS.
使用Flume-ng自带的HDFS Sink写HDFS的方案一直很稳定,目录按天分,写脚本预先创建目录、加Hive分区,使用hive进行分析。
但是由于可能对当天数据有统计需求hdfs.rollInterval设的比较小,目前是2分钟,每天都会有大量小文件,hive处理速度十分慢。

Flume-ng 找人写了个简单的入mysql的插件,单加了一个队列,把日志文件切分后按列送进mysql,插件要求数据库insert使用prepare批量处理insert。
Continue reading “Infobright 企业版数据导入和数据擦写实验” »

Infobright 企业版数据导入和数据擦写实验 by @sskaje: https://sskaje.me/2015/01/infobright-iee-data-testing/

Incoming search terms: