D语言，WEB 日志正则分解,管道处理进入数据库
$ cat match.d import std.regex;import std.stdio;import std.string;import std.array;void main(){auto r = regex(`^(\S+) (\S+) (\S+) \[(.+)\] "([^"]+)" ([0-9]{3}) ([0-9]+) "([^"]+)" "([^"]+)" "([^"]+)"`);foreach(line; stdin.byLine){ foreach(m; match(line, r)){auto c = m.captures;c.popFront();auto value = join(c, "\",\"");auto sql = format("insert into log(remote_addr,unknow,remote_user,time_local,request,status,body_bytes_sent,http_referer,http_user_agent,http_x_forwarded_for) value(\"%s\");", value );writeln(sql);}}}

?编译

$ dmd match.d 编译优化（减肥）$ strip match

测试

$ cat access.log | ./match输出类似insert into log(remote_addr,unknow,remote_user,time_local,request,status,body_bytes_sent,http_referer,http_user_agent,http_x_forwarded_for) value("192.168.2.76","-","-","19/Mar/2013:15:24:20 +0800","GET /favicon.ico HTTP/1.1","404","193","-","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0","-");

??

用法

?

$ cat access.log | match | mysql -hlocalhost -ulog -p123456 logging

?

高级用法， 实时处理日志

首先创建一个管道寻该日志文件写入管道中cat ?管道名?| match | mysql -hlocalhost -ulog -p123456 logging ?这样就可以实现实时日志插入。

上面程序稍加修改即可实现Hbase,?Hypertable?插入

?

?

?

?

?

?

'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

将 log_format 格式直接生成SQL