Max's Blog » 2008

Nov 05 2008

i have a dream

Category: 乱up当秘笈 — ssmax @ 13:45:26

奥巴马获胜了，马丁路德金的dream也实现得差不多了吧。

今天奥巴马一获胜，我们的流量立刻上升了一倍多，看多还是很多人关注世界大事的。

Comments (0)

Nov 04 2008

nginx 和 syslog-ng 配合

Category: 技术 — ssmax @ 11:11:10

nginx本来不支持syslog的，虽然有个补丁，试了一下，配置性太差，error log 和access log 不能分开配置。。。

然后就想了一个方案，用syslog-ng ，也不用改nginx的源代码了，syslog-ng可以支持pipe输入

The sources, destinations, and filters available in syslog-ng are listed below. For details, see The syslog-ng Administrator Guide .

Name	Description
internal()	Messages generated internally in syslog-ng.
unix-stream()	Opens the specified unix socket in `SOCK_STREAM` mode and listens for incoming messages.
unix-dgram()	Opens the specified unix socket in `SOCK_DGRAM` mode and listens for incoming messages.
file()	Opens the specified file and reads messages.
pipe(), fifo	Opens the specified named pipe and reads messages.
tcp()	Listens on the specified TCP port for incoming messages.
udp()	Listens on the specified UDP port for incoming messages.
tcp6()	Listens on the specified TCP port for incoming messages over IPv6.
udp6()	Listens on the specified UDP port for incoming messages over IPv6.
sun-stream(), sun-streams()	Opens the specified `STREAMS` device on Solaris systems and reads incoming messages.

Table 1.1. Source drivers available in syslog-ng

开始，先建立一个fifo文件（就是pipe啦）。。。

mkfifo logs/access.pipe

然后把nginx设置为

access_log logs/access.pipe main;

syslog-ng配置加上

source s_all {
        # message generated by Syslog-NG
        internal();
        # standard Linux log source (this is the default place for the syslog()
        # function to send logs to)
        unix-stream(“/dev/log”);
        # messages from the kernel
        file(“/proc/kmsg” log_prefix(“kernel: “));
        # use the following line if you want to receive remote UDP logging messages
        # (this is equivalent to the “-r” syslogd flag)
        # udp();
        pipe(“/home/nginx/logs/access.pipe”)
};

这样子就多了一个输入了，syslog-ng用file模式打开管道也可以的，不知道两者性能上面有没有差别，好像是没有什么区别的，另外一种fifo模式，就是pipe了，呵呵。

需要注意一下log_fifo_size的大小，这个是指的缓存多少行，而不是缓存多大。

如果不用pipe，用直接用access.log文件也是可以的，不过就是触发的时间太长，而且重启syslog-ng的时候可能会造成log重复。

设置真实文件的时候就需要注意多一些选项了，最重要就是follow_freq()，多久去tail一次，因为文件是没有poll的：

一般真实文件的例子

source s_tail { file("/var/log/apache/access.log" 
follow_freq(1) flags(no-parse)); };

9.1.1. Options common for every source

Some parameters affecting message parsing are common for all sources:

Name	Type	Default	Description
flags()	set of [no-parse,kernel]	empty set	Specifies log parsing flags. `no-parse` completely disables syslog message parsing and processes the complete line as the message part of a syslog message. Other information (timestamp, host, etc.) is added automatically. This flag is useful for parsing files not complying to the syslog format. `kernel` makes the source default to the `LOG_KERN \| LOG_CRIT` priority if not specified otherwise.
follow_freq()	number	-1	Indicates that the source should be checked periodically instead of being polled. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng will not attempt to use `poll()` on the file, but checks whether the file changed every time the `follow_freq()` interval (in seconds) has elapsed.
keep_timestamp()	yes or no	yes	Specifies whether syslog-ng should accept the timestamp received from the peer. If disabled, the time of reception will be used instead.
log_fetch_limit()	number	The value specified by the global `log_fetch_limit()` option, which defaults to `10`.	The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if `log_fetch_limit()` is too high.
log_iw_size()	number	100	The size of the initial window, this value is used during flow control.
log_msg_size()	number	The value specified by the global `log_msg_size()` option, which defaults to `8192`.	Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.
log_prefix()	string		A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding `kernel:` to the kernel messages on Linux.
optional()	yes or no		Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the `pipe()`, `unix-dgram`, and `unix-stream` drivers.
pad_size()	number	0	Specifies input padding. Some operating systems (such as HP-UX) pad all 0 messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). Syslog-ng will pad reads from the associated device to the number of bytes set in `pad_size()`. Mostly used on HP-UX where `/dev/log` is a named pipe and every write is padded to 2048 bytes.
time_zone()	timezone in the form +/-HH:MM		The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.