sysfence - system resources guard for Linux

sysfence <configuration file> [<configuration file> ...]

Sysfence is a resource monitoring tool designed for Linux machines. While running as daemon it checks resource levels and makes desired action if some values exceed safety limits.

Sysfence can be used for notifying system administrators when something goes wrong, stopping services when system performance is dropping too low and starting them when it's going up again, periodically restarting memory-leaking processes, dumping system statistics in critical situations.

Sysfence can monitor following resource levels: load average, used and free memory amount, used and free swap space.

Sysfence reads it's configuration from file(s) specified in argument list. Config files may contain one or more rules describing conditions and actions to be performed.

Rule has syntax like this:

if { resource1 > limit1 or { resource2 < limit2 and resource3 < limit3 } } run once 'command-to-be-run'

The block enclosed within {} brackets describes condition. When it's result is TRUE, following command is invoked.

The once keyword is optional. If present, the command is executed only once after condition becomes TRUE. Next execution will take place only if condition becomes FALSE and then TRUE again. Without once keyword, command is invoked periodically, after every resource check that gives TRUE, no matter what was the condition result before.

Command specified right after run keyword is passed to /bin/sh, so it may contain more than one instruction or even whole script. But be careful - rule checking is suspended unless command execution has been completed! (Other rules are unaffected.)

As resources, following ones can be given:

la1

- load average during last minute.

la5

- load average during last 5 minutes.

la15

- load average during last 15 minutes.

memfree

- lower limit for free memory amount.

memused

- upper limit for memory used by processes.

swapfree

- lower limit for free swap space.

swapused

- upper limit for swap space in use.

Do you have problems with MySQL server choking and freezing whole system? I do. To find queries that cause problems, you may use:

if { la1 >= 8.00 } run once 'echo "SHOW FULL PROCESSLIST" | mysql | mail my@email.com'

Of course, that wouldn't prevent your system from being blocked, but following rule could. MySQL will be restarted if LA for last minute is over 10.0 or LA for last five minutes is over 6.0.

if { la1 >= 10.00 or la5 >= 6.0 } run '/etc/rc.d/init.d/mysql stop; sleep 120; /etc/rc.d/init.d/mysql start'

We may also restart some services that probably have memory leaks and use lots of swap space if not restarted periodically. Let's assume that 256MB of used swap is enough to give our Zope server a break.

if { swapused >= 256M } run '/etc/rc.d/init.d/zope restart'

We may also alert admins... Notice that you don't have to be r00t:

if { la15 > 4.0 and { swapfree < 64M or memfree < 128M } } run 'echo "i wish you were here..." | sendsms +48ADMINCELLPHONE'

Using sysfence version 0.7 or later you may give rule a name that will be used in logs:

rule "high load" { la1 > 3.0 and la15 > 2.0 } log

rule keyword has the same meaning as if. There are also synonymes for other keywords. Detailed list is included within sysfence package.

You can find an example config file in /usr/share/doc/sysfence/example.conf.

Sysfence was written by Michal Saban (emes at pld-linux org) and Mirek Kopertowski (m.kopertowski at post pl)

This manual page was created by Lukasz Jachowicz <honey@debian.org>, for the Debian project (but may be used by others). It is based on the http://sysfence.sf.net/ page.