spong-message - send out alerts when there is a problem

spong-message [--debug] [--file filename | --message detailed message text] color host service time message

spong-message [--debug] --color|--status color --host hostname --service service --time time --summary summary message text [--file filename | --message detailed message text]

This program is called by the spong-server to send out alerts. The spong-server only makes a quick determination for send out alerts. It's only criteria is a change in status of a service. The spong-message make a more thorough determination of transmitting the alerts according to the message rules defined in the spong.message configuration file. spong-message also has throttling mechanisms to prevent an excessive number of messages from being send within a short amount of time.

--debug

Enables the printing of detailed debugging lines. Can be used in conjunction with --test to test new messaging rules.

--color|--status color

Specified the status color of the event being reported. color can be green, yellow or red.

--host hostname

The hostname of the server that is being reported on.

--service service

The name of the service that is being reported on.

--time time

The time of the event being reported in epoch format (i.e. time()). This is the summary message field of the event being reported on.

--file filename

The name of a file to read the detailed message text from. If the filename is '-', the text is read from stdin. Detailed message text of the event being reported.

The following parameters can be specified on the command line with the accompanying command line parameters.

color

the status color of the message (red, yellow, or green)

host

the hostname being of the alert

service

the name of the service of the alert

time

the date/time (in time() format) of the problem

message

a summary line of the problem

When spong-message is called, the information passed in the arguments is run through a list of rules which determine who is contacted, when they are contacted and how often. The information is also run through a number of checks to determine if the message should be sent.

A small database in the $SPONGTMP directory is kept so that spong-message can keep track how many pages have been sent, when was the last page sent, etc. These checks help to direct problems to the correct people, and also help to throttle messages when there are wide-spread problems (such as a networking outage).

If you are going to be performing maintenance on a machine, or have standard down time for a machine, you can specify that down time in the spong.hosts file using the down attribute in a %HOSTS variable. If a problem is reported during the time indicated, spong-message will not send any messages.

spong-message also checks for any acknowledgements active for a machine. If there an active acknowledgement found for a machine and service, no messages will be sent.

spong-message uses the %HUMANS entries defined in the contacts attributes of the messaging rules of spong.message to determine who is to be contacted. A list of contacts is generated from all of the message rules that are matched. (See the spong.message and spong.hosts documentation for information on the file formats.)

Notification messages are formatted by message templates in the CW%TEMPLATES configuration variable in the spong.message file. The 'DEFAULT' template is use is no other template is found. Templates override can be defined for any contact, message module, or a combination of the two. See MESSAGE TEMPLATES in spong.message for information on the file format.

spong-message alerts people via the messaging modules that are installed. New messaging functions can be easily created. See the Message Modules section in the Developer Guide.

spong.hosts

spong.hosts defined attributes for two things of important to <spong-message> 1) the hosts that Spong is monitoring (%HOSTS), and 2) the contacts that are responsible for the various hosts and how to contact them (%HUMANS). See spong.hosts for a full description of all of the file formats.

spong.message

This file hold the rules that determine who is to be contacted, when and how often. There are some of the important configuration variables in the spong.message file 1) how the messaging rules are to be scanned ($RULES_MATCH), 2) the messaging rules ($MESSAGING_RULES), 3) how to format the messages being sent (%TEMPLATES).

From spong.hosts:

%HUMANS

The %HUMANS configuration variable hold all of possible message recipients (the humans) and the information necessary on how to contact them. Each human contact can be a person, a group of people, or really anything you want. Each human that is defined should have name attribute associated with it. name is the name or description of the contact. To send out any notifications at least messaging attribute must be define for the human. A messaging attribute consists of a message module name as a key and contact information as a value. See EXAMPLES for a detailed %HUMANS example.

%HOSTS

The %HOSTS configuration variables can hold a list of regularly scheduled maintenance periods (down) for each host. Any alerts that are generated during a maintenance period will be silently discarded.

From spong.message:

$RULES_MATCH

$RULES_MATCH determines how the rules in $MESSAGING_RULES are scanned. If it is FIRST_MATCH the rules will be scanned until the first rule that matches the messaging criteria. A value of ALL means that all of the rules are scan with the contacts of all matching rules being adding into the list of contacts to notify. If the value is OLD, then spong-message will fall back to the messaging code used in Spong versions 2.0 - 2.1 for compatibility.

$MESSAGING_RULES

$MESSAGING_RULES contains the rules that how, who and how often contacts are notified. See $MESSAGING_RULES for the full rules syntax.

%TEMPLATES

CW%TEMPLATES determined how notification message are formatted. A template format consists a subject and message template strings. Both strings are not required as some message module will not require a subject or a body. See $MESSAGING_RULES for the full rules syntax.

From spong.conf:

$SEND_MESSAGE

$SEND_MESSAGE defines when spong-message is called by spong-server. It is not really specific to spong-message, but I think it is useful to describe its behavior here. This variable can contain one of four valid values. If it is RED, then spong-message is called for every time a system or service reports a problem. If its value is CHANGE, then spong-message is only called when there is a change of state . If this values is RED-CHANGE, then spong-message is called every time a service or service reports a problem and when the condition is cleared. (going from green/yellow to red, and then again going from red to green/yellow). If its value is NONE, then spong-message is never called.

$MESSAGES_PER_HOUR

$MESSAGES_PER_HOUR is the maximum number of messages that are sent to the same person in an hour. All message past this number are just logged to the history file, but are not sent. This helps to prevent against message overload such as when you have a networking problem, and everything appears to go red at once. The default value is 5.

$IDENT_MESSAGES_PER_HOUR

$IDENT_MESSAGES_PER_HOUR is the maximum number of identical messages that are sent to the same person in an hour. The default value is 3.

spong.conf, spong.hosts, $SPONGTMP/message-db, spong.message

Here are some examples to show you possible configurations.

spong.hosts

   %HUMANS = (
      'unix-staff' => { 'name'   => 'Midrange On-call Staff',
                        'email'  => 'its-unix@school.edu'
                       },

      'georgew'    => { 'name'   => 'George Wilson',
                        'email'  => 'georgew@school.edu',
                        'skytel' => '1234567' }, 
   );

   %HOSTS = (

     'grad.cs.school.edu' =>      { services => 'ftp smtp http',
                                   'down'   => ["*:05:30-06:30",
                                                 "0:00:00-04:00" ] },

     'www.school.edu' =>         { services => 'ftp smtp http' },

   );

spong.message

    $RULES_MATCH = 'FIRST-MATCH';

    $MESSAGING_RULES = [

       { hosts => [ 'gard.cs.school.edu' ],
         contacts => [ 'unix-staff'],
         exclude_colors => ['green','yellow'],
       },

       { hosts => ['www.school.edu'],
         contacts => [ 'georgew:email',
                       { rcpt=>'georgew:skytel', repeat=>900, }, ]
       },

    ];

    %TEMPLATES = (

      'DEFAULT' => { subject => 'spong - !!COLOR!! !!HOST!! !!SERVICE!!',
                     body    =>
    '!!DATETIME!!
    !!COLOR!! !!HOST!! !!SERVICE!!
    !!SUMMARY!!',
                   },

    'email'  => { subject => 'spong - !!COLOR!! !!HOST!! !!SERVICE!!',
                  body    =>
    'Host !!HOST!! service has been reported !!COLOR!!.
    Summary: !!SUMMARY!!

    Spong Web Page:      !!WWWSPONG!!
    Service Detail Page: !!WWWSPONG!!/service/!!HOST!!/!!SERVICE!!

    Status event details:
    !!DETAILED!!
    ',
              },
    );

Perl v5.003 or greater is required.

To receive pages, you currently must have a pager that can be contacted electronically (via email or web interface).

No know bugs.

spong-server, spong.hosts, spong.conf, spong.message, spong-message-mod-template, strftime(3)

Ed Hill <ed-hill@uiowa.edu>, Unix System Administrator, The University of Iowa

Stephen L Johnson <sjohnson@monsters.org>

Based on code/ideas from Sean MacGuire (BB), and Helen Harrison (Pong). Ed Hill original converted Big Brother (http://www.bb4.com) into Perl which diverged from Big Brother to become Spong. Ed Hill continued Spong development until version 2.1. Stephen L Johnson took over development in October, 1999 with his changes which became Spong 2.5.