nnadmin - nn database administration

nnadmin [ commands ]

nnadmin is a control program for the nnmaster(1M) daemon which is responsible for building and maintaining the database used by the nn(1) news reader.

nnadmin allows you to display extracts from the log file, display the "raw" contents of the database, make consistency checks on the database, instruct the running nnmaster to expire one or more groups, alter the options of the running nnmaster, and much more.

nnadmin runs in two modes: interactive and non-interactive.

In interactive mode, simple one line menus are used to show the available operations which are then selected by typing the letter associated with the command (normally the first letter in the command name).

In non-interactive mode, the commands argument will be used as a series of key-strokes which are interpreted exactly as if they were typed in from the keyboard in interactive mode. For example, to stop the nnmaster, the following invokation of nnadmin can be used:

nnadmin MK

which will select the (M)aster submenu from the main menu, and then the (K)ill entry from the submenu.

In non-interactive mode, the menus are not displayed and the commands are not echoed! nnadmin will exit when there are no more key-strokes to be read from the commands argument. It is not possible to specify a group name in the commands argument, so the functionalities of nnadmin that relates to specific groups are only available in interactive mode.

Some "dangerous" commands will require that you confirm them by following them by "Y" on the command line. The most noteable are IY (initialize database) and EY (expire all groups). These commands will be marked with a [Y] following the command name.

You can also invoke an interactive nnadmin using the :admin command in nn.

At all prompts you can hit `!' to spawn a subshell.

The working directory of the subshell will be changed to the database directory when invoked from the MASTER or DUMP menus, and it will changed to the group's spool directory (if it exists) when invoked from the GROUP menu.

From the main menu (identified by the ADMIN prompt) you can select the following operations:

C)onf

Show current configuration parameters such as directories, files, programs, network usage, etc.

E)xpire [Y]

Send a request to the nnmaster daemon to schedule (and run) expire for all groups in the database.

G)roups

Enter the GROUP submenu.

I)nit [Y]

Send a request to the nnmaster daemon to recollect all groups in the database.

L)og

Enter the LOG submenu.

M)aster

Enter the MASTER submenu.

Q)uit

Quit nnadmin.

S)tat

Print general statistics about the database. See the section on Database Statistics below.

U)pdate

Update the incore copy of the database master index.

V)alidate

Make a thorough consistency check on the database. If inconsistencies are found in a group, you will be asked whether a request should be sent to the nnmaster daemon to recollect the group (in non-interactive mode, requests will be sent automatically for all corrupted groups).

W)akeup

Send a wakeup signal to the nnmaster daemon to have it receive messages sent to it, perform the required actions, and then collect articles as necessary.

Z (silent validation)

This operation is identical to the Validate operation, expect that no output is produced during the consistency check; this operation is used by the nnmaster to execute the -C option.

The master menu (identified by the MASTER prompt) provides access to overall database information, and to send control messages to the nnmaster daemon.

C)heck

In interactive mode and in verbose batch mode (nnadmin MC), print a message telling whether nnmaster is running or not. In silent batch mode (nnadmin =MC) exit with a status code of 0 if nnmaster is running and 1 otherwise; this may be useful is administrative scripts.

D)ump

Enter the DUMP submenu.

F)iles

Print a listing (using ls(1)) of all the data and index files in the database.

G)roup

Print the master index entry for a single group identified by its internal group number.

K)ill

Stop the nnmaster when it has finished its current task.

O)ptions

Change the runtime options of the running nnmaster daemon. Currently, only the value of the -r and -e options can be modified.

S)tat

Print general statistics about the database. See the section on Database Statistics below.

T)race

Turn the trace option -t on or off in the running nnmaster.

The dump menu (identified by the DUMP prompt) allows you to print the master index entry for various selections of groups in the database.

A)ll

Print all groups in the database.

E)mpty

Print the empty groups in the database.

H)oles

Print the groups where the `min' field in the active file is not the first article saved in the database (because it doesn't exist or because it is ignored for some other reason, e.g. bad or old).

I)gnored

Print groups which are ignored, either in the GROUPS file or because of some other condition (mainly no spool directory).

N)on-empty

Print the non-empty groups in the database.

V)alid

Print the groups which are present in the active file.

in(W)alid

Print the groups in the database which are not present in the active file.

The log menu (identified by the LOG prompt) enables you the extract specific entries from the log file, and to truncate the log file.

The entries in the log file share the following format: <class>: <date> <time> (<user>): <message> where <class> identifies the message class, the <date> and <time> specify when the entry was made, the <user> specifies who created the entry (the letter "M" denote the nnmaster), and the <message> is the text of the entry.

To extract the log file entries of a specific class, simply enter the letter identifying the class:

A - admin to master communication

This class of messages are related to the sending of messages from an nnadmin program to the nnmaster daemon.

B - bad articles

Reports about bad articles which have been ignored or removed (controlled by the -b and -B options to nnmaster).

C - collection statistics

Statistics about collection of new articles. The message has the format:

Collect: nnn art, ppp gr, ttt s

meaning that nnn articles in ppp groups were collected in ttt seconds (real time).

E - fatal errors

Fatal errors encountered during operation. These errors require manual intervention to be fixed (some of the fatal errors occur if thing that "cannot happen" happens anyway, and may indicate a bug in the software).

M - nnmaster messages.

Master start/stop messages.

N - NNTP related messages

Various messages related to the NNTP part of the nnmaster, mostly about lost connections and failed attempts to connect to the NNTP server. These messages should only appear if you use NNTP, and your NNTP server is down for some reason.

O - old articles

Reports related to ignoring (and removing) old articles when building the database (controlled by the -O and -B options to nnmaster).

R - reports

Non-fatal error which enables the nnmaster to continue operation, but may prevent a user to run nn (file access problems). Reported problems should be checked. The most common report message will probably be

some.group: no directory

which indicates that the spool directory for that group has disappeared (most likely because it has been rmgroup'ed).

T - trace output

Messages produced as a result of using the -t option on the nnmaster. This is primarily for debugging purposes.

U - usage statistics

If nn is compiled with the STATISTICS option enabled, an entry will be made in the log file every time a user has spent more than five minutes on news reading. The message will have the following format:

USAGE hours.minutes

Since it is possible to suspend nn, or leave the terminal while nn is active, nn tries to be intelligent when it calculates the usage time so it will reflect the actual time spent on news reading. The usage statistics can be summarized using the nnusage(1M) program.

V - validation errors

When inconsistencies are detected in the database during validation, an entry for each corrupted group will be entered in the log file.

X - expire statistics

Messages similar to the Collect statistics reporting the result of running expire on the database. Reports related to ignoring, removing, renumbering, and reactivation of groups are also given class X.

To extract a specific entry class, grep(1) is used, so it may take a while on a large log file.

There are also a few special operations on the log file:

G)roup

Extract the entries which refers to a specified group.

(1-9) tail

Invoke tail(1) to extract the last 10-90 entries in the log file.

space

Equivalent to 1 (list last 10 lines of log).

(.) all

Display the complete log file.

(@) clean [Y]

Move the Log file to Log.old, and create a new empty Log file. If you want to clean out the old log file as well, simply repeat the clean operation (this will result in an empty Log.old file.)

When you enter the group menu (identified by the GROUP prompt), nnadmin will prompt you for the name of a news group, which you can enter with the usual completion feature described in the nn(1) manual. You can then perform the following operations on the specified group:

C)lear_flag

Clear a group specific flag. See the section on group flags below.

D)ata

Dump the contents of the data file containing the extracted article headers for the group.

E)xpire

Request the nnmaster to run expire on the group.

F)iles

List the files (using ls(1)) containing the index and data for the group.

G)roup

Switch to another group.

H)eader

Dump the master index entry for the group.

R)ecollect

Request the nnmaster to recollect all articles in the group.

S)et_flag

Set a group specific flag. See the section on group flags below.

V)alidate

Perform validation on the group's database information.

Z)ap [Y]

Remove group from news system - this will be done by running the rmgroup program which must reside in the NEWS_LIB directory. Of course, this should be done with great caution.

You can set and clear the following flags for individual groups to control the future behaviour of nnmaster on that group.

Notice that these flags will be reset to their default value if you reinitialize the database using nnmaster -I. To change these flags permanently, they should be set or cleared in the GROUPS file.

A)lways_digest

Normally, nnmaster will only attempt to split digests into individual articles if it can easily recognize an article as a digest. This requires that the word "digest" appears somewhere in the subject line, and that one of the first few lines in the body of the article loosely matches the subject. A few news groups frequently receives digests which break one or both of these requirements. To have nnmaster split these digests into individual articles anyway, you can turn on the "always digest" flag on these news groups. This will instruct nnmaster to treat all articles in the group as digests (naturally, articles which are then found not to contain other articles are still treated as normal articles.)

C)ontrol

This is a special flag for the control group. It indicates that the "Newsgroups:" field in the article header cannot be trusted (it does not specify the groups to which the article has been posted.)

D)irectory missing

This flag indicates that the spool directory for the news group cannot be found (the group has probably been removed with rmgroup(1M)). It is set automatically be the nnmaster if it cannot access the directory. When the flag is set, nnmaster completely ignores the group, so it can be used to disable news collection in specific groups. If you recreate the group or the directory manually, you must also clear this flag to have the nnmaster recognize the group again.

M)oderated

Indicates that the group is moderated. This flag is normally initialized automatically from the active file, and it should not be changed lightly.

N)ever_digest

This is the opposite of the "always digest" flag; when set, the nnmaster will never attempt to split any articles in that group into subarticles.

When you select the (S)tat operation in the main or master menus, you will get some general statistics about the database:

initialized

The time when the database was last rebuild using nnmaster -I.

last_scan, last_size

The time stamp on the active file and its size the last time the nnmaster read it.

no of groups

The total number of groups in the database.

Articles

The total number of articles in all groups. This is not an exact number, because it will count split digests as a single article (making the number too small), and it may count some articles that have been expired (making the number too large).

Disk usage

The total number of (1 kbyte) disk blocks occupied by the database.

The master index entries displayed when you select the (H)eader operation in the master and group menus contain the following information:

group_name group_number

The first line of the display will show the name of the group and the internal group number which is used to identify the group in the database.

first/last art

This is the numbers of the first and last article that are currently stored in the database.

active info

This is the numbers of the first and last article in the news system as read from the active file. They will normally match the numbers above, but they may differ while the nnmaster is working on the group (or it has not yet collected all the articles in the group).

Offsets: index->..., data->...

These values show the starting position for the next write operation on the index and data files. They are primarily used for consistency checking and recovery after a system crash, but after an "expire by rewrite" operation (expire method 2) which is performed "in-situ", the data and index files may physically be longer than the actual data stored in them.

Flags:

This shows the current flags set for this group. If no flags are set, the field is omitted from the display. One extra flag which was not explained above is the BLOCKED flag; it is a temporary locking flag set on a group by the nnmaster while it is updating the database files for that group to prevent nn clients to access that group.

When you select the (D)ata operation on the group menu, you will get a combined display of the raw data and index files for that group. The index file contains a single 32 bit value for each existing article number. This value is an offset into the data file pointing to the header for the corresponding article.

When nn want to access the article from number N to the last article, it looks up the offset for article number N in the index file, and uses this as the starting point for reading article header information in the data file. It then simply reads to the end of the data file in which the article headers for articles number N+1, N+2, and so on follows immediately after the header for article number N.

The article header information is presented in a very terse form; each of the output lines are described below for reference purposes:

offset = xxxx , article # = nnnnn (type)

This shows the offset into the data file and the article number. The offset is stored in the index file for quick access. If no type is printed it is a normal article. Other types are: "digest header" and "digest sub-article".

xpost(count): nnn, nnn, nnn, ...

Cross-postings to other groups are encoded as a list of internal group numbers.

ts=nn hp=nn fp=nn lp=nn ref=nn[+Re] lines=nn

These values are used by nn to sort, present, and access an article:

ts is the time stamp on the article; it is a simple encoding of the posting date and time found in the Date: field.

hp, fp, and lp are offsets into the file containing the article text: the header position, first text position, and last text position. The first will be zero for normal articles, but not for articles in a split digest. The last will be equal to the length of the file for normal articles, but not inside digests.

ref is the number of references on the Reference: line. If "+Re" follows the number, the subject line contained a "Re:" prefix which has been removed.

Sender(length): name

The name of the sender in "ready to print" format, i.e. reduced to 16 characters as explained in the nn manual.

Subj(length): subject

This is the full subject line from the article header (except for Re: prefixes in various formats).

The $db, $lib, and $news used below are synonyms for the DB_DIRECTORY, LIB_DIRECTORY, and the news system's lib directories respectively.

$db/MASTER Database master index

$db/GROUPS News group names in MASTER file order

$db/DATA/nnn.x Index file for group number nnn

$db/DATA/nnn.d Data file for group number nnn

$master/GATE Message channel from nnadmin to nnmaster

$master/MPID The process id of the nnmaster daemon.

$Log The log file (truncate it regularly!)

The MASTER file contains a record for each news group, occurring in the same sequence as the group names in the GROUPS file. The sequence also defines the group numbers used to identify the files in the database and in a few other places.

The GATE file will be created by nnadmin when needed, and removed by nnmaster when it has read it. Therefore, to send a message to the nnmaster requires that you are allowed to write in the $master directory.

nn(1), nncheck(1), nngrep(1), nntidy(1)

nnquery(1), nnusage(1), nnmaster(8)

The GATE file is created with the owner and modes of the user that runs nnadmin which may cause problems if the owner of the nnmaster process (normally "news") is not allowed to read the created GATE file (a "umask" of 022 is ok.) Unless you allow ordinary users to create files in the LIB directory where the GATE file resides, only the owner of the directory (normally "news") and "root" can use nnadmin to send messages to the nnmaster. However, to send a wakeup signal to the master, anybody can run

nnmaster -w

The user interface is completely out of line with the rest of the nn family, and the way to run nnadmin in the non-interactive mode is a bit bizarre. This is not likely to change, because I believe there are more important things to do!

Kim F. Storm, Texas Instruments A/S, Denmark

E-mail: storm@texas.dk