uqwk - Collect news & mail messages for offline reading

uqwk [a-plethora-of-options]

uqwk is a program which collects all a user's unread news and/or mail, and formats it into a packet for offline reading, saving telephone and connect charges. Simple Offline Usenet Packet (SOUP), ZipNews, and the deprecated QWK packet formats are supported. Mail can be read from classic Unix mbox format mail files, MMDF mail files, or maildir mailboxes. News is either read directly from a locally available file system, or obtained using NNTP.

uqwk also accepts reply packets, so replies can be mailed or posted, depending whether the message is marked private (email) or public (news).

uqwk also supports a small offline command language, so the contents of the user's .newsrc file can be viewed and manipulated offline.

Note that many varieties of uqwk can be built: not all options are applicable to (and available) in every one. Specifically, the NNTP-related options are only available in uqwk builds using NNTP, the options related to reading news directly from disk only to non-NNTP builds. Furthermore, the options concerning scoring and NNTP authentication are only present if support for scoring, resp. NNTP authentication, was compiled in.

-p

Print options. uqwk will show its version number and the options that were compiled in, will examine all appropriate environment variables and command line options, print the values of all run-time options, then exit. This is useful to see what uqwk thinks it is going to do before you actually run it (and what setup it is offering).

-v

identical to "-p".

-?

same as "-v".

+r

Read only. Normally, uqwk will empty the user's mail spool file and update the user's .newsrc file to reflect the fact that mail and news have been read. If this switch is specified, uqwk will not touch these files. This is useful for testing.

-r

Do not execute in read-only mode. This is the default.

+m

Do mail. uqwk will process all mail in the user's mail spool file and convert it into an offline packet.

-m

Don't do mail. This is the default.

+n

Do news. Using the user's .newsrc file and the news system's active file, uqwk will collect all unread news articles in all subscribed newsgroups into the packet. This can generate a large number of messages and large packet files.

-n

Don't do news. This is the default.

+L

Operate in SOUP (also known as HDPF or SLNP) mode. The files comprising a Simple Offline Usenet Packet (SOUP), specifically, AREAS and *.MSG, will be created instead of a QWK packet. The reply file, if any, will be interpreted as an SOUP "replies" file instead of a QWK reply packet. This is the default.

-L

Do not operate in SOUP mode. If (and only if) neither SOUP nor ZipNews mode is selected, uqwk will operate in QWK mode.

+z

Operate in ZipNews mode. The files comprising a ZipNews news packet, specifically, "user.JN", "user.NWS", and "user.IDX", will be created instead of a QWK packet.

-z

Do not operate in ZipNews mode. If (and only if) neither SOUP nor ZipNews mode is selected, uqwk will operate in QWK mode. This is the default.

-Iposting_agent

Specifies the program used when posting messages (usually /usr/lib/news/inews). uqwk adds a `-h' flag when calling it, except in plain (non-expert) QWK mode.

+Vn

The value of n determines whether articles are validated before posting, whether this checking is verbose, and whether the result is respected or ignored:

+V Equivalent to `+V3' (default) +V0 No article validation +V1 Verbose article validation, ignoring the outcome +V2 Silent article validation, failing articles are not posted +V3 Verbose article validation, articles containing fatal errors are not posted

Whenever checking is performed, bad articles are appended to the file `dead.articles' in the uqwk home directory. This option is only available when uqwk was compiled with the `GNKSA' option, in which case the default value is `3'.

-V

Equivalent to `+V0', as defined above.

+x

Honor Xref header lines in news articles. When this switch is in effect, uqwk will only pack a crossposted article in the first newsgroup in which it is encountered. This can reduce the size of the packet but may introduce unexpected behaviour on the part of the offline reader. This is the default.

-x

Do not honor Xref lines.

+F

Ignore error on nonexisting news messages; proceed instead of aborting. This is the default.

-F

Signal an error and abort on missing news messages.

-Mmailspool

Look in the directory fImailspool for the mailbox file (or maildir mailbox). The default is /var/spool/mail.

-fmailfile

Look for mail in the file mailfile. The default is a file (or maildir mailbox) with the same name as the user's login name in fImailspool; however, the MAIL environment variable is used when set. This switch overrides the -M switch.

-aactivefile

Use activefile for the list of all available newsgroups and article numbers. The default is /usr/local/news/lib/active. This option is not available if uqwk obtains articles from a news server using NNTP.

-Snewsdir

Look for news articles in the directory newsdir. The default is /var/spool/news. This option is not available if uqwk obtains articles from a news server using NNTP.

-Nnewsrcfile

Use newsrcfile for the list of all newsgroups to which the user is subscribed and the list of article numbers which have been read. The default is $HOME/.newsrc.

-uusername

By default uqwk uses the getpwuid() system call to determine the proper user name to use. This switch overrides the results of that call.

-Hhomedir

By default, uqwk uses the getpwuid() system call to determine the user's home directory. The home directory is where the files comprising the offline packet will be created. It is also where uqwk looks for the .newsrc file. This switch may be used to override the results of the getpwuid() call.

-Bmaxblocks

Stop processing news articles if the size of the offline packet exceeds maxblocks 128-byte blocks. This is useful since large amounts of unread news can create large packets. Use a maxblocks value of zero to suppress this check. The default is 4096 blocks (half a megabyte).

-Dngfile

Process newsgroups in the order specified in ngfile. ngfile is expected to contain one newsgroup name per line.

-Usumfile

Operate in "summary" mode. Instead of packing articles, uqwk creates a summary file, sumfile, which consists of the newsgroup name, article number, and subject line of each unread article. All summarized articles are marked as read.

-Eselfile

Operate in "selection" mode. Instead of using the .newsrc file to determine which articles to pack, uqwk reads selfile, a file in the same format as the summary file. (The actual subjects are optional in selfile.) The idea is that a summary may be created by using -U, downloaded, edited or processed by additional software to select desired articles, uploaded, and fed to the -E switch for offline article selection.

-Rreplyfile

Process replyfile as a reply packet. Messages and articles created by the offline reader must be uploaded as a reply packet and then processed by this switch to be mailed or posted. If operating in SOUP mode, this file should be the REPLIES file. uqwk will remove replyfile when it has been processed unless it is running in read-only mode. If operating in ZipNews mode, this switch should specify the directory in which the ZipNews reply files reside. No default reply file is set.

-dhostname

Use the specified hostname when processing ZipNews reply packets. This should be the fully-qualified domain name of the Unix host. The default is "nowhere".

+h

Include headers. Since the QWK specification places limits on the sizes of certain header fields such as To:, From:, and Subject:, uqwk can include all message and article headers in the body of the message so all fields are visible. This is the default. (This switch only applies to the QWK packet format.)

-h

Do not include headers in messages. (QWK only.)

-llen

Truncate the names of newsgroups to len characters. Some offline readers cannot handle long newsgroup names. Newsgroup names are truncated after having been translated. The default is 15 characters. Specify a value of zero for unlimited newsgroup name lengths. (QWK only.)

+e

Include an entry for every subscribed newsgroup in the CONTROL.DAT file. (QWK only).

-e

Do not include an entry in CONTROL.DAT for empty newsgroups. This is the default. (QWK only.)

+W

Activate the Blue Wave kludge. The Blue Wave QWK reader does not like the personal.ndx file that uqwk creates by default. If this switch is specified uqwk will create a regular ???.ndx file that Blue Wave likes. (QWK only.)

-W

Do not activate the Blue Wave kludge. This is the default.

+X

Operate in expert QWK mode. The QWK header is ignored and all header lines are taken from the body of the message. In the hands of an expert, this permits the creation of RFC822 compliant mail messages and RFC1036 compliant news articles using QWK readers, which is difficult or impossible otherwise in many cases. This is intended for people who really know what they're doing. (QWK only.)

-X

Do not operate in expert QWK mode. This is the default.

-ttablefile

Translate newsgroup names as specified in tablefile. Each line in tablefile is expected to have two fields separated by white space: The old name of the newsgroup and the name to which it will be translated. (QWK only.)

-bbbsname

Specify the BBS Name to be entered into the QWK packet. The default is "unknown BBS". (QWK only.)

-ccity

Specify the BBS City to be entered into the QWK packet. The default is "Anytown, USA". (QWK only.)

-Pphone

Specify the BBS Phone Number to be entered into the QWK packet. The default is "555-1212". (QWK only.)

-ssysop

Specify the BBS Sysop Name to be entered into the QWK packet. The default is "Joe Sysop". (QWK only.)

-ibbsid

Specify the BBS ID to be entered into the QWK packet. The BBS ID is important since it will be checked against the BBS ID string in any incoming reply packets. If the two do not match, the reply packet will not be processed. The BBS ID consists of an integer, a comma, and a string of less than nine characters, with no spaces. The default is "0,SOMEBBS". (QWK only.)

+k

Perform kill/score processing. The global score file will be read (if it exists), as well as the score file for the current group, and each article will becored'. See the "USING SCORE FILES" section below. This is the default.

-k

Do not perform kill/score processing.

-Tn

Set global threshold to signed integer n. The default is 0.

-Kkilldir

Look for score files in killdir. The default is $HOME/.uqwk.

+Ausername:password

Perform NNTP authentication, specifying the username and password to use for that purpose. (NNTP only)

-A

Don't do NNTP authentication. This is the default. (NNTP only)

Most of the run-time options can also be controlled by environment variables. If an option is specified by both an environment variable and a command-line option, the command-line option is honored.

Note that, as with options, not all variables are meaningful in al possible incarnations of the program.

UQ_DO_MAIL

Determines whether or not to process mail. Should be set to `1' (do mail) or `0' (don't do mail).

UQ_DO_NEWS

Determines whether or not to process news. Should be set to `1' (do news) or `0' (don't do news).

UQ_INC_HDRS

Determines whether or not to include headers in messages. Should be set to `1' (include headers) or `0' (don't include headers).

UQ_EVERY_MODE

Determines whether or not to include an entry in CONTROL.DAT for empty newsgroups. Should be set to `1' (include empty groups) or `0' (don't include empty groups).

UQ_PRT_OPTS

Determines whether or not to just print the values of run-time options and stop. Should be set to `1' (print options) or `0' (don't print options).

UQ_READ_ONLY

Determines whether or not to run in read-only mode. Should be set to `1' (read-only) or `0' (not read-only).

UQ_WAF_MODE

Determines whether the .newsrc file should be interpreted as a Waffle "join" file. Should be set to `0' (normal) or `1' (Waffle mode).

UQ_SOUP_MODE

Determines whether or not to run in SOUP mode. Should be set to `0' (don't run in SOUP mode) or `1' (run in SOUP mode).

UQ_ZIP_MODE

Determines whether or not to run in ZipNews mode. Should be set to `0' (don't run in ZipNews mode) or `1' (run in ZipNews mode).

UQ_INEWS_PATH

Specifies the posting agent (usually inews) to be used. uqwk calls it with a `-h' command line switch, except in non-expert QWK mode.

UQ_VALIDATE

Determines the level of message validation. Applicable only with GNKSA support compiled in. See the `+V' switch for details.

UQ_XRF_MODE

Determines whether or not to honor Xref lines. Should be set to `0' (do not honor Xref lines) or `1' (honor Xref lines). UQ_IGNORE0FD Determines whether or not to ignore nonexisting news messages; `0' to abort, `1' to ignore.

UQ_BW_KLUDGE

Determines whether or not to activate the Blue Wave kludge. Should be set to `0' (do not activate the Blue Wave kludge) or `1' (activate the Blue Wave kludge).

UQ_XPRT_MODE

Determines whether or not to operate in QWK expert mode. Should be set to `0' (do not run in expert mode) or `1' (run in expert mode).

UQ_MAIL_DIR

Specifies the directory where the mailbox file (or maildir type mailbox) resides.

UQ_MAIL_FILE

Specifies the mailbox file (or maildir mailbox).

UQ_ACT_FILE

Specifies the name of the news system's active file.

UQ_NEWS_DIR

Specifies the directory where the news spool resides. UQ_USER_NAME Specifies the username of the person running uqwk.

UQ_HOME_DIR

Specifies the home directory, where the packet files will be created, and where uqwk expects to find the .newsrc file.

UQ_BBS_NAME

Specifies the BBS name to be entered into the QWK packet.

UQ_BBS_CITY

Specifies the BBS city to be entered into the QWK packet.

UQ_BBS_PHONE

Specifies the BBS phone number to be entered into the QWK packet.

UQ_BBS_SYSOP

Specifies the BBS sysop name to be entered into the QWK packet.

UQ_BBS_ID

Specifies the BBS ID to be entered into the QWK packet.

UQ_NRC_FILE

Specifies the name of the user's .newsrc file.

UQ_NG_FILE

Specifies the name of the desired newsgroups file.

UQ_MAX_BLKS

Specifies the maximum size of the offline packet in 128-byte blocks.

UQ_TRN_FILE

Specifies the name of the newsgroup name translation table.

UQ_GRP_LEN

Specifies the maximum length of newsgroup names.

UQ_REP_FILE

Specifies the name of the reply packet, if any.

UQ_HOST_NAME

Specifies the host name for ZipNews replies.

UQ_SUM_FILE

Specifies the name of the summary file, if any. Also causes uqwk to execute in summary mode.

UQ_SEL_FILE

Specifies the name of the selection file, if any. Also causes uqwk to execute in selection mode.

UQ_DO_KILLS

Determines whether or not to perform kill/score processing. Should be set to `1' (do scoring) or `0' (don't do scoring).

UQ_KILL_THRESHOLD

Specifies the global threshold. Should be set to a signed integer.

UQ_KILL_DIR

Specifies the directory where the score files are.

UQ_AUTH_USER

Specifies the username to use for NNTP authentication. If, and only if, both a username and a password are set, NNTP authentication will be attempted.

UQ_AUTH_PASS

Specifies the password to use for NNTP authentication. If, and only if, both a username and a password are set, NNTP authentication will be attempted.

If support for scoring was compiled in, uqwk can "score articles" while collecting news, so that articles whose "score" is below a certain threshold can be excluded from the packet. The rules according to which articles are scored are specified by the user in so-called score files.

The format of these score files is based on the score files used by some offline readers (notably YARN). Each newsgroup can have a score file; its name is the name of the group. There is also a global score file, that applies to all groups, which is called "global".

Getting started

If you find some class of articles you wish to exclude from a particular newsgroup, use any text editor to create a file named after it. Each line of such a file contains a score, a place to look for a keyword or regular expression, and the keyword.

For example, to exclude all crossposted articles from talk.bizarre, create a file called "talk.bizarre" in the directory $HOME/.uqwk, with the following line in it:

-10 Newsgroups: ,

The effect of this line in this file would be that all articles in talk.bizarre which' "Newsgroups: " header contains a `,' would get -10 added to their initial score of 0. With a threshold of 0, that means that unless other lines add enought to compensate, these articles will not be included.

Score file format

Each score file carries the name of the group it applies to, and consists of a series of score lines. Blank lines and lines beginning with `#' are ignored. Lines can have any of the following formats:

killthreshold n

specifies an integer score n that articles must meet or exceed in order to be included in the packet. This line is optional, if not present the global kill threshold is used. (which defaults to 0 if not set).

score header pattern

constitutes an actual scoring line; score is added to an article's score if the remainder of the line matches. To match, the specified pattern must match with the articles's indicated header. The specified pattern is always interpreted as a regular expression; for reasons of compatibility patterns can be preceded with the word "pattern".

The global score file is processed first. Its patterns are applied to all articles. Score lines are processed in sequence. An article that scores below the kill threshold will not be included.

Example alt.usenet.offline-reader score file

# only include articles explicitly selected killthreshold 1

# select messages about FurriNews +2 Subject: FurriNews # select messages about uqwk +2 Subject: uqwk

# eliminate messages about crapware -666 Subject: microsoft -666 Subject: netscape

Limitations, additional notes

It should be noted that only headers are scanned for computing scores. Furthermore, all scanning is done using POSIX regex functions -- even when a simple substring search could have sufficed.

If, while processing a reply packet, uqwk encounters a message to the username "UQWK", the body of the message will be interpreted as a small command language, used to display newsgroup names and subscribe or unsubscribe to newsgroups. The results of the execution of the commands will be mailed back to the originating user.

This is what the command language looks like:

HELP

List all the available commands.

SUBSCRIBE newsgroup

Subscribe to the named newsgroup.

UNSUBSCRIBE newsgroup

Unsubscribe from the named newsgroup. UNSUBSCRIBE ALL may be used to unsubscribe from all newsgroups. UNSUBSCRIBE ALL is also the only way to create a new .newsrc if it does not already exist.

CATCHUP newsgroup

Mark all articles in the named newsgroup as read.

GROUPS

List all newsgroups to which the user is currently subscribed.

ALLGROUPS

List all the available newsgroups and the number of articles in each one.

SHELL command

Execute the specified shell command. Anything written to standard output will be mailed back to the originating user.

In QWK mode, it is possible to send mail to long email addresses by including, as the first thing in the body of the message, a line of the form:

To: user@site.com

(The space after the colon is important.)

The handling of MMDF mail files is kludgy and should be fixed.

In QWK mode, if the .newsrc file is modified significantly between the time a packet is created and the time replies to that packet are processed, it is possible that articles may be posted to the wrong newsgroup. If you use uqwk along with a conventional online newsreader, it may be wise to maintain a separate .newsrc file strictly for uqwk.

uqwk does not do mail locking. New mail messages that arrive while packing mail can therefore be lost. If this worries you (it should), there are a few ways of preventing message loss. One approach (see the `getsoup' script, listed below, for an example) is to move your messages to a temporary file, and let uqwk collect the messages from that file instead of directly from the system mailbox. Since messages might get appended at any time, locking must be applied even while the mailbox is being moved. Another approach is to explicitly lock the mailbox before, and unlock it after, running uqwk -- i.e. like this:

lockfile -l666 -ml; trap "lockfile -mu" 1 2 3 13 15 uqwk +m lockfile -mu; trap "" 1 2 3 13 15

A third approach to run uqwk in read-only mode and clean up your mailbox afterwards; in which case one can but hope one really knows what one's doing.

uqwk does not support the ZipNews offline commands. To perform offline configuration using uqwk, send mail to the username "UQWK" in the format described in the COMMAND LANGUAGE section above.

It may be convenient for users to employ simple shell scripts for preparing packets using uqwk, and for handling reply packets. A few examples might prove instructive.

getsoup - collect mail and news into a SOUP packet

#!/bin/sh

UQWKARGS="-B0 -m +L"

# What to do? Set variables to enable. #UQ_DO_MAIL=1 UQ_DO_NEWS=1 export UQ_DO_MAIL UQ_DO_NEWS

# NNTPAUTH="yes please!"

# When reading mail from the system mailbox, locking # will have to be done somehow. If your Unix system # does not have lockfile, but does have /usr/ucb/mail, # use UQ_LOCKING="ucbmail". See below.

UQ_LOCKING="lockfile"

# # No user servicable parts below (I hope) #

prg=`basename $0`

if [ -z "$UQ_DO_MAIL" -a -z "$UQ_DO_NEWS" ] then echo "$prg: nothing to do" >&2 exit fi

UQ_HOME_DIR=$HOME/.uqwk UQ_NRC_FILE=$HOME/.newsrc export UQ_HOME_DIR UQ_NRC_FILE

# Do things in a nice, cozy subdirectory if [ ! -d $UQ_HOME_DIR ] then echo "$prg: $UQ_HOME_DIR does not exist, creating it" mkdir -p $UQ_HOME_DIR if [ ! -d $UQ_HOME_DIR ] then echo "$prg: failed to create $UQ_HOME_DIR" >&2 echo "$prg: bailing out" >&2 exit 1 fi fi cd $UQ_HOME_DIR

soupfile=soup.zip

if [ -f $soupfile ] ; then echo "Hmmm, $soupfile already exists." rm -i $soupfile if [ -f $soupfile ] ; then exit 1 fi fi

if [ -n "$UQ_DO_NEWS" ] then UQWKARGS="$UQWKARGS +n" # Get auth stuff (nntp auth mode only) if [ -n "$NNTPAUTH" ] then while [ -z "$UQ_AUTH_PASS" ] do echo "Authentication for server $NNTPSERVER;" echo "Username: $USER, password: \c" stty -echo read UQ_AUTH_PASS; echo stty echo done # Pass these through environment UQ_AUTH_USER=$USER; export UQ_AUTH_USER UQ_AUTH_PASS fi fi

if [ -n "$UQ_DO_MAIL" ] then UQWKARGS="$UQWKARGS +m"

# Read mail from the system mail box? if [ -z "$UQ_MAIL_FILE" ] then # uqwk does not do mailbox locking. Locking options: # - procmail's nifty "lockfile" utility # - /usr/ucb/mail ("ucbmail") UQ_MAIL_FILE=/tmp/$prg.$USER.mbx.$$; export UQ_MAIL_FILE

case $UQ_LOCKING in "lockfile") lockfile -l666 -ml; trap "lockfile -mu" 1 2 3 13 15 mv $MAIL $UQ_MAIL_FILE lockfile -mu; trap "" 1 2 3 13 15 ;; "ucbmail") ( echo 's1-$' $UQ_MAIL_FILE ; echo q ) | /usr/ucb/mail >/dev/null ;; *) echo "$prg: no, or unknown, locking method" >&2 echo "$prg: bailing out" >&2 exit 1 ;; esac fi fi

# # Now we know how to call uqwk, proper args 'n' all # uqwk $UQWKARGS; err=$? if [ $err -ne 0 ] then echo "$prg: fatal error in uqwk" >&2 echo "$prg: proceed with caution" >&2 fi

# Create the SOUP packet # if [ "`echo *.MSG`" = "*.MSG" ] then echo "Sorry, nothing to get." rm -f AREAS exit fi zip -m $soupfile AREAS *.MSG mv $soupfile $HOME

echo "Done, you can upload <ftp://$USER@ftp.`domainname`/$soupfile> now."

putsoup - process a SOUP reply packet

#!/bin/sh

prg=`basename $0` rf=reply.zip; cd $UQ_HOME_DIR

if [ ! -f $rf ]; then echo "$prg: no reply packet found" >&2 exit 1 fi

unzip -ju $rf 2>/dev/null

if uqwk -m -n +L -RREPLIES then echo "$prg: fatal uqwk error, proceed with caution" >&2 fi

echo "Done. If all went well, $rf can safely be deleted." rm -i $rf

inews(8), lockfile(1), mail(1), procmail(1)

uqwk was inspired by a conversation with Harry Lockwood <lockwood@world.std.com>.

QWK format was designed by Mark "Sparky" Herring <mark.herring@nashville.com>. The creator of SOUP is Rhys Weatherley <rhys@cs.uq.oz.au>. The creator of ZipNews is Jack Kilday <jkilday@nlbbs.com>.

Thanks to Patrick Y. Lee <patlee@panix.com> for the QWK documentation. Many thanks also to the beta-testers: Karl J. Vesterling <kjv@exucom.com> and Brian J. Coan <brian@igc.apc.org>.

Many thanks to Ken Whedbee <kcw@grumpy.ksc.nasa.gov> for the NNTP modifications.

Many other people, too numerous to list here, contributed bug fixes and suggestions for improvement.

Steve Belczyk, steve1@genesis.nred.ma.us, seb3@gte.com.

Copyright (C) 1993-1994 by steve belczyk. Permission to use, copy, modify and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation. This software is provided "as is" without expressed or implied warranty.

Jeroen Scheerder, js@cwi.nl, js@xs4all.nl, js@phil.ruu.nl.

Jim Tittsler <7j1ajh@amsat.org> came up with decent scoring code. Jos den Bekker <josdb@xs4all> provided valuable suggestions and bugfixes, most notably regarding error handling.