newsx - news server exchange; post and fetch news articles

newsx [ options ] [ spoolname [ [ hostname ] port ] ]

newsx is a client connecting to an NNTP server, posting outgoing articles batched by the news system, while also fetching new incoming articles from the same newsserver.

It provides the NNTP capabilities required for serving a small local news spool such as those that may be found on installations with NNTP access through limited ISP accounts only, as well as being suitable for exchanging news with supplemental news sources for full scale news servers.

The program would normally be run by user u.

When started, newsx will go through the spoolname outgoing spool queue, and attempt to post all articles therein to the NNTP server hostname.

If a username and password for use with the simple authorization protocol have been supplied, they will be sent to the server first. If the -r option is specified, a MODE READER command will be sent.

When the outgoing transfer is finished, articles will be removed from the outgoing spool queue if successful transfer has been acknowledged by the NNTP host, if the article was specified by the host as a duplicate (already posted), or if a spooled article can no longer be found.

The fetch phase is then initiated. Articles available from the newsserver will be fetched and placed in the incoming spool, ready to be injected into the news system by newsrun (C News) or rnews -U (INN).

The groups are visited in the sequence that they appear in the host incoming state file, see in.hosts(5). This may be used actively to prioritize between newsgroups. A tag mechanism controlled by the -e option is available to limit the number of groups that will be fetched.

Group names not present in the active file will not be fetched. Every group name is checked against the spoolname entries in the sys (C News) or newsfeeds (INN) file, as well as any --groups option, and only groups allowed there will actually be fetched. For C News, the group name is also checked against the ME entry, whereas in INN the ME entry is just prepended to the spoolname entry.

In this way, the sys/newsfeed file is used to control which newsgroups are actually received from the remote site, in the same manner that the local server determines which outgoing article will go where.

If an -e option hasn't been specified, the local active file is then traversed to see if there are groups not mentioned in the current host newsrc file. Again, only groups allowed by sys or newsfeeds is actually fetched. Since these groups don't have a starting point specified, all available articles will be fetched.

To avoid transferring cross-posted articles more than once, articles with messags IDs already fetched in the same session will not be fetched again.

To make multiple news sources to be used in an efficient manner, articles present in the history database will be skipped.

The pull phase can be omitted entirely by using the -g option.

Statistics on the connection will be logged to syslog (or stderr with -d). The posted article count is the total number of articles actually posted. The duplicate count is how many message IDs from posted messages that were found to already exist on the remote system. The missing count is those posted articles that could no longer be found locally, usually because they were canceled locally, or perhaps since they were expired already.

The connection time and the total number of characters transferred via the NNTP layer is logged, in addition to the average speed of article transfer. The speed given is the net rate, the calculation being based on the actual number of characters belonging to news article head and body parts.

A log file may be specified with the -l option, and a folder of posted messages maintained with the -f option.

options

As detailed below.

spoolname

The name of the outgoing spool queue. If not specified, the value of the NNTPSERVER environment variable will be used.

hostname

The hostname of the remote NNTP server to connect to. If not specified, the name of the spool will be used.

port

The port used for connecting to hostname. Will usually not need to be specified, and port "nntp" will be assumed. The port name "telnet" will receive special treatment.

-f folder, --posted folder

Posted articles will be appended to the standard mail folder indicated. Note that the article will be added to the folder the first time posting of an article is attempted. The actual status of the posting will appear in the log file.

-l logfile, --log logfile

Log information about successfully posted articles to the file indicated.

--stat statfile

A summary of article exchange statistics will be logged to the file indicated.

--scanlogs

Obey locks set by the INN scanlogs utility. This is currently only meaningfull for use with a modified scanlogs that understands newsx logs.

-a authfile, --auth authfile

If specified, the username and password combination found in authfile will be sent to the server as part of the simple authorization protocol when the connection is first made. The file should consist of a line with two blank-separated words, the first being the username, the second the password. Do ensure that this file only has read access only for those users that need to know.

--authgeneric

Authentication with the server is performed using AUTHINFO GENERIC. This works by spawning a child process via /bin/sh string, where string is set in the environment variable NNTPAUTH. This process is expected to handle the authentication portion of the connection.

-r, --reader

A MODE READER command is sent to the remote server. This is used when connecting to remote INN hosts that needs it, and is the default state when using --ihave. With ihave, the command is issued between the ihave and fetch phase, otherwise the command is issued initially.

--no-reader

No MODE READER command is sent to the remote server. This is the default state when not using --ihave.

--readbeforeauth

Send the MODE READER command before attempting to authenticate via an AUTHINFO command. Otherwise, it is sent after the authentication. This option will obviously have no effect unless you also specify both --reader and one of --authgeneric or --auth authfile.

-w chatfile, --chat chatfile

Use chatfile to control any special connect sequence that may be required before the NNTP protocol is connected. Only implemented for the "telnet" protocol.

-x command, --exec command

If specified, the command will be executed after the host port has been opened. This option is used in rare cases when special tricks has to be employed to get an NNTP connection activated.

-y program, --connect program

If specified, connect via a bidirectional pipe to program instead of to a socket. The hostname and port arguments will be ignored.

-p, --no-post

Do not attempt to do any posting.

--ihave

Use NNTP IHAVE mechanism for submitting news messages, instead of the default POST. If this is for forwarding news in general, and not necessarily news generated locally, the setting of the --fail and --bounce option should be carefully reviewed.

-m, --no-msgid

Remove any message-ID header field from the posted message. This should cause the receiving end to generate a fresh message-ID. It should never been used together with --ihave.

-k, ----keep-path

Keep the Path when posting. In posting mode, the Path is by default stripped together with the NNTP-Posting-Host and Xref header items, so that the path of the news article as posted will appear to start at the actual posting host. With --ihave, the path is always included.

--continue

Ignore (but report) errors about non-existing outgoing spools, and continue with news fetching instead.

--max-path len

All outgoing articles are checked to see if the length of

their paths are not too long. This is an extra precaution against inadvertently submitting foreign articles for posting. This option sets the maximum path length allowed. The default value is 1, consistent with a simple, local, newsspool. To disable this feature, set it to an impossibly large value, like 999.

-g, --no-fetch

Do not attempt to fetch any news articles.

-e end_tag, --end end_tag

Specify a tag in the incoming host state file, see in.hosts(5). News fetching will end when a corresponding tag line has been reached. The tag line consists of a colon followed by the tag itself. This is a useful tool for dividing the newsgroups into several categories. Only a few newsgroups may be fetched during prime time, whereas the full list can be fetched once every night, for instance.

-W winsize, --window winsize

Specify the window size with respect to how many outstanding NNTP requests will be allowed at any one time. The default setting is 10. A value of 0 will disable windowing (aka. streaming) completely, making debugging easier. Increasing the window can significantly improve the article transfer speed. The effect is particulary pronounced over slow links and when many of the articles are already present in the local spool, where an improvement in speed of over 10 times has been observed compared to running with windowing disabled. All this at the penalty of an increased host newsserver load, obviously.

-b minspool, --size minspool

Specify the minimum incoming spool size. After every new group, if the spool file has reached the byte size specified, a new spool batch will be made. Setting this value to 1 will cause every newsgroup to be placed in a separate spool.

--rnews

Pipe incoming messages directly to rnews instead of placing them into the in-coming spool.

--pipe-to program

Pipe incoming messages to program instead of placing them into the in-coming spool. A separate program and pipe is initiated for each incoming message.

--rnews-to program

Like --pipe-to, but the pipe is continuous for each session, with article seperation is in rnews style.

-z, --sync

Omit actual pulling of news articles. Instead, simply update the host incoming state file to synchronize with the currently highest available article numbers from the server in question. This option can be useful when a fresh local spool is initiated. No article posting will be attempted. See also in.hosts(5).

-Z, --syncnew

Omit actual pulling of news articles in previously unseen groups only. The net effect is that newly added groups will start fetching from now on, instead of from the first article available at the host server.

--reset

Ignore the latest article information in the host incoming state file and attempt fetching all articles available from the server, subject to the normal history lookup constraints. This option is useful if the remote spool has been reconfigured. See also in.hosts(5).

--filter filterprog

Enable message filtering via the program filterprog. Uses the highwind interface model, where each article is given to the filter in NNTP-fashion, and the filter responds with a 335 to accept, and a 435 to reject a message.

--mfilter filterprog

Enable filtering by message ID. The filterprog will be invoked once for each unseen message, with the message ID as a command line argument. A non-zero return status will prevent the message from being transferred. This function is obsolete, and will probably be removed in later versions.

-X num, --maxnew num

Fetch at most num articles from each new and previously unseen newsgroup.

--maxart num

Fetch at most num articles from each newsgroup. Note that this might cause articles to be silently lost.

--no-path

Unless the exclusion pattern is set to newsx, all incoming articles are checked to see if their path is consistent with the exclusions given in f. This option omits this check.

--forget-inactive

Remove information in the incoming hosts state file about newsgroups that are removed from the a file. See also in.hosts(5).

--minfree N

Don't fetch news if there are less than N kbytes free space in the news spool.

--group list

Specify pattern for groups to be fetched that will apply in addition to the ones in f. The syntax is basically the same as for INN newsfeeds(5). It is recommended to put list in single quotes, since shells may otherwise treat the ! character in strange ways. For a list that specifies only negations, an initial '*' clause will be assumed.

This options is particularily useful to specify unsymmetric behaviour, i.e.

groups where there will be outgoing traffic, but no incoming articles. For instance, to explicitly avoid fetching of control groups, use:

--groups '!control.*'

It can also be useful to specify that only a subset of groups should be fetched for the occasion, e.g:

--groups 'comp.*,!*.advocacy'

It is recommended to put the argument for --groups in single quotes.

-c, --cnews

Set to C News mode as opposed to INN. Controls details regarding handling of spool and lock files.

-i, --inn

Set to INN mode as opposed to C News. Controls details regarding handling of spool and lock files.

--newline

Uses single newline character as line terminator instead of the carriage return, line feed sequence.

-v, --version

Display program version. Include this when reporting bugs to http://www.kvaleberg.com/bug/

--help

Give a very brief usage summary.

--no-ps

Do not update the process status display.

Note: Unless otherwise specified, these options are in addition to the usual actions and do not imply --no-post or --no-fetch. Specify those switches if desired.

--list listname

Obtain a list of newsgroups from the newsserver to listname. The list consists of one line per newsgroup, containing the newsgroup name, a blank, and a letter showing the group status (usually 'y' for an active group, 'm' for a moderated group).

If the first character is a '|',

listname is assumed to be a program path which will receive the list as standard input (you will need to put the entire argument in quotes to escape the usual shell interpretation of '|'). If it is a single '-', the list will appear on the standard output. Otherwise it will be assumed to be a filename.

--newlist listname

Obtain a list of newsgroups not currently present on the local server from the newsserver to listname. The list is also subject to the limitations of the sys or newsfeeds file. The operation is otherwise as for the --list option.

--desc descname

Obtain a list of newsgroups descriptions from the newsserver to descname. Only descriptions for locally active newsgroups relevant to the newsfeed in question will be fetched.

The list consists of one line per newsgroup, containing the newsgroup name,

a tab, and a short textual description. A pipe may be specified in descname kist as for the --list option.

--alldesc descname

Obtain a list of all newsgroups descriptions available from the newsserver to descname. The operation is otherwise as for the --desc option.

-t timeout, --timeout timeout

Specify timeout for TCP/IP and lockfile operations, in seconds. The default value is 600.

--fail time

Specify the maximum age of an article. Whenever an attempt of posting an article fails, and the specified time limit is exceeded, the article will be declared as failed, and a bounce message will be generated. The default unit is hours; the suffixes s for seconds, m for minutes, h for hours, d for days and w for weeks would change this. They can also be combined, e.g. "4h30m". In absence of this option, no fail by age will be performed. Setting the time to zero will bounce messages after the first failed attempt.

--bounce addr

Specify the destination address for messages bounced due to the fail time being exceeded. If addr is specified as none, no bounce will be generated. The default value, poster, causes messages to be returned to the sender. This setting should definitely only be used in situations where newsx is utilized for posting locally generated news.

--attach how

Specifies the way the original message is handled for bounced messages:

mime

Send the original as a Mime attachement. This is the default behaviour.

yes

Append the original to the mail message itself.

no

Do not attach the original message.

The following options are not for ordinary use: They would only be used for special configurations where newsx is used as a sort of gateway to inject news messages from other sources.

--inews

Pipe incoming messages directly to inews, inject them into the local spool via the same interface as is used for local new messages.

--inews-options options

Specify command line options for --inews. The leading '-' must be included. The default inews options are "-hOS".

--add-header header

Add a specific header to incoming messages. The header will be added exactly as specified, adding a trailing newline.

The exception is for the special 4 letter string "Path".

This will be replaced by a header line that says "Path: HOSTNAME!not-for-mail" if the incoming message contains no Path:. This can be quite essential for news gateways to prevent injected messages to be retured back to the source.

--profile file

Read a newsx profile from file . This file can contain command line options to specify default newsx behaviour. See newsx.conf(5) for an overview of the format.

Any option may be used in profile files. Some options are really only meaningfull in profile files:

--spoolname spoolname

Specify a default spoolname.

--hostname hostname

Specify a default hostname.

--port port

Specify a default port for connecting to hostname.

Configuration to the local news system peculiarities is traditionally done at build-time, but it is possible to specify at runtime in various ways.

--config

Read and obey the standard news configuration file in c. This allows newsx to adapt to changes in the local newsserver configuration dynamically.

--config-is configfile

Like --config, except that the configuration filename is specified. A list of files may also be specified, seperated by colons. In this case, the first file on this list that exists will be used.

--home newshome

Use newshome as the news home directory instead of the default n. A full path must be specified.

-s spooldir, --spool spooldir

Use spooldir as the news article spool directory instead of the default s. A full path must be specified.

--togo togofile

Specify the name of the C News togo file for outgoing news. No effect in INN mode. Required only if different from the default value o/spoolname/togo. A leading '/' specifices a full path, a leading '.' is relative from the current directory, otherwise the name is relative to o/spoolname.

--batch batchdir

Use batchdir as the outgoing news article spool directory instead of the default b. A leading '/' specifices a full path, a leading '.' is relative from the current directory, otherwise the name is relative to s.

-h historydb, --history historydb

Specify the name of the history database file, used for checking if news articles are already present in the spool. Required only when using a name different from the default h. An empty argument "" will prevent any referrence to the news history database.

--active activefile

Specify the name of the active newsgroup file for incoming news. Required only if different from the default value a. A leading '/' specifices a full path, a leading '.' specifies a path relative to the current directory, otherwise the name is relative to n.

--newsfeeds newsfeedsfile

Specify the name of the newsfeed specification file. This will override the default both in C News (sys) and INN (newsfeeds) modes. File name conventions as for --active.

--incoming incomingdir

Use incomingdir as the incoming news article spool directory instead of the default i. Fine name conventions as for --active, except that the default directory is s.

--inhosts inhostsdir

Use inhostsdir as the news host active file directory instead of the default j. Name conventions as for --incoming. See also in.hosts(5).

--hostconfig hostconfigdir

Use hostconfigdir as the directory for the host specific profile. instead of the default k. Name conventions as for --incoming. See also newin.hosts(5) and newsx.conf(5).

--debug level

Set debug level. Any level different from zero diverts reporting to stderr instead of syslog, and turns on extra debugging output. Level 2 will output a '.' for every new article. Level 3 will log all NNTP commands. Level 4 will produce the same state information that is available via the process status display. Going to level 5 will produce additional debug information, level 6 will include history database debug, while level 7 also will show all communications over the NNTP socket, article content included.

-d, --verbose

Enable debugging, and sets the debug level according to the number of times it is specified.

-n, --dry-run

No-action flag, will "fake" an NNTP connection, and leave the outgoing batch untouched. Probably useful only with --debug for debugging and dry-testing.

--verify

Used together with --configor --config-is, causes the configuration to be verified, but not adapted to.

-o, --keepold

Keep the previous outgoing spool in an ".old" file. This might seem like a useful option for ordinary use, but the problem is that a non-empty file might cause a news watch program to believe there is a stale outgoing spool.

-u, --no-force

After the timeout of lockfile operations, just give up and do not attempt to unlock the lockfiles and remove the stale process.

--no-queue

Do not queue up for access to a news host. If a lock already exists for the specified host, just give up and don't queue up for it.

--locks locksdir

Use locksdir as the lock file directory instead of the default l. Fine name conventions as for --active, except that the default directory is n. If the string none is supplied, no locking will occur. All this should of course be used only if you really, really know what you are doing.

-q "msgid", --enquire "msgid"

Enquires whether the msgid is present in the local history database. The msgid should include the angle brackets. Implies --no-post and --no-fetch.

--no-hostlock

Do not implement the newsx host access lock. Should only be used if you really know what you are doing.

--no-next

Do not use the NNTP NEXT command. This causes newsx not to attempt to use NEXT for filling out gaps in the article sequence.

--missing "num"

Tunes the number of missing articles in a row before a NEXT command is issued, instead of a sequential STAT. A value of zero will cause NEXT to be used extensively. The default value is 0 if no window, 2 if a window is specified.

--keep-fake

Some remote news servers generate faked news articles. Normally, you would want newsx to skip them, but with this option you can actually fetch them.

The -w option specifies that a special connect sequence is required, and that a script for this sequence can be found in the chatfile. This file consists of lines that contains pattern receive and send pairs, separated by blanks. The patterns may be enclosed in quotes. The script is currently only available for the "telnet" protocol.

A typical invocation line would be: newsx -w chat.acme acme login.acme.net telnet

The file chat.acme could look like this: # login for acme.net NNTP

login: myusername

Password: mypassword

$ "exec telnet news.acme.net nntp"

ECHO

The script will have to be adapted for local conditions, of course.

The special tag ECHO specifies that line echo should be suppressed, required since the host telnet operates in line mode.

Since this file usually contains passwords and other sensitive information, ensure that read access to it is limited.

The ps process status will also show the current newsx status. During the actual news transfer phase, it will show which group and article number that is being fetched. A continuous status can be obtained by; newsq -c

This feature can be disabled via the --no-ps option.

For INN, the following configuration items will be used:

$INND

Signals an INN configuration.

$NEWSBIN

Directory for INN binaries.

$NEWSHOME

News home directory.

$PATHETC

Directory for news configuration files. Defaults to $NEWSHOME if not available.

$SPOOL

If $SPOOL contains the string "/articles", the incoming host directory $INHOSTS will be formed by replacing it with "/inhosts". Otherwise, "$SPOOL/in.hosts" will be used.

$BATCH

Outgoing spool.

$INCOMING

Incoming spool.

$ACTIVE

Active file.

$HISTORY

History database.

$LOCKS

DIrectory for lockfiles.

$NEWSFEEDS

Newsfeeds file.

$EXTENDEDDBZ

True is history database in extended format. Only relevant for early INN 2.x where there is no libinn, otherwise ignored.

For C News:

$NEWSBIN

$NEWSCTL

Equivalent to INN $NEWSHOME. $ACTIVE will be "$NEWSCTL/active". $HISTORY will be "$NEWSCTL/history". $NEWSFEEDS will be "$NEWSCTL/sys". $LOCKS and $PATHETC will be "$NEWSCTL".

$NEWSARTS

Equivalent to INN $SPOOL. $BATCH will be "$NEWSARTS/out.going". $INCOMING will be "$NEWSARTS/in.coming". $INHOSTS will be "$NEWSARTS/in.hosts".

The -l log file of posted articles will contain one line for each article. To be compatible with the INN format, each field is separated by a tabstop:

Month

Standard 3 letter abbreviation.

Date

2 digits

HH:MM:SS

Local time.

Year

4 digits.

Spool

Name of outgoing spool.

<Msgid>

Within angle brackets.

Filename

As it appears in the spool file.

Sender

The value of the Sender-field, or From-field if no Sender is available.

Status

"OK" for successfully posted items, "Duplicate" for items already posted.

Lines

Number of lines in the posting.

Status

Message status. "OK" if everything went well, otherwise an error message appears.

The incoming spool consists of files with the file name ending by ".t". Each file can contain many articles. Each article is prefixed by the following header: #! rnews N

The number N is the number of bytes of the article proper, not counting the header line. The article follows after the header line. It appears exactly as received from the news server except that CR/LF sequences are replaced by proper newlines, and the terminating full stop is not included.

Incoming message headers will pass through without modification, with the exception that the text "newsx" will be added if not there already when the exclusion pattern for the newshost is set to "newsx".

Most outgoing message header items will also pass through, altough some header items are handled specially:

From:

Will be used for indentifying the sender if everything else fails.

Message-ID:

Will be used for logging purposes. The --no-msgid will remove this item for messages being posted.

NNTP-Posting-Date:

NNTP-Posting-Host:

Will always be removed for messages being posted.

Path:

Will be removed for messages being posted, unless the ----keep-path option is given.

Reply-to:

Will be used for indentifying the sender.

Sender:

Will be used for indentifying the sender.

Xref:

Will always be removed from outgoing messages.

X-Server-Date:

X-Trace:

X-Complaints-To:

Will always be removed for messages being posted.

newsx returns error codes as follows: 0 - Successful completion

1 - General system error

2 - Incorrect arguments supplied.

3 - Error connecting to remote host

4 - NNTP Protocol error

5 - Errors accessing outgoing spool file.

6 - Errors accessing host newsrc file.

7 - Errors accessing incoming spool file.

8 - Errors accessing local active and sys files.

9 - Socket access error (e.g. connection timeout)

10 - Errors accessing authinfo file.

11 - Connect script failure.

12 - Option -q did not find the message ID.

13 - Wait for lock timed out.

Distribution fields are ignored by newsx, and the handling of them left to the news server.

Report any bugs to http://www.kvaleberg.com/bug/

See also in.hosts(5).

e/newsx.conf

Newsx standard profile. If it exists, this profile file will be read before the command line arguments are parsed. See newsx.conf(5).

c

The standard news configuration file.

For C News:

o/spoolname/togo

Spool file

o/spoolname/togo.old

Previous spool file

o/spoolname/LOCKb

Lock file

i/.tmp.$$

Incoming spool, temporary file. $$ is the process ID.

i/T.$$.S.t

Incoming spool, when finished and ready for newsrun. T is the current Unix time, $$ is the process ID, and, S is a sequence number.

a

Local active file, contains a list over all currently active newsgroups in the local spool. This is used as a basis for constructing a list over which newsgroups will actually be requested.

n/sys

Newsserver configuration file. This is used as a filter against the local active list, deciding which newsgroups which will actually be pulled. The basic format of each specification is:

site/exclusions:grouplist/distlist:flags:cmd

If the exclusions field is set to newsx, newsx will modify the path of incoming news to include the "newsx" pattern.

The site field identifies the remote news source.

The grouplist is a comma separated list of group pattern names. A !-prefix signifies exclusions, and the name all matches anything.

h

h.pag

h.dir

Local news spool database.

For INN:

o/spoolname

Spool file

o/spoolname.old

Previous spool file

l/LOCK.spoolname

Lock file

a

Local active file, contains a list over all currently active newsgroups in the local spool. This is used as a basis for constructing a list over which newsgroups will actually be requested.

f

Newsserver configuration file. This is used as a filter against the local active list, deciding which newsgroups which will actually be pulled. The basic format of each specification is:

site/exclusions:grouplist/distlist:flags:param

If the exclusions field is set to newsx, newsx will modify the path of incoming news to include the "newsx" pattern.

The site field identifies the remote news source.

The grouplist is a comma separated list of group pattern names. A !-prefix signifies groups not exchanged, and the name * matches anything.

h

h.pag

h.dir

Local news spool article database.

Egil Kvaleberg <egil@kvaleberg.no>

newsq(1), in.hosts(5)

C News: news(5), newsdb(5), newssys(5), rnews(8)

INN: rnews(1), inews(1), wildmat(3), active(5), history(5), newsfeeds(5), ctlinnd(8)

RFC-977 - Network News Transfer Protocol (NNTP)

RFC-1036 - Usenet Article Format

http://www.kvaleberg.com/ISP-Hookup-HOWTO.html

http://www.kvaleberg.com/newsx.html