NAME

newsfeeds - Determine where Usenet articles are sent

IN A NUTSHELL

The newsfeeds file configures what innd does with incoming accepted articles.

A minimalist newsfeeds file which parameters an outgoing NNTP feed to "news.server.com", using path identity "server" (as it appears in Path header fields), of all the articles you receive is:

    # Mandatory line, with no exclusion patterns defined.
    ME:::
    # A real-time feed through innfeed of all your articles.
    news.server.com/server\
        :*\
        :Tm:innfeed!
    # The channel feed entry for innfeed.
    # Change "/usr/lib/news/bin" to the directory path
    # of the innfeed program.
    innfeed!:!*:Tc,Wnm*:/usr/lib/news/bin/innfeed

In order to find an external feed, you can ask in the news.admin.peering newsgroup. Several news administrators will certainly respond and gracefully provide you with a news feed.

Other entries may be added in newsfeeds depending on your needs. See the examples below and also in the archive(8), controlchan(8), news2mail(8), ninpaths(8) and perl-nocem(8) man pages for the most widely used.

If you exclude local hierarchies or newsgroups from the feed, make sure to add "!control,!control.*" so that cancel articles or any kind of control articles related to excluded hierarchies or newsgroups are not propagated to the remote peer. Its newsfeeds entry then looks like:

    news.server.com/server\
        :*,!local.*,@*bina*,!control,!control.*\
        :Tm:innfeed!

After any changes, run "inncheck" to perform basic syntax checks, and reload this configuration file with the following command:

    ctlinnd reload newsfeeds 'new peer'

DESCRIPTION

The file pathetc/newsfeeds specifies how incoming articles should be distributed to other programs and files on the server. It is parsed by the InterNetNews server innd(8) when it starts up, or when directed to by ctlinnd(8). innd doesn't send articles to remote sites itself, so newsfeeds doesn't directly determine which remote news servers articles are sent to. Instead, it specifies what batch files should be created or which programs should be run (and what information should be sent to them), and then this information is used by programs like innxmit(8) and innfeed(8) to feed articles to remote sites.

The newsfeeds file isn't used solely to set up feeding accepted articles to remote sites but also to pass them (or bits of information about them) to any local programs or files that want that data. For example, controlchan(8), a daemon that processes incoming control messages, runs out of newsfeeds, as could a news to mail gateway.

The file is interpreted as a set of lines, parsed according to the following rules: if a line ends with a backslash, the backslash, the newline, and any whitespace at the start of the next line is deleted. This is repeated until the entire "logical" line is collected. If the logical line is blank or starts with a number sign ("#"), it is ignored.

All other lines are interpreted as feed entries. An entry should consist of four colon-separated fields; two of the fields may have optional sub-fields, marked off by a slash. Fields or sub-fields that take multiple parameters should be separated by a comma. Extra whitespace can cause problems and should be avoided. Except for the site names, case is significant. The format of an entry is:

    sitename[/exclude,exclude,...]\
        :pattern,pattern...[/distribution,distribution...]\
        :flag,flag...\
        :parameter

Each field is described below.

The sitename is the name of the site to which a news article can be sent. It is used for writing log entries and for determining if an article should be forwarded to a site. (A "site" is the generic term for some destination of newsfeed data; it often corresponds to a remote news peer, but doesn't have to. For example, a local archiving program run from newsfeeds is also a "site".) If sitename already appears in the article's Path header field body, then the article will not be sent to the site. The name is usually whatever the remote site uses to identify itself in the Path header field body, but can be almost any word.

Be careful, though, to avoid having the sitename accidentally match a path identity in the Path header field unintentionally. For this reason, special local entries (such as archivers or gateways) should probably end with an exclamation point to make sure that they do not have the same name as any real site. For example, "gateway" is an obvious name for the local entry that forwards articles out to a mailing list. If a site with the name "gateway" posts an article, when the local site receives the article it will see the name in the Path header field and not send the article to its own "gateway" entry. Since "gateway!" can't appear as an individual path identity in the Path header field (as "!" is the delimiter), that would be a better thing to use for sitename.

(Another way to avoid this problem is with the "Ap" flag; see the description below.)

If an entry has an exclusion sub-field, the article will not be sent to that site if any of exclude appear in the Path header field body. (It's sometimes convenient to have the sitename be an abbreviated form of the name of the remote site, since all the sitenames to which an article is sent are written to the log and using shorter sitenames can therefore improve performance for large servers. In this case, the path identities of those sites should be given as exclude entries and the "Ap" flag used so that the abbreviated sitename doesn't accidentally match some other path identities in Path header fields.)

The same sitename can be used more than once and the appropriate action will be taken for each entry that should receive the article, but this is recommended only for program feeds to avoid confusion. Case is not significant in site names.

The comma-separated pattern specifies which groups to send to the site; it is interpreted to build a "subscription list" for the site. The default subscription is to get all groups carried by the server. It is a uwildmat pattern supporting poison ("@") wildcards; see the libinn_uwildmat man page for full details on the pattern matching language. pattern will be matched against every newsgroup carried by the server and all newsgroups that match will be added to the subscription list for the site.

Normally, a given article (or information about it) is sent to a site if any of the newsgroups to which the article was posted are in that site's subscription list. If a newsgroup matches an "@" pattern in pattern, then not only is it not added to the subscription list, but any articles crossposted to that newsgroup also will not be sent to that site even if other newsgroups to which it was crossposted are in that site's subscription list. This is called a poison pattern (because matching groups are "poisoned").

For example, to receive all comp.* groups, but only comp.sources.unix within the sources newsgroups, the following pattern can be used:

    comp.*,!comp.sources.*,comp.sources.unix

Note that the trailing ".*" is required; the pattern has to match the whole newsgroup name. "comp.sources.*" could be written "comp.sources*" and would exclude the newsgroup comp.sources (if it exists) as well as the groups in the comp.sources.* hierarchy, but note that this would also exclude a newsgroup named comp.sources-only (whereas the above pattern would add that group to the site subscription list since it matches "comp.*" and none of the other patterns).

For another example, to feed alt.* and misc.* to a given site but not any articles posted to alt.binaries.warez (even if they're also crossposted to other alt.* or misc.* groups), the following pattern can be used:

    alt.*,@alt.binaries.warez,misc.*

Note, however, that if you reversed the "alt.*" and "@alt.binaries.warez" entries, this pattern would be equivalent to "alt.*,misc.*", since the last matching pattern determines whether a given newsgroup matches and the poison logic only applies if the poison entry is the last matching entry.

Control messages follow slightly different propagation rules than normal articles; see innd(8) for the details. Note that most subscriptions should have "!junk,!control,!control.*" in their pattern list due to those propagation rules (and since "junk" is a special internal newsgroup; see wanttrash in inn.conf(5) for more details on what it's used for) and that the best way to keep control messages local to a site is with a distribution.

A subscription can be further modified by specifying distributions that the site should or should not receive. The default is to send all articles to all sites that subscribe to any of the groups where it has been posted, but if an article has a Distribution header field and any distributions are specified, then they are checked according to the following rules:

1.

If the Distribution header field matches any of the values in the sub-field, the article is sent.

2.

If a distribution starts with an exclamation point, and it matches the Distribution header field, the article is not sent.

3.

If the Distribution header field does not match any distribution in the site's entry and no negations were used, the article is not sent.

4.

If the Distribution header field does not match any distribution in the site's entry and any distribution started with an exclamation point, the article is sent.

If an article has more than one distribution specified, then each one is handled according according to the above rules. If any of the specified distributions indicate that the article should be sent, it is; if none do, it is not sent. In other words, the rules are used as a logical or.

It is almost definitely a mistake to have a single feed that specifies distributions that start with an exclamation point along with some that don't.

Distributions are text words, not patterns; entries like "*" or "all" have no special meaning.

The flag field is described in "FLAG VALUES". The interpretation of the parameter field depends on the type of feed and is explained in more detail in "FEED TYPES". It can be omitted for some types of feeds.

The site named "ME" is special. There must be exactly one such entry, and it should be the first entry in the file. If the "ME" entry has an exclusion sub-field, incoming articles are rejected completely if any of the names specified in that exclusion sub-field appear in their Path header field bodies. If the "ME" entry has a subscription list, that list is prepended to the subscription list of all other entries. For example, "*,!control,!control.*,!junk,!foo.*" could be used to set the default subscription list for all other feeds so that local postings are not propagated unless "foo.*" explicitly appears in the site's subscription list. This feature tends to be somewhat confusing since the default subscription is prepended and can be overridden by other patterns.

If the "ME" entry has a distribution sub-field, only articles that match that distribution list are accepted and all other articles with a distribution are rejected. A common use for this is to put something like "/!local" in the "ME" entry to reject local postings from other misconfigured sites. The distribution sub-field of "ME" has no effect on the acceptance or rejection of articles that do not have a Distribution header field.

An empty "ME" entry is possible, in which case no exclusion patterns will be defined.

Finally, it is also possible to set variables in newsfeeds and use them later in the file. A line starting with "$" sets a variable. For example:

    $LOCALGROUPS=local.*,example.*
    $CONTROLGROUPS=control,control.*

This sets the variable "LOCALGROUPS" to "local.*,example.*" and the variable "CONTROLGROUPS" to "control,control.*". They can later be used elsewhere in the file, such as in a site entry like:

    news.example.com:$LOCALGROUPS:Tf,Wnm:

which is then completely equivalent to:

    news.example.com:local.*,example.*:Tf,Wnm:

Variables aren't solely simple substitution. If either "!" or "@" immediately precedes the variable and the value of the variable contains commas, that character will be duplicated before each comma. This somewhat odd-sounding behavior is designed to make it easier to use variables to construct feed patterns. The utility becomes more obvious when you observe that the line:

    news.example.net:*,@$LOCALGROUPS,!$CONTROLGROUPS:Tf,Wnm:

is therefore equivalent to:

    news.example.net:*,@local.*,@example.*,!control,!control.*:Tf,Wnm:

which (as explained below) excludes all of the groups in $LOCALGROUPS and unwanted control articles from the feed to that site.

FLAG VALUES

The flags parameter specifies miscellaneous parameters, including the type of feed, what information should be sent to it, and various limitations on what articles should be sent to a site. They may be specified in any order and should be separated by commas. Flags that take values should have the value immediately after the flag letter with no whitespace. The valid flags are:

< size

An article will only be sent to this site if it is less than size bytes long. The default is no limit.

> size

An article will only be sent to this site if it is greater than size bytes long. The default is no limit.

A checks

An article will only be sent to this site if it meets the requirements specified in checks, which should be chosen from the following set. checks can be multiple letters if appropriate. Note that this flag is not effective on funnel targets; it has to be used on every funnel entry (for instance, Af is not effective on the innfeed! funnel target and therefore has to be specified on every funnelled news site).

B high/low

If a site is being fed by a file, channel, or exploder (see below), the server will normally start trying to write the information as soon as possible. Providing a buffer may give better system performance and help smooth out overall load if a large batch of news comes in. The value of this flag should be two numbers separated by a slash. high specifies the point at which the server can start draining the feed's I/O buffer, and low specifies when to stop writing and begin buffering again; the units are bytes. The default is to do no buffering, sending output as soon as it is possible to do so.

C count

If this flag is specified, an article will only be sent to this site if the number of groups it is posted to, plus the square of the number of groups followups would appear in, is no more than count. 30 is a good value for this flag, allowing for instance:

F name

Specifies the name of the file that should be used if it's necessary to begin spooling for the site (see below). If name is not an absolute path, it is taken to be relative to pathoutgoing in inn.conf. If name is a directory, the file togo in that directory will be used as the file name.

G count

If this flag is specified, an article will only be sent to this site if it is posted to no more than count newsgroups. This has the problem of filtering out many FAQs, as well as newsgroup creation postings and similar administrative announcements. Either the C flag or the U flag is a better solution.

H count

If this flag is specified, an article will only be sent to this site if it has count or fewer sites in its Path header field body. This flag should only be used as a rough guide because of the loose interpretation of the Path header field; some sites put the poster's name in the header field body, and some sites that might logically be considered to be one hop become two because they put the posting workstation's name in the header field body. The default value for count if not specified is one. (Also see the O flag, which is sometimes more appropriate for some uses of this flag.)

I size

The flag specifies the size of the internal buffer for a file feed. If there are more file feeds than allowed by the system, they will be buffered internally in least-recently-used order. If the internal buffer grows bigger than size bytes, however, the data will be written out to the appropriate file. The default value is 16 KB.

N status

Restricts the articles sent to this site to those in newsgroups with the moderation status given by status. If status is "m", only articles in moderated groups are sent; if status is "u", only articles in unmoderated groups are sent.

O originator

If this flag is specified, an article will only be sent to this site if it contains an Injection-Info header field (or an X-Trace header field if no Injection-Info header field is found) and the first field of this header field matches originator. originator is a uwildmat expression without commas or a list of such expressions, separated by "/". The article is never sent if the first character of the pattern begins with "@" and the rest of the pattern matches. One use of this flag is to restrict the feed to locally generated posts by using an originator pattern that matches the Injection-Info header field added by the local server.

P priority

The nice priority that this channel or program feed should receive. This should be a positive number between 0 and 20 and is the priority that the new process will run with. This flag can be used to raise the priority to normal if you're using the nicekids parameter in inn.conf.

Q hashfeed

Specifies the hashfeed match expression for this site. It must be in the form "value/mod" or "start-end/mod". The Message-ID of the article is hashed using MD5, which results in a 128-bit hash. The lowest 32 bits are then taken by default as the hashfeed value (which is an integer). If the hashfeed value modulus "mod" plus one equals "value" or is between "start" and "end", the article will be fed to this site. All these numbers must be integers.

It is a deterministic way to control the flow of articles and to split a feed. For instance:

    Q1/2      Feeds about 50% of all articles to this site.
    Q2/2      Feeds the other 50% of all articles.
    

Another example with three sites:

    Q1-3/10   Feeds about 30% of all articles.
    Q4-5/10   Feeds about 20% of all articles.
    Q6-10/10  Feeds about 50% of all articles.
    

If this flag is specified multiple times, the contents will be logically "OR"ed together (just one match is needed).

You can use an extended syntax of the form "value/mod_offset" or "start-end/mod_offset". As MD5 generates a 128-bit return value, it is possible to specify from which byte-offset the 32-bit integer used by hashfeed starts. The default value for "offset" is "_0" and thirteen overlapping values from "_0" to "_12" can be used. Only up to four totally independent values exist: "_0", "_4", "_8" and "_12".

Therefore, it allows generating a second level of deterministic distribution. Indeed, if a news server is fed "Q1/2", it can go on splitting thanks to "Q1-3/9_4" for instance. Up to four levels of deterministic distribution can be used.

The algorithm is compatible with the one used by Diablo 5.1 and up. If you want to use the legacy quickhashing method used by Diablo before 5.1, you can put an "@" sign just after the Q flag (for instance "Q@1-3/10", but the distribution of the messages is not perfect with this legacy method whose use is discouraged and for which offsets cannot be used).

S size

If the amount of data queued for the site gets to be larger than size bytes, the server will switch to spooling, appending to a file specified by the F flag, or pathoutgoing/sitename if F is not specified. Spooling usually happens only for channel or exploder feeds, when the spawned program isn't keeping up with its input.

T type

This flag specifies the type of feed for this site. type should be a letter chosen from the following set:

    c        Channel
    f        File
    l        Log entry only
    m        Funnel (multiple entries feed into one)
    p        Program
    x        Exploder
    

Each feed is described below in "FEED TYPES". The default is Tf, for a file feed.

U count

If this flag is specified, an article will only be sent to this site if followups to this article would be posted to no more than count newsgroups. (Also see C for a more complex way of handling this.)

W items

For a file, channel, or exploder feed, this flag controls what information will be sent to this site. For a program feed, only the asterisk ("*") has any effect. items should be chosen from the following set:

FEED TYPES

innd provides four basic types of feeds: log, file, program, and channel. An exploder is a special type of channel. In addition, several entries can feed into the same feed; these are funnel feeds, which refer to an entry that is one of the other types. Funnel feeds are partially described above with the description of the W* flag. A funnel feed gets every article that would be sent to any of the feeds that funnel into it and normally include the W* flag in their flags so that the program processing that feed knows which sites received which articles. The most common funnel feed is innfeed(8).

Note that the term "feed" is technically a misnomer, since the server doesn't transfer articles itself and only writes data to a file, program, or log telling another program to transfer the articles.

The simplest feed is a log feed (Tl). Other than a mention in the news log file, pathlog/news, no data is written out. This is equivalent to a Tf entry writing to /dev/null, except that no file is ever opened. Flushing a log feed does nothing.

A file feed (Tf) is the next simplest type of feed. When the site should receive an article, the specified data is written out to the file named by the parameter field. If parameter is not an absolute path, it is taken to be relative to pathoutgoing in inn.conf. If parameter is not given, it defaults to pathoutgoing/sitename. The file name should be unique (two file feeds should not ever point to the same file).

File feeds are designed for use by external programs that periodically process the written data. To cooperate with innd properly, such external programs should first rename the batch file and then send a flush command for that site to innd using ctlinnd(8). innd will then write out any buffered data, close the file, and reopen it (under the original name), and the program can process the data in the renamed file at its leisure. File feeds are most frequently used in combination with nntpsend(8).

A program feed (Tp) spawns a given program for every article that the site receives. The parameter field must be the command line to execute, and should contain one instance of %s, which will be replaced by the storage API token of the article (the actual article can be retrieved by the program using sm(1)). The program will not receive anything on standard input (unlike earlier versions of INN, where the article is sent to the program on stdin), and standard output and error from the program will be set to the error log (pathlog/errlog). innd will try to avoid spawning a shell if the command has no shell meta-characters; this feature can be defeated if necessary for some reason by appending a semi-colon to the end of the command. The full path name of the program to be run must be specified unless the command will be run by the shell (and it is strongly recommended that the full path name always be specified regardless).

If a program feed is the target of a funnel, and if W* appears in the flags of the site, a single asterisk may be present in the parameter and will be replaced by a space-separated list of names of the sites feeding into the funnel which received the relevant article. If the site is not the target of a funnel, or if the W* flag is not used, the asterisk has no special meaning.

Flushing a program feed does nothing.

For a channel (Tc) or exploder (Tx) feed, the parameter field again names the process to start. As with program feeds, the full path to the program must be specified. However, rather than spawning the program for every article, it is spawned once and then whenever the site receives an article, the data specified by the site flags is written to the standard input of the spawned program. Standard output and error are set as with program feeds. If the process exits, it will be restarted automatically. If the process cannot be started, the server will spool input to a file named pathoutgoing/sitename and will try to start the process again later.

When a channel or exploder feed is flushed, the server closes its end of the pipe to the program's standard input. Any pending data that has not been written will be spooled; see the description of the S flag above. The server will then spawn a new instance of the program. No signal is sent to the program; it is up to the program handling a channel or exploder feed to notice end of file on its standard input and exit appropriately.

Exploders are a special type of channel feed. In addition to the channel feed behavior described above, exploders can also be sent command lines. These lines start with an exclamation point and their interpretation is up to the exploder. The following commands are generated automatically by the server:

    !newgroup group
    !rmgroup group
    !flush
    !flush site

These commands are sent whenever the ctlinnd(8) command of the same name is received by the server. In addition, the ctlinnd(8) "send" command can be used to send an arbitrary command line to an exploder. The primary exploder is buffchan(8).

Finally, Tm feeds are the input to a funnel. The parameter field of the site should name the site handling articles for all of the funnel inputs.

EXAMPLES

The syntax of the newsfeeds file is so complex because you can specify a staggering variety of feeds. INN is capable of interacting with a wide variety of programs that do various things with news articles. Far and away the most common two entries in newsfeeds, however, are file feeds for nntpsend(8) and funnel feeds for innfeed(8).

The former look like this:

    feed.example.com:*,!control,!control.*,!junk:Tf,Wnm:

which generates a file named pathoutgoing/feed.example.com containing one line per article consisting of the storage API token, a space, and the message ID.

The latter look like this:

    feed.example.com:*,!control,!control.*,!junk:Tm:innfeed!

Very similar, except that this is the input to a funnel feed named "innfeed!". One could also write this as:

    example/feed.example.com:*,!control,!control.*,!junk:Ap,Tm:innfeed!

(note the Ap so that articles that contain just "example" in the Path header field body will still be sent), which is completely equivalent except that this will be logged in pathlog/news as going to the site "example" rather than "feed.example.com".

The typical feed entry for innfeed(8) is a good example of a channel feed that's the target of various funnel feeds (make sure the path to innfeed is properly set):

    innfeed!:!*:Tc,Wnm*:<pathbin in inn.conf>/innfeed -y

Note that the pattern for this feed is just "!*" so that it won't receive any articles directly. The feed should only receive those articles that would go to one of the funnel feeds that are feeding into it. innfeed(8) will receive one line per article on its standard input containing the storage API token, the message ID, and a space-separated list of sites that should receive that article.

Here's a more esoteric example of a channel feed:

    watcher!:*:Tc,Wbnm\
        :exec awk '$1 > 1000000 { print "BIG", $2, $3 }' \
         >> <pathlog>/bigart

This receives the byte size of each article along with the storage API token and Message-ID, and appends to a bigart log file a line for every article that's over a million bytes. This is actually rather a strange way to write this since INN can do the size check itself; the following is equivalent:

    watcher!:*:Tc,>1000000,Wbnm\
        :exec awk '{ print "BIG", $2, $3}' >> <pathlog>/bigart

Here's a cute, really simple news to mail gateway that also serves as an example of a fairly fancy program feed:

    mailer!:!*:W*,Tp\
        :sm %s | innmail -s "News article" *

Remember that %s is replaced by the storage API token, so this retrieves the article and pipes it into innmail (which is safer than programs like Mail(1) because it doesn't parse the body for tilde commands) with a given subject line. Note the use of "*" in the command line and W* in the flags; this entry is designed to be used as the target of funnel feeds such as:

    peter@example.com:news.software.nntp:Tm:mailer!
    sue@example.com:news.admin.misc:Tm:mailer!

Suppose that the server receives an article crossposted between news.admin.misc and news.software.nntp. The server will notice that the article should be sent to the site "peter@example.com" and the site "bob@example.com", both of which funnel into "mailer!", so it will look at the "mailer!" site and end up executing the command line:

    sm @...@ | innmail -s "News article" \
        peter@example.com sue@example.com

which will mail the article to both Peter and Sue.

Finally, another very useful example of a channel feed: the standard entry for controlchan(8). Make sure its path is properly set.

    controlchan!\
        :!*,control,control.*,!control.cancel\
        :AC,Tc,Wnsm:<pathbin in inn.conf>/controlchan

This program only wants information about articles posted to a control newsgroup other than control.cancel, which due to the sorting of control messages described in innd(8) will send it all control messages except for cancel messages. controlchan gets the storage API token, the name of the sending site (for processing old-style ihave and sendme control messages, be sure to read about logipaddr in controlchan(8)), and the message ID for each article.

For many other examples, including examples of the special "ME" site entry, see the example newsfeeds file distributed with INN. Also see the install documentation that comes with INN for information about setting up the standard newsfeeds entries used by most sites.

HISTORY

Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews. Reformatted and rewritten in POD by Russ Allbery <eagle@eyrie.org>.

SEE ALSO

active(5), buffchan(8), controlchan(8), ctlinnd(8), inn.conf(5), innd(8), innfeed(8), innxmit(8), libinn_uwildmat(3), nntpsend(8).