innfeed.conf - Configuration file for innfeed
The configuration file innfeed.conf in pathetc is
used to control the innfeed(8) program. It is a fairly free-format
file that consists of three types of entries: key:value,
peer and group. Comments are from the hash character
"#" to the end of the line.
key:value entries are a keyword and a value
separated by a colon (which can itself be surrounded by whitespace). For
example:
max-connections: 10
A legal key starts with a letter and contains only letters,
digits, and the "_" and
"-" characters. There are 5 different
types of values: integers, floating-point numbers, characters, booleans, and
strings.
Integer and floating-point numbers are as to be expected, except
that exponents in floating-point numbers are not supported. A boolean value
is either "true" or
"false" (case is not significant). A
character value is a single-quoted character as defined by the C-language. A
string value is any other sequence of characters. If the string needs to
contain whitespace, then it must be quoted with double quotes, and uses the
same format for embedding non-printing characters as normal C-language
string.
Peer entries look like:
peer <name> {
# body ...
}
The word "peer" is required. The
<name> is the same as the site name in INN's newsfeeds
configuration file. The body of a peer entry contains some number (possibly
zero) of key:value entries.
Group entries look like:
group <name> {
# body ...
}
The word "group" is required.
The <name> is any string valid as a key. The body of a group
entry contains any number of the three types of entries. So
key:value pairs can be defined inside a group, and peers can
be nested inside a group, and other groups can be nested inside a group.
key:value entries that are defined outside of all
peer and group entries are said to be at "global
scope". There are global key:value entries that apply to
the process as a whole (for example the location of the backlog file
directory), and there are global key:value entries that act as
defaults for peers. When innfeed looks for a specific value in a peer
entry (for example, the maximum number of connections to set up), if the
value is not defined in the peer entry, then the enclosing groups are
examined for the entry (starting at the closest enclosing group). If there
are no enclosing groups, or the enclosing groups do not define the
key:value, then the value at global scope is used.
A small example could be:
# Global value applied to all peers that have
# no value of their own.
max-connections: 5
# A peer definition. "uunet" is the name used by innd
# in the newsfeeds configuration file.
peer uunet {
ip-name: usenet1.uu.net
}
peer vixie {
ip-name: gw.home.vix.com
max-connections: 10 # Override global value.
}
# A group of two peers which can handle more connections
# than normal.
group fast-sites {
max-connections: 15
# Another peer. The "max-connections" value from the
# "fast-sites" group scope is used. The "ip-name" value
# defaults to the peer's name.
peer data.ramona.vix.com {
}
peer bb.home.vix.com {
max-connections: 20 # He can really cook.
}
}
Given the above configuration file, the defined peers would have
the following values for the max-connections key:
uunet 5
vixie 10
data.ramona.vix.com 15
bb.home.vix.com 20
innfeed ignores key:value pairs it is not
interested in. Some configuration file values can be set via a command-line
option, in which case that setting overrides the settings in the file.
Configuration files can be included in other configuration files
via the syntax:
$INCLUDE filename
There is a maximum nesting depth of 10.
For a fuller example configuration file, see the supplied
innfeed.conf.
The following listing show all the keys that apply to the process
as whole. These are not required (compiled-in defaults are used where
needed).
- news-spool
- This key requires a pathname value and defaults to patharticles in
inn.conf. It specifies where the top of the article spool is. This
corresponds to the -a command-line option.
- input-file
- This key requires a pathname value. It specifies the pathname (relative to
the backlog-directory value) that should be read in funnel-file
mode. This corresponds to giving a filename as an argument on the
command-line (i.e. its presence also implies that funnel-file mode should
be used).
The default is unset; innfeed then runs in channel or
batch mode.
- pid-file
- This key requires a pathname value and defaults to innfeed.pid. It
specifies the pathname (relative to pathrun in inn.conf)
where the pid of the innfeed process should be stored. This
corresponds to the -p command-line option.
- debug-level
- This key defines the debug level for the process. Default is
0. A non-zero number generates a lot of messages
to stderr, or to the config-defined log-file. This corresponds to
the -d command-line option.
If a file named innfeed.debug exists in the
pathlog directory (as set in inn.conf), then
debug-level is automatically set to 1.
This is a cheap way of avoiding continual reloading of the
newsfeeds file when debugging. Note that debug messages still go
to log-file.
- debug-shrinking
- This key requires a boolean value and defaults to false (the debug file is
allowed to grow without bound). If set to true, this file is truncated
when its size reaches a certain limit. See backlog-limit for more
details.
- initial-sleep
- This key requires a positive integer. The default value is
2. It defines the number of seconds to wait when
innfeed (or a fork) starts, before beginning to open connections to
remote hosts.
- fast-exit
- This key requires a boolean value and defaults to false. If set to true,
when innfeed receives a SIGTERM or SIGQUIT signal, it will close
its listeners as soon as it can, even if it means dropping articles.
- use-mmap
- This key requires a boolean value and defaults to true. When
innfeed is given file names to send (a fairly rare use case)
instead of storage API tokens, it specifies whether mmaping should be used
if innfeed has been built with mmap(2) support. If article
data on disk is not in NNTP-ready format (CR/LF at the end of each line),
then after mmaping, the article is read into memory and fixed up, so
mmaping has no positive effect (and possibly some negative effect
depending on your system), and so in such a case this value should be
"false", which corresponds to the
-M command-line option.
- log-file
- This key requires a pathname value and defaults to innfeed.log. It
specifies where any logging messages that could not be sent via
syslog(3) should go (such as those generated when a positive value
for debug-value is used). This corresponds to the -l
command-line option.
This pathname is relative to pathlog in
inn.conf.
- log-time-format
- This key requires a format string suitable for strftime(3). It is
used for messages sent via syslog(3) and to the status-file.
Default value is "%a %b %d %H:%M:%S
%Y".
- backlog-directory
- This key requires a pathname value and defaults to innfeed. It
specifies where the current innfeed process should store backlog
files. This corresponds to the -b command-line option.
This pathname is relative to pathspool in
inn.conf.
- backlog-highwater
- This key requires a positive integer value and defaults to
5. It specifies how many articles should be kept
on the backlog file queue before starting to write new entries to
disk.
- backlog-ckpt-period
- This key requires a positive integer value and defaults to
30. It specifies how many seconds elapse between
checkpoints of the input backlog file. Too small a number will mean
frequent disk accesses; too large a number will mean after a crash,
innfeed will re-offer more already-processed articles than
necessary.
- backlog-newfile-period
- This key requires a positive integer value and defaults to
600. It specifies how many seconds elapse before
each check for externally generated backlog files that are to be picked up
and processed.
- backlog-rotate-period
- This key requires a positive integer value and defaults to
60. It specifies how many seconds elapse before
innfeed checks for a manually created backlog file and moves the
output backlog file to the input backlog file.
- dns-retry
- This key requires a positive integer value and defaults to
900. It defines the number of seconds between
attempts to re-lookup host information that previously failed to be
resolved.
- dns-expire
- This key requires a positive integer value and defaults to
86400. It defines the number of seconds between
refreshes of name to address DNS translation. This is so long-running
processes do not get stuck with stale data, should peer IP addresses
change.
- gen-html
- This key requires a boolean value and defaults to false. It specifies
whether the status-file should be HTML-ified.
- status-file
- This key requires a pathname value and defaults to innfeed.status.
An absolute pathname can be used. It specifies the pathname (relative to
pathhttp when gen-html is true; otherwise, pathlog as
set in inn.conf) where the periodic status of the innfeed
process should be stored. This corresponds to the -S command-line
option.
- connection-stats
- This key requires a boolean value and defaults to false. If the value is
true, then whenever the transmission statistics for a peer are logged,
each active connection logs its own statistics. This corresponds to the
-z command-line option.
- host-queue-highwater
- This key requires a positive integer value and defaults to
10. It defines how many articles will be held
internally for a peer before new arrivals cause article information to be
spooled to the backlog file.
- stats-period
- This key requires a positive integer value and defaults to
600. It defines how many seconds innfeed
waits between generating statistics on transfer rates.
- stats-reset
- This key requires a positive integer value and defaults to
43200. It defines how many seconds innfeed
waits before resetting all internal transfer counters back to zero (after
logging one final time). This is so a innfeed process running more
than a day will generate "final" stats that will be picked up by
logfile processing scripts.
- initial-reconnect-time
- This key requires a positive integer value and defaults to
30. It defines how many seconds to first wait
before retrying to reconnect after a connection failure. If the next
attempt fails too, then the reconnect time is approximately doubled until
the connection succeeds, or max-reconnect-time is reached.
- max-reconnect-time
- This key requires an integer value and defaults to
3600. It defines the maximum number of seconds to
wait between attempt to reconnect to a peer. The initial value for
reconnection attempts is defined by initial-reconnect-time, and it
is doubled after each failure, up to this value.
- stdio-fdmax
- This key requires a non-negative integer value and defaults to
0. If the value is greater than zero, then
whenever a network socket file descriptor is created and it has a value
less than this, the file descriptor will be dup'ed to bring the
value up greater than this. This is to leave lower numbered file
descriptors free for stdio. Certain systems, Sun's in particular, require
this. SunOS 4.1.x usually requires a value of 128 and Solaris requires a
value of 256. The default if this is not specified, is
0.
The following keys are used with imapfeed to authenticate
to a remote host. Several parameters may be included at global scope:
- deliver-authname
- The authname is who you want to authenticate as.
- deliver-password
- This is the appropriate password for authname.
- deliver-username
- The username is who you want to "act" as, that is, who is
actually going to be using the server.
- deliver-realm
- In this case, the "realm" is the realm in which the specified
authname is valid. Currently this is only needed by the DIGEST-MD5 SASL
mechanism.
- deliver-rcpt-to
- A printf(3)-style format string for creating the envelope recipient
address. The pattern MUST include a single string specifier which will be
replaced with the newgroup (e.g.
"bb+%s"). The default is
"+%s".
- An optional printf(3)-style format string for creating a To: header
field to be prepended to the article. The pattern MUST include a single
string specifier which will be replaced with the newgroup (e.g.
"post+%s@domain"). If not specified, the
To: header field will not be prepended.
All the key:value pairs mentioned in this section
can be specified at global scope. They may also be specified inside a group
or peer definition. Note that when peers are added dynamically (i.e. when
innfeed receives an article for an unspecified peer), it will add the
peer site using the parameters specified at global scope.
No keys are currently required. They all have a default value, if
not present in the configuration file.
The following keys are optional:
- article-timeout
- This key requires a non-negative integer value. The default value is
600. If no articles need to be sent to the peer
for this many seconds, then the peer is considered idle and all its active
connections are torn down.
- response-timeout
- This key requires a non-negative integer value. The default value is
300. It defines the maximum amount of time to wait
for a response from the peer after issuing a command.
- initial-connections
- This key requires a non-negative integer value. The default value is
1. It defines the number of connections to be
opened immediately when setting up a peer binding. A value of
0 means no connections will be created until an
article needs to be sent.
- max-connections
- This key requires a positive integer value. The default value is
2 but may be increased if needed or for large
feeds. It defines the maximum number of connections to run in parallel to
the peer. A value of 0 specifies an unlimited
number of maximum connections. In general, use of an unlimited number of
maximum connections is not recommended. Do not ever set
max-connections to zero with dynamic-method 0 set, as this
will saturate peer hosts with connections.
- close-period
- This key requires a positive integer value and defaults to
86400. It is the maximum number of seconds a
connection should be kept open. Some NNTP servers do not deal well with
connections being held open for long periods.
- dynamic-method
- This key requires an integer value between 0 and 3. The default value is
3. It controls how connections are opened, up to
the maximum specified by max-connections. In general (and
specifically, with dynamic-method 0), a new connection is opened
when the current number of connections is below max-connections,
and an article is to be sent while no current connections are idle.
Without further restraint (i.e. using dynamic-method 0), in
practice this means that max-connections connections are
established while articles are being sent. Use of other
dynamic-method settings imposes a further limit on the amount of
connections opened below that specified by max-connections. This
limit is calculated in different ways, depending of the value of
dynamic-method.
Users should note that adding additional connections is not
always productive -- just because opening twice as many
connections results in a small percentage increase of articles accepted
by the remote peer, this may be at considerable resource cost both
locally and at the remote site, whereas the remote site might well have
received the extra articles sent from another peer a fraction of a
second later. Opening large numbers of connections is considered
antisocial.
The meanings of the various settings are:
- 0 (no method)
- Increase of connections up to max-connections is unrestrained.
- 1 (maximize articles per second)
- Connections are increased (up to max-connections) and decreased so
as to maximize the number of articles per second sent, while using the
fewest connections to do this.
- 2 (set target queue length)
- Connections are increased (up to max-connections) and decreased so
as to keep the queue of articles to be sent within the bounds set by
dynamic-backlog-low and dynamic-backlog-high, while using
the minimum resources possible. As the queue will tend to fill if the site
is not keeping up, this method ensures that the maximum number of articles
are offered to the peer while using the minimum number of connections to
achieve this.
- 3 (combination)
- This method uses a combination of methods 1 and 2 above. For sites
accepting a large percentage of articles, method 2 will be used to ensure
these sites are offered as complete a feed as possible. For sites
accepting a small percentage of articles, method 1 is used, to minimize
remote resource usage. For intermediate sites, an appropriate combination
is used.
- dynamic-backlog-low
- This key requires a floating-point value between 0 and 100. It represents
(as a percentage) the low water mark for the host queue. If the host queue
falls below this level while using dynamic-method 2 or 3, and if 2
or more connections are open, innfeed will attempt to drop
connections to the host. An Infinite Impulse Response (IIR) filter is
applied to the value to prevent connection flap (see
dynamic-filter). The default value is 20.0.
This value must be smaller than dynamic-backlog-high.
- dynamic-backlog-high
- This key requires a floating-point value between 0 and 100. It represents
(as a percentage) the high water mark for the host queue. If the host
queue rises above this level while using dynamic-method 2 or 3, and
if less than max-connections are open to the host, innfeed
will attempt to open further connections to the host. An Infinite Impulse
Response (IIR) filter is applied to the value to prevent connection flap
(see dynamic-filter). The default value is
50.0. This value must be larger than
dynamic-backlog-low.
- dynamic-backlog-filter
- This key requires a floating-point value between 0 and 1. It represents
the filter coefficient used by the Infinite Impulse Response (IIR) filter
used to implement dynamic-method 2 and 3. The default value of this
filter is 0.7, giving a time constant of 1/(1-0.7)
articles. Higher values will result in slower response to queue fullness
changes; lower values in faster response.
- max-queue-size
- This key requires a positive integer value. The default value is
20. It defines the maximum number of articles to
process at one time when using streaming to transmit to a peer. Larger
numbers mean more memory consumed as articles usually get pulled into
memory (see the description of use-mmap).
- streaming
- This key requires a boolean value. Its default value is true. It defines
whether streaming commands are used to transmit articles to the
peers.
- no-check-high
- This key requires a floating-point number which must be in the range [0.0,
100.0]. When running transmitting with the streaming commands,
innfeed attempts an optimization called "no-CHECK mode".
This involves not asking the peer if it wants the article, but just
sending it. This optimization occurs when the percentage of the articles
the peer has accepted gets larger than this number. If this value is set
to 100.0, then this effectively turns off no-CHECK
mode, as the percentage can never get above 100.0. If this value is too
small, then the number of articles the peer rejects will get bigger (and
your bandwidth will be wasted). The default value of
95.0 usually works pretty well.
- no-check-low
- This key requires a floating-point number which must be in the range [0.0,
100.0], and it must be smaller that the value for no-check-high.
When running in no-CHECK mode, as described above, if the percentage of
articles the remote server accepts drops below this number, then the
no-CHECK optimization is turned off until the percentage gets above the
no-check-high value again. If there is small difference between
this and the no-check-high value (less than about 5.0), then
innfeed may frequently go in and out of no-CHECK mode. If the
difference is too big, then it will make it harder to get out of no-CHECK
mode when necessary (wasting bandwidth). Keeping this to between 5.0 and
10.0 less than no-check-high usually works pretty well. The default
value is 90.0.
- no-check-filter
- This is a floating-point value representing the time constant, in
articles, over which the CHECK/no-CHECK calculations are done. The default
value is 50.0, which will implement an Infinite
Impulse Response (IIR) filter of time constant 50. This roughly equates to
making a decision about the mode over the previous 50 articles. A higher
number will result in a slower response to changing percentages of
articles accepted; a lower number will result in a faster response.
- port-number
- This key requires a positive integer value. It defines the TCP/IP port
number to use when connecting to the remote. Usually, port number 119 is
used, which is the default value.
- force-ipv4
- This key requires a boolean value. By default, it is set to false. Setting
it to true is the same as setting bindaddress6 to
"none" and removing bindaddress
from "none" if it was set.
- drop-deferred
- This key requires a boolean value. By default, it is set to false. When
set to true, and a peer replies with code 431 or 436 (try again later),
innfeed just drops the article and does not try to re-send it. This
is useful for some peers that keep on deferring articles for a long time
to prevent innfeed from trying to offer the same article over and
over again.
- min-queue-connection
- This key requires a boolean value. By default, it is set to false. When
set to true, innfeed will attempt to use a connection with the
least queue size (or the first empty connection). If this key is set to
true, it is recommended that dynamic-method be set to
0. This allows for article propagation with the
least delay.
- no-backlog
- This key requires a boolean value. It specifies whether spooling should be
enabled (false, the default) or disabled (true). Note that when
no-backlog is set, articles reported as spooled are actually
silently discarded.
- backlog-limit
- This key requires a non-negative integer value. If the number is
0 (the default), then backlog files are allowed to
grow without bound when the peer is unable to keep up with the article
flow. If this number is greater than 0, then it specifies the size (in
bytes) the backlog file should get truncated to when the backlog file
reaches a certain limit. The limit depends on whether
backlog-factor or backlog-limit-highwater is used.
This parameter also applies to the debug file when
debug-shrinking is set to true, and has the same effect on this
file as the one has on backlog files.
- backlog-factor
- This key requires a floating-point value, which must be larger than 1.0.
It is used in conjunction with the peer key backlog-limit. If
backlog-limit has a value greater than zero, then when the backlog
file gets larger than the value backlog-limit *
backlog-factor, then the backlog file will be truncated to the size
backlog-limit.
For example, if backlog-limit has a value of
1000000, and backlog-factor has a value
of 2.0, then when the backlog file gets to be
larger than 2000000 bytes in size, it will be truncated to 1000000
bytes. The front portion of the file is removed, and the trimming
happens on line boundaries, so the final size may be a bit less than
this number. If backlog-limit-highwater is defined too, then
backlog-factor takes precedence. The default value of
backlog-factor is 1.1.
This parameter also applies to the debug file when
debug-shrinking is set to true, and has the same effect on this
file as the one has on backlog files.
- backlog-limit-highwater
- This key requires a positive integer value that must be larger than the
value for backlog-limit. The default value is
0.
If the size of the backlog file gets larger than this value
(in bytes), then the backlog file will be shrunk down to the size of
backlog-limit. If both backlog-factor and
backlog-limit-highwater are defined, then the value of
backlog-factor is used.
This parameter also applies to the debug file when
debug-shrinking is set to true, and has the same effect on this
file as the one has on backlog files.
- backlog-feed-first
- This key requires a boolean value. By default it is set to false. When set
to true, the backlog is fed before new files. This is intended to enforce
in-order delivery, so setting this to true when initial-connections
or max-connections is more than 1 is inconsistent.
- bindaddress
- This key requires a string value. It specifies which outgoing IPv4 address
innfeed should bind the local end of its connection to. It must be
an IPv4 address in dotted-quad format (nnn.nnn.nnn.nnn),
"any", or
"none". If not set or set to
"any", innfeed defaults to
letting the kernel choose this address. If set to
"none", innfeed will not use IPv4
for outgoing connections to peers in this scope (i.e. it forces IPv6).
If not set in innfeed.conf, innfeed defaults to
the value of sourceaddress from inn.conf (which by default
is unset).
- bindaddress6
- This key requires a string value. It behaves like bindaddress
except for outgoing IPv6 connections. It must be in numeric IPv6 format
(note that a value containing colons must be enclosed in double quotes),
"any", or
"none". If set to
"none", innfeed will not use IPv6
for outgoing connections to peers in this scope.
If not set in innfeed.conf, innfeed defaults to
the value of sourceaddress6 from inn.conf (which by
default is unset).
- username
- This key requires a string value. If the value is defined, then
innfeed tries to authenticate by AUTHINFO USER and this value used
for user name. password must also be defined, if this key is
defined.
- password
- This key requires a string value. The value is the password used for
AUTHINFO PASS. username must also be defined, if this key is
defined.
As previously explained, the peer definitions can contain
redefinitions of any of the key:value pairs described in the
section about global peer defaults above. There is one
key:value pair that is specific to a peer definition.
- ip-name
- This key requires a word value. The word is either one of the host's
FQDNs, or the dotted-quad IP address of the peer for IPv4, or the
colon-separated IP address of the peer for IPv6. If this value is not
specified, then the name of the peer in the enclosing peer block is
taken to also be its ip-name.
If innfeed gets a SIGHUP signal, then it will reread the
configuration file. All values at global scope except for
backlog-directory can be changed (although note that
bindaddress and bindaddress6 changes will only affect new
connections).
Any new peers are added and any missing peers have their
connections closed.
The log file is also reopened.
For a comprehensive example, see the sample innfeed.conf
distributed with INN and installed as a starting point.
Here are examples of how to format values:
eg-string: "New\tconfig\tfile\n"
eg-long-string: "A long string that goes
over multiple lines. The
newline is kept in the
string except when quoted
with a backslash \
as here."
eg-simple-string: A-no-quote-string
eg-integer: 10
eg-boolean: true
eg-char: 'a'
eg-ctrl-g: '\007'
Written by James Brister <brister@vix.com> for InterNetNews.
Converted to POD by Julien Elie.
Earlier versions of innfeed (up to 0.10.1) were shipped
separately; innfeed is now part of INN and shares the same version
number. Please note that the innfeed.conf format has changed
dramatically since version 0.9.3.
$Id: innfeed.conf.pod 10179 2017-09-18
20:13:48Z iulius $