This FAQ assumes that the reader is familiar with the basic operation of Cyclone and has had a chance to install and setup a news server running Cyclone. It is not intended to replace the documentation, but instead, is intended to be used as a supplement and provide answers to common and advanced questions.
In this version, this FAQ mainly deals with questions about Cyclone. While some sections exist for Typhoon and Breeze, it is not the intended focus of the FAQ at the current time. As these products mature, the FAQ will be extended to directly cover these products. Until then, sections for Typhoon and Breeze are explicitly labeled as such.
This FAQ also assumes that you are running the latest version of Cyclone. Features and tricks documented in this FAQ may not work for older versions.
Any comments or questions for the FAQ are welcome. Please direct correspondence to <faq@highwind.com>
Thanks to the following people for assistance during the creation of
this FAQ:
"HighWind Software", "Cyclone NewsRouter", "Typhoon NewsServer", and "Breeze NewsServer" are trademarks of HighWind Software, Inc.
The following list is a list of questions that future versions of the FAQ will answer. If you have any suggestions for these questions or other questions, please mail them to <faq@highwind.com>
An important consideration to take into account when purchasing hardware to run Cyclone is that most people purchase a much faster CPU and much disk space than they need. It is often better to spend more money on network resources and memory, and possibly a few smaller fast disks. Cyclone is quite different than other news systems since it is extremely fast. Scalability may not even be an issue for most sites. For servers with less than 20 feeds, almost any hardware will be adequate.
The following assumes a machine which is handing both incoming and outgoing full feeds. Important benchmarks for a Cyclone system are the number of feed objects and the number of connections a server can handle.
Specialized servers such as aggregating servers (which handle lots of incoming feeds but relatively few outgoing feeds), exploding servers (which handle only a few incoming feeds but lots of outgoing feeds), and servers which are handling less than full feeds will take up less resources than servers handling both incoming and outgoing full feeds. Current statistics for a full Usenet feed are listed below (8.1).
To measure the approximate number of connections on a Cyclone server,
use:
> netstat -n | egrep 'nntp|119' | grep -c ESTABLISHED
The number of feeds is considered to be the number of feed objects. Often, full outgoing feeds will be split into multiple feed objects. It is reported when the server starts up via syslog, or as the text response to running the validate program.
CPU power is not usually of critical importance. Memory, disk, and network usually become system bottlenecks before a Cyclone server becomes CPU bound. Note that UltraSPARC based systems have a better bus and I/O systems than older SS5s or SS20s. Using filtering, subscription directives, active files and other features will all radically affect CPU usage.
In general, a SS20, low end Ultra, or fast Pentium will scale up to about 100 feed objects. Beyond that you will want an Ultra 2, Pentium Pro, and multiple machines. Although you should get an adequate CPU, bigger issues in terms of raw capacity are the I/O issues.
There are known problems with certain Pentium configurations. Certain SCSI controllers have problems dealing with the heavy traffic generated by Cyclone. Consult Sun for the latest info.
There are no hard rules for how much memory a Cyclone server requires. As always, more memory is always better. Also, configuring lots of swap space spread across many drives (2.2) is important. As a general rule, 0.5MB to 1MB for each connection is sufficient. Exceeding this rule is fine, as long as it is within reasonable limits. Using certain history options such as "-cache" or "-bighistory" will also increase memory usage. If filters are being used, be sure to account for the memory usage of the filters. Often filters use a lot of memory with various databases used in the filtering process.
More spindles and more controllers are always helpful for Cyclone servers. The disk configuration regulates not only the speed in receiving and transmitting articles, but also provides an upper limit to how much backlog you can store for other feeds. It is recommended that people use 4GB or smaller disks for Cyclone servers in order to provide a reasonable number of spindles. Disks 7200 RPM or faster will result in a performance increase over 5400 RPM disks. Striping disks (2.1) will help realize better performance from disks.
SCSI is highly recommended, and systems with UltraSCSI or Wide interfaces will be able to better utilize their disks. Differential drives will result in a more stable bus and allow greater cable lengths.
The best way to judge what a server can do is to compare it against machine configurations currently in production. Below are the configurations of a few Cyclone servers currently in production.
A standard configuration could be a SS20/712 (SPARC 20 w/ two 70 Mhz SuperSPARC processors) with 352MB RAM, 6x2GB 5400 RPM disks spread across two controllers and 100 Mbit Fast Ethernet (full duplex). It should be able to handle 50 full feeds spread across 100 feed objects and about 300 connections without much trouble. It should scale further by increasing the amount of memory.
A configuration for a big machine could be an Ultra 2/2170 (Ultra 2 w/ two 170 Mhz UltraSPARC-1 processors) with 512MB RAM, 12x2GB 5400 RPM disks on a SPARC Storage Array 102 and 100 Mbit Fast Ethernet (full duplex). It should be able to handle 80 full feeds spread across 150 feed objects and about 600 connections without much trouble. Additional memory and faster disks would allow the machine to scale further.
There are usually five major factors that affect how "fast" a news
server is. By "fast" it is meant how quickly a server can receive
articles and re-propagate them to other peers. These factors are:
Disk is used for storing the articles and the history database. Since the history database often becomes a hot spot, striping it across multiple disks is always helpful (2.1). Other options including using the "-cache" option to store the history database in memory (4.1.4), or using a solid state disk for the history database.
The more memory a server has, the more data can be cached in memory. Cyclone makes use of all available memory in an adaptive manner and will happily use all the memory that a server has free.
A fast server will usually want at least a 100 Mbit Fast Ethernet connection to the local network. A full duplex connection to a dedicated switch port is always nice, or alternatively, using multiple Fast Ethernet cards for load balancing will work well. Cyclone supports using multiple Ethernet cards with the "OutgoingInterface" (4.3.3) directive and the "-interface" (4.3.1) option.
How well connected the server's LAN is to the Internet is of vital importance in determining how fast a server is. An ISP which has a single T1 will find that T1 largely saturated by news traffic and will have a hard time keeping up with more than 2 to 3 full feeds, no matter how fast the machine's hardware is. Keep in mind, that for many sites, 2 to 3 full feeds from fast Cyclone peers will be perfectly fine and will only lag the fastest of sites by 5 to 10 seconds. (Assuming that the server and network can keep up to the traffic levels).
Assuming that your server can keep up physically, your news peering arrangements determine how fast you get the news. Peering with lots of other fast servers will result in them "racing" one another as they try to send you news articles faster than other sites. It is not uncommon for a well connected site to be offered the same article 10 times in a single second. As mentioned before, a server only needs two or three full feeds to be up to date and current.
Cyclone does balancing by writing articles in a round robin fashion across all available spool disks. For most cases, if you have more than 3 drives, striping is not necessary for the spool objects.
Striping data is used to balance traffic across all available disks. Hardware RAID solutions are preferable, but software RAID0 using Solstice Online DiskSuite or Veritas Volume Manager will also increase performance keeping in mind that in RAID0 the loss of any one disk will cause a catastrophic loss of data. Software RAID5 is NOT recommended; if reliability is required, use RAID0+1 (striping with mirroring), or use a hardware RAID system. Cyclone also works with NFS mounted spools (such as those from a NetApp file server).
In terms of striping, it is a good idea to keep the spool objects and the history database on separate stripes or filesystems. A recommended configuration for a system with 6x2GB on two controllers, would be four disks striped together for spool objects, and the other two striped together for history, logs, and outgoing queues. Sites using the "-fifo" style of history (4.1.3) will want to consider using striping.
There are many varying opinions on what stripe size to use. The recommended advice from a Veritas support engineer is to use a stripe size of 64KB for partitions with spool objects, and a stripe size of 16KB or 32KB for other Cyclone partitions. It is a good idea to not mix disks used for spool objects and other things on large systems; that is, use a certain number of disks for spool object stripes, and use different disks for the other files used by Cyclone.
Spreading swap across multiple disks (assuming they are the same speed) will help increase performance. Solaris allows the use of swap files which can be placed on striped partitions for increased performance. Generally, configuring the machine with twice as much swap as there is physical memory works well. (If there is 256MB RAM on the machine, configure 512MB swap on the machine). For systems with 1GB or more of memory, using 1x to 1.5x physical memory for swap space, works well.
For more information on using swap files, refer to the "swap" and "mkfile" man pages.
It is a good idea to ensure that you are running the latest recommended patches. Newer kernel patches often have fixes for the TCP/IP stack or for the thread library. Sites running Solaris 2.5 or 2.5.1 will definitely want to consider upgrading to the ISS upgrade or Solaris 2.6 for network improvements. Solaris 2.6 provides some TCP improvements which have made a noticeable impact on existing Cyclone installations. Although all HighWind products work, some software packages are not compatible with Solaris 2.6 at present time. The ISS upgrade has a downside; by using it, it will take you out of the "core" Solaris patch path. Therefore, if you have problem with an application, solutions may be limited.
Officially, the Solaris recommend patches for all versions and platforms are located at <ftp://sunsolve.sun.com/pub/patches/>
The following sections provide general advice for tuning all versions of Solaris.
The /etc/system file allows you to modify kernel values. The system
must be reset to have these changes take effect. Some suggested
tuning parameters include:
The commands are separated from the rest of the text with --'s.
* How often to check buffers before flushing to disk set autoup=360 * time (in seconds) after which buffers will be saved to disk set tune_t_fsflushr=30
These will cause the OS to flush buffers to disk less often.
Increasing table and cache sizes can sometime provide to be helpful, but raising them too high might hurt performance by wasting resources and causing some operations to become extremely slow. Unless you are really sure, we would not recommend altering the Solaris defaults for table or cache sizes. Also remember that altering these values can cause you to lose more data if there is a crash since the time between updates has been increased.
ndd commands are used to set parameters for the TCP/IP stack. They are usually added to the /etc/init.d/inetinit script which is run when the system is booting.
The commands are separated from the rest of the text with --'s.
# Increase the default send/receive windows; helps on fast networks # mostly Also helps on high latency networks ndd -set /dev/tcp tcp_recv_hiwat 65536 ndd -set /dev/tcp tcp_xmit_hiwat 65536 ndd -set /dev/tcp tcp_cwnd_max 65535 # limit connection timeouts to 30 seconds ndd -set /dev/tcp tcp_ip_abort_cinterval 30000 # allows quicker restarts and resource saver ndd -set /dev/tcp tcp_close_wait_interval 30000
On systems with lots of memory, putting the history file (with the "-history" option to Cyclone) in /tmp is almost the same as putting the history file in a RAM disk and will often result increased performance. In order to minimize the effects of a system crash, it is recommended that the history file be copied back to disk at least once every 30 minutes.
A better idea is to use the "-cache" option (4.1.4) in Cyclone. This will keep large parts of the history database in memory and automatically write them back at statistics time. This should present better performance than using /tmp as there will only be a single copy of the history data in memory.
For Cyclone, it is usually sufficient to log "news.info" and "news.err" priorities to files. Logging "news.info" is only needed if the server has been started with the "-detailed" option. For servers where only the most serious errors should be reported should log "news.crit".
An example /etc/syslog.conf configuration would be:
news.info /news/cyclone/log/news.info news.err /news/cyclone/log/news.err
The "syslog.conf" file should only have TABs separating the log levels and the filenames, using SPACEs will cause errors.
For sites running a large number of Cyclone servers, syslog allows you to log messages to remote hosts. This can be very useful as it provides centralized logging.
Sites might want to move the spools and history outside the default Cyclone install area. This makes upgrading and having multiple installs on one machine a lot easier.
Cyclone supports three different history mechanisms. All three are self-expiring. This means that there is never any down time to clean up the history database or purge old entries. Also, they are all of a fixed size; your history database will never require more space than when it is first created.
The three history mechanisms are not compatible with each other and the history file must be started from scratch whenever a different history mechanism is selected.
The following table compares the various history mechanisms Cyclone supports, ranked in order of relative speed for doing lookups. The memory usage field does not take into account servers using the "-cache" option in which case the memory usage may be higher.
| database | # of articles | disk | memory |
| REGULAR | ~8 million | 34MB | 16-32MB |
| BIGHISTORY | ~16 million | 68MB | 16-64MB |
| FIFO | 4 million | 67MB | 8-16MB |
| FIFO | 8 million | 112MB | 8-16MB |
| FIFO | 12 million | 168MB | 8-16MB |
| FIFO | 16 million | 218MB | 8-16MB |
| FIFO | 24 million | 319MB | 8-16MB |
The "regular" history is the default history database used by Cyclone. It uses "time and access biased" self-expiration. This means that when Cyclone inserts a new message ID into the history database, it has to choose an old one to overwrite. This is done by identifying a cluster of IDs that can be replaced and choosing the oldest and least accessed one from the cluster. Since this database relies on clusters of message IDs, the retention time for an individual message ID is not predictable. Although the mechanism works very well, on very high-traffic highly-connected sites with some slower peers, the server might re-accept (and re-distribute) some small number of articles seen just a couple of days ago. Re-propagating a very small number of articles is not a concern for the vast majority of news sites. However, if you are deploying customer facing boxes, or are concerned about this, there are two other alternatives.
The "big" history, invoked with the "-bighistory" option, uses the same algorithms as the regular history database but on a database that is twice the size (and has twice the capacity). A 4x improvement in the retention time of message IDs has been demonstrated on large sites. This option does use more memory than the regular history database.
For very large peering machines, it would probably be wise to consider using the "-bighistory" option instead of the regular history since it will minimize the number of duplicates your peers will see.
The FIFO history, invoked with the "-fifo" option, uses a "first-in-first-out" algorithm to ensure that the history database always contains the most recent message IDs. It takes up more disk space than the previous two mechanisms, but guarantees predictable retention of message IDs and uses less memory than either of the "time and access biased" history mechanisms.
When using the FIFO style of a history database, you may want to adjust the size of the database beyond the default size of 8 million articles. Use the "-queue" option to specify the number of million of articles that will be stored in the history database. (Note that 2**20 == 1 million articles for the purpose of describing the history database size). Beyond the default size of 8 million articles, the size of the history database file increases linearly based upon its capacity. Memory usage for the FIFO history database is constant for all capacities. The chart in section (4.1) best documents the disk and memory usage of using different sizes of FIFO history databases.
FIFO queues of different sizes are not compatible. If you change the size of your FIFO queue you must restart your history. It is possible to use the "-queue" option to use a history database smaller than the default size of 8 million articles.
For this option to have any effect, the "-fifo" option must be explicitly specified.
For systems with a lot of memory, Cyclone has a "-cache" option which will store large portions of the history database in memory. This will result in a performance boost for systems with lots of free memory. The "-cache" option will automatically write the history database back to disk at statistics time (as specified by the "-update" option).
If you need to relocate the history database file (normally stored in the bin directory), invoke Cyclone with the "-history" option specifying the new location. This is useful for moving the history file onto a different partition such as /tmp, a separate filesystem, or even a solid state disk.
In order to control news routing, Cyclone provides a number of simple but powerful options that determine how the server will stamp news articles. It is important to notify remote feeds of what path alias to exclude from feeding you; if this isn't done, sites will end up offering you a lot of extra articles and will occasionally result in articles you have already seen being sent to you (11.3).
[A general note on path stamping: Usenet sites use path stamps to determine which sites should be offered articles (among other things). Every outgoing feed for a site will have an associated list of path stamps. Assuming that none of these path stamps appear in the "Path:" header of the message, it will be queued to be offered to the site (as long as it passes all other criteria for this outgoing feed).]
Normally, Cyclone stamps incoming articles with the full hostname of the server. Using the "-masquerade" option will allow you to stamp an alternative path line on incoming articles.
The "-alias" option allows you to specify multiple aliases by separating them with a "!" on the command line. This will cause Cyclone to check for each alias listed with the "-alias" option in the "Path:" header of each incoming article, and only add the missing path stamps.
For example:
Cyclone is started with the following options:
"-masquerade newsfeed.site.com -alias site.com!news.site.com"
IN: border.site.com!hello!usenet OUT: newsfeed.site.com!site.com!news.site.com!border.site.com!hello!usenet
IN: border.site.com!site.com!hello!usenet OUT: newsfeed.site.com!news.site.com!border.site.com!site.com!hello!usenet
IN: border.site.com!site.com!hello!news.site.com!usenet OUT: newsfeed.site.com!border.site.com!site.com!hello!news.site.com!usenet
Consider a company (company.com) which has two news servers on opposite sides of the country. They are set up to feed each other on the internal company backbone and you wish to split your external feeds geographically. Both have official names such as "n001.bc.company.com" and "n101.on.company.com", and also DNS entries "news-west" and "news-east" that point to the appropriate machine.
You would then start the west coast machine with: "-masquerade news-west.company.com -alias news.company.com" and the east coast machine with: "-masquerade news-east.company.com -alias news.company.com"
Then, when peering you tell peers to not send you any news stamped with "news.company.com" and have sites closer to the east coast machine feed "news-east.company.com" and have sites closer to the west coast machine feed "news-west.company.com". This configuration would be considered an ideal setup.
To further continue the example, assume that things on the west coast get very busy and you wish to expand. To expand you add another machine with another official names such as "n002.bc.company.com".
You now want to look at your "Path:" line differently. Have remote sites still alias out "news.company.com" so they do not send you articles your network has already seen. Internally, you want to use the alias "news-west.company.com" as your west coast cluster of machines since this will prevent your east coast machines from getting an article from the west coast and then refeeding it to another west coast machine. This is important since you do not want an article coming in on the west coast, going to the east coast, and them going back to the west coast.
To make these changes you want to change the "news-west.company.com" DNS entry such that it has multiple A addresses and points to both west coast machines and round-robins. This will make sites feeding your west coast machines balance between the two machines.
You then change your original "news-west" machine
(n001.bc.company.com) to run as:
"-masquerade news-west-1.company.com -alias
news-west.company.com!news.company.com"
and set up the new west coast machine (n002.bc.company.com) as:
"-masquerade news-west-2.company.com -alias
news-west.company.com!news.company.com"
Then, configure the west coast machines to only feed each other using the "news-west-1" and "news-west-2" aliases. This will allow them to exchange articles locally. You will also need to set up the individual DNS entries for "news-west-1" and "news-west-2".
You now have a "cluster" on the west coast that stamps the "news-west.company.com" alias and still preserves the "news.company.com" alias overall.
Multiple IP addresses will allow you to shape your network traffic in a number of ways.
By using the "-interface" option to Cyclone, you can run Cyclone on a specific network address as opposed to listening to all addresses. This allows you to run two servers, both on port 119 (the standard NNTP port), on the same machine.
For servers which do not have full duplex Ethernet, you can use two Ethernet cards and addresses to spread traffic out and prevent collisions. Have people feed one of the IP addresses, and then specify the other interface in each of the feed objects with the "OutgoingInterface" directive.
There are two ways to easily load balance traffic across multiple network interfaces.
For your server, create a DNS entry which has A addresses for each of the network cards. This will roughly balance incoming traffic across your two interfaces. Then, specify for half your feeds to use one address for outgoing traffic, and have the other half using the second address, using the "OutgoingInterface" directive.
Another option is to create a separate DNS entry for each address, and then for each incoming feed make a DNS entry pointing to one of the addresses.
For example, if your news server has the IP addresses "10.10.10.1" and "10.10.10.2", with names "newsfeed-hme0.usenet.com" and "newsfeed-hme1.usenet.com", and you are being fed by "both news.company.com" and "news.internet.net" you can make DNS CNAMEs such as: "company-in.usenet.com" which is a CNAME for "newsfeed-hme0.usenet.com" "internet-in.usenet.com" which is a CNAME for "newsfeed-hme1.usenet.com" Then instruct "news.company.com" to feed "company-in.usenet.com" and "news.internet.net" to "feed internet-in.usenet.com". You can then balance feeds across either network interface as you wish by changing the DNS entries.
In general, you want to have as large as possible spool objects, spread out across as many disks as possible. If you are not using some sort of RAID, you can still spread traffic across multiple disks by placing a single spool object per disk. It is best to create the spool objects soon after the filesystem is created (with newfs) to ensure that the data is located contiguously on the file system.
When using larger disks (4GB or larger) with multiple spool objects, it is best to spread apart the spool object entries in the "cyclone.conf" file. This will spread out the load across all of your disks better since Cyclone round-robins article writes to all the spools in the order listed in the "cyclone.conf" file.
Most sites should continue to use UFS for the filesystems as opposed to using Veritas File System for Cyclone servers. vxfs has not proven to be a large performance win with Cyclone. Also, if you have more than 4 drives, software RAID will seldom increase performance.
Cyclone allows sites to encode their spools for security reasons by using the "Encoded" directive for a spool object. This will result in a performance loss and it is recommended that people do not use this option unless government regulations or local management requires it.
There are a number of options available for sites which need to quickly
create new spool objects. One option is to use "bin/validate" to create
the spools by running the following command line:
> bin/validate -spools
This can be done while a Cyclone server is running. After the spools have been created, the Cyclone server can be restarted and it will begin to use the new spool objects.
Spool objects are regular files, so another option is to copy existing spool objects. This is a good technique for systems striping a spool partition between multiple disks. First, create a single blank spool with "validate -spools", then, simply use cp to copy this spool to the other locations specified in the "cyclone.conf" configuration file.
Do not try to copy spool objects between machines of different architectures. Specifically, you cannot use spool objects created on a SPARC based system on an Intel based system, or spool objects created on a SS20 on an UltraSPARC based system. This is especially true with NFS mounted spools.
To delete spool objects, simply remove the entries for the spool objects in question from the "cyclone.conf" configuration file and restart the server. After the server has been restarted, remove the spool object files by using "rm".
The most simple way to clear outgoing backlogs without accepting new articles is to start the server on a port other than the standard NNTP port. This is done by using the "-port" option when starting Cyclone. For example, to start the server on port 120, modify "bin/start" so that Cyclone is started with "-port 120".
The XRef options are used to integrate Cyclone with Typhoon and Breeze. The "Xref:" header in a Usenet message specifies the article number for each newsgroup the message is posted in. Cyclone has three options for how to handle the "Xref:" header. They are specified by adding the "XrefAction" directive in the "cyclone.conf" file. For example, to generate "Xref:" headers for all posts passing through the Cyclone server use "XrefAction Generate".
Consider the following diagram to give an overview of the roles of the various servers in a multi-level master/slave configuration.
border border \ / master / \ transport transport / | \ | \ leaf leaf leaf leaf leaf
The border, master and transport servers all run Cyclone, using the "XrefAction" directive with values of "Remove", "Generate" and "Pass" respectively. The leaf servers run Typhoon or Breeze and are set to accept "Xref:" headers as generated by the master server. This is done by setting the "XrefAction Parse" directive in Typhoon or Breeze.
This is the regular mode of operation for most Cyclone servers. If an "Xref:" header is in the message, it will be dropped and the article will be propagated to other sites without the "Xref:" header. For Cyclone servers operating solely as external border servers, this is the recommended mode of operation. The use of this option does not require an active file.
This option is used for Cyclone servers acting as a master server. It is responsible for generating the article numbers for all slaves or leaf nodes. This use of this option requires an active file. Newsgroups which do not appear in the active file will not have an "Xref:" value generated for the article.
This option specifies that the message will pass through the Cyclone server with the "Xref:" header intact. This is mainly of use for internal distribution servers who are being fed by a master Cyclone server and are feeding other distribution servers or leaf nodes. It does not require an active file.
Active files are used by Cyclone sites who want to limit propagation of Usenet articles to a selected set of newsgroups or which are acting as a master server (using XrefAction Generate). Active files are typically located in the "bin" directory and are specified by using the "-active" option to Cyclone. The "-active" option must be followed by the filename of the active file.
The active file is of the standard format:
NEWSGROUP HIGH LOW MODE
where mode is typically "y", "n" or "m" (without the quotes). The "y"
mode specifies that the newsgroup allows local posts, the "n" option
specifies that the newsgroup does not allow local posts, and the "m"
option specifies that the newsgroups is carried and is moderated.
Cyclone servers can be configured to drop unapproved postings in
moderated groups (4.9).
By simply supplying your Cyclone system with a list of newsgroups names it will populate the rest of the fields for you. This will mark all groups as unmoderated which can cause a lot of problems. If you just add a group name on the end of the file, Cyclone will do the right thing and add the newsgroup with the appropriate starting feeds (not include the newsgroup mode).
When a message is received that does not match any newsgroups in the active file, it is only propagated if the feed has the "AllowJunk" directive set to "True". Alternatively, the "DefaultAllowJunk" directive in "cyclone.conf" can be set to provide a default for all outgoing feeds.
When a site is doing "Xref:" header generation (4.7.2), the active file is updated to reflect the new high number of the newsgroups when an article is posted.
ISC is the official maintainer for the "Big-8" set of newsgroups and keeps up to date versions of both active and newsgroups files. They can be found at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
For sites which wish to stop propagating unapproved articles in moderated newsgroups there is a "DropUnapprovedArticles" directive which when set to "True" will drop any articles that are marked as moderated (with the "m" type in the active file) that does not have an "Approved:" header. The value of the header is not verified, it is only checked for existence.
A clever technique for sites which wish to stop propagating unapproved articles in moderated newsgroups without using a large active file is to have a small active file with only moderated newsgroups in it. Then, specify "DefaultAllowJunk True" which will result in the propagation of all articles not in the active file.
Cyclone supports filtering based on the results of an arbitrary program. This is a very powerful option and gives you complete flexibility in implementing any sort of spam filtering that you wish. To use these feature, invoke Cyclone with the "-program" option and specify the filename of the executable of the filter. By default, Cyclone passes the message header to the filter and waits for a response. By using the "-body" option, Cyclone will send the entire article with body to the filter. The documentation has more details on the "protocol" used to communicate between Cyclone and the filter.
HighWind Software is committed to reducing spam on Usenet and has provided links to a number of ready-to-run filters for Cyclone on their anti-spam page at <http://www.highwind.com/antispam.html>
To selectively feed filtered articles to certain sites, add a "DropFilteredArticles False" directive in your cyclone.conf. Then, for feeds you wish to feed filtered articles add a "AllowJunk True" directive to the feed entry. If you wish to only filter articles for certain feeds, set the default value of "AllowJunk" to "True", and then explicitly add "AllowJunk False" for feeds which will get filtered. Remember that articles that do not match the active file are also considered junk.
By using the "-body" option of Cyclone, you can instruct Cyclone to send the body of every article along with the header. As usual, the article information will be terminated by a line containing only ".\r\n".
To allow your filter to optimize, the "-body" option also causes Cyclone to check for an early response from your filter and terminate an article body prematurely. For example, if your filter sends it's "335\r\n" or "435\r\n" response after reading only a few lines of the article body, Cyclone will notice this and attempt to send the ".\r\n" as soon as possible.
With this optimization, if your filter can make a decision early, you should have it write that decision and go into "read and discard until .\r\n" mode. This optimization also works for messages with long headers.
With article propagation times to decreasing, it is often desirable to limit the age of articles that your site accepts and stores. Cyclone will only store an article that it receives if it is recent enough to be sent to any peer. This is specified by the "MaxAge" directive on outgoing feed. Specifically, any article older than the largest value of "MaxAge" will not be stored. Use the "DefaultMaxAge" directive to set a default value for all feeds; if no feeds explicitly use the "MaxAge" directive, the value of the "DefaultMaxAge" directive will be the oldest article that is stored.
Splitting outgoing feeds is the process of making a feed to another site span across multiple feed objects, each of which might have multiple streams. This is very useful because it allows you to prioritize certain messages for peers, or reserve a certain number of streams for small or for large articles. For systems feeding sites with limited bandwidth, feeding articles of all sizes will tend to "lock up" channels when large articles are sent causing small articles to be backlogged behind the large ones.
Splitting outgoing feeds depends heavily on the type of feed. This section has been divided to show configuration options for a wide variety of feed types.
When splitting outgoing feeds, sites might want to use different interfaces for the different feed objects for the same site using the "OutgoingInterface" directive (4.3.3).
Have a separate feed object for each set of newsgroups. Usually, a split of "Big 8", alt.binaries.*, alt.*, everything else, will work well. Use the "Subscription" directive to split feeds based on newsgroup.
The "Big 8" are considered to be comp.*, humanities.*, misc.*, news.*, rec.*, sci.*, soc.*, talk.*.
Have a separate feed object for small and large articles will prevent large articles from clogging up the smaller article streams. Use the "MinLength" and "MaxLength" directives to split feeds based on article size.
The single best value to split feeds at is either 4000 bytes or 8000 bytes depending on your pagesize. Both the operating system and Cyclone deal with memory in terms of pages each which are either 4096 or 8192 and are determined by your machine type. The best way to find out what the pagesize on your machine is to simply run the "/bin/pagesize" command. It will return either 4096 or 8192. It is suggested that people split on 4000 bytes or 8000 bytes, respectively, to give Cyclone a small bit of buffer space for extra information. Splitting at this size is a big win because the majority of articles that are being feeded for a particular set of streams will all fit in a single page of memory.
For sites which are trying to maximum article throughput to remote peers, splitting feeds at smaller value such as 1000 bytes or 1400 bytes will also increase throughput by dedicating streams to really small articles. Sites might also like to split at 64000 bytes for similar reasons.
First and foremost, the number of streams you use should be dictated by the remote system policy. If the remote site asks you to use 3 connections or less, do not try to use more than 3 connections. In many cases, using more streams will make things worse. Sites that are heavily backlogged or resource bound will experience problems no matter how many streams you use and by using many streams you are effectively just trying to force more articles through than the other machines feeding. This is a bad situation as it ultimately reduces the other machines capacity; it is a much better idea to turn DOWN the number of feeds to the site.
The number of streams to use depends heavily on the peering relation and the resources of the other machine. For most cases, 5 streams is more than enough to feed most sites. If feeding a reader machine which has no other feeds, it is best to experiment with a different number of streams and find the smallest number at which the reader machine does not backlog (or backlogs the least) and use that.
For really fast sites feeding other really fast sites, you might wish to increase the number of streams providing the other site can keep up. By adding more streams, you increase the potential bandwidth to the other site. It is recommended that you do NOT use more than 12 streams to any particular site without express permission.
Sites feeding machines that have applied fixed limits on the number of incoming streams might want to use one stream less than the maximum so that they can use the additional stream when needed to run "nntpTime" or other diagnostics.
This topic is well covered in the documentation and this section mainly just adds some additional comments. For reader machines that can keep up you might want to increase the backlog up to the approximate number of articles that your spool space can hold. By increasing the backlog past this, if the site goes down for an extended time and then comes back up there will be a lot of "Expired" articles which are relatively expensive to handle. The approximate number of articles that your spool space can hold can be calculated by looking at traffic over a week or two and find the average article size. Then divide your total spool space across all spool objects and divide by the average article size. The resulting number is the approximate number of articles your spool can hold at current traffic levels.
(As of October 1997, the average article size for sites which accepted articles up to 1MB in size was about 12000 bytes).
For sites that constantly drop articles that you are not obligated to provide backlog for, you might want to try using "non-spooling" feeds (5.4). This way, you at least are always sending recent articles to the other site.
To do "non-spooling" feeds where no backlogs are kept for an outgoing feed, you want to specify a "MaxDepth" of 395 or lower. There is little use in lowering this value any lower since you still want to be able to handle bursts of traffic. If you use non-spooling feeds, also called "best-shot" feeds, you will usually drop some articles to sites due to bursts of traffic or slight slowdowns on the other end.
The backup features of Cyclone (enabled by using the directives "BackupHostName", "BackupPortNumber" and "BackupTimeInterval") are very powerful and allow you to build fault-tolerant feeding systems.
This is the most common operation where are you feeding customer servers from one machine and you wish to continue to feed the customers if that machine goes down. To do this, you will need to define a primary machine and backup machine (or you could have multiple primary machines with each backing up another machine). Configure the primary machine with a regular feeds.conf. On the backup machine, configure it with the same feeds.conf entries for each site you want to backup for and add the backup directives for each feed entry. To add the backup directives, you want to have "BackupHostName" refer to the primary machine, and similarly "BackupPortNumber" refer to the port number on the primary machine. Then have "BackupTimeInterval" set to how long you wish to wait in between polling the primary server. Remember to make sure that the backup machine can connect to each customer server.
It is also possible to have a single machine backup for many machines by just adding the feed objects for each machine being backed up for and adding the appropriate backup directives (referring to the primary machine for the feed in question). This is useful if you have news machines scattered through out a network and you wish to provide fault tolerant news service for all of them. Just setup a single backup server at a central location and constantly feed it.
The backup feeds are also useful for feeding multiple customer boxes. If you are feeding a company which has two machines, "news-1.company.com" and "news-2.company.com", and you normally wish to feed "news-1" unless it goes down at which point you wish to feed "news-2", here is how you would setup such a configuration. On your feed machine add the following feed objects.
<Feed> IncomingHostname news.company.com OutgoingHostname news-1.company.com OutgoingFeedname company Aliases news.company.com NumberOfStreams 5 </Feed>
<Feed> OutgoingHostname news-2.company.com OutgoingFeedname company-backup BackupHostName news-1.company.com BackupPortNumber 119 BackupTimeInterval 600 Aliases news.company.com NumberOfStreams 5 </Feed>
This then feeds "news-2" only if "news-1" is down for more than 10 minutes.
To combat the problems of spam on Usenet, cancels are being issued for messages which exceed certain limits. Cancels are also used for retro moderation of newsgroups for off-topic articles. Express cancel feeds are a way of sending cancels to a remote site "out of band". They are distributed in separate streams which will not get used for other, larger articles. Since the large majority of cancels are sent automatically, the delay between the original posting and the cancel is often under a minute. Most modern news servers can be configured to accept cancels automatically and add a history entry for the message being canceled if it has not yet arrived. This allows you to save disk and network resources since you only need to process one article instead of two.
Cancel feeds in Cyclone are handled by using the "SubjectFilter"
directive. To send a site both a regular feed, and a cancel feed, use
a set of feed objects structured like this:
<Feed> IncomingHostname news.company.com OutgoingHostname news.company.com OutgoingFeedname company-normal Aliases company SubjectFilter cmsg?cancel* NumberOfStreams 4 </Feed>
<Feed> IncomingHostname news.company.com OutgoingHostname news.company.com OutgoingFeedname company-cancel Aliases company SubjectFilter *, !cmsg?cancel* NumberOfStreams 1 </Feed>
By adding a delay (5.12) of 5-15 minutes to the first feed, the resources used on the remote site is greatly reduced since it will only need to handle one message (the cancel) for each article of spam, instead of two (the original article and the associated cancel). This can turn out to be a big win for sites whose readers servers are having trouble keeping up. Usually no more than one stream is necessary to handle cancels.
NoCeM messages are another option for dealing with spam. Most spam cancelers also send out NoCeM messages on a regular basis for all the messages they cancel. Two major benefits of NoCeM messages are that they require far less resources to handle since a single message can cancels thousands of articles, and they are PGP signed so that there is a very low chance of forgery. More information on NoCeM can be found at the CancelMoose home page (<http://www.nocem.org/>).
NoCeM messages are posted to a small number of high bandwidth groups. Also, cancels messages for which NoCeM messages have been generated are usually stamped with "nocemed". This means that other feed objects to sites with NoCeM feeds can have the path stamp "nocemed" aliased out.
To send a feed of only NoCeM messages, use a feed object such as:
<Feed> OutgoingHostname news.company.com OutgoingFeedname company-nocem Aliases company Subscription !*, news.lists.filters, alt.nocem.misc NumberOfStreams 1 </Feed>
Delaying the regular feed (5.12) is highly recommended since NoCeM messages are usually only posted every 15 minutes and will lag behind regular postings on a Cyclone fast site. Usually no more than one stream is necessary to handle all NoCeM messages.
For feeds which handle spam using a method other than cancels might
find it beneficial to not feed those messages. Besides using a
"SubjectFilter" to not send cancel messages (5.6.1), an administrator
might find it preferable to only stop sending a particular subset of
cancels. Spam cancelers usually indicate the type of message being
canceled by including a pseudo-site in the path line which can be
filtered by adding that pseudo-site in the "Aliases" (5.8.2) list for
the outgoing feed. Common pseudo-sites include:
| Pseudo-Site | Description |
| cyberspam | Spam/EMP cancels (universal) |
| spewcancel | Spew cancels |
| mmfcancel | Make.Money.Fast cancels |
| bincancel | Binary (in a non-Binary group) cancels |
| retromod | Retro-Moderation cancels |
Multiple pseudo-sites can be used in the same article, and cyberspam is usually included in all such cancels. More information on cancel messages can be found in Tim Skirvin's FAQ which can be found in the newsgroup "news.answers" or at <http://www.uiuc.edu/ph/www/tskirvin/faqs/cancel.html>
An example of a feed that just filters out spewcancels is:
<Feed> OutgoingHostname news.company.com OutgoingFeedname company Aliases company, spewcancel </Feed>
Also, sites can use the "IncomingPathFilter" directive to automatically drop all cancels of particular type which is useful for a Cyclone server running as a master server that is also doing its own spam filtering.
There are three different directives used to specify outgoing feed content by newsgroup: "Subscription", "AdditionalSubscription", and "FilterSubscription". A message is queued to be offered if it matches at least one newsgroup in the combination of "Subscription" and "AdditionalSubscription" and does not match any newsgroup in "FilterSubscription". A subscription match (from now on, considered to be the union of "Subscription" and "AdditionalSubscription") only needs to have a match on a single newsgroup to have the article queued to be offered. That is, if a feed has a subscription of "news.*, !news.admin.*" and a message is crossposted to "news.software.nntp,news.admin.misc" it will still be sent since the newsgroup "news.software.nntp" still matches the subscription. On the other hand, if the feed has a subscription of "news.*", and a "FilterSubscription" of "news.admin.*", the article would NOT be sent since "news.admin.misc" matches the "FilterSubscription" of "news.admin.*".
The use of the "Path:" header is the main way to prevent circular loops in Usenet and to prevent waste by reoffering the same articles. It is very important to remember that DNS entries (as specified by OutgoingHostname) and path aliases (as specified by Aliases) are entirely different things. Many sites will have a DNS entry of their news server such as "news.company.com", but stamp articles with "company.com".
If the "Aliases" directive is missing for a feed object, the value of the "OutgoingHostname" directive is used as the default value for "Aliases". If an "Aliases" directive is present, it does NOT automatically assume a default value from the "OutgoingHostname" directive. The best solution is to always use an "Aliases" directive specifying exactly what path stamps you wish to exclude.
The following entry will offer any articles which have not yet been
stamped with "news.company.com":
<Feed> OutgoingHostname news.company.com OutgoingFeedname company NumberOfStreams 5 </Feed>
If that site has a direct feed from "usenet.org" and does not want you
to send articles which have passed through "usenet.org" as well, you
need to change your feed entry to:
<Feed> OutgoingHostname news.company.com OutgoingFeedname company Aliases news.company.com, usenet.org NumberOfStreams 5 </Feed>
Both the new name "usenet.org", and the old default value "news.company.com" must be included as the value of the "Aliases" directive.
Besides using Aliases path line exclusions for filtering out cancel messages (5.6.3), sites might wish to filter out messages which have passed through another peer of the remote site. That is, if there is an outgoing feed for "news.company.com" which is being fed by both "news.usenet.com" and you, you might want to exclude both "news.company.com" and "news.usenet.com" when feeding the remote site. This way, you're not offering the site articles that it will probably get from another peer.
To configure such a feed, use a feed object such as:
<Feed> OutgoingHostname news.company.com OutgoingFeedname company Aliases company.com NumberOfStreams 5 </Feed>
Incoming feed objects where the "StampIncomingIPAddress" directive is set to "True" will have articles sent from that "IncomingHostname" stamped with the IP address from which the articles were sent on the path line. This is most useful for feed objects which have an "IncomingHostname" directive with a wildcard and allows administrators to track the path of the message. Cyclone will also stamp the IP address on an "IncomingHostname *" feed to prevent abuse of "open" servers.
Cyclone supports expected path stamps for incoming feeds to reduce the possibility of forged posts getting injected at your server. That is, it allows you to verify the "news.company.com" is only stamping messages as "news.company.com" before sending them to you instead of "news.business.org". To use this feature, add an "IncomingExpectedPathStamp" directive which lists all valid path stamps for the incoming feed. Messages which come from this feed but do not match the path stamps listed are marked with an additional path stamp of the "IncomingHostName" value for the feed object with ".MISMATCH" appended to it.
Sites which send you articles which do not match the expected path stamps will show up as such in the incoming statistics for the feed.
For sites which do not wish to propagate messages which have been marked by other sites as mismatched, it is possible to set up in the "cyclone.conf" file a "DefaultIncomingPathFilter" directive with the value of "*.MISMATCH".
The "MaxChecks" directive specifies the maximum number of "CHECKs" that will be sent to a feed in streaming mode before waiting for a response. Cyclone has adaptive algorithms which will find the optimal number of "CHECKs" to send to maximum article throughput up to this value. Most sites do not need to change the value from its default value of 35.
Sites which are feeding large numbers of articles to other sites might find it beneficial to increase the "MaxChecks" value up to its maximum of 125. It is also wise for delayed feeds (5.12) that will have a lot of refused articles. That is, if you are delaying a feed for 30 minutes to another site with full feeds, you will be usually getting 99% of the articles you offer refused. Therefore, it makes sense to up the value of "MaxChecks" to the maximum allowable value of 125.
For feeds to well connected sites or to sites over high latency links where bandwidth is expensive might wish to lower the value of "MaxChecks" down to 15 or even lower. Lowering it beyond 5 will not have any effect. Another alternative for sites lowering the "MaxChecks" is to consider turning off streaming (5.11.1).
For feeds to well connected sites or to sites where rejections are expensive, turning off streaming can save a lot of bandwidth, and increase article throughput. To do so, set the "AllowStreaming" directive for the outgoing feed to "False".
Using delayed feeds is a very good way to provide redundant news services for other sites. By setting up a feed with a "DelayTime" directive with a value of "600" (10 minutes) to a site with other feeds, you will offer them articles 10 minutes after receiving them and "fill in the holes" to their newsfeed. This also greatly reduces the resource usage on your server since most of the time you will just be sending CHECKs instead of the full articles.
Delayed feeds can also be very effective in a multi-level news system environment where you can delay posts from getting sent to the leaf nodes in order to send NoCeM messages or cancels (5.6) before the actual messages. For NoCeM messages, a delay of 10 to 15 minutes is usually needed (since the most frequent NoCeM notices are usually only posted every 5 minutes). For cancels, the delay can be much lower (1 to 5 minutes for a well connected sites) since cancels are issued immediately.
Using very small delay values (30 to 60 seconds) are useful for reducing bandwidth usage that is often associated with "racing" other fast sites.
Local posting feeds are feeds in which you only send articles that originated from your site, or another close by peer. This is usually done by specifying a low number with the "MaxHops" directive since a local post will only go through a small number of local machines (and thus have a small number of path entries) before it reaches your border machine for transport to other servers. A good value for sites is usually a "MaxHops" value of "5". This will usually work for sites which have three levels in their news system (border, master, leaf). Sites with more than three levels may find it difficult to accurately to do local posting feeds and will more than likely prefer to rely on standard feeds.
Small sites can support local posting feeds by using the "Aliases" directive and aliasing out articles to peers from other peers (5.8.2). This is usually reasonable for sites with 4 to 5 peers. To do this, for each peer which you wish to send only local posts, alias out all of your other peers path stamps for the outgoing feed object.
To prevent peers from using too much of your resources by opening an excessive number of streams to your server, use the "MaxIncomingNumberOfStreams" directive to provide a maximum number of incoming streams they can open. Using the "DefaultMaxIncomingNumberOfStreams" will allow you to provide a default value for all feeds. A value of "0" will disable connection limits. Be sure to let the peer know of the limit so they don't try to waste resources by opening up too many connections to you. It is reasonable to set "DefaultMaxIncomingNumberOfStreams 25" in your "cyclone.conf" to prevent feeds from getting out of hand.
See the web page <http://www.highwind.com/faq/> for the small script "summarize-text.pl" which will allow you to generate quick summary statistics suitable for text display or mailing. The script accepts both incoming and outgoing data at the same time and outputs a joint report. There is also an option to have a "mapping" file which will map hostnames to sitenames. That way, you can display the statistics for the server "city-big-hub-news-in.company.com" simply as "company"'s statistics.
For example, to do a daily report:
> cat log/stats.in log/stats.out | tools/summarize-text.pl
Using the statistics generated by the "-group" option of Cyclone allows you to get a good idea of how traffic is distributed across newsgroups and allows you to plan for upgrades on the reader machine. Use the "tools/group-stats.pl" script to generate summaries from this data.
Outgoing articles that are queued for being offered will end up being labeled as rejected, refused, accepted, dropped, or expired. These terms are often confusing for new news administrators; here is a brief description of all of them.
offered: The article was offered to the other site with an "IHAVE" or "CHECK". This number is the sum of "IHAVEs" and "CHECKs" and does not include articles sent only with a "TAKETHIS" (which happens when streaming and the remote site is accepting most of the articles).
rejected: The article was sent in entirety to the remote site and which point the remote site decided it did not want the article because it was a duplicate, invalid or didn't match an active file entry. These are expensive in terms of bandwidth and resources since you actually sent the article and then found out you should not of.
refused: The article was offered with an "IHAVE" or "CHECK" and the remote site responded that it had the article. These are very inexpensive.
accepted: The article was sent in entirety to the remote site and it accepted the article for further processing or delivery.
dropped: The article was never offered because the backlog for the feed exceeded the "MaxDepth" value for the feed. These are relatively inexpensive.
expired: The article was offered but then not sent since the article was not in the spool. These are relatively expensive since you need to go to the spool to find out that you cannot send the article. It is much more desirable to have dropped articles instead of expired articles.
Since Cyclone uses a highly adaptive engine to offer articles, it may determine that a remote site appears to want all the articles that you offering. In that case, Cyclone will stop offering (using CHECK) the articles and simply send them (using TAKETHIS). Since the offered count does not include "TAKETHIS" counts, the remote site can accept more articles than are offered.
The "-detailed" option logs lots of additional information to the syslog at the "LOG_INFO" level. It is useful for sites which want to know what is happening to their server on a regular basis. Among other things, it logs all connects and disconnects from remote sites.
The "log-report.pl" script is a useful tool to find data on incoming traffic on your Cyclone server. It is best used to provide information on what is happening "right now" to the server, and also equally useful to provide information on how the server is performing over a long period of time. It does this by using the output generated when Cyclone is run with the "-log" option.
For Cyclone, "log-report.pl" has three major modes of operation which are describe in detail below. For Typhoon and Breeze, there is an additional mode of operation supported. Multiple modes of operation can be included at the same time.
For example:
> cat log/log | log-report.pl -r -d -s
The Incoming Article Rate mode, enabled with "-r", gives you a very good way of determining what is happening to your server right now. It reports, on a per second basis, how many articles were received in that second. This is most useful when used with a tail of the log file.
For example, on a busy server after being restarted:
> tail -500 log/log | log-report.pl -r
Report from Sat Oct 11 13:47:43 1997 to Sat Oct 11 13:47:53 1997, 11s
Incoming Article Rate
Arts Bytes Second Diff Second (string)
* 22 89402 876592063 0s Sat Oct 11 13:47:43 1997
40 131239 876592064 1s Sat Oct 11 13:47:44 1997
36 181163 876592065 1s Sat Oct 11 13:47:45 1997
48 278438 876592066 1s Sat Oct 11 13:47:46 1997
51 219660 876592067 1s Sat Oct 11 13:47:47 1997
57 489518 876592068 1s Sat Oct 11 13:47:48 1997
24 162125 876592069 1s Sat Oct 11 13:47:49 1997
46 184725 876592070 1s Sat Oct 11 13:47:50 1997
82 182531 876592071 1s Sat Oct 11 13:47:51 1997
43 324761 876592072 1s Sat Oct 11 13:47:52 1997
* 51 263242 876592073 1s Sat Oct 11 13:47:53 1997
The first and last seconds are marked with a star to indicate that they may not represent complete data for the second in question. The "Diff" column indicates the number of seconds in between the current second and the last second; during non-busy periods for servers that are not taking a full feed, there might be several seconds in between articles.
The Incoming Article Rate mode is the default mode of operation (this can be changed easily by modifying the log-report.pl program).
The window option, enabled with "-w", allows you to see how your server is performing in granularity greater than a single second. Instead of reporting things on a per second basis, it gives you the totals over a specified number of seconds. By using the per second option, enabled with "-p", it is possible to translate these numbers into per second averages.
For example, the above data with a 5 second window:
> tail -500 log/log | log-report.pl -r -w5
Report from Sat Oct 11 13:47:43 1997 to Sat Oct 11 13:47:53 1997, 11s
Incoming Article Rate, 5 second window
Articles Bytes Start End Final Second (string)
197 899902 876592063 876592067 Sat Oct 11 13:47:47 1997
252 1343660 876592068 876592072 Sat Oct 11 13:47:52 1997
And using the per second option:
> tail -500 log/log | log-report.pl -r -w5 -p
Report from Sat Oct 11 13:47:43 1997 to Sat Oct 11 13:47:53 1997, 11s
Incoming Article Rate, 5 second window
Articles Bytes Start End Final Second (string)
39.4 179980.4 876592063 876592067 Sat Oct 11 13:47:47 1997
50.4 268732.0 876592068 876592072 Sat Oct 11 13:47:52 1997
Finally, an example of reporting over an hour window (per second
option):
> cat log/log | log-report.pl -r -w3600 -p
Report from Sun Oct 12 00:00:00 1997 to Sun Oct 12 19:27:59 1997, 19h 28m
Incoming Article Rate, 3600 second window
Articles Bytes Start End Final Second (string)
12.4 148786.2 876639600 876643199 Sun Oct 12 00:59:59 1997
11.1 114229.9 876643200 876646799 Sun Oct 12 01:59:59 1997
10.6 133532.1 876646800 876650399 Sun Oct 12 02:59:59 1997
10.8 134094.2 876650400 876653999 Sun Oct 12 03:59:59 1997
10.1 95392.6 876654000 876657599 Sun Oct 12 04:59:59 1997
10.5 111867.8 876657600 876661199 Sun Oct 12 05:59:59 1997
10.1 107188.4 876661200 876664799 Sun Oct 12 06:59:59 1997
10.3 116411.0 876664800 876668399 Sun Oct 12 07:59:59 1997
10.7 121961.4 876668400 876671999 Sun Oct 12 08:59:59 1997
12.0 152981.1 876672000 876675599 Sun Oct 12 09:59:59 1997
12.1 168975.7 876675600 876679199 Sun Oct 12 10:59:59 1997
13.4 158965.8 876679200 876682799 Sun Oct 12 11:59:59 1997
13.1 174116.1 876682800 876686399 Sun Oct 12 12:59:59 1997
11.9 176317.3 876686400 876689999 Sun Oct 12 13:59:59 1997
11.5 143768.7 876690000 876693599 Sun Oct 12 14:59:59 1997
11.5 149317.7 876693600 876697199 Sun Oct 12 15:59:59 1997
11.3 138777.0 876697200 876700799 Sun Oct 12 16:59:59 1997
11.9 135741.3 876700800 876704399 Sun Oct 12 17:59:59 1997
12.8 131200.2 876704400 876707999 Sun Oct 12 18:59:59 1997
Note that there should be no space between the "-w" and the window size.
The Incoming Article Source mode, enabled with "-s", allows you to see where articles are coming from. By default it shows source based on the IncomingHostname entry that it matches, but by using the "-ss" option, it also splits source into the IP address. This is useful for sites which have multiple news servers feeding you under the same IncomingHostname entry. The per second option, enabled with "-p" (used in conjunction with -s or -ss), can also be used; this gives you per second averages for each site.
For example, to see where the most recent articles you have received
come from:
> tail -500 log/log | log-report.pl -s
Or to see the average number of articles per second and bytes per
second for each IP address:
> cat log/log | log-report.pl -ss -p
The Incoming Article Distribution mode, enabled with "-d", gives you a break down of articles that arrive at your server. Articles are organized into different groups based upon their size. It provides a great way of seeing where your large and small articles come from (when combined with the regular expressions) and also provides a good way to tune outgoing feeds split by size. The sizes used to organize articles by are easily customizable at the top of the "log-report.pl" script.
For example:
> cat log/log | log-report.pl -d
Report from Sun Oct 12 00:00:00 1997 to Sun Oct 12 19:27:59 1997, 19h 28m
Incoming Article Distribution by Size
# MaxSize Count Cum. Count Bytes Cum. Bytes
1 100 0 0 0 0
2 200 0 0 0 0
3 300 10 10 2590 2590
4 400 143 153 54739 57329
5 500 7857 8010 3657482 3714811
6 600 51224 59234 28819970 32534781
7 700 128128 187362 83206570 115741351
8 800 69742 257104 52457597 168198948
9 900 48636 305740 41101056 209300004
10 1000 69849 375589 65924721 275224725
11 1100 39587 415176 41956192 317180917
12 1200 38851 454027 44450135 361631052
13 1300 26743 480770 33441334 395072386
14 1400 31182 511952 42189176 437261562
15 1500 46826 558778 67894159 505155721
16 1600 32595 591373 50385568 555541289
17 1700 19944 611317 32837030 588378319
18 1800 13709 625026 23967515 612345834
19 1900 10649 635675 19692344 632038178
20 2000 8987 644662 17512050 649550228
21 2500 31933 676595 71054086 720604314
22 3000 16260 692855 44350457 764954771
23 3500 19199 712054 61354826 826309597
24 4000 6422 718476 23819708 850129305
25 5000 7913 726389 35510637 885639942
26 6000 3913 730302 21054682 906694624
27 8000 4622 734924 32214945 938909569
28 12000 6673 741597 66379769 1005289338
29 16000 5203 746800 74044364 1079333702
30 32000 10619 757419 255223735 1334557437
31 48000 10009 767428 395133081 1729690518
32 64000 8376 775804 460057285 2189747803
33 128000 12329 788133 1097598891 3287346694
34 256000 6425 794558 1103409348 4390756042
35 512000 5174 799732 2121227268 6511983310
36 1024000 4562 804294 3118684193 9630667503
37 1536000 62 804356 76807573 9707475076
38 2048000 0 804356 0 9707475076
TOTAL 804356 9707475076
This report shows that about 90% of the articles are less than 8000 bytes in length, but only account for 10% of the entire volume. It also shows that the space for all articles between 8000 bytes and 64000 bytes, and those less than 8000 bytes, is approximately equal despite a large difference in the number of articles.
This option only works with Typhoon and Breeze log files.
The Spool Usage mode, enabled with "-u", is a good way to see how your spools are being utilized. This is most useful for Typhoon or Breeze sites which have certain spools for certain hierarchies or certain size groups. The size of each spool must be specified by modifying the "log-report.pl" script. While not perfect, it is reasonable to assume that if you use a large enough sample size, it will generally reflect the data for a longer period of time. This allows you to see where articles are being filed and give you an approximation of how long a particular spool will last.
For example:
> cat log/log | log-report.pl -u
Report from Sun Oct 12 19:05:25 1997 to Sun Oct 12 20:34:17 1997, 1h 28m
Spool Usage
Spool Capacity Articles Bytes % Cap Lifetime % Art % Byt
1 2048000000 44452 60883178 3.0% 2d 1h 88.7% 10.6%
2 2048000000 2836 252926162 12.3% 11h 59m 5.7% 43.9%
3 2048000000 2827 262487450 12.8% 11h 33m 5.6% 45.5%
TOTAL 50115 576296790
This report shows data for three 2GB spools. The first is for articles less than 8000 bytes, the last two are for articles larger than 8000 bytes. The sample includes data for approximately 90 minutes or 50000 articles. During that time, 3% of the spool for small articles was filled, giving it a lifetime of just over 2 days; the other two each were filled about 12.5%, giving them a lifetime of about 12 hours. This data also reflects the findings from the distribution mode that 90% of the articles are less than 8000 bytes but they only account for 10% of the total volume.
While some of this information can be obtained from careful use of the distribution mode, it is much more difficult to calculate for sites which have many spools split by size and also by subscription. This mode gives you a good approximation based on real data.
To show a report of the last X articles ("right now" report), tail the
last X lines of the current log file. "log-report.pl" correctly deals
with an incomplete last line. For example, to view a report on the
last 500 articles:
> tail -500 log/log | tools/log-report.pl
To show a report on a longer period either increase the parameter to
tail or simply use the entire log file. This also works well for
archived log files. This can take a fair amount of time due to the
size of the log file. For example:
> cat log/archive/log.971018 | tools/log-report.pl
The "log-report.pl" utility also has a wide variety of options that
specify what data will be reported on. All the different modes of
operation allow the addition of regular expressions to select
particular entries from the log file. Multiple regular expressions
can be searched for, and a line is only used to tabulate results if it
matches at least one of the regular expressions. Regular expressions
are most useful in searching for particular IncomingHostnames, IP
addresses of sending server, or message IDs. For example, to only
tabulate on articles from "company.com" or "server.org":
> cat log/log | tools/log-report.pl company.com server.org
Or to search for articles originating from ClariNet:
> cat log/log | tools/log-report.pl '@.*clari.net'
(The quotes are used to prevent shell expansion on the *)
The "nntpTime" utility is the single best way to determine how well your Cyclone server (and yours peers) are running. Sending 100 articles, by using "-loop 100", is a very good way to measure how well your server is running at that instant. While times will vary depending on server load, hardware configuration and software configuration, taking several measurements at different times will give you a good idea of your server's performance. In general, low double digit "nntpTimes" for all metrics is considered excellent. The "IHAVE" response time refers to the time to perform the history lookup and will generally be higher for systems running with the "-fifo" history. The article response time is usual a direct reflection of how fast and how busy your disks are.
On a VERY fast server (under normal load conditions), you might see an
nntpTime such as:
> nntpTime -server news -loop 100
DNS + Connect: 9 msec Welcome Message: 8 msec Average IHAVE Response (100 trys) : 4 msec Average Article Response (100 x 4172 byte articles): 12 msec
For EXTREMELY fast sites, nntpTime has a microsecond option that can be invoked by using the "-u" option.
The main reason why Cyclone can take a long time to restart, especially after being up for long periods is because it must reresolve all the DNS entries. The time to look up DNS entries is highly variable and a single bad entry can slow down the entire process. One solution is to prime the DNS server's cache by executing a "bin/validate -dnstime" prior to restarting Cyclone. For convenience, sites might want to add a line which does the "bin/validate -dnstime" prior to stopping the server in the "bin/restart" script.
Using the "bin/validate" program with the "-dnstime" will show you how long it takes to perform all your DNS lookups. Adding the "-detailed" option will show you how long each individual entry takes to resolve and can be used to indicate trouble spots. For sites with continually poor DNS lookup times, consider using the "-nodns" option. For sites using the "-nodns" option, be sure to replace all full hostnames with their IP addresses.
Sites with a lot of peers and reliable DNS might want to reduce the time in between DNS reloads. This is useful when you have lots of peers and you want to notice DNS changes more quickly. The default value is to update the DNS every 6 hours. This is done by specifying the time in between DNS reloads as an argument to the "-dnsexpire" of Cyclone. Remember to specify the time in seconds.
The data listed in this section is current to October 1997. Expect numbers listed to increase in the future.
Daily traffic for a full feed is at about 10GB to 12GB per day, or about 800,000 to 1,000,000 articles per day. Traffic has been steadily increasing for the last couple of years. A listing of real world article distributions during this period can be found in section (6.5.3).
A couple of full feeds will easily saturate a full T1. Busy servers can expect to see 400-600 KB/s outgoing averaged over a full day. Really large sites with many full feeds could easily use in excess of 100GB for news transport.
The normal incoming and outgoing stats file for a Cyclone server (along with the "-groups" stats) are usually quite small. The incoming and outgoing stats file for busy sites will usually not total more than 1MB per day. Similarly, the "-groups" stats data is also usually under 1MB per day.
The size of the "-log" and "-paths" data can fluctuate quite a lot due to fluctuations in daily traffic. Generally, one can expect the daily size for a "-log" file to be 80MB to 110MB, and the daily size for a "-paths" file to be 100MB to 140MB.
Using the "-detailed" option will result in a lot of info being logged via syslog. This is especially true for busy sites with many connections since each connection generates connection and disconnection logs.
The Usenet backbone is extremely fast and also very resilient. In terms of speed, the majority of the top 150 machines on the Freenix list are separated by no more than 10 to 15 seconds. The very fastest machines (at least the top 25, possibly more) are most often in sync to under 3 to 5 seconds. A busy site with many peers could easily see the same article offered 20 times within the first second; and a fast Cyclone server can turn around an article (time from being first offered to time first offering) in under 250ms.
The best way to get support from HighWind Software for Cyclone, or any other HighWind product, is to send email to <support@highwind.com>. Another good forum is the Cyclone mailing list. To join, send a mail with "subscribe cyclone" (without the quotes) in the body to <majordomo@jabberwocky.bbnplanet.com>. Similarly, the Typhoon and Breeze mailing list can be joined by sending a mail with "subscribe typhoon" in the body to the same address. Suggestions for new frequently asked questions or other FAQ material should be mailed to <faq@highwind.com>.
This question is often asked by sites who were using other feeding software prior to running Cyclone and never noticed a backlog before. The answer is Usually that you weren't getting as many articles/bytes before. The best way to verify this is through the use of the "nntpTime" utility. Cut down on other feeds to the reader box and let Cyclone do its job without much competition. Using "express" cancels combined with delayed feeds (5.6) or doing filtering on your Cyclone server (4.10) can alleviate many problems with backlog to a reader machine. Another option is to try out Typhoon or Breeze for your reader machine.
There are three main reasons why your peers might see duplicate articles being offered from your Cyclone server.
The first occurs if you are feeding a server which handles express cancels by not accepting the original article if the cancel arrive first. Since cancels are posted so quickly, the article and the cancel could be offered in the same "CHECK" window. The remote server would respond to the "CHECKs" by saying to send the articles and Cyclone would send both articles simultaneously. If the cancel is accepted a fraction earlier than the real article, this will generate a duplicate error when the real article is actually accepted. This can be overcome by using express cancel feeds (5.6) or by shutting off streaming.
The second occurs if the other peers is being fed by a number of fast sites that are closely in sync (as are the majority of the Cyclone servers in the top 50 of the Freenix list). If some of the peers are using streaming, the often end up "racing" each other as they all offer the article at the same time and then send it. This can be overcome by having the peer cut down on the number of incoming streams (5.14) or have some of the feeds be delayed by a minute or so (5.12).
Finally, because of the inconsistencies of Usenet, some sites might refeed you old articles which you previously saw but have now expired out of your history database. Reducing your incoming "MaxAge" (4.11) or using either of the "-bighistory" (4.1.2) or "-fifo" (4.1.3) styles of history can help greatly.
An alternative to using the "-update" value for specifying stat periods, sites might wish to handle stats through a cron script. A benefit of this is it allows you to restart the server if the configuration has changed while still generating stats on the hour.
To do this, make a root cronjob for the top of the hour. The script should check the modified time of the configuration files, and if they have been modified in the last hour, execute the "bin/restart" script. Otherwise, if they have not been modified, execute the "bin/statsnow" script. You might also wish to generate HTMLized reports using the "tools/summarize.pl" script at this point after the stats have been written. Remember to set the "-update" time to a large value (perhaps 2 hours in case the cron job fails). Also, for security, make the scripts owned by root and non-writable.
Included below is an example of an improved "bin/rotate" script for use to be run right before the end of the day. It also restarts the server to ensure that the files are flushed before the end of the day. Also, create the subdirectories "old", "logdata", "groupdata" and "pathdata" in the log directory.
The "../bin/restart" line can be replaced with a "../bin/statsnow" for
sites which do not wish to restart the server daily; this also allows
you to run the rotate script as news instead of root. You will want
to invoke the rotate BEFORE midnight, so an example crontab is:
58 11 * * * /news/cyclone/bin/rotate
If you are running rotate script as root, be sure to make it owned by root and non-writable for security.
#!/bin/sh
#
# Cyclone Rotate - modified
# Copyright (c) 1996-1997, HighWind Software, Inc.
#
cd `/usr/bin/dirname $0`
DS=`/bin/date +%Y%m%d`
#
# Move log/log ->log/logdata/log.YYMMDD
#
if [ -r ../log/log ]; then
/bin/mv -f ../log/log ../log/logdata/log.$DS
fi
#
# Move log/paths ->log/pathdata/paths.YYMMDD
#
if [ -r ../log/paths ]; then
/bin/mv -f ../log/paths ../log/pathdata/paths.$DS
fi
# The following three need to be updated one last time
sleep 10
../bin/restart
sleep 20
#
# Move log/stats.in ->log/old/stats.in.YYMMDD
#
if [ -r ../log/stats.in ]; then
/bin/mv -f ../log/stats.in ../log/old/stats.in.$DS
fi
#
# Move log/stats.out ->log/old/stats.out.YYMMDD
#
if [ -r ../log/stats.out ]; then
/bin/mv -f ../log/stats.out ../log/old/stats.out.$DS
fi
#
# Move log/groups ->log/groupdata/groups.YYMMDD
#
if [ -r ../log/groups ]; then
/bin/mv -f ../log/groups ../log/groupdata/groups.$DS
fi
To ensure proper startup and shutdown of your Cyclone server at system startup and shutdown, an "init.d" script should be stored in "/etc/init.d" and also hardlinked into "/etc/rc2.d" (for startup) and into "/etc/rc0.d" (for shutdown).
An example file has been included below. To use it, change "/news/cyclone/bin/start" to the location of your "bin/start" script.
Save the modified file as "/etc/init.d/cyclone". Then hardlink it for
startup and shutdown with:
> ln /etc/init.d/cyclone /etc/rc2.d/S99cyclone
> ln /etc/init.d/cyclone /etc/rc0.d/K12cyclone
#!/sbin/sh
#
# Cyclone NewsRouter Startup/shutdown script - /etc/init.d/cyclone
#
case $1 in
'start')
if [ -f /news/cyclone/bin/start ]
then
echo "Starting Cyclone NewsRouter"
/news/cyclone/bin/start
fi
;;
'stop')
if [ -f /news/cyclone/bin/stop ]
then
echo "Stopping Cyclone NewsRouter"
/news/cyclone/bin/stop
fi
;;
'restart')
if [ -f /news/cyclone/bin/restart ]
then
echo "Restarting Cyclone NewsRouter"
/news/cyclone/bin/restart
fi
;;
*)
echo "usage: /etc/init.d/cyclone {start|restart|stop}"
;;
esac
Remember to define one or more "UpstreamHostnames" if you want your posts to propagate to other Usenet systems. This is especially true if you have "XrefAction" set to "Parse". Also remember to not put your own name in this list. The system will store locally posted articles when in "XrefAction Generate" mode, and when in "XrefAction Parse" mode, the master will re-propagate the article to you. Make sure that your master server is one of the hosts listed in the "UpstreamHostnames" list.
The post's "Newsgroups:" header might match an entry in the active file but not match a spool object subscription. Be sure you have at least one spool object with a "Subscription" of "*" (and a "FilterSubscription" of "!*") to ensure that all groups listed in the active file are stored.
Unless your machine experiences a catastrophic failure, Cyclone should never crash; in regular operation, Cyclone should never exit unless it is specifically stopped or killed. If your Cyclone server crashes or dies, contact HighWind immediately at <support@highwind.com>. In order to assist HighWind in tracking the problem as quickly as possible, please provide include in your email the hardware and software configuration (including the OS version) of the machine.
Cyclone handles crashes or unexpected shutdowns (such as power failures) very well. You should be able to simply restart the server and it will clean up any corrupt data. Sites running "-cache" might re-accept some duplicate articles received since the last history file write.
Note that copying over a binary that is currently running will cause a SIGBUS error and kill the server on Solaris. This is the standard operation for Solaris if a running binary is overwritten. Either stop the server, replace the binary and restart the server; or move the old binary, copy a new binary in, and restart the server to prevent this.
Solaris has a problem that will result in syslog() messages being "lost" if a large number of messages are logged at the same time. This usually does not occur, but can occasionally happen during system startup or during other periods of heavy logging.
Occasionally, a syslog appears of the form: