• Configuration Tutorial
  Provides step-by-step instructions on setting up a basic system configuration.
  
  
• Configuration File Specification
  The "final word" on system configuration.
  
  
• Advanced Configuration Examples
  Provides detailed examples of common advanced site configurations.
  
  
• GUI Configuration
  Provides instructions on setting up a Web-based GUI for your customers.
  
  

HighWind has tried to make the system very easy to configure. These simple step-by-step instructions will jump-start your site configuration.

1. Using the cyclone.conf-EXAMPLE file as a guide, create a cyclone.conf file in the root of the install area. In that file, create a number of "Spool" objects to hold articles. Each "Spool" object must have a SpoolName, a Path, and a KiloBytes directive. These directives specify a unique name, a location, and a size for each spool. We recommend placing a large "Spool" object on each disk in your system.

   For example, the following lines create a 1 GigaByte spool called "Main" that will be put in the file "/news/spool/bigSpool".

   <Spool>
   SpoolName Main
   Path /news/spool/bigSpool
   KiloBytes 1000000
   </Spool>
   

2. Using a text editor, create a feeds.conf file in the top of the install area. For each machine with which this system will exchange news, create an empty "Feed" Object. (Comment lines begin with a '#' character. Blank Lines are ignored.)
   

   <Feed>
   # First Machine
   </Feed>
   
   <Feed>
   # Second Machine
   </Feed>
   

3. For each machine that is feeding this system news, add an IncomingHostname directive.
   

   <Feed>
   # First Machine (Sends news to us)
   IncomingHostname first.machine.com
   </Feed>
   
   <Feed>
   # Second Machine (Doesn't send news to us)
   </Feed>
   

4. For each machine to which we are sending news, add an OutgoingHostname and a Subscription directive. The Subscription directive uses wildcard characters to specify which newsgroups should be sent to this machine.
   

   <Feed>
   # First Machine (Sends and Receives news to and from us)
   IncomingHostname first.machine.com
   
   # first.machine.com gets "Everything but the alt hierarchy"
   OutgoingHostname first.machine.com
   Subscription *, !alt.*
   </Feed>
   
   <Feed>
   # Second Machine (Receives news from us)
   
   # second.machine.com gets "Just the comp and rec hierarchies"
   OutgoingHostname second.machine.com
   Subscription comp.*, rec.*
   </Feed>
   

5. You are FINISHED! You just created a configuration file that set up a Bidirectional feed to first.machine.com and a Unidirectional feed to second.machine.com. All that is left is to run the "bin/start" command script to bootstrap the system.

   When you start Cyclone for the very first time, it will create the spools you have specified in the cyclone.conf file, do not be alarmed if this takes a significant amount of time.

At startup, the system reads the cyclone.conf and feeds.conf files located in the top of the install area.

In both files, Blank lines and lines beginning with a '#' character (except #include lines) are ignored. Entries in configuration files can be continued across multiple lines by ending the lines with the '\' character. In addition, the values "True" and "Yes" and "False" and "No" are case-insensitive and interchangeable.

The cyclone.conf file specifies one or more "spool" objects. These objects determine the size and location of the article spools. Each spool object contains directives indicating the properties of that spool.

On startup, Cyclone will allocate space and check the integrity of all spool objects. We recommend creating multiple spool objects with a total of at least 1 GigaByte of spool space. In addition, putting spool objects on separate disk drives will let Cyclone maximize performance by spreading the system load evenly across multiple drives.

<Spool> .... </Spool>

These are the spool delimiters that begin and end each spool object.

SpoolName

Specifies a unique name for this spool object. No two spools can have the same "SpoolName". No slashes or whitespace are allowed in this value.

Path path_to_file

Specifies the location of this spool object. The specified path can be absolute, or relative to the "bin" directory. This file will be created, expanded, and/or validated when the system starts up.

KiloBytes 1000000

Specifies the size in kilobytes of this spool object. Individual spool objects can be as small as 10000 (10 MegaBytes) and as large as 2,000,000 (2 GigaBytes).

Encoded False

Specifies whether the articles inside the spool objects should be stored in a "non-human-readable" form. This directive is optional and defaults to "False". Unless required by local authorities, HighWind Software recommends you don't enable this feature.

In addition to the "spool" objects, cyclone.conf specifies the DEFAULT values for directives that can appear in the feeds.conf file. The following directives are also present in the cyclone.conf file.

XrefAction

The XrefAction directive can be set to three values:

• Remove - Removes any "Xref:" header present in an article. This is the default and is suitable for most installations.
• Pass - Allows any "Xref:" header present in an article to pass through "untouched" by the system.
• Generate - Removes and Generates an "Xref:" header line with an entry for each newsgroup present in the Active File.

DropUnapprovedArticles

If the DropUnapprovedArticles directive is set to "True" any article posted to a moderated group (as specified in the Active File) will be dropped unless it has an Approved: header line.

DropFilteredArticles

If the DropFilteredArticles directive is set to "True" any article that is Filtered by a filtering program (See Custom Article Filtering) will be will be dropped. Otherwise, the article will be considered "junk" and will only be sent to feeds that have the AllowJunk directive set to "Yes".

The feeds.conf file contains one or more "feed" objects. These objects determine the sites with which this system will exchange news. Each feed object contains directives indicating the properties of that feed.

Since a "feed" can be unidirectional or bidirectional, the incoming and outgoing directions are specified separately. This allows you (in a single "logical feed") to receive and send news from different hosts.

<Feed> .... </Feed>

These are the feed delimiters that begin and end each feed object.

#include path_to_file

Specifies another feed configuration file to be loaded at this point in the file. The specified path can be absolute, or relative to the "bin" directory. (Full recursion is permitted).

IncomingHostname hostname.company.com

Specifies the hostname, IP address, or Glob-style Expression specifying a machine or (in the case of an expression) a collection of machines that is feeding this system news. No Default value can be supplied.

OutgoingHostname news.company.com

Specifies the hostname (or IP address) of the machine to which we are feeding news. You can have multiple outgoing feeds to the same host by creating multiple feed objects with identical "OutgoingHostname" values but different "OutgoingFeedName" values. No Default value can be supplied.

OutgoingFeedName feed_name

Specifies a unique name for this outgoing feed. This allows you to have more than one outgoing feed for a host. No two feeds can have the same "OutgoingFeedName". No slashes or whitespace are allowed in this value. The Default value is simply the value of the "OutgoingHostname" directive.

Subscription *, !alt.*, !soc.*

Simple Glob-style Expressions specifying the groups for this outgoing feed. The subscription begins "empty" and is built in the order expressions are specified. Groups are added to the subscription unless the expression begins with '!' which causes the matching groups to be removed from the subscription. The "DefaultSubscription" directive in the cyclone.conf file provides a default value for this directive. Review the following examples:

clari.*, comp.*, rec.*
	Will send all articles posted to the "clari", "comp",
	and "rec" hierarchies.

*, !alt.*, !soc.*
	Will send all articles except those posted to the "alt"
	and "soc" hierarchies.

comp.*, !comp.lang.*
	Will send all articles posted to the "comp" hierarchy except those 
	posted to the "comp.lang" hierarchy.

alt.*, !alt.binaries.*, alt.binaries.pictures.*
	Will send all articles posted to the "alt" hierarchy but only those
	posted to the "alt.binaries.pictures" hierarchy within the
	"alt.binaries" hierarchy.

The following characters are valid in each of the expressions:
	!     At the beginning of a pattern, inverses its meaning.
	*     Matches any string, including the empty string.
	?     Matches any single character.
	\     Quotes any of the above characters.

AdditionalSubscription !clari.*

Simple Glob-style Expressions that get appended to the Subscription directive. Since this directive is NOT accessible via the GUI, it is particularly useful for filtering out "premium" newsgroups. For example, setting this directive to "!clari.*" will prevent "clari" groups from being sent REGARDLESS of what the Subscription directive specifies. The "DefaultAdditionalSubscription" directive in the cyclone.conf file is optional and provides a default value for this directive.

FilterSubscription !*

Simple Glob-style Expressions specifying the groups on this outgoing feed that should be eliminated. If an article belongs to a group that matches this expression, it will be dropped REGARDLESS of whether it belongs to other groups. The "DefaultFilterSubscription" directive in the cyclone.conf file provides a default value for this directive.

SubjectFilter !*

Simple Glob-style Expressions specifying the subjects on this outgoing feed that should not be sent. If an article has a subject that matches this expression, it will not be sent. The "DefaultSubjectFilter" directive in the cyclone.conf file provides a default value for this directive. Please remember that these are case-sensitive Glob-style patterns. Spaces will not be interpreted.

Aliases news.company.com, news2.company.com, news3.company.com

Names all of the machines that are "equivalent" for this outgoing feed. If the "Path:" field of an article indicates that it has already been to one of these sites, it won't be sent to this feed. If you decide to use this directive, don't forget to include the value of "OutgoingHostname" in your list of machines. Please remember that the value for this directive is case-sensitive. The Default is the value of the "OutgoingHostname" directive.

NumberOfStreams 1

Number of simultaneous connections to use when feeding articles to this outgoing feed. This value can be set as high as 100. The "DefaultNumberOfStreams" directive in the cyclone.conf file provides a default value for this directive.

MaxIncomingNumberOfStreams 0

Maximum number of simultaneous incoming connections to allow on this incoming feed. Setting this value to 0 implies you do NOT want to impose a limit. The "DefaultMaxIncomingNumberOfStreams" directive in the cyclone.conf file provides a default value for this directive.

StampIncomingIPAddress False

Specifies whether or not the system should stamp the "Path:" header line with the IP address of the host sending articles. The "DefaultStampIncomingIPAddress" directive in the cyclone.conf file provides a default value for this directive.

IncomingExpectedPathStamp news.company.com

This directive is aimed at cutting down on FAKE "Path:" header lines. This optional directive specifies a list of strings that could appear as the first entry in the "Path:" header line in all articles on this feed. If an article arrives and has a "Path:" header line that doesn't begin with an entry from this list, the system will prepend the first entry in the list suffixed by the additional phrase ".MISMATCH". There is no default value for this directive.

MaxChecks 125

Upper limit on the number of NNTP "check" messages that can be sent before waiting for a response from this outgoing feed. This value can be set as high as 125. Cyclone will automatically regulate the actual number of "check" messages sent to maximize article transfer rates. The "DefaultMaxChecks" directive in the cyclone.conf file provides a default value for this directive.

DelayTime 0

Number of seconds to delay each article before transmission to a remote site. Although this value can be set as high as 86,400 (one day), it is usually MORE than adequate to use a value between 30 and 600 (30 seconds and 10 minutes). The "DefaultDelayTime" directive in the cyclone.conf file provides a default value for this directive.

MaxLength 524288

Maximum length (in bytes) of a valid article for this outgoing feed. Longer articles will not be sent. This value can be set as high as 4Mb (4194304 bytes). The "DefaultMaxLength" directive in the cyclone.conf file provides a default value for this directive.

MinLength 0

Minimum length (in bytes) of a valid article for this outgoing feed. Shorter articles will not be sent. This value can be set as high as 4Mb (4194304 bytes). The "DefaultMinLength" directive in the cyclone.conf file provides a default value for this directive.

MaxAge 21

Maximum Age (in days) of a valid article for this outgoing feed. Articles that are stamped with a "Date:" field older than this will not be sent. This value can be set as high as 1 year (365 days). The "DefaultMaxAge" directive in the cyclone.conf file provides a default value for this directive.

MinAge 0

Minimum Age (in days) of a valid article for this outgoing feed. Articles that are stamped with a "Date:" field younger than this will not be sent. This value can be set as high as 1 year (365 days). The "DefaultMinAge" directive in the cyclone.conf file provides a default value for this directive.

MaxHops 100

Maximum number of hosts a valid article for this outgoing feed can pass through (as indicated by the "Path:" header line). This value can be set as high as 1,000,000 hosts. The "DefaultMaxHops" directive in the cyclone.conf file provides a default value for this directive.

MinHops 0

Minimum number of hosts a valid article for this outgoing feed can pass through (as indicated by the "Path:" header line). This value can be set as high as 1,000,000 hosts. The "DefaultMinHops" directive in the cyclone.conf file provides a default value for this directive.

MaxConnectionDuration 86400

Maximum time (in seconds) to keep a connection open to the OutgoingHostName. The "DefaultMaxConnectionDuration" directive in the cyclone.conf file provides a default value for this directive. Please remember that the GUI uses MINUTES instead of seconds.

MaxDepth 10000

Maximum number of articles to keep in the queue for this outgoing feed. If the queue reaches MaxDepth, the oldest articles are removed to make room for newer articles. This value can be set as high as 2,000,000 articles. Regardless of the value set by this directive, the backlog capacity of the system is bounded by the spool capacity. The "DefaultMaxDepth" directive in the cyclone.conf file provides a default value for this directive.

We recommend increasing MaxDepth for HIGH-SPEED peers. Larger MaxDepth settings are useful for occasional network outages or slowdowns. See the Capacity Section of the Performance Guide for more details on setting MaxDepth.

MaxCrossPost 100

Maximum number of groups to which an article can belong for this outgoing feed. Articles posted to more than this number of groups will not be sent. The "DefaultMaxCrossPost" directive in the cyclone.conf file provides a default value for this directive.

PortNumber 119

Port to connect to for this outgoing feed. In general, you shouldn't have to change this from the RFC defined NNTP port (119). The "DefaultPortNumber" directive in the cyclone.conf file provides a default value for this directive.

OutgoingInterface news.company.com

Specifies the local hostname (or IP address) of the network interface to use for this outgoing feed. On single-homed machines (those with a single network interface), you shouldn't have to use this directive. However, if you have a multi-homed machine, this directive allows you to control which interface is used. No Default value can be supplied.

AllowStreaming Yes

Specifies whether or not the system should attempt to use efficient NNTP streaming transfers for this outgoing feed. The "DefaultAllowStreaming" directive in the cyclone.conf file provides a default value for this directive.

AllowTryAgain Yes

Specifies whether or not the system should attempt to resend articles that are temporarily refused by this outgoing feed (via NNTP 431 and 436 response codes). In addition, on incoming feeds, this specifies whether or not the system will send the 431 and/or 436 response codes to remote sites. The "DefaultAllowTryAgain" directive in the cyclone.conf file provides a default value for this directive.

AllowJunk No

If an Active File is present, this specifies whether or not the system should send articles that do not match the Active File to this outgoing feed. The "DefaultAllowJunk" directive in the cyclone.conf file provides a default value for this directive.

MaxBytesPerSecond 0

Maximum speed to send news in BYTES per second for this outgoing feed. This limit is enforced by calculating an average of the transmission speed. Setting this value to 0 implies you do NOT want to impose a limit on the transmission speed. The "DefaultMaxBytesPerSecond" directive in the cyclone.conf file provides a default value for this directive.

IncomingSubjectFilter !*

Simple Glob-style Expressions specifying the subjects on this incoming feed that should be eliminated. If an article has a subject that matches this expression, it will be dropped. The "DefaultIncomingSubjectFilter" directive in the cyclone.conf file provides a default value for this directive. Please remember that these are case-sensitive Glob-style patterns. Spaces will not be interpreted.

IncomingGroupFilter !*

Simple Glob-style Expressions specifying the groups on this incoming feed that should be eliminated. If an article belongs to a group that matches this expression, it will be dropped REGARDLESS of whether it belongs to other groups. The "DefaultIncomingGroupFilter" directive in the cyclone.conf file provides a default value for this directive.

IncomingPathFilter !*

Simple Glob-style Expressions specifying the hostnames on this incoming feed that should be eliminated. If a hostname appears in an article's Path: header line that matches this expression, it will be dropped immediately. The "DefaultIncomingPathFilter" directive in the cyclone.conf file provides a default value for this directive.

Distribution world,usa,na,ne

Simple Glob-style Expressions specifying the allowed values for the distribution field in articles for this outgoing feed. Articles that arrive with no specification of their distribution will be treated as if they had a distribution of "world". The "DefaultDistribution" directive in the cyclone.conf file provides a default value for this directive.

BackupHostName main-feeder.company.com
BackupPortNumber 119
BackupTimeInterval 600

Specifies the hostname (or IP address) and Port to monitor for this outgoing feed. If a connection can not be established to this host and port, Cyclone will begin transmitting data to the host specified by the OutgoingHostName directive. Once a connection can be established, Cyclone will stop transmitting data. A connection is attempted every "BackupTimeInterval" seconds. No Default value can be supplied.

This feature is VERY useful when implementing fault-tolerant News systems. You can have secondary machines monitor your primary machines and only activate feeds if the primary machines become unreachable.

AuthInfoUser username

Username to use for "AuthInfo style" NNTP authentication for this outgoing feed. Do NOT use this directive unless authentication is required. No Default value can be supplied.

AuthInfoPass password

Password to use for "AuthInfo style" NNTP authentication for this outgoing feed. Do NOT use this directive unless authentication is required. No Default value can be supplied.

• Standard Article Filtering

• Custom Article Filtering (ANTI-SPAM)

• Preventing FAKED Articles

• Sites with Multiple NewsServers

• Redundant Feeds and Disaster Recovery

• Large Configurations

• Article Routing and Appearance of your News Network

Review the following examples:

On startup, the system will spawn a copy of the program specified by the "-program" option. When an article is received and ready to be considered for outgoing feeds, a copy of the header lines terminated by a line containing ".\r\n" will be sent to the standard input of the filtering program.

After the header is sent to the filtering program, the system will attempt to read 5 bytes from the standard output of the filtering program. If the 5 bytes are "335\r\n", the article will be accepted. If the 5 bytes are "435\r\n", the article will be discarded.

It is important to remember that the filtering program should NOT exit. In addition, since the system waits for the filtering program to make a decision about each article, speed is very important.

Here is a simple example of a filtering program. Although this example is written in PERL, filtering programs can be written in any language. This example is included as sample_filter.pl in the bin directory.

#!/usr/local/bin/perl
#
$| = 1; # Flush STDOUT
$good_article = "335\r\n";
$bad_article = "435\r\n";

# Initialize
$result = $good_article;

# Loop on input
while($line = <>) {
    # If the Subject contains "MAKE MONEY FAST", reject the article.
    if ($line =~ /^Subject:/) {
	if ($line =~ /MAKE MONEY FAST/) {
	    $result = $bad_article;
	} else {
	    $result = $good_article;
	}
    }

    if ($line eq ".\r\n") { print $result; }
}

If you are feeding a site that has multiple newsservers, you will want to specify all the newsservers at that site in the Aliases directive. This will avoid the unnecessary transmission of articles to that site by treating all the newsservers at that site as being "equivalent" to the host you are actually feeding.

In the following example, "company.com" has 3 newsservers ("news1.company.com", "news2.company.com", and "news3.company.com"). We are tasked with receiving articles from "news1.company.com" and sending the "comp" newsgroups to "news2.company.com" that have "world" or "usa" distribution.

By deploying multiple machines, you can provide a high level of redundancy, failover, and disaster recovery. By feeding each of your customers with a primary feed and one or more secondary feeds from alternate sites, you can provide uninterrupted service even in the face of catastrophic failures. Cyclone provides two ways to do this efficiently:

• By using the DelayTime directive, you can "time-shift" your secondary feeds to avoid having your machines "race" each other to the customer. Because secondary feeds are "delayed", the primary feed provides the articles to the customer while secondary feeds get "Refused". Only if the primary feed becomes unreachable will secondary feed articles be accepted. Because secondary feeds are "delayed", it is a good idea to give them a much larger MaxDepth than that of the primary feed.

• For even greater bandwidth efficiency, you can use the BackupHostName, BackupPortNumber, and BackupTimeInterval directives to monitor your primary feeds. Doing this will cause NO traffic from the secondary sites unless the primary site becomes unreachable.

In the following example, we show an example of how to set up a primary and secondary feeds to "news.company.com".

If your machine feeds many sites, or your site configuration is being managed by more than one administrator, it can be desirable to partition the configuration across multiple files. This can be done via the #include directive.

In the following example, we show a sample feeds.conf file that uses multiple #include directives to include many configuration files. In turn, these configuration files can include other configuration files. Using this technique, the system configuration can be delegated to files under the control of different administrators.

Large sites generally have a collection of interconnected News machines. By default, articles passing through any of these systems will have their Path: header line augmented with the Fully Qualified Domain Name of the local host. Since the presence of a hostname in the Path: header line effects the routing of an article, controlling what goes into that line is sometimes desirable. Administrators have two options for controlling the way Cyclone updates the Path: line:

-masquerade: This option to cycloned lets the administrator use an argument name instead of the Fully Qualified Domain Name in the Path: header line.

-alias: This option to cycloned gives the administrator a way to add an argument name to the Path: header line if it isn't already present. This option is usually used to add a common name that represents your collection of news machines.

It is worth noting that these options are simple but very powerful. Clever use of these options (including the use of multiple names separated by '!' characters) allows you to optimize the routing in your news network in ways that are not immediately obvious.

Contact HighWind Support if you would like additional assistance or advice in using these options.

• Step-by-Step GUI Configuration

• Advanced GUI Configuration Options

These instructions will guide you through the steps necessary to set up a Web-based GUI control page for each of your customers.

1. Using the #include directive, organize your Cyclone configuration file into a single tier hierarchy. That is, have a master feeds.conf file that uses the #include directive to include slave feed configuration files.

   By using this simple hierarchy, you can isolate any customer's portion of the system configuration to a single configuration file.

2. Identify or install a CGI capable Web Server on a machine that has access to the Cyclone configuration file hierarchy created in the previous step. Ensure that the Web Server process has write permissions to the slave configuration files mentioned in the previous step. See the Advanced Section for other alternatives.

3. Read and follow the documentation inside the modifyfeed.conf file. These instructions will give you details on how you can control the actions of your customer's GUI.

4. Once this is complete, you should setup a cron(1M) entry to run the Cyclone bin/restart command script at regular intervals so that changes made through the GUI will take effect. We suggest a 12 hour interval.

The cgi-bin/modifyfeed.cgi program does not require a locally installed Cyclone system. For that reason, it can be installed on any Web Server on any machine on your network.

By exploiting this feature, you can use a central Web Server to manage all the configuration files for every Cyclone system on your network. If you do this, you can use rdist(1) or a similar system to distribute the resulting configuration files out to your Cyclone machines.

If you choose to rdist(1) your configuration files out to remote Cyclone machines, we suggest, in addition to using cron, modifying the bin/restart command script to only restart the system if the configuration files have changed. By taking the small amount of time to set this up, you can remotely control dozens of Cyclone machines from a central console.

To protect the configuration of your Cyclone system, we recommend using Web-based password protection on each of your client Web subdirectories. In addition, using SSL or similar technology will provide further protection from unauthorized access.

Back to Cyclone Documentation