Characters in the ISO 8859-1 (Latin 1) character set are mapped to their closest ASCII equivalent before further examination and indexing. (Individual indexing modules may also do their own character mapping.)

Stop words, words that occur too frequently or have no information content, are not indexed. (There is a default built-in set of a few hundred such words.) Additionally, several heuristics are used to determine which words should not be indexed.

First, a word is checked to see if it looks like an acronym. A word is considered an acronym only if it starts with a capital letter and is composed exclusively of capital letters, digits, and punctuation symbols, e.g., ``AT&T.'' If a word looks like an acronym, it is indexed and no further checks are done.

Second, there are several other checks that are applied. A word is not indexed if it:

1.

Is less than CWWord_Min_Size letters. (Default is 4.)

2.

Contains less than CWWord_Min_Vowels vowels. (Default is 1.)

3.

Contains more than CWWord_Max_Consec_Same of the same character consecutively (not including digits). (Default is 2.)

4.

Contains more than CWWord_Max_Consec_Consonants consecutive consonants. (Default is 5.)

5.

Contains more than CWWord_Max_Consec_Vowels consecutive vowels. (Default is 4.)

6.

Contains more than CWWord_Max_Consec_Puncts consecutive punctuation characters. (Default is 1.)

Via the FilterFile configuration file variable, files matching particular patterns can be filtered prior to indexing. Via the FilterAttachment configuration file variable, e-mail attachments whose MIME types match particular patterns can be filtered prior to indexing. (See FILTERS in swish++.conf(4).)

In order to add words from new documents to an existing index++, either the entire set of documents can be reindexed or the new documents alone can be incrementally indexed. In many cases, reindexing everything is sufficient since index++ is really fast. For a very large document set, however, this may use too many resources.

However, there is a pitfall for incremental indexing: if any of the -f, --word-files, -p, or --word-percent options or WordFilesMax or WordPercentMax variables are used, then words that are too frequent are discarded. If new documents are added containing very few of those words, then they could no longer be too frequent. However, there is no way to get them back since they were discarded.

The way around this problem is not to discard any words by specifying 101%. However, because no words are discarded, the size of the index file will be larger, perhaps significantly so.

It is possible that, in practice, the loss of words may not be that important especially if new documents are very similar to old documents and that words that were too frequent in the old set would also be too frequent in new set.

Another way around this problem is to do periodic full indexing.

This module simply indexes plain text files performing character mapping and word determination as has already been described.

Additional processing is done for HTML and XHTML files. The additional processing is:

1.

Character and numeric (decimal and hexadecimal) entity references are converted to their ASCII character equivalents before further examination and indexing. For example, ``résumé'' becomes ``resume'' before indexing.

2.

If a matched set of CW<TITLE> ... CW</TITLE> tags is found within the first CWTitleLines lines of the file (default is 12), then the text between the tags is stored in the generated index file as the file's title rather than the file's name. (Every non-space whitespace character in the title is converted to a space; leading and trailing spaces are removed.)

3.

If an HTML or XHTML element contains a CWCLASS attribute whose value is among the set of class names specified as those not to index (via one or more of either the -C or --no-class option or the ExcludeClass variable), then all the text up to the tag that ends the element will not be indexed.

For an element that has an optional end tag, ``the tag that ends the element'' is either the element's end tag or a tag of another element that implicitly ends it; for an element that does not have an end tag, ``the tag that ends the element'' is the element's start tag. (See the EXAMPLES.)

All elements from the HTML 4.0 specification (including deprecated elements), Ruby elements, plus common, browser-specific elements are recognized; unrecognized elements are ignored. (See the -H or --dump-html option.)

4.

If an HTML or XHTML element contains a CWTITLE attribute, then the words specified as the value of the CWTITLE attribute are indexed.

5.

If an CWAREA, CWIMG, or CWINPUT element contains an CWALT attribute, then the words specified as the value of the CWALT attribute are indexed.

6.

If a CWMETA element contains both a CWNAME and CWCONTENT attribute, then the words specified as the value of the CWCONTENT attribute are indexed associated with the meta name specified as the value of the CWNAME attribute.

(However, if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words specified as the value of the CWCONTENT attribute are still indexed, but not associated with the meta name.)

(See also the -m, --meta, -M, and --no-meta options or the IncludeMeta or ExcludeMeta variables.) Meta names can later be queried against specifically using search++(1).

7.

If a CWTABLE element contains a CWSUMMARY attribute, then the words specified as the value of the CWSUMMARY attribute are indexed.

8.

If an CWOBJECT element contains a CWSTANDBY attribute, then the words specified as the value of the CWSTANDBY attribute are indexed.

9.

All other HTML or XHTML tags and comments (anything between CW< and CW> characters) are discarded.

In compliance with the HTML specification, any one of no quotes, single quotes, or double quotes may be used to contain attribute values and attributes can appear in any order. Values containing whitespace, however, must be quoted. The specification is vague as to whether whitespace surrounding the CW= is legal, but index++ allows it.

ID3 tags are used to store audio meta information for MP3 files (generally). Since audio files contain mostly binary information, only the ID3 tag text fields are indexed. ID3 tag versions 1.x and 2.x (through 2.4) are supported (except for encrypted frames). If a file contains both 1.x and 2.x tags, only the 2.x tag is indexed. The processing done for files containing an ID3 tag is:

1.

If a title field is found, then the value of the title is stored in the generated index file as the file's title rather than the file's name. (Every non-space whitespace character in the title is converted to a space; leading and trailing spaces are removed.)

2.

Words that are the value of fields are indexed associated with the field name as a meta name. (However, if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words specified as the value of the field are still indexed, but not associated with the field.)

(See also the -m, --meta, -M, and --no-meta options or the IncludeMeta or ExcludeMeta variables.) Meta names can later be queried against specifically using search++(1).

For ID3v1.x, the recommended fields to be indexed are: album, artist, comments, genre, and title.

For ID3v2.2, the recommended text fields (with reassignments) to be indexed are: com=comments, tal=album, tcm=composer, tco=genre, tcr=copyright, ten=encoder, txt=lyricist, tt1=content, tt2=title, tt3=subtitle, ipl=musicians, tot=original-title, tol=original-lyricist, toa=original-artist, tp1=artist, tp2=performers, tp3=conductor, tpb=publisher, txx=user, slt=lyrics, and ult=lyrics.

For ID3v2.4, the recommended text fields (with reassignments) to be indexed are: comm=comments, talb=album, tcom=composer, tcon=genre, tcop=copyright, tenc=encoder, text=lyricist, tipl=people, tit1=content, tit2=title, tit3=subtitle, tmcl=musicians, tmoo=mood, toal=original-title, toly=original-lyricist, tope=original-artist, town=owner, tpe1=artist, tpe2=performers, tpe3=conductor, tpub=publisher, tsst=set-subtitle, txxx=user, user=terms, sylt=lyrics, and uslt=lyrics.

ID3v2.3 is the same as 2.4 except replace tmcl=musicians with ipls=musicians.

All text fields (with reassignments) for all versions of ID3 can (and should) be specified concurrently so it need not be known in advance which version(s) of ID3 MP3 files are encoded with.

3.

For ID3v2.x, text fields that are compressed are uncompressed prior to indexing.

4.

For ID3v2.x, Unicode text that is encoded in either UTF-8 or UTF-16 (either big- or little-endian) is decoded prior to indexing.

Additional processing is done for LaTeX files. If a \\title command is found within the first CWTitleLines lines of the file (default is 12), then the value of the title is stored in the generated index file as the file's title rather than the file's name. (Every non-space whitespace character in the title is converted to a space; leading and trailing spaces are removed.)

Additional processing is done for mail and news files. The additional processing is:

1.

If a Subject header is found within the first CWTitleLines lines of the file (default is 12), then the value of the subject is stored in the generated index file as the file's title rather than the file's name. (Every non-space whitespace character in the title is converted to a space; leading and trailing spaces are removed.)

2.

Words that are the value of a header are indexed associated with the header name as a meta name. (However, if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words specified as the value of the header are still indexed, but not associated with the header.)

(See also the -m, --meta, -M, and --no-meta options or the IncludeMeta or ExcludeMeta variables.) Meta names can later be queried against specifically using search++(1).

The recommended headers to be indexed are: Bcc, Cc, Comments, Content-Description, From, Keywords, Newsgroups, Resent-To, Subject, and To.

3.

MIME attachments are indexed.

4.

Text that is in the text/enriched content type is converted to plain text prior to indexing.

5.

Text that is encoded as either quoted-printable or base-64 is decoded prior to indexing.

6.

Unicode text that is encoded in either the UTF-7 or UTF-8 character set is decoded prior to indexing.

7.

Text in vCards is indexed such that the values of types (fields) are associated with the types as meta names.

The recommended vCard types (with reassignments) to be indexed are: adr=address, categories, class, label=address, fn=name, nickname, note, org, role, and title.

Indexing mail and news files is most effective only when there is exactly one message per file. While Usenet news files are usually this way, mail files are not. Mail files, e.g., mailboxes, are usually comprised of multiple messages. Such files would need to be split up into files of individual messages prior to indexing since there's no point in indexing a single mailbox: every search result would return a rank of 100 for the same file. Therefore, the splitmail++(1) utility is included in the SWISH++ distribution.

Additional processing is done for Unix manual page files. The additional processing is:

1.

If a NAME section heading macro (CW.SH) is found within the first CWTitleLines lines of the file (default is 12), then the contents of the next line are stored in the generated index file as the file's title rather than the file's name. (Every non-space whitespace character in the title is converted to a space; leading and trailing spaces as well as backslash sequences, such as CW\f2, are removed.)

2.

Words that are in a section are indexed associated with the name of the section as a meta name. (However, if either the -A or --no-assoc-meta options or the AssociateMeta variable is specified, then the words in a section are still indexed, but not associated with the section heading.)

Spaces in multi-word section headings are converted to dashes, e.g., ``see also'' becomes ``see-also'' as a meta name. (See also the -m, --meta, -M, and --no-meta options or the IncludeMeta or ExcludeMeta variables.) Meta names can later be queried against specifically using search++(1).

The recommended sections to be indexed are: AUTHOR, BUGS, CAVEATS, DESCRIPTION, DIAGNOSTICS, ENVIRONMENT, ERRORS, EXAMPLES, EXIT-STATUS, FILES, HISTORY, NAME, NOTES, OPTIONS, RETURN-VALUE, SEE-ALSO, SYNOPSIS, and WARNINGS.

This module simply indexes rich text format files without all formatting commands.

All these example assume you change your working directory to your web server's document root prior to indexing.

To index all HTML and text files on a web server: index++ -v3 -e 'html:*.*htm*' -e 'text:*.txt' . To index all files not under directories named CWCVS: find . -name CVS -prune -o -type f -a -print | index++ -e 'html:*.*htm*' -

When using the Windows command interpreter, single quotes around filename patterns don't work; you must use double quotes: index++ -v3 -e "html:*.*htm*" -e "text:*.txt" . This is a problem with Windows, not SWISH++. (Double quotes will also work under Unix.)

In an HTML or XHTML document, there may be sections that should not be indexed. For example, if every page of a web site contains a navigation menu such as: <SELECT NAME="menu"> <OPTION>Home <OPTION>Automotive <OPTION>Clothing <OPTION>Hardware </SELECT> or a common header and footer, then, ordinarily, those words would be indexed for every page and therefore be discarded because they would be too frequent. However, via either the -C or --no-class option or the ExcludeClass variable, one or more class names can be specified and then HTML or XHTML elements belonging to one of those classes will not have the text up to the tag that ends them indexed. Given a class name of, say, CWno_index, the above menu can be changed to: <SELECT NAME="menu" CLASS="no_index"> and then everything up to the CW</SELECT> tag will not be indexed.

For an HTML element that has an optional end tag (such as the CW<P> element), the text up to the tag that ends it will not be indexed, which is either the element's own end tag or a tag of some other element that implicitly ends it. For example, in: <P CLASS="no_index"> This was the poem that Alice read: <BLOCKQUOTE> <B>Jabberwocky</B><BR> `Twas brillig, and the slithy toves<BR> Did gyre and gimble in the wabe;<BR> All mimsy were the borogoves,<BR> And the mome raths outgrabe. </BLOCKQUOTE> the CW<BLOCKQUOTE> tag implicitly ends the CW<P> element (as do all block-level elements) so the only text that is not indexed above is: ``This was the poem that Alice read.''

For an HTML or XHTML element that does not have an end tag, only the text within the start tag will not be indexed. For example, in: <IMG SRC="home.gif" ALT="Home" CLASS="no_index"> the word ``Home'' will not be indexed even though it ordinarily would have been if the CWCLASS attribute were not there.

(See Filters under EXAMPLES in swish++.conf(4).)