hunspell - format of Hunspell dictionaries and affix files

Hunspell(1) requires two files to define the language that it is spellchecking. The first file is a dictionary containing words for the language, and the second is an "affix" file that defines the meaning of special flags in the dictionary.

A dictionary file (*.dic) contains a list of words, one per line. The first line of the dictionaries (except personal dictionaries) contains the word count. Each word may optionally be followed by a slash ("/") and one or more flags, which represents affixes or special attributes. Default flag format is a single (usually alphabetic) character. In a Hunspell dictionary file, there is also an optional morphological field separated by tabulator. Morphological desciptions have custom format.

An affix file (*.aff) may contain a lot of optional attributes. For example, SET is used for setting the character encodings of affixes and dictionary files. TRY sets the change characters for suggestions. REP sets a replacement table for multiple character corrections in suggestion mode. PFX and SFX defines prefix and suffix classes named with affix flags.

The following affix file example defines UTF-8 character encoding. `TRY' suggestions differ from the bad word with an English letter or an apostrophe. With these REP definitions, Hunspell can suggest the right word form, when the misspelled word contains f instead of ph and vice versa.

SET UTF-8
TRY esianrtolcdugmphbyfvkwzESIANRTOLCDUGMPHBYFVKWZ'

REP 2 REP f ph REP ph f

PFX A Y 1 PFX A 0 re .

SFX B Y 2 SFX B 0 ed [^y] SFX B y ied y

There are two affix classes in the dictionary. Class A defines an `re-' prefix. Class B defines two `-ed' suffixes. First suffix can be added to a word if the last character of the word isn't `y'. Second suffix can be added to words terminated with an `y'. (See details later.) The following dictionary file uses these affix classes.

3
hello
try/B
work/AB

All accepted words with this example: hello, try, tried, work, worked, rework, reworked.

SET encoding

Set character encoding of words and morphemes in affix and dictionary files. Possible values: UTF-8, ISO8859-1 - ISO8859-10, ISO8859-13 - ISO8859-15, KOI8-R, KOI8-U, microsoft-cp1251, ISCII-DEVANAGARI.

FLAG value

Set flag type. Default type is the extended ASCII (8-bit) character. `UTF-8' parameter sets UTF-8 encoded Unicode character flags. The `long' value sets the double extended ASCII character flag type, the `num' sets the decimal number flag type. Decimal flags numbered from 1 to 65535, and in flag fields are separated by comma.

COMPLEXPREFIXES

Set twofold prefix stripping (but single suffix stripping) for agglutinative languages with right-to-left writing system.

LANG langcode

Set language code. In Hunspell may be language specific codes enabled by LANG code. At present there are az_AZ, de_DE, hu_HU, TR_tr specific codes in Hunspell (see the source code).

TRY characters

Hunspell can suggest right word forms, when those differs from the bad form by one TRY character. The parameter of TRY is case sensitive.

NOSUGGEST flag

Words signed with NOSUGGEST flag are not suggested. Proposed flag for vulgar and obscene words.

MAXNGRAMSUGS num

Set number of n-gram suggestions. Value 0 switches off the n-gram suggestions.

NOSPLITSUGS

Disable split-word suggestions.

SUGSWITHDOTS

Add dot(s) to suggestions, if input word terminates in dot(s). (Not for OpenOffice.org dictionaries, because OpenOffice.org has an automatic dot expansion mechanism.)

REP number_of_replacement_definitions

REP what replacement

We can define language-dependent phonetic information in the affix file (.aff) by a replacement table. First REP is the header of this table and one or more REP data line are following it. With this table, Hunspell can suggest the right forms for the typical faults of spelling when the incorrect form differs by more, than 1 letter from the right form. For example a possible English replacement table definition to handle misspelled consonants:

REP 8
REP f ph
REP ph f
REP f gh
REP gh f
REP j dg
REP dg j
REP k ch
REP ch k

Replacement table is also usable in robust morphological analysis (accepting bad forms) or stricter compound word support in spell checking (forbidding generated compound words, if they are also simple words with typical fault).

MAP number_of_map_definitions

MAP string_of_related_chars

We can define language-dependent information on characters that should be considered related (ie. nearer than other chars not in the set) in the affix file (.aff) by a character map table. With this table, Hunspell can suggest the right forms for words, which incorrectly choose the wrong letter from a related set more than once in a word.

For example a possible mapping could be for the German umlauted ü versus the regular u; the word Frühstück really should be written with umlauted u's and not regular ones

MAP 1
MAP uü

COMPOUND number_of_compound_definitions

COMPOUND compound_pattern

Define custom compound patterns with a regex-like syntax. The first COMPOUND is a header with the number of the following COMPOUND definitions. Compound patterns consist compound flags and star or question mark meta characters. A flag followed by a `*' matches a word sequence of 0 or more matches of words signed with this compound flag. A flag followed by a `?' matches a word sequence of 0 or 1 matches of a word signed with this compound flag. See tests/compound*.* examples. Note: `*' and `?' metacharacters work only with the default 8-bit character and the UTF-8 FLAG types. Note II: COMPOUND flags haven't been compatible with the COMPOUNDFLAG, COMPOUNDBEGIN, etc. compound flags yet (use these flags on different words).

COMPOUNDMIN num

Minimum length of words in compound words. Default value is 3 letters.

COMPOUNDFLAG flag

Words signed with COMPOUNDFLAG may be in compound words (except when word shorter than COMPOUNDMIN). Affixes with COMPOUNDFLAG also permits compounding of affixed words.

COMPOUNDBEGIN flag

Words signed with COMPOUNDFLAG (or with a signed affix) may be first elements in compound words.

COMPOUNDLAST flag

Words signed with COMPOUNDFLAG (or with a signed affix) may be last elements in compound words.

COMPOUNDMIDDLE flag

Words signed with COMPOUNDFLAG (or with a signed affix) may be middle elements in compound words.

ONLYINCOMPOUND flag

Suffixes signed with ONLYINCOMPOUND flag may be only inside of compounds (Fuge-elements in German, fogemorphemes in Swedish). ONLYINCOMPOUND flag works also with words (see tests/onlyincompound.*).

COMPOUNDPERMITFLAG flag

Prefixes are allowed at the beginning of compounds, suffixes are allowed at the end of compounds by default. Affixes with COMPOUNDPERMITFLAG may be inside of compounds.

COMPOUNDFORBIDFLAG flag

Suffixes with this flag forbid compounding of the affixed word.

COMPOUNDROOT flag

COMPOUNDROOT flag signs the compounds in the dictionary (Now it is used only in the Hungarian language specific code).

COMPOUNDWORDMAX number

Set maximum word count in a compound word. (Default is unlimited.)

CHECKCOMPOUNDDUP

Forbid word duplication in compounds (eg. foofoo).

CHECKCOMPOUNDREP

Forbid compounding, if the (usually bad) compound word may be a non compound word with a REP fault. Useful for languages with `compound friendly' orthography.

CHECKCOMPOUNDCASE

Forbid upper case characters at word bound in compounds.

CHECKCOMPOUNDTRIPLE

Forbid compounding, if compound word contains triple letters (eg. foo|ox or xo|oof). Bug: missing multi-byte character support in UTF-8 encoding (works only for 7-bit ASCII characters).

CHECKCOMPOUNDPATTERN number_of_checkcompoundpattern_definitions

CHECKCOMPOUNDPATTERN endchars beginchars

Forbid compounding, if first word in compound ends with endchars, and next word begins with beginchars.

COMPOUNDSYLLABLE max_syllable vowels

Need for special compounding rules in Hungarian. First parameter is the maximum syllable number, that may be in a compound, if words in compounds are more than COMPOUNDWORDMAX. Second parameter is the list of vowels (for calculating syllables).

SYLLABLENUM flags

Need for special compounding rules in Hungarian.

PFX flag cross_product number

PFX flag stripping prefix condition morphological_description

SFX flag cross_product number

SFX flag stripping suffix condition morphological_description

An affix is either a prefix or a suffix attached to root words to make other words. We can define affix classes with arbitrary number affix rules. Affix classes are signed with affix flags. The first line of an affix class definition is the header. The fields of an affix class header:

(0) Option name (PFX or SFX)

(1) Flag (name of the affix class)

(2) Cross product (permission to combine prefixes and suffixes). Possible values: Y (yes) or N (no)

(3) Line count of the following rules.

Fields of an affix rules:

(0) Option name

(1) Flag

(2) stripping characters from beginning (at prefix rules) or end (at suffix rules) of the word

(3) affix (optionally with flags of continuation classes, separated by a slash)

(4) condition.

Zero stripping or affix are indicated by zero. Zero condition is indicated by dot. Condition is a simplified, regular expression-like pattern, which must be met before the affix can be applied. (Dot signs an arbitrary character. Characters in braces sign an arbitrary character from the character subset. Dash hasn't got special meaning, but circumflex (^) next the first brace sets the complementer character set.)

(5) Custom morphological description.

CIRCUMFIX flag

Affixes signed with CIRCUMFIX flag may be on a word when this word also has a prefix with CIRCUMFIX flag and vice versa.

FORBIDDENWORD flag

This flag signs forbidden word form. Because affixed forms are also forbidden, we can substract a subset from set of the accepted affixed and compound words.

KEEPCASE flag

Forbid capitalized and uppercased forms of words signed with KEEPCASE flags. Useful for special ortographies (measurements and currency often keep their case in uppercased texts) and writing systems (eg. keeping lower case of IPA characters).

LEMMA_PRESENT flag

Generally, there are dictionary words as lemmas in output of morphological analysis. Sometimes dictionary words are not lemmas, but affixed (not real) stems and virtual stems. In this case lemmas (real stems) need to put into morphological description, and forbid not real lemmas in morphological analysis adding LEMMA_PRESENT flag to dictionary words.

NEEDAFFIX flag

This flag signs virtual stems in the dictionary. Only affixed forms of these words will be accepted by Hunspell. Except, if the dictionary word has a homonym or a zero affix. NEEDAFFIX works also with prefixes and prefix + suffix combinations (see tests/pseudoroot5.*).

PSEUDOROOT flag

Deprecated. (Former name of the NEEDAFFIX option.)

WORDCHARS characters

WORDCHARS extends tokenizer of Hunspell command line interface with additional word character. For example, dot, dash, n-dash, numbers, percent sign are word character in Hungarian. (This attribute haven't worked with UTF-8 encoding yet.)

Hunspell's affix rules have got an optional morphological description field. There is a similar optional field in dictionary file, separated by tabulator:

 word/flags    morphology

We define a simple resource with morphological informations.

Affix file:

 SFX X Y 1
 SFX X 0 able . +ABLE

Dictionary file:

 drink/X   [VERB]

Test file:

 drink
 drinkable

Test:

 $ hunmorph test.aff test.dic test.txt
 drink:     drink[VERB]
 drinkable: drink[VERB]+ABLE

You can see in the example, that the analyzer concatenates the morphological fields in item and arrangement style.

Ispell's original algorithm strips only one suffix. Hunspell can strip another one yet.

The twofold suffix stripping is a significant improvement in handling of immense number of suffixes, that characterize agglutinative languages.

Extending the previous example by adding a second suffix (affix class Y will be the continuation class of the suffix `able'):

 SFX Y Y 1
 SFX Y 0 s . +PLUR

SFX X Y 1 SFX X 0 able/Y . +ABLE

Dictionary file:

 drink/X   [VERB]

Test file:

 drink
 drinkable
 drinkables

Test:

 $ hunmorph test.aff test.dic test.txt
 drink:      drink[VERB]
 drinkable:  drink[VERB]+ABLE
 drinkables: drink[VERB]+ABLE+PLUR

Theoretically with the twofold suffix stripping needs only the square root of the number of suffix rules, compared with a Hunspell implementation. In our practice, we could have elaborated the Hungarian inflectional morphology with twofold suffix stripping. (Note: In Hunlex preprocessor's grammar can be use not only twofold, but multiple suffix slitting.)

Hunspell can handle more than 65000 affix classes. There are two new syntax for giving flags in affix and dictionary files.

FLAG long command sets 2-character flags:

  FLAG long
  SFX Y1 Y 1
  SFX Y1 0 s 1

Dictionary record with the Y1, Z3, F? flags:

  foo/Y1Z3F?

FLAG num command sets numerical flags separated by comma:

  FLAG num
  SFX 65000 Y 1
  SFX 65000 0 s 1

Dictionary example:

  foo/65000,12,2756

Hunspell's dictionary can contain repeating elements that is homonyms:

 work/A    [VERB]
 work/B    [NOUN]

An affix file:

 SFX A Y 1
 SFX A 0 s . +SG3

SFX B Y 1 SFX B 0 s . +PLUR

Test file:

 works

Test:

 > works
 work[VERB]+SG3
 work[NOUN]+PLUR

This feature also gives a way to forbid illegal prefix/suffix combinations in difficult cases.

An interesting side-effect of multi-step stripping is, that the appropriate treatment of circumfixes now comes for free. For instance, in Hungarian, superlatives are formed by simultaneous prefixation of leg- and suffixation of -bb to the adjective base. A problem with the one-level architecture is that there is no way to render lexical licensing of particular prefixes and suffixes interdependent, and therefore incorrect forms are recognized as valid, i.e. *legvén = leg + vén `old'. Until the introduction of clusters, a special treatment of the superlative had to be hardwired in the earlier HunSpell code. This may have been legitimate for a single case, but in fact prefix--suffix dependences are ubiquitous in category-changing derivational patterns (cf. English payable, non-payable but *non-pay or drinkable, undrinkable but *undrink). In simple words, here, the prefix un- is legitimate only if the base drink is suffixed with -able. If both these patters are handled by on-line affix rules and affix rules are checked against the base only, there is no way to express this dependency and the system will necessarily over- or undergenerate.

In next example, suffix class R have got a prefix `continuation' class (class P).

PFX P Y 1
PFX P   0 un . [prefix_un]+

SFX S Y 1 SFX S 0 s . +PL

SFX Q Y 1 SFX Q 0 s . +3SGV

SFX R Y 1 SFX R 0 able/PS . +DER_V_ADJ_ABLE

Dictionary:

2
drink/RQ	[verb]
drink/S	[noun]

Morphological analysis:

> drink
drink[verb]
drink[noun]
> drinks
drink[verb]+3SGV
drink[noun]+PL
> drinkable
drink[verb]+DER_V_ADJ_ABLE
> drinkables
drink[verb]+DER_V_ADJ_ABLE+PL
> undrinkable
[prefix_un]+drink[verb]+DER_V_ADJ_ABLE
> undrinkables
[prefix_un]+drink[verb]+DER_V_ADJ_ABLE+PL
> undrink
Unknown word.
> undrinks
Unknown word.

Conditional affixes implemented by a continuation class are not enough for circumfixes, because a circumfix is one affix in morphology. We also need CIRCUMFIX option for correct morphological analysis.

# circumfixes: ~ obligate prefix/suffix combinations
# superlative in Hungarian: leg- (prefix) AND -bb (suffix)
# nagy, nagyobb, legnagyobb, legeslegnagyobb
# (great, greater, greatest, most greatest)

CIRCUMFIX X

PFX A Y 1 PFX A 0 leg/X .

PFX B Y 1 PFX B 0 legesleg/X .

SFX C Y 3 SFX C 0 obb . +COMPARATIVE SFX C 0 obb/AX . +SUPERLATIVE SFX C 0 obb/BX . +SUPERSUPERLATIVE

Dictionary:

1
nagy/C	[MN]

Analysis:

> nagy
nagy[MN]
> nagyobb
nagy[MN]+COMPARATIVE
> legnagyobb
nagy[MN]+SUPERLATIVE
> legeslegnagyobb
nagy[MN]+SUPERSUPERLATIVE

Allowing free compounding yields decrease in precision of recognition, not to mention stemming and morphological analysis. Although lexical switches are introduced to license compounding of bases by Ispell, this proves not to be restrictive enough. For example:

# affix file
COMPOUNDFLAG X

2
foo/X
bar/X

With this resource, foobar and barfoo also are accepted words.

This has been improved upon with the introduction of direction-sensitive compounding, i.e., lexical features can specify separately whether a base can occur as leftmost or rightmost constituent in compounds. This, however, is still insufficient to handle the intricate patterns of compounding, not to mention idiosyncratic (and language specific) norms of hyphenation.

The Hunspell algorithm currently allows any affixed form of words, which are lexically marked as potential members of compounds. Hunspell improved this, and its recursive compound checking rules makes it possible to implement the intricate spelling conventions of Hungarian compounds. For example, using COMPOUNDWORDMAX, COMPOUNDSYLLABLE, COMPOUNDROOT, SYLLABLENUM options can be set the noteworthy Hungarian `6--3' rule. Further example in Hungarian, derivate suffixes often modify compounding properties. Hunspell allows the compounding flags on the affixes, and there are two special flags (COMPOUNDPERMITFLAG and (COMPOUNDFORBIDFLAG) to permit or prohibit compounding of the derivations.

Suffixes with this flag forbid compounding of the affixed word.

We also need several Hunspell features for handling German compounding:

# German compounding

# set language to handle special casing of German sharp s

LANG de_DE

# compound flags

COMPOUNDBEGIN U COMPOUNDMIDDLE V COMPOUNDEND W

# Prefixes are allowed at the beginning of compounds, # suffixes are allowed at the end of compounds by default: # (prefix)?(root)+(affix)? # Affixes with COMPOUNDPERMITFLAG may be inside of compounds. COMPOUNDPERMITFLAG P

# for German fogemorphemes (Fuge-element) # Hint: ONLYINCOMPOUND is not required everywhere, but the # checking will be a little faster with it.

ONLYINCOMPOUND X

# forbid uppercase characters at compound word bounds CHECKCOMPOUNDCASE

# for handling Fuge-elements with dashes (Arbeits-) # dash will be a special word

COMPOUNDMIN 1 WORDCHARS -

# compound settings and fogemorpheme for `Arbeit'

SFX A Y 3 SFX A 0 s/UPX . SFX A 0 s/VPDX . SFX A 0 0/WXD .

SFX B Y 2 SFX B 0 0/UPX . SFX B 0 0/VWXDP .

# a suffix for `Computer'

SFX C Y 1 SFX C 0 n/WD .

# for forbid exceptions (*Arbeitsnehmer)

FORBIDDENWORD Z

# dash prefix for compounds with dash (Arbeits-Computer)

PFX - Y 1 PFX - 0 -/P .

# decapitalizing prefix # circumfix for positioning in compounds

PFX D Y 29 PFX D A a/PX A PFX D Ä ä/PX Ä . . PFX D Y y/PX Y PFX D Z z/PX Z

Example dictionary:

4
Arbeit/A-
Computer/BC-
-/W
Arbeitsnehmer/Z

Accepted compound compound words with the previous resource:

Computer
Computern
Arbeit
Arbeits-
Computerarbeit
Computerarbeits-
Arbeitscomputer
Arbeitscomputern
Computerarbeitscomputer
Computerarbeitscomputern
Arbeitscomputerarbeit
Computerarbeits-Computer
Computerarbeits-Computern

Not accepted compoundings:

computer
arbeit
Arbeits
arbeits
ComputerArbeit
ComputerArbeits
Arbeitcomputer
ArbeitsComputer
Computerarbeitcomputer
ComputerArbeitcomputer
ComputerArbeitscomputer
Arbeitscomputerarbeits
Computerarbeits-computer
Arbeitsnehmer

This solution is still not ideal, however, and will be replaced by a pattern-based compound-checking algorithm which is closely integrated with input buffer tokenization. Patterns describing compounds come as a separate input resource that can refer to high-level properties of constituent parts (e.g. the number of syllables, affix flags, and containment of hyphens). The patterns are matched against potential segmentations of compounds to assess wellformedness.

Problems with the 8-bit encoding

Both Ispell and Myspell use 8-bit ASCII character encoding, which is a major deficiency when it comes to scalability. Although a language like Hungarian has a standard ASCII character set (ISO 8859-2), it fails to allow a full implementation of Hungarian orthographic conventions. For instance, the '--' symbol (n-dash) is missing from this character set contrary to the fact that it is not only the official symbol to delimit parenthetic clauses in the language, but it can be in compound words as a special 'big' hyphen.

MySpell has got some 8-bit encoding tables, but there are languages without standard 8-bit encoding, too. For example, a lot of African languages have non-latin or extended latin characters.

Similarly, using the original spelling of certain foreign names like Ångström or Molière is encouraged by the Hungarian spelling norm, and, since characters 'Å' and 'è' are not part of ISO 8859-2, when they combine with inflections containing characters only in ISO 8859-2 (like elative -bo=l, allative -to=l or delative -ro=l with double acute), these result in words (like Ångströmro=l or Molière-to=l.) that can not be encoded using any single ASCII encoding scheme.

The problems raised in relation to 8-bit ASCII encoding have long been recognized by proponents of Unicode. Unfortunately, switching to Unicode (e.g., UTF-16 encoding) would require a great deal of code optimization and would have an impact on the efficiency of the algorithm. The Dömölki algorithm used in checking affixing conditions utilizes 256-byte character arrays, which would grow to 64k with Unicode encoding. Since online affixing for a richly agglutinative language can easily have several hundred such arrays (in the case of the standard Hungarian resources we use, this number is ca. 300 or more since redundant storage of structurally identical affix patterns improves efficiency), switching to Unicode would incur high resource costs. Nonetheless, it is clear that trading efficiency for encoding-independence has its advantages when it comes a truly multi-lingual application, therefore it was among our plans for a long while to extend the architecture in this direction.

A hybrid solution

Recently we implemented successfully a memory and time efficient Unicode handling. In non-UTF-8 character encodings Hunspell works with the original 8-bit algorithms, but with UTF-8 encoded dictionary and affix file Hunspell uses a hybrid string manipulation and condition checking to support Unicode:

Affixes and words are stored in UTF-8, during the analysis are handled in mostly UTF-8, in condition checking and suggestion are converted to UTF-16.

Dömölki-algorithm is used for storing and checking 7-bit ASCII (ISO 646) condition characters, and sorted UTF-16 lists for other Unicode characters of condition patterns.

Hunspell have supported only the first 65534 character (Basic Multilingual Plane) of Unicode Standard yet.

hunspell (1), ispell (1), ispell (4)