GutenMark Wordlists Page Attractively formatting Project Gutenberg texts home features download usage FAQ changes bugs links developer

Contents

What are wordlists and namelists?
What are they good for?
What wordlists are available?
Massaging the wordlists
Configuring

What are wordlists and namelists?

What are they good for?

• The ALL-CAPS style of italicizing.  [NOTE:  GutenMark can support this feature without wordlists, but using wordlists makes the support better.]  Many PG etexts represent italicized words in all-capital letters, as in "I can't believe SHE did that!"  In print, this should be rendered as "I can't believe she did that!"  Unfortunately, it is not obvious how the italicized word should be capitalized, or for that matter, if they even should be italicized.  Consider the following examples:  "I can't believe JOHN did that!" and "I can't believe NASA did that!"  These should be rendered "I can't believe John did that!" and "I can't believe NASA did that!"  GutenMark handles this by means of wordlists/namelists.  Since the wordlists contain the word "she", but not "She" or "SHE", GutenMark knows that SHE must be converted to she.  Since the wordlists contain "John" but not "JOHN", JOHN is converted to John.  Finally, since the wordlists actually contain the word "NASA", GutenMark understands that it should be left unchanged.
• Italicizing foreign words.  In English, it is correct (though it seems to me to be increasingly rarer) to italicize foreigns words and phrases, such as "the military's junta was unsuccessful."  Obviously, wordlists can be used to locate these non-english words.
• Restoration of diacritical marks.  Most PG etexts discard diacritical marks.  For example, the name "Schrödinger" would normally appear in a PG etext as "Schrodinger."  By consulting the wordlists, the correct form is known and can be restored.
• Hyphenation.  [NOTE:  This feature is not yet implemented.]  PG etexts contain so-called "hard carriage-returns" at the ends of the text lines, which GutenMark is forced to remove in order to re-justify the paragraphs.  If a word in the PG etext was broken across two lines by hyphenation (a practice that PG does not recommend), then a false hyphen might appear in the re-justified etext.  For example, if one text line ended with "in-" and the next began with "visible", then the marked-up HTML would contain "in-visible."  GutenMark can examine the wordlists to determine that "invisible" is a correct form, and that the hyphen can therefore be removed.

What wordlists are available?

Massaging the wordlists

The format of a GutenMark wordlist is simple:

• It is an ASCII text file, which has been compressed with the GNU gzip program.
• It contains a line for each word.  The lines can't contain any whitespace, or anything other than the word itself.
• The words should be capitalized as follows:  If a word must be in all-caps, like "NASA", then put it in all-caps.  If the word requires some special capitalization, such as "John" or "MacMurray", then capitalize it accordingly.  For normal words that are usually in lower-case, but are capitalized at the beginnings of sentences, use all lower-case.
• The words can contain any character in the following table, but not leading or trailing apostrophes.  The table includes both numerical codes (for non-ASCII characters) and the characters themselves, but the characters may or may not appear correctly, depending on your browser and its settings:

Unfortunately, the process of creating a wordlist will not be easy for most people, and since it varies from case to case it cannot be described in detail here.  It will be easiest for those with programming experience, and such knowledge is assumed in the next couple of paragraphs.

Most of the existing wordlists were created from language databases for the *nix spell-checker program called "ispell".  (Click here for more information.)  Ispell databases don't contain wordlists as such, but do contain word data and so-called "affix" files.  By combining these two, with the ispell '-e' command-line switch, a wordlist can be produced.   Some existing ispell databases don't incorporate diacritical marks directly, but expect them to encoded by some funky sequence of characters.  For example, 'Schrödinger' might appear as 'Schro"dinger'. GutenMark wordlists must contain the former rather than the latter.

For this reason and others, ispell wordlists always need some additional post-processing to be acceptable to GutenMark.  The post-processing for the existing GutenMark wordlists is performed by various little utility programs (listed in a table above) provided by GutenMark.  All of the utilities are simple command-line filters.  For more info, I fear you must look at the actual source code for the utilities.  Fortunately, this source code is quite simple.

It's probably unlikely that anyone will actually want to create a wordlist.  But if you do, you might want to tell me about it, so that I can add post the wordlist here for download.

Configuring

• Namelists for the language the etext is in.
• Namelists for other languages.
• Wordlists for the language the etext is in.
• Wordlists for other languages, in order of descreasing probability of finding words from that language within the etext.

When running GutenMark as a command-line program, GutenMark.cfg is by default located in the directory from which you run GutenMark.  (This can be overridden with the "--config" command-line option in later versions.)  When running GutenMark under the control of GUItenMark, the configuration file is by default in the "GutConfigs" subdirectory of the installation directory.  (This can optionally be overriden within GUItenMark.)  The wordlists are also usually located in the same directory as the program, but need not be if the configuration file is edited appropriately.  If in GutenMark.cfg you specify the wordlists by pathname—i.e., by filename plus directory—GutenMark will look for the wordlist files only in the exact locations you have specified.  On the other hand, if in GutenMark.cfg you specify the wordlists only by filename without directory, GutenMark will look for the wordlists first in the current directory and then in the directory containing the GutenMark executable.  The default configuration file does not contain directory names.

Important note:  Prior to version 20020721, there was a bug in which wordlists specified in GutenMark.cfg without directories were sought only in the current directory.  Therefore, if wordlists were put into the executable directory as recommended by the installation instructions, they could not be found without modifying the default configuration file to include the full pathnames of the wordlists.  In versions 20020721 and later, this problem has been fixed.