FreeBSD Manual Pages
ANNOGEN(1) General Commands Manual ANNOGEN(1)
Annotator Generator is an examples-driven generator of fast text annotators.
"Annotate" in this context means to add pronunciation or other information to
each word, and/or to split text into words in a language that does not use
spaces.
O You supply a corpus of pre-annotated texts for Annotator Generator to
work out the rules and exceptions
O Annotator Generator creates table-driven code in C, Java, Javascript,
Dart or Python with 2 and 3 compatibility
O The resulting program should be able to annotate any text that contains
words or phrases similar to those found in the examples
O It can output the annotations alone or it can combine them with the
original text using HTML Ruby markup or simple braces
O If anything is unclear (didn't happen in the examples, or there's not
enough context to figure out which example should be applied) then the
program will leave it unannotated so you can pass it to a backup anno-
tation program if you have one.
O If you have no backup annotator then try setting the -y option, which
makes Annotator Generator try harder to find context-independent rules
with context-dependent exceptions, so as to annotate as much text as
possible.
O Generated annotators can act as filters for Web Adjuster; options are
also provided for generating Android apps, browser extensions, and
clipboard annotators for Windows and Windows Mobile, or you could for-
mat the annotations on a Unix terminal
-h, --help
show this help message and exit
--infile=
Filename of a text file (or a compressed .gz, .bz2, .xz or .bz3 file or
URL) to read the input examples from. If this is not specified, standard
input is used.
--incode=
Character encoding of the input file (default utf-8)
--mstart=
The string that starts a piece of text with annotation markup in the input
examples; default <ruby><rb>
--mmid=
The string that occurs in the middle of a piece of markup in the input ex-
amples, with the word on its left and the added markup on its right (or the
other way around if mreverse is set); default </rb><rt>
--mend=
The string that ends a piece of annotation markup in the input examples;
default </rt></ruby>
-r, --mreverse
Specifies that the annotation markup is reversed, so the text before mmid
is the annotation and the text after it is the base text
--no-mreverse
Cancels any earlier --mreverse option in Makefile variables etc
--end-pri=
Treat words that occur in the examples before this delimeter as having
"high priority" for Yarowsky-like seed collocations (if these are in use).
Normally the Yarowsky-like logic tries to identify a "default" annotation
based on what is most common in the examples, with the exceptions indicated
by collocations. If however a word is found in a high-priority section at
the start, then the first annotation found there will be taken as the ideal
"default" even if it's in a minority in the examples; everything else will
be taken as an exception.
-s, --spaces
Set this if you are working with a language that uses whitespace in its
non-markedup version (not fully tested). The default is to assume that
there will not be any whitespace in the language, which is correct for Chi-
nese and Japanese.
--no-spaces
Cancels any earlier --spaces option in Makefile variables etc
-c, --capitalisation
Don't try to normalise capitalisation in the input. Normally, to simplify
the rules, the analyser will try to remove start-of-sentence capitals in
annotations, so that the only remaining words with capital letters are the
ones that are always capitalised such as names. (That's not perfect: some
words might always be capitalised just because they never occur mid-sen-
tence in the examples.) If this option is used, the analyser will instead
try to "learn" how to predict the capitalisation of all words (including
start of sentence words) from their contexts.
--no-capitalisation
Cancels any earlier --capitalisation option in Makefile variables etc
-w, --annot-whitespace
Don't try to normalise the use of whitespace and hyphenation in the example
annotations. Normally the analyser will try to do this, to reduce the risk
of missing possible rules due to minor typographical variations.
--no-annot-whitespace
Cancels any earlier --annot-whitespace option in Makefile variables etc
--keep-whitespace=
Comma-separated list of words (without annotation markup) for which white-
space and hyphenation should always be kept even without the --annot-white-
space option. Use when you know the variation is legitimate. This option
expects words to be encoded using the system locale (UTF-8 if it cannot be
detected).
--suffix=
Comma-separated list of annotations that can be considered optional suf-
fixes for normalisation
--suffix-minlen=
Minimum length of word (in Unicode characters) to apply suffix normalisa-
tion
--post-normalise=
Filename or URL of an optional Python module defining a dictionary called
'table' mapping integers to integers for arbitrary single-character normal-
isation on the Unicode BMP. This can reduce the size of the annotator. It
is applied in post-processing (does not affect rules generation itself).
For example this can be used to merge the recognition of Full, Simplified
and Variant forms of the same Chinese character in cases where this can be
done without ambiguity, if it is acceptable for the generated annotator to
recognise mixed-script words should they occur. If any word in the examples
has a different annotation when normalised than not, the normalised version
takes precedence.
--glossfile=
Filename of an optional text file (or compressed .gz, .bz2, .xz or .bz3
file or URL) to read auxiliary "gloss" information. Each line of this
should be of the form: word (tab) annotation (tab) gloss. Extra tabs in the
gloss will be converted to newlines (useful if you want to quote multiple
dictionaries). When the compiled annotator generates ruby markup, it will
add the gloss string as a popup title whenever that word is used with that
annotation (before any reannotator option is applied). The annotation field
may be left blank to indicate that the gloss will appear for all other an-
notations of that word. The entries in glossfile do not affect the annota-
tion process itself, so it's not necessary to completely debug glossfile's
word segmentation etc.
-C, --gloss-closure=
If any Chinese, Japanese or Korean word is missing from glossfile, search
its closure of variant characters also, using the Unihan variants file (or
URL) specified by this option
--no-gloss-closure
Cancels any earlier --gloss-closure option in Makefile variables etc
-M, --glossmiss-omit
Omit rules containing any word not mentioned in glossfile. Might be useful
if you want to train on a text that uses proprietary terms and don't want
to accidentally 'leak' those terms (assuming they're not accidentally in-
cluded in glossfile also). Words may also be listed in glossfile with an
empty gloss field to indicate that no gloss is available but rules using
this word needn't be omitted.
--no-glossmiss-omit
Cancels any earlier --glossmiss-omit option in Makefile variables etc
--words-omit=
File (or compressed .gz, .bz2, .xz or .bz3 file or URL) containing words
(one per line, without markup) to omit from the annotator. Use this to make
an annotator smaller if for example if you're working from a rules file
that contains long lists of place names you don't need this particular an-
notator to recognise but you still want to keep them as rules for other an-
notators, but be careful because any word on such a list gets omitted even
if it also has other meanings (some place names are also normal words).
--manualrules=
Filename of an optional text file (or compressed .gz, .bz2, .xz or .bz3
file or URL) to read extra, manually-written rules. Each line of this
should be a marked-up phrase (in the input format) which is to be uncondi-
tionally added as a rule. Use this sparingly, because these rules are not
taken into account when generating the others and they will be applied re-
gardless of context (although a manual rule might fail to activate if the
annotator is part-way through processing a different rule); try checking
messages from --diagnose-manual.
--c-filename=
Where to write the C, C#, Python, Javascript, Go or Dart program. Defaults
to standard output, or annotator.c in the system temporary directory if
standard output seems to be the terminal (the program might be large, espe-
cially if Yarowsky-like indicators are not used, so it's best not to use a
server home directory where you might have limited quota).
--c-compiler=
The C compiler to run if generating C and standard output is not connected
to a pipe. The default is to use the "cc" command which usually redirects
to your "normal" compiler. You can add options (remembering to enclose this
whole parameter in quotes if it contains spaces), but if the C program is
large then adding optimisation options may make the compile take a long
time. If standard output is connected to a pipe, then this option is ig-
nored because the C code will simply be written to the pipe. You can also
set this option to an empty string to skip compilation. Default: cc -o an-
notator
--outcode=
Character encoding to use in the generated parser (default utf-8, must be
ASCII-compatible i.e. not utf-16)
--rulesFile=
Filename of a JSON file to hold the accumulated rules. Adding .gz, .bz2,
.xz or .bz3 for compression is acceptable. If this is set then either
--write-rules or --read-rules must be specified.
--write-rules
Write rulesFile instead of generating a parser. You will then need to rerun
with --read-rules later.
--no-write-rules
Cancels any earlier --write-rules option in Makefile variables etc
--read-rules
Read rulesFile from a previous run, and apply the output options to it. You
should still specify the input formatting options (which should not
change), and any glossfile or manualrules options (which may change), but
no input is required.
--no-read-rules
Cancels any earlier --read-rules option in Makefile variables etc
-E, --newlines-reset
Have the annotator reset its state on every newline byte. By default new-
lines do not affect state such as whether a space is required before the
next word, so that if the annotator is used with Web Adjuster's htmlText
option (which defaults to using newline separators) the spacing should be
handled sensibly when there is HTML markup in mid-sentence.
--no-newlines-reset
Cancels any earlier --newlines-reset option in Makefile variables etc
-z, --compress
Compress annotation strings in the C code. This compression is designed for
fast on-the-fly decoding, so it saves only a limited amount of space (typi-
cally 10-20%) but might help if RAM is short.
--no-compress
Cancels any earlier --compress option in Makefile variables etc
-Z, --zlib
Compress the embedded data table using zlib (or pyzopfli if available), and
include code to call zlib to decompress it on load. Useful if the runtime
machine has the zlib library and you need to save disk space but not RAM
(the decompressed table is stored separately in RAM, unlike --compress
which, although giving less compression, at least works 'in place'). Once
--zlib is in use, specifying --compress too will typically give an addi-
tional disk space saving of less than 1% (and a runtime RAM saving that's
greater but more than offset by zlib's extraction RAM). If generating a
Javascript annotator with zlib, the decompression code is inlined so
there's no runtime zlib dependency, but startup can be ~50% slower so this
option is not recommended in situations where the annotator is frequently
reloaded from source (unless you're running on Node.js in which case load-
ing is faster due to the use of Node's "Buffer" class).
--no-zlib
Cancels any earlier --zlib option in Makefile variables etc
-l, --library
Instead of generating C code that reads and writes standard input/output,
generate a C library suitable for loading into Python via ctypes. This can
be used for example to preload a filter into Web Adjuster to cut
process-startup delays.
--no-library
Cancels any earlier --library option in Makefile variables etc
-W, --windows-clipboard
Include C code to read the clipboard on Windows or Windows Mobile and to
write an annotated HTML file and launch a browser, instead of using the de-
fault cross-platform command-line C wrapper. See the start of the generated
C file for instructions on how to compile for Windows or Windows Mobile.
--no-windows-clipboard
Cancels any earlier --windows-clipboard option in Makefile variables etc
--java=
Instead of generating C code, generate Java, and place the *.java files in
the directory specified by this option. The last part of the directory
should be made up of the package name; a double slash (//) should separate
the rest of the path from the package name, e.g. --java=/path/to/wher-
ever//org/example/annotator and the main class will be called Annotator.
--android=
URL for an Android app to browse (--java must be set). If this is set, code
is generated for an Android app which starts a browser with that URL as the
start page, and annotates the text on every page it loads. Use file:///an-
droid_asset/index.html for local HTML files in the assets directory; a
clipboard viewer is placed in clipboard.html, and the app will also be able
to handle shared text. If certain environment variables are set, this op-
tion can also compile and sign the app using Android SDK command-line tools
(otherwise it puts a message on stderr explaining what needs to be set)
--android-template=
File (or URL) to use as a template for Android start HTML. This option im-
plies --android=file:///android_asset/index.html and generates that in-
dex.html from the file specified (or from a built-in default if the special
filename 'blank' is used). The template file may include URL_BOX_GOES_HERE
to show a URL entry box and related items (offline-clipboard link etc) in
the page, in which case you can optionally define a Javascript function
'annotUrlTrans' to pre-convert some URLs from shortcuts etc; also enables
better zoom controls on Android 4+, a mode selector if you use --annota-
tion-names, a selection scope control on recent-enough WebKit, and a visi-
ble version stamp (which, if the device is in 'developer mode', you may
double-tap on to show missing glosses). VERSION_GOES_HERE may also be in-
cluded if you want to put it somewhere other than at the bottom of the
page. If you do include URL_BOX_GOES_HERE you'll have an annotating Web
browser app that allows the user to navigate to arbitrary URLs: as of 2020,
this is acceptable on Google Play and Huawei AppGallery (non-China only
from 2022), but not Amazon AppStore as they don't want 'competition' to
their Silk browser.
--gloss-simplify=
A regular expression matching parts of glosses to remove when generating a
'3-line' format in apps, but not for hover titles or popups. Default re-
moves parenthesised expressions if not solitary, anything after the first
slash or semicolon, and the leading word 'to'. Can be set to empty string
to omit simplification.
-L, --pleco-hanping
In the Android app, make popup definitions link to Pleco or Hanping if in-
stalled
--no-pleco-hanping
Cancels any earlier --pleco-hanping option in Makefile variables etc
--bookmarks=
Android bookmarks: comma-separated list of package names that share our
bookmarks. If this is not specified, the browser will not be given a book-
marks function. If it is set to the same value as the package specified in
--java, bookmarks are kept in just this Android app. If it is set to a
comma-separated list of packages that have also been generated by annogen
(presumably with different annotation types), and if each one has the same
android:sharedUserId attribute in AndroidManifest.xml's 'manifest' tag
(you'll need to add this manually), and if the same certificate is used to
sign all of them, then bookmarks can be shared across the set of browser
apps. But beware the following two issues: (1) adding an android:share-
dUserId attribute to an app that has already been released without one
causes some devices to refuse the update with a 'cannot install' message
(details via adb logcat; affected users would need to uninstall and rein-
stall instead of update, and some of them may not notice the instruction to
do so); (2) this has not been tested with Google's new "App Bundle"
arrangement, and may be broken if the Bundle results in APKs being signed
by a different key. In June 2019 Play Console started issuing warnings if
you release an APK instead of a Bundle, even though the "size savings" they
mention are under 1% for annogen-generated apps.
-e, --epub
When generating an Android browser, make it also respond to requests to
open EPUB files. This results in an app that requests the 'read external
storage' permission on Android versions below 6, so if you have already re-
leased a version without EPUB support then devices running Android 5.x or
below will not auto-update past this change until the user notices the up-
date notification and approves the extra permission.
--no-epub
Cancels any earlier --epub option in Makefile variables etc
--android-print
When generating an Android browser, include code to provide a Print option
(usually print to PDF) and a simple highlight-selection option. The Print
option will require Android 4.4, but the app should still run without it on
earlier versions of Android.
--no-android-print
Cancels any earlier --android-print option in Makefile variables etc
--known-characters=
When generating an Android browser, include an option to leave the most
frequent characters unannotated as 'known'. This option should be set to
the filename or URL of a UTF-8 file of characters separated by newlines,
assumed to be most frequent first, with characters on the same line being
variants of each other (see --freq-count for one way to generate it). Words
consisting entirely of characters found in the first N lines of this file
(where N is settable by the user) will be unannotated until tapped on.
--freq-count=
Name of a file to write that is suitable for the known-characters option,
taken from the input examples (which should be representative of typical
use). Any post-normalise table provided will be used to determine which
characters are equivalent.
--android-audio=
When generating an Android browser, include an option to convert the selec-
tion to audio using this URL as a prefix, e.g. https://exam-
ple.org/speak.cgi?text= (use for languages not likely to be supported by
the device itself). Optionally follow the URL with a space (quote care-
fully) and a maximum number of words to read in each user request. Setting
a limit is recommended, or somebody somewhere will likely try 'Select All'
on a whole book or something and create load problems. You should set a
limit server-side too of course.
--extra-js=
Extra Javascript to inject into sites to fix things in the Android browser
app. The snippet will be run before each scan for new text to annotate. You
may also specify a file to read: --extra-js=@file.js or --ex-
tra-js=@file1.js,file2.js (or URLs; do not use // comments in these files,
only /* ... */ because newlines will be replaced), and you can create vari-
ants of the files by adding search-replace strings: --ex-
tra-js=@file1.js:search:replace,file2.js
--tts-js
Make Android 5+ multilingual Text-To-Speech functions available to extra-js
scripts (see TTSInfo code for details)
--no-tts-js
Cancels any earlier --tts-js option in Makefile variables etc
--existing-ruby-js-fixes=
Extra Javascript to run in the Android browser app or browser extension
whenever existing RUBY elements are encountered; the DOM node above these
elements will be in the variable n, which your code can manipulate or re-
place to fix known problems with sites' existing ruby (such as common
two-syllable words being split when they shouldn't be). Use with caution.
You may also specify a file or URL to read: --exist-
ing-ruby-js-fixes=@file.js
--existing-ruby-lang-regex=
Set the Android app or browser extension to remove existing ruby elements
unless the document language matches this regular expression. If
--sharp-multi is in use, you can separate multiple regexes with comma and
any unset will always delete existing ruby. If this option is not set at
all then existing ruby is always kept.
--existing-ruby-shortcut-yarowsky
Set the Android browser app to 'shortcut' Yarowsky-like collocation deci-
sions when adding glosses to existing ruby over 2 or more characters, so
that words normally requiring context to be found are more likely to be
found without context (this may be needed because adding glosses to exist-
ing ruby is done without regard to context)
--extra-css=
Extra CSS to inject into sites to fix things in the Android browser app.
You may also specify a file or URL to read --extra-css=@file.css
--app-name=
User-visible name of the Android app
--compile-only
Assume the code has already been generated by a previous run, and just run
the compiler
--no-compile-only
Cancels any earlier --compile-only option in Makefile variables etc
-j, --javascript
Instead of generating C code, generate JavaScript. This might be useful if
you want to run an annotator on a device that has a JS interpreter but
doesn't let you run your own binaries. The JS will be table-driven to make
it load faster. See comments at the start for usage.
--no-javascript
Cancels any earlier --javascript option in Makefile variables etc
-6, --js-6bit
When generating a Javascript annotator, use a 6-bit format for many ad-
dresses to reduce escape codes in the data string by making more of it
ASCII
--no-js-6bit
Cancels any earlier --js-6bit option in Makefile variables etc
-8, --js-octal
When generating a Javascript annotator, use octal instead of hexadecimal
codes in the data string when doing so would save space. This does not com-
ply with ECMAScript 5 and may give errors in its strict mode.
--no-js-octal
Cancels any earlier --js-octal option in Makefile variables etc
-9, --ignore-ie8
When generating a Javascript annotator, do not make it backward-compatible
with Microsoft Internet Explorer 8 and below. This may save a few bytes.
--no-ignore-ie8
Cancels any earlier --ignore-ie8 option in Makefile variables etc
-u, --js-utf8
When generating a Javascript annotator, assume the script can use UTF-8 en-
coding directly and not via escape sequences. In some browsers this might
work only on UTF-8 websites, and/or if your annotation can be expressed
without the use of Unicode combining characters.
--no-js-utf8
Cancels any earlier --js-utf8 option in Makefile variables etc
--browser-extension=
Name of a Chrome or Firefox browser extension to generate. The extension
will be placed in a directory of the same name (without spaces), which may
optionally already exist and contain icons like 32.png and 48.png to be
used.
--browser-extension-description=
Description field to use when generating browser extensions
--manifest-v3
Use Manifest v3 instead of Manifest v2 when generating browser extensions
(tested on Chrome only, and requires Chrome 88 or higher). This is now re-
quired for all Chrome Web Store uploads.
--gecko-id=
a Gecko (Firefox) ID to embed in the browser extension
--dart
Instead of generating C code, generate Dart. This might be useful if you
want to run an annotator in a Flutter application.
--no-dart
Cancels any earlier --dart option in Makefile variables etc
--dart-datafile=
When generating Dart code, put annotator data into a separate file and open
it using this pathname. Not compatible with Dart's "Web app" option, but
might save space in a Flutter app (especially along with --zlib)
-Y, --python
Instead of generating C code, generate a Python module. Similar to the
Javascript option, this is for when you can't run your own binaries, and it
is table-driven for fast loading.
--no-python
Cancels any earlier --python option in Makefile variables etc
--reannotator=
Shell command through which to pipe each word of the original text to ob-
tain new annotation for that word. This might be useful as a quick way of
generating a new annotator (e.g. for a different topolect) while keeping
the information about word separation and/or glosses from the previous an-
notator, but it is limited to commands that don't need to look beyond the
boundaries of each word. If the command is prefixed by a # character, it
will be given the word's existing annotation instead of its original text,
and if prefixed by ## it will be given text#annotation. The command should
treat each line of its input independently, and both its input and its out-
put should be in the encoding specified by --outcode.
-A, --reannotate-caps
When using --reannotator, make sure to capitalise any word it returns that
began with a capital on input
--no-reannotate-caps
Cancels any earlier --reannotate-caps option in Makefile variables etc
--sharp-multi
Assume annotation (or reannotator output) contains multiple alternatives
separated by # (e.g. pinyin#Yale) and include code to select one by number
at runtime (starting from 0). This is to save on total space when shipping
multiple annotators that share the same word grouping and gloss data, dif-
fering only in the transcription of each word.
--no-sharp-multi
Cancels any earlier --sharp-multi option in Makefile variables etc
--annotation-names=
Comma-separated list of annotation types supplied to sharp-multi (e.g.
Pinyin,Yale), if you want the Android app etc to be able to name them. You
can also set just one annotation names here if you are not using
sharp-multi.
--annotation-map=
Comma-separated list of annotation-number overrides for sharp-multi, e.g.
7=3 to take the 3rd item if a 7th is selected
--annotation-postprocess=
Extra code for post-processing specific annotNo selections after retrieving
from a sharp-multi list (@file or @url allowed)
-o, --allow-overlaps
Normally, the analyser avoids generating rules that could overlap with each
other in a way that would leave the program not knowing which one to apply.
If a short rule would cause overlaps, the analyser will prefer to generate
a longer rule that uses more context, and if even the entire phrase cannot
be made into a rule without causing overlaps then the analyser will give up
on trying to cover that phrase. This option allows the analyser to generate
rules that could overlap, as long as none of the overlaps would cause ac-
tual problems in the example phrases. Thus more of the examples can be cov-
ered, at the expense of a higher risk of ambiguity problems when applying
the rules to other texts. See also the -y option.
--no-allow-overlaps
Cancels any earlier --allow-overlaps option in Makefile variables etc
-y, --ybytes=
Look for candidate Yarowsky seed-collocations within this number of bytes
of the end of a word. If this is set then overlaps and rule conflicts will
be allowed when seed collocations can be used to distinguish between them,
and the analysis is likely to be faster. Markup examples that are com-
pletely separate (e.g. sentences from different sources) must have at least
this number of (non-whitespace) bytes between them.
--ybytes-max=
Extend the Yarowsky seed-collocation search to check over larger ranges up
to this maximum. If this is set then several ranges will be checked in an
attempt to determine the best one for each word, but see also ymax-thresh-
old and ymax-limitwords.
--ymax-threshold=
Limits the length of word that receives the narrower-range Yarowsky search
when ybytes-max is in use. For words longer than this, the search will go
directly to ybytes-max. This is for languages where the likelihood of a
word's annotation being influenced by its immediate neighbours more than
its distant collocations increases for shorter words, and less is to be
gained by comparing different ranges when processing longer words. Setting
this to 0 means no limit, i.e. the full range will be explored on all
Yarowsky checks.
--ymax-limitwords=
Comma-separated list of words (without annotation markup) for which the
ybytes expansion loop should run at most two iterations. This may be useful
to reduce compile times for very common ambiguous words that depend only on
their immediate neighbours. Annogen may suggest words for this option if it
finds they take inordinate time to process.
--ybytes-step=
The increment value for the loop between ybytes and ybytes-max
-k, --warn-yarowsky
Warn when absolutely no distinguishing Yarowsky seed collocations can be
found for a word in the examples
--no-warn-yarowsky
Cancels any earlier --warn-yarowsky option in Makefile variables etc
-K, --yarowsky-all
Accept Yarowsky seed collocations even from input characters that never oc-
cur in annotated words (this might include punctuation and example-separa-
tion markup)
--no-yarowsky-all
Cancels any earlier --yarowsky-all option in Makefile variables etc
--yarowsky-multiword
Check potential multiword rules for Yarowsky seed collocations also. With-
out this option (default), only single-word rules are checked.
--no-yarowsky-multiword
Cancels any earlier --yarowsky-multiword option in Makefile variables etc
--yarowsky-thorough
Recheck Yarowsky seed collocations when checking if any multiword rule
would be needed to reproduce the examples. This could risk 'overfitting'
the example set.
--no-yarowsky-thorough
Cancels any earlier --yarowsky-thorough option in Makefile variables etc
--yarowsky-half-thorough
Like --yarowsky-thorough but check only what collocations occur within the
proposed new rule (not around it), less likely to overfit
--no-yarowsky-half-thorough
Cancels any earlier --yarowsky-half-thorough option in Makefile variables
etc
--yarowsky-debug=
Report the details of seed-collocation false positives if there are a large
number of matches and a small number of false positives, optionally with a
lesser large number of matches when the false positive doesn't cut across
word boundaries. Default 1000:1:750 (setting a single number N is equiva-
lent to 1000:N:750, where N=0 omits reporting; setting X:N is equivalent to
X:N:X). Occasionally these mismatches might be due to typos in the corpus,
so it might be worth a check.
--allow-exceptions=
Filename (or URL) of any known exeptions for --yarowsky-debug checks (de-
fault allow-exceptions.txt)
--normalise-debug=
When --capitalisation is not in effect. report words that are usually capi-
talised but that have at most this number of lower-case exceptions (default
1) for investigation of possible typos in the corpus
--allow-caps-exceptions=
Filename (or URL) of any known exeptions for --normalise-debug checks (de-
fault allow-caps-exceptions.txt)
--debug-dir=
Directory in which to write reports of possible typos etc (defaults to cur-
rent directory)
--normalise-cache=
Optional file to use to cache the result of normalisation. Adding .gz,
.bz2, .xz or .bz3 for compression is acceptable.
-1, --single-words
Do not generate any rule longer than 1 word, although it can still have
Yarowsky seed collocations if -y is set. This speeds up the search, but at
the expense of thoroughness. You might want to use this in conjuction with
-y to make a parser quickly.
--no-single-words
Cancels any earlier --single-words option in Makefile variables etc
--max-words=
Limits the number of words in a rule. 0 means no limit. --single-words is
equivalent to --max-words=1. If you need to limit the search time, and are
using -y, it should suffice to use --single-words for a quick annotator or
--max-words=5 for a more thorough one (or try 3 if --yarowsky-half-thorough
is in use).
--multiword-end-avoid=
Comma-separated list of words (without annotation markup) that should be
avoided at the end of a multiword rule (e.g. sandhi likely to depend on the
following word)
-d, --diagnose=
Output some diagnostics for the specified word. Use this option to help an-
swer "why doesn't it have a rule for...?" issues. This option expects the
word without markup and uses the system locale (UTF-8 if it cannot be de-
tected).
--diagnose-limit=
Maximum number of phrases to print diagnostics for (0 means unlimited). De-
fault: 10
-m, --diagnose-manual
Check and diagnose potential failures of --manualrules
--no-diagnose-manual
Cancels any earlier --diagnose-manual option in Makefile variables etc
-q, --diagnose-quick
Ignore all phrases that do not contain the word specified by the --diagnose
option, for getting a faster (but possibly less accurate) diagnostic. The
generated annotator is not likely to be useful when this option is present.
--no-diagnose-quick
Cancels any earlier --diagnose-quick option in Makefile variables etc
--priority-list=
Instead of generating an annotator, use the input examples to generate a
list of (non-annotated) words with priority numbers, a higher number mean-
ing the word should have greater preferential treatment in ambiguities, and
write it to this file (or compressed .gz, .bz2, .xz or .bz3 file). If the
file provided already exists, it will be updated, thus you can amend an ex-
isting usage-frequency list or similar (although the final numbers are pri-
orities and might no longer match usage-frequency exactly). The purpose of
this option is to help if you have an existing word-priority-based text
segmenter and wish to update its data from the examples; this approach
might not be as good as the Yarowsky-like one (especially when the same
word has multiple readings to choose from), but when there are integration
issues with existing code you might at least be able to improve its
word-priority data.
-t, --time-estimate
Estimate time to completion. The code to do this is unreliable and is prone
to underestimate. If you turn it on, its estimate is displayed at the end
of the status line as days, hours or minutes.
--no-time-estimate
Cancels any earlier --time-estimate option in Makefile variables etc
-0, --single-core
Use only one CPU core even when others are available on Unix
--no-single-core
Cancels any earlier --single-core option in Makefile variables etc
--cores-command=
Command to run when changing the number of CPU cores in use (with new num-
ber as a parameter); this can run a script to pause/resume any lower-prior-
ity load
-p, --status-prefix=
Label to add at the start of the status line, for use if you batch-run
annogen in multiple configurations and want to know which one is currently
running
Legal considerations
Annotator code will contain individual words and some phrases from the
original corpus (and these can be read even by people who do not have the
unannotated version); with regards to copyright law, I expect the annotator
code will count as an "index" to the collection, the copyright of which ex-
ists separately to that of the original collection, but laws do vary by
country and I am not a solicitor so please act judiciously.
Legally obtaining that original annotated corpus is up to you. If you are
in the UK the government says non-commercial text mining is allowed (terms
of use prohibiting non-commercial mining are unenforceable), provided you:
1. respect network stability (i.e. wait a long time between each down-
load),
2. connect directly to the publisher (this law bypasses the publisher's
terms of use, not those of third-party search engines like Google),
3. use the result only for mining, not for republishing the original text
(so you can't publish your unprocessed crawl dumps either),
4. and still respect any prohibitions against sharing whatever mining
tools you made for the site (as this law is only about text mining, not
about the sharing of tools).
Laws outside the UK are different (and I'm not a lawyer) so check care-
fully. Gao et al 2020's paper on "The Pile"
https://arxiv.org/abs/2101.00027 claims published crawl dumps with limited
processing might be permissible under American copyright law as transforma-
tive fair use, but I'm not sure how legally watertight their argument is:
it might be safer to keep unlicensed parts of the corpus private and pub-
lish only the resulting index.
If the website's terms don't actually prohibit writing an unpublished
scraper for non-commercial mining purposes, perhaps you won't need a legal
exception for the crawling part--but you should still respect their band-
width and do it slowly, both for moral reasons (it's the right thing to do)
and pragmatic ones (you won't want their sysadmins and service providers
taking action against you).
Citation
If you need to cite a peer-reviewed paper:
Silas S. Brown. Web Annotation with Modified-Yarowsky and Other Algorithms.
Overload 112 (December 2012) pp.4-7.
Silas S. Brown April 2026 ANNOGEN(1)
Legal considerations | Citation
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=annogen&sektion=1&manpath=FreeBSD+Ports+15.1.quarterly>