org.omegat.tokenizer

Class HunspellTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.HunspellTokenizer

All Implemented Interfaces:

ITokenizer

public class HunspellTokenizer
extends BaseTokenizer

Methods for tokenize string.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

DEFAULT_TOKENS_COUNT, EMPTY_STRING_LIST, EMPTY_TOKENS_LIST, shouldDelegateTokenizeExactly, TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor and Description
HunspellTokenizer()

Method Summary

Modifier and Type	Method and Description
java.lang.String[]	getSupportedLanguages() Return an array of language strings (xx-yy) indicating the tokenizer's supported languages.
protected org.apache.lucene.analysis.TokenStream	getTokenStream(java.lang.String strOrig, boolean stemsAllowed, boolean stopWordsAllowed)
static void	loadPlugins()
static void	unloadPlugins()

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getEffectiveLanguage, getProjectLanguage, getStandardTokenStream, printTest, test, tokenize, tokenizeByCodePoint, tokenizeByCodePointToStrings, tokenizeToStrings, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HunspellTokenizer

public HunspellTokenizer()

Method Detail

getTokenStream

protected org.apache.lucene.analysis.TokenStream getTokenStream(java.lang.String strOrig,
                                                                boolean stemsAllowed,
                                                                boolean stopWordsAllowed)
                                                         throws java.io.IOException

Specified by:

getTokenStream in class BaseTokenizer

Throws:

java.io.IOException

getSupportedLanguages

public java.lang.String[] getSupportedLanguages()

Description copied from interface: ITokenizer

Return an array of language strings (xx-yy) indicating the tokenizer's supported languages. Meant for tokenizers for which the supported languages can only be determined at runtime, like the HunspellTokenizer.

Indicate that this should be used by setting the Tokenizer annotation to contain only Tokenizer.DISCOVER_AT_RUNTIME.

Specified by:

getSupportedLanguages in interface ITokenizer

Overrides:

getSupportedLanguages in class BaseTokenizer

loadPlugins

public static void loadPlugins()

unloadPlugins

public static void unloadPlugins()