Package org.omegat.tokenizer

Class LuceneGermanTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneGermanTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneGermanTokenizer
extends BaseTokenizer

LuceneGermanTokenizer class provides tokenization functionality specifically designed for German language text. It extends the BaseTokenizer class and implements a customizable token stream that supports stemming and includes options for enabling or disabling stop words during the tokenization process. This tokenizer uses a German-specific analyzer to produce the appropriate tokens.

The tokenizer is set as the default for German language ("de") based on the language annotation.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneGermanTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneGermanTokenizer

public LuceneGermanTokenizer()