Package org.omegat.tokenizer

Class LuceneRussianTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneRussianTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneRussianTokenizer
extends BaseTokenizer

The LuceneRussianTokenizer class provides text tokenization specifically for the Russian language. This tokenizer utilizes the Lucene library and supports stemming and stop word filtering functionality based on the provided configuration.

It extends the BaseTokenizer class and overrides the base tokenization behavior to accommodate language-specific rules and features for Russian text.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneRussianTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneRussianTokenizer

public LuceneRussianTokenizer()