Package org.omegat.tokenizer

Class LuceneRomanianTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneRomanianTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneRomanianTokenizer
extends BaseTokenizer

LuceneRomanianTokenizer is a tokenizer specifically designed for the Romanian language. It extends the BaseTokenizer class and provides tokenization capabilities for Romanian text. This class supports optional stemming and stop word filtering based on its configuration.

The tokenizer utilizes the RomanianAnalyzer class from Apache Lucene to process text when stemming and/or stop words are enabled. In cases where stemming is disabled, a standard token stream is provided instead.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneRomanianTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneRomanianTokenizer

public LuceneRomanianTokenizer()