Package org.omegat.tokenizer

Class LuceneFinnishTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneFinnishTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneFinnishTokenizer
extends BaseTokenizer

Tokenizer class for the Finnish language that extends the base tokenizer functionality. It provides tokenization capabilities tailored to the Finnish language, with support for stemming and stop-word filtering based on configuration.

The class utilizes Lucene's FinnishAnalyzer for processing the text input. It determines whether stemming and stop-word filtering are applied, depending on the parameters provided.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneFinnishTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneFinnishTokenizer

public LuceneFinnishTokenizer()