Package org.omegat.tokenizer

Class LuceneLatvianTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneLatvianTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneLatvianTokenizer
extends BaseTokenizer

A tokenizer for processing Latvian language text using Lucene's analysis capabilities. This class extends the BaseTokenizer and provides tokenization for Latvian text with options for stemming and the inclusion/exclusion of stop words.

This tokenizer utilizes the LatvianAnalyzer from Lucene for tokenization when stemming is enabled. If stemming is disabled, a standard token stream is used.

The tokenizer can be configured to include or exclude stop words and stems in the tokenization process: - When stemming is allowed, the LatvianAnalyzer is used, and the inclusion of stop words depends on the specified configuration. - When stemming is not allowed, the tokenizer falls back to the standard token stream implementation.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneLatvianTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneLatvianTokenizer

public LuceneLatvianTokenizer()