Package org.omegat.tokenizer

Class LuceneBrazilianTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneBrazilianTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneBrazilianTokenizer
extends BaseTokenizer

A tokenizer implementation for processing Brazilian Portuguese text with optional stemming and stop word removal capabilities. This class leverages Lucene's BrazilianAnalyzer and CharArraySet for tokenization tasks.

The behavior of the tokenizer is controlled via the method parameters and ensures that it is adaptable to a variety of text processing needs specific to Brazilian Portuguese.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneBrazilianTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneBrazilianTokenizer

public LuceneBrazilianTokenizer()