Package org.omegat.tokenizer

Class LuceneSpanishTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneSpanishTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneSpanishTokenizer
extends BaseTokenizer

A tokenizer implementation for the Spanish language using Lucene's analysis tools. This tokenizer provides support for tokenizing Spanish text, optionally allowing stemming and stop words removal.

The class utilizes Lucene's SpanishAnalyzer for Spanish-specific tokenization functionalities. If stemming is enabled, it applies stemming in conjunction with a configurable stop words set. Otherwise, it falls back to a standard token stream.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneSpanishTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneSpanishTokenizer

public LuceneSpanishTokenizer()