Package org.omegat.tokenizer

Class LuceneBasqueTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneBasqueTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneBasqueTokenizer
extends BaseTokenizer

Tokenizer class specifically designed for processing Basque text. It extends the BaseTokenizer and provides functionality to tokenize Basque input text.

This tokenizer uses the BasqueAnalyzer to perform tokenization, optionally applying stemming and stop word removal based on the parameter configurations. If stemming is not allowed, it defaults to standard tokenization.

This class is annotated with Tokenizer to associate it with the Basque language using the language code "eu".

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneBasqueTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneBasqueTokenizer

public LuceneBasqueTokenizer()