Package org.omegat.tokenizer

Class LuceneIrishTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneIrishTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneIrishTokenizer
extends BaseTokenizer

A tokenizer for processing text in the Irish language. This class extends the functionality of the BaseTokenizer to provide tokenization specifically tailored for Irish. It handles token streams with optional stemming and stop-word removal based on the provided parameters.

The tokenizer utilizes Lucene's IrishAnalyzer when stemming is enabled. Stop-word support is configurable to include the default Irish stop-word set or none at all. If stemming is disabled, a standard token stream is generated.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneIrishTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneIrishTokenizer

public LuceneIrishTokenizer()