Package org.omegat.tokenizer

Class LuceneEnglishTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneEnglishTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneEnglishTokenizer
extends BaseTokenizer

The LuceneEnglishTokenizer class provides a tokenization implementation for the English language using Lucene's EnglishAnalyzer. It is integrated as the default tokenizer for the "en" (English) language in OmegaT.

This tokenizer can handle stop words and word stemming based on the provided configuration flags. It supports tokenization of text using either Lucene's standard tokenization or with additional analysis features such as stemming and stop word removal.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneEnglishTokenizer()

Method Summary

Modifier and Type	Method	Description
static void	loadPlugins()	Register plugin into OmegaT.
static void	unloadPlugins()

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneEnglishTokenizer

public LuceneEnglishTokenizer()

Method Details

loadPlugins

public static void loadPlugins()

Register plugin into OmegaT.

unloadPlugins

public static void unloadPlugins()