Package org.omegat.tokenizer

Class LuceneIndonesianTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneIndonesianTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneIndonesianTokenizer
extends BaseTokenizer

A tokenizer implementation designed for the Indonesian language, utilizing Lucene's IndonesianAnalyzer to provide specific tokenization and processing of text data. This tokenizer supports stemming and stop-word removal based on configuration.

This class extends the BaseTokenizer and overrides the getTokenStream method to produce a token stream for Indonesian text based on the provided parameters.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneIndonesianTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneIndonesianTokenizer

public LuceneIndonesianTokenizer()