Package org.omegat.tokenizer

Class LuceneThaiTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneThaiTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneThaiTokenizer
extends BaseTokenizer

LuceneThaiTokenizer is a tokenizer implementation for processing Thai language text. It extends the BaseTokenizer and provides tokenization support for Thai text, utilizing the ThaiAnalyzer from Apache Lucene.

This tokenizer allows optional stemming and the inclusion of stop words during the tokenization process.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneThaiTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneThaiTokenizer

public LuceneThaiTokenizer()