Package org.omegat.tokenizer

Class LuceneTurkishTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneTurkishTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneTurkishTokenizer
extends BaseTokenizer

The LuceneTurkishTokenizer provides tokenization for Turkish text. This tokenizer utilizes Lucene's TurkishAnalyzer to process text into tokens, supporting optional stemming and stop words removal based on the provided configuration. It is designed to integrate with the Lucene framework for text analysis.

This class extends the functionality of the BaseTokenizer and overrides the required method for obtaining the token stream for Turkish language processing.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneTurkishTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneTurkishTokenizer

public LuceneTurkishTokenizer()