Package org.omegat.tokenizer

Class LuceneHindiTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneHindiTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneHindiTokenizer
extends BaseTokenizer

LuceneHindiTokenizer is a tokenizer implementation for the Hindi language, extending the base functionality of the BaseTokenizer. It processes input text to generate tokens while optionally applying stemming and stopword filtering based on the provided configuration.

This class utilizes the Lucene HindiAnalyzer for tokenization and supports the inclusion or exclusion of stop words and stemming during the tokenization process.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneHindiTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneHindiTokenizer

public LuceneHindiTokenizer()