Package org.omegat.tokenizer

Class LuceneHungarianTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneHungarianTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneHungarianTokenizer
extends BaseTokenizer

A tokenizer class for processing Hungarian text using the Lucene library. This class extends the BaseTokenizer and provides functionality for tokenizing Hungarian text, with optional stemming and stop-word filtering based on the provided configurations.

This tokenizer uses the HungarianAnalyzer from Lucene to perform text analysis when stemming is enabled. If stemming is disabled, a standard token stream is returned.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneHungarianTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneHungarianTokenizer

public LuceneHungarianTokenizer()