Package org.omegat.tokenizer

Class LucenePersianTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LucenePersianTokenizer

All Implemented Interfaces:

ITokenizer

public class LucenePersianTokenizer
extends BaseTokenizer

A tokenizer for processing Persian text using Apache Lucene's PersianAnalyzer.

This tokenizer provides token streams specifically tailored for the Persian language with options to enable or disable stemming and stop word processing. It leverages the default stop word set provided in the PersianAnalyzer.

The class is designed to handle text tokenization requirements for Persian, considering the unique linguistic characteristics of the language.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LucenePersianTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LucenePersianTokenizer

public LucenePersianTokenizer()