Package org.omegat.tokenizer

Class LuceneArabicTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneArabicTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneArabicTokenizer
extends BaseTokenizer

A tokenizer implementation for tokenizing Arabic text based on Lucene's ArabicAnalyzer. This tokenizer extends BaseTokenizer and can generate tokens with optional stemming and/or stop word removal based on the provided parameters.

The class is annotated with @Tokenizer and explicitly supports the Arabic language ("ar").

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneArabicTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneArabicTokenizer

public LuceneArabicTokenizer()