Package org.omegat.tokenizer

Class LuceneFrenchTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneFrenchTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneFrenchTokenizer
extends BaseTokenizer

The LuceneFrenchTokenizer is a specialized tokenizer intended for processing French language text with Lucene. It extends the functionality of the BaseTokenizer and integrates language-specific tokenization features.

This tokenizer provides support for stop words and stemming, enabling enhanced linguistic analysis for French text. When stop words and/or stemming are enabled, it utilizes the FrenchAnalyzer from the Lucene library. If stemming is disabled, standard tokenization is applied.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneFrenchTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneFrenchTokenizer

public LuceneFrenchTokenizer()