Package org.omegat.tokenizer

Class LuceneDanishTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneDanishTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneDanishTokenizer
extends BaseTokenizer

The LuceneDanishTokenizer class extends the BaseTokenizer and provides tokenization functionality for Danish language text processing. It uses the Lucene DanishAnalyzer to handle language-specific tokenization, including stemming and stop words, when enabled.

This tokenizer is marked as the default tokenizer for Danish ("da") language support and works with customizable options for stemming and stop word removal.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneDanishTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneDanishTokenizer

public LuceneDanishTokenizer()