Package org.omegat.tokenizer

Class LuceneCJKTokenizer

java.lang.Object

org.omegat.tokenizer.BaseTokenizer

org.omegat.tokenizer.LuceneCJKTokenizer

All Implemented Interfaces:

ITokenizer

public class LuceneCJKTokenizer
extends BaseTokenizer

Tokenizer implementation specifically designed for processing CJK (Chinese, Japanese, Korean) text. It extends the BaseTokenizer to provide tokenization functionality tailored for CJK languages.

This tokenizer utilizes specific analysis tools like CJKAnalyzer, CJKBigramFilter, CJKWidthFilter, and LowerCaseFilter to process and create tokens suitable for CJK text indexing and searching.

Nested Class Summary

Nested classes/interfaces inherited from interface org.omegat.tokenizer.ITokenizer

ITokenizer.StemmingMode

Field Summary

Fields inherited from class org.omegat.tokenizer.BaseTokenizer

TOKENIZER_DEBUG_PROVIDER

Constructor Summary

Constructors

Constructor	Description
LuceneCJKTokenizer()

Method Summary

Methods inherited from class org.omegat.tokenizer.BaseTokenizer

getSupportedLanguages, tokenizeVerbatim, tokenizeVerbatimToStrings, tokenizeWords, tokenizeWordsToStrings

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneCJKTokenizer

public LuceneCJKTokenizer()