Package org.omegat.tokenizer

Class LuceneJapaneseTokenizer

java.lang.Object
org.omegat.tokenizer.BaseTokenizer
org.omegat.tokenizer.LuceneJapaneseTokenizer
All Implemented Interfaces:
ITokenizer
public class LuceneJapaneseTokenizer extends BaseTokenizer
A tokenizer implementation that specializes in processing Japanese text using Apache Lucene. This tokenizer is designed to handle text analysis for the Japanese language, with support for stemming, stop word filtering, and OmegaT-style tag handling. It extends the BaseTokenizer and integrates with Lucene's Japanese language processing tools.

Features: 1. Provides tokenization for Japanese text using Lucene's JapaneseAnalyzer and JapaneseTokenizer. 2. Supports optional stemming and stop word filtering during tokenization. 3. Handles OmegaT-style tags by either blanking out or reassembling them as needed. 4. Includes a custom LuceneJapaneseTokenizer.TagJoiningFilter to reassemble fragmented tags during tokenization.

  • Constructor Details

    • LuceneJapaneseTokenizer

      public LuceneJapaneseTokenizer()
      Initializes the tokenizer and sets up default configurations. Tags are not delegated for exact tokenization by default.