Package org.omegat.tokenizer
package org.omegat.tokenizer
-
ClassDescriptionBase class for Lucene-based tokenizers.Methods for tokenize string.Methods for tokenize string.Interface for tokenize string engine.A tokenizer implementation for tokenizing Arabic text based on Lucene's ArabicAnalyzer.Tokenizer implementation for processing Armenian text.Tokenizer class specifically designed for processing Basque text.A tokenizer implementation for processing Brazilian Portuguese text with optional stemming and stop word removal capabilities.The LuceneBulgarianTokenizer class is a tokenizer implementation designed for processing Bulgarian text.Tokenizer implementation for processing Catalan language text.Tokenizer implementation specifically designed for processing CJK (Chinese, Japanese, Korean) text.A tokenizer implementation for the Czech language, extending the functionality of the BaseTokenizer.The LuceneDanishTokenizer class extends the BaseTokenizer and provides tokenization functionality for Danish language text processing.LuceneDutchTokenizer is a specialized tokenizer for processing Dutch language text.The LuceneEnglishTokenizer class provides a tokenization implementation for the English language using Lucene's EnglishAnalyzer.Tokenizer class for the Finnish language that extends the base tokenizer functionality.The LuceneFrenchTokenizer is a specialized tokenizer intended for processing French language text with Lucene.Tokenizer implementation for the Galician language based on Lucene.LuceneGermanTokenizer class provides tokenization functionality specifically designed for German language text.A tokenizer specifically designed for tokenizing Greek text.LuceneHindiTokenizer is a tokenizer implementation for the Hindi language, extending the base functionality of the BaseTokenizer.A tokenizer class for processing Hungarian text using the Lucene library.A tokenizer implementation designed for the Indonesian language, utilizing Lucene's IndonesianAnalyzer to provide specific tokenization and processing of text data.A tokenizer for processing text in the Irish language.Tokenizer implementation for processing Italian language text using Apache Lucene.A tokenizer implementation that specializes in processing Japanese text using Apache Lucene.A tokenizer for processing Latvian language text using Lucene's analysis capabilities.A tokenizer implementation for the Norwegian language, tailored for use with text processing tasks.A tokenizer for processing Persian text using Apache Lucene's
PersianAnalyzer.A tokenizer implementation for the Polish language using the Lucene analysis library.LucenePortugueseTokenizer provides a tokenization mechanism for processing Portuguese text.LuceneRomanianTokenizer is a tokenizer specifically designed for the Romanian language.The LuceneRussianTokenizer class provides text tokenization specifically for the Russian language.LuceneSmartChineseTokenizer is a tokenizer class for handling Chinese text processing using the SmartChineseAnalyzer or HMMChineseTokenizer from the Lucene library.A tokenizer implementation for the Spanish language using Lucene's analysis tools.The LuceneSwedishTokenizer class provides functionality to tokenize Swedish text strings while optionally applying stemming and stop word filtering.LuceneThaiTokenizer is a tokenizer implementation for processing Thai language text.The LuceneTurkishTokenizer provides tokenization for Turkish text.Annotation to indicate the languages for which a tokenizer is intended for use.BreakIterator for word-breaks with OmegaT heuristics, based on an instance of BreakIterator implementing word breaks.