Package org.omegat.tokenizer

Class DefaultTokenizer

java.lang.Object
org.omegat.tokenizer.DefaultTokenizer
All Implemented Interfaces:
ITokenizer
public class DefaultTokenizer extends Object implements ITokenizer
Methods for tokenize string.

  • Field Details

    • EMPTY_TOKENS_LIST

      public static final Token[] EMPTY_TOKENS_LIST

    • EMPTY_STRINGS_LIST

      public static final String[] EMPTY_STRINGS_LIST

  • Constructor Details

    • DefaultTokenizer

      public DefaultTokenizer()

  • Method Details

    • tokenizeWords

      public Token[] tokenizeWords(String strOrig, ITokenizer.StemmingMode stemmingMode)
      Breaks a string into word-only tokens. Numbers, tags, and other non-word tokens are NOT included in the result. Stemming can be used depending on the supplied ITokenizer.StemmingMode.

      This method is used to find fuzzy matches and glossary entries.

      Results can be cached for better performance.

      Specified by:
      tokenizeWords in interface ITokenizer

    • tokenizeWordsToStrings

      public String[] tokenizeWordsToStrings(String str, ITokenizer.StemmingMode stemmingMode)
      Description copied from interface: ITokenizer
      Breaks a string into word-only strings. Numbers, tags, and other non-word tokens are NOT included in the result. Stemming can be used depending on the supplied ITokenizer.StemmingMode.

      When stemming is used, both the original word and its stem may be included in the results, if they differ. (The stem will come first.)

      This method used for dictionary lookup.

      Results are not cached.

      Specified by:
      tokenizeWordsToStrings in interface ITokenizer

    • tokenizeVerbatim

      public Token[] tokenizeVerbatim(String strOrig)
      Description copied from interface: ITokenizer
      Breaks a string into tokens. Numbers, tags, and other non-word tokens are included in the result. Stemming is NOT used.

      This method is used to mark string differences in the UI and to tune similarity.

      Results are not cached.

      Specified by:
      tokenizeVerbatim in interface ITokenizer

    • tokenizeVerbatimToStrings

      public String[] tokenizeVerbatimToStrings(String str)
      Description copied from interface: ITokenizer
      Breaks a string into strings. Numbers, tags, and other non-word tokens are included in the result. Stemming is NOT used.

      This method is used to mark string differences in the UI and for debugging purposes.

      Results are not cached.

      Specified by:
      tokenizeVerbatimToStrings in interface ITokenizer

    • getWordBreaker

      public static BreakIterator getWordBreaker()
      Returns an iterator to break sentences into words.

    • isContains

      public static boolean isContains(Token[] tokensList, Token tokenForCheck)
      Check if array contains token.

    • isContainsAll

      public static boolean isContainsAll(Token[] tokensList, Token[] listForFind, boolean notExact)
      Check if the listForFind tokens are present in tokensList.
      Parameters:
      tokensList - a list of tokens to be searched
      listForFind - a list of tokens to search in tokensList
      notExact - is true if the tokens in listForFind can be non-contiguous or in a different order in the tokensList. If false, tokens must be exactly the same.
      Returns:
      true if the tokens in listForFind are found in tokensList

    • searchAll

      public static List<Token[]> searchAll(Token[] tokensList, Token[] listForFind, boolean notExact)
      Find and return all tokens in tokensList that match the tokens in listForFind.
      Parameters:
      tokensList - a list of tokens to be searched
      listForFind - a list of tokens to search in tokensList
      notExact - is true if the tokens in listForFind can be non-contiguous or in a different order in the tokensList. If false, tokens must be exactly the same.
      Returns:
      A list containing each hit of the matched tokens. Each token array represents a different instance of listForFind that was found in tokensList.

    • getSupportedLanguages

      public String[] getSupportedLanguages()
      Description copied from interface: ITokenizer
      Return an array of language strings (xx-yy) indicating the tokenizer's supported languages. Meant for tokenizers for which the supported languages can only be determined at runtime, like the HunspellTokenizer.

      Indicate that this should be used by setting the Tokenizer annotation to contain only Tokenizer.DISCOVER_AT_RUNTIME.

      Specified by:
      getSupportedLanguages in interface ITokenizer