Package org.omegat.tokenizer

Annotation Interface Tokenizer

@Retention(RUNTIME) public @interface Tokenizer
Annotation to indicate the languages for which a tokenizer is intended for use.

  • Required Element Summary

    Required Elements
    Modifier and Type
    Required Element
    Description
    String[]
    languages
    The languages supported by the tokenizer.

  • Optional Element Summary

    Optional Elements
    Modifier and Type
    Optional Element
    Description
    boolean
    isDefault
    When multiple tokenizers support the same language, this indicates the one that should be preferred above the others.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    DISCOVER_AT_RUNTIME
    Value for the languages() member that indicates that the supported languages should be determined at runtime via BaseTokenizer.getSupportedLanguages().

  • Field Details

  • Element Details

    • languages

      String[] languages
      The languages supported by the tokenizer. E.g. LuceneCJKTokenizer supports { zh, ja, ko }.

    • isDefault

      boolean isDefault
      When multiple tokenizers support the same language, this indicates the one that should be preferred above the others.
      Default:
      false