Tokenizer (OmegaT 6.0.0 API)

```
@Retention(RUNTIME)
public @interface Tokenizer
```
Annotation to indicate the languages for which a tokenizer is intended for use.

Field Summary

Fields
Modifier and Type	Fields	Description
`static java.lang.String`	`DISCOVER_AT_RUNTIME`	Value for the `languages()` member that indicates that the supported languages should be determined at runtime via `BaseTokenizer.getSupportedLanguages()`.

Required Element Summary

Required Elements
Modifier and Type Required Element Description

java.lang.String[] languages
The languages supported by the tokenizer.

Optional Element Summary

Optional Elements
Modifier and Type	Optional Element	Description
`boolean`	`isDefault`	When multiple tokenizers support the same language, this indicates the one that should be preferred above the others.

- Field Detail
  - DISCOVER_AT_RUNTIME
```
static final java.lang.String DISCOVER_AT_RUNTIME
```
    Value for the languages() member that indicates that the supported languages should be determined at runtime via BaseTokenizer.getSupportedLanguages().
- Element Detail
  - languages
```
java.lang.String[] languages
```
    The languages supported by the tokenizer. E.g. LuceneCJKTokenizer supports { zh, ja, ko }.
- - isDefault
```
boolean isDefault
```
    When multiple tokenizers support the same language, this indicates the one that should be preferred above the others.
    
    Default:
    
    false