Type Alias TokenizerFunction

TokenizerFunction: (token: string, language?: string) => string[]

Type declaration

- (token: string, language?: string): string[]
- Parameters
  - token: string
  - Optionallanguage: string
  Returns string[]