Tokenizer

java.lang.Object

com.reign.kat.lib.voice.speech.Tokenizer

public class Tokenizer extends Object

Allows us to parse text into a series of generalized tokens which make up the structure of the sentence.

Constructor Summary

Constructors

Constructor

Description

Tokenizer()
Method Summary

Modifier and Type

Method

Description

static void

init(int tableVersion, String tokenTablePath)

Initializes the Tokenizer and creates the necessary token bindings

static TokenResult

tokenize(String[] text)

Attempts to convert a list of Strings into a valid series of tokens

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details
- Tokenizer
  
  public Tokenizer()
Method Details
- init
  
  public static void init(int tableVersion, String tokenTablePath)
  
  Initializes the Tokenizer and creates the necessary token bindings
  
  Parameters:
  
  tableVersion - tokentable format version to use. (will throw an error if we try to read an incompatible file version)
  
  tokenTablePath - path to tokentable.json
- tokenize
  
  public static TokenResult tokenize(String[] text)
  
  Attempts to convert a list of Strings into a valid series of tokens
  
  Parameters:
  
  text - List of strings to convert
  
  Returns:
  
  List of tokens created from text