WordTokenizer (Deep Netts Core Pro 4.0.0 API)

java.lang.Object

deepnetts.tokenizers.WordTokenizer

All Implemented Interfaces:: Tokenizer

public class WordTokenizer extends Object implements Tokenizer

Constructor Summary

Constructors

Constructor

Description

WordTokenizer(WordTokenizerParameters wordTokenizerParameters)
Method Summary

Modifier and Type

Method

Description

String

decode(List<Integer> indices)

List<Integer>

encode(String text)

static WordTokenizerParameters

train(String text)

Methods inherited from class Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details
- WordTokenizer
  
  public WordTokenizer(WordTokenizerParameters wordTokenizerParameters)
Method Details
- encode
  
  public List<Integer> encode(String text)
  
  Specified by:
  
  encode in interface Tokenizer
- decode
  
  public String decode(List<Integer> indices)
  
  Specified by:
  
  decode in interface Tokenizer
- train
  
  public static WordTokenizerParameters train(String text)