Interface Tokenizer

All Known Implementing Classes:

BPETokenizer, WordTokenizer

public interface Tokenizer

Method Summary

Modifier and Type	Method	Description
String	decode(List<Integer> indices)
List<Integer>	encode(String text)

Method Details

encode

List<Integer> encode(String text)

decode

String decode(List<Integer> indices)