Class BPETokenizer

java.lang.Object

deepnetts.tokenizers.bpe.BPETokenizer

All Implemented Interfaces:

Tokenizer

public class BPETokenizer
extends Object
implements Tokenizer

Constructor Summary

Constructors

Constructor	Description
BPETokenizer(BPETokenizerParameters bpeTokenizerParameters)

Method Summary

Modifier and Type	Method	Description
String	decode(List<Integer> indices)
List<Integer>	encode(String text)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BPETokenizer

public BPETokenizer(BPETokenizerParameters bpeTokenizerParameters)

Method Details

decode

public String decode(List<Integer> indices)

Specified by:

decode in interface Tokenizer

encode

public List<Integer> encode(String text)

Specified by:

encode in interface Tokenizer