Class BPETokenizerParameters

java.lang.Object

deepnetts.tokenizers.bpe.BPETokenizerParameters

public class BPETokenizerParameters
extends Object

Constructor Summary

Constructors

Constructor	Description
BPETokenizerParameters(Map<Integer,byte[]> vocab, Map<Pair<Integer,Integer>, Integer> merges)

Method Summary

Modifier and Type	Method	Description
Map<Pair<Integer,Integer>, Integer>	getMerges()
Map<Integer,byte[]>	getVocab()
void	setMerges(Map<Pair<Integer,Integer>, Integer> merges)
void	setVocab(Map<Integer,byte[]> vocab)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BPETokenizerParameters

public BPETokenizerParameters(Map<Integer,byte[]> vocab,
 Map<Pair<Integer,Integer>, Integer> merges)

Method Details

getVocab

public Map<Integer,byte[]> getVocab()

getMerges

public Map<Pair<Integer,Integer>, Integer> getMerges()

setVocab

public void setVocab(Map<Integer,byte[]> vocab)

setMerges

public void setMerges(Map<Pair<Integer,Integer>, Integer> merges)