Class LayerNorm

java.lang.Object

deepnetts.net.layers.AbstractLayer<TensorBase, TensorBase, TensorBase>

deepnetts.net.layers.LayerNorm

All Implemented Interfaces:

Backward, Forward, Layer<TensorBase>, Serializable

public class LayerNorm
extends AbstractLayer<TensorBase, TensorBase, TensorBase>

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
Tensor3D	deltaInputs

Fields inherited from class AbstractLayer

activation, activationType, backwardImpl, batchMode, batchSize, biases, cudaHandles, deltaBiases, deltas, deltaWeights, depth, forwardImpl, gradients, height, inputs, learningRate, mode, momentum, networkType, nextLayer, numThreads, optimizer, optimizerType, outputs, prevDeltaBiases, prevDeltaWeights, prevLayer, randomWeightsType, regL1, regL2, threadPool, trainable, weights, width

Constructor Summary

Constructors

Constructor	Description
LayerNorm(int embeddingDim)
LayerNorm(int embeddingDim, float epsilon)

Method Summary

Modifier and Type	Method	Description
void	applyWeightChanges()	Applies weight changes to current weights Must be diferent for convolutional does nothing for MaxPooling Same for FullyConnected and OutputLayer
void	backward()	This method should implement backward pass in subclasses
void	calculateOutputError()
void	forward()	This method should implement forward pass in subclasses
void	init()	This method should implement layer initialization in subclasses, when a layer is added to the network (create weights, outputs, deltas, randomization etc.).
void	initBuffers(int embeddingDim, int sequenceLength, int batchSize)
void	initTransientFields()
void	initWeights(int embeddingDim)

Methods inherited from class AbstractLayer

getActivation, getActivationType, getBackwardAcc, getBatchSize, getBiases, getDeltaBiases, getDeltas, getDeltaWeights, getDepth, getForwardAcc, getGradients, getHeight, getL1Regularization, getL1WeightSum, getL2Regularization, getL2WeightSum, getLearningRate, getMode, getMomentum, getNetworkType, getNextLayer, getNumThreads, getOptimizer, getOptimizerType, getOutputs, getPrevDeltaBiases, getPrevDeltaWeights, getPrevlayer, getWeights, getWidth, isBatchMode, isTrainable, setActivationType, setBatchMode, setBatchSize, setBiases, setCudaHandles, setDeltas, setL1Regularization, setL2Regularization, setLearningRate, setMode, setMomentum, setNetworkType, setNextlayer, setOptimizerType, setOutputs, setPrevDeltaWeights, setPrevLayer, setThreadPool, setTrainable, setWeights, setWeights

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

deltaInputs

public Tensor3D deltaInputs

Constructor Details

LayerNorm

public LayerNorm(int embeddingDim)

LayerNorm

public LayerNorm(int embeddingDim,
 float epsilon)

Method Details

init

public void init()

Description copied from class: AbstractLayer

This method should implement layer initialization in subclasses, when a layer is added to the network (create weights, outputs, deltas, randomization etc.).

Specified by:

init in class AbstractLayer<TensorBase, TensorBase, TensorBase>

forward

public void forward()

Description copied from class: AbstractLayer

This method should implement forward pass in subclasses

Specified by:

forward in interface Forward

Specified by:

forward in interface Layer<TensorBase>

Specified by:

forward in class AbstractLayer<TensorBase, TensorBase, TensorBase>

backward

public void backward()

Description copied from class: AbstractLayer

This method should implement backward pass in subclasses

Specified by:

backward in interface Backward

Specified by:

backward in interface Layer<TensorBase>

Specified by:

backward in class AbstractLayer<TensorBase, TensorBase, TensorBase>

applyWeightChanges

public void applyWeightChanges()

Description copied from class: AbstractLayer

Applies weight changes to current weights Must be diferent for convolutional does nothing for MaxPooling Same for FullyConnected and OutputLayer

Specified by:

applyWeightChanges in class AbstractLayer<TensorBase, TensorBase, TensorBase>

initTransientFields

public void initTransientFields()

Overrides:

initTransientFields in class AbstractLayer<TensorBase, TensorBase, TensorBase>

initWeights

public void initWeights(int embeddingDim)

initBuffers

public void initBuffers(int embeddingDim,
 int sequenceLength,
 int batchSize)

calculateOutputError

public void calculateOutputError()