This section describes the layers that calculate the value of specified activation functions for each of their inputs.
CPtr<CBaseLayer> NEOML_API CreateActivationLayer( const CActivationDesc& activation );
Creates a layer that calculates the activation function specified by the activation
parameter.
To create a layer, one must specify its type and, optionally, parameters. All configurable activation layers have a CParam
structure with all possible settings.
class CActivationDesc {
public:
// For non-parametrized activations or default parameters
CActivationDesc( TActivationFunction type );
// For explicitly setting activation parameters. 'Param' must be a 'CParam' struct from the correspondent layer
template<class Param>
CActivationDesc( TActivationFunction type, const Param& param );
// Changing/setting parameters of the selected activation.
// 'Param' must be a 'CParam' struct from the correspondent layer.
template<class Param>
void SetParam( const Param& param );
/// etc.
};
enum TActivationFunction {
AF_Linear = 0,
AF_ELU,
AF_ReLU,
AF_LeakyReLU,
AF_Abs,
AF_Sigmoid,
AF_Tanh,
AF_HardTanh,
AF_HardSigmoid,
AF_Power,
AF_HSwish,
AF_GELU,
AF_Exp,
AF_Log,
AF_Count
};
TActivationFunction constant | Class name | Activation function |
---|---|---|
AF_Linear |
CLinearLayer | a linear activation function: ax + b |
AF_ELU |
CELULayer | ELU activation function |
AF_ReLU |
CReLULayer | ReLU activation function |
AF_LeakyReLU |
CLeakyReLULayer | LeakyReLU activation function |
AF_Abs |
CAbsLayer | abs(x) activation function |
AF_Sigmoid |
CSigmoidLayer | sigmoid activation function |
AF_Tanh |
CTanhLayer | tanh activation function |
AF_HardTanh |
CHardTanhLayer | HardTanh activation function |
AF_HardSigmoid |
CHardSigmoidLayer | HardSigmoid activation function |
AF_Power |
CPowerLayer | pow(x, exp) activation function |
AF_HSwish |
CHSwishLayer | h-swish activation function |
AF_GELU |
CGELULayer | x * F( X < x ) activation function, where X ~ N(0, 1) |
AF_Exp |
CExpLayer | exp activation function |
AF_Log |
CLogLayer | log activation function |
AF_Erf |
CErfLayer | erf activation function |
AF_Count |
This is an auxiliary constant: it contains the number of supported activation functions. |