Swish function information

The swish function is a mathematical function defined as follows:

\operatorname {swish} (x)=x\operatorname {sigmoid} (\beta x)={\frac {x}{1+e^{-\beta x}}}.

^[1]

where β is either constant or a trainable parameter depending on the model. For β = 1, the function becomes equivalent to the Sigmoid Linear Unit^[2] or SiLU, first proposed alongside the GELU in 2016. The SiLU was later rediscovered in 2017 as the Sigmoid-weighted Linear Unit (SiL) function used in reinforcement learning.^[3]^[1] The SiLU/SiL was then rediscovered as the swish over a year after its initial discovery, originally proposed without the learnable parameter β, so that β implicitly equalled 1. The swish paper was then updated to propose the activation with the learnable parameter β, though researchers usually let β = 1 and do not use the learnable parameter β. For β = 0, the function turns into the scaled linear function f(x) = x/2.^[1] With β → ∞, the sigmoid component approaches a 0-1 function pointwise, so swish approaches the ReLU function pointwise. Thus, it can be viewed as a smoothing function which nonlinearly interpolates between a linear function and the ReLU function.^[1] This function uses non-monotonicity, and may have influenced the proposal of other activation functions with this property such as Mish.^[4]

When considering positive values, Swish is a particular case of sigmoid shrinkage function defined in ^[5] (see the doubly parameterized sigmoid shrinkage form given by Equation (3) of this reference).

^ ^a ^b ^c ^d Cite error: The named reference Ramachandran-Zoph-Le_2017_v2 was invoked but never defined (see the help page).
^ Cite error: The named reference Hendrycks-Gimpel_2016 was invoked but never defined (see the help page).
^ Cite error: The named reference Elfwing-Uchibe-Doya_2017 was invoked but never defined (see the help page).
^ Misra, Diganta (2019). "Mish: A Self Regularized Non-Monotonic Neural Activation Function". arXiv:1908.08681 [cs.LG].
^ Atto, Abdourrahmane M.; Pastor, Dominique; Mercier, Gregoire (March 2008). "Smooth sigmoid wavelet shrinkage for non-parametric estimation". 2008 IEEE International Conference on Acoustics, Speech and Signal Processing (PDF). pp. 3265–3268. doi:10.1109/ICASSP.2008.4518347. ISBN 978-1-4244-1483-3. S2CID 9959057.

[Ramachandran-Zoph-Le_2017_v2-1] Cite error: The named reference Ramachandran-Zoph-Le_2017_v2 was invoked but never defined (see the help page).

[Hendrycks-Gimpel_2016-2] Cite error: The named reference Hendrycks-Gimpel_2016 was invoked but never defined (see the help page).

[Elfwing-Uchibe-Doya_2017-3] Cite error: The named reference Elfwing-Uchibe-Doya_2017 was invoked but never defined (see the help page).

[4] Misra, Diganta (2019). "Mish: A Self Regularized Non-Monotonic Neural Activation Function". arXiv:1908.08681 [cs.LG].

[5] Atto, Abdourrahmane M.; Pastor, Dominique; Mercier, Gregoire (March 2008). "Smooth sigmoid wavelet shrinkage for non-parametric estimation". 2008 IEEE International Conference on Acoustics, Speech and Signal Processing (PDF). pp. 3265–3268. doi:10.1109/ICASSP.2008.4518347. ISBN 978-1-4244-1483-3. S2CID 9959057.

Swish function information

and 22 Related for: Swish function information

Swish function

Sigmoid function

Swish

List of mathematical abbreviations

Activation function

Ride cymbal

Backpropagation

Dedekind eta function

Universal approximation theorem

List of search engines

SWF

List of Wild Kratts episodes

BankID

Mobile payment

List of animal sounds

Cum shot

Mullvad

Brooklyn Immersionists

Stunt cock

BRCA1

Aban Palace

MobilePay