Machine Learning

Pytorch/Matlab ML Toolbox    

Followings are a kind of cheatsheet that I frequently refer to when I am writing/reading Pytorch code. These are not my own note. These would lead you to Pytorch and Matlab official document page or someone else note in internet.

• torch.nn (Pytorch)
• Activation Function/Transfer Function (Matlab ML Toolbox)
• compet - Competitive transfer function.
• elliotsig - Elliot sigmoid transfer function.
• hardlim - Positive hard limit transfer function.
• hardlims - Symmetric hard limit transfer function.
• logsig - Logarithmic sigmoid transfer function.
• netinv - Inverse transfer function.
• poslin - Positive linear transfer function.
• purelin - Linear transfer function.
• radbas - Radial basis transfer function.
• radbasn - Radial basis normalized transfer function.
• satlin - Positive saturating linear transfer function.
• satlins - Symmetric saturating linear transfer function.
• softmax - Soft max transfer function.
• tansig - Symmetric sigmoid transfer function.
• tribas - Triangular basis transfer function.
• Activation Functions (Pytorch)
• torch.nn.AdaptiveLogSoftmaxWithLoss
• torch.nn.ELU
• torch.nn.Hardshrink
• torch.nn.Hardtanh
• torch.nn.LeakyReLU
• torch.nn.LogSigmoid
• torch.nn.LogSoftmax
• torch.nn.MultiheadAttention
• torch.nn.PReLU
• torch.nn.ReLU
• torch.nn.ReLU6
• torch.nn.RReLU
• torch.nn.SELU
• torch.nn.Sigmoid
• torch.nn.Softmin
• torch.nn.Softmax
• torch.nn.Softmax2d
• torch.nn.Softplus
• torch.nn.Softshrink
• torch.nn.Softsign
• torch.nn.Tanh
• torch.nn.Tanhshrink
• torch.nn.Threshold
• Loss Functions (Pytorch)
• torch.nn.BCELoss (Binary Cross Entropy)
• torch.nn.BCEWithLogitsLoss (combines a Sigmoid layer and the BCELoss in one single class)
• torch.nn.CosineEmbeddingLoss
• torch.nn.CrossEntropyLoss
• torch.nn.CTCLoss (Connectionist Temporal Classification loss)
• torch.nn.KLDivLoss (Kullback-Leibler divergence Loss)
• torch.nn.HingeEmbeddingLoss
• torch.nn.L1Loss (Mean Absolute Error)
• torch.nn.MarginRankingLoss
• torch.nn.MSELoss(Mean Square Error)
• torch.nn.MultiLabelMarginLoss
• torch.nn.MultiLabelSoftMarginLoss
• torch.nn.MultiMarginLoss
• torch.nn.NLLLoss (negative log likelihood loss)
• torch.nn.PoissonNLLLoss (Negative log likelihood loss with Poisson distribution of target)
• torch.nn.SmoothL1Loss
• torch.nn.SoftMarginLoss
• torch.nn.TripletMarginLoss
• Perform Function/Loss Functions (Matlab ML Toolbox)
• mae / Mean absolute error performance function
• mse / Mean squared normalized error performance function
• sae / Sum absolute error performance function
• sse / Sum squared error performance function
• torch.optim / Weight Update Algorithm
• torch.optim.Adadelta (Adaptive Learning Rate Method)
• torch.optim.Adagrad (Adaptive Subgradient Methods)
• torch.optim.Adam (Adam algorithm)
• torch.optim.AdamW (AdamW algorithm)
• torch.optim.SparseAdam (lazy version of Adam algorithm suitable for sparse tensors)
• torch.optim.Adamax (Adamax algorithm,a variant of Adam based on infinity norm).
• torch.optim.ASGD(Averaged Stochastic Gradient Descent)
• torch.optim.LBFGS (L-BFGS algorithm)
• torch.optim.RMSprop (RMS prop algorithm)
• torch.optim.Rprop (resilient backpropagation algorithm)
• torch.optim.SGD (Stochastic Gradient Descent)
• Training Function / Weight Update Algorithm (Matlab ML Toolbox)
• trainb / Batch training with weight and bias learning rules
• trainbu / Batch unsupervised weight/bias training
• trainc / Cyclical order weight/bias training
• trainr / Random order incremental training with learning functions
• trains / Sequential order incremental training with learning functions
• traingd / Gradient descent backpropagation
• traingdm / Gradient descent with momentum backpropagation
• traingdx / Gradient descent with momentum and adaptive learning rate backpropagation
• trainlm / Levenberg-Marquardt backpropagation
• trainru / Unsupervised random order weight/bias training
• trainscg / Scaled conjugate gradient backpropagation