Loss Capabilities in TensorFlow

Final Up to date on July 15, 2022

Loss metric is essential for neural networks. As all machine studying mannequin is a optimization downside or one other, the loss is the target operate to reduce. In neural networks, the optimization is finished with gradient descent and backpropagation. However what are loss features and the way are they affecting our neural networks?

On this submit, we’ll cowl what loss features are and go into some generally used loss features and how one can apply them to your neural networks.

After studying this text, you’ll be taught:

• What are loss features and the way they’re completely different from metrics
• Frequent loss features for regression and classification issues
• Easy methods to use loss features in your TensorFlow mannequin

Let’s get began!

Loss features in TensorFlow.
Photograph by Ian Taylor. Some rights reserved.

Overview

This text is break up into 5 part; they’re:

• What are loss features?
• Imply absolute error
• Imply squared error
• Categorical cross-entropy
• Loss features in apply

What are loss features?

In neural networks, loss features assist optimize the efficiency of the mannequin. They’re normally used to measure some penalty that the mannequin is incurring on its predictions, such because the deviation of the prediction away from the bottom fact label. Loss features are normally differentiable throughout their area (however it’s allowed that the gradient is undefined just for very particular factors, reminiscent of x = 0, which is mainly ignored in apply). Within the coaching loop, we differentiate them with respect to parameters and used these gradients for our backpropagation and gradient descent steps to optimize our mannequin on the coaching set.

Loss features are additionally barely completely different from metrics. Whereas loss features can inform us the efficiency of our mannequin, they won’t be of direct curiosity or simply explainable by people. That is the place metrics are available in. Metrics reminiscent of accuracy are far more helpful for people to know the efficiency of a neural community regardless that they won’t be good decisions for loss features since they won’t be differentiable.

Within the following, let’s discover some widespread loss features, particularly, the imply absolute error, imply squared error, and categorical cross entropy.

Imply Absolute Error

The imply absolute error (MAE) measures absolutely the distinction between predicted values and the bottom fact labels and takes the imply of the distinction throughout all coaching examples. Mathematically, it is the same as $frac{1}{m}sum_{i=1}^mlverthat{y}_i–y_irvert$ the place $m$ is the variety of coaching examples, $y_i$ and $hat{y}_i$ are the bottom fact and predicted values respectively, and we’re averaging over all coaching examples. The MAE is rarely adverse, however it will be zero provided that the prediction matched the bottom fact completely. It’s an intuitive loss operate and may additionally be used as one in all our metrics, particularly for regression issues since we might need to reduce the error in our predictions.

Let’s take a look at what the imply absolute error loss operate seems to be like graphically:

Imply absolute error loss operate, floor fact at x = 0 and x-axis symbolize the anticipated worth

Just like activation features, we’re normally additionally interested by what the gradient of the loss operate seems to be like since we’re utilizing the gradient in a while to do backpropagation to coach our mannequin’s parameters.

Imply absolute error loss operate (blue) and gradient (orange)

We discover that there’s a discontinuity within the gradient operate for the imply absolute loss operate however we are inclined to ignore it because it happens solely at x = 0 which in apply hardly ever occurs since it’s the likelihood of a single level in a steady distribution.

Let’s check out how one can implement this loss operate in TensorFlow utilizing the the Keras losses module:

which provides us 2.0 because the output as anticipated, since $ frac{1}{2}(lvert 2-1rvert + lvert 3-0rvert) = frac{1}{2}(4) = 4 $. Subsequent, let’s discover one other loss operate for regression fashions with barely completely different properties, the imply squared error.

Imply Squared Error

One other in style loss operate for regression fashions is the imply squared error (MSE), which is the same as $frac{1}{m}sum_{i=1}^m(hat{y}_i–y_i)^2$. It’s much like the imply absolute error because it additionally measures the deviation of the anticipated worth from the bottom fact worth. Nevertheless, the imply squared error squares this distinction (all the time non-negative since sq. of actual numbers are all the time non-negative), which provides it barely completely different properties.

One notable one is that the imply squared error favors a lot of small errors over a small variety of massive errors, which results in fashions which have much less outliers or not less than outliers which can be much less extreme than fashions skilled with a imply absolute error. It’s because a big error would have a considerably bigger influence on the error, and consequently the gradient of the error, when in comparison with a small error.

Graphically,

Imply squared error loss operate, floor fact at x = 0 and x-axis symbolize the anticipated worth

Then, wanting on the gradient,

Imply squared error loss operate (blue) and gradient (orange)

Discover that bigger errors would result in a bigger magnitude for the gradient and likewise a bigger loss. Therefore, for instance, two coaching examples that deviate from their floor truths by 1 unit would result in a lack of 2, whereas a single coaching instance that deviates from its floor fact by 2 items would result in a lack of 4, therefore having a bigger influence.

Let’s take a look at how one can implement the imply squared loss in TensorFlow.

which provides the output 5.0 as anticipated since $frac{1}{2}[(2-1)^2 + (3-0)^2] = frac{1}{2}(10) = 5$. Discover that the second instance with a predicted worth of three and precise worth of 0 contributes 90% of the error beneath the imply squared error vs 75% of the error beneath imply absolute error.

Generally, you might even see individuals use root imply squared error (RMSE) as a metric. That is to take the sq. root of MSE. From the attitude of a loss operate, MSE and RMSE are equal.

Each MAE and MSE are measuring values in a steady vary. Therefore they’re for regression issues. For classification issues, we are able to use categorical cross-entropy.

Categorical Cross-entropy

The earlier two loss features are for regression fashions, the place the output may very well be any actual quantity. Nevertheless, for classification issues, there’s a small, discrete set of numbers that the output may take. Moreover, the quantity that we use to label-encode the courses are arbitrary, and with no semantic that means (e.g. if we used the labels 0 for cat, 1 for canine, and a pair of for horse, it doesn’t symbolize {that a} canine is half cat and half horse). Subsequently it shouldn’t have an effect on the efficiency of the mannequin.

In a classification downside, the mannequin’s output is a vector of likelihood for every class. In Keras fashions, normally we anticipate this vector to be “logits”, i.e., actual numbers to be remodeled to likelihood utilizing softmax operate, or the output of a softmax activation operate.

The cross-entropy between two likelihood distributions is a measure of the distinction between the 2 likelihood distributions. Exactly, it’s $-sum_i P(X = x_i) log Q(X = x_i)$ for likelihood $P$ and $Q$. In machine studying, we normally have the likelihood $P$ supplied by the coaching knowledge and $Q$ predicted by the mannequin, which $P$ is 1 for the right class and 0 for each different class. The expected likelihood $Q$, nevertheless, is normally valued between 0 and 1. Therefore when used for classification issues in machine studying, this components might be simplified into: $$textual content{categorical cross entropy} = – log p_{gt}$$ the place $p_{gt}$ is the model-predicted likelihood of the bottom fact class for that exact pattern.

Cross-entropy metric have a adverse signal as a result of $log(x)$ tends to adverse infinity as $x$ tends to zero. We would like a better loss when the likelihood approaches 0 and a decrease loss when the likelihood approaches 1. Graphically,

Categorical cross entropy loss operate, the place x is the anticipated likelihood of the bottom fact class

Discover that the loss is strictly 0 if the likelihood of the bottom fact class is 1 as desired. Additionally, because the likelihood of the bottom fact class tends to 0, the loss tends to optimistic infinity as nicely, therefore considerably penalizing dangerous predictions. You may acknowledge this loss operate for logistic regression, and they’re related besides the logistic regression loss is particular to the case of binary courses.

Categorical cross entropy loss operate (blue) and gradient (orange)

Now, wanting on the gradient of the cross entropy loss,

Wanting on the gradient, we are able to see that the gradient is usually adverse which can be anticipated since to lower this loss, we might need the likelihood on the bottom fact class to be as excessive as potential, and recall that gradient descent goes in the other way of the gradient.

There are two alternative ways to implement categorical cross entropy in TensorFlow. The primary technique takes in one-hot vectors as enter,

This provides the output as 0.2876821 which is the same as $-log(0.75)$ as anticipated. The opposite method of implementing the explicit cross entropy loss in TensorFlow is utilizing a label-encoded illustration for the category, the place the category is represented by a single non-negative integer indicating the bottom fact class as an alternative.

which likewise offers the output 0.2876821.

Now that we’ve explored loss features for each regression and classification fashions, let’s check out how we use loss features in our machine studying fashions.

Loss Capabilities in Apply

Let’s discover how we are able to use loss features in apply. We’ll discover this by means of a easy dense mannequin on the MNIST digit classification dataset.

First, we get the obtain the info from Keras datasets module,

Then, we construct our mannequin,

And we take a look at the mannequin structure outputted from the above code,

We will then compile our mannequin, which can be the place we introduce the loss operate. Since it is a classification downside, we’ll use the cross entropy loss. Specifically, because the MNIST dataset in Keras datasets is represented as a label as an alternative of an one-hot vector, we’ll use the SparseCategoricalCrossEntropy loss.

And eventually, we practice our mannequin.

And our mannequin efficiently trains with the next output:

And that’s one instance of how one can use a loss operate in a TensorFlow mannequin.

Additional Studying

Under are the documentation of the loss features from TensorFlow/Keras:

Conclusion

On this submit, you may have seen loss features and the function that they play in a neural community. You have got additionally seen some in style loss features utilized in regression and classification fashions, in addition to how one can use the cross entropy loss operate in a TensorFlow mannequin.

Particularly, you discovered:

• What are loss features and the way they’re completely different from metrics
• Frequent loss features for regression and classification issues
• Easy methods to use loss features in your TensorFlow mannequin