What is "metrics" in Keras?
Solution 1
So in order to understand what metrics
are, it's good to start by understanding what a loss
function is. Neural networks are mostly trained using gradient methods by an iterative process of decreasing a loss
function.
A loss
is designed to have two crucial properties - first, the smaller its value is, the better your model fits your data, and second, it should be differentiable. So, knowing this, we could fully define what a metric
is: it's a function that, given predicted values and ground truth values from examples, provides you with a scalar measure of a "fitness" of your model, to the data you have. So, as you may see, a loss
function is a metric, but the opposite doesn't always hold. To understand these differences, let's look at the most common examples of metrics
usage:
Measure a performance of your network using non-differentiable functions: e.g. accuracy is not differentiable (not even continuous) so you cannot directly optimize your network w.r.t. to it. However, you could use it in order to choose the model with the best accuracy.
Obtain values of different loss functions when your final loss is a combination of a few of them: Let's assume that your loss has a regularization term which measures how your weights differ from
0
, and a term which measures the fitness of your model. In this case, you could usemetrics
in order to have a separate track of how the fitness of your model changes across epochs.Track a measure with respect to which you don't want to directly optimize your model: so - let's assume that you are solving a multidimensional regression problem where you are mostly concerned about
mse
, but at the same time you are interested in how acosine-distance
of your solution is changing in time. Then, it's the best to usemetrics
.
I hope that the explanation presented above made obvious what metrics are used for, and why you could use multiple metrics in one model. So now, let's say a few words about mechanics of their usage in keras
. There are two ways of computing them while training:
Using
metrics
defined while compilation: this is what you directly asked. In this case,keras
is defining a separate tensor for each metric you defined, to have it computed while training. This usually makes computation faster, but this comes at a cost of additional compilations, and the fact that metrics should be defined in terms ofkeras.backend
functions.Using
keras.callback
: It is nice that you can useCallbacks
in order to compute your metrics. As each callback has a default attribute ofmodel
, you could compute a variety of metrics usingmodel.predict
or model parameters while training. Moreover, it makes it possible to compute it, not only epoch-wise, but also batch-wise, or training-wise. This comes at a cost of slower computations, and more complicated logic - as you need to define metrics on your own.
Here you can find a list of available metrics, as well as an example on how you could define your own.
Solution 2
As in keras metrics page described:
A metric is a function that is used to judge the performance of your model
Metrics are frequently used with early stopping callback to terminate training and avoid overfitting
Solution 3
Reference: Keras Metrics Documentation
As given in the documentation page of keras metrics
, a metric
judges the performance of your model. The metrics
argument in the compile
method holds the list of metrics that needs to be evaluated by the model during its training and testing phases.
Metrics like:
binary_accuracy
categorical_accuracy
sparse_categorical_accuracy
top_k_categorical_accuracy
andsparse_top_k_categorical_accuracy
are the available metric functions that are supplied in the metrics
parameter when the model is compiled.
Metric functions are customizable as well. When multiple metrics need to be evaluated it is passed in the form of a dictionary
or a list
.
One important resource you should refer for diving deep into metrics can be found here
Solution 4
From an implementation point of view, losses and metrics are actually identical functions in Keras:
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow.keras as Keras
>>> print(Keras.losses.mean_squared_error == Keras.metrics.mean_squared_error)
True
>>> print(Keras.losses.poisson == Keras.metrics.poisson)
True
Solution 5
Loss helps find the best solution your model can produce. Metric actually tells us how good it is. Imagine, we found the regression line (that has the least minimum squared error). Is that a good enough solution? This is what the metric will answer(considering the shape and spread of data, ideally!).
DragonKnight
Updated on June 07, 2022Comments
-
DragonKnight almost 2 years
It is not yet clear for me what
metrics
are (as given in the code below). What exactly are they evaluating? Why do we need to define them in themodel
? Why we can have multiple metrics in one model? And more importantly what is the mechanics behind all this? Any scientific reference is also appreciated.model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['mae', 'acc'])
-
DragonKnight over 6 yearswould you provide a reference to see what is its mechanic?
-
Nic Cottrell almost 5 yearsWhen you say "track a measure" - is this so we can visualize the metric(s) after training has taken place to see how quickly/smoothly the model has trained?
-
Tom N Tech almost 5 yearsAny references on how to implement or use batch-wise or training-wise metrics?
-
spencer741 about 4 yearsJust as an addition to this useful answer for keras related things... "In addition to the metrics above, you may use any of the loss functions described in the loss function page as metrics." ... from the keras docs. The great explanation by @Marcin Możejko said "a loss function is a metric." It isn't noticeable in the Keras docs that the same functions are available for loss and metrics, but they do say it towards the bottom of the docs in smaller print.