Skip to main content

Why Evaluation Metrics Matters


This is a follow up article on "The Importance of F1 Score" in which we understood the technical aspects of evaluating a Machine Learning model. In this article we will understand how different evaluation metrics can help us in designing solutions based on the problem statement and domain.

I will like to distinguish evaluation metrics with respect to the following criteria:

When there is life involved:

In case of aircraft or ships, where a Machine Learning algorithm misses to detect a failure can be a costly affair, since there is life involved. Also in the case of detecting cancer, a failure to predict a true positive can lead to delay in treatment, and hence can be life threatening.

In cases such as detecting failures in aircrafts, even if we get false positives is acceptable, but not able to detect true positives can be too expensive in terms of loss of life. Thus it becomes important to use Recall as a measure, as any positive detection becomes important, even if it turns out to be false positive. It is reasonable to waste resources on analyzing a false positive in such cases.

When there is cost involved:

In cases where the cost of responding to a false positive is too high, and where there are no lives involved, it is acceptable even if few failures are not detected. Take for example, a power plant is located in a village. Responding to a false positive from a sensor can be costly, as the transportation cost to check whether the fault has really occurred is too high. In such a scenario it becomes important to consider Precision as a measure. Thus the model should be trained for high precision, as the quality of the detection by the Machine Learning algorithm matters.


When two or more models need to be compared:

In cases where we need to perform comparison between two or more models going with a single metric like Precision or Recall may not be right. In such cases it is advisable to consider the harmonic mean of both the metrics, and hence F1 score matters in such scenarios. The model with high F1 score should be considered, except when the first two criteria are involved.

Understanding evaluation metrics can help you in deciding what model to choose, which metric matters for which domain. It will also help you to interact with your client and understand different Machine Learning algorithms better. Also it gives you the ability of understanding a problem statement well, thus helping you to implement the correct Machine Learning solution.

Evaluation metrics should be the main area of focus while designing Machine Learning solutions.


By,
Siddhesh Wagle,
Research Consultant,
Cere Labs





 

Comments

Post a Comment

Popular posts from this blog

GPU - The brain of Artificial Intelligence

Machine Learning algorithms require tens and thousands of CPU based servers to train a model, which turns out to be an expensive activity. Machine Learning researchers and engineers are often faced with the problem of running their algorithms fast. Although initially invented for processing graphics in computer games, GPUs today are used in machine learning to perform feature detection from vast amount of unlabeled data. Compared to CPUs, GPUs take far less time to train models that perform classification and prediction. Characteristics of GPUs that make them ideal for machine learning Handle large datasets Needs far less data centre infrastructure Can be specialized for specific machine learning needs Perform vector computations faster than any known processor Designed to perform data parallel computation NVIDIA CUDA GPUs today are used to build deep learning image processing tools for  Adobe Creative Cloud. According to NVIDIA blog future Adobe applicati

Understanding Projection Pursuit Regression

The following article gives an overview of the paper "Projection Pursuit Regression” published by Friedman J. H and Stuetzle W. You will need basic background of Machine Learning and Regression before understanding this article. The algorithms and images are taken from the paper. ( http://www.stat.washington.edu/courses/stat527/s13/readings/FriedmanStuetzle_JASA_1981.pdf )  What is Regression? Regression is a machine learning technology used to predict a response variable given multiple predictor variables or features. The main distinction is that the response to be predicted is any real value and not just any class or cluster name. Hence though similar to Classification in terms of making a prediction, it is largely different given what it’s predicting.  A simple to understand real world problem of regression would be predicting the sale price of a particular house based on it’s square footage, given that we have data of similar houses sold in that area in the past. T

Understanding Generative Adverserial Networks - Part 1

This is a two part series on understanding Generative Adversarial Networks (GANs). This part deals with the conceptual understanding of GANs. In the second part we will try to understand the mathematics behind GANs. Generative networks have been in use for quite a while now. And so have discriminative networks. But only in 2014 did someone get the brilliant idea of using them together. These are the generative adversarial networks. This kind of deep learning model was invented by Ian Goodfellow . When we work with data already labelled, it’s called supervised learning. It’s much easier compared to unsupervised learning, which has no predefined labels, making the task more vague.  "Generative Adversarial Networks is the most interesting idea in the last ten years in Machine Learning." - Yann LeCun In this post, we’ll discuss what GANs are and how they work, at a higher , more abstract level. Since 2014, many variations of the traditional GAN have co