why does scikitlearn says F1 score is ill-defined with FN bigger than 0?

python machine-learning statistics scikit-learn

42,430

Solution 1

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py

F1 = 2 * (precision * recall) / (precision + recall)

precision = TP/(TP+FP) as you've just said if predictor doesn't predicts positive class at all - precision is 0.

recall = TP/(TP+FN), in case if predictor doesn't predict positive class - TP is 0 - recall is 0.

So now you are dividing 0/0.

Solution 2

Precision, Recall, F1-score and Accuracy calculation

- In a given image of Dogs and Cats

  * Total Dogs - 12  D = 12
  * Total Cats - 8   C = 8

- Computer program predicts

  * Dogs - 8  
    5 are actually Dogs   T.P = 5
    3 are not             F.P = 3    
  * Cats - 12
    6 are actually Cats   T.N = 6 
    6 are not             F.N = 6

- Calculation

  * Precision = T.P / (T.P + F.P) => 5 / (5 + 3)
  * Recall    = T.P / D           => 5 / 12

  * F1 = 2 * (Precision * Recall) / (Precision + Recall)
  * F1 = 0.5

  * Accuracy = T.P + T.N / P + N
  * Accuracy = 0.55

Wikipedia reference

42,430

Tim

Elitists are oppressive, anti-intellectual, ultra-conservative, and cancerous to the society, environment, and humanity. Please help make Stack Exchange a better place. Expose elite supremacy, elitist brutality, and moderation injustice to https://stackoverflow.com/contact (complicit community managers), in comments, to meta, outside Stack Exchange, and by legal actions. Push back and don't let them normalize their behaviors. Changes always happen from the bottom up. Thank you very much! Just a curious self learner. Almost always upvote replies. Thanks for enlightenment! Meanwhile, Corruption and abuses have been rampantly coming from elitists. Supportive comments have been removed and attacks are kept to control the direction of discourse. Outright vicious comments have been removed only to conceal atrocities. Systematic discrimination has been made into policies. Countless users have been harassed, persecuted, and suffocated. Q&A sites are for everyone to learn and grow, not for elitists to indulge abusive oppression, and cover up for each other. https://softwareengineering.stackexchange.com/posts/419086/revisions https://math.meta.stackexchange.com/q/32539/ (https://i.stack.imgur.com/4knYh.png) and https://math.meta.stackexchange.com/q/32548/ (https://i.stack.imgur.com/9gaZ2.png) https://meta.stackexchange.com/posts/353417/timeline (The moderators defended continuous harassment comments showing no reading and understanding of my post) https://cs.stackexchange.com/posts/125651/timeline (a PLT academic had trouble with the books I am reading and disparaged my self learning posts, and a moderator with long abusive history added more insults.) https://stackoverflow.com/posts/61679659/revisions (homework libels) Much more that have happened.

Updated on July 09, 2022

Comments

Tim almost 2 years
I run a python program that calls sklearn.metrics's methods to calculate precision and F1 score. Here is the output when there is no predicted sample:
```
/xxx/py2-scikit-learn/0.15.2-comp6/lib/python2.6/site-packages/sklearn/metr\
ics/metrics.py:1771: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples.
  'precision', 'predicted', average, warn_for)

/xxx/py2-scikit-learn/0.15.2-comp6/lib/python2.6/site-packages/sklearn/metr\
ics/metrics.py:1771: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.
  'precision', 'predicted', average, warn_for)
```
When there is no predicted sample, it means that TP+FP is 0, so
- precision (defined as TP/(TP+FP)) is 0/0, not defined,
- F1 score (defined as 2TP/(2TP+FP+FN)) is 0 if FN is not zero.
In my case, sklearn.metrics also returns the accuracy as 0.8, and recall as 0. So FN is not zero.

But why does scikilearn says F1 is ill-defined?

What is the definition of F1 used by Scikilearn?
- Zaccharie Ramzi almost 5 years
  
  please mark the answer as accepted.