mean() of column in pandas DataFrame returning inf: how can I solve this?
Solution 1
not so familiar with pandas but if you convert to a numpy array it works, try
np.asarray(valores.iloc[:,5], dtype=np.float).mean()
Solution 2
NaN
values should not matter when computing the mean of a pandas.Series
. Precision is also irrelevant. The only explanation I can think of is that one of the values in valores
is equal to infinity.
You could exclude any values that are infinite when computing the mean like this:
import numpy as np
is_inf = valores.iloc[:, 5] == np.inf
valores.ix[~is_inf, 5].mean()
Solution 3
If the elements of the pandas series are strings you get inf
and the mean result. In this specific case you can simply convert the pandas series elements to float
and then calculate the mean. No need to use numpy.
Example:
valores.iloc[:,5].astype(float).mean()
Solution 4
I had the same problem with a column that was of dtype 'o', and whose max value was 9999. Have you tried using the convert_objects
method with the convert_numeric=True
parameter? This fixed the problem for me.
Augusto Ribas
Biologist, interested in R language, Latex, Ecology and Evolution
Updated on June 04, 2022Comments
-
Augusto Ribas almost 2 years
I'm trying to implement some machine learning algorithms, but I'm having some difficulties putting the data together.
In the example below, I load a example data-set from UCI, remove lines with missing data (thanks to the help from a previous question), and now I would like to try to normalize the data.
For many datasets, I just used:
valores = (valores - valores.mean()) / (valores.std())
But for this particular dataset the approach above doesn't work. The problem is that the mean function is returning
inf
, perhaps due to a precision issue. See the example below:bcw = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data', header=None) for col in bcw.columns: if bcw[col].dtype != 'int64': print "Removendo possivel '?' na coluna %s..." % col bcw = bcw[bcw[col] != '?'] valores = bcw.iloc[:,1:10] #mean return inf print valores.iloc[:,5].mean()
My question is how to deal with this. It seems that I need to change the type of this column, but I don't know how to do it.