How to avoid decoding to str: need a bytes-like object error in pandas?
28,025
Your data has NaNs
(not a number).
You can either drop them first:
documents = documents.dropna(subset=['content'])
Or, you can fill all NaNs
with an empty string, convert the column to string type and then map your string based function.
documents['content'].fillna('').astype(str).map(preprocess)
This is because your function preprocess has function calls that accept string only data type.
Edit:
How do I know that your data contains NaNs? Numpy nan are considered float values
>>> import numpy as np
>>> type(np.nan)
<class 'float'>
Hence, you get the error
TypeError: decoding to str: need a bytes-like object, float found
Author by
wayne64001
Updated on January 07, 2022Comments
-
wayne64001 over 2 years
Here is my code :
data = pd.read_csv('asscsv2.csv', encoding = "ISO-8859-1", error_bad_lines=False); data_text = data[['content']] data_text['index'] = data_text.index documents = data_text
It looks like
print(documents[:2]) content index 0 Pretty extensive background in Egyptology and ... 0 1 Have you guys checked the back end of the Sphi... 1
And I define a preprocess function by using gensim
stemmer = PorterStemmer() def lemmatize_stemming(text): return stemmer.stem(WordNetLemmatizer().lemmatize(text, pos='v')) def preprocess(text): result = [] for token in gensim.utils.simple_preprocess(text): if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3: result.append(lemmatize_stemming(token)) return result
And when I use this function:
processed_docs = documents['content'].map(preprocess)
It appears
TypeError: decoding to str: need a bytes-like object, float found
How to encode my csv file to byte-like object or how to avoid this kind of error?