python charmap codec can't decode byte X in position Y character maps to <undefined>

Question

I'm experimenting with python libraries for data analysis,the problem i'm facing is this exception

UnicodeDecodeError was unhandled by user code Message: 'charmap' codec can't decode byte 0x81 in position 165: character maps to < undefined>

I have looked into answers with similar issues and the OP seems to be either reading text with different encoding or printing it.

In my code the error shows up at import statement,that's what confuses me.

I'm using python 64 bit 3.3 on Visual Studio 2015 and geotext is the library where it shows the error.

Kindly point as to where to look to deal with this error.

Solution

Here is how I solved this issue (its applicable for geotext 0.3.0)

Check the traceback:

Traceback (most recent call last): File "pythonTwitterTest.py", line 5, in from process.processData import * File "C:\OwaisWorkx\Courses\5th Semester\Project\pythonTwitterTest\pythonTwitterTest\process\processData.py", line 1, in from geotext import GeoText # for classifying and seperating City , Country and States/Provinces File "c:\Python33\lib\site-packages\geotext__init__.py", line 7, in from .geotext import GeoText File "c:\Python33\lib\site-packages\geotext\geotext.py", line 87, in class GeoText(object): File "c:\Python33\lib\site-packages\geotext\geotext.py", line 103, in GeoText index = build_index() File "c:\Python33\lib\site-packages\geotext\geotext.py", line 77, in build_index cities = read_table(get_data_path('cities15000.txt'), usecols=[1, 8]) File "c:\Python33\lib\site-packages\geotext\geotext.py", line 54, in read_table for line in lines: File "c:\Python33\lib\site-packages\geotext\geotext.py", line 51, in lines = (line for line in f if not line.startswith(comment)) File "c:\Python33\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 165: character maps to

This shows the error is actually in the geotext.py file

so open that up

geotext.py and goto line 45: Change from this

with open(filename, 'r') as f

to this

with open(filename, 'r', encoding='utf-8') as f:

P.S: Solution taken from Python-forum.io