"utf8 "\x96" does not map to Unicode at <somefile.pl> at line no - 321" Error in Perl

Question

I am setting up Perl application . I am getting this error. "utf8 "\x96" does not map to Unicode at" Can anybody let me know the cause and solution. Am i missing any configuration or it is my installation problem ?

Following is the code :

open(FILE,"<:encoding(UTF-8)",$self->{BASEDIR}.$self->{FILENAME}) || die "could not open file $basedir$filename - $!";

Solution 1

The character 0x96 is not a valid UTF-8 encoding. There is a block of code points just above 0x80 that, in UTF-8, encodes the start of a 2- or 3-byte character.

The input you are reading must not be UTF-8, and is most likely Latin1 or CP1252.

You will need to convert the input data to UTF-8, however one does that in Perl (it's been a long time since I did any Perl and it didn't use UTF-8 by default when I was writing Perl :-)

Solution 2

I suspect that something you believe to be encoded in UTF-8 is not, in fact, encoded with UTF-8.

Solution 3

Just putting this info out there in case it helps someone in the future.

If you're working with a Microsoft product, this can be caused by non-US characters (European, Chinese, etc). For instance, if someone sends you an excel spreadsheet of data that you need to process and it's saved in .csv format, those characters can be outside of the utf-8 range if it wasn't saved properly.

Fortunately, at least in Excel for Mac v. 15, it is possible to take that data and "save as" specifically a CSV UTF-8 file - it's in the list of options. This is a separate option from the other CSV file option. This will convert non-US characters into the UTF-8 charset and solve this issue.

Comments

[Admin]

What should I do now? Because i have set up application on another PC then it works but on another one its not working.

[Jim Garrison]

It would help if you edited your post to include more information, such as the code around where you get the error and a sample of the data that causes the problem. Otherwise we're just guessing.

[Admin]

Its not fatal error. Page is getting displayed properly but this message is dipslyed at the top.

[Jim Garrison]

Is the input file UTF-8? The message indicates that it is not.

[MacMartin]

in vim I checked :set fileencoding end :set fileencodings to see, that the file in this case was not utf8 but latin1! after changing this with the previous commands and saving - it was read correctly