Python: UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

Question

I'm currently have an issue with my python 3 code.

 replace_line('Products.txt', line, tenminus_str)

Is the line I'm trying to turn into utf-8, however when I try to do this like I would with others, I get errors such as no attribute ones and when I try to add, for example...

.decode("utf8")

...to the end of it, I still get errors that it is using ascii. I also tried other methods that worked with other lines such as adding io. infront and adding a comma with

encoding = 'utf8'

The function that I am using for replace_line is:

def replace_line(file_name, line_num, text):
    lines = open(file_name, 'r').readlines()
    lines[line_num] = text
    out = open(file_name, 'w')
    out.writelines(lines)
    out.close()

How would I fix this issue? Please note that I'm very new to Python and not advanced enough to do debugging well.

EDIT: Different fix to this question than 'duplicate'

EDIT 2:I have another error with the function now.

File "FILELOCATION", line 45, in refill replace_line('Products.txt', str(line), tenminus_str) 

File "FILELOCATION", line 6, in replace_line lines[line_num] = text

TypeError: list indices must be integers, not str 

What does this mean and how do I fix it?

Solution 1

codecs module is just what you need. detail here

import codecs
def replace_line(file_name, line_num, text):
    f = codecs.open(file_name, 'r', encoding='utf-8')
    lines = f.readlines()
    lines[line_num] = text
    f.close()
    w = codecs.open(file_name, 'w', encoding='utf-8')
    w.writelines(lines)
    w.close()

Solution 2

Change your function to:

def replace_line(file_name, line_num, text):
    with open(file_name, 'r', encoding='utf8') as f:
        lines = f.readlines()
    lines[line_num] = text
    with open(file_name, 'w', encoding='utf8') as out:
        out.writelines(lines)

encoding='utf8' will decode your UTF-8 file correctly.

with automatically closes the file when its block is exited.

Since your file started with \xef it likely has a UTF-8-encoding byte order mark (BOM) character at the beginning. The above code will maintain that on output, but if you don't want it use utf-8-sig for the input encoding. Then it will be automatically removed.

Solution 3

Handling coding problems You can try adding the following settings to your head

---

import sys
reload(sys)
sys.setdefaultencoding('utf-8')
Type = sys.getfilesystemencoding()

Solution 4

Try adding encoding='utf8' if you are reading a file

with open("../file_path", encoding='utf8'):
         # your code

View more solutions