Python - UnicodeEncodeError: 'charmap' codec can't encode characters in position 85-89: character maps to <undefined>

Question

I am trying to see if I can transfer the output of urllib.request.urlopen() to a text file just to look at it. I tried decoding the output into a string so I can write into a file, but apparently the original output included some Korean characters that are not translating properly into the string.

So far I have:

from urllib.request import urlopen

openU = urlopen(myUrl)
pageH = openU.read()
openU.close()
stringU = pageH.decode("utf-8")

f=open("test.txt", "w+")
f.write(stringU)

I do not get any errors until the last step at which point it says:

Traceback (most recent call last):  
  File "<stdin>", line 1, in <module>  
  File "C:\Users\Chae\AppData\Local\Programs\Python\Python36\lib\encodings\cp1252.py", line 19, in encode  
  return codecs.charmap_encode(input,self.errors,encoding_table)[0] 
UnicodeEncodeError: 'charmap' codec can't encode characters in position 85-89: character maps to `<undefined>`

Is there a way to get the string to also include Korean or if not, how do I skip the characters causing problems and write the rest of the string into the file?

Solution

Does it matter to you what the file encoding is? If not, then use utf-8 encoding:

f=open("test.txt", "w+", encoding="utf-8")
f.write(stringU)

If you want the file to be cp1252-encoded, which apparently is the default on your system, and to ignore unencodable values, add errors="ignore":

f=open("test.txt", "w+", errors="ignore")
f.write(stringU)

Comments

[Zac1]

Where do you come from genius?! This , encoding="utf-8" helped, after searching for hours reg. this issue!

[greendino]

still having the same exact issue after encode("utf-8")