How to convert a string to utf-8 in Python
Solution 1
In Python 2
>>> plain_string = "Hi!"
>>> unicode_string = u"Hi!"
>>> type(plain_string), type(unicode_string)
(<type 'str'>, <type 'unicode'>)
^ This is the difference between a byte string (plain_string) and a unicode string.
>>> s = "Hello!"
>>> u = unicode(s, "utf-8")
^ Converting to unicode and specifying the encoding.
In Python 3
All strings are unicode. The unicode
function does not exist anymore. See answer from @Noumenon
Solution 2
If the methods above don't work, you can also tell Python to ignore portions of a string that it can't convert to utf-8:
stringnamehere.decode('utf-8', 'ignore')
Solution 3
Might be a bit overkill, but when I work with ascii and unicode in same files, repeating decode can be a pain, this is what I use:
def make_unicode(inp):
if type(inp) != unicode:
inp = inp.decode('utf-8')
return inp
Solution 4
Adding the following line to the top of your .py file:
# -*- coding: utf-8 -*-
allows you to encode strings directly in your script, like this:
utfstr = "ボールト"
Solution 5
If I understand you correctly, you have a utf-8 encoded byte-string in your code.
Converting a byte-string to a unicode string is known as decoding (unicode -> byte-string is encoding).
You do that by using the unicode function or the decode method. Either:
unicodestr = unicode(bytestr, encoding)
unicodestr = unicode(bytestr, "utf-8")
Or:
unicodestr = bytestr.decode(encoding)
unicodestr = bytestr.decode("utf-8")
Related videos on Youtube
Bin Chen
Updated on September 21, 2021Comments
-
Bin Chen over 2 years
I have a browser which sends utf-8 characters to my Python server, but when I retrieve it from the query string, the encoding that Python returns is ASCII. How can I convert the plain string to utf-8?
NOTE: The string passed from the web is already UTF-8 encoded, I just want to make Python to treat it as UTF-8 not ASCII.
-
Mudassir over 13 yearsTry this link http://evanjones.ca/python-utf8.html
-
boatcoder over 7 yearsI think a better title would be How to coerce a string to unicode without translation?
-
devssh over 5 yearsIn 2018, python 3 if you get ascii decode error do
"some_string".encode('utf-8').decode('utf-8')
-
-
Gopakumar N G over 10 years,I am getting the following error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 2: invalid start byte
This is my code: ret=[] for line in csvReader: cline=[] for elm in line: unicodestr = unicode(elm, 'utf-8') cline.append(unicodestr) ret.append(cline) -
jfs almost 9 yearsIt is not what OP asks. But avoid such string literals anyway. It creates Unicode string in Python 3 (good) but it is a bytestring in Python 2 (bad). Either add
from __future__ import unicode_literals
at the top or useu''
prefix. Don't use non-ascii characters inbytes
literals. To get utf-8 bytes, you couldutf8bytes = unicode_text.encode('utf-8')
later if it is necessary. -
Noumenon over 8 yearsNone of this applies in Python 3, all strings are unicode and
unicode()
doesn't exist. -
智障的人 about 8 yearsKind of bumping this, but thanks. This fixed an issue where I was trying to print unicode and was getting �s.
-
Tanguy over 6 yearsHow to you convert
u
back to astr
format (convertu
back tos
)? -
Haroldo_OK about 6 yearsThis code will only work as long as the text does not contain non-ascii characters; a simple accented character on the string will make it fail.
-
saran3h over 5 yearsGot AttributeError: 'str' object has no attribute 'decode'
-
duhaime over 5 years@saran3h it sounds like you're using Python 3, in which case Python should handle encoding issues for you. Have you tried reading your document without specifying an encoding?
-
Ortal Turgeman over 5 years@jfs how will
from __future__ import unicode_literals
help me to convert a string with non-ascii characters to utf-8? -
jfs over 5 years@OrtalTurgeman I'm not answering the question. Look, it is a comment, not an answer. My comment addresses the issue with the code in the answer. It tries to create a bytestring with non-ascii characters on Python 2 (it is a SyntaxError on Python 3 — bytes literals forbid that).
-
Sha2b over 4 yearsHi, if you have
"2340"
in a string variable, and you want to print the unicode characterU+2340
(⍀), is there any way to do that? -
Vishesh Mangla almost 4 yearsPython by default picks system encoding. In windows 10 it's cp1252 which is different from utf-8. I wasted few hours on it while using codecs.open() in py 3.8
-
Gino Mempin almost 3 yearsWhat is
unidecode
? Is it this pypi.org/project/Unidecode? Please provide info if it's a 3rd-party package, and how to install/use it. -
U12-Forward over 2 years@Sha2b
chr(0x2340)
gives:⍀
-
Mike Pennington over 2 yearsThis no longer works, as written... the
unicode
type doesn't exist in python3