TypeError: a bytes-like object is required, not 'str' when writing to a file in Python3

Question

I've very recently migrated to Py 3.5. This code was working properly in Python 2.7:

with open(fname, 'rb') as f:
    lines = [x.strip() for x in f.readlines()]

for line in lines:
    tmp = line.strip().lower()
    if 'some-pattern' in tmp: continue
    # ... code

After upgrading to 3.5, I'm getting the:

TypeError: a bytes-like object is required, not 'str'

error on the last line (the pattern search code).

I've tried using the .decode() function on either side of the statement, also tried:

if tmp.find('some-pattern') != -1: continue

- to no avail.

I was able to resolve almost all 2:3 issues quickly, but this little statement is bugging me.

Solution 1

You opened the file in binary mode:

with open(fname, 'rb') as f:

This means that all data read from the file is returned as bytes objects, not str. You cannot then use a string in a containment test:

if 'some-pattern' in tmp: continue

You'd have to use a bytes object to test against tmp instead:

if b'some-pattern' in tmp: continue

or open the file as a textfile instead by replacing the 'rb' mode with 'r'.

Solution 2

You can encode your string by using .encode()

Example:

'Hello World'.encode()

As the error describes, in order to write a string to a file you need to encode it to a byte-like object first, and encode() is encoding it to a byte-string.

Solution 3

Like it has been already mentioned, you are reading the file in binary mode and then creating a list of bytes. In your following for loop you are comparing string to bytes and that is where the code is failing.

Decoding the bytes while adding to the list should work. The changed code should look as follows:

with open(fname, 'rb') as f:
    lines = [x.decode('utf8').strip() for x in f.readlines()]

The bytes type was introduced in Python 3 and that is why your code worked in Python 2. In Python 2 there was no data type for bytes:

>>> s=bytes('hello')
>>> type(s)
<type 'str'>

Solution 4

You have to change from wb to w:

def __init__(self):
    self.myCsv = csv.writer(open('Item.csv', 'wb')) 
    self.myCsv.writerow(['title', 'link'])

to

def __init__(self):
    self.myCsv = csv.writer(open('Item.csv', 'w'))
    self.myCsv.writerow(['title', 'link'])

After changing this, the error disappears, but you can't write to the file (in my case). So after all, I don't have an answer?

Source: How to remove ^M

Changing to 'rb' brings me the other error: io.UnsupportedOperation: write

Solution 5

For this small example, adding the only b before 'GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\n\n' solved my problem:

import socket

mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send(b'GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\n\n')

while True:
    data = mysock.recv(512)
    if (len(data) < 1):
        break
    print (data);

mysock.close()

What does the 'b' character do in front of a string literal?

View more solutions

Comments

[wescpy]

If you peek at the various documents that ppl have linked to, you'll see that everything "worked" in Py2 because default strings were bytes whereas in Py3, default strings are Unicode, meaning that any time you're doing I/O, esp. networking, byte strings are the standard, so you must learn to move b/w Unicode & bytes strings (en/decode). For files, we now have "r" vs. "rb" (and for 'w' & 'a') to help differentiate.

[Martijn Pieters]

@wescpy: Python 2 has 'r' vs 'rb' too, switching between binary and text file behaviours (like translating newlines and on certain platforms, how the EOF marker is treated). That the io library (providing the default I/O functionality in Python 3 but also available in Python 2) now also decodes text files by default is the real change.

[wescpy]

@MartijnPieters: Yes, agreed. In 2.x, I only used the 'b' flag when having to work with binary files on DOS/Windows (as binary is the POSIX default). It's good that there is a dual purpose when using io in 3.x for file access.

[ericOnline]

r does not work with zipfile 's .open(). Example: def get_aoi1(zip): z = zipfile.ZipFile(zip) for f in z.namelist(): with z.open(f, 'r') as rptf: for l in rptf.readlines(): if l.find("$$") != -1: return l.split('=') else: return print(l) test = get_aoi1('testZip.zip')

[Martijn Pieters]

@ericOnline ZipFile.open() docs explicitly state that only binary mode is supported (Access a member of the archive as a binary file-like object). You can wrap the file object in io.TextIOWrapper() to achieve the same effect.

[Martijn Pieters]

@ericOnline also, don’t use .readlines() when you can iterate over the file object directly. Especially when you only need info from a single line. Why read everything into memory when that info could be found in the first buffered block?

[Mark Ransom]

Python 2 does indeed have a type for bytes, it's just confusingly called str while the type for text strings is called unicode. In Python 3 they changed the meaning of str so that it was the same as the old unicode type, and renamed the old str to bytes. They also removed a bunch of cases where it would automatically try to convert from one to the other.

[Mark Ransom]

Note that the coding comment at the top of the file doesn't affect the way bytes or encode works, it only changes the way characters in your Python source are interpreted.

[jma]

This comment was quite useful in the context of using fd.subprocess.Popen(); fd.communicate(...);.

[Skippy le Grand Gourou]

If concatenation to a string is needed afterwards (TypeError: can only concatenate str (not "bytes") to str) : "Hello "+("World".encode()).decode() (same with join() obviously).

[Peter Mortensen]

Why does it work? The OP has left the building ("Last seen more than 5 years ago"), so perhaps somebody else can chime in?

[Peter Mortensen]

Why does that work?

[Peter Mortensen]

Why does that work? An explanation would be in order. (But without "Edit:", "Update:", or similar - the answer should appear as if it was written today.)

[Peter Mortensen]

Why does that work?

[Theofilos Papapanagiotou]

You cannot write a string to a file, you need to encode the string to a byte-like object to do so. By running the encode() method of a string, we get the encoded version of it in the default encoding, which is usually utf-8.