Read .csv file from URL into Python 3.x - _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
Solution 1
The problem relies on urllib
returning bytes. As a proof, you can try to download the csv file with your browser and opening it as a regular file and the problem is gone.
A similar problem was addressed here.
It can be solved decoding bytes to strings with the appropriate encoding. For example:
import csv
import urllib.request
url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"
ftpstream = urllib.request.urlopen(url)
csvfile = csv.reader(ftpstream.read().decode('utf-8')) # with the appropriate encoding
data = [row for row in csvfile]
The last line could also be: data = list(csvfile)
which can be easier to read.
By the way, since the csv file is very big, it can slow and memory-consuming. Maybe it would be preferable to use a generator.
EDIT: Using codecs as proposed by Steven Rumbalski so it's not necessary to read the whole file to decode. Memory consumption reduced and speed increased.
import csv
import urllib.request
import codecs
url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"
ftpstream = urllib.request.urlopen(url)
csvfile = csv.reader(codecs.iterdecode(ftpstream, 'utf-8'))
for line in csvfile:
print(line) # do something with line
Note that the list is not created either for the same reason.
Solution 2
Even though there is already an accepted answer, I thought I'd add to the body of knowledge by showing how I achieved something similar using the requests
package (which is sometimes seen as an alternative to urlib.request
).
The basis of using codecs.itercode()
to solve the original problem is still the same as in the accepted answer.
import codecs
from contextlib import closing
import csv
import requests
url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"
with closing(requests.get(url, stream=True)) as r:
reader = csv.reader(codecs.iterdecode(r.iter_lines(), 'utf-8'))
for row in reader:
print row
Here we also see the use of streaming provided through the requests
package in order to avoid having to load the entire file over the network into memory first (which could take long if the file is large).
I thought it might be useful since it helped me, as I was using requests
rather than urllib.request
in Python 3.6.
Some of the ideas (e.g using closing()
) are picked from this similar post
Chris
I am a Biostatistician in the Department of Family and Community Medicine at the University of Toronto. I collaborate with family physicians and help them with the design and analysis of their primary care research projects.
Updated on November 18, 2020Comments
-
Chris over 3 years
I've been struggling with this simple problem for too long, so I thought I'd ask for help. I am trying to read a list of journal articles from National Library of Medicine ftp site into Python 3.3.2 (on Windows 7). The journal articles are in a .csv file.
I have tried the following code:
import csv import urllib.request url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv" ftpstream = urllib.request.urlopen(url) csvfile = csv.reader(ftpstream) data = [row for row in csvfile]
It results in the following error:
Traceback (most recent call last): File "<pyshell#4>", line 1, in <module> data = [row for row in csvfile] File "<pyshell#4>", line 1, in <listcomp> data = [row for row in csvfile] _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
I presume I should be working with strings not bytes? Any help with the simple problem, and an explanation as to what is going wrong would be greatly appreciated.
-
Steven Rumbalski over 10 yearsIncorrect.
StringIO
is a Python 2 module. Answer needs to be for Python 3. This is particularly important because of how Python 3 handles strings. -
HennyH over 10 years@StevenRumbalski I assume using docs.python.org/3.4/library/io.html#io.StringIO would be alright then?
-
Steven Rumbalski over 10 years+1. However, something feels wrong about having to read all the data before decoding it. Does Python 3 offer anything that allows this to be done as a generator?
-
Steven Rumbalski over 10 years
StringIO
does not accept bytes:TypeError: initial_value must be str or None, not bytes
. -
Steven Rumbalski over 10 yearsFigured it out. The Python 3 way to stream this is to use
codecs.iterdecode
. -
Diego Herranz over 10 yearsAdded a version of the snippet using codecs to make use of generators.
-
HennyH over 10 years@StevenRumbalski see my updated answer, which doesn't read in the whole file or use stringIO
-
white_gecko about 10 yearswith
responseHeader = response.info()
you can even get the response header from where you can get the correct encoding e.g. withencoding = responseHeader['Content-Type'].split(';')[1].split('=')[1]
which you can use for decoding the responseresponse.read().decode(encoding)
, so you don't have to hardcode the encoding and react to different encodings