Using io.BufferedReader on a stream obtained with open()?
Solution 1
By the looks of your print
statement, you're using Python 2. On that version, a file
is not a valid argument to the BufferedReader
constructor:
Under Python 2.x, this is proposed as an alternative to the built-in
file
object, but in Python 3.x it is the default interface to access files and streams. (1)
You should use io.open
instead:
>>> f = io.open(".bashrc", "rb")
If you do this, there's no need to explicitly wrap it in a BufferedReader
since that's exactly what io.open
returns by default:
>>> type(f)
<type '_io.BufferedReader'>
See its docs for details; there's a buffering
argument that controls the buffering.
In Python 3, open
is io.open
so the two I/O libraries have been merged back into one. It seems that io
was added to Python 2.6 mostly for forward compatibility.
Solution 2
You can set the amount of buffering in bytes by passing the buffering
argument to open:
import sys
srcfile = sys.argv[1]
with open(srcfile, 'rb', buffering=30) as f:
print(f.peek(30))
print(f.read(20))
This is a BufferedReader
:
>>> with open("test.txt", 'rb', buffering=30) as f:
... type(f)
<class '_io.BufferedReader'>
Note that, by default, it's buffered to 1
- line buffered.
Solution 3
In Python2, if you have to use file
object as returned by open
(or e.g. provided by some module routines which you cannot modify), you can use file descriptor obtained by fileno()
for io.FileIO
constructor, then pass io.FileIO
object to io.BufferedReader
constructor.
So, you sample code can be rewritten as follows:
import sys
import io
srcfile = sys.argv[1]
with open(srcfile, 'rb') as f:
fio = io.FileIO(f.fileno())
fbuf = io.BufferedReader(fio)
print fbuf.read(20)
Jason S
Updated on July 09, 2022Comments
-
Jason S almost 2 years
I want to use a buffered stream because I want to use a
peek()
method to peek ahead but use my stream with another method that expects a file-like object. (I'd useseek()
but may have to handle piped-in I/O that doesn't support random access.)But this test case fails:
AttributeError: 'file' object has no attribute '_checkReadable'
import sys import io srcfile = sys.argv[1] with open(srcfile, 'rb') as f: fbuf = io.BufferedReader(f) print fbuf.read(20)
What's going on and how do I fix it? I thought BufferedReader was intended to buffer a stream. If so, why does the
open()
function not return something that's compatible with it? -
Jason S about 12 yearssure, that's fine, but eventually I'm going want to buffer around other sources of input that might not be buffered... or is everything just buffered by default? sys.stdin? network streams?
-
Gareth Latty about 12 yearsWell, then you can do what you did and do
BufferedReader(f)
- as it's already aBufferedReader
it will work. -
Fred Foo about 12 yearsThis is the Python 3 solution; in Python 2 the OP should use
io.open
. -
Gareth Latty about 12 years@larsmans Ah, I didn't know this was a 3.x feature, you are correct.
-
Fred Foo about 12 yearsCorrection: this is actually a pretty valid solution also in Python 2, but
open
will return afile
, not aBufferedReader
in 2.x, so there's nopeek
method. -
Jason S about 12 yearsthere's no need to wrap for opening a file, but what if I use sys.stdin instead?
-
Fred Foo about 12 years@JasonS: then the hack that
username
pointed to is valid. Orio.open("/dev/stdin")
if your platform has that file (but in either case stay clear ofsys.stdin
).