Reading one integer at a time using python

Question

How can I read int from a file? I have a large(512MB) txt file, which contains integer data as:

0 0 0 10 5 0 0 140
0 20 6 0 9 5 0 0

Now if I use c = file.read(1), I get only one character at a time, but I need one integer at a time. Like:

c = 0
c = 10
c = 5
c = 140 and so on...

Any great heart please help. Thanks in advance.

Solution 1

Here's one way:

with open('in.txt', 'r') as f:
  for line in f:
    for s in line.split(' '):
      num = int(s)
      print num

By doing for line in f you are reading bit by bit (using neither read() all nor readlines). Important because your file is large.

Then you split each line on spaces, and read each number as you go.

You can do more error checking than that simple example, which will barf if the file contains corrupted data.

As the comments say, this should be enough for you - otherwise if it is possible your file can have extremely long lines you can do something trickier like reading blocks at a time.

Solution 2

512 MB is really not that large. If you're going to create a list of the data anyway, I don't see a problem with doing the reading step in one go:

my_int_list = [int(v) for v in open('myfile.txt').read().split()]

if you can structure your code so you don't need the entire list in memory, it would be better to use a generator:

def my_ints(fname):
    for line in open(fname):
        for val in line.split():
            yield int(val)

and then use it:

for c in my_ints('myfile.txt'):
    # do something with c (which is the next int)