Fastest stdin/out IO in python 3?

Question

I've been solving a few problems on SPOJ.pl using python 3.1.2 and some peoples fast result on simple problems makes me wonder if there is a faster way to handle input and output.

I've tried using

input()
print()

and

sys.stdin.readline()
sys.stdout.write()

or rather

for line in sys.stdin:
    #Handle input
    sys.stdout.write(output)

to process each line. I've also tried to collect all output in lists and print all at once when everything is processed.

But all of these produce similar execution times.

Is there any faster way to handle input and output from stdin/out?

Solution 1

The following will probably be fastest:

1. Read all the input at once using os.read(0, some_big_enough_number).

2. Process the output, collecting the results in a list results.

3. Write all the output at once using os.write(1, "".join(results)).

I remember one case where I noticed that os.read() and os.write() are sometimes faster than using Python I/O, but I don't remember the details.

Solution 2

SPOJ lets you choose among a variety of programming languages. Are you comparing your execution time to other solutions written in other programming languages?

Just for fun, I submitted the following solutions to the first problem (codename TEST) to compare run-times.

C++ solution (G++ 4.3.2)

#include <iostream>
int main ( int, char** )
{
     for ( int number=0; (std::cin >> number) && (number != 42); ) {
         std::cout << number << std::endl;
     }
}

See the submission.

Python (2.5) solution

import sys
for line in sys.stdin:
    number = int(line)
    if number == 42:
        break
    print number

See the submission.

Conclusion

I'm not 100% sure that this gets the absolute best performance in either languages, but there's not so much code in there to optimize.

I get time 0.00 measurement for the C++ and 0.04 measurement for the Python code. Assuming the sequence of numbers submitted to both programs is the same, I think comparison of run-times against solutions in other languages is almost meaningless (see next paragraph).

Now, this is only true for simple problems. Most advanced problems require choosing the right algorithm for the problem and choosing the wrong one has drastic consequences. In those cases, carefully crafted Python solutions might still be slower than carefully crafted C++ solutions but the good Python solution will beat a naïve solution written in any other language.

Solution 3

Probably not.

In the end, print will call sys.stdout.write(). But since print is a built in function, probably implemented in C, it might even be faster than calling sys.stdout.write().

Since all IO has to go through the object which sys.stdout returns, that's the bottleneck. The same is true for sys.stdin.

There are no magic tricks to make this faster.

If you need faster IO, try these things:

• Write to a file instead
• Use buffered IO (pipe stdout using the buffer command in a small shell script).
• Use a memory mapped file
• Make sure the process which reads your output can keep up. For example the DOS console is pretty slow. If you pipe the output through a slow command, that can block your python process.

[EDIT] Seems like SPOJ.pl is some kind of programmer shootout site. In this case, I/O speed is not the culprit: You have used a bad algorithm to solve the problem.

The speed difference between a good and a fair performance can easily be between 10 and 100'000 times. By changing a few lines of code, I could once make code run in less than 5 seconds that took 45 minutes before.

Comments

[Xavier Combelle]

I think that if people have fastest results than you the cause is not in your I/O stuff

[while]

It was my initial guess but it is also the case for problems that only need to redirect some of the input to output directly. This is why I asked the question.

[Ned Deily]

Python 3.1.2 is not a particularly good version of Python to be using for performance measurements. Python 3.2.2 is current for Python 3 and there have been many significant performance improvements made between those releases.

[while]

Thanks, although I can't do any of this on SPOJ.pl since I submit my code and it then runs against test cases on their servers. What confuses me is that people get a lot faster execution times than me even if the problem can be solved by just printing specific things from the input.

[André Caron]

@while: Try writing your solution in C and see if you can match the response times. I don't know how SPOJ.pl computes run times, but you might be paying for the Python interpreter to start up and byte compile your Python script...

[while]

It is always faster in C but I'm comparing with the other solutions written in python.

[while]

Thanks, You're absolutely right but I wasn't comparing different languages. Check the solution times in python 3.1.2 for this problem: spoj.pl/ranks/ACT/lang=PYTH%203.1.2 All I do is redirecting input to output but I still get 0.14 and the fastest is 0.02. That's why I thought there might be a faster way of handling io which made me a bit curious.

[while]

It doesn't really work since I don't know the length of the input though.

[Sven Marnach]

@while: Just use a number that is sufficient, like 1000000000. It just has to be enough to read all the input.

[Xavier Combelle]

That is because the output of real program is very short in fact and the input may be very long

[while]

Seems to be slightly faster when testing on my machine (python 3.2.2). Can't get it to work on SPOJ (python 3.1.2) though for some reason but thanks anyway.

[André Caron]

@while: it's not uncommon for these hosts to limit the amount of memory available to your process. Try reading in blocks of some reasonable size. You'll end up having less I/O anyways.

[Kaerber]

Well, the point of such problems is they are NOT solved by just printing specific things from the input, that is why people get a lot faster execution times.