How to write to a file using non blocking IO?

Question

I want to write to a file using a non-blocking method in Python. On some googling, I found that the language supports fcntl in order to do so, but the method to implement the same is not very clear to me.

This is the code snippet (I don't know where I am going wrong):

import os, fcntl
nf = fcntl.fcntl(0,fcntl.F_UNCLK)
fcntl.fcntl(0,fcntl.F_SETFL , nf | os.O_NONBLOCK )
nf = open ("test.txt", 'a') 
nf.write ( " sample text \n")

Is this the correct way to perform a non-blocking IO operation on a file? I doubt it. Also, could you suggest any other modules in Python which allow me to do so?

Solution 1

This is how you turn non-blocking mode on for a file in UNIX:

fd = os.open("filename", os.O_CREAT | os.O_WRONLY | os.O_NONBLOCK)
os.write(fd, "data")
os.close(fd)

On UNIX, however, turning on non-blocking mode has no visible effect for regular files! Even though the file is in non-blocking mode, the os.write call won't return immediately, it will sleep until the write is complete. To prove it to yourself experimentally, try this:

import os
import datetime

data = "\n".join("testing\n" * 10 for x in xrange(10000000))
print("Size of data is %d bytes" % len(data))

print("open at %s" % str(datetime.datetime.now()))
fd = os.open("filename", os.O_CREAT | os.O_WRONLY | os.O_NONBLOCK)
print("write at %s" % str(datetime.datetime.now()))
os.write(fd, data)
print("close at %s" % str(datetime.datetime.now()))
os.close(fd)
print("end at %s" % str(datetime.datetime.now()))

You'll notice that the os.write call does take several seconds. Even though the call is non-blocking (technically, it's not blocking, it's sleeping), the call is not asynchronous.

---

AFAIK, there is no way to write to a file asynchronously on Linux or on Windows. You can simulate it, however, using threads. Twisted has a method named deferToThread for this purpose. Here's how you use it:

from twisted.internet import threads, reactor

data = "\n".join("testing\n" * 10 for x in xrange(10000000))
print("Size of data is %d bytes" % len(data))

def blocking_write():
    print("Starting blocking_write")
    f = open("testing", "w")
    f.write(data)
    f.close()
    print("End of blocking_write")

def test_callback():
    print("Running test_callback, just for kicks")

d = threads.deferToThread(blocking_code)
reactor.callWhenRunning(cc)
reactor.run()

Solution 2

Writes are cached by the OS and dumped to disk after a few seconds. That is, they are "not blocking" already. You don't have to do anything special.

Comments

[jcollado]

possible duplicate of Asynchronous file writing possible in python?

[Rahul]

No its not , i need to keep it simple by just using fcntl :)

[Flimm]

What if the file is actually mounted on a network share? Surely, the call would only return after the acknowledgements have been received?

[jcea]

Depends of the remote file system and semantic implemented, synchronous or asynchronous. There are examples of both, or even things like "sync on close".

[Sean]

How would I gain access to the reactor object from within a LineReciver or other such protocol built by a server factory?

[strcat]

A non-blocking write can be done to a regular file with POSIX AIO or Windows IOCP.