What's the difference between io.open() and os.open() on Python?

Question

I realised that the open() function I've been using was an alias to io.open() and that importing * from os would overshadow that.

What's the difference between opening files through the io module and os module?

Solution 1

io.open() is the preferred, higher-level interface to file I/O. It wraps the OS-level file descriptor in an object that you can use to access the file in a Pythonic manner.

os.open() is just a wrapper for the lower-level POSIX syscall. It takes less symbolic (and more POSIX-y) arguments, and returns the file descriptor (a number) that represents the opened file. It does not return a file object; the returned value will not have read() or write() methods.

From the os.open() documentation:

This function is intended for low-level I/O. For normal usage, use the built-in function open(), which returns a “file object” with read() and write() methods (and many more).

Solution 2

Absolutely everything:

• os.open() takes a filename as a string, the file mode as a bitwise mask of attributes, and an optional argument that describes the file permission bits, and returns a file descriptor as an integer.

• io.open() takes a filename as a string or a file descriptor as an integer, the file mode as a string, and optional arguments that describe the file encoding, buffering used, how encoding errors and newlines are handled, and if the underlying FD is closed when the file is closed, and returns some descendant of io.IOBase.

Solution 3

os.open is very similar to open() from C in Unix. You're unlikely to want to use it unless you're doing something much more low-level. It gives you an actual file descriptor (as in, a number, not an object).

io.open is your basic Python open() and what you want to use just about all the time.

Solution 4

In Python 2, the built-in open and io.open were different (io.open was newer and supported more things). In Python 3, open and io.open are now the same thing (they got rid of the old built-in open), so you should always use open. Code that needs to be compatible with Python 2 and 3 might have a reason to use io.open.

Below code to validate this.

import io
with io.open("../input/files.txt") as f:
    text = f.read().lower()

with open('../input/files.txt', encoding='utf-8') as f2:
    text2 = f2.read().lower()

print(type(f))
print(type(f2))
# <class '_io.TextIOWrapper'>
# <class '_io.TextIOWrapper'>

Solution 5

To add to the existing answers:

I realised that the open() function I've been using was an alias to io.open()

open() == io.open() in Python 3 only. In Python 2 they are different.

While with open() in Python we can obtain an easy-to-use file object with handy read() and write() methods, on the OS level files are accessed using file descriptors (or file handles in Windows). Thus, os.open() should be used implicitly under the hood. I haven't examined Python source code in this regard, but the documentation for the opener parameter, which was added for open() in Python 3.3, says:

A custom opener can be used by passing a callable as opener. The underlying file descriptor for the file object is then obtained by calling opener with (file, flags). opener must return an open file descriptor (passing os.open as opener results in functionality similar to passing None).

So os.open() is the default opener for open(), and we also have the ability to specify a custom wrapper around it if file flags or mode need to be changed. See the documentation for open() for an example of a custom opener, which opens a file relative to a given directory.

View more solutions

Comments

[Gio Borje]

Does that mean if I take some of my simple C file IO code, prepend os. to the stdio functions, and change the extension to .py, the code will execute without errors?

[Owen]

I highly highly doubt it, but I'd love to see how it goes (just don't break anything).

[Karl Knechtel]

Although I have to wonder, why would anyone choose Python for low-level I/O?

[cdhowie]

Maybe Python is the language in which the coder is the most comfortable. Or maybe they are writing an addon for another Python program.

[0 _]

Why not? Low-level manipulation doesn't dictate in what language it should we expressed.

[cdhowie]

@IoannisFilippidis Because if you are going to use low-level interfaces, then why are you even using Python to begin with? Unless what you're doing can't be accomplished using higher-level interfaces, or the higher-level interfaces have severe performance issues, you should use the higher-level interfaces in the language you are using -- otherwise you should just use a different language altogether.

[0 _]

Python is more readable and flexible than writing in C, and there can be computations that benefit from accessing scientific packages written in Python.

[cdhowie]

@IoannisFilippidis That's my point. If you want the flexibility of Python, then you probably want to be using its high-level I/O interfaces anyway -- not doing so results in less portable, less readable, and less maintainable code.

[0 _]

My point is that you do want to write low-level code in Python, among other, high-level code, to which Python is more suited.

[0 _]

In any case, I would likely write it in Cython, so C, because flat is better than nested (PEP 20), and I've done so in the past, so the point here was more for the sake of discussion.

[AXO]

For an example of why you might want to use os.open see: stackoverflow.com/a/45368120/2705757 (setting file permission upon creation, without using os.chmod)

[Hejazzman]

Using high or low-level interfaces is almost totally orthogonal to the language you're using. There's no reason low level interfaces must be used from C/C++ or so. High/low level interfaces are about level of detail you want to talk to system at , not about performance or static typing or anything of the short. Python, JS, Lisp, and even shell scripts can talk to low level interfaces just as well.

[Hejazzman]

@cdhowie that's a false dichotomy. You might want to use Python AND perform low level control of the system. It's also wrong that "not doing so results in less portable, less readable, and less maintainable code". Compared to what? Talking to the same low level APIs in C? No, it will be more portable and readable to talk to them from Python. If you mean "compared to using high level interfaces" that's a moot point, if what you need to do (business goal) involves low level interfaces. Low/High level APIs != Low/High level languages.