Copy file with pathlib in Python

Question

I try to copy a file with pathlib

import pathlib
import shutil

my_file=pathlib.Path('/etc/hosts')
to_file=pathlib.Path('/tmp/foo')
shutil.copy(my_file, to_file)

I get this exception:

/home/foo_egs_d/bin/python /home/foo_egs_d/src/test-pathlib-copy.py
Traceback (most recent call last):
  File "/home/foo_egs_d/src/test-pathlib-copy.py", line 6, in <module>
    shutil.copy(my_file, to_file)
  File "/usr/lib/python2.7/shutil.py", line 117, in copy
    if os.path.isdir(dst):
  File "/home/foo_egs_d/lib/python2.7/genericpath.py", line 41, in isdir
    st = os.stat(s)
TypeError: coercing to Unicode: need string or buffer, PosixPath found

Process finished with exit code

... how to copy file with pathlib in Python 2.7?

Solution 1

To use shutil.copy:

import pathlib
import shutil

my_file = pathlib.Path('/etc/hosts')
to_file = pathlib.Path('/tmp/foo')

shutil.copy(str(my_file), str(to_file))  # For Python <= 3.7.
shutil.copy(my_file, to_file)  # For Python 3.8+.

The problem is pathlib.Path create a PosixPath object if you're using Unix/Linux, WindowsPath if you're using Microsoft Windows.

With older versions of Python, shutil.copy requires a string as its arguments. For them, use the str function here.

Solution 2

The cause for shutil.copy() not working is that you are not using the latest Python, Python 3.6 shutil.copy() can handle Path objects (or subclasses thereof). That for older versions of Python this throws an error is because those implementations of shutil expect string arguments for copy, and not pathlib.Path type arguments.

What you actually want to be able to write is:

my_file.copy(to_file)

You can subclass Path to include such a method, and adapt the creation of my_file. I find it easier to just graft/monkey-patch/duck-punch it on the existing pathlib.Path

from pathlib import Path


def _copy(self, target):
    import shutil
    assert self.is_file()
    shutil.copy(str(self), str(target))  # str() only there for Python < (3, 6)

Path.copy = _copy

You can put this code anywhere you like, as long as it gets executed before calling the .copy method on any of the Path instances. The argument to .copy() can be a file or a directory.

Solution 3

Since Python 3.5, without importing shutil, you can do:

from pathlib import Path

dest = Path('dest')
src = Path('src')
dest.write_bytes(src.read_bytes()) #for binary files
dest.write_text(src.read_text()) #for text files

For Python 2.7, pathlib2 provides the read_bytes, read_text, write_bytes and write_text methods.

The file will be loaded in memory, so this method is not suitable for files larger than the machines available memory.

As per the comments, one can use write_bytes and read_bytes to copy text files, but if you need to deal with the encoding at copy time write_text an read_text present the advantage of two extra parameters:

• encoding is the name of the encoding used to decode or encode the file
• errors is an optional string that specifies how encoding and decoding errors are to be handled

They both have the same meaning as in open().

Solution 4

How shutil was converted to accept pathlib.Path objects in Python 3.6

As mentioned at in this answer, shutil in Python 3.6 can take pathlib.Path objects.

Since this felt quite magic, I decided to investigate a little bit how it was implemented to see if I would be able to reuse this magic on my own classes.

The improvement was a result of PEP 519.

This generalized a lot of stdlib functionality, and documentation was not consistently updated as a result, including most of shutil which as of 3.7 only documents support in a single function. Welcome to the joys of dynamic typing.

Where documented, the stlib links to the glossary for "path-like objects".

An object representing a file system path. A path-like object is either a str or bytes object representing a path, or an object implementing the os.PathLike protocol. An object that supports the os.PathLike protocol can be converted to a str or bytes file system path by calling the os.fspath() function; os.fsdecode() and os.fsencode() can be used to guarantee a str or bytes result instead, respectively. Introduced by PEP 519.

and that then links to the documentation of os.PathLike:

An abstract base class for objects representing a file system path, e.g. pathlib.PurePath.

New in version 3.6.

abstractmethod __fspath__()

Return the file system path representation of the object.

The method should only return a str or bytes object, with the preference being for str.

The key implementation commits seem to be:

• 3f9183b5aca568867f37c38501fca63911580c66 which basically changes the lowest level path manipulation point at Modules/posixmodule.c
• 568be63248614a2cdd7666a67ddfd16e817f7db9 which adds __fspath__ to pathlib.PurePath

If you want to implement your own path-like classes, you can do it like:

#!/usr/bin/env python3

class MyPath:
    def __init__(self, path):
        self.path = path
    def __fspath__(self):
        return self.path

with open(MyPath('f'), 'w'):
    pass

Tested in Python 3.6.7, Ubuntu 18.10.

Solution 5

You might use pathlib3x - it offers a backport of the latest (at the date of writing this answer Python 3.11.a0) Python pathlib for Python 3.6 or newer, and a few additional functions like copy, copy2, etc ...

$> python -m pip install pathlib3x
$> python
>>> import pathlib3x as pathlib
>>> my_file = pathlib.Path('/etc/hosts')
>>> to_file = pathlib.Path('/tmp/foo')
>>> my_file.copy(to_file)

you can find it on github or PyPi

---

^{Disclaimer: I'm the author of the pathlib3x library.}

View more solutions

Comments

[Anthon]

This works without throwing an error on Python 3.6

[guettli]

@Anthon we use Python 2.7.

[guettli]

If I understood the problem you talk about correctly, then the code is not portable: as_posix() does not work on window. OK, /etc/hosts does not exist on windows, but the path object could be created from some config which defines a windows path "C:\....". Is there no portable solution?

[Remi Guan]

@guettli Actually if you're using my_file.as_posix() on Windows, it'll return C:/etc/hosts. So that doesn't matter.

[guettli]

Don't take this personal: I though pathlib was made to make things easier. I guess I stick with using plain old strings like I used to do.

[Andrew Wagner]

I would also have expected Pathlib to be able to copy a file, given that it can move/rename and unlink/delete files.

[guettli]

AFAIK this is called Monkey-Patching. Why does pathlib not provide this?

[Anthon]

@guettli Probably for the same reason as that you have to import copy from shutil: someone thought it was not necessary. I have a range of such monkey patches in my ruamel.std.pathlib library, and some supporting routines for transitioning to pathlib (which with the changes in 3.6 are partly superfluous)

[guettli]

Is there a way to do this with Python 2.7?

[Jacques Gaudin]

There is something for Python 2.7, namely pathlib2 but I haven't tried it. pypi.python.org/pypi/pathlib2. The read_bytes and write_bytes methods are in the source code, so I presume they work.

[Jacques Gaudin]

@GeorgeSovetov Thanks I added it to the answer.

[Wayne Werner]

Note that the dest.write_bytes(src.read_bytes()) actually works for text files, too. Honestly I'd just do that, since that's literally what shutil.copy does under the hood (not the only thing it does, but some of it)

[Jacques Gaudin]

@WayneWerner Sure but write_text has a few more arguments, namely encoding and errors to help with text file manipulation. See github.com/python/cpython/blob/3.6/Lib/pathlib.py#L1206

[Wayne Werner]

@JacquesGaudin I guess it depends on whether or not you want to deal with encoding issues at the the time, or if you just want a 1:1 copy of the original file

[Jacques Gaudin]

@WayneWerner Absolutely, I have amended the answer to mention them and make the choice clearer. Thanks.

[Steven Rumbalski]

With this method a sufficiently large file would crash Python with a MemoryError. shutil.copy doesn't have this problem because it uses shutil.copyfileobj which buffers larger files into smaller chunks.

[Mr. B]

Not perfect for larger files, but to_file.write_bytes(my_file.read_bytes())

[Anthon]

Using shutil.copy() you will have two copies: the original and the target. If you use Path.rename your original is gone, so this is not related to what the op wants at all.

[Bram Vanroy]

I'd say pathlib does not provide this functionality because that's not what it's meant for - just as os.path was not meant for file handling itself. The module's functionality is about file handling and file's metadata, but not about file handling, as far as I know.

[Nathan]

Copying with read_bytes and write_bytes should preserve the file exactly, but if you do read_text and write_text, do newlines potentially get changed? E.g. if I'm on Windows, will "\n" be converted to "\r\n"? This is what happens with a standard open("fname").read(), and it's why I prefer to read/write bytes if just copying a file.

[Anthon]

@mr.zog I don't think your intended audience will get notified that you wrote a comment when you just put in a name. You should consult the help on formatting of comments where it clearly states to use the @ character

[mr.zog]

@Bram Vanroy "The module's functionality is about file handling and file's metadata, but not about file handling" is a bit contradictory. What did you intend to convey?

[Bram Vanroy]

@mr.zog There's indeed a mistake in that comment. What I meant to say was that pathlib is not meant to do file manipulation actions such as copying or moving.

[Tim M.]

@AndrewWagner the difference, if I had to guess, is that moving and deleting files are purely filesystem operations, where you just update filesystem metadata. Copying involves actually writing out new data, so maybe the pathlib people thought it was out of scope.

[SwimBikeRun]

@guettli This will be fixed once this goes in github.com/python/cpython/pull/15326

[patrick-mooney]

This copies the file's contents, but not file permissions, ownership, access policies, or other file metadata. OP's comment actually says that s/he wants to "copy the file," not just the data it contains.

[Oliver]

Although Path is primarily designed to represent a file path, it also provides mkdir() and exists() and such, so it is natural to expect it to do a lot of what shutil does. A proper separation of concerns would probably have Path and File and Folder with File/Folder being created from a Path and File have getPath() etc. Then once you have a File or Folder it's pretty obvious it makes sense to have copy(), copytree() etc on those objects instead of on Path. I use Path everywhere I can, btw.

[AMC]

@KevinGuan Can you update this, or you would you mind if I did?

[SwimBikeRun]

If anyone is wondering why shutil.move doesn't work the same way in Python<3.9, this is fixed in Python3.9 as of my contribution :D

[Conchylicultor]

For compatibility with other pathlib-like API, os.fspath should be used rather than str. See python.org/dev/peps/pep-0519

[acl]

For info: "older Python" means Python 3.7, "newer Python" is Python 3.8 and above.

[Timo]

I thought that using python at least 3.6 is ok for omitting strings in shutil.copy, see the comment from Anthon above.