Create missing directories in ftplib storbinary

Question

I was using pycurl to transfer files over ftp in python. I could create the missing directories automatically on my remote server using:

c.setopt(pycurl.FTP_CREATE_MISSING_DIRS, 1)

for some reasons, I have to switch to ftplib. But I don't know how to to the same here. Is there any option to add to storbinary function to do that? or I have to create the directories manually?

Solution 1

FTP_CREATE_MISSING_DIRS is a curl operation (added here). I'd hazard a guess that you have to do it manually with ftplib, but I'd love to be proven wrong, anyone?

I'd do something like the following: (untested, and need to catch ftplib.all_errors)

ftp = ... # Create connection

# Change directories - create if it doesn't exist
def chdir(dir): 
    if directory_exists(dir) is False: # (or negate, whatever you prefer for readability)
        ftp.mkd(dir)
    ftp.cwd(dir)

# Check if directory exists (in current location)
def directory_exists(dir):
    filelist = []
    ftp.retrlines('LIST',filelist.append)
    for f in filelist:
        if f.split()[-1] == dir and f.upper().startswith('D'):
            return True
    return False

Or you could do directory_exists like this: (a bit harder to read?)

# Check if directory exists (in current location)
def directory_exists(dir):
    filelist = []
    ftp.retrlines('LIST',filelist.append)
    return any(f.split()[-1] == dir and f.upper().startswith('D') for f in filelist)

Solution 2

I know it's kind of an old post but I just needed this and came up with a very simple function. I'm new to Python so I'd appreciate any feedback.

from ftplib import FTP

ftp = FTP('domain.com', 'username', 'password')

def cdTree(currentDir):
    if currentDir != "":
        try:
            ftp.cwd(currentDir)
        except IOError:
            cdTree("/".join(currentDir.split("/")[:-1]))
            ftp.mkd(currentDir)
            ftp.cwd(currentDir)

Usage example:

cdTree("/this/is/an/example")

Solution 3

I tried adding this as a comment to the @Alex L 's answer, but it was too long. You need to descend recursively when changing directory if you want to create directories on the way. E.g.

def chdir(ftp, directory):
    ch_dir_rec(ftp,directory.split('/')) 

# Check if directory exists (in current location)
def directory_exists(ftp, directory):
    filelist = []
    ftp.retrlines('LIST',filelist.append)
    for f in filelist:
        if f.split()[-1] == directory and f.upper().startswith('D'):
            return True
    return False

def ch_dir_rec(ftp, descending_path_split):
    if len(descending_path_split) == 0:
        return

    next_level_directory = descending_path_split.pop(0)

    if not directory_exists(ftp,next_level_directory):
        ftp.mkd(next_level_directory)
    ftp.cwd(next_level_directory)
    ch_dir_rec(ftp,descending_path_split)

Solution 4

I am using the following lines to resolve missing directory paths for FTP file copy

import os
ftps = FTP_TLS('ftps_server')
ftps.connect()
ftps.login()

destination_dir_path = 'some/dir/path'          # directory path on FTP
dir_path = ''
for i in destination_dir_path.split('/'):
    dir_path = os.path.join(dir_path,i)
    if i not in ftps.nlst(os.path.dirname(dir_path)):
        ftps.mkd(dir_path)                      # create directory on the FTP
ftps.storbinary(...)                            # store file using the binary mode

Solution 5

An alternative is to simply loop through each of the path elements, create the next and change into the newly-created directory. My use case was fairly straightforward though as I was copying items from one FTP server to another.

def create_ftp_path(session: ftplib.FTP, required_dir: str):
    required_dir = required_dir.split('/')[:-1]
    for path_item in required_dir:
        if path_item.strip() == '':
            continue
        path_item = path_item.replace('/', '')
        try:
            session.cwd(path_item)
        except:
            session.mkd(path_item)
            session.cwd(path_item)

Considerations:

• This function assumes you have already changed directory for your FTP session to some base path and the required_dir is a path from that base path.
• required_dir includes file name as the last element.
• I'm removing any / characters because in my case they were causing 553 permission denied exception.
• The exception handling is lacking, but in my case upload validation is happening further in the code so even if it fails it will be caught further down.

View more solutions

Comments

[AliBZ]

Thank you, although it was not exactly what I was looking for, but it was a good answer. Thanx ;)

[xApple]

very nice ! dir is a python built-in you might want to change that variable name... also you want to catch specific exceptions, not all of them

[xApple]

No, you don't have to do it manually. You could just call the makedirs method in the ftputil package instead.

[lecnt]

Thanks you xApple for your feedback. I replaced 'dir' and restricted to only catch IOError exceptions.

[xApple]

I think you forgot to replace an instance of the dir variable.

[lecnt]

Oops, fixed it. Thank you for catching it. Now it's perfect, it just needs to be up voted. :)

[Alex L]

@lecnt - a couple more suggestions: lower_case_with_underscores is preferred for function/variable names - see pep8. Also it's better to use os.path functions for manipulating paths, e.g. os.path.normpath(os.path.join(current_dir, '..'))

[sushh]

smart solution. Exactly what i was looking for.!

[Mykhailo Seniutovych]

Note this solution only works with top level directories and does not work with sub-directories

[Julian Drago]

I'd like to highlight that this also answers another common question: how do I check if a ftp directory exists. I think this way is more Pythonic than the answer of listing the directory contents then looking for the directory name. Instead, try to change to that directory, and catch the error, in line with "ask for forgiveness" rather than "ask for permission".

[tommy.carstensen]

In my opinion try and except is very pythonic. This question deals with it: stackoverflow.com/questions/7604636

[dloeckx]

This no longer works (with me). I had to change ioerror to ftplib.error_perm, and ftp.mkd(currentDir) to ftp.mkd(currentDir.split("/")[-1]).