How can I ignore or remove ".ipynb_checkpoints" in colab?

Question

My code in tf.keras is given below. I want to retrieve a file(Xscale.npy) in each sub_directory(component_0, component_1) of model_cnn folder.

root_dir = '/content/drive/My Drive/DeepCID/model_cnn'

    i=0
    for (root, dirs, files) in os.walk(root_dir):
      for d in dirs:
        print(dirs) 
        os.chdir(os.path.join(root, d))
        print(os.getcwd())
        datafile3 = './Xscale.npy'
        Xscale = np.load(datafile3)

The error message is,

['.ipynb_checkpoints', 'component_0', 'component_1']
/content/drive/My Drive/DeepCID/model_cnn/.ipynb_checkpoints
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-862f78aebef9> in <module>()
     57     print(os.getcwd())
     58     datafile3 = './Xscale.npy'
---> 59     Xscale = np.load(datafile3)
     60     Xtest = (Xtest0 - Xscale[0])/Xscale[1]
     61 

/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    426         own_fid = False
    427     else:
--> 428         fid = open(os_fspath(file), "rb")
    429         own_fid = True
    430 

FileNotFoundError: [Errno 2] No such file or directory: './Xscale.npy'

I recognize that '.ipynb_checkpoints' is the problem. But, when I look into the folder, there is no .ipynb_checkpoints file or folder.

My drive in Colab is

My questions are

1) How can I ignore .ipynb_checkpoints when accessing a file in sub_directories?

2) Why is .ipynb_checkpoints file not visible in colab disk?

Thanks in advance, D.-H.

Solution 1

Change your code to the following.

1) Check if it is a hidden file

2) Not use os.chdir as it is not necessary.

root_dir = '/content/drive/My Drive/DeepCID/model_cnn'
datafile3 = 'Xscale.npy'

i=0
for (root, dirs, files) in os.walk(root_dir):
    for d in dirs:
        if not d.startswith('.'):
            dir_path = os.path.join(root, d)
            file_path = os.path.join(dir_path, datafile3)
            Xscale = np.load(file_path)

There are more elegant ways to do this in terms of getting absolute file paths but I wanted to minimize the amount of code changed.

An alternative method uses pathlib.

from pathlib import Path

root_dir = '/content/drive/My Drive/DeepCID/model_cnn'
datafile3 = 'Xscale.npy'

i=0
for (root, dirs, files) in os.walk(root_dir):
    for d in dirs:
        if not d.startswith('.'):
            fp = Path(root) / d / datafile3
            Xscale = np.load(str(fp))

Solution 2

I was having the same issue. This might be happening because during the upload, it is also uploading the .ipynb_checkpoint which is a hidden folder. A fix for this is that you can manually delete the folder using

rmdir /content/drive/My Drive/DeepCID/model_cnn/.ipynb_checkpoints in a cell just before executing the code you mention.

Comments

[gold_cy]

the file isn't visible in the GUI because it is a dot file which means it is hidden by default. also no need to switch directories that breaks things, just feed the path to the item...

[Dong-Ho]

If possible, could you show me your more elegant way to do this?

[gold_cy]

added the cleaner way to do it which uses pathlib