Better way to find absolute paths during os.walk()?

Question

I am practicing with the os module and more specifically os.walk(). I am wondering if there is an easier/more efficient way to find the actual path to a file considering this produces a path that suggests the file is in the original folder when os.walk() is first ran:

import os

threshold_size = 500

for folder, subfolders, files in os.walk(os.getcwd()):
    for file in files:
        filePath = os.path.abspath(file)
        if os.path.getsize(filePath) >= threshold_size:
            print filePath, str(os.path.getsize(filePath))+"kB"

This is my current workaround:

import os

threshold_size = 500

for folder, subfolders, files in os.walk(os.getcwd()):
    path = os.path.abspath(folder)
    for file in files:
        filePath = path + "\\" + file
        if os.path.getsize(filePath) >= threshold_size:
            print filePath, str(os.path.getsize(filePath))+"kB"

For shaktimaan, this:

for folder, subfolders, files in os.walk(os.getcwd()):
    for file in files:
        filePath = os.path.abspath(file)
        print filePath

produces this(most of these files are in a subfolder of projects, not projects itself):

C:\Python27\projects\ps4.py
C:\Python27\projects\ps4_encryption_sol.py
C:\Python27\projects\ps4_recursion_sol.py
C:\Python27\projects\words.txt
C:\Python27\projects\feedparser.py
C:\Python27\projects\feedparser.pyc
C:\Python27\projects\news_gui.py
C:\Python27\projects\news_gui.pyc
C:\Python27\projects\project_util.py
C:\Python27\projects\project_util.pyc
C:\Python27\projects\ps5.py
C:\Python27\projects\ps5.pyc
C:\Python27\projects\ps5_test.py
C:\Python27\projects\test.py
C:\Python27\projects\triggers.txt
C:\Python27\projects\ps6.py
C:\Python27\projects\ps6_pkgtest.py
C:\Python27\projects\ps6_solution.py
C:\Python27\projects\ps6_visualize.py
C:\Python27\projects\ps6_visualize.pyc
C:\Python27\projects\capitalsquiz1.txt
C:\Python27\projects\capitalsquiz2.txt
C:\Python27\projects\capitalsquiz3.txt
C:\Python27\projects\capitalsquiz4.txt
C:\Python27\projects\capitalsquiz5.txt
C:\Python27\projects\capitalsquiz_answers1.txt
C:\Python27\projects\capitalsquiz_answers2.txt
C:\Python27\projects\capitalsquiz_answers3.txt
C:\Python27\projects\capitalsquiz_answers4.txt
C:\Python27\projects\capitalsquiz_answers5.txt
C:\Python27\projects\quiz.py
C:\Python27\projects\file2.txt
C:\Python27\projects\regexes.txt
C:\Python27\projects\regexsearch.py
C:\Python27\projects\testfile.txt
C:\Python27\projects\renamedates.py

Solution 1

I think there you mistook what abspath does. abspath just convert a relative path to a complete absolute filename.

For e.g.

os.path.abspath(os.path.join(r"c:\users\anonymous\", ".."))
#produces this output : c:\users

Without any other information, abspath can only form an absolute path from the only directory it can know about, for your case the current working directory. So currently what it is doing is it joins os.getcwd() and your file

So what you would have to do is:

for folder, subfolders, files in os.walk(os.getcwd()):
    for file in files:
        filePath = os.path.join(os.path.abspath(folder), file)

Solution 2

Your work around should work fine, but a simpler way to do this would be:

import os

threshold_size = 500

root = os.getcwd()
root = os.path.abspath(root) # redunant with os.getcwd(), maybe needed otherwise
for folder, subfolders, files in os.walk(root):
    for file in files:
        filePath = os.path.join(folder, file)
        if os.path.getsize(filePath) >= threshold_size:
            print filePath, str(os.path.getsize(filePath))+"kB"

The basic idea here is that folder will be an absolute normalized path if the argument to os.walk is one and os.path.join will produce an absolute normalized path if any of the arguments is an absolute path and all the following arguments are normalized.

The reason why os.path.abspath(file) doesn't work in your first example is that file is a bare filename like quiz.py. So when you use abspath it does essentially the same thing os.path.join(os.getcwd(), file) would do.

Solution 3

This simple example should do the trick. I have stored the result in a list, because for me it's quite handy to pass the list to a different function and execute different operations on a list.

import os
directory = os.getcwd()
list1 = []

for root, subfolders, files in os.walk(directory):
  list1.append( [ os.path.join(os.path.abspath(root), elem) for elem in files if elem ])
# clean the list from empty elements
final_list = [ x for x in list1 if x != [] ]

Comments

[shaktimaan]

You should use filePath = os.path.join(path, file) instead of filePath = path + "\\" + file

[Jordan]

Oh duh! I just learned about the join method and it didn't even cross my mind. How come the os.path.abspath isn't returning the correct absolute path though?

[shaktimaan]

I tried running this on my system and os.path.abspath is returning correct path. Can you upload the output that you get while using abspath??

[Reishin]

rtfm, there is even example with obtaining full path.

[Priya]

Thank you @shaktimaan.. This explanation and along with code, helped me to understand concept better