How to get the summarized sizes of directories and their subdirectories?

Question

I have the following code

import nltk
from nltk.corpus.reader import TaggedCorpusReader

corpus_root = 'C:/Python27'
reader = TaggedCorpusReader(corpus_root, 'test.txt')
print reader.words()
print reader.tagged_words()

and the problem is that in the results give me value 'None' in each word..

[('And', 'None'), ('now', 'None')..

and when i have this code

import nltk
text = nltk.word_tokenize("And now for something completely different")
nltk.pos_tag(text)

the result is right...

[('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'), ('completely', 'RB'), ('different', 'JJ')]

What is going wrong??

Solution 1

This does what you're looking for:

du -sh /*

What this means:

• -s to give only the total for each command line argument.
• -h for human-readable suffixes like M for megabytes and G for gigabytes (optional).

• /* simply expands to all directories (and files) in /.

  Note: dotfiles are not included; run shopt -s dotglob to include those too.

Also useful is sorting by size:

du -sh /* | sort -h

Here:

• -h ensures that sort interprets the human-readable suffixes correctly.

Solution 2

I often need to find the biggest directories, so to get a sorted list containing the 20 biggest dirs I do this:

du -m /some/path | sort -nr | head -n 20

In this case the sizes will be reported in megabytes.

Solution 3

I like to use Ncdu for that, you can use the cursor to navigate and drill down through the directory structure it works really well.

Solution 4

The existing answers are very helpful, maybe some beginner (like me) will find this helpful as well.

1. Very basic loop, but for me this was a good start for some other size related operations:

   for each in $(ls) ; do du -hs "$each" ; done
   

2. Very similar to the first answer and nearly the same result as 1.), but it took me some time to understand the difference of * to ./* if in a subdirectory:

   du -sh ./*
   

Solution 5

The following du invocation should work on BSD systems:

du -d 1 /

View more solutions

Comments

[Ryan]

The --total flag was helpful for me. E.g. du -sh --total applications/*. askubuntu.com/a/465436/48214

[linuxfan]

My du (Ubuntu 10.4) doesn't have a -d option. What system are you on?

[2ndkauboy]

On my openSUSE it doesn't have a -d option either :(

[Philipp]

OK, then it's a BSD option only (I'm on OS X).

[Dummy00001]

Right portable option combination on BSD/*NIX is du -sk /*. I hate the -k stuff soooo much. Linux' -h totally rocks.

[Philipp]

If you have dot-directories in the root directory, you can use shopt -s dotglob to include them in the count.

[psur]

It's very usefull, because it's simple and you can place what path you want instead of /*, e.g. ./ for current directory or ./* for each item in current directory.

[Xedecimal]

Here's a way to get it more readable du -sh /some/path | sort -hr | head -n 20

[chrisan]

@Xedecima the problem with using h is the sort doesn't know how to handle different sizes. For example 268K is sorted higher than 255M, and both are sorted higher than 2.7G

[Xedecimal]

The -h (human readable) argument on the 'sort' command should properly read these values. Just like du's -h flag exports them. Depending on what you're running I'm guessing.

[Vishnu Kumar]

in other systems, its --max-depth

[podarok]

--threshold ^^^ this option is not availavle on linux

[dingzhihu]

@podarok It's available on OpenSUSE 13.2 Linux. Try to find a more recent version of your distribution or compile a more recent version of the package yourself.

[podarok]

It doen't work on Ubuntu LTS (14.04). It is the most recent one ))

[dingzhihu]

@podarok Which version of GNU coreutils? Mine is 8.24.

[podarok]

Sorry. It was not on Ubuntu 14.04. It was Centos 6.6. Lost in consoles....

[ganesh]

Yes, that can be very slow. But you can cache the output from a single find and work from that. No need to access the filesystem several times.

[dingzhihu]

@Hennes: How do you cache file sizes in a shell script?

[ganesh]

Caching might have been a bad term. I was thinking of something like done in this port superuser.com/a/597173/121352 where we scan the disks contents once into a mapping and then continue using data from that mapping rather than hitting the disk again.

[relascope]

@psur or you can use ./*/ to get only subfolders and not all items

[Kamil Maciorowski]

I agree one can expand this example and do tricks like getting the size of "certain subsets of directories, according to filetypes" etc.; it may seem a good starting point. Nevertheless this solution is flawed from the start. To every user who would like to use this method I recommend reading answers and comments to this question as well as the article linked there. I don't say you cannot do it at all. Know the limitations, that's all.

[Vu Anh]

Sorted version: du -sh /* | sort -h

[SDsolar]

Nice tip on the sort -h

[SDsolar]

Works in Ubuntu 16.04. Nice tip.

[c1phr]

If you're using alpine, -h isn't an available flag on sort. sort -nr should get you the same result, leading to du -sh /* | sort -nr.

[linuxfan]

@c1phr If your sort doesn't have -h, you need to leave it off from du as well, otherwise the sorting will mix up kilo/mega/gigabytes. du -s /* | sort -nr.

[deviant]

sudo du -haxt 1G / | sort -hr | head -30

[joseluisq]

This should throw the top ten of biggest directories: du -hS /var/lib/docker | sort -rh | head -10

[Ciro Santilli Путлер Капут 六四事]

Awesome. Keywords: du meets ncurses. You can use b to drop into a shell in the directory.

[machineaddict]

for each does not work as it appends console characters (eg \033[) to the list of folders

[Martin]

@machineaddict not sure what you mean. I use this all the time, works for me just fine.

[machineaddict]

try to run your command starting with for each. it will not work

[Martin]

i run the command exactly as written here. starting with for each. works.

[userlond]

I've ended with du -hs /* | sort -hr. It outputs sizes in readable format (5GB, 6.6M) and sorts values descending.

[bobbogo]

Personal plea: please never parse the output of ls. Just use ordinary bash filename exopansion. A simple for f in * is safer, more accurate, faster etc.