cannot 'ls' /mnt directory

Question

I have a CentOS-6 machine that I'm working on that is running a live database. It's having a problem with the /mnt directory. I cannot ls the directory for some reason. stat is working and shows this output:

  File: `/mnt/'
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: ca02h/51714d    Inode: 16321       Links: 9
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2011-09-23 06:50:20.000000000 -0500
Modify: 2012-10-29 11:46:33.000000000 -0500
Change: 2012-10-29 11:46:33.000000000 -0500

I can also cd into the directory and issue both the touch and mkdir commands successfully (or at least without errors. However, if I try to run ls in the directory it just hangs. At that point neither Ctrl+C nor Ctrl+Z have any effect and my system locks up. I have to close my ssh window and reconnect to do anything else.

I'm at a bit of a loss here, does anyone have any ideas what might be causing this? Or, does anyone know how I can try to fix it, without rebooting the machine (I've encountered this once before on the same directory on a different machine and it was corrected by rebooting) since this is a live machine, I can't actually reboot.

These are the filesystems which are mounted in that directory:

blob.XXXXX.com:/blend on /mnt/blend type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
blob.XXXXX.com:/new_log on /mnt/new_log type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
blob.XXXXX.com:/new_backup on /mnt/new_backup type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
blob.XXXXX.com:/vz on /mnt/vz type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
blob.XXXXX.com:/git on /mnt/git type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)

Interestingly enough, the output of strace ls /mnt shows the mount directories in there.

Solution

One thing is, ls attempts to stat() every file within the directory (which includes, in this case, the root of every mounted file system), which neither of the commands you attempted successfuly do so. So the problem is likely with one of the mounted filesystems, and not with /mnt itself.

To know more, you need to figure out what filesystems you have mounted in /mnt, and try each of them individually (ls -d /mnt/foo) to see which one causes the problem.

Comments

[Admin]

What filesystems do you have mounted?

[Admin]

pastebin.com/xgqYsrLC

[Admin]

Have you managed to get it to freeze under strace? (with --color=auto or some other option)

[CRThaze]

Neither of them seem to be having an issue.

[Random832]

so if you ls -d everything in /mnt, one at a time, none of them freeze? And if you ls /mnt again it still freezes again?

[Random832]

Can you pastebin the strace output? Maybe this will show where it freezes.

[CRThaze]

Yup, that's about right. Here's the strace outupt: pastebin.com/k39WGub7

[CRThaze]

Yet, I was able to stat every file in the directory manually. (there were only about 7 as can be seen in the strace output, unless I'm missing something)

[Ciprian Tomoiagă]

@Gilles'SO-stopbeingevil' that's an amazing insight ! Thanks ! running \ls (to bypass aliases) runs it instantly, whereas the color made it hang due to NFS not being available