mkfs refuses to format device in the claim it's mounted when actually it is not

Question

This is the problem:

root@ip-10-126-247-82:~# mkfs.ext4 /dev/xvda3
mke2fs 1.41.14 (22-Dec-2010)
/dev/xvda3 is mounted; will not make a filesystem here!

And this is the debugging:

root@ip-10-126-247-82:~# mount
/dev/xvda1 on / type ext4 (rw)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)

Further more, device /dev/xvda2 the kernel will reformat and xvda1, xvda2, xvda3 are different devices

root@ip-10-126-247-82:~# ls -la /dev/xvda*
brw-rw---- 1 root disk 202, 1 2011-12-21 18:54 /dev/xvda1
brw-rw---- 1 root disk 202, 2 2011-12-22 10:33 /dev/xvda2
brw-rw---- 1 root disk 202, 3 2011-12-21 18:54 /dev/xvda3


root@ip-10-126-247-82:~# cat /proc/partitions 
major minor  #blocks  name

 202        1   10485760 xvda1
 202        2  356485632 xvda2
 202        3     917504 xvda3

It won't format xvda1 (correct)

root@ip-10-126-247-82:~# mkfs.ext4 /dev/xvda1 
mke2fs 1.41.14 (22-Dec-2010)
/dev/xvda1 is mounted; will not make a filesystem here!

It will format xvda2 (correct)

root@ip-10-126-247-82:~# mkfs.ext4 /dev/xvda2
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
22282240 inodes, 89121408 blocks
4456070 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
2720 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

It won't format xvda3 (incorrect)

root@ip-10-126-247-82:~# mkfs.ext4 /dev/xvda3
mke2fs 1.41.14 (22-Dec-2010)
/dev/xvda3 is mounted; will not make a filesystem here!

-- EDIT:

Adding lsof debug as @Janne Pikkarainen suggests:

root@ip-10-126-247-82:~# lsof -n | grep '202,3'
root@ip-10-126-247-82:~# lsof -n | grep 'xvda3'
root@ip-10-126-247-82:~# 

Solution 1

It seems that the mount point is for swap.

I believed I've been working on a m1.large when actually I was working on c1.medium which has only 1 ephemeral storage attached.

root@ip-10-126-247-82:~# python
Python 2.7.2+ (default, Oct  4 2011, 20:03:08) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto.utils
>>> boto.utils.get_instance_metadata()['block-device-mapping']
{'ami': 'sda1', 'root': '/dev/sda1', 'ephemeral0': 'sda2', 'swap': 'sda3'}

The mapping here is a bit mixed up (don't know why). sda -> xvda. From this you can see that / is mapped to /dev/xvda1, ephemeral0 is mapped to /dev/xvda2 and swap is mapped to /dev/xvda3. I believe that the swap space is managed by Xen, which is the reason why I can't reformat / unmount it.

I can further confirm this by

root@ip-10-126-247-82:~# cat /proc/swaps 
Filename                                Type            Size    Used    Priority
/dev/xvda3                              partition       917500  0       -1

But now I'm not sure if swap is being used, as it does not appear in mount output.

-- EDIT:

General info regrading EC2, on c1.medium you get free swap space, on m1.large you don't. By "free" I mean you can unmount it and then reclaim it as another ephemeral storage, just a hack :)

swapoff -a
mkfs.ext4 /dev/xvda3
mount /dev/xvda3 /mnt
df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            9.9G  1.6G  7.8G  17% /
udev                  849M  4.0K  849M   1% /dev
tmpfs                 342M  172K  342M   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  854M     0  854M   0% /run/shm
/dev/xvda3            882M   17M  821M   3% /mnt

Solution 2

It is possible to mount something without it appearing to /etc/mtab at all by using -n switch in mount.

Does lsof -n | grep xvda3 return something?

Comments

[Admin]

cat /proc/mounts

[Janne Pikkarainen]

Try swapon -s.