Doesn't metadata occupy any size?

Question

I put 4096 characters in a text file and save it.Since every character is 1 byte, Size of the text file must be 4K byte.As you see below that's OK:

I connect my flash memory to my computer. the free space on the flash memory is 1,717,518,336 bytes :

I created a copy of the file in my flash memory. And again take a look at the free space. it has 1,717,514,240 bytes free space :

Let see what is the difference :

1,717,518,336 - 1,717,514,240 = 4096 bytes

---

My question :

Q1:

As you see in the last picture above, the only space that the file occupy on the flash, is the space for its contents [characters].So where is metadata file ?

I mean, when I move the file to another computer, how it understand name of file, Owner of file, Date created and modified and ... ?

Doesn't it occupy any size?!!

Q2:

Can I see the metadata file in the flash memory?

Appreciate your time and consideration.

Solution 1

Yes, metadata occupies space. On NTFS it occupies 1024 bytes, to be specific. However, the information is not stored in the file, but in the master file table MFT. Specifically in MFT record #4 $AttrDef.

See this Technet article for details: table 3.5 holds all MFT records defined.

When a volume is formatted with NTFS, a Master File Table (MFT) file and other pieces of metadata are created. Metadata are the files NTFS uses to implement the file system structure. NTFS reserves the first 16 records of the MFT for metadata files.

NTFS creates a file record for each file and a directory record for each directory created on an NTFS volume. The MFT includes a separate file record for the MFT itself. These file and directory records are stored on the MFT. The attributes of the file are written to the allocated space in the MFT. Besides file attributes, each file record contains information about the position of the file record in the MFT.

Note that other file systems can and do deal differently with metadata.

EDIT: It has been pointed out in the comments section that this answer is missing the point because the OP asked for metadata on FAT32 filesystems, not NTFS. If I knew how, I would remove the "correct answer" attribute. Therefore I provide additional information that answers the question with regard to FAT32.

FAT32 saves simple metadata such as visibility or modification time for each file and folder in an entry in the parent folder of the the file or folder, creating a tree down from the root folder of the FS. As pointed out with regards to NTFS this is not a file but saved within the folder data structure. The entry originally was 32 byte large and contained the following attributes:

Name (8.3) xxxxxxxx.yyy. (88 bits)

Attribute byte (8 bits of information, described later in this section).

One reserved byte.

Create time (24 bits).

Create date (16 bits).

Last access date (16 bits).

Two reserved bytes.

Last modified time (16 bits).

Last modified date (16 bits).

Starting cluster number in the file allocation table (16 bits).

File size (32 bits).

The list was taken from this Microsoft Technet article and pertains to FAT16. Since the cluster size of FAT32 can be 32 bits and the name of files can be longer than 8.3 the table is not wholly accurate. To accomodate long file names and larger disks FAT32 modifies some behaviour which can be read up in the Wikipedia here but the basic idea holds.

Solution 2

Doesn't it occupy any size?!!

Yes, but it's a small entry in a large pre-allocated block. That block is counted in the "used" portion of your disk. Adding an entry inside that block doesn't require the block to be expanded.

Depending on the filesystem, eventually the block will be filled and extended somehow after a lot of filenames are added.

Can I see the metadata file in the flash memory?

Not easily

As Ruslan and Blorgbeard commented. You can install a hex viewer such as HxD which will allow you to view (and edit - be very careful) the raw filesystem data. But you'll have to do the interpretation yourself concerning which bytes belong to which filesystem structures. For that you would need some good documentation of the specific filesystem used on the flash disk. FAT32 is likely to be simpler to understand than any of the many variants of NTFS. See Understanding FAT32 Filesystems for example.

Solution 3

Metadata is not stored (nor reported by usual file management tools) as files, it is stored on filesystem's data on disk.

Depending on the nature / version of the filesystem, each entry will take some amount of disk space to represent the metadata information.

Moreover alongside the space allocated in the Master File Table, some filesystems will also keep journal about files change (taking extra space), and some filesystems can even be extended with extra fields containing special purpose metadata.

So technically metadata takes disk space, but it is not taken in account by most file management utilities which works calling system's API that query the filesystem for file space, not for metadata space.

And this abstraction is only the tip of the iceberg, as filesystem itself works on an abstraction of underlying physical disk space, provided by low level disk routines, so actually only the disk's internal logic will know how many actual memory positions are available for higher levels and how many are marked as not reliable, reserved, or used for checksum.

Solution 4

The metadata depends on the file system. The most basic file systems usually used on exchangeable media is based on a DOS file system (FAT). DOS doesn't have users, and permissions. Or, more correctly, some of that information is carried in the 8th data bit of the file name. The only resource overhead that is used in a basic DOS FS is to account for the blocks of the file, or the directory that it is in - and that block is probably already committed for handling ".", already. IOW, adding an empty file wouldn't add another bit of storage, but it would change some existing consumed and allocated bits.

When you use more advanced file systems, with journalling and users, you get metadata, and journal entries and possibly a forensic chain to recover previous file versions, etc. Then a small file can explode in used storage.

So, check the FS type. If it is FAT, then you probably don't have a user recorded in metadata on the media. Hence... no space used. :)

A good example of a FAT FS is an open source implementation - and you can see that the list of operations available does not include "get/set user" and "check access by user". No identity = no storage of that identity.

Solution 5

Where are metadata kept?

When we talk about metadata, there are two types of metadata.

The first type includes created date, last modified date, last accessed date. Depending on file system (i.e. NTFS / FAT / Ext3 etc...) there will be different "metadata" available, for example Windows owner and permission on NTFS.

The first type applies to all files, e.g. the .txt file in your example.

All file systems pre-allocate spaces to contain these metadata at the Master File Table (MFT for NTFS, some other names for other file system), which is not directly accessible by users. Since MFT is not counted as part of drive capacity, there is no "additional" storage space required anyway.

The second type of "metadata" is defined by the file type or application. For example, Office documents keeps "Author", "Subject" and other metadata; JPEG images keeps a set of EXIF data including "date picture taken", "camera model", "shutter speed"; while MP3 sound contains "album", "track #", "bitrate"...

The second type DOES take up additional space, because these "metadata" forms part of the file.

---

Different size in different drive

When your text file is on C:\ it takes up 4K. It becomes 1K in size when you put it on your flash drive H:\. This is because different "block size" for the different partitions.

Files are allocated spaces in block. Therefore, on a file system of block size 4K, 1 byte is allocated 4K while 4,097 bytes (4K + 1 byte) is allocated 8K.

Apparently your C: is formatted with 4K block size while H: is formatted with 1K block size, resulting in the difference.

View more solutions

Comments

[TheGoodUser]

Thank you dear friend. Can I see the file that metadata store in it? and edit it with a tool, for example with a hex viewer?

[TheGoodUser]

Not easily!? So I can? :D how?

[user5249203]

@TheGood If deprived of Google, I'd probably start by booting Linux and using dd to copy the raw flash filesystem to a file on another disk then use a hex-viewer to examine it in conjunction with a good reference-work on the specific filesystem (if not proprietary and undisclosed). I might make a metadata change with touch, repeat the dd and use a binary diff.

[Cthulhu]

Good thing I'm not deprived of Google.

[Rob Moir]

@TheGoodUser-Sp metadata, on NTFS at least, isn't stored in "a file" in the conventional sense, which is what I get the impression you are looking for. There isn't something tucked away in the \windows folder somewhere that we're all just not telling you about; file metadata is an integral part of the filesystem itself.

[jnovacho]

In this case the Flash Drive is formatted as FAT-32. So the metada are stored in File Allocation Table (FAT) en.wikipedia.org/wiki/File_Allocation_Table

[Ruslan]

There's no need to change OS to use a hex-viewer on a hard drive directly. Just use a decent hex viewer. (It seems even hiew can do it, but I'm not sure since I was on windows long ago).

[Kaslai]

It's important to note that the allocation tables for NTFS and FAT are preallocated. It differs from format to format, but several sectors are usually occupied with empty preallocated blocks that are to be used for storing metadata in order to reduce fragmentation.

[Ben Voigt]

What makes you think metadata is fixed size? The access control list certainly can vary quite a lot in complexity; I fail to see how it can always fit in 1024 bytes (along with all the other metadata like access and modification times)

[Blorgbeard]

HxD will definitely do it, on Windows. Extras menu, Open Disk.

[bjanssen]

@Ben Voigt: I see where you are coming from. That's bad phrasing on my part, so let me clarify. The metadata record in the MFT is always 1024 bytes, if used or not. The actual used space is tracked in the Allocated Size attribute. If the metadata is smaller than 1024 bytes the record is padded, if it is larger another record is created and linked to the now primary record. You can read a treatment of this in the Wikipedia article en.wikipedia.org/wiki/… In summary you are right in saying there can be more metadata than fits in 1KB.

[Kenneth L]

Whether the metadata is copied depends on how you do it. For example robocopy gives you option whether to copy timestamps, attributes, owner information, access control lists (ACL) etc. It also depends on whether the file system on your flash memory supports it (e.g. FAT32 does not keep owner info and ACL so they are discarded anyway). Metadata is not in the form of a file so you cannot see the raw data; if you want to see low level things you may need dd in Linux but how to interpret them is another story.

[user5249203]

@Blorgbeard, Ruslan: Thanks, I've updated the answer to include your suggestions.

[user1686]

@RobM: Technically $MFT is a file in the root directory, isn't it? ntfs-3g allows one to read it under Linux.

[Rob Moir]

@grawity yes you're right but I think referring to it in that manner in the context of this question is just going to confuse the OP.

[bjanssen]

@JezC: You are right, I missed this part. Even the screenshot states FAT32. So I have added some information with regards to saving simple metadata in FAT-FS.

[user541686]

FYI, I don't think $AttrDef contains what you think it contains.