Explorer.exe is accessing $Mft (NTFS Master File Table) and causing excessive hard disk grinding in a directory containing ~10,000,000 files

Question

I have an internal HDD which contains a directory, which contains a series of folders, which contain a total of about about 10 million small text files. Whenever I open this directory in Windows Explorer, the hard drive goes absolutely nuts and sounds like it's having a heart attack.

Resource Monitor shows that it's due to a roughly 11.5 MB/sec access to the drive's NTFS Master File Table (see below screenshot):

Indexing has been disabled for this directory, and all its subdirectories, as it is known that drive indexing and large numbers of small files are a volatile combination. Likewise, the directory only contains a couple folders (each of which house a large number of files), so none of the files in question are ever being displayed onscreen.

As such, I am a bit baffled as to why this intense reading occurs. It slows down any other processes which are working with the HDD to a near-standstill whenever it happens, so I have recently started navigating the drive and its contents via Command Prompt/PowerShell, which, for whatever reason, does not seem to trigger the NTFS reading-frenzy.

I'm not really a tech person, and so I don't know the details of how Windows 7 accesses drives to display files and folders, so I have a couple questions:

• Why is the NTFS Master File Table being accessed, even though none of the files in question ever appear on screen, and none of them are ever being opened?

• Is it absolutely necessary?

• If not, is there a way to disable it? If so, what negative consequences would result?

• Is there a more efficient (in terms of reducing disk wear and unnecessary file table access) way to set up a directory whose sole purpose is to house a large number of files?

Additional info: The drive is healthy (no S.M.A.R.T errors, and CrystalDiskInfo says there have been no problems), and is not a boot drive.

Solution

A good way to store this many files is by adding some directory structure to it.

For example see the "algorithm" described here: https://serverfault.com/a/95454/5109

Comments

[cybernard]

have you ever defragged this hdd?

[DumpsterDoofus]

@cybernard: No, but I don't see why that would be a problem, considering that 1) the files have all been added recently, and not many deletions or modifications have occurred, and more importantly 2) none of the files are ever displayed or opened during the operation (only the 10 or 11 folders containing them appear onscreen).

[cybernard]

have you ever done a chkdsk /r To see if the drive has gained any bad sectors over time?

[Psycogeek]

if the files are so small that they fit in the MFT , that is where they will be stored. I think the size is around 600bytes or smaller. The Idea is you already have a specific space allotted for each file record, if the data also will fit in the file record , that is where it will be put. There are better explainations on the sites that explain the file systems. With that many file items, your MFT may become fragmented. In most normal situations the MFT is actually way to big. You have a rare use case here where it might actually be too small

[Psycogeek]

Windows 7 now parses much more of the file information than XP ever did when hopping into the folder/directory for the files. It probably follows along with other "Pre-Do" stuff that attempts to speed up the system, and can instead slow the system , when you were not needing to Do what it pre-doed. It does not matter how the explorer is set to view the info it still bumbles through it all. Therefore a good answer is the one Tamay provided, have it in smaller groups better organised.

[cybernard]

You need a defragger that can handle the MFT table like perfectdisk by raxco. Dividing, 10m files into folders and subfolders might help, but the MFT keeps records of all files.

[phuclv]

@Psycogeek it depends on how much space is left on the MFT after storing the file's metatdata, but typically Files smaller than approximately 900 bytes are stored within the directory entry at the MFT technet.microsoft.com/en-us/library/cc781134(WS.10).aspx