Max (practical) files per folder on a Windows file share?

Question

I support an internal application that stores files on a Windows Server 2003 SP2 file share. Because of how it is currently configured to store files, one folder has ~116,000 files in it (another has ~65,000, and the other folders have less, but still several thousand in each). The application has become very slow to write files to.

The file layout is configurable to an extant, so I'm trying to come up with a better plan. Does anyone have any experience on how many items per folder SMB can handle before it starts to become unusable? In this case, it's been slow for quite a while, but it didn't start become unberable until the folder exceeded 100,000 files.

Solution 1

It's more dependent on bandwidth and latency (especially latency) than it is on the number of files and scaling in the algorithms being used to enumerate the directory. There is no "magic number", I guess, is what I'm saying.

The SMB protocol is hideous for requiring lots and lots of round trips. That number of files, with double the latency, would be many times more than twice as slow, for example.

You've done the benchmarking by accident for your LAN, your network infrastructure's latency, and your server computer's IO subsystem latency. You've obviously found a "magic number". I'd get that directory pared down until performance gets better. There's no other way!

Solution 2

Evan's right, there is no magic number. It depends on the app and the server. Upgrading to server 2008 will help, and is the first thing I'd do, as long as the client is vista or better as they use SMBv2. I've got shares that have 500,000 files that browse like crap but, since the users only use direct paths provided, work fine. On the same server I've got shares with 100,000 files that users have no issues with.

Comments

[Jeff Hardy]

The "client" is actually another server (also 2k3), but 2k8 isn't option for now (next year, maybe).

[raja]

While these will affect NTFS performance, they will not affect SMB performance. In addition I would not remove last access update on a server unless you have really good reason to (or want to live with the inability to see when a file was last modified)- However these are fine on a workstation (with the same caveat about time- which most users don't need)

[Spence]

Last accessed time isn't last modified time. Even so, you shouldn't really "trust" last modified time anyway. In NT unprivileged users can set any of the MAC times to whatever they want.