SMB2 traffic crashes network?

Question

We've been having significant network slowdown issues over the past few weeks, primarily on a Friday morning. We run Windows 7 client machines, with Windows Server 2008 R2 servers.

What generally happens is the network starts to slow down massively at 08:55 and resumes normal speeds at around 09:20

This affects everything on the network from logging on, resetting passwords, opening programs and files etc. On my client machine, Physical Memory usage remains at around 40% (normal) and CPU usage hovers around 0-10% idle.

The servers show memory usage spikes massively and remains quite intense during the times mentioned above.

I have taken several wireshark captures, both during the slowdown and when the network operates fine.

One of the main things I noticed is the increase in SMB2 entries in the wireshark log during the slowdown.

Record Time         Source          Destination     Protocol Length Info
382    3.976460000  10.47.35.11     10.47.32.3      SMB2     362    Create Request File: pcross\My Documents
413    4.525047000  10.47.35.11     10.47.32.3      SMB2     146    Close Request File: pcross\My Documents
441    5.235927000  10.47.32.3      10.47.35.11     SMB2     298    Create Response File: pcross\My Documents\Downloads
442    5.236199000  10.47.35.11     10.47.32.3      SMB2     260    Find Request File: pcross\My Documents\Downloads SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern: *;Find Request File: pcross\My Documents\Downloads SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern: *
573    6.327634000  10.47.35.11     10.47.32.3      SMB2     146    Close Request File: pcross\My Documents\Downloads
703    7.664186000  10.47.35.11     10.47.32.3      SMB2     394    Create Request File: pcross\My Documents\Downloads\WestlandsProspectus\P24 __ P21.pdf

These are some of the SMB2 records from a list of a couple of hundred which original from my computer with a destination of the fileserver.

One of the interesting things to note is the last entry in the examples above is for a PDF file. That file was not open anywhere on my computer, or on anyone elses. No folders with the files in were open either.

When I took another capture when the network was running fine, there were hardly any SMB2 entries, and the ones that were displayed were mainly from Wireshark.

We currently have around 800 computers, 90 Macs and 200 Laptops and Netbooks. Our concern is if this traffic is happening on my computer, is it happening on other computers, and if so, would those computers be adding to the slow network issues?

Again, this only happens during certain times. We're pretty sure its not the our antivirus. Is there anything to narrow down whats initializing this SMB traffic during the particular times?

Or if anyone has any extra advice, or links to resources it would be appreciate.

Edit After looking at wireshark logs on a couple of other computers, there is a definate increase in SMB traffic for adobe photoshop files on the server from my computer. It only seems to be scanning for photoshop files and related files (such as settings etc). I have CS2 ( :( ) on my computer, but the other guys have CS6, and some computers dont even have photoshop and still getting bogged down.

Solution 1

Were you using the Windows quota-ing system (File Server Resource Manager)? Our network is having similar issues and i'm getting wireshark logs similar to what you posted.

Solution 2

You think it's originating from your PC? Have you tried unplugging your Ethernet cord and seeing if it fixes the network wide issue?

I think you may be taking a too narrow view on a network wide issue by only looking at individual Wireshark logs, have you tried reviewing your switch and router logs and seeing if there are any errors?

To find the source of SMB traffic that is suspected to cause the issue I would run a netstat -a and look for what program is using TCP port 445 (Wikipedia also says UDP ports 137, 138 & TCP ports 137, 139) I would do this on both your workstation and on the file server. I would also set up some network related performance monitors on the file server to see if that is spiking during this time as well.

I don't think it would be safe to settle on SMB just yet without seeing the same sort of traffic on multiple workstations. The fact that it is happening at a specific time makes it seem like there is a scheduled task , program, or backup running at that time. WSUS settings and Windows Update GPOs have caused this issue for me in the past, I would double-check those.

It really sounds like the best solution would be to set up some sort of SNMP monitoring / NMS on all of the workstations / servers and their NICs. Quest Foglight and Solarwinds NPM, can do this. After monitoring SNMP traffic you would be able to see what interfaces are having high utilization during those trouble times. Buyer beware, this can be expensive. Quest Foglight will let you monitor up to 200 interfaces, so that may be enough for a good sample.

Comments

[Zoredache]

Do you have roaming profiles/redirected folders setup on your network? Are you using offline files on those tons of laptops? Are you just looking at the initial sync when you login?

[Phil Cross]

We do have roaming profiles and redirected folders, all staff and students work and profiles are kept on the server, staff laptops sync up with the network when they plug in, however my computer (and the other network admins) save work both to the network and the C drive. I could understand it if it was a sync issue, but it would appear that its all photoshop files being requested from my computer rather than anything else. This happens on the other network admins computers, just photoshop files being requested.

[Phil Cross]

thanks for your reply. We're open to other possibilities other than SMB traffic. We've taken logs from a few other PCs (Network Admins) and it would appear their getting the same traffic on their logs during the slowdown time. I've edited my original post to mention that the only SMB traffic which increases during the slowdown is files related to adobe and photoshop (which we've upgraded to CS6 when the slowdown happened). We've looked at scheduled tasks, but unfortunately not found any! We've got a consultant coming on friday to have a look see. Hopefully they can shed some light!

[Brett Larson]

Make sure to post back when you find out!

[Phil Cross]

Turns out it was our staff quota-ing system! It was doing audits of our system to notify us of staff and students exceeding the quota.