Why do I only get 300-400 Mbit/s on my 1000 Mbit/s network?

Question

My setup is like this:

Laptop <--> Router <--> Desktop

They all support gigabit Ethernet. I'm using the 'iperf' TCP/IP benchmark on the laptop and desktop, and I've never gotten more than 400 Mbit/s raw throughput. The laptop is an Intel Core 2 Duo 2 GHz running Windows XP, the desktop is an Intel Core 2 quad 2.4 GHz running Windows 7. What things can I look at to maximize the network speed between them? Software settings? A way to know if my cable is not up to specifications? How to know if the gigabit Ethernet controller in one of the devices just isn't capable of hitting the max speed?

Solution 1

1 Gbit/s is the theorectical maximum of this connection.

Many things can affect this.

Expect a hit in performance from your on-board NICs, as they will most likely be (based on device descriptions "Laptop" and "Desktop") host-based adapters that rely on the CPU to process the network traffic. Therefore speed will be affected by the CPU doing other things (like getting the data from the HDDs to the networking sub-system for example).

Also, depending on what the "router" is, it may be trying to process the data as it passes from LAN port to LAN port, so its processing speeds may also be affecting the data transfer speeds, regardless of how fast its 'ports' are.

If I were you my next test would be to get myself a 'just long enough' CAT6 crossover patch cable and wire the notebook directly to the PC (you may be able to get away with a straight-through cable, as many NIC's support auto-crossover these days).

That way you (at least) can pretty easily eliminate or identify the router (and/or existing wiring) as a bottleneck.

Solution 2

iperf uses a pretty small window size by default.

Increase the window or run it in UDP mode and it will easily saturate a 1 Gbit/s connection.

Solution 3

Jeff Atwood has some good analysis of why real-world throughput of a Gigabit network is closer to, in his calculations, 30MB/s. (big 'B', not little 'b')

...you definitely shouldn't expect the perfect scaling we achieved moving from 10baseT to 100baseT. Without any major tweaking, you'll get only a fraction of the tenfold bandwidth improvement you might expect

Solution 4

If everything is gigabit, you could try to enable jumbo frames. One of my friends did this with his home computer and noticed a huge increase in throughput. However, there are some drawbacks for low-latency applications like VOIP, as mentioned in the article.

Solution 5

Did you say you're using a router? is that an off the shelf consumer grade router (not switch)?

I'd say thats most likely your bottleneck. To verify this, Plug the laptop directly into the desktop. Set your Laptop and desktop to use a static IP address like 192.168.1.10 (laptop), 192.168.1.11 (desktop).

This should allow the laptop to talk directly to the desktop. Run iperf between them and report the result back here.

That should work as most modern network cards can auto-sense transmit/receive wires unlike the old days when you used to have to build a crossover cable.

If you see a marked increase then your router is the culprit. Some of them have switches on the back, others can actually route between ports. Many of these routers are terrible performers. If you do find this is the problem then if I were you I'd either upgrade the router (which is probably fine for internet use) or go and buy yourself a GigE switch and cascade that onto the router switch. Plug your laptop and desktop into the switch. That way, they will bypass the router to talk to each other.

Make sure you change your laptop & desktop back to DHCP when you've completed the experiment.

To give you an idea, you should get I would think 650Mbit/s and above. iperf doesn't use the harddrive unless you tell it to. So it's memory to memory or really testing the network.

On of the other bottlenecks is actually how quick your CPU can transfer data from the memory to the network card. Networks that are on PCI express seem to operate better. I think laptops are generally fairly cheaply built these days so if you don't see much above that then this could be the other bottleneck.

Also, consumer grade switches can often only cope switching 1GB between any two ports whereas high end managed switches can switch at wire speed on all ports simultaneously. That's part of the reason they cost so much more.

View more solutions

Comments

[user23307]

That "pcattcp" tool he is using is a piece of crap. My 900mhz linux netbook(which is probably as fast as his machine was in 2005) does 300MB/s over loopback.

[davr]

Didn't think of that, I'll try that.

[davr]

That's why I mentioned 'iperf', but I guess it's not that common of a tool. Basically it tests network speed directly, without reading/writing to the Hard Drive, so that's not the bottleneck there. I do realize that HDD can be a bottleneck in real world usage though.

[davr]

Yeah, iperf gives noticably better performance over loopback than pcattcp does. I'll stick to iperf for testing for now unless I hear of somethign else that's better.

[eleven81]

See en.wikipedia.org/wiki/… for two possible pinouts for making a gigabit crossover cable.

[Ƭᴇcʜιᴇ007]

Good info for sure. In this case if the user's not used to making a cable, and the goal is to eliminate possible slow-downs then he's probably better off using a factory-made cable. Nothing much worse then a loose crimp. ;)

[davr]

Also I already get higher than his 240mbit, and that article is 5 years old, so it's not as relevant anymore I think, seeing as how technology should have advanced greatly since then.

[davr]

Just FYI, with gigabit ethernet, crossover cables are no longer needed. All gigabit ethernet controllers are required to be able to 'autosense' the cable, so you can use a standard cable to connect to gigabit ethernet devices together. I'll try connecting my laptop straight to the desktop and see if there's a difference.

[davr]

Yeah, I'm not using a 'dumb' hub. I think most people these days will use intelligent switches or routers which handle connecting devices directly to each other, eliminating collisions. Also my test involves only two devices connected to a single router, so collisions wouldn't be a problem in my particular case.

[TheEruditeTroglodyte]

davr - excuse while I play devils' advocate for a moment . . . What prevents your two hosts from attempting to talk at the same time? Also, you didn't mention what kind of data is being transmitted . . . is it almost exclusively TCP or are there lots of data streaming (audio, video, etc)? Are both computers streaming data simultaneously or is there a more precisely demarcated client/server relationship?

[davr]

I'm using a special TCP/IP benchmarking program where only one computer sends, and the other listens. But even so, Ethernet is full duplex...if you have only two machines, they can both send and receive full bandwidth to each other at the same time.

[hookenz]

You won't saturate it with consumer grade hardware. The CPU generally gets limited by the PCI or PCI-Express bus. i.e. CPU can't move data from RAM to network card quick enough. Server grade hardware seems to do better. I know, I'm a programmer and I've been trying to do just this. Getting close but now limited by my own hardware.

[hookenz]

The increase should not that dramatic for most applications. All this will do is mean less TCP or UDP protocol overhead per packet. Also, for best results your switch/routers should also support jumbo frames. I doubt his would.

[hookenz]

Well, considering that the loopback interface is virtual (nothing even gets transfered to the network card) you'd expect much better performance (basically it's then process to process and no HDD touched). I'm getting 8.09 Gbits/s (1035Mbytes/s) over the loopback interface.

[davr]

Thanks for the info, please update your answer if you find any settings which affect the speed on windows

[rayking]

worked for me, was banging my head for an hour trying to figure out why Windows iperf gives me 250mbit/sec and Ubuntu booted on the same machine gives me 925mbit/sec. turns out iperf on windows defaults to 8k tcp window size and to 88k window size on linux.

[Nowak Grzegorz]

@Omry, iperf on windows is a dog from several perspectives. I have mostly given up and only run iperf under linux

[user1696603]

this makes me wonder if iperf isn't the only part of Windows that defaults to a smaller packet size than Linux, see superuser.com/a/95587/16966

[ebrahim.mr]

@davr could you have any success?