Having an issue trying to get Gigabit speed across my network (Ubuntu Server)

Question

I've just started looking into the network speeds at my office, the entire network is setup to be "Gigabit". This includes Gb switches, Gb Network cards and Cat 5e cabling.

I'm not expecting the full speed, I just want more than ~90 Mb/s.

I've been running some tests with iperf the linux tools and checking the hardware with ethtool. I have 3 servers and when doing my checks/test I discovered that the two backup servers can access each other at around 450 Mb/s but when using either one of them to connect and test the main server, I only get the 90Mb/s even though ethtool shows the networking card running at 1000/Full.

The only difference between all the server/networking cards is the "Port" which ethtool shows. On the two backup servers the "Port" is shown as MII yet on the other it's shown as "Twisted Pair".

When using ethtool -s to manually set the "Port" to MII on the main server, it looses all connectivity and does not show "Speed" or "Duplex".

Anyway, Am i doing something wrong? Is there a specific reason my main server cannot use Gb when there appears to be no difference except the "Port"?

Solution 1

Let's start with the simplest test - swap out the patch cable attaching the slow server with one of the ones on a fast server.

If it fixes things, then perhaps you have a bad cable?

The next thing that I would look at is the configuration on the switch port. Make sure that it's the same on the slow server's port as it is on the fast. I'd also switch the slow server to a fast servers port and test, as it might be a bad port.

Next I'd look at the cards - are they the same make and model? are they configured the same? Try swapping one from a fast server to the slow server. You might have a bad card, or out dated/poor drivers for one of the cards.

Solution 2

As @Luis pointed out, 'b' = bit, and 'B' = byte. iperf helps us out by reporting times in Mbits/sec:

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.01 GBytes   865 Mbits/sec

Does ifconfig eth0 show any errors on either host? From your reports, it seems as if either the CPU is overloaded on one of the two iperf machines, or the switch (please tell me it's not multiple switches) between them is overloaded, or you have a 100Mbit constriction somewhere between your hosts. I've found people cascaded off of a small 100Mbit switch one too many times, so it's good to know where your cable runs start and finish. Are you using a smart switch? Does it show that your ports are active at gigabit speeds?

Once you figure out your problem, you might want to enable jumbo frames on your hosts in order to get even better throughput when you're making bulk data transfers, provided you have a reasonably-modern switch.

Comments

[user94217]

Right guys, BIOS was no luck. Didn't have a compatible NIC handy. Could not upgrade the firmware. As a last chance effort I decided to use the Ubuntu 11.10 desktop live cd and test iperf on it. I couldn't believe it! 490Mb/s!!!! So i'm currently upgrading the kernel but it's not the latest. I think a re-istall will have to be done if this doesn't work. So I can get Gb using Ubuntu and the current hardware which means between 10.10 and 11.10 this issue has been fixed. As it stands it might be a while before I'm running Gb but I know it's possible now. Thanks you all!!!

[hookenz]

What switches are you using?

[hookenz]

Ahh, tp link. If you connect the two backup servers together (minus the switch) what's the throughput like? Some switches can't do wire speed switching across all ports simultaneously. TP Link is somewhat of a home brand. I'm not saying that's the issue but you need to eliminate that as the issue.

[user94217]

Thanks for the swift reply dude. I used the measurements right in my case. Gigabit = 1000 Mb not MB. On 10/100 i get ~ 90 Mb/s with Gigabit I should get 300Mb/s+. 1 gigabit = 128 megabytes so really anything around 50MB/s would be awesome. I'll check out the MTU and see what the results of your listed commands show. :) Thanks again!

[user94217]

Sorry for my poor explanation. I'm after the bits not the bytes. Its a Gigabit connection i'm after. I get ~90 Megabits per second from the main server and ~400 Megabits from the backup server to backup server. It's a single 48 port Gb switch TP Link, not sure of the model. It's only handling ~ 20 connections and only 4 are using the Gigabit. Every port is Gigabit also because I have a few which only have 2-4 Gb ports on. I've tested the network and i'm sure that the issue lies with the Main server itself.

[user94217]

Thanks dude. Replaced the cable for a brand new one, I always narrow it down as much as possible. I cannot compare cards as the backup servers are just "NAS" boxes we put together running Ubuntu. Cards in them are high-end desktop Gb. The server is decent HP not sure of the model. I've also tried multiple ports, even the ones the backup servers were using. I've tested on my PC and a laptop both with Gb and it's all good. I've tried manually setting it but it appears to AutoNeg to 1000/FD anyway. It's a weird one for me. Think i'll check the BIOS and then try another NIC just incase. Thanks.

[user94217]

Tomorrow I will be checking the BIOS options to see there is no restriction there. I'm also going to look at getting a compatable NIC to test it with as the two connections it uses are on-board. I'll have to check the activity lights tomorrow. I know the backup servers use Gb without any issue and it's the same switch and type of cable. I've looked into Jumbo Frames but I'm not trying to move data at super speeds I'm just looking to increase the speed to match the backup servers. I'm simply trying to get what I should be able to out of the equipment I have. Thanks.

[tsykoduk]

yeah. If the cards are different manufacture/models that very well could be a clue. Can you put one of the cards that's in your NAS box into the server? Everything is pointing to that being the delta. Good luck! ;)

[user94217]

The cards are different for 4 machines I have tested with Ethtool. The main server should have the best spec NIC's by a large margin yet that's the one i'm having issues with. The server only has PCI-x available so I think i'll have to buy a compatible NIC. This is a valuable production server and I want to ensure stability. The only difference between them is that the main shows Twisted Pair and the others show MII under the "Ports" section. Can these be incompatible? I can achieve Gb for all but the Twisted Paired NIC.

[Luis Bruno]

Same MTU everywhere. See the statistics using ethtool -S eth0 and eth1 on MAIN, and eth0 on BACKUP.

[Luis Bruno]

PS: are both the MAIN links 1Gbps?

[user94217]

Hey Luis, yes both network connections are 1 Gbps but the eth0 is only plugged into a 10/100 router, and i'm not bothered about that connection. I think my next step is going be to upgrade the Kernel of the OS and then the firmware of the NIC. If that doesn't work I think i'll just buy a PCI-x NIC. Thanks dude.

[Luis Bruno]

Yeah, just wanted to make sure you're not testing against the wrong IP address; perhaps I'm preaching to the choir here? :) A quick ifdown eth0 would solve that, if you have local console access.

[tsykoduk]

Honestly, I'd test one of the working cards in the primary server. Rule out hardware (and if they are the same OS revision, drivers). No need to buy anything for testing. If the test shows that the less expensive card works, and you are concerned about failure. PCI cards will work in PCIx slots, so a test seems reasonable.

[user94217]

I've had my run around's because of mismatching stuff before. I make sure to rule out my stupidity before I post online :) You've been a great help dude, I wouldn't have tried the stuff I did today without posting it here. I've commented above because I'm not sure how to let everyone know it's pretty much done now. Thanks Luis :)