Extremely slow network transfers in one direction

21,511

I have seen this before, caused by a duplex mismatch. The spec for Gigabit ethernet requires both sides to use autonegotiate, so make sure that the NIC's on the server and clients are set to auto negotiate, and the same for your switch ports. Setting it to 1000/Full explicitly won't do it.

I see from one of your outputs:

negotiated 1000baseT-HD flow-control, link ok

That looks like half-duplex. What happens when both sides aren't set to auto is that the autonegotiate phase between the two connections correctly detects the 1000mbps speed, but can't discover the duplex mode so it uses the default which is half duplex. So you have 2 devices trying to comunicate, one in full duplex and the other in half duplex. The net result is that the traffic from one side causes a lot of packet errors forcing the packets to be resent and killing the transfer rate in that direction.

I fixed a problem between a server and a NAS box this way, one card has been set to auto and the other explicitly to 1000/Full.

Share:
21,511

Related videos on Youtube

4026
Author by

4026

Updated on September 18, 2022

Comments

  • 4026
    4026 over 1 year

    File transfers from my home server (running Ubuntu 12.04 Server) to any other PC on my LAN (all running Windows 7) are extremely slow, but file transfers in the other direction (uploading to the server) are fast.

    I observed this with both Samba and SFTP transfers, so decided to verify it with iperf, and got the same results. The output on the server looked like this:

    iperf -s
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 85.3 KByte (default)
    ------------------------------------------------------------
    [  4] local 192.168.1.95 port 5001 connected with 192.168.1.82 port 1309
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-10.0 sec   416 MBytes   349 Mbits/sec
    ------------------------------------------------------------
    Client connecting to 192.168.1.82, TCP port 5001
    TCP window size: 46.1 KByte (default)
    ------------------------------------------------------------
    [  4] local 192.168.1.95 port 59704 connected with 192.168.1.82 port 5001
    [  4]  0.0-12.2 sec  1.50 MBytes  1.03 Mbits/sec
    

    The client (My Win 7 PC) produced this output for the same run:

    iperf.exe -c 192.168.1.95 -r
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 64.0 KByte (default)
    ------------------------------------------------------------
    ------------------------------------------------------------
    Client connecting to 192.168.1.95, TCP port 5001
    TCP window size: 64.0 KByte (default)
    ------------------------------------------------------------
    [  4] local 192.168.1.82 port 1309 connected with 192.168.1.95 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-10.0 sec   416 MBytes   349 Mbits/sec
    [  4] local 192.168.1.82 port 5001 connected with 192.168.1.95 port 59704
    [  4]  0.0-12.2 sec  1.50 MBytes  1.03 Mbits/sec
    

    Similar results are observed from other clients. The computers are all connected to the same Gigabit wired switch. Exchanging the ports and cables between them gives the same result.

    Running the transfers to and from a ramdisk set up on the server doesn't alter the transfer speeds, either; I don't believe the hard disk is causing the problem.

    I've done my best to google around similar problems, but I haven't found any relevant solutions yet. Does anyone have any ideas?


    The following are some dumps of relevant debug commands, in case they're of any use to people more knowledgeable than me.

    ethtool output for the ethernet card on the server:

    ethtool eth0
    Settings for eth0:
            Supported ports: [ TP MII ]
            Supported link modes:   10baseT/Half 10baseT/Full
                                    100baseT/Half 100baseT/Full
                                    1000baseT/Half 1000baseT/Full
            Supported pause frame use: No
            Supports auto-negotiation: Yes
            Advertised link modes:  10baseT/Half 10baseT/Full
                                    100baseT/Half 100baseT/Full
                                    1000baseT/Half 1000baseT/Full
            Advertised pause frame use: Symmetric Receive-only
            Advertised auto-negotiation: Yes
            Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                                 100baseT/Half 100baseT/Full
                                                 1000baseT/Half 1000baseT/Full
            Link partner advertised pause frame use: Symmetric Receive-only
            Link partner advertised auto-negotiation: Yes
            Speed: 1000Mb/s
            Duplex: Full
            Port: MII
            PHYAD: 0
            Transceiver: internal
            Auto-negotiation: on
            Supports Wake-on: pumbg
            Wake-on: g
            Current message level: 0x00000033 (51)
                                   drv probe ifdown ifup
            Link detected: yes
    

    Mii-tool output:

    sudo mii-tool
    eth0: negotiated 1000baseT-HD flow-control, link ok
    

    List of hardware returned by lspci:

    sudo lspci
    00:00.0 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
    00:00.1 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
    00:00.2 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
    00:00.3 Host bridge: VIA Technologies, Inc. PT890 Host Bridge
    00:00.4 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
    00:00.7 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
    00:01.0 PCI bridge: VIA Technologies, Inc. VT8237/VX700 PCI Bridge
    00:08.0 Network controller: Broadcom Corporation BCM4306 802.11b/g Wireless LAN Controller (rev 03)
    00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet (rev 10)
    00:0b.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet (rev 10)
    00:0f.0 IDE interface: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80)
    00:0f.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
    00:10.0 USB controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
    00:10.1 USB controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
    00:10.2 USB controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
    00:10.3 USB controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
    00:10.4 USB controller: VIA Technologies, Inc. USB 2.0 (rev 86)
    00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800/K8T890 South]
    00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
    01:00.0 VGA compatible controller: VIA Technologies, Inc. CN700/P4M800 Pro/P4M800 CE/VN800 Graphics [S3 UniChrome Pro] (rev 01)
    

    List of loaded drivers returned by lsmod:

    sudo lsmod
    Module                  Size  Used by
    usb_storage            39646  0
    vesafb                 13516  1
    ppdev                  12849  0
    arc4                   12473  2
    snd_via82xx            24718  0
    gameport               15060  1 snd_via82xx
    snd_ac97_codec        110213  1 snd_via82xx
    b43                   342801  0
    ac97_bus               12642  1 snd_ac97_codec
    snd_pcm                80916  2 snd_via82xx,snd_ac97_codec
    snd_timer              28931  1 snd_pcm
    snd_page_alloc         14108  2 snd_via82xx,snd_pcm
    snd_mpu401_uart        13865  1 snd_via82xx
    snd_rawmidi            25424  1 snd_mpu401_uart
    snd_seq_device         14172  1 snd_rawmidi
    snd                    62218  7 snd_via82xx,snd_ac97_codec,snd_pcm,snd_timer,snd_mpu401_uart,snd_rawmidi,snd_seq_device
    mac80211              436493  1 b43
    soundcore              14635  1 snd
    serio_raw              13027  0
    i2c_viapro             12969  0
    cfg80211              178877  2 b43,mac80211
    bcma                   25651  1 b43
    parport_pc             32114  1
    shpchp                 32265  0
    mac_hid                13077  0
    via_cputemp            13031  0
    hwmon_vid              12723  1 via_cputemp
    lp                     17455  0
    parport                40930  3 ppdev,parport_pc,lp
    pata_via               13428  0
    ssb                    50691  1 b43
    r8169                  56396  0
    sata_via               13495  2
    

    Driver info for the Gigabit card:

    sudo modinfo r8169
    filename:       /lib/modules/3.2.0-51-generic-pae/kernel/drivers/net/ethernet/realtek/r8169.ko
    version:        6.017.00-NAPI
    license:        GPL
    description:    RealTek RTL-8169 Gigabit Ethernet driver
    author:         Realtek and the Linux r8169 crew <[email protected]>
    srcversion:     231149B837AF2B63F7C0271
    alias:          pci:v00001186d00004300sv00001186sd00004C00bc*sc*i*
    alias:          pci:v000010ECd00008169sv*sd*bc*sc*i*
    alias:          pci:v000010ECd00008167sv*sd*bc*sc*i*
    depends:
    vermagic:       3.2.0-51-generic-pae SMP mod_unload modversions 686
    parm:           speed:force phy operation. Deprecated by ethtool (8). (array of int)
    parm:           duplex:force phy operation. Deprecated by ethtool (8). (array of int)
    parm:           autoneg:force phy operation. Deprecated by ethtool (8). (array of int)
    parm:           rx_copybreak:Copy breakpoint for copy-only-tiny-frames (int)
    parm:           use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
    parm:           debug:Debug verbosity level (0=none, ..., 16=all) (int)
    

    ...and uname -a for good measure:

    uname -a
    Linux hotblack 3.2.0-51-generic-pae #77-Ubuntu SMP Wed Jul 24 20:40:32 UTC 2013 i686 i686 i386 GNU/Linux
    

    If anyone thinks any other debug output would be helpful to have, please don't hesitate to ask.


    Update

    For want of a better way to test a different driver, I've also tried dropping back to an older kernel on boot, which returned r8169 to v2.3LK-NAPI, but the problem remained the same.

    I have, however, made an interesting discovery. If I use ethtool to force the card back to 100Mb...

    sudo ethtool -s eth0 speed 100 autoneg on duplex full
    sudo ethtool eth0
    Settings for eth0:
            Supported ports: [ TP ]
            Supported link modes:   10baseT/Half 10baseT/Full
                                    100baseT/Half 100baseT/Full
                                    1000baseT/Full
            Supported pause frame use: No
            Supports auto-negotiation: Yes
            Advertised link modes:  10baseT/Half 10baseT/Full
                                    100baseT/Half 100baseT/Full
            Advertised pause frame use: Symmetric Receive-only
            Advertised auto-negotiation: Yes
            Speed: 100Mb/s
            Duplex: Full
            Port: Twisted Pair
            PHYAD: 0
            Transceiver: internal
            Auto-negotiation: on
            MDI-X: Unknown
            Supports Wake-on: pumbg
            Wake-on: g
            Current message level: 0x00000033 (51)
                                   drv probe ifdown ifup
            Link detected: yes
    

    ...then iperf shows the connection working fine in both directions:

    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 85.3 KByte (default)
    ------------------------------------------------------------
    [  4] local 192.168.1.95 port 5001 connected with 192.168.1.82 port 2437
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-10.0 sec   105 MBytes  87.7 Mbits/sec
    ------------------------------------------------------------
    Client connecting to 192.168.1.82, TCP port 5001
    TCP window size: 83.9 KByte (default)
    ------------------------------------------------------------
    [  4] local 192.168.1.95 port 40844 connected with 192.168.1.82 port 5001
    [  4]  0.0-10.0 sec   113 MBytes  94.4 Mbits/sec
    

    I'm not certain exactly what to conclude from this, but it is at least further evidence that the upload speed isn't being constrained by the hardware...

    • RobotHumans
      RobotHumans almost 11 years
      Have you played around with downloading from a ram backed filesystem? I'm guessing it's choking on disk seek/reads.
    • 4026
      4026 almost 11 years
      Interesting. I presumed iperf wouldn't be constrained by the hard-disk. Am I wrong about that?
    • 4026
      4026 almost 11 years
      Yep, just tried it and confirmed: files transferred to and from a ramdisk on the server are just as fast transferring to the server, and just as slow downloading from it again. The problem doesn't seem to be with the disk.
    • RobotHumans
      RobotHumans almost 11 years
      You might want to edit that somewhat relevant piece of data in to your question proper. It's good to know.
    • RobotHumans
      RobotHumans almost 11 years
      I dropped a link to your question in chat. It seems like a really interesting bug (not in the sense of off-topic-ness, but in the sense of needing to be fixed or worked around). UF has a couple questions on this and it seems to be related to the driver for your device, not Ubuntu itself.
    • 4026
      4026 almost 11 years
      Thanks. I've edited in a note about the ramdisk test. I'll keep looking around and see if I can find some other drivers to try, to see if that makes any difference...
    • ceinmart
      ceinmart over 6 years
      Perhaps this is very old thread , I got into the same problem, but with vmware esxi , what is curious, is exactly with the same NIC model , Realtek 8169 ... not found any solution yet.. set set speed to 100MB just make all worst