speedtest-cli very slow although network speed is fine

Question

I'm encountering a strange problem here. On my local homeserver(Debian 9.9), speedtest-cli and it's python pendant are freaking slow. Since I use it to monitor my ISP connection stability, this is a problem.

My first thought was my server or speedtest in general have a problem, but here starts the fun part.

Speedtest from my PC(Windows 10), directly connected to the router:

fast.com 370 Mbit/s 
speedtest.net(server 15819) 242 Mbit/s down, 50,78 Mbit/s up

Speedtest from my notebook(Arch Linux), directly connected to the router:

fast.com 310 Mbit/s
speedtest.net(server 15819) 240 Mbit/s down, 50,8 Mbit/s up.

So far so good, but regarding the Debian server, directly connected to the router:

speedtest-cli(server 15819) 3,88 Mbit/s down, 3,69Mbit/s up

So maybe the server is broken... But no, the connection speed is just fine.

wget --output-document=/dev/null https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.9.0-amd64-netinst.iso
leads to a download speed of about 200 Mbit/s.

Speedtest directly on the router too results in about 250 Mbit/s.

I already tried pruging speedtest-cli, rebooting the server and so on.

Any idea what is going on here? How is it possible, the commandline spedtest fails this spectacular, while the rest of the network setup is just fine?

Solution 1

I was messing with this - and have speedtest-cli and speedtest++ installed. (Unfortunately, my GUI PCs are behind a homeplug connection, but trust me when I say the rest). Its probably down to the protocol that's being used.

Here is the result from the version in the repository from Ubuntu 16.04

[email protected]_router:/etc$ speedtest-cli
Retrieving speedtest.net configuration...
Testing from Singtel Fiber (XXX.XXX.XXX.XXX)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Singtel (Singapore) [3.91 km]: 230.77 ms
Testing download speed................................................................................
Download: 648.47 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.15 Mbit/s

Interestingly the version in git has better results for uploads, though still not consistent with the speedtest++ and ookla's speed test cli results

[email protected]_router:~/speedtestgittest$ ./speedtest_cli_git.py
Retrieving speedtest.net configuration...
Testing from Singtel Fiber (xxx.xxx.xxx.xxx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by MyRepublic (Singapore) [3.91 km]: 233.882 ms
Testing download speed................................................................................
Download: 607.12 Mbit/s
Testing upload speed................................................................................................
Upload: 59.29 Mbit/s

As per the speedtest-CLI github page

There is the potential for this tool to report results inconsistent with Speedtest.net. There are several concepts to be aware of that factor into the potential inconsistency:

Speedtest.net has migrated to using pure socket tests instead of HTTP based tests

Ookla seem aware of that too

Performance - all of the open source versions use an ancient version of our test engine that predates our Flash test. Yeah it's that old. This is not ideal for fast connections. We've also seen popular versions that are impacted by DNS timings and all sorts of other methodology issues. In addition, most are written in high level languages, which typically have issues hitting high bandwidth levels.

Though I suspect some excessively deep performance profiling is needed to determine whether its the protocol or the language.

I've been using speedtest++ until about 5 minutes after I first wrote this answer, and the results are loads better.

[email protected]_router:~$ SpeedTest
Speedtest ++
SpeedTest++ version 1.14
Speedtest.net command line interface
Info: https://github.com/taganaka/SpeedTest
Author: Francesco Laurita <[email protected]>
IP: XXX.XXX.XXX.XXX ( Singtel Fiber ) Location: [<redacted>, <redacted>]
Finding fastest server... 9575 Servers online
............
Server: Singapore speedtest.singnet.com.sg:8080 by Singtel (3.91278 km from you): 1 ms
Ping: 1 ms.
Jitter: 0 ms.
Determine line type (2) ........................
Fiber / Lan line type detected: profile selected fiber
Testing download speed (32) ................................................................................................................................................................................................................................................................................................
Download: 989.61 Mbit/s
Testing upload speed (12) .................................................................................................................................................................................................................................................................................................................................................................................................................................................
Upload: 998.73 Mbit/s

Contrast this with the official client (and their blog suggests someone has written an app to monitor historical speed for you), and does CSV and TSV output

[email protected]_router:~$ speedtest
   Speedtest by Ookla
     Server: Singtel - Singapore (id = 13623)
        ISP: Singtel Fiber
    Latency:     1.91 ms   (0.05 ms jitter)
   Download:   932.83 Mbps (data used: 479.6 MB)
     Upload:   941.23 Mbps (data used: 424.7 MB)
Packet Loss:     0.0%
 Result URL: <redacted>

As you can tell... the speed is substancially different.

I recall the gui test being comparable to the cli test when I messed with it.

Speedtest ++ and the official clients use a raw sockets/custom protocol that works better than the older protocol at finding line speed.

Solution 2

The devs seem to be already beaten up by such requests and so post these lines in the readme on github:

Different versions of Python will execute certain parts of the code faster than others. CPU and Memory capacity and speed will play a large part in inconsistency between Speedtest.net and even other machines on the same network. Issues relating to inconsistencies will be closed as wontfix and without additional reason or context.

So the solution here is to use a non-Python based benchmarks on a stable and strong RAM/CPU boxes. Googling showed me one of the solutions: a c++ based cli speed test on https://github.com/taganaka/SpeedTest

Solution 3

speedtest-cli is completely broken, reports wildly innacurate results, and the developer 'sivel' on GitHub has resolutely refused to address any of the dozens of issues filed on GitHub about this issue, immediately closing and locking these issues without comment.

Comments

[Michael Frank]

Which version of speedtest-cli are you using? As I understand it, there is one that doesn't work, and one that does.

[r4ptor]

1.0.0 Cloning the github repository and running it directly results in this behavior too. As one can see in this grafana graph, the testing worked fine until 2019-05-03. Nothing really has changed since then except a bandwidth upgrade from the ISP.

[tvdo]

Could be the TCP parameters on that machine...

[Seandex]

The latest version of speedtest-cli has been working just fine. Avoid to install it from apt.