How to (really) disable NCQ in Linux

linux kernel drivers sata

22,170

Solution 1

Thanks to @frostschutz, I could measure the write performance in Linux without NCQ feature. The kernel boot parameter libata.force=noncq disabled NCQ completely.

Regarding my Seagate 6TB write performance problem, there was no change in speed. Linux still reaches 180 MiB/s.

But then I had another idea:
The Linux driver does not use transfers of 32 MiB chunks. The kernel buffer is much smaller, especially if NCQ with 32 queues is enabled (32 queues * 32 MiB => 1 GiB AHCI buffer).

So I tested my SATA controller with 256 KiB transfers and voilà, it's possible to reach 185 MiB/s.

So I guess the Seagate ST6000AS0002 firmware is not capable of handling big ATA burst transfers. The ATA standard allows up to 65.536 logical blocks, which equals 32 MiB.

SMR - Shingled Magnetic Recording

Another possibility for the bad write performance could be the shingled magnetic recording technique, which is used by Seagate in these archive devices. Obviously, I triggered a rare effect with my FPGA implementation.

Solution 2

Setting the queue depth to 1 (/sys/block/sd*/device/queue_depth) disables NCQ, it is not necessary to use the kernel parameter libata.force=noncq (which can only be set at boot).

commit 360f654e7cda850034f3f6252a7a7cff3fa77356

 Date:   Sat Sep 30 19:45:00 2006 +0900

 [PATCH] libata: turn off NCQ if queue depth is adjusted to 1
 
 Turn off NCQ if queue depth is adjusted to 1.

22,170

Paebbels

Patrick Lehmann studied computer science at Technische Universität Dresden, Germany. His professional career already found its foundation here when he was already teaching as a tutor computer engineering and computer architecture Later on, he specialized in digital design, FPGA technology, and high-speed communication solutions like Serial-ATA, Gigabit Ethernet, or PCI Express. He was sharing his gained knowledge in labs, research articles, and on social platforms. The focus of his research work is was on in-memory database systems, the Serial-ATA protocol implementation, and embedding FPGAs into a Cloud infrastructure. Since 2017, Patrick Lehmann is working for PLC2 GmbH as a instructor in the topics of VHDL, OSVVM, FPGA technology as well as high-speed communication. As a consultant and “fire fighter” he helps critical projects to bring on track. In cooperation with PLC2 Design GmbH, he is a senior system architect for FPGA-based solutions, team leader in FPGA design projects, and technical project advisor. Mr. Lehmann is one of the developers and maintainers of the PoC-Library, a platform and vendor independent open source IP core library. He is a contributor to the GHDL project as well, a free VHDL simulator and synthesis tool. In 2016, he started an initiative called "Open Source VHDL Group", whose aim is a free collection of VHDL packages. As a maintainer of more than 40 open source repositories and owner of several GitHub organizations, he is one of circa 20 people worldwide driving the open source community for EDA related tooling. Furthermore, Mr. Lehmann is very active in the IEEE P1076 "VHDL Analysis and Standardization Group" since 2014. He detailed and wrote major parts of the language changes for the current VHDL-2019 revision. In 2017, he became an IEEE Standards Association member and was announced vice-chair of the IEEE P1076 working group. He managed to register VHDL as an open source pilot project. In cooperation with IEEE SA, the working group successfully publishes all VHDL language packages as the first open-source standard in the history of IEEE. In 2021, the IEEE P1076 working group for VHDL-2025 was approved. One of its main goals is to create a new collaborative and open-source publishing flow at IEEE SA, so the whole standard might become open source.

Updated on September 18, 2022

Comments

Paebbels over 1 year
I implemented my own Serial-ATA Host-Bus-Adapter (HBA) in VHDL and programmed it onto a FPGA. A FPGA is chip which can be programmed with any digital circuit. It's also equipped with serial transceivers to generate high speed signals for SATA or PCIe.

This SATA controller supports SATA 6 Gb/s line rates and uses ATA-8 DMA-IN/OUT commands to transfer data in up to 32 MiB chunks to and from the device. The design is proven to work at maximum speed (e.g. Samsung SSD 840 Pro -> over 550 MiB/s).

After some tests with several SSD and HDD devices, I bought a new Seagate 6 TB Archive HDD (ST6000AS0002). This HDD reaches up to 190 MiB/s read performance, but only 30 to 40 MiB/s write performance!

So I dug deeper and measured the transmitted frames (yes that's possible with a FPGA design). As far as I can tell, the Seagate HDD is ready to receive the first 32 MiB of a transfer in one piece. This transfer happens at maximum line speed of 580 MiB/s. After that, the HDD stalls the remaining bytes for over 800 ms! Then the HDD is ready to receive the next 32 MiB and stalls again for 800 ms. All in all an 1 GiB transfer needs over 30 seconds, which equals to circa 35 MiB/s.

I assume that this HDD has a 32 MiB write cache, which is flushed in between the burst cycles. Data transfers with less than 32 MiB don't show this behavior.

My controller uses DMA-IN and DMA-OUT command to transfer data. I'm not using the QUEUED-DMA-IN and QUEUED-DMA-OUT command, which are used by NCQ capable AHCI controllers. Inplementing AHCI and NCQ on a FPGA platform is very complex and not needed by my application layer.

I would like to reproduce this scenario on my Linux PC, but the Linux AHCI driver has NCQ enabled by default. I need to disable NCQ, so I found this website describing how to disable NCQ, but it doesn't work.

The Linux PC still reaches 190 MiB/s write performance.
```
> dd if=/dev/zero of=/dev/sdb bs=32M count=32
1073741824 bytes (1.1 GB) copied, 5.46148 s, 197 MB/s
```
I think there is a fault in the article from above: Reducing the NCQ queue depth to 1 does not disable NCQ. It just allows the OS the use only one queue. It can still use QUEUED-DMA-** commands for the transfer. I need to realy disable NCQ so the driver issues DMA-IN/OUT commands to the device.

So here are my questions:
1. How can I disable NCQ?
2. If NCQ queue depth = 1, is Linux's AHCI driver using QUEUED-DMA-** or DMA-** commands?
3. How can I check if NCQ is disable, because changing /sys/block/sdX/device/queue_depth is not reported in dmesg?
CR. over 6 years

In my experience, disabling NCQ gives a large boost in performance. I have tried this on desktop systems, servers, you name it. Even using 100% "server" high-performance hardware that you would think would benefit from NCQ. Nope, it's all worst than just disabling it. IMHO NCQ is one of the worst things to happen to hard-drives. I have never once seen it provide a benefit under any circumstances, be it dedicated RAID card or on-board chipset.
Paebbels over 6 years

Have you done burst operations or random access? NCQ has no impact on burst operations but improves random access.
Paebbels over 6 years

Sorry, but you haven't answered my question. Another question is, what hardware have you used a simple consumer mainboard or a workstation/server mainboard or dedicated RAID controller. Many implementations do not support as many outstanding requests as designed by NCQ / AHCI.
xhienne over 3 years

Better than redirecting the reader to a patch, tell us how to do this: echo 1 > /sys/block/sdX/device/queue_depth
Simon over 3 years

It was supposed to be a comment on another answer, but I'm not allowed to make comments. The patch is not a "redirect", it's a citation.
xhienne over 3 years

Comments masqueraded as answers are not allowed. But this is actually a useful answer that needs more upvotes, especially now that you have added how to disable NCQ on a live system. Thank you.